Showing posts with label Data Guard Broker. Show all posts
Showing posts with label Data Guard Broker. Show all posts

May 13, 2024

ORA-01111: name for data file is unknown – rename to correct file

 

Recently I asked one of my junior colleague to add a datafile in a tablespace. The database had a physical standby database running. When she added the datafile , the parameter “standby_file_management” in standby database server was set to manual. So after addition of the datafile the file was created like “UNNAMED00006”

And since then the  dataguard synchronization was broken.

In alert log the event of datafile addition was showing like this

2024-04-05T05:48:11.121082-07:00

File #6 added to control file as 'UNNAMED00006' because

the parameter STANDBY_FILE_MANAGEMENT is set to MANUAL

The file should be manually created to continue.

PR00 (PID:5572): MRP0: Background Media Recovery terminated with error 1274

2024-04-05T05:48:11.183592-07:00

Errors in file E:\APP\diag\rdbms\edwprddr\edwprdint\trace\edwprdint_pr00_5572.trc:

ORA-01274: cannot add data file that was originally created as 'E:\APP\ORADATA\EDWPRD_INT\SYSTEM02.DBF'

PR00 (PID:5572): Managed Standby Recovery not using Real Time Apply

Recovery interrupted!

Recovery stopped due to failure in applying recovery marker (opcode 17.30).

 

By checking the dataguard broker , we found

DGMGRL> show configuration

 

Configuration - my_dg_config

 

  Protection Mode: MaxPerformance

  Members:

  edwprd_int - Primary database

    edwprddr   - Physical standby database

      Error: ORA-16810: multiple errors or warnings detected for the member

 

Fast-Start Failover:  Disabled

 

Configuration Status:

ERROR   (status updated 48 seconds ago)

 

 

Solution: To resolve this issue, we took following actions

First we needed to identify which file id was not created properly.

So login to primary database and look for file# 6 which we retrieved from alert log.

SQL>Select file#, name from v$datafile where file#=6;

     FILE# NAME

---------- -------------------------------------------------------

         6 E:\APP\ORADATA\EDWPRD_INT\SYSTEM02.DBF

 

So , now we will have to recreate the data file in proper location as it was not created in proper location because standby_file_management parameter set to manual.

Login to Standby database

SQL> SQL> alter database recover managed standby database cancel;

alter database recover managed standby database cancel

*

ERROR at line 1:

ORA-16136: Managed Standby Recovery not active

 

This command we used to cancel the MRP but , it failed as MRP process was not active .

 

Now create a datafile in proper location by referring current 'UNNAMED00006' file

 

SQL> alter database create datafile 'E:\APP\ORACLE\PRODUCT\19.3.0\DBHOME_1\DATABASE\UNNAMED00006' as 'E:\APP\ORADATA\EDWPRD_INT\SYSTEM02.dbf';

 

Database altered.

 

SQL>alter system set standby_file_management='AUTO';

 

Now start the MRP process, it should start applying all the archives and put the database back in sync

SQL> Alter database recover managed standby database disconnect from session;

Database altered.

 

 

 

Now after checking the dataguard broker , status showed success

DGMGRL> show configuration

 

Configuration - my_dg_config

 

  Protection Mode: MaxPerformance

  Members:

  edwprd_int - Primary database

    edwprddr   - Physical standby database

 

Fast-Start Failover:  Disabled

 

Configuration Status:

SUCCESS   (status updated 35 seconds ago)

 

 

April 10, 2023

Handy Oracle Data guard Broker Commands Part-II

Following command will validate the configuration of the specified database.

DGMGRL> validate database 'db_name';

DGMGRL> validate database proddr;

 

  Database Role:     Physical standby database

  Primary Database:  prod

 

  Ready for Switchover:  Yes

  Ready for Failover:    Yes (Primary Running)

 

  Flashback Database Status:

    prod  :  On

    proddr:  Off

 

  Managed by Clusterware:

    prod  :  NO

    proddr:  NO

    Validating static connect identifier for the primary database prod...

    The static connect identifier allows for a connection to database "prod".

 

DGMGRL>

 

 

Following command will display the current Fast-Start Failover configuration.

DGMGRL> show fast_start failover

 

Fast-Start Failover:  Disabled

 

  Protection Mode:    MaxPerformance

  Lag Limit:          30 seconds

 

  Threshold:          30 seconds

  Active Target:      (none)

  Potential Targets:  (none)

  Observer:           (none)

  Shutdown Primary:   TRUE

  Auto-reinstate:     TRUE

  Observer Reconnect: (none)

  Observer Override:  FALSE

 

Configurable Failover Conditions

  Health Conditions:

    Corrupted Controlfile          YES

    Corrupted Dictionary           YES

    Inaccessible Logfile            NO

    Stuck Archiver                  NO

    Datafile Write Errors          YES

 

  Oracle Error Conditions:

    (none)

 

 

 

The following command is used to convert database to snapshot standby.

DGMGRL> convert database 'db_name' to snapshot standby;

DGMGRL> convert database sbdb to Snapshot standby;

Converting database "sbdb" to a Snapshot Standby database, please wait...

Database "sbdb" converted successfully

 

The following command is used to convert database to physical standby.

DGMGRL> convert database 'db_name' to physical standby;

 

DGMGRL> convert database sbdb to Physical standby;

Converting database "sbdb" to a Physical Standby database, please wait...

Operation requires shut down of instance "PROD" on database "sbdb"

Shutting down instance "PROD"...

Connected to "SBDB"

Database closed.

Database dismounted.

ORACLE instance shut down.

Operation requires start up of instance "PROD" on database "sbdb"

Starting instance "PROD"...

Connected to an idle instance.

ORACLE instance started.

Connected to "SBDB"

Database mounted.

Connected to "SBDB"

Database "sbdb" converted successfully

 

The following command validates static connect identifier for given db_name

DGMGRL> VALIDATE STATIC CONNECT IDENTIFIER FOR SBDB;

Oracle Clusterware is not configured on database "sbdb".

Connecting to database "sbdb" using static connect identifier "(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=az-psdb1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=SBDB_DGMGRL)(INSTANCE_NAME=PROD)(SERVER=DEDICATED)(STATIC_SERVICE=TRUE)))" ...

Succeeded.

The static connect identifier allows for a connection to database "sbdb".

 

DGMGRL>

 

Execute sql statement from DGMGRL prompt. Select command doesn’t work from this prompt.

 

DGMGRL> sql "alter system switch logfile"

Succeeded.

DGMGRL>

 

 

March 13, 2023

Handy Oracle Data guard Broker Commands Part-I

 

Handy Oracle Data guard Broker Commands: -

Here,

Primary database is : ORCL

Standby Database is : ORCLDR


Login DGMGRL utility

dgmgrl / [Here connection happens through os authentication]

or

dgmgrl sys/<sys_password>@<connect dentifier>

 

 

To check current status and configuration

DGMGRL> show configuration;

 

Configuration - my_dg_config

 

  Protection Mode: MaxPerformance

  Members:

  orcl   - Primary database

    orcldr - Physical standby database

 

Fast-Start Failover:  Disabled

 

Configuration Status:

SUCCESS   (status updated 16 seconds ago)

 

 

To Display information of members under Dataguard configuration

DGMGRL> show database orcl

 

Database - orcl

 

  Role:               PRIMARY

  Intended State:     TRANSPORT-ON

  Instance(s):

    orcl

 

Database Status:

SUCCESS

 

DGMGRL> show database orcldr

 

Database - orcldr

 

  Role:               PHYSICAL STANDBY

  Intended State:     APPLY-ON

  Transport Lag:      0 seconds (computed 0 seconds ago)

  Apply Lag:          0 seconds (computed 0 seconds ago)

  Average Apply Rate: 30.00 KByte/s

  Real Time Query:    OFF

  Instance(s):

    orcl

 

Database Status:

SUCCESS

 

 

Performing a Switchover using dataguard broker

DGMGRL> switchover to orcldr;

Performing switchover NOW, please wait...

Operation requires a connection to database "orcldr"

Connecting ...

Connected to "ORCLDR"

Connected as SYSDBA.

New primary database "orcldr" is opening...

Operation requires start up of instance "orcl" on database "orcl"

Starting instance "orcl"...

Connected to an idle instance.

ORACLE instance started.

Connected to "orcl"

Database mounted.

Switchover succeeded, new primary is "orcldr"

 

 

To Display detailed information of configuration

DGMGRL> show configuration verbose

 

Configuration - my_dg_config

 

  Protection Mode: MaxPerformance

  Members:

  orcl   - Primary database

    orcldr - Physical standby database

 

  Properties:

    FastStartFailoverThreshold      = '30'

    OperationTimeout                = '30'

    TraceLevel                      = 'USER'

    FastStartFailoverLagLimit       = '30'

    CommunicationTimeout            = '180'

    ObserverReconnect               = '0'

    FastStartFailoverAutoReinstate  = 'TRUE'

    FastStartFailoverPmyShutdown    = 'TRUE'

    BystandersFollowRoleChange      = 'ALL'

    ObserverOverride                = 'FALSE'

    ExternalDestination1            = ''

    ExternalDestination2            = ''

    PrimaryLostWriteAction          = 'CONTINUE'

    ConfigurationWideServiceName    = 'orcl_CFG'

 

Fast-Start Failover:  Disabled

 

Configuration Status:

SUCCESS

 

 

To Display properties and a brief information of the database

DGMGRL> show database verbose orcldr

 

Database - orcldr

 

  Role:               PHYSICAL STANDBY

  Intended State:     APPLY-ON

  Transport Lag:      0 seconds (computed 1 second ago)

  Apply Lag:          0 seconds (computed 1 second ago)

  Average Apply Rate: 23.00 KByte/s

  Active Apply Rate:  765.00 KByte/s

  Maximum Apply Rate: 2.26 MByte/s

  Real Time Query:    OFF

  Instance(s):

    orcl

 

  Properties:

    DGConnectIdentifier             = 'orcldr'

    ObserverConnectIdentifier       = ''

    FastStartFailoverTarget         = ''

    PreferredObserverHosts          = ''

    LogShipping                     = 'ON'

    RedoRoutes                      = ''

    LogXptMode                      = 'ASYNC'

    DelayMins                       = '0'

    Binding                         = 'optional'

    MaxFailure                      = '0'

    ReopenSecs                      = '300'

    NetTimeout                      = '30'

    RedoCompression                 = 'DISABLE'

    PreferredApplyInstance          = ''

    ApplyInstanceTimeout            = '0'

    ApplyLagThreshold               = '30'

    TransportLagThreshold           = '30'

    TransportDisconnectedThreshold  = '30'

    ApplyParallel                   = 'AUTO'

    ApplyInstances                  = '0'

    StandbyFileManagement           = ''

    ArchiveLagTarget                = '0'

    LogArchiveMaxProcesses          = '0'

    LogArchiveMinSucceedDest        = '0'

    DataGuardSyncLatency            = '0'

    LogArchiveTrace                 = '0'

    LogArchiveFormat                = ''

    DbFileNameConvert               = ''

    LogFileNameConvert              = ''

    ArchiveLocation                 = ''

    AlternateLocation               = ''

    StandbyArchiveLocation          = ''

    StandbyAlternateLocation        = ''

    InconsistentProperties          = '(monitor)'

    InconsistentLogXptProps         = '(monitor)'

    LogXptStatus                    = '(monitor)'

    SendQEntries                    = '(monitor)'

    RecvQEntries                    = '(monitor)'

    HostName                        = 'EGC-MUN-STBY-PH'

    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=SERVER1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ORCLDR_DGMGRL)(INSTANCE_NAME=orcl)(SERVER=DEDICATED)))'

    TopWaitEvents                   = '(monitor)'

    SidName                         = '(monitor)'

 

  Log file locations:

    Alert log               : D:\ORACLE\diag\rdbms\orcldr\orcl\trace\alert_orcl.log

    Data Guard Broker log   : D:\ORACLE\diag\rdbms\orcldr\orcl\trace\drcorcl.log

 

Database Status:

SUCCESS

 

January 30, 2023

ORA-01017: invalid username/password during convert database to physical standby

 

Convert database to physical standby command errored with ORA-01017: invalid username/password; logon denied

 

So during execution of convert to physical standby using dgbroker utility , an error of ORA-01017: invalid username/password; logon denied is observed.

[oracle@server1]$ dgmgrl /

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Jan 19 02:11:48 2023

Version 19.16.0.0.0

 

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

 

Welcome to DGMGRL, type "help" for information.

Connected to "PROD"

Connected as SYSDG.

DGMGRL> connect sys@PROD

Password:

Connected to "PROD"

Connected as SYSDBA.

DGMGRL> CONVERT DATABASE sbdb to PHYSICAL STANDBY;

Converting database "sbdb" to a Physical Standby database, please wait...

Operation requires shut down of instance "PROD" on database "sbdb"

Shutting down instance "PROD"...

ORA-01017: invalid username/password; logon denied

 

 

 

 

Solution:-

There could be a few possibilities.

1.       Make sure the sys user password you are using is correct.

2.       The password file in both primary and standby database is same and valid.

3.       Sometimes error can be observed if case sensitive parameter is different between primary and standby

 

However after making sure all the above points , we were still failing to convert the snapshot standby database into physical standby.

 

To resolve this, we need to connect to dgmgrl utility as sys user and then perform the conversion.

 

 

 

[oracle@az-pspddb2 flashback]$ dgmgrl /

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Jan 19 02:11:48 2023

Version 19.16.0.0.0

 

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

 

Welcome to DGMGRL, type "help" for information.

Connected to "PROD"

Connected as SYSDG.

DGMGRL> connect sys@PROD

Password:<Provide sys password>

Connected to "PROD"

Connected as SYSDBA.

              DGMGRL> CONVERT DATABASE sbdb to PHYSICAL STANDBY;

              Converting database "sbdb" to a Physical Standby database, please wait...

Operation requires shut down of instance "PROD" on database "sbdb"

              Shutting down instance "PROD"...

             Connected to "SBDB"

             Database closed.

Database dismounted.

ORACLE instance shut down.

Operation requires start up of instance "PROD" on database "sbdb"

              Starting instance "PROD"...

             Connected to an idle instance.

ORA-01034: ORACLE not available

Process ID: 0

Session ID: 0 Serial number: 0

 

            ORACLE instance started.

            Connected to "SBDB"

             Database mounted.

 

 

 

January 23, 2023

How to fix ORA-16853: apply lag has exceeded specified threshold in 19c


We had a situation where in multi standby database environment , one of the standby database was lagging behind from the primary database. The log gap between primary and standby database was pretty huge. However, in primary we had the archives present so eventually after disabling/enabling LOG_ARCHIVE_DEST_3(As this was pointing to the 2nd standby site) the RFS process started transporting the archives to the 2nd standby database.

 

After executing following command the MRP process started applying logs in the standby site

SQL> alter database recover managed standby database disconnect from session;

 

However, when we checked in dataguard broker console, it was throwing a warning

ORA-16853: apply lag has exceeded specified threshold

 

DGMGRL> show configuration; 

Configuration - jde_dg_config 

  Protection Mode: MaxPerformance

  Members:

  prod   - Primary database

    proddr - Physical standby database

    sbdb   - Physical standby database

      Warning: ORA-16853: apply lag has exceeded specified threshold 

Fast-Start Failover:  Disabled

 Configuration Status:

WARNING   (status updated 52 seconds ago)

 

 

Solution:-

The reason of above warning is self explanatory, as one of our standby db (sbdb in this case) was out of sync for long time, the threshold value has been exceeded hence the warning.

 

[oracle@server1]$ dgmgrl /

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Wed Jan 18 23:15:18 2023

Version 19.16.0.0.0

 Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

 Welcome to DGMGRL, type "help" for information.

Connected to "PROD"

Connected as SYSDG.

 DGMGRL> edit database sbdb set property ApplyLagThreshold=0;

Property "applylagthreshold" updated

DGMGRL> edit database sbdb set property TransportLagThreshold=0;

Property "transportlagthreshold" updated


 

DGMGRL> show database verbose sbdb 

Database - sbdb

  Role:               SNAPSHOT STANDBY

  Transport Lag:      0 seconds (computed 1 second ago)

  Apply Lag:          60 minutes 16 seconds (computed 1 second ago)

  Instance(s):

    PROD

   Properties:

    DGConnectIdentifier             = 'sbdb'

    ObserverConnectIdentifier       = ''

    FastStartFailoverTarget         = ''

    PreferredObserverHosts          = ''

    LogShipping                     = 'ON'

    RedoRoutes                      = ''

    LogXptMode                      = 'ASYNC'

    DelayMins                       = '0'

    Binding                         = 'optional'

    MaxFailure                      = '0'

    ReopenSecs                      = '300'

    NetTimeout                      = '30'

    RedoCompression                 = 'DISABLE'

    PreferredApplyInstance          = ''

    ApplyInstanceTimeout            = '0'

    ApplyLagThreshold               = '0'

    TransportLagThreshold           = '0'

    TransportDisconnectedThreshold  = '30'

    ApplyParallel                   = 'AUTO'

    ApplyInstances                  = '0'

    StandbyFileManagement           = ''

    ArchiveLagTarget                = '0'

    LogArchiveMaxProcesses          = '0'

    LogArchiveMinSucceedDest        = '0'

    DataGuardSyncLatency            = '0'

    LogArchiveTrace                 = '0'

    LogArchiveFormat                = ''

    DbFileNameConvert               = ''

    LogFileNameConvert              = ''

    ArchiveLocation                 = ''

    AlternateLocation               = ''

    StandbyArchiveLocation          = ''

    StandbyAlternateLocation        = ''

    InconsistentProperties          = '(monitor)'

    InconsistentLogXptProps         = '(monitor)'

    LogXptStatus                    = '(monitor)'

    SendQEntries                    = '(monitor)'

    RecvQEntries                    = '(monitor)'

    HostName                        = 'az-psdvdb1'

    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server2)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=SBDB_DGMGRL)(INSTANCE_NAME=PROD)(SERVER=DEDICATED)))'

    TopWaitEvents                   = '(monitor)'

    SidName                         = '(monitor)'

   Log file locations:

    Alert log               : /u01/oracle/diag/rdbms/sbdb/PROD/trace/alert_PROD.log

    Data Guard Broker log   : /u01/oracle/diag/rdbms/sbdb/PROD/trace/drcPROD.log

 Database Status:

SUCCESS 

DGMGRL>

 

So, we can resolve  the warning by setting ApplyLagThreshold and TransportLagThreshold to Zero .          

May 9, 2020

How to apply RUR Patch in Dataguard Environment in 19c

Applying RUR(Release Update Revision) Patch in a dataguard (Physical Standby) environment.

Here is a brief explanation on how to apply RUR (Release Update Revision) in a dataguard environment
Here I will be  applying RUR 19.5.1.0 on the Primary and standby databases.

Primary database Server : primary
Standby database Server : standby

Primary db_unique_name : orclcdb
Standby db_unique_name : orclstd

April 17, 2020

Rolling Forward Standby Database When Archive log is missing in Primary DB

Problem statement :- standby database goes out of sync as the archive logs are missing from primary site.


Environment:


Primary:

Primary database node     : primary.soumya.com
Database name              : orclcdb
Primary DB UNIQUE NAME     : orclcdb
Primary database instances : orclcdb
DB Version: 19.0.0 

Standby:

Standby database node     : standby.soumya.com
Database name              : orclcdb
Standby DB UNIQUE NAME     : orclstd
Standby database instances : orclcdb
DB Version: 19.0.0 

April 15, 2020

ORA-16698 member has a LOG_ARCHIVE_DEST_n parameter with SERVICE attribute set

In primary DATABASE:-
Primary DB:- orclcdb
Standby DB:- orclstd


oracle@primary flashback]$ dgmgrl sys/admin123#@orclcdb
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Mon Apr 13 19:05:31 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "orclcdb"
Connected as SYSDBA.

DGMGRL> CREATE CONFIGURATION my_dg_config AS PRIMARY DATABASE IS orclcdb CONNECT IDENTIFIER IS orclcdb;

April 13, 2020

How to create physical standby database on Oracle 19c


OS Version:-Oracle Linux 7
Oracle version:- Oracle Database 19c Enterprise Edition Release 19.3.0 64bit

System Configuration

primary server:-

edit the following files

cat /etc/hosts

[oracle@primary ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.110   primary.soumya.com      primary
192.168.0.120   standby.soumya.com      standby

Oracle Database 26ai Installation Using RPM on Oracle Linux 9 (OEL 9) – Step-by-Step Guide

  In this post I will describe the installation of Oracle Database 26ai 64-bit on Oracle Linux 9 (OL8) 64-bit. The installation requires a m...