OEM 13c Agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'

 

Problem description:-

Recently for a customer we had performed a DR Drill  .  The customer had OEM agent installed in all oracle database server . Now after the drill was successful there was a requirement to restore one of the server from VM snapshot backup which was taken before the drill. Now after restoration of that vm , the OEM agent for that target was showing status as 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'

Solution:-

To resolve this problem , we did the following.

Logged into target DB server where agent status was showing as follows

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. --------------------------------------------------------------- Agent Version          : 13.5.0.0.0 OMS Version            : (unknown) Protocol Version       : 12.1.0.1.0 Agent Home             : E:/Agent13c/agent_inst/agent_inst Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0 Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib Agent Process ID       : 6428 Parent Process ID      : 5376 Agent URL              : https://WMDR.com:3872/emd/main/ Local Agent URL in NAT : https://WMDR.com:3872/emd/main/ Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload Started at             : 2024-05-13 03:30:26 Started by user        : WMDR16$ Operating System       : Windows version 10.0 (amd64) Number of Targets      : 6 Last Reload            : (none) Last successful upload                       : (none) Last attempted upload                        : (none) Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload           : 672 Size of XML files pending upload(MB)         : 0.91 Available disk space on upload filesystem    : 44.17% Collection Status                            : Collections enabled Heartbeat Status                             : Agent is blocked Blocked Reason         : Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Last attempted heartbeat to OMS              : 2024-05-13 04:55:01 Last successful heartbeat to OMS             : (none) Next scheduled heartbeat to OMS              : 2024-05-13 04:58:01   --------------------------------------------------------------- Agent is Running and Ready

As we can see the Hearbeat status is showing “Agent is Blocked”  and reason behind block is agent is out of sync with its repository.

To resolve this problem, We logged into OMS Server and executed following

E:\>cd E:\u01\app\middleware\bin   E:\u01\app\middleware\bin>set OMS_HOME=E:\u01\app\middleware   E:\u01\app\middleware\bin>SET JAVA_HOME=E:\u01\app\agent\agent_13.5.0.0.0\oracle_common\jdk   E:\u01\app\middleware\bin>SET ORACLE_HOME=E:\u01\app\middleware   E:\u01\app\middleware\bin>emcli login -username=sysman Enter password :   Login successful   E:\u01\app\middleware\bin>emcli sync Synchronized successfully   E:\u01\app\middleware\bin>emcli resyncAgent -agent=WMDR.com:3872 Resync job RESYNC_20240513083609008 successfully submitted

After a while, when we checked the oem hearbeat status was showing OK.

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. --------------------------------------------------------------- Agent Version          : 13.5.0.0.0 OMS Version            : 13.5.0.0.0 Protocol Version       : 12.1.0.1.0 Agent Home             : E:/Agent13c/agent_inst/agent_inst Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0 Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib Agent Process ID       : 2660 Parent Process ID      : 7172 Agent URL              : https://WMDR.com:3872/emd/main/ Local Agent URL in NAT : https://WMDR16.com:3872/emd/main/ Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload Started at             : 2024-05-13 05:33:22 Started by user        : WMDR16$ Operating System       : Windows version 10.0 (amd64) Number of Targets      : 6 Last Reload            : (none) Last successful upload                       : 2024-05-13 05:37:19 Last attempted upload                        : 2024-05-13 05:37:19 Total Megabytes of XML files uploaded so far : 0.08 Number of XML files pending upload           : 3 Size of XML files pending upload(MB)         : 0 Available disk space on upload filesystem    : 44.15% Collection Status                            : Collections enabled Heartbeat Status                             : Ok Last attempted heartbeat to OMS              : 2024-05-13 05:36:42 Last successful heartbeat to OMS             : 2024-05-13 05:36:42 Next scheduled heartbeat to OMS              : 2024-05-13 05:37:43   --------------------------------------------------------------- Agent is Running and Ready

The “visiblepw” is not set the sudoers file and as a result, the user will not be able to run sudo over ssh

While installing agent in target host for oracle enterprise manager 13c, following error is observed

Solution:-

This issue has two different solutions

1.       Change the entry “Defaults !visiblepw” to “Defaults visiblepw” in the /etc/sudoers file and re-try Agent Deployment. This needs to be done in host server.

2. You can also set the "oracle.sysman.prov.agentpush.enablePty" property to true in the "/data2/app/oracle/middleware/sysman/prov/agentpush/agentpush.properties" file, which is present on the OMS host,

And  pass -S argument to the sudo command

 /usr/bin/sudo -S -u %RUNAS% %COMMAND%

Now retry again the agent deployment , this time it should succeed. 

Enterprise manager 13c: Error During Agent Installation

While installing agent in target host for oracle enterprise manager 13c, following error is observed

EM 12c, 13c: Error During Agent Installation : Ensure central inventory is owned by install user, has read and write permission

Solution:-

Make sure in inventory_loc in /etc/oraInst.loc file points to a valid inventory location and it is writable by the installing user in the  target host server.

inventory_loc=/DEV/R12DEV/db/tech_st/19.3.0/oraInventory

where <valid inventory location> must be writable by the installing user. 

 

Steps to change sysman password in Oracle Enterprise Manager 12c

Stop oms . Do not include '-all' as the Admin Server needs to be up during this operation. cd /u01/app/oracle/product/MiddlewareR5/oms/bin/ [oracle@em bin]$ ./emctl stop oms Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down

Modify the SYSMAN password:- emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd <repository sys user password> -new_pwd <new sysman repository password> Example:- [oracle@em bin]$ ./emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd welcome1  -new_pwd welcome2# Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved. Changing passwords in backend ... Passwords changed in backend successfully. Updating repository password in Credential Store... Successfully updated Repository password in Credential Store. Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'. Successfully changed repository password. [oracle@em bin]$

Stop the admin server on OMS machine [oracle@em bin]$ ./emctl stop oms -all Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Already Stopped AdminServer Successfully Stopped Oracle Management Server is Down

Start the OMS [oracle@em bin]$ ./emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up Check OMS status [oracle@em bin]$ ./emctl status oms Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved. WebTier is Up Oracle Management Server is Up