OEM 13c Agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'

 



Problem description:-

Recently for a customer we had performed a DR Drill  .  The customer had OEM agent installed in all oracle database server . Now after the drill was successful there was a requirement to restore one of the server from VM snapshot backup which was taken before the drill. Now after restoration of that vm , the OEM agent for that target was showing status as 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'



Solution:-

To resolve this problem , we did the following.


Logged into target DB server where agent status was showing as follows

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 5

Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.

---------------------------------------------------------------

Agent Version          : 13.5.0.0.0

OMS Version            : (unknown)

Protocol Version       : 12.1.0.1.0

Agent Home             : E:/Agent13c/agent_inst/agent_inst

Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log

Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0

Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib

Agent Process ID       : 6428

Parent Process ID      : 5376

Agent URL              : https://WMDR.com:3872/emd/main/

Local Agent URL in NAT : https://WMDR.com:3872/emd/main/

Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload

Started at             : 2024-05-13 03:30:26

Started by user        : WMDR16$

Operating System       : Windows version 10.0 (amd64)

Number of Targets      : 6

Last Reload            : (none)

Last successful upload                       : (none)

Last attempted upload                        : (none)

Total Megabytes of XML files uploaded so far : 0

Number of XML files pending upload           : 672

Size of XML files pending upload(MB)         : 0.91

Available disk space on upload filesystem    : 44.17%

Collection Status                            : Collections enabled

Heartbeat Status                             : Agent is blocked

Blocked Reason         : Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console.

Last attempted heartbeat to OMS              : 2024-05-13 04:55:01

Last successful heartbeat to OMS             : (none)

Next scheduled heartbeat to OMS              : 2024-05-13 04:58:01

 

---------------------------------------------------------------

Agent is Running and Ready


As we can see the Hearbeat status is showing “Agent is Blocked”  and reason behind block is agent is out of sync with its repository.

To resolve this problem, We logged into OMS Server and executed following


E:\>cd E:\u01\app\middleware\bin

 

E:\u01\app\middleware\bin>set OMS_HOME=E:\u01\app\middleware

 

E:\u01\app\middleware\bin>SET JAVA_HOME=E:\u01\app\agent\agent_13.5.0.0.0\oracle_common\jdk

 

E:\u01\app\middleware\bin>SET ORACLE_HOME=E:\u01\app\middleware

 

E:\u01\app\middleware\bin>emcli login -username=sysman

Enter password :

 

Login successful

 

E:\u01\app\middleware\bin>emcli sync

Synchronized successfully

 

E:\u01\app\middleware\bin>emcli resyncAgent -agent=WMDR.com:3872

Resync job RESYNC_20240513083609008 successfully submitted


After a while, when we checked the oem hearbeat status was showing OK.

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 5

Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.

---------------------------------------------------------------

Agent Version          : 13.5.0.0.0

OMS Version            : 13.5.0.0.0

Protocol Version       : 12.1.0.1.0

Agent Home             : E:/Agent13c/agent_inst/agent_inst

Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log

Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0

Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib

Agent Process ID       : 2660

Parent Process ID      : 7172

Agent URL              : https://WMDR.com:3872/emd/main/

Local Agent URL in NAT : https://WMDR16.com:3872/emd/main/

Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload

Started at             : 2024-05-13 05:33:22

Started by user        : WMDR16$

Operating System       : Windows version 10.0 (amd64)

Number of Targets      : 6

Last Reload            : (none)

Last successful upload                       : 2024-05-13 05:37:19

Last attempted upload                        : 2024-05-13 05:37:19

Total Megabytes of XML files uploaded so far : 0.08

Number of XML files pending upload           : 3

Size of XML files pending upload(MB)         : 0

Available disk space on upload filesystem    : 44.15%

Collection Status                            : Collections enabled

Heartbeat Status                             : Ok

Last attempted heartbeat to OMS              : 2024-05-13 05:36:42

Last successful heartbeat to OMS             : 2024-05-13 05:36:42

Next scheduled heartbeat to OMS              : 2024-05-13 05:37:43

 

---------------------------------------------------------------

Agent is Running and Ready




No comments:

Post a Comment