Batch script to monitor ORDS service and send mail notification it goes down

 

Hi Everyone,

 

Recently I came across an issue where ords service on windows was getting stopped due to some reason. It was hard to monitor as they had no mechanism to monitor this.

 

To mitigate the problem, I wrote a batch script which will monitor ords service’s uptime on windows . The script will send a mail once notifying service has gone down.

 

Save the following script in .bat format and schedule it as per your requirement in task scheduler.

 

@echo off

setlocal enabledelayedexpansion

set my_date=%date%

set my_time=%TIME%

set my_hours=%my_time:~0,2%

set my_minutes=%my_time:~3,2%

set my_seconds=%my_time:~6,2%

 

REM Define variables

set APEX_URL=http://server1.example.com:7799/ords

set MAIL_RECIPIENT='soumya.das@testmail.com'

set SMTP_SERVER=smtp.office365.com

set SMTP_PORT=587

set MAIL_SENDER=notification@example.com

set MAIL_SUBJECT="Oracle APEX URL Status"

set SMTP_USER=emr-notification@example.com

set SMTP_PASSWORD=xyz123#

set SUBJECT="Apex down Alert"

set HOSTNAME=%COMPUTERNAME%

 

REM Check if the URL is accessible

curl -s -o NUL -w "%%{http_code}" %APEX_URL% > D:\response.txt

 

REM Check if D:\response.txt exists

if not exist D:\response.txt (

    REM If D:\response.txt is not present, send email notification

               powershell.exe -command "Send-MailMessage -From '%MAIL_SENDER%' -To %MAIL_RECIPIENT% -Subject '%SUBJECT%' -Body 'Oracle APEX URL is not accessible at %HOSTNAME% .Apex Service is down.' -SmtpServer '%SMTP_SERVER%' -Port %SMTP_PORT% -UseSsl -Credential (New-Object PSCredential('%SMTP_USER%', (ConvertTo-SecureString '%SMTP_PASSWORD%' -AsPlainText -Force)))"

)

 

REM Read the HTTP status code

set /p HTTP_STATUS=<D:\response.txt

 

REM Check if HTTP status code is 000, indicating a failure

if "%HTTP_STATUS%" equ "503" (

    powershell.exe -command "Send-MailMessage -From '%MAIL_SENDER%' -To %MAIL_RECIPIENT% -Subject '%SUBJECT%' -Body 'Oracle APEX URL is not accessible at %HOSTNAME%.Apex Service is down.' -SmtpServer '%SMTP_SERVER%' -Port %SMTP_PORT% -UseSsl -Credential (New-Object PSCredential('%SMTP_USER%', (ConvertTo-SecureString '%SMTP_PASSWORD%' -AsPlainText -Force)))"

) else if "%HTTP_STATUS%" equ "000" (

              powershell.exe -command "Send-MailMessage -From '%MAIL_SENDER%' -To %MAIL_RECIPIENT% -Subject '%SUBJECT%' -Body 'Oracle APEX URL is not accessible at %HOSTNAME%.Apex Service is down.' -SmtpServer '%SMTP_SERVER%' -Port %SMTP_PORT% -UseSsl -Credential (New-Object PSCredential('%SMTP_USER%', (ConvertTo-SecureString '%SMTP_PASSWORD%' -AsPlainText -Force)))"

)

 

REM Cleanup temporary files

del D:\response.txt

 

endlocal

 

How to apply oracle enterrprise manager13c Release 5 Update 22 Patch

How to apply RU Patch on Oracle Enterprise Manager 13.5

 

In this post I will be showing step by step guide to apply RU patch on Oracle Enterprise Manager 13.5 .  I will be applying  patch for April 2024(Patch no :- 36335368 )

Here I will be installing the patch on OMS server which is running on Windows.

 

1.     Download the patch (36335368) from My oracle support(MOS) and upload it into OMS Server

2.     Ensure OMSPatcher version atleast 13.9.5.20.0 or at later version.

To check omspatcher version.

Login to OMS Server and navigate to OMS HOME and execute following

D:\app\oem13c\middleware\OMSPatcher>set ORACLE_HOME=D:\app\oem13c\middleware

 

D:\app\oem13c\middleware\OMSPatcher>omspatcher version

 

Your environment has been set.

OMSPatcher Version: 13.9.4.4.0

OPlan Version: 12.2.0.1.16

OsysModel build: Tue Apr 28 18:16:31 PDT 2020

 

OMSPatcher succeeded.

 

As we can see the omspatcher version is lower than the required version, we will have to upgrade it.

Before upgrade, First shutdown the oms server

cd D:\app\oem13c\middleware\bin

D:\app\oem13c\middleware\bin>emctl stop oms -all

Oracle Enterprise Manager Cloud Control 13c Release 5

Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.

Stopping Oracle Management Server...

WebTier is Down

Oracle Management Server Successfully Stopped

Oracle Management Server is Down

JVMD Engine is Down

 

Now,we will take backup of existing omspatcher.

cd D:\app\oem13c\middleware\

D:\app\oem13c\middleware>ren OMSPatcher OMSPatcher_old

 

 

Download OMSPatcher patch(19999993) from my oracle support. And place it inside middleware home and unzip it.

Now check the version

cd D:\app\oem13c\middleware\OMSPatcher

D:\app\oem13c\middleware\OMSPatcher>omspatcher version

OMSPatcher Version: 13.9.5.20.0

OPlan Version: 12.2.0.1.16

OsysModel build: Tue Apr 28 18:16:31 PDT 2020

 

OMSPatcher succeeded.

 

3.    On Windows, apply Oracle Repository Creation Utility patch 33053642

C:\software>cd 33053642

 

C:\software\33053642>D:\app\oem13c\middleware\OPatch\opatch apply

Oracle Interim Patch Installer version 13.9.4.2.5

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

 

 

Oracle Home       : D:\app\oem13c\middleware

Central Inventory : C:\Program Files\Oracle\Inventory

   from           :

OPatch version    : 13.9.4.2.5

OUI version       : 13.9.4.0.0

Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_15-40-16PM_1.log

 

 

OPatch detects the Middleware Home as "D:\app\oem13c\middleware"

 

Verifying environment and performing prerequisite checks...

OPatch continues with these patches:   33053642

 

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

 

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = 'D:\app\oem13c\middleware')

 

 

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '33053642' to OH 'D:\app\oem13c\middleware'

 

Patching component oracle.rcu.shared, 12.2.1.4.0...

 

Patching component oracle.rcu.shared, 12.2.1.4.0...

Patch 33053642 successfully applied.

Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_15-40-16PM_1.log

 

OPatch succeeded.

 

4.     Apply the 12.2.1.4.0 version of below JDBC Patches on OMS server before proceeding with Enterprise Manager 13c Release 5 Update 22 patch 36335368 or later

Patch 35430934

Patch 34153238

Patch 31657681

C:\Windows\system32>cd C:\software\35430934

 

C:\software\35430934>D:\app\oem13c\middleware\OPatch\opatch apply

Oracle Interim Patch Installer version 13.9.4.2.5

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

 

 

Oracle Home       : D:\app\oem13c\middleware

Central Inventory : C:\Program Files\Oracle\Inventory

   from           :

OPatch version    : 13.9.4.2.5

OUI version       : 13.9.4.0.0

Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-16-26PM_1.log

 

 

OPatch detects the Middleware Home as "D:\app\oem13c\middleware"

 

Verifying environment and performing prerequisite checks...

OPatch continues with these patches:   35430934

 

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

 

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = 'D:\app\oem13c\middleware')

 

 

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '35430934' to OH 'D:\app\oem13c\middleware'

 

Patching component oracle.javavm.jrf, 19.3.0.0.0...

Patch 35430934 successfully applied.

Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-16-26PM_1.log

 

OPatch succeeded.

 

C:\software>cd 34153238

 

C:\software\34153238>D:\app\oem13c\middleware\OPatch\opatch apply

Oracle Interim Patch Installer version 13.9.4.2.5

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

 

 

Oracle Home       : D:\app\oem13c\middleware

Central Inventory : C:\Program Files\Oracle\Inventory

   from           :

OPatch version    : 13.9.4.2.5

OUI version       : 13.9.4.0.0

Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-18-42PM_1.log

 

 

OPatch detects the Middleware Home as "D:\app\oem13c\middleware"

 

Verifying environment and performing prerequisite checks...

OPatch continues with these patches:   34153238

 

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

 

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = 'D:\app\oem13c\middleware')

 

 

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '34153238' to OH 'D:\app\oem13c\middleware'

 

Patching component oracle.javavm.jrf, 19.3.0.0.0...

Patch 34153238 successfully applied.

Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-18-42PM_1.log

 

OPatch succeeded.

 

C:\software>cd 31657681

 

C:\software\31657681>D:\app\oem13c\middleware\OPatch\opatch apply

Oracle Interim Patch Installer version 13.9.4.2.5

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

 

 

Oracle Home       : D:\app\oem13c\middleware

Central Inventory : C:\Program Files\Oracle\Inventory

   from           :

OPatch version    : 13.9.4.2.5

OUI version       : 13.9.4.0.0

Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-21-27PM_1.log

 

 

OPatch detects the Middleware Home as "D:\app\oem13c\middleware"

 

Verifying environment and performing prerequisite checks...

OPatch continues with these patches:   31657681

 

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

 

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = 'D:\app\oem13c\middleware')

 

 

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '31657681' to OH 'D:\app\oem13c\middleware'

 

Patching component oracle.javavm.jrf, 19.3.0.0.0...

Patch 31657681 successfully applied.

Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-21-27PM_1.log

 

OPatch succeeded.

 

5.     Once all 3 patches are applied, we will proceed with OMS RU patch apply process.

 

Start OMS server.

D:\app\oem13c\middleware\bin>emctl start oms

Oracle Enterprise Manager Cloud Control 13c Release 5

Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.

Windows service OracleManagementServer_EMGC_OMS1_1 successfully started

Oracle Management Server is Up

JVMD Engine is Up

 

6.     Analyze  RU patch 36335368

D:\app\oem13c\middleware\OMSPatcher>omspatcher apply C:\software\36335368 -analyze

OMSPatcher Automation Tool

Copyright (c) 2017, Oracle Corporation.  All rights reserved.

 

 

OMSPatcher version : 13.9.5.20.0

OUI version        : 13.9.4.0.0

Running from       : D:\app\oem13c\middleware

Log file location  : D:\app\oem13c\middleware\cfgtoollogs\omspatcher\opatch2024-05-24_15-55-08PM_1.log

 

 

Patch contents are fine. Continuing patching session

Checking if Repository Create Utility (RCU) component patch exist in the current Oracle Home.

Repository Create Utility (RCU) component patch 33053642 exist in the current Oracle Home.

OMSPatcher log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_15-55-16PM_analyze.log

 

Please enter OMS weblogic admin server URL(t3s://CSPRDINT01.com:7101):>

Please enter OMS weblogic admin server username(weblogic):>

Please enter OMS weblogic admin server password:>

 

Enter DB user name : sys

Enter 'SYS' password :

Checking if current repository database is a supported version

Current repository database version is supported

 

 

Prereq "checkComponents" for patch 36329009 passed.

 

Prereq "checkComponents" for patch 36329231 passed.

 

Prereq "checkComponents" for patch 36329196 passed.

 

Prereq "checkComponents" for patch 36329020 passed.

 

Prereq "checkComponents" for patch 36329033 passed.

 

Prereq "checkComponents" for patch 36329152 passed.

 

Configuration Validation: Success

 

 

Running apply prerequisite checks for sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" and Oracle Home "D:\app\oem13c\middleware"...

Sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" are successfully analyzed for Oracle Home "D:\app\oem13c\middleware"

 

 

Complete Summary

================

 

 

All log file names referenced below can be accessed from the directory "D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_15-55-08PM_SystemPatch_36335368_1"

 

Prerequisites analysis summary:

-------------------------------

 

The following sub-patch(es) are applicable:

 

             Featureset                                             Sub-patches                                                                                  Log file

             ----------                                             -----------                                                                                  --------

  oracle.sysman.top.oms   36329033,36329196,36329020,36329009,36329231,36329152   36329033,36329196,36329020,36329009,36329231,36329152_opatch2024-05-24_15-55-45PM_1.log

 

 

The following sub-patches are incompatible with components installed in the OMS system:

35582217,34430509,34706773,36329066,35854914,36161799,36329046,36329161,36329220

 

 

 

Log file location: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_15-55-16PM_analyze.log

 

OMSPatcher succeeded.

 

D:\app\oem13c\middleware\OMSPatcher>

 

7.     As the analyze result was successful, we can now proceed and apply the patch.

Apply RU patch:-

D:\app\oem13c\middleware\OMSPatcher>omspatcher apply C:\software\36335368

 

OMSPatcher Automation Tool

Copyright (c) 2017, Oracle Corporation.  All rights reserved.

 

 

OMSPatcher version : 13.9.5.20.0

OUI version        : 13.9.4.0.0

Running from       : D:\app\oem13c\middleware

Log file location  : D:\app\oem13c\middleware\cfgtoollogs\omspatcher\opatch2024-05-24_16-36-53PM_1.log

 

 

Patch contents are fine. Continuing patching session

Checking if Repository Create Utility (RCU) component patch exist in the current Oracle Home.

Repository Create Utility (RCU) component patch 33053642 exist in the current Oracle Home.

OMSPatcher log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_16-36-59PM_apply.log

 

Please enter OMS weblogic admin server URL(t3s://CSPRDINT01.amnm.com:7101):>

Please enter OMS weblogic admin server username(weblogic):>

Please enter OMS weblogic admin server password:>

 

Enter DB user name : sys

Enter 'sys' password :

Checking if current repository database is a supported version

Current repository database version is supported

 

 

Prereq "checkComponents" for patch 36329009 passed.

 

Prereq "checkComponents" for patch 36329231 passed.

 

Prereq "checkComponents" for patch 36329196 passed.

 

Prereq "checkComponents" for patch 36329020 passed.

 

Prereq "checkComponents" for patch 36329033 passed.

 

Prereq "checkComponents" for patch 36329152 passed.

 

Configuration Validation: Success

 

 

Running apply prerequisite checks for sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" and Oracle Home "D:\app\oem13c\middleware"...

Sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" are successfully analyzed for Oracle Home "D:\app\oem13c\middleware"

 

To continue, OMSPatcher will do the following:

[Patch and deploy artifacts]   : Apply sub-patch(es) [ 36329009 36329020 36329033 36329152 36329196 36329231 ]

                                 Apply RCU artifact with patch "D:\app\oem13c\middleware\.omspatcher_storage\36329009_May_8_2024_01_15_01\original_patch";

                                 Apply RCU artifact with patch

                                 Register MRS artifact "commands";

                                 Register MRS artifact "omsPropertyDef";

                                 Register MRS artifact "targetType";

                                 Register MRS artifact "chargeback";

                                 Register MRS artifact "default_collection";

                                 Register MRS artifact "assoc";

                                 Register MRS artifact "jobTypes";

                                 Register MRS artifact "systemStencil";

                                 Register MRS artifact "procedures";

                                 Register MRS artifact "discovery";

                                 Register MRS artifact "EcmMetadataOnlyRegistration";

                                 Register MRS artifact "swlib";

                                 Register MRS artifact "OracleCertifiedTemplate";

                                 Register MRS artifact "namedQuery";

                                 Register MRS artifact "TargetPrivilege";

                                 Register MRS artifact "CredStoreMetadata";

                                 Register MRS artifact "storeTargetType";

                                 Register MRS artifact "gccompliance";

                                 Register MRS artifact "compression";

                                 Register MRS artifact "eventSpecificCustmzn";

                                 Register MRS artifact "report";

                                 Register MRS artifact "namedsql";

                                 Register MRS artifact "runbooks";

                                 Register MRS artifact "SecurityClassManager";

                                 Register MRS artifact "derivedAssocs"

 

 

Do you want to proceed? [y|n]

y

User Responded with: Y

Stopping the OMS.....

Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_16-43-57PM_SystemPatch_36335368_9\stop_oms_2024-05-24_16-43-57PM.log

 

 

Applying sub-patch(es) "36329009,36329020,36329033,36329152,36329196,36329231"

Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_16-37-24PM_1.log

 

Starting the ADMIN .....

Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_16-49-14PM_SystemPatch_36335368_12\start_admin_2024-05-24_16-49-14PM.log

 

 

Starting the oms

Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_17-48-11PM_SystemPatch_36335368_108\start_oms_2024-05-24_17-48-11PM.log

 

 

Complete Summary

================

 

 

All log file names referenced below can be accessed from the directory "D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_17-48-11PM_SystemPatch_36335368_108"

 

Patching summary:

-----------------

 

Binaries of the following sub-patch(es) have been applied successfully:

 

                        Featureset                                             Sub-patches                                                                                  Log file

                        ----------                                             -----------                                                                                  --------

  oracle.sysman.top.oms_13.5.0.0.0   36329009,36329020,36329033,36329152,36329196,36329231   36329009,36329020,36329033,36329152,36329196,36329231_opatch2024-05-24_16-37-24PM_1.log

 

 

The following sub-patches are incompatible with components installed in the OMS system:

35582217,34430509,34706773,36329066,35854914,36161799,36329046,36329161,36329220

 

Deployment summary:

-------------------

 

The following artifact(s) have been successfully deployed:

 

                        Artifacts                                                               Log file

                        ---------                                                               --------

                              SQL                rcu_applypatch_original_patch_2024-05-24_16-54-55PM.log

                              SQL                rcu_applypatch_original_patch_2024-05-24_16-59-12PM.log

                              SQL                rcu_applypatch_original_patch_2024-05-24_16-59-43PM.log

                              SQL                rcu_applypatch_original_patch_2024-05-24_17-00-16PM.log

                              SQL                rcu_applypatch_original_patch_2024-05-24_17-00-55PM.log

                              SQL                rcu_applypatch_original_patch_2024-05-24_17-01-48PM.log

           MRS-default_collection            emctl_register_default_collection_2024-05-24_17-33-45PM.log

  

 

Log file location: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_16-36-59PM_apply.log

 

OMSPatcher succeeded.

 

8.     Once patch application is done, perform a emcli sync

D:\app\oem13c\middleware\bin>emcli login -username=sysman

Enter password :

 

Login successful

 

D:\app\oem13c\middleware\bin>emcli sync

Synchronized successfully

 


Batch script to kill blocked session

An oracle database administrator often face a situation where a deadlock is observed in database. A deadlock occurs when two or more sessions are waiting for data locked by each other, resulting in all the sessions being blocked.

As a DBA, the only way you can resolve a lock by killing either the blocking session or blocked session.

To automate this process, I have prepared a batch script for windows which will kill the session after 10 mins of getting locked.

Step 1. Lets prepare the sql first which will find out the session that are locked more than 10mins

SET LINESIZE 1000

SET PAGESIZE 1000

SET FEEDBACK OFF

 

SELECT s.sid, s.serial#, ROUND((SYSDATE - s.logon_time) * 24 * 60) AS logon_time_minutes

FROM v$session s, v$session_wait w

WHERE s.sid = w.sid

AND w.event = 'enq: TX - row lock contention'

AND s.status = 'ACTIVE'

AND (SYSDATE - s.logon_time) * 24 * 60 >= 10;

exit;


Lets save the above sql as blocked_sessions_query.sql

Step 2. I will now prepare the batch script which will kill the session based on output retrieved from above sql


@echo on

 

REM Oracle database connection details

 

SET ORACLE_USER=sys

SET ORACLE_PASSWORD=welcome123#

SET ORACLE_SID=ORCL

SET ORACLE_HOME=D:\oracle\product\19.3.0\dbhome_1

 

REM Define log file path

SET LOG_FILE=F:\script_log.txt

 

REM SQL file containing the query to identify blocked sessions

SET SQL_FILE=F:\blocked_sessions_query.sql

 

%ORACLE_HOME%\bin\sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SID% as sysdba @%SQL_FILE% > F: \session_id.txt

 

 

REM Execute SQL query using SQL*Plus and fetch results

FOR /F "tokens=1,2,3" %%a IN (F:\session_id.txt) DO (

    REM Set session details

    set sid=%%a

    set serial#=%%b

    set logon_time_minutes=%%c

)

    REM If session has been active for at least 15 minutes, kill the session

    if %logon_time_minutes% GEQ 15 (

        echo Killing session %sid% (serial#: %serial#%)

        echo ALTER SYSTEM KILL SESSION '%sid%,%serial#%' immediate; | %ORACLE_HOME%\bin\sqlplus -S %ORACLE_USER%/%ORACLE_PASSWORD%@%ORACLE_SID% as sysdba

        REM Call email notification function here

    )

 

 

exit /b

 


Save the above batch file as deadlock_kill.bat

Step 3. Now schedule the above script in task scheduler

















Click on and provide password to set it.


Disclaimer:- This script should be tested properly in uat/test environment and it is not advised to run it directly on production database. The script automatically selects the session and kills it. However before killing the session, it is recommended you must send the information to application team to get their approval on which session to be killed.


OEM 13c Agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'

 



Problem description:-

Recently for a customer we had performed a DR Drill  .  The customer had OEM agent installed in all oracle database server . Now after the drill was successful there was a requirement to restore one of the server from VM snapshot backup which was taken before the drill. Now after restoration of that vm , the OEM agent for that target was showing status as 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'



Solution:-

To resolve this problem , we did the following.


Logged into target DB server where agent status was showing as follows

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 5

Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.

---------------------------------------------------------------

Agent Version          : 13.5.0.0.0

OMS Version            : (unknown)

Protocol Version       : 12.1.0.1.0

Agent Home             : E:/Agent13c/agent_inst/agent_inst

Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log

Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0

Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib

Agent Process ID       : 6428

Parent Process ID      : 5376

Agent URL              : https://WMDR.com:3872/emd/main/

Local Agent URL in NAT : https://WMDR.com:3872/emd/main/

Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload

Started at             : 2024-05-13 03:30:26

Started by user        : WMDR16$

Operating System       : Windows version 10.0 (amd64)

Number of Targets      : 6

Last Reload            : (none)

Last successful upload                       : (none)

Last attempted upload                        : (none)

Total Megabytes of XML files uploaded so far : 0

Number of XML files pending upload           : 672

Size of XML files pending upload(MB)         : 0.91

Available disk space on upload filesystem    : 44.17%

Collection Status                            : Collections enabled

Heartbeat Status                             : Agent is blocked

Blocked Reason         : Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console.

Last attempted heartbeat to OMS              : 2024-05-13 04:55:01

Last successful heartbeat to OMS             : (none)

Next scheduled heartbeat to OMS              : 2024-05-13 04:58:01

 

---------------------------------------------------------------

Agent is Running and Ready


As we can see the Hearbeat status is showing “Agent is Blocked”  and reason behind block is agent is out of sync with its repository.

To resolve this problem, We logged into OMS Server and executed following


E:\>cd E:\u01\app\middleware\bin

 

E:\u01\app\middleware\bin>set OMS_HOME=E:\u01\app\middleware

 

E:\u01\app\middleware\bin>SET JAVA_HOME=E:\u01\app\agent\agent_13.5.0.0.0\oracle_common\jdk

 

E:\u01\app\middleware\bin>SET ORACLE_HOME=E:\u01\app\middleware

 

E:\u01\app\middleware\bin>emcli login -username=sysman

Enter password :

 

Login successful

 

E:\u01\app\middleware\bin>emcli sync

Synchronized successfully

 

E:\u01\app\middleware\bin>emcli resyncAgent -agent=WMDR.com:3872

Resync job RESYNC_20240513083609008 successfully submitted


After a while, when we checked the oem hearbeat status was showing OK.

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 5

Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved.

---------------------------------------------------------------

Agent Version          : 13.5.0.0.0

OMS Version            : 13.5.0.0.0

Protocol Version       : 12.1.0.1.0

Agent Home             : E:/Agent13c/agent_inst/agent_inst

Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log

Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0

Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib

Agent Process ID       : 2660

Parent Process ID      : 7172

Agent URL              : https://WMDR.com:3872/emd/main/

Local Agent URL in NAT : https://WMDR16.com:3872/emd/main/

Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload

Started at             : 2024-05-13 05:33:22

Started by user        : WMDR16$

Operating System       : Windows version 10.0 (amd64)

Number of Targets      : 6

Last Reload            : (none)

Last successful upload                       : 2024-05-13 05:37:19

Last attempted upload                        : 2024-05-13 05:37:19

Total Megabytes of XML files uploaded so far : 0.08

Number of XML files pending upload           : 3

Size of XML files pending upload(MB)         : 0

Available disk space on upload filesystem    : 44.15%

Collection Status                            : Collections enabled

Heartbeat Status                             : Ok

Last attempted heartbeat to OMS              : 2024-05-13 05:36:42

Last successful heartbeat to OMS             : 2024-05-13 05:36:42

Next scheduled heartbeat to OMS              : 2024-05-13 05:37:43

 

---------------------------------------------------------------

Agent is Running and Ready