How to apply oracle enterrprise manager13c Release 5 Update 22 Patch

How to apply RU Patch on Oracle Enterprise Manager 13.5

 

In this post I will be showing step by step guide to apply RU patch on Oracle Enterprise Manager 13.5 .  I will be applying  patch for April 2024(Patch no :- 36335368 )

Here I will be installing the patch on OMS server which is running on Windows.

 

1.     Download the patch (36335368) from My oracle support(MOS) and upload it into OMS Server

2.     Ensure OMSPatcher version atleast 13.9.5.20.0 or at later version.

To check omspatcher version.

Login to OMS Server and navigate to OMS HOME and execute following

D:\app\oem13c\middleware\OMSPatcher>set ORACLE_HOME=D:\app\oem13c\middleware   D:\app\oem13c\middleware\OMSPatcher>omspatcher version   Your environment has been set. OMSPatcher Version: 13.9.4.4.0 OPlan Version: 12.2.0.1.16 OsysModel build: Tue Apr 28 18:16:31 PDT 2020   OMSPatcher succeeded.

 

As we can see the omspatcher version is lower than the required version, we will have to upgrade it.

Before upgrade, First shutdown the oms server

cd D:\app\oem13c\middleware\bin D:\app\oem13c\middleware\bin>emctl stop oms -all Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. Stopping Oracle Management Server... WebTier is Down Oracle Management Server Successfully Stopped Oracle Management Server is Down JVMD Engine is Down

 

Now,we will take backup of existing omspatcher.

cd D:\app\oem13c\middleware\ D:\app\oem13c\middleware>ren OMSPatcher OMSPatcher_old

 

Download OMSPatcher patch(19999993) from my oracle support. And place it inside middleware home and unzip it.

Now check the version

cd D:\app\oem13c\middleware\OMSPatcher D:\app\oem13c\middleware\OMSPatcher>omspatcher version OMSPatcher Version: 13.9.5.20.0 OPlan Version: 12.2.0.1.16 OsysModel build: Tue Apr 28 18:16:31 PDT 2020   OMSPatcher succeeded.

 

3.    On Windows, apply Oracle Repository Creation Utility patch 33053642

C:\software>cd 33053642   C:\software\33053642>D:\app\oem13c\middleware\OPatch\opatch apply Oracle Interim Patch Installer version 13.9.4.2.5 Copyright (c) 2024, Oracle Corporation.  All rights reserved.     Oracle Home       : D:\app\oem13c\middleware Central Inventory : C:\Program Files\Oracle\Inventory    from           : OPatch version    : 13.9.4.2.5 OUI version       : 13.9.4.0.0 Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_15-40-16PM_1.log     OPatch detects the Middleware Home as "D:\app\oem13c\middleware"   Verifying environment and performing prerequisite checks... OPatch continues with these patches:   33053642   Do you want to proceed? [y|n] y User Responded with: Y All checks passed.   Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = 'D:\app\oem13c\middleware')     Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '33053642' to OH 'D:\app\oem13c\middleware'   Patching component oracle.rcu.shared, 12.2.1.4.0...   Patching component oracle.rcu.shared, 12.2.1.4.0... Patch 33053642 successfully applied. Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_15-40-16PM_1.log   OPatch succeeded.

 

4.     Apply the 12.2.1.4.0 version of below JDBC Patches on OMS server before proceeding with Enterprise Manager 13c Release 5 Update 22 patch 36335368 or later

Patch 35430934

Patch 34153238

Patch 31657681

C:\Windows\system32>cd C:\software\35430934   C:\software\35430934>D:\app\oem13c\middleware\OPatch\opatch apply Oracle Interim Patch Installer version 13.9.4.2.5 Copyright (c) 2024, Oracle Corporation.  All rights reserved.     Oracle Home       : D:\app\oem13c\middleware Central Inventory : C:\Program Files\Oracle\Inventory    from           : OPatch version    : 13.9.4.2.5 OUI version       : 13.9.4.0.0 Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-16-26PM_1.log     OPatch detects the Middleware Home as "D:\app\oem13c\middleware"   Verifying environment and performing prerequisite checks... OPatch continues with these patches:   35430934   Do you want to proceed? [y|n] y User Responded with: Y All checks passed.   Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = 'D:\app\oem13c\middleware')     Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '35430934' to OH 'D:\app\oem13c\middleware'   Patching component oracle.javavm.jrf, 19.3.0.0.0... Patch 35430934 successfully applied. Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-16-26PM_1.log   OPatch succeeded.

 

C:\software>cd 34153238   C:\software\34153238>D:\app\oem13c\middleware\OPatch\opatch apply Oracle Interim Patch Installer version 13.9.4.2.5 Copyright (c) 2024, Oracle Corporation.  All rights reserved.     Oracle Home       : D:\app\oem13c\middleware Central Inventory : C:\Program Files\Oracle\Inventory    from           : OPatch version    : 13.9.4.2.5 OUI version       : 13.9.4.0.0 Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-18-42PM_1.log     OPatch detects the Middleware Home as "D:\app\oem13c\middleware"   Verifying environment and performing prerequisite checks... OPatch continues with these patches:   34153238   Do you want to proceed? [y|n] y User Responded with: Y All checks passed.   Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = 'D:\app\oem13c\middleware')     Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '34153238' to OH 'D:\app\oem13c\middleware'   Patching component oracle.javavm.jrf, 19.3.0.0.0... Patch 34153238 successfully applied. Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-18-42PM_1.log   OPatch succeeded.

 

C:\software>cd 31657681   C:\software\31657681>D:\app\oem13c\middleware\OPatch\opatch apply Oracle Interim Patch Installer version 13.9.4.2.5 Copyright (c) 2024, Oracle Corporation.  All rights reserved.     Oracle Home       : D:\app\oem13c\middleware Central Inventory : C:\Program Files\Oracle\Inventory    from           : OPatch version    : 13.9.4.2.5 OUI version       : 13.9.4.0.0 Log file location : D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-21-27PM_1.log     OPatch detects the Middleware Home as "D:\app\oem13c\middleware"   Verifying environment and performing prerequisite checks... OPatch continues with these patches:   31657681   Do you want to proceed? [y|n] y User Responded with: Y All checks passed.   Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = 'D:\app\oem13c\middleware')     Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '31657681' to OH 'D:\app\oem13c\middleware'   Patching component oracle.javavm.jrf, 19.3.0.0.0... Patch 31657681 successfully applied. Log file location: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_13-21-27PM_1.log   OPatch succeeded.

 

5.     Once all 3 patches are applied, we will proceed with OMS RU patch apply process.

 

Start OMS server.

D:\app\oem13c\middleware\bin>emctl start oms Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. Windows service OracleManagementServer_EMGC_OMS1_1 successfully started Oracle Management Server is Up JVMD Engine is Up

 

6.     Analyze  RU patch 36335368

D:\app\oem13c\middleware\OMSPatcher>omspatcher apply C:\software\36335368 -analyze OMSPatcher Automation Tool Copyright (c) 2017, Oracle Corporation.  All rights reserved.     OMSPatcher version : 13.9.5.20.0 OUI version        : 13.9.4.0.0 Running from       : D:\app\oem13c\middleware Log file location  : D:\app\oem13c\middleware\cfgtoollogs\omspatcher\opatch2024-05-24_15-55-08PM_1.log     Patch contents are fine. Continuing patching session Checking if Repository Create Utility (RCU) component patch exist in the current Oracle Home. Repository Create Utility (RCU) component patch 33053642 exist in the current Oracle Home. OMSPatcher log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_15-55-16PM_analyze.log   Please enter OMS weblogic admin server URL(t3s://CSPRDINT01.com:7101):> Please enter OMS weblogic admin server username(weblogic):> Please enter OMS weblogic admin server password:>   Enter DB user name : sys Enter 'SYS' password : Checking if current repository database is a supported version Current repository database version is supported     Prereq "checkComponents" for patch 36329009 passed.   Prereq "checkComponents" for patch 36329231 passed.   Prereq "checkComponents" for patch 36329196 passed.   Prereq "checkComponents" for patch 36329020 passed.   Prereq "checkComponents" for patch 36329033 passed.   Prereq "checkComponents" for patch 36329152 passed.   Configuration Validation: Success     Running apply prerequisite checks for sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" and Oracle Home "D:\app\oem13c\middleware"... Sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" are successfully analyzed for Oracle Home "D:\app\oem13c\middleware"     Complete Summary ================     All log file names referenced below can be accessed from the directory "D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_15-55-08PM_SystemPatch_36335368_1"   Prerequisites analysis summary: -------------------------------   The following sub-patch(es) are applicable:                Featureset                                             Sub-patches                                                                                  Log file              ----------                                             -----------                                                                                  --------   oracle.sysman.top.oms   36329033,36329196,36329020,36329009,36329231,36329152   36329033,36329196,36329020,36329009,36329231,36329152_opatch2024-05-24_15-55-45PM_1.log     The following sub-patches are incompatible with components installed in the OMS system: 35582217,34430509,34706773,36329066,35854914,36161799,36329046,36329161,36329220       Log file location: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_15-55-16PM_analyze.log   OMSPatcher succeeded.   D:\app\oem13c\middleware\OMSPatcher>

 

7.     As the analyze result was successful, we can now proceed and apply the patch.

Apply RU patch:-

D:\app\oem13c\middleware\OMSPatcher>omspatcher apply C:\software\36335368   OMSPatcher Automation Tool Copyright (c) 2017, Oracle Corporation.  All rights reserved.     OMSPatcher version : 13.9.5.20.0 OUI version        : 13.9.4.0.0 Running from       : D:\app\oem13c\middleware Log file location  : D:\app\oem13c\middleware\cfgtoollogs\omspatcher\opatch2024-05-24_16-36-53PM_1.log     Patch contents are fine. Continuing patching session Checking if Repository Create Utility (RCU) component patch exist in the current Oracle Home. Repository Create Utility (RCU) component patch 33053642 exist in the current Oracle Home. OMSPatcher log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_16-36-59PM_apply.log   Please enter OMS weblogic admin server URL(t3s://CSPRDINT01.amnm.com:7101):> Please enter OMS weblogic admin server username(weblogic):> Please enter OMS weblogic admin server password:>   Enter DB user name : sys Enter 'sys' password : Checking if current repository database is a supported version Current repository database version is supported     Prereq "checkComponents" for patch 36329009 passed.   Prereq "checkComponents" for patch 36329231 passed.   Prereq "checkComponents" for patch 36329196 passed.   Prereq "checkComponents" for patch 36329020 passed.   Prereq "checkComponents" for patch 36329033 passed.   Prereq "checkComponents" for patch 36329152 passed.   Configuration Validation: Success     Running apply prerequisite checks for sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" and Oracle Home "D:\app\oem13c\middleware"... Sub-patch(es) "36329033,36329196,36329020,36329009,36329231,36329152" are successfully analyzed for Oracle Home "D:\app\oem13c\middleware"   To continue, OMSPatcher will do the following: [Patch and deploy artifacts]   : Apply sub-patch(es) [ 36329009 36329020 36329033 36329152 36329196 36329231 ]                                  Apply RCU artifact with patch "D:\app\oem13c\middleware\.omspatcher_storage\36329009_May_8_2024_01_15_01\original_patch";                                  Apply RCU artifact with patch                                  Register MRS artifact "commands";                                  Register MRS artifact "omsPropertyDef";                                  Register MRS artifact "targetType";                                  Register MRS artifact "chargeback";                                  Register MRS artifact "default_collection";                                  Register MRS artifact "assoc";                                  Register MRS artifact "jobTypes";                                  Register MRS artifact "systemStencil";                                  Register MRS artifact "procedures";                                  Register MRS artifact "discovery";                                  Register MRS artifact "EcmMetadataOnlyRegistration";                                  Register MRS artifact "swlib";                                  Register MRS artifact "OracleCertifiedTemplate";                                  Register MRS artifact "namedQuery";                                  Register MRS artifact "TargetPrivilege";                                  Register MRS artifact "CredStoreMetadata";                                  Register MRS artifact "storeTargetType";                                  Register MRS artifact "gccompliance";                                  Register MRS artifact "compression";                                  Register MRS artifact "eventSpecificCustmzn";                                  Register MRS artifact "report";                                  Register MRS artifact "namedsql";                                  Register MRS artifact "runbooks";                                  Register MRS artifact "SecurityClassManager";                                  Register MRS artifact "derivedAssocs"     Do you want to proceed? [y|n] y User Responded with: Y Stopping the OMS..... Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_16-43-57PM_SystemPatch_36335368_9\stop_oms_2024-05-24_16-43-57PM.log     Applying sub-patch(es) "36329009,36329020,36329033,36329152,36329196,36329231" Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\opatch\opatch2024-05-24_16-37-24PM_1.log   Starting the ADMIN ..... Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_16-49-14PM_SystemPatch_36335368_12\start_admin_2024-05-24_16-49-14PM.log     Starting the oms Please monitor log file: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_17-48-11PM_SystemPatch_36335368_108\start_oms_2024-05-24_17-48-11PM.log     Complete Summary ================     All log file names referenced below can be accessed from the directory "D:\app\oem13c\middleware\cfgtoollogs\omspatcher\2024-05-24_17-48-11PM_SystemPatch_36335368_108"   Patching summary: -----------------   Binaries of the following sub-patch(es) have been applied successfully:                           Featureset                                             Sub-patches                                                                                  Log file                         ----------                                             -----------                                                                                  --------   oracle.sysman.top.oms_13.5.0.0.0   36329009,36329020,36329033,36329152,36329196,36329231   36329009,36329020,36329033,36329152,36329196,36329231_opatch2024-05-24_16-37-24PM_1.log     The following sub-patches are incompatible with components installed in the OMS system: 35582217,34430509,34706773,36329066,35854914,36161799,36329046,36329161,36329220   Deployment summary: -------------------   The following artifact(s) have been successfully deployed:                           Artifacts                                                               Log file                         ---------                                                               --------                               SQL                rcu_applypatch_original_patch_2024-05-24_16-54-55PM.log                               SQL                rcu_applypatch_original_patch_2024-05-24_16-59-12PM.log                               SQL                rcu_applypatch_original_patch_2024-05-24_16-59-43PM.log                               SQL                rcu_applypatch_original_patch_2024-05-24_17-00-16PM.log                               SQL                rcu_applypatch_original_patch_2024-05-24_17-00-55PM.log                               SQL                rcu_applypatch_original_patch_2024-05-24_17-01-48PM.log            MRS-default_collection            emctl_register_default_collection_2024-05-24_17-33-45PM.log      Log file location: D:\app\oem13c\middleware\cfgtoollogs\omspatcher\36335368\omspatcher_2024-05-24_16-36-59PM_apply.log   OMSPatcher succeeded.

 

8.     Once patch application is done, perform a emcli sync

D:\app\oem13c\middleware\bin>emcli login -username=sysman Enter password :   Login successful   D:\app\oem13c\middleware\bin>emcli sync Synchronized successfully

 

OEM 13c Agent status shows 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'

 

Problem description:-

Recently for a customer we had performed a DR Drill  .  The customer had OEM agent installed in all oracle database server . Now after the drill was successful there was a requirement to restore one of the server from VM snapshot backup which was taken before the drill. Now after restoration of that vm , the OEM agent for that target was showing status as 'Diagnose for Agent Blocked (Bounce Counter Mismatch)'

Solution:-

To resolve this problem , we did the following.

Logged into target DB server where agent status was showing as follows

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. --------------------------------------------------------------- Agent Version          : 13.5.0.0.0 OMS Version            : (unknown) Protocol Version       : 12.1.0.1.0 Agent Home             : E:/Agent13c/agent_inst/agent_inst Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0 Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib Agent Process ID       : 6428 Parent Process ID      : 5376 Agent URL              : https://WMDR.com:3872/emd/main/ Local Agent URL in NAT : https://WMDR.com:3872/emd/main/ Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload Started at             : 2024-05-13 03:30:26 Started by user        : WMDR16$ Operating System       : Windows version 10.0 (amd64) Number of Targets      : 6 Last Reload            : (none) Last successful upload                       : (none) Last attempted upload                        : (none) Total Megabytes of XML files uploaded so far : 0 Number of XML files pending upload           : 672 Size of XML files pending upload(MB)         : 0.91 Available disk space on upload filesystem    : 44.17% Collection Status                            : Collections enabled Heartbeat Status                             : Agent is blocked Blocked Reason         : Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered. Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Last attempted heartbeat to OMS              : 2024-05-13 04:55:01 Last successful heartbeat to OMS             : (none) Next scheduled heartbeat to OMS              : 2024-05-13 04:58:01   --------------------------------------------------------------- Agent is Running and Ready

As we can see the Hearbeat status is showing “Agent is Blocked”  and reason behind block is agent is out of sync with its repository.

To resolve this problem, We logged into OMS Server and executed following

E:\>cd E:\u01\app\middleware\bin   E:\u01\app\middleware\bin>set OMS_HOME=E:\u01\app\middleware   E:\u01\app\middleware\bin>SET JAVA_HOME=E:\u01\app\agent\agent_13.5.0.0.0\oracle_common\jdk   E:\u01\app\middleware\bin>SET ORACLE_HOME=E:\u01\app\middleware   E:\u01\app\middleware\bin>emcli login -username=sysman Enter password :   Login successful   E:\u01\app\middleware\bin>emcli sync Synchronized successfully   E:\u01\app\middleware\bin>emcli resyncAgent -agent=WMDR.com:3872 Resync job RESYNC_20240513083609008 successfully submitted

After a while, when we checked the oem hearbeat status was showing OK.

E:\Agent13c\agent_inst\agent_13.5.0.0.0\bin>emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 5 Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. --------------------------------------------------------------- Agent Version          : 13.5.0.0.0 OMS Version            : 13.5.0.0.0 Protocol Version       : 12.1.0.1.0 Agent Home             : E:/Agent13c/agent_inst/agent_inst Agent Log Directory    : E:/Agent13c/agent_inst/agent_inst\sysman\log Agent Binaries         : E:\Agent13c\agent_inst\agent_13.5.0.0.0 Core JAR Location      : E:\Agent13c\agent_inst\agent_13.5.0.0.0\jlib Agent Process ID       : 2660 Parent Process ID      : 7172 Agent URL              : https://WMDR.com:3872/emd/main/ Local Agent URL in NAT : https://WMDR16.com:3872/emd/main/ Repository URL         : https://OEM-DBMGR.com:1159/empbs/upload Started at             : 2024-05-13 05:33:22 Started by user        : WMDR16$ Operating System       : Windows version 10.0 (amd64) Number of Targets      : 6 Last Reload            : (none) Last successful upload                       : 2024-05-13 05:37:19 Last attempted upload                        : 2024-05-13 05:37:19 Total Megabytes of XML files uploaded so far : 0.08 Number of XML files pending upload           : 3 Size of XML files pending upload(MB)         : 0 Available disk space on upload filesystem    : 44.15% Collection Status                            : Collections enabled Heartbeat Status                             : Ok Last attempted heartbeat to OMS              : 2024-05-13 05:36:42 Last successful heartbeat to OMS             : 2024-05-13 05:36:42 Next scheduled heartbeat to OMS              : 2024-05-13 05:37:43   --------------------------------------------------------------- Agent is Running and Ready

How to Resolve OEM Agent ‘Availability Evaluation Error’

 

Issue : While registering a database as a target in OEM 13c the agent was failing during start with following error  ‘Availability Evaluation Error’

Solution:-

To resolve the issue make I followed the following steps in following order .

Run this from db target server which you are trying to add as a target in OEM

cd $AGENT_HOME/agent_inst/bin $ ping omshost $ nslookup omshost $ emctl pingOMS $ emctl config agent addinternaltargets $ emctl secure agent $ emctl config agent listtargets $ emctl clearstate agent $ emctl upload agent $ emctl reload agent $ emctl start agent   The agent should show UP now, otherwise continue with the following steps:   $ emctl stop agent $ emctl secure agent $ emctl clearstate agent $ emctl secure agent $ emctl start agent $ emctl upload agent $ emctl reload agent $ emctl pingOMS $ emctl status agent

 

 

The user does not have the privileges to run commands as user "root" using the Privilege Delegation tool "/usr/bin/sudo"

While installing agent in target host for oracle enterprise manager 13c, following error is observed 

"The user  does not have the privileges to run commands as user "root" using the Privilege Delegation tool "/usr/bin/sudo" 

Solution:-

1. Ignore this warning and proceed with installation. Execute <agent Oracle Home>/root.sh manually after logging in as root.

OR

Ensure the sudo privileges for <AGENT_INSTALL_USER> to execute the command set in sudoers file.

a. Enable Sudo user setup on target host without password.

    As root user login to the target server:

# visudo
Locate the following line in the file mentioned
root    ALL=(ALL)       ALL

Add the following line below the line mentioned .Here r12devdb is the user that would be used to connect the server and continue with the installation.

r12devdb  ALL=(ALL)       ALL

 

2. Continue installation in console.