How to Apply Oracle 11gR2 GRID PSU/CPU Patching

**********************  Patching in GRID Home ***** Supplemental Readme - Grid Infrastructure Release Update 12.2.0.1.x / 18c (Doc ID 2246888.1)

Step:1 Stop the CRS services on node 1

/orahps/11.2.0/grid/bin/crsctl stop crs

Step:2 : Unlock inventory for patching.

/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock

Change patch permission.

chown -R grid:oinstall /orahps/software

su - grid

Step:3: 

/orahps/11.2.0/grid/OPatch/ocm/bin/emocmrsp -output /orahps/software/ocm.rsp

/orahps/11.2.0/grid/OPatch/opatch version

Step:4 Take exiting patch applied backup.

/orahps/11.2.0/grid/OPatch/opatch lsinventory  -oh /orahps/11.2.0/grid > /tmp/grid.lsinventory.before

/orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid

Step:5 Applying GRID Patch.

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/24006111

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/23054319

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/22502505

/orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid

exit

Step:6 Perform follwoing steps after patching.

/orahps/11.2.0/grid/rdbms/install/rootadd_rdbms.sh

/orahps/11.2.0/grid/crs/install/rootcrs.pl -patch

/orahps/11.2.0/grid/bin/crsctl check crs

Step: 7 This below command will check your Grid home is locked and not if not locked then automatically will locked. 

Note: This command has to be run only patching window.

GRID_CHECK=$(stat -c %U /orahps/11.2.0/grid); if [[ $GRID_CHECK == root ]]; then echo -e \\nPASS: Grid Home is locked\\n; else  echo -e \\nFAIL: Grid Home is not locked. Locking Grid Home now . . .\\n;/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock; /orahps/11.2.0/grid/crs/install/rootcrs.pl -patch; fi

******************  PATCH LOGS ****************************
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/ocm/bin/emocmrsp -output /orahps/software/ocm.rsp
OCM Installation Response Generator 10.3.7.0.0 - Production
Copyright (c) 2005, 2012, Oracle and/or its affiliates.  All rights reserved.

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y
The OCM configuration response file (/orahps/software/ocm.rsp) was successfully created.

>> Applied patches.

grid@testbn03:~$ /orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/24006111
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /orahps/11.2.0/grid
Central Inventory : /orahps/app/oraInventory
   from           : /orahps/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-50-56AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   17478514  18031668  18522509  19121551  19769489  20299013  20760982  21352635  21948347  22502456  23054359  24006111

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/orahps/11.2.0/grid')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sdo, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.sdo.locator, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18031668' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.ldap.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.crs, 11.2.0.4.0...

Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18522509' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '19121551' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.ordim.client, 11.2.0.4.0...

Patching component oracle.ordim.jai, 11.2.0.4.0...
Applying sub-patch '19769489' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ] , [ oracle.sysman.agent, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.util, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20299013' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ] , [ oracle.rdbms.oci, 11.2.0.4.0 ] , [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.sysman.common, 10.2.0.4.5...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.sysman.common.core, 10.2.0.4.5...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20760982' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '21352635' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...
Applying sub-patch '21948347' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.5 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '22502456' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.rdbms.olap, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.oraolap.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Applying sub-patch '23054359' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '24006111' to OH '/orahps/11.2.0/grid'

Patching component oracle.sqlplus.ic, 11.2.0.4.0...

Patching component oracle.sqlplus, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
+ PATH=/bin:/usr/bin:/usr/ccs/bin
+ export PATH
+ lib=/orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so
+ makefile=/orahps/11.2.0/grid/sqlplus/lib/ins_sqlplus.mk
+ so_ext=so
+ target=dlopenlib
+ var=''
+ basename /orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so .so
+ libname=libsqlplus
+ dirname /orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so
+ dir=/orahps/11.2.0/grid/sqlplus/lib
+ [ var '=' dlopenlib ]
+ [ -f /orahps/11.2.0/grid/sqlplus/lib/libsqlplus.a ]
+ dir2=/orahps/11.2.0/grid/sqlplus/lib/
+ [ '' '!=' '' ]
+ make -f /orahps/11.2.0/grid/sqlplus/lib/ins_sqlplus.mk dlopenlib _FULL_LIBNAME=/orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so _LIBNAME=libsqlplus _LIBDIR=/orahps/11.2.0/grid/sqlplus/lib/ _LIBNAME_LIBS='$(libsqlplusLIBS)' _LIBNAME_EXTRALIBS='$(libsqlplusEXTRALIBS)'
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver

Composite patch 24006111 successfully applied.
OPatch Session completed with warnings.
Log file location: /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-50-56AM_1.log

OPatch completed with warnings.
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/23054319
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2019, Oracle Corporation.  All rights reserved.

Oracle Home       : /orahps/11.2.0/grid
Central Inventory : /orahps/app/oraInventory
   from           : /orahps/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-52-19AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   23054319

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/orahps/11.2.0/grid')

Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying interim patch '23054319' to OH '/orahps/11.2.0/grid'

Patching component oracle.crs, 11.2.0.4.0...

OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver


Patch 23054319 successfully applied.
OPatch Session completed with warnings.
Log file location: /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-52-19AM_1.log

OPatch completed with warnings.
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/22502505
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /orahps/11.2.0/grid
Central Inventory : /orahps/app/oraInventory
   from           : /orahps/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-53-01AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   22502505

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/orahps/11.2.0/grid')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying interim patch '22502505' to OH '/orahps/11.2.0/grid'

Patching component oracle.usm, 11.2.0.4.0...
Patch 22502505 successfully applied.
Log file location: /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-53-01AM_1.log

OPatch succeeded.
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid
22502505;ACFS Patch Set Update : 11.2.0.4.160419 (22502505)
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
24006111;Database Patch Set Update : 11.2.0.4.161018 (24006111)

OPatch succeeded.

grid@testdbn03:~$ exit
logout
root@testdbn03:~# /orahps/11.2.0/grid/rdbms/install/rootadd_rdbms.sh

root@testdbn03:~# /orahps/11.2.0/grid/crs/install/rootcrs.pl -patch

Using configuration parameter file: /orahps/11.2.0/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

root@testdbn03:~# /orahps/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

root@testdbn03:~# GRID_CHECK=$(stat -c %U /orahps/11.2.0/grid); if [[ $GRID_CHECK == root ]]; then echo -e \\nPASS: Grid Home is locked\\n; else  echo -e \\nFAIL: Grid Home is not locked. Locking Grid Home now . . .\\n;/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock; /orahps/11.2.0/grid/crs/install/rootcrs.pl -patch; fi

PASS: Grid Home is locked

root@testdbn03:~#

grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid
22502505;ACFS Patch Set Update : 11.2.0.4.160419 (22502505)
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
24006111;Database Patch Set Update : 11.2.0.4.161018 (24006111)

OPatch succeeded.
grid@testdbn03:~$ exit

grid@testdbn04:~$ /orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid
22502505;ACFS Patch Set Update : 11.2.0.4.160419 (22502505)
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
24006111;Database Patch Set Update : 11.2.0.4.161018 (24006111)

OPatch succeeded.