OS Watcher Installation in RAC

 Step:1 Download and untar the oswbb812.tar under the grid user in RAC on the both nodes. Follow the OS Watcher User's Guide (Doc ID 1531223.1)

/oraoci/app/oracle/OSW

oswbb         oswbb812.tar

Step: 2 Enable the private interconnet logs. create the private.net

grid@test:/oraoci/app/oracle/OSW/oswbb$ cat private.net

#Solaris Example

######################################################################

echo "zzz ***"`date`

traceroute -r -F 172.17.0.91

traceroute -I -r -F 172.17.0.92

traceroute -r -F 172.17.0.93

traceroute -I -r -F 172.17.0.94

######################################################################

# DO NOT DELETE THE FOLLOWING LINE!!!!!!!!!!!!!!!!!!!!!

######################################################################

rm locks/lock.file

Step:3 Start the OS watcher service.

30 --> Log generation in every 30 second

168 --> Log retention policy.

grid@test:/oraoci/app/oracle/OSW/oswbb$ nohup ./startOSWbb.sh 10 168 zip &

How to setup SSL certificate for apache

 Step:1 Generate key for Apache in linux.

[root@hostname test]# openssl genrsa -out postrga.key 2048

Generating RSA private key, 2048 bit long modulus

.............................+++

..............................................................+++

e is 65537 (0x10001)

[root@hostname test]# ls

postrga.key

Step:2 Generate csr from postrga key.

[root@hostname test]# openssl req -new -key postrga.key -out postrga.csr

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [XX]:QA

State or Province Name (full name) []:DOHA

Locality Name (eg, city) [Default City]:Doha

Organization Name (eg, company) [Default Company Ltd]:B bank

Organizational Unit Name (eg, section) []:IT

Common Name (eg, your name or your server's hostname) []:hostname

Email Address []:

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:welcome123

An optional company name []:NA

[root@hostname test]# ls -lrt

total 8

-rw-r-----. 1 root root 1679 Oct  5 09:22 postrga.key

-rw-r-----. 1 root root 1058 Oct  5 09:25 postrga.csr

Step:3 Validate key.

[root@hostname test]# openssl rsa -in postrga.key -check

RSA key ok

writing RSA key

-----BEGIN RSA PRIVATE KEY-----

MIIEpAIBAAKCAQEAzIXG3I02jOPsGdlqMj1h2U3/vCZoXfEdPY/0uZpM5FHjuo38

0UggfO88ZtMECgsPwEy2mmJ9w55S+/aJoojwdDyZgT2VOPIwRsheTh0zD9s374Y8

ZSTbmCSVIPeunw6N5iUW7qYH99VgLPo87hwbJauetDRxDF9A1OT0lTCxI/dO3dHb

/mYGdc1tbmpifENrgYZJmHKRHiKWVNDrzI9FyyjIUF2ziCFXQf0c+ZaIT0zj7L5H

Uv0rvJkFKfc6QQdbfryOtqhTVIpE81TQ9ygw3M0pARReo4pj4VyYWvw5/orCpmQ0

kE5w2j2aAkLbieWpMut4MwtEo0MzaePiDsZ7qwIDAQABAoIBABdEoPs2lZ4x0Aqg

0ZQlp5wxY4bkV7tYUxtN5Wm4REc1idMPlFO7APc1wIlyOGNsHr6h+6RU0DoRKR/B

TKdRLQoMWup5NyDjbNqgsRnx+H01U4fMndJj/xewEcdOZm9yGHK2hYVhHkbz350+

WvA8Alqmy8d0r+fYbmfKJIlUoNKWfC7lVTndg6WWCVqhJ/Dtgd8zHf9c1eH2XcOM

NNTTlLhpOc/TKyCeQChpbUeqLm71eJuMOzKHo71ZhuSUBWmQvnTZ3XgvRpJ/2nt9

ut5TTzz0XBS/x/UMoTOVBeCuAp3IiXfTWHQZ2tEeKHVOb/HIGf5To7XHydyT++PI

kgoLmgECgYEA6a2yQErTjLn//jkXpilC0EuDJPp1o92fstLPDokRqJJKLrf06Q5A

Adp5oEImjfqIxu0JiB6q5duHL0NxYF5Zi0XCdjYL4+NZq2GQMwz02VX61A9jQ6dj

oe2Y/GQp6HNFPJRsEH7PHDrJ7r7pIiazS2yCYsDaNWY/d3KbdibvrfMCgYEA4A8c

K6kKUhLRUPiIe/pecziWd3wa9hU9reFAwKlh0YxxM6dYe8nDpEhrq1/VZj2PHYNi

qUXNqbDR9u1KLXhxG9d0hcD/XLP9nQZ/up3nEKaPjKpq5qFZohUPpRt+tAXxxch3

CFhI+0Aa5NOYK6VkgLaPeAG6aS7OJ+wkg3ApEWkCgYEAq/nRfc+T8bXR2N9bsEYd

jf6E/en2OisZJRpQmHn4LP+86LUp4XtDUBnh+Te1+DeyAfThVvR7Ab+WtgoEns/R

RtaIT0dnaMghbifXYDrg2DKW7wqEG35NrRaXyOhOj2yGQBdbkKhC9Juvka2fFis1

OQUa0GLuNO6TNu15zHLLYlECgYEA0/r3uAbyELxk9Kpu1n0FpxU+pkTwbSmjdULL

tHuMZGjvIOOOv28gfelWlIjSqSxMnquTSHsWHRmBTJErwOFWgfTR9UmnT9mqdEmV

xsVlu0caQQdqpFC7MLGkX50nvsJw/6Ktm2OLiMoZp6zZJ1IPSAdwrSVVa/YwbWzp

DsGq2SECgYA3Q6U0ZQAyUZyYUVXXr1a6O1rtdLcZFQ2GhHTOEMnS10LsHULH20Oz

6iUsqs7Pq/cp24QJPix/9TJUGILJUDRKF9MLIHEwa8QgI1nZPCHUGfbvd6GM3bPi

Ox1ESgJahisOE+6uE2Y6kHbr3C7X/UF8sSZ3SGYy/VRybDT5DiQjCw==

-----END RSA PRIVATE KEY-----

Step: 4 Validate generate CSR.

[root@hostname test]# openssl req -text -noout -verify -in postrga.csr

verify OK

Certificate Request:

    Data:

        Version: 0 (0x0)

        Subject: C=QA, ST=DOHA, L=Doha, O=B bank, OU=IT, CN=hostname

        Subject Public Key Info:

            Public Key Algorithm: rsaEncryption

                Public-Key: (2048 bit)

                Modulus:

                    00:cc:85:c6:dc:8d:36:8c:e3:ec:19:d9:6a:32:3d:

                    61:d9:4d:ff:bc:26:68:5d:f1:1d:3d:8f:f4:b9:9a:

                    4c:e4:51:e3:ba:8d:fc:d1:48:20:7c:ef:3c:66:d3:

                    04:0a:0b:0f:c0:4c:b6:9a:62:7d:c3:9e:52:fb:f6:

                    89:a2:88:f0:74:3c:99:81:3d:95:38:f2:30:46:c8:

                    5e:4e:1d:33:0f:db:37:ef:86:3c:65:24:db:98:24:

                    95:20:f7:ae:9f:0e:8d:e6:25:16:ee:a6:07:f7:d5:

                    60:2c:fa:3c:ee:1c:1b:25:ab:9e:b4:34:71:0c:5f:

                    40:d4:e4:f4:95:30:b1:23:f7:4e:dd:d1:db:fe:66:

                    06:75:cd:6d:6e:6a:62:7c:43:6b:81:86:49:98:72:

                    91:1e:22:96:54:d0:eb:cc:8f:45:cb:28:c8:50:5d:

                    b3:88:21:57:41:fd:1c:f9:96:88:4f:4c:e3:ec:be:

                    47:52:fd:2b:bc:99:05:29:f7:3a:41:07:5b:7e:bc:

                    8e:b6:a8:53:54:8a:44:f3:54:d0:f7:28:30:dc:cd:

                    29:01:14:5e:a3:8a:63:e1:5c:98:5a:fc:39:fe:8a:

                    c2:a6:64:34:90:4e:70:da:3d:9a:02:42:db:89:e5:

                    a9:32:eb:78:33:0b:44:a3:43:33:69:e3:e2:0e:c6:

                    7b:ab

                Exponent: 65537 (0x10001)

        Attributes:

            unstructuredName         :NA

            challengePassword        :welcome123

    Signature Algorithm: sha256WithRSAEncryption

         0a:65:b6:45:ee:16:2d:f7:c4:8a:9c:22:56:a8:f9:dd:01:34:

         dc:dd:25:ba:91:7d:21:b1:d9:40:02:a7:9d:53:17:30:c6:06:

         36:0e:6f:ea:ab:04:47:69:ac:9e:a3:4f:0a:38:59:75:2f:95:

         62:a1:b2:eb:85:73:94:c2:55:c8:47:ea:c8:97:a7:a0:28:69:

         ff:e3:7e:e8:36:0b:01:f4:b7:46:97:76:b1:2a:14:8e:46:16:

         66:2b:f1:2e:53:a2:5a:11:c5:3c:25:03:dc:aa:fd:d3:a7:96:

         b8:97:b9:6d:36:ff:ef:72:83:58:24:e4:05:97:8c:fa:87:22:

         12:ca:e1:c6:f4:7d:40:dc:c3:1b:59:0e:60:7f:cd:87:af:4b:

         38:59:a6:50:5b:7d:bf:d6:65:97:d3:e2:30:6c:ab:d8:ee:cf:

         b8:c2:ca:9a:8d:2f:68:fa:05:92:17:6d:8e:67:7b:c3:d0:8b:

         57:f9:7f:c6:44:5d:12:4a:27:a2:c7:4f:b0:2e:43:df:f1:d4:

         c9:94:87:e9:5e:21:82:ed:d7:56:1d:5d:bb:70:81:72:f8:b6:

         f5:ad:8b:d1:2d:f5:38:68:33:b6:2e:0c:2c:33:f1:80:44:46:

         0c:8b:f0:5b:52:91:53:81:9d:74:f5:5c:7d:d6:14:af:3c:18:

         8d:55:18:a9

[root@hostname test]# ls

postrga.csr  postrga.key

Step: 5 Send csr file to Digicert authority to get the cert file.

Step: 6 Copy all the files in below path which is given by Digicert authority.

[root@hostname test]# ls /root/test/

postrga.csr  postrga.key

Step:6 Now you can configure this path /root/test/ in ssl.

How to generate DDL in Oracle Database

SYNTAX:

select dbms_metadata.get_ddl('TABLE','<TABLE_NAME>','<SCHEMA_NAME>') from dual;

COMMAND:

select dbms_metadata.get_ddl('TABLE','EMPLOYEE','HR') from dual;

ORA-20005: object statistics are locked (stattype = ALL)

 ACTIVITY:

Gather statistics on table levels:

SQL> EXEC DBMS_STATS.gather_table_stats('HR', 'CUSTOMER', estimate_percent => 15);

ERROR:

*

ERROR at line 1:

ORA-20005: object statistics are locked (stattype = ALL)

ORA-06512: at "SYS.DBMS_STATS", line 23829

ORA-06512: at "SYS.DBMS_STATS", line 23880

ORA-06512: at line 1

SOLUTIONS:

SQL>select owner,table_name,STATTYPE_LOCKED from dba_tab_statistics where table_name='CUSTOMER' and owner='HR';

OWNER        TABLE_NAME                     STATT

------------ ------------------------------ -----

HR     CUSTOMER ALL

Elapsed: 00:00:00.15

+++++++++ To unlock_table_stats  +++++++++++++

SQL>EXEC DBMS_STATS.unlock_table_stats('HR','CUSTOMER');

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.11

SQL>select owner,table_name,STATTYPE_LOCKED from dba_tab_statistics where table_name='CUSTOMER' and owner='HR';

OWNER        TABLE_NAME                     STATT

------------ ------------------------------ -----

HR     CUSTOMER

Elapsed: 00:00:00.01

SQL>EXEC DBMS_STATS.gather_table_stats('HR', 'CUSTOMER', estimate_percent => 15);

PL/SQL procedure successfully completed.

NodeManager is not coming up after SSL Setup

ERROR:

 After SSL implementation NodeManager is not startup.

java.lang.RuntimeException: Cannot convert identity certificate

  at com.certicom.tls.interfaceimpl.CertificateSupport.addAuthChain(Unknown Source)

  at com.certicom.net.ssl.SSLContext.addAuthChain(Unknown Source)

  at com.bea.sslplus.CerticomSSLContext.addIdentity(Unknown Source)

  at weblogic.security.utils.SSLContextWrapper.addIdentity(SSLContextWrapper.java:144)

  at weblogic.nodemanager.server.SSLListener.init(SSLListener.java:53)

  at weblogic.nodemanager.server.NMServer.start(NMServer.java:206)

  at weblogic.nodemanager.server.NMServer.main(NMServer.java:377)

  at weblogic.NodeManager.main(NodeManager.java:31)

 weblogic.nodemanager.server.NMServer main

SEVERE: Fatal error in node manager server

java.lang.RuntimeException: Cannot convert identity certificate

  at com.certicom.tls.interfaceimpl.CertificateSupport.addAuthChain(Unknown Source)

  at com.certicom.net.ssl.SSLContext.addAuthChain(Unknown Source)

  at com.bea.sslplus.CerticomSSLContext.addIdentity(Unknown Source)

  at weblogic.security.utils.SSLContextWrapper.addIdentity(SSLContextWrapper.java:144)

  at weblogic.nodemanager.server.SSLListener.init(SSLListener.java:53)

  at weblogic.nodemanager.server.NMServer.start(NMServer.java:206)

  at weblogic.nodemanager.server.NMServer.main(NMServer.java:377)

  at weblogic.NodeManager.main(NodeManager.java:31)

+ set +x

SOLUTIONS:

Configure Custom keystore in nodemanager.properties and enable JSSE for nodemanager.

1. Navigate to WL_HOME\server\bin

2. Take back-up of nodemanager.properties

3. Append below lines in nodemanager.properites.

----------------------------------------

KeyStores=CustomIdentityAndCustomTrust

CustomIdentityKeyStoreFileName=Identity_Keystore EX:/ofm/Oracle/Middleware/keystores/keystore.jks

CustomIdentityKeyStorePassPhrase=Identity_Keystore_Password

CustomIdentityAlias=Identity_Keystore_Alias EX: server_cert

CustomIdentityPrivateKeyPassPhrase=Private_Key_Used_When_Creating_Certificate EX: /ofm/Oracle/Middleware/keystores/keystore.jks

-------------------------------------------

4. Take back-up of startNodeManager.sh and edit the startNodeManager.sh with below JAVA_OPTIONS to enable JSSE.

-----------------------------

JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.security.SSL.enableJSSE=true"

export JAVA_OPTIONS

----------------------------------

5. Restart the Nodemanager and Admin server.

6. Once the nodemanager become Reachable in Admin console, start the Forms and Reports server.

SSL (HTTPS) Implementation in WebLogic and Managed servers.

 All the commands below reference $MIDDLEWARE_HOME for FMW 11g. If using FMW 12c, replace $MIDDLEWARE_HOME with $ORACLE_HOME.

1. Create a directory, for example: $MIDDLEWARE_HOME/keystores

cd /oracle/ofm/Oracle/Middleware ---> (MW_HOME)

$mkdir keystores

2. Run the following to set the environment on UNIX:

ofm11g@host:/ofm/Oracle/Middleware$ cd /ofm/Oracle/Middleware/user_projects/domains/PWCUAT/bin

$./setDomainEnv.sh

3. Create a keystore and private key, by executing the following command:

Syntax: CN=host.DOMAIN.com (Common Name), OU=organization  (Organization Unit), O=Organization (Organization), L=organization  Street (Locality Unit), ST=Doha, (State Province) C=QA (Country)"

ofm11g@host:/ofm/Oracle/Middleware/keystores$ keytool -genkey -alias server_cert -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -dname "CN=host.DOMAIN.com, OU=organization , O=Organization, L=organization  Street, ST=Doha, C=QA" -keypass password -keystore keystore.jks -storepass password

ofm11g@host:/ofm/Oracle/Middleware/keystores$ ls

keystore.jks

4. At this point take a backup of the keystore e.g: keystore.jks

ofm11g@host:/ofm/Oracle/Middleware/keystores$ cp -pr keystore.jks keystore.jks-org

5. To view the contents of the keystore created, execute the following command:

ofm11g@host:/ofm/Oracle/Middleware/keystores$ keytool -list -v -keystore keystore.jks -storepass password

6. Create a Certificate Signing Request (CSR) using the following command:

keytool -certreq -v -alias server_cert -file hpsuaterver.csr -sigalg SHA256withRSA -keypass password -storepass password -keystore keystore.jks

ofm11g@host:/ofm/Oracle/Middleware/keystores$ keytool -certreq -v -alias server_cert -file hpsuaterver.csr -sigalg SHA256withRSA -keypass password -storepass password -keystore keystore.jks

Certification request stored in file <hpsuaterver.csr>

Submit this to your CA

ofm11g@host:/ofm/Oracle/Middleware/keystores$ ls

hpsuaterver.csr   keystore.jks      keystore.jks-org

Make sure you use the same -alias, -storepass and -keypass passwords from Step 3.

The CSR (server.csr) created looks like this:

-----BEGIN NEW CERTIFICATE REQUEST-----

MIIxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

-----END NEW CERTIFICATE REQUEST-----

7. Send this CSR to a Certificate Authority (CA) of your choice. They will provide two certificate server certificate and root certificate.

8. Once you have received the Certificate back you will need to import this along with the Trusted Root CA certificate(s) that signed it, into your keystore.

Take the server certificate and save it a file called server.cer. Take the Certificate Authority's root certificate and save to a file called rootCA.cer in your keystore directory e.g $MIDDLEWARE_HOME/keystores. Repeat this step for any more Root CA certificates in the chain e.g rootCA2.cer etc..

ofm11g@host:/ofm/Oracle/Middleware/keystores$ mv certnew.cer server.cer

ofm11g@host:/ofm/Oracle/Middleware/keystores$ mv UATroot.cer rootCA.cer

ofm11g@host:/ofm/Oracle/Middleware/keystores$ ls -lrt

total 26

-rw-r--r--   1 ofm11g   hps         2240 Jul 26 14:28 keystore.jks-org

-rw-r--r--   1 ofm11g   hps         2240 Jul 26 14:28 keystore.jks

-rw-r--r--   1 ofm11g   hps         1035 Jul 26 14:39 hpsuaterver.csr

-rw-r--r--   1 ofm11g   hps         2090 Jul 27 13:40 server.cer

-rw-r--r--   1 ofm11g   hps         1328 Jul 27 13:40 rootCA.cer

ofm11g@host:/ofm/Oracle/Middleware/keystores$

9. Import the CA's root certificate into your keystore using the following command:

Syntax: keytool -import -v -noprompt -trustcacerts -alias <alias> -file <rootca_file> -keystore <keystore> -storepass <password>

ofm11g@host:/ofm/Oracle/Middleware/keystores$ keytool -import -v -noprompt -trustcacerts -alias rootcacert -file rootCA.cer -keystore keystore.jks -storepass password

Certificate was added to keystore

[Storing keystore.jks]

If there are other intermediate trust certificates, repeat this for each trust certificate using a different alias each time.

10. Import the Server Certificate into your keystore using the following command:

Syntax: keytool -import -v -noprompt -trustcacerts -alias <alias> -file <rootca_file> -keystore <keystore> -storepass <password>

ofm11g@host:/ofm/Oracle/Middleware/keystores$ keytool -import -v -alias server_cert -file server.cer -keystore keystore.jks -keypass password -storepass password

Certificate reply was installed in keystore

[Storing keystore.jks]

Make sure you use the same -alias from Step 3.

11. To view the contents of the keystore, execute the following command:

Syntax: keytool -list -v -keystore keystore.jks -storepass <PASSWORD>

ofm11g@host:/ofm/Oracle/Middleware/keystores$ keytool -list -v -keystore keystore.jks -storepass password

12. At this point the keystore is now ready for use. To use this Keystore with WLS please refer back to the Master Note for your version:

Configuring Oracle WebLogic Server (10.3.x - 12.1.x) to Use SSL in Fusion Middleware 11g/12c (Doc ID 1235653.1)

Step II: Configure WebLogic Server for SSL

The steps below take you through configuring SSL for a Managed Server.

The steps assumes the reader understands how to start the Admin Server and Managed Server.

1. Start the Admin Server in the Domain

2. Login to the WLS console e.g: http://10.0.00.11:7001/console

3. Select 'Environment' -> 'Servers' and click on the server you want to configure

4. Select the 'Keystores' tab

5. Select 'Keystore -> 'Change'

6.Select 'Custom Identity and Custome Trust'from the drop down list and click 'Save'

7. Enter the relevant information in the Keystores page:

'Custom Identity Keystore' : /ofm/Oracle/Middleware/keystores/keystore.jks

'Custom Identity Keystore' : JKS

'Custom Identity Keystore Passphrase' : password

'Confirm Custom Identity Keystore Passphrase' : password

'Custom Trust Keystore' : /ofm/Oracle/Middleware/keystores/keystore.jks

'Custom Trust Keystore Type' : JKS 

'Custom Trust Keystore Passphrase' : password

'Confirm Custom Trust Keystore Passphrase' : password

Click 'Save'

13. Select the 'SSL' tab and enter the relevant information:

'Private Key Alias' : server_cert

'Private Key Password' : password

'Confirm Private Key Password': password

Click 'Save'

14. Select 'Environment' -> 'Servers' and click on the Managed Server configured

 In the 'General' tab:

Check 'SSL Listen Port Enabled'

'SSL Listen Port' : <port> e.g 7012 (make sure this is not used by another process)

Click Save

And Click on adnvace and check the Use JSSE SSL and save the changes.

15. Implement the SSL for managed servers. Follow the steps from 12 to 15 for remaining servers.

16 . Click on Environment->Servers-> AdminServer or Managed Server-> Click on SSL -> Click on Advance --> HostnameVerification "Node" --> Click on save -> Activate the changes.

17. Add SSL parameter in nodemanager.

# Added following parameters in nodemanager.properties

KeyStores=CustomIdentityAndCustomTrust

CustomIdentityKeyStoreFileName=/ofm/Oracle/Middleware/keystores/keystore.jks

CustomIdentityKeyStorePassPhrase=password

CustomIdentityAlias=server_cert

CustomIdentityPrivateKeyPassPhrase=password

# added this parameter in startNodeManager.sh

JAVA_OPTIONS="-Dweblogic.security.SSL.enableJSSE=true ${JAVA_OPTIONS}"

export JAVA_OPTIONS

# Validate your certificate.

cd $DOMAIN_HOME/bin

-bash-3.2$ . ./setDomainEnv.sh

-bash-3.2$ java  utils.ValidateCertChain -jks server_cert /ofm/Oracle/Middleware/keystores/keystore.jks

Cert[0]: CN=host.DOMAIN.com,OU=organization ,O=Organization,L=organization  Street,ST=Doha,C=QA

Cert[1]: CN=Organization-BBSFADS001-CA,dc=Organization,dc=local

Certificate chain appears valid

17. Ask your network/security team to allow new SSL ports (7012/9011/9012) between your machine and server.

18. Restart all the weblogic server services.

19. Test the below new SSL URL should be open.

https://10.0.00.11:7012/console

20. Disable http port and restart the services.

1. In WebLogic Administration Console:

2. Click Lock and Edit.

3. Select Environment, Clusters, and select cluster_forms.

4. Select Configuration, and the Replication tab.

5. Select secure replication enabled.

6. Click Save.

7. Click Activate Changes.

Please do the above steps for cluster_reports also.

Note: Please take config.xml back-up before doing any changes in weblogic console.

++++++++++++++Start the Application  services. +++++++++++++====

echo "Starting up the AdminServer ..."

nohup $DOMAIN_HOME/bin/startWebLogic.sh  &

sleep 60

echo "Starting Node Manager ..."

nohup $WL_HOME/server/bin/startNodeManager.sh &

sleep 5

echo "Starting Forms Server 11G................."

nohup $DOMAIN_HOME/bin/startManagedWebLogic.sh WLS_FORMS https://10.0.00.12:7012 &

sleep 60

echo "Starting Reports Server 11G................."

nohup sh $DOMAIN_HOME/bin/startManagedWebLogic.sh WLS_REPORTS https://10.0.00.12:7012 &

sleep 90

echo "Starting OPMN ALL ............................"

opmnctl startall

=====================

Oracle Reference Documents:

=====================

Configuring Oracle WebLogic Server (10.3.x - 12.1.x) to Use SSL in Fusion Middleware 11g/12c (Doc ID 1235653.1)

How To Create a Java Keystore via Keytool in FMW 11g/12c (Doc ID 1230333.1)

After SSL Implementation in WebLogic Not listening for SSL, java.io.IOException: Cannot convert identity certificate

 ERROR:

<Jul 28, 2020 10:06:39 AM GMT+03:00> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias server_cert from the JKS keystore file /ofm/Oracle/Middleware/keystores/keystore.jks.>

<Jul 28, 2020 10:06:39 AM GMT+03:00> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>

<Jul 28, 2020 10:06:39 AM GMT+03:00> <Emergency> <Security> <BEA-090034> <Not listening for SSL, java.io.IOException: Cannot convert identity certificate.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Emergency> <Security> <BEA-090034> <Not listening for SSL, java.io.IOException: Cannot convert identity certificate.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Emergency> <Security> <BEA-090034> <Not listening for SSL, java.io.IOException: Cannot convert identity certificate.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Emergency> <Security> <BEA-090034> <Not listening for SSL, java.io.IOException: Cannot convert identity certificate.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Emergency> <Security> <BEA-090034> <Not listening for SSL, java.io.IOException: Cannot convert identity certificate.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Error> <WebLogicServer> <BEA-000297> <Inconsistent security configuration, java.lang.RuntimeException: Cannot convert identity certificate>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Emergency> <Security> <BEA-090034> <Not listening for SSL, java.io.IOException: Cannot convert identity certificate.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 172.30.0.130:7001 for protocols iiop, t3, ldap, snmp, http.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on 10.0.11.146:7001 for protocols iiop, t3, ldap, snmp, http.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on 172.30.0.66:7001 for protocols iiop, t3, ldap, snmp, http.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 10.0.11.144:7001 for protocols iiop, t3, ldap, snmp, http.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <Server> <BEA-002613> <Channel "Default[5]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <Server> <BEA-002613> <Channel "Default[4]" is now listening on 172.30.2.2:7001 for protocols iiop, t3, ldap, snmp, http.>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <WebLogicServer> <BEA-000329> <Started WebLogic Admin Server "AdminServer" for domain "PWCUAT" running in Production Mode>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>

<Jul 28, 2020 10:06:40 AM GMT+03:00> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>

https port is not coming up after SSL implementation in WebLogic.

SOLUTIONS:

1. Navigate to [managed server or Admin server ] > Configuration > SSL > Advanced.

2. Check "Use JSSE SSL".

3. Save and Activate changes.

No replication server channel for WLS_FORMS java.lang.AssertionError: No replication server channel for WLS_FORMS

 ERROR:

<Aug 23, 2020 11:56:43 AM GMT+03:00> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: No replication server channel for WLS_FORMS

java.lang.AssertionError: No replication server channel for WLS_FORMS

at weblogic.cluster.replication.ReplicationManagerServerRef.initialize(ReplicationManagerServerRef.java:128)

at weblogic.cluster.replication.ReplicationManagerServerRef.<clinit>(ReplicationManagerServerRef.java:84)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:170)

at weblogic.rmi.internal.BasicRuntimeDescriptor.getServerReferenceClass(BasicRuntimeDescriptor.java:469)

Truncated. see log file for complete stacktrace

SOLUTIONS:

1. In WebLogic Administration Console:

2. Click Lock and Edit.

3. Select Environment, Clusters, and select cluster_forms.

4. Select Configuration, and the Replication tab.

5. Select secure replication enabled.

6. Click Save.

7. Click Activate Changes.

Please do the above steps for cluster_reports also.

Note: Please take config.xml back-up before doing any changes in weblogic console.

ORA-32017: failure in updating SPFILE ORA-16179: incremental changes to "log_archive_dest_1" not allowed with SPFILE

 ERROR:

SQL> alter system set log_archive_dest_1='/oradata/archive' scope=spfile;

alter system set log_archive_dest_1='/oradata/archive' scope=spfile

*

ERROR at line 1:

ORA-32017: failure in updating SPFILE

ORA-16179: incremental changes to "log_archive_dest_1" not allowed with SPFILE

Solutions:

SQL> alter system set log_archive_dest_1='location=/oradata/archive' scope=both;

System altered.

Convert Oracle Database from No Archive Mode to Archive Mode

 Step:1 Check the current archive status.

SQL> archive log list;

Database log mode              No Archive Mode

Automatic archival             Disabled

Archive destination            USE_DB_RECOVERY_FILE_DEST

Oldest online log sequence     210

Current log sequence           212

Step:2 Change the archive log location.

SQL> alter system set log_archive_dest_1='location=/oradata/archive' scope=both;

System altered.

SQL> archive log list;

Database log mode              No Archive Mode

Automatic archival             Disabled

Archive destination            /oradata/archive

Oldest online log sequence     210

Current log sequence           212

Step:3 Stop the database and start in Mount mode.

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area 6157238272 bytes

Fixed Size                  8615744 bytes

Variable Size            1291847872 bytes

Database Buffers         4848615424 bytes

Redo Buffers                8159232 bytes

Database mounted.

Step:4 Change database from no archivelog to archivelog .

SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> archive log list;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /oradata/archive

Oldest online log sequence     210

Next log sequence to archive   212

Current log sequence           212

ORA-01591: lock held by in-doubt distributed transaction 167.3.155988

 ERROR:

ORA-02354: error in exporting/importing data

ORA-01591: lock held by in-doubt distributed transaction 167.3.155988

SOLUTIONS:

SQL> SELECT LOCAL_TRAN_ID, STATE FROM DBA_2PC_PENDING;

LOCAL_TRAN_ID          STATE

---------------------- ----------------

167.3.155988           prepared

SQL> execute DBMS_TRANSACTION.PURGE_LOST_DB_ENTRY('167.3.155988');

BEGIN DBMS_TRANSACTION.PURGE_LOST_DB_ENTRY('167.3.155988'); END;

*

ERROR at line 1:

ORA-06510: PL/SQL: unhandled user-defined exception

ORA-06512: at "SYS.DBMS_TRANSACTION", line 105

ORA-06512: at line 1

SQL> SELECT local_tran_id FROM dba_2pc_pending;

LOCAL_TRAN_ID

----------------------

167.3.155988

SQL> rollback force '167.3.155988';

Rollback complete.

SQL> commit;

Commit complete.

SQL> begin dbms_transaction.purge_lost_db_entry('167.3.155988');

  2  end;

  3  /

PL/SQL procedure successfully completed.

SQL> SELECT * FROM dba_2pc_pending;

no rows selected

SQL> commit;

Commit complete.

SQL> SELECT * FROM dba_2pc_pending;

no rows selected

Re-Initiate your process.

ERROR at line 1: ORA-03113: end-of-file on communication channel ERROR: ORA-03114: not connected to ORACLE

ERROR:
======
sqlplus -s APPS/***** @/mwdb-pl/ebsap/fs1/EBSapps/appl/ad/12.0.0/patch/115/sql/adsqlwrapper.sql '/mwdb-pl/ebsap/fs1/EBSapps/appl/ad/12.0.0/patch/115/sql/ADZDWRKR.sql &un_apps &systempwd CUTOVER 0 3 10'
Connected.
PL/SQL procedure successfully completed.
Connected.
Session altered.
PL/SQL procedure successfully completed.
declare
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
ERROR:
ORA-03114: not connected to ORACLE
ERROR:
ORA-03114: not connected to ORACLE
ERROR:
ORA-03114: not connected to ORACLE

CAUSE:
======

Database Auditing is Enabled.

SOLUTIONS:
=========

Disable Auditing with the following steps:

1- SHOW PARAMETER AUDIT

2- Check audit_trail='DB','EXTENDED'

none or false - Auditing is disabled.
db or true - Auditing is enabled, with all audit records stored in the database audit trial (SYS.AUD$).
db,extended - As db, but the SQL_BIND and SQL_TEXT columns are also populated.
xml- Auditing is enabled, with all audit records stored as XML format OS files.
xml,extended - As xml, but the SQL_BIND and SQL_TEXT columns are also populated.
os- Auditing is enabled, with all audit records directed to the operating system's audit trail.

3-  NOAUDIT ALL;

4-  alter system set audit_trail='NONE';

5- Reboot database

6- Retest the issue, it should go successful.

ORA-20000: Please manually fix the following invalid packages: "PACKAGE BODY SYSTEM.APPS_ARRAY_DDL"

ERROR:
======

ATTENTION: All workers either have failed or are waiting:

FAILED: file adinvset.pls on worker  1.

ERROR at line 1:
ORA-20000: Please manually fix the following invalid packages: "PACKAGE BODY
SYSTEM.APPS_ARRAY_DDL"
ORA-06512: at "SYSTEM.AD_INVOKER", line 776
ORA-06512: at "SYSTEM.AD_COMPILE", line 103
ORA-06512: at "SYSTEM.AD_INVOKER", line 762
ORA-06512: at line 2

SOLUTIONS:
=========

Please check list of invalid objects and compile.

SELECT owner, object_name FROM all_objects
WHERE object_name= 'APPS_ARRAY_DDL'
AND object_type = 'PACKAGE';

>> execute below steps from application.
SQL>show user
APPS
SQL>@/mwdb-pl/ebsap/fs1/EBSapps/appl/ad/12.0.0/admin/sql/adinvset.pls Superbarwa20 10 0 TRUE FALSE

Connect with sysdba
SQL>utlrp.sql

>> Restart failed worker from adctrl utility.

Autoconfig failed in EBS R12.1.3 with 19c database

ERROR:
======
WARNING: [CVM Error Report]
The following report lists errors encountered during CVM Phase
      <filename>  <return code where appropriate>
  /mwdb-pl/ebsap/fs1/EBSapps/appl/fnd/12.0.0/bin/txkCfgUtlfileDir.sh  1

 No. of scripts failed in CVM phase: 1

Database Tier:
==========

>>Set the PDB environment:

$. erpdev_>host>.env
$perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=$CONTEXT_FILE \ -oraclehome=$ORACLE_HOME -outdir=$ORACLE_HOME/appsutil/log -mode=syncUtlFileDir
Enter the full path of Oracle Home: /mwdb-pl/oracle/19.3.0.0/db_3
Enter the APPS Password:

SQL> select value from v$parameter where name='utl_file_dir';

VALUE
--------------------------------------------------------------------------------
/mwdb-pl/oracle/19.3.0.0/temp/erpdev,/mwdb-pl/oracle/19.3.0.0/db_3/appsutil/outbound/erpdev_bbhoeccdbdev,/tmp

Application Tier:
============

>> Validate the s_appltmp entries in run/patch context file environment.

OLD Entries:
=========
<APPLTMP oa_var="s_appltmp">/mwdb-pl/ebsap/fs1/inst/apps/erpdev_bbhoeccdbdev/appltmp</APPLTMP>
<APPLPTMP oa_var="s_applptmp" osd="UNIX">/usr/tmp</APPLPTMP>

NEW: Entries:
==========
<APPLRGF oa_var="s_applrgf">/mwdb-pl/ebsap/fs1/inst/apps/erpdev_bbhoeccdbdev/logs/appl/rgf</APPLRGF>
<APPLTMP oa_var="s_appltmp">/mwdb-pl/oracle/19.3.0.0/temp/erpdev</APPLTMP>

Execute adautoconfig.sh in application tier and sure it's completed successfully.

Error in adop session

ERROR:
======
    [ERROR]     Error occurred while executing <adpatch  workers=8    options=hotpatch    flags=autoskip   console=no interactive=no  defaultsfile=/mwdb-pl/ebsap/fs1/EBSapps/appl/admin/erpdev/adalldefaults.txt patchtop=/mwdb-pl/ebsap/fs_ne/EBSapps/patch/13543062 driver=u13543062.drv logfile=u13543062.log>
    [ERROR]     Please check the adpatch log files.
    [STATEMENT] Autopatch completed with errors/warnings. Please check logfiles
    [STATEMENT] SQL Statement:       update ad_adop_sessions
        set status='F'
         where adop_session_id = 2 and appltop_id = 2233 and node_name='bbhoeccdbdev'

    [STATEMENT] SQL Statement:         update ad_adop_sessions
        set apply_end_date=to_date('07-07-2020 12:25:38','DD-MM-YYYY HH24:MI:SS')
        where  adop_session_id=2

    [STATEMENT] APPLY Phase END TIME: 07-07-2020 12:25:38
    [STATEMENT] [START 2020/07/07 12:25:42] Unlocking sessions table
      [STATEMENT] SQL stmt: <       begin
          AD_ZD_ADOP.UNLOCK_SESSIONS_TABLE('bbhoeccdbdev',60,2);
       end;
[STATEMENT] Online patching tool cannot proceed when a previous patching session is incomplete
[STATEMENT] Please ensure no pending patching sessions exist before trying a new patch
[ERROR]     Unrecoverable error occured. Exiting the current session.
[STATEMENT] [START 2020/07/07 13:46:35] Unlocking sessions table
[STATEMENT] [END   2020/07/07 13:46:36] Unlocking sessions table
[STATEMENT] Log file: /adop_20200707_134510.log
[STATEMENT] [START 2020/07/07 13:46:41] Unlocking sessions table
[STATEMENT] [END   2020/07/07 13:46:43] Unlocking sessions table

CAUSE:
======
Node Name       Node Type       Phase       Status          Started                        Finished                       Elapsed
--------------- --------------- ----------- --------------- ------------------------------ ------------------------------ ------------
bbhoeccdbdev    master          APPLY       FAILED          07-JUL-20 12:24:49 +03:00      07-JUL-20 12:25:38 +03:00      0:00:49
                                PREPARE     NOT APPLICABLE
                                CUTOVER     NOT APPLICABLE
Session was in incomplete in ad_adop_sessions tables.

SOLUTIONS:
==========

>> Check the adop cycle status.

SQL>select adop_session_id,status from ad_adop_sessions where status='F';

Output:
adop_session_id status
2 F

>> Update the status to completed.

SQL>update ad_adop_sessions set status='C' where status='F';

SQL>commit;

$adop -status
Node Name       Node Type       Phase       Status          Started                        Finished                       Elapsed
--------------- --------------- ----------- --------------- ------------------------------ ------------------------------ ------------
bbhoeccdbdev    master          APPLY       ACTIVE          07-JUL-20 12:24:49 +03:00      07-JUL-20 12:25:38 +03:00      0:00:49

Now you can start the adop patch.

ORA-00604: error occurred at recursive SQL level 1 ORA-01450: maximum key length (3215) exceeded

ERROR:

======

SQL> alter index CSM.CSM_DASHBOARD_SEARCH_COLS_U1 rebuild online;

alter index CSM.CSM_DASHBOARD_SEARCH_COLS_U1 rebuild online

*

ERROR at line 1:

ORA-00604: error occurred at recursive SQL level 1

ORA-01450: maximum key length (3215) exceeded

SOLUTIONS:

==========

SQL> alter index CSM.CSM_DASHBOARD_SEARCH_COLS_U1 rebuild;

Index altered.

FAILED: File ADZDPREP.sql In Online Patching with ORA-44303: service name exists

ERROR:

======

sqlplus -s APPS/***** @/mwdb-pl/ebsap/fs1/EBSapps/appl/ad/12.0.0/patch/115/sql/adsqlwrapper.sql '/mwdb-pl/ebsap/fs1/EBSapps/appl/ad/12.0.0/patch/115/sql/ADZDPREP.sql &un_apps &pw_apps &systempwd'

Connected.

PL/SQL procedure successfully completed.

Connected.

Session altered.

PL/SQL procedure successfully completed.

Commit complete.

begin

*

ERROR at line 1:

ORA-44303: service name exists

ORA-06512: at "SYS.DBMS_SERVICE_ERR", line 21

ORA-06512: at "SYS.DBMS_SERVICE", line 316

ORA-06512: at "APPS.AD_ZD_PREP", line 388

ORA-01403: no data found

ORA-06512: at "APPS.AD_ZD_PREP", line 382

ORA-06512: at line 4

CAUSE:

======

Found to have ebs_patch already in all_services

SQL> select SERVICE_ID,NAME from all_services;

SERVICE_ID NAME

---------- ----------------------------------------------------------------

  1 SYS$BACKGROUND

  2 SYS$USERS

  5 ebs_patch

Online Enable patch will try to create the ebs_patch service in the all_services table. 

If the service is already present in all_services table, then the patch worker will fail while executing ADZDPREP.sql script with below error.

SOLUTIONS:

==========

Check the output of Below query

select SERVICE_ID,NAME from dba_services;

If ebs_patch exists then Please perform below action plan.

begin

  DBMS_SERVICE.STOP_SERVICE(service_name =>'ebs_patch');

  DBMS_SERVICE.DELETE_SERVICE(service_name =>'ebs_patch');

end;

Please check below query and make sure that "ebs_patch" does not exist.

select SERVICE_ID,NAME from dba_services;

If no entry found then commit the session.

Then, restart the patch.

ORA-20100: ORA-20100: File o0148195.tmp creation failed. after upgrade database to 19c

ERROR:
======

SQL> EXEC FND_STATS.GATHER_SCHEMA_STATISTICS ('ALL');
BEGIN FND_STATS.GATHER_SCHEMA_STATISTICS ('ALL'); END;

*
ERROR at line 1:
ORA-20100: ORA-20100: File o0148195.tmp creation failed.
File could not be opened or operated on as requested.
Action: Make sure the directory - /mwdb-pl/oracle/19.3.0.0/temp/erpdev - is a
valid directory with write permissions and is accessible from the database
server node
ORA-06512: at "APPS.FND_FILE", line 319
ORA-06512: at "APPS.FND_FILE", line 364
ORA-06512: at "APPS.FND_FILE", line 421
ORA-06512: at "APPS.FND_STATS", line 585
ORA-06512: at "APPS.FND_STATS", line 1044
ORA-06512: at "APPS.FND_STATS", line 1044
ORA-06512: at "APPS.FND_STATS", line 1275
ORA-06512: at "APPS.FND_STATS", line 1275
ORA-06512: at "APPS.FND_STATS", line 815
ORA-06512: at line 1

Solutions:
======

Step:1 Check UTL_FILE_DIR path sis exits.

SQL> select value from v$parameter where name='utl_file_dir';

VALUE
--------------------------------------------------------------------------------
/mwdb-pl/oracle/19.3.0.0/temp/erpdev,/mwdb-pl/oracle/19.3.0.0/db_3/appsutil/outb
ound/erpdev_host,/tmp


Step 2: Change the permission.

cd /mwdb-pl/oracle/19.3.0.0/temp
chmod 777 erpdev

Step: 3 Updating UTL_FILE_DIR in an Oracle E-Business Suite Instance on Oracle Database 19c (2525754.1)

oradev$perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=$CONTEXT_FILE \
-oraclehome=$ORACLE_HOME -outdir=$ORACLE_HOME/appsutil/log -mode=addUtlFileDir

Step: 4 Re-execute step.1

DB Upgrade to 19c using DBUA

ERROR:
======
Error in preupgrade tool execution. ERROR - Unable to run the preupgrade due to:ERROR - Unable to run preupgrade due to:
ORA-06512: at "SYS.UTL_FILE", line 106
ORA-06512: at "SYS.UTL_FILE", line 746
ORA-06512: at "SYS.DBMS_PREUP", line 3352
ORA-06512: at "SYS.DBMS_PREUP", line 10500
ORA-06512: at line 8

declare
*
ERROR at line 1:
ORA-29284: file read error
ORA-06512: at line 56

SOLUTION:
==========
1) unset the env variable ORA_NLS10
unset ORA_NLS10

Verify:
echo $ORA_NLS10

2) Re-run DBUA upgrade.

DB Pre-clone failed perl adpreclone.pl dbTier

ERROR:
======
$perl adpreclone.pl dbTier
Can't locate strict.pm in @INC (@INC contains: /mwdb-pl/oracle/11.2.0.4/db_2/perl/lib/5.8.3 /mwdb-pl/oracle/11.2.0.4/db_2/perl/lib/site_perl/5.8.3 /mwdb-pl/oracle/11.2.0.4/db_2/appsutil/perl ../lib/5.10.0/sun4-solaris-thread-multi-64 ../lib/5.10.0 ../lib/site_perl/5.10.0/sun4-solaris-thread-multi-64 ../lib/site_perl/5.10.0 .) at adpreclone.pl line 34.
BEGIN failed--compilation aborted at adpreclone.pl line 34.

Cause:
====
PERL5LIB path is not updated in context environment file.

Solutions:
======
Update PERL5LIB parmater value in erpdev_<Host>.env file.

OLD: PERL5LIB=/mwdb-pl/oracle/11.2.0.4/db_2/perl/lib/5.8.3:/mwdb-pl/oracle/11.2.0.4/db_2/perl/lib/site_perl/5.8.3:/mwdb-pl/oracle/11.2.0.4/db_2/appsutil/perl

NEW: PERL5LIB=/mwdb-pl/oracle/11.2.0.4/db_2/perl/lib/5.10.0:/mwdb-pl/oracle/11.2.0.4/db_2/perl/lib/site_perl/5.10.0:/mwdb-pl/oracle/11.2.0.4/db_2/appsutil/perl

Set the environment and execute  again command. perl adpreclone.pl dbTier

#############################################################
StageDBTier Finished at Sun Jun 14 13:28:47 GMT+03:00 2020
Status: Completed Successfully
#############################################################

APPLY PATCHES IN RAC GRID/ORACLE DB HOME

/*FOLLOWING STEPS TO APPLY PATCHES IN RAC GRID/ORACLE DB HOME ENVIRONMENT */

Grid Home: /orahps/12.2.0/grid

Oracle Home: /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome

>> Analyze patches from root user.

---- Using below command can analyze both home grid and DB Home in once
/orahps/12.2.0/grid/OPatch/opatchauto apply /rmanhps/software/patches/28828733 -analyze

-- Using below command can analyze grid patch only.

/orahps/12.2.0/grid/OPatch/opatchauto apply /rmanhps/software/patches/28828733 -analyze -oh /orahps/12.2.0/grid

-- Using below command can analyze Oracle DB Home only.

/orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome/OPatch/opatchauto apply /rmanhps/software/patches/28828733 -analyze -oh

/orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome

>> How to check rollingpatch method.

root@testdbn01:~# /orahps/12.2.0/grid/bin/crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1819134257].

Step:1 Create ocm response file using grid user.

/orahps/12.2.0/grid/OPatch/ocm/bin/emocmrsp -output /orahps/software/ocm.rsp

>> Stop crs from root user

/orahps/12.2.0/grid/bin/crsctl stop crs

/orahps/12.2.0/grid/crs/install/rootcrs.sh -prepatch

>> Give grid owner ship to PSU patches.

chown -R grid:oinstall /rmanhps/software/patches
su - grid

>> Check Opatch version from grid user.

/orahps/12.2.0/grid/OPatch/opatch version

>> Check applied patches from grid user.

/orahps/12.2.0/grid/OPatch/opatch lspatches -oh /orahps/12.2.0/grid

>> Login with Grid user and apply these patches in Grid home.

$ /orahps/12.2.0/grid/OPatch/opatch apply -oh /orahps/12.2.0/grid -local -silent /rmanhps/software/patches/28828733/28822515
$ /orahps/12.2.0/grid/OPatch/opatch apply -oh /orahps/12.2.0/grid -local -silent /rmanhps/software/patches/28828733/28870605
$ /orahps/12.2.0/grid/OPatch/opatch apply -oh /orahps/12.2.0/grid -local -silent /rmanhps/software/patches/28828733/28864846
$ /orahps/12.2.0/grid/OPatch/opatch apply -oh /orahps/12.2.0/grid -local -silent /rmanhps/software/patches/28828733/26839277
$ /orahps/12.2.0/grid/OPatch/opatch apply -oh /orahps/12.2.0/grid -local -silent /rmanhps/software/patches/28828733/28566910

>> Check patches are applied and not using below query.

/orahps/12.2.0/grid/OPatch/opatch lspatches -oh /orahps/12.2.0/grid

******************************* Process How to apply Patches in Oracle RAC Database Home ******************************

>> Change owner ship oracle for patches.

#chown -R oracle:oinstall /rmanhps/software/patches
#su - oracle

>> Check opatch version.

/orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome/OPatch/opatch version

>> Check applied patches.

/orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome/OPatch/opatch lspatches -oh /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome

>> Execute pre-patch command from oracle user.

$ /rmanhps/software/patches/28828733/28870605/custom/scripts/prepatch.sh -dbhome /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome

>> Applied Patches in oracle home from oracle user.

$ /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome/OPatch/opatch apply -oh /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome -local -silent /rmanhps/software/patches/28828733/28822515

$ /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome/OPatch/opatch apply -oh /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome -local -silent /rmanhps/software/patches/28828733/28870605

>> Execute postpatch command after applying patches.

$ /rmanhps/software/patches/28828733/28870605/custom/scripts/postpatch.sh -dbhome /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome 

/orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome/OPatch/opatch lspatches -oh /orahps/12.2.0/db/app/product/12.2.0/hpsrac_dbhome

**********   Below steps are applicable only after grid patching.

/orahps/12.2.0/grid/rdbms/install/rootadd_rdbms.sh

/orahps/12.2.0/grid/crs/install/rootcrs.sh -postpatch

/orahps/12.2.0/grid/bin/crsctl check crs

>> This command will lock invertory if its unlock.

GRID_CHECK=$(stat -c %U /orahps/12.2.0/grid); if [[ $GRID_CHECK == root ]]; then echo -e \\nPASS: Grid Home is locked\\n; else  echo -e \\nFAIL: Grid Home is not locked. Locking Grid Home now . . .\\n;/orahps/12.2.0/grid/crs/install/rootcrs.sh -unlock; /orahps/12.2.0/grid/crs/install/rootcrs.sh -patch; fi


/rmanhps/software/patches/28828733/28822515
/rmanhps/software/patches/28828733/28870605
/rmanhps/software/patches/28828733/28864846
/rmanhps/software/patches/28828733/26839277
/rmanhps/software/patches/28828733/28566910


*****************************  Applying Patches in 11GR2 ************************
>> Execute following steps from root user.

/orahps/11.2.0/grid/bin/crsctl stop crs

/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock

>> Change ownership of patch.

chown -R grid:oinstall /orahps/software
su - grid

>> Create Grid response file from grid user.

/orahps/11.2.0/grid/OPatch/ocm/bin/emocmrsp -output /orahps/software/ocm.rsp

>> Check opatch version and inventory backed up.

/orahps/11.2.0/grid/OPatch/opatch version

/orahps/11.2.0/grid/OPatch/opatch lsinventory  -oh /orahps/11.2.0/grid > /tmp/grid.lsinventory.before

/orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid

>> Apply following patches in Grid home from grid user.

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/24006111

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/23054319

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/22502505

/orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid

********* Applicable following below steps duing patching of grid and oracle home *******************

>> Execute following steps from root user.

/orahps/11.2.0/grid/rdbms/install/rootadd_rdbms.sh

/orahps/11.2.0/grid/crs/install/rootcrs.pl -patch

/orahps/11.2.0/grid/bin/crsctl check crs

>> This command will lock inventory if it is unlock.

GRID_CHECK=$(stat -c %U /orahps/11.2.0/grid); if [[ $GRID_CHECK == root ]]; then echo -e \\nPASS: Grid Home is locked\\n; else  echo -e \\nFAIL: Grid Home is not locked. Locking Grid Home now . . .\\n;/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock; /orahps/11.2.0/grid/crs/install/rootcrs.pl -patch; fi

How to Apply PSU/CPU Patching in Oracle RDBMS Home

Steps how to apply patches in oracle home 11GR2/2246888.1)

Step:1 Change ownership of patches. 

chown -R oracle:oinstall /u01/patches/BP
su - oracle

/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/opatch version

Step:2 Check versions before starting

/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/opatch lsinventory -oh /u04/app/orapoam/product/11.2.0.4/POAM > /tmp/database1.lsinventory.before
/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/opatch lspatches -oh /u04/app/orapoam/product/11.2.0.4/POAM

Step:3 Stop the database and listener before start below process

/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/ocm/bin/emocmrsp -output /u01/patches/BP/ocm.rsp

/u01/patches/BP/27980213/27441052/custom/server/27441052/custom/scripts/prepatch.sh -dbhome /u04/app/orapoam/product/11.2.0.4/POAM

Step:4 Applied patches.

/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/opatch napply -oh /u04/app/orapoam/product/11.2.0.4/POAM -local -silent -ocmrf /u01/patches/BP/ocm.rsp /u01/patches/BP/27980213/27734978

/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/opatch napply -oh /u04/app/orapoam/product/11.2.0.4/POAM -local -silent -ocmrf /u01/patches/BP/ocm.rsp /u01/patches/BP/27980213/27441052/custom/server/27441052

/u01/patches/BP/27980213/27441052/custom/server/27441052/custom/scripts/postpatch.sh -dbhome /u04/app/orapoam/product/11.2.0.4/POAM

>> Install OHVM patches if it is required.

Install OJVM
/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/opatch apply -oh /u04/app/orapoam/product/11.2.0.4/POAM -local -silent -ocmrf /u01/patches/BP/ocm.rsp /u01/patches/ONEOFF/27923163

Check Results
/u04/app/orapoam/product/11.2.0.4/POAM/OPatch/opatch lspatches -oh /u04/app/orapoam/product/11.2.0.4/POAM
exit

How to Apply Oracle 11gR2 GRID PSU/CPU Patching

**********************  Patching in GRID Home ***** Supplemental Readme - Grid Infrastructure Release Update 12.2.0.1.x / 18c (Doc ID 2246888.1)

Step:1 Stop the CRS services on node 1

/orahps/11.2.0/grid/bin/crsctl stop crs

Step:2 : Unlock inventory for patching.

/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock

Change patch permission.

chown -R grid:oinstall /orahps/software

su - grid

Step:3: 

/orahps/11.2.0/grid/OPatch/ocm/bin/emocmrsp -output /orahps/software/ocm.rsp

/orahps/11.2.0/grid/OPatch/opatch version

Step:4 Take exiting patch applied backup.

/orahps/11.2.0/grid/OPatch/opatch lsinventory  -oh /orahps/11.2.0/grid > /tmp/grid.lsinventory.before

/orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid

Step:5 Applying GRID Patch.

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/24006111

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/23054319

/orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/22502505

/orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid

exit

Step:6 Perform follwoing steps after patching.

/orahps/11.2.0/grid/rdbms/install/rootadd_rdbms.sh

/orahps/11.2.0/grid/crs/install/rootcrs.pl -patch

/orahps/11.2.0/grid/bin/crsctl check crs

Step: 7 This below command will check your Grid home is locked and not if not locked then automatically will locked. 

Note: This command has to be run only patching window.

GRID_CHECK=$(stat -c %U /orahps/11.2.0/grid); if [[ $GRID_CHECK == root ]]; then echo -e \\nPASS: Grid Home is locked\\n; else  echo -e \\nFAIL: Grid Home is not locked. Locking Grid Home now . . .\\n;/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock; /orahps/11.2.0/grid/crs/install/rootcrs.pl -patch; fi

******************  PATCH LOGS ****************************
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/ocm/bin/emocmrsp -output /orahps/software/ocm.rsp
OCM Installation Response Generator 10.3.7.0.0 - Production
Copyright (c) 2005, 2012, Oracle and/or its affiliates.  All rights reserved.

Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y
The OCM configuration response file (/orahps/software/ocm.rsp) was successfully created.

>> Applied patches.

grid@testbn03:~$ /orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/24006111
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /orahps/11.2.0/grid
Central Inventory : /orahps/app/oraInventory
   from           : /orahps/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-50-56AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   17478514  18031668  18522509  19121551  19769489  20299013  20760982  21352635  21948347  22502456  23054359  24006111

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/orahps/11.2.0/grid')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sdo, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.sdo.locator, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18031668' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.ldap.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.crs, 11.2.0.4.0...

Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18522509' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '19121551' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.ordim.client, 11.2.0.4.0...

Patching component oracle.ordim.jai, 11.2.0.4.0...
Applying sub-patch '19769489' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ] , [ oracle.sysman.agent, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.util, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20299013' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ] , [ oracle.rdbms.oci, 11.2.0.4.0 ] , [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.sysman.common, 10.2.0.4.5...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.sysman.common.core, 10.2.0.4.5...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20760982' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '21352635' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...
Applying sub-patch '21948347' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.5 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '22502456' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.rdbms.olap, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.oraolap.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Applying sub-patch '23054359' to OH '/orahps/11.2.0/grid'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '24006111' to OH '/orahps/11.2.0/grid'

Patching component oracle.sqlplus.ic, 11.2.0.4.0...

Patching component oracle.sqlplus, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
+ PATH=/bin:/usr/bin:/usr/ccs/bin
+ export PATH
+ lib=/orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so
+ makefile=/orahps/11.2.0/grid/sqlplus/lib/ins_sqlplus.mk
+ so_ext=so
+ target=dlopenlib
+ var=''
+ basename /orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so .so
+ libname=libsqlplus
+ dirname /orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so
+ dir=/orahps/11.2.0/grid/sqlplus/lib
+ [ var '=' dlopenlib ]
+ [ -f /orahps/11.2.0/grid/sqlplus/lib/libsqlplus.a ]
+ dir2=/orahps/11.2.0/grid/sqlplus/lib/
+ [ '' '!=' '' ]
+ make -f /orahps/11.2.0/grid/sqlplus/lib/ins_sqlplus.mk dlopenlib _FULL_LIBNAME=/orahps/11.2.0/grid/sqlplus/lib/libsqlplus.so _LIBNAME=libsqlplus _LIBDIR=/orahps/11.2.0/grid/sqlplus/lib/ _LIBNAME_LIBS='$(libsqlplusLIBS)' _LIBNAME_EXTRALIBS='$(libsqlplusEXTRALIBS)'
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver

Composite patch 24006111 successfully applied.
OPatch Session completed with warnings.
Log file location: /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-50-56AM_1.log

OPatch completed with warnings.
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/23054319
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2019, Oracle Corporation.  All rights reserved.

Oracle Home       : /orahps/11.2.0/grid
Central Inventory : /orahps/app/oraInventory
   from           : /orahps/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-52-19AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   23054319

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/orahps/11.2.0/grid')

Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying interim patch '23054319' to OH '/orahps/11.2.0/grid'

Patching component oracle.crs, 11.2.0.4.0...

OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ld: warning: symbol '_init' not found, but .init section exists - possible link-edit without using the compiler driver
ld: warning: symbol '_fini' not found, but .fini section exists - possible link-edit without using the compiler driver


Patch 23054319 successfully applied.
OPatch Session completed with warnings.
Log file location: /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-52-19AM_1.log

OPatch completed with warnings.
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch napply -oh /orahps/11.2.0/grid -local -silent -ocmrf /orahps/software/ocm.rsp /orahps/software/24436338/22502505
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /orahps/11.2.0/grid
Central Inventory : /orahps/app/oraInventory
   from           : /orahps/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.15
OUI version       : 11.2.0.4.0
Log file location : /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-53-01AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   22502505

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/orahps/11.2.0/grid')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying interim patch '22502505' to OH '/orahps/11.2.0/grid'

Patching component oracle.usm, 11.2.0.4.0...
Patch 22502505 successfully applied.
Log file location: /orahps/11.2.0/grid/cfgtoollogs/opatch/opatch2019-04-07_08-53-01AM_1.log

OPatch succeeded.
grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid
22502505;ACFS Patch Set Update : 11.2.0.4.160419 (22502505)
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
24006111;Database Patch Set Update : 11.2.0.4.161018 (24006111)

OPatch succeeded.

grid@testdbn03:~$ exit
logout
root@testdbn03:~# /orahps/11.2.0/grid/rdbms/install/rootadd_rdbms.sh

root@testdbn03:~# /orahps/11.2.0/grid/crs/install/rootcrs.pl -patch

Using configuration parameter file: /orahps/11.2.0/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

root@testdbn03:~# /orahps/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

root@testdbn03:~# GRID_CHECK=$(stat -c %U /orahps/11.2.0/grid); if [[ $GRID_CHECK == root ]]; then echo -e \\nPASS: Grid Home is locked\\n; else  echo -e \\nFAIL: Grid Home is not locked. Locking Grid Home now . . .\\n;/orahps/11.2.0/grid/crs/install/rootcrs.pl -unlock; /orahps/11.2.0/grid/crs/install/rootcrs.pl -patch; fi

PASS: Grid Home is locked

root@testdbn03:~#

grid@testdbn03:~$ /orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid
22502505;ACFS Patch Set Update : 11.2.0.4.160419 (22502505)
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
24006111;Database Patch Set Update : 11.2.0.4.161018 (24006111)

OPatch succeeded.
grid@testdbn03:~$ exit

grid@testdbn04:~$ /orahps/11.2.0/grid/OPatch/opatch lspatches  -oh /orahps/11.2.0/grid
22502505;ACFS Patch Set Update : 11.2.0.4.160419 (22502505)
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
24006111;Database Patch Set Update : 11.2.0.4.161018 (24006111)

OPatch succeeded.