How to Start and Stop Local Listener and SCAN Listener (RAC Basic Command)

                      There are following RAC basic commands

How to Start and Stop Local Listener and SCAN Listener

First check which services are running from which nodes.

oragrid$ps -eaf|grep tns

---------How to set environment if environment is not set---------

bash-4.3$ which srvctl
/u02/grid/product/11.2.0/bin/srvctl

bash-4.3$ export PATH=$ORACLE_HOME/bin:$PATH

/u02/grid/product/11.2.0/bin/lsnrctl
--How to STOP Local Listener:- Run this command from one by one all nodes

oragrid$srvctl stop LISTENER -n adupgdb1
oragrid$srvctl stop LISTENER -n oracledb1
oragrid$srvctl stop LISTENER -n oracledb2

--How to STOP SCAN Listener  ---- Run only from any one node

oragrid$srvctl stop scan_listener

----- How to START Local Listener ------

oragrid$srvctl start LISTENER -n adupgdb1
oragrid$srvctl start LISTENER -n oracledb1

oragrid$srvctl start LISTENER -n oracledb2


-----How to start SCAN Listener ------

oragrid$srvctl start scan_listener

-----------------------------------------------------------------------

How to bounced Single SCAN Listener one by one on nodes

Run from the grid user.

bash-4.3$ srvctl status scan_listener -i 3
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node adupgdb1
bash-4.3$ srvctl stop scan_listener -i 3
bash-4.3$ srvctl status scan_listener -i 3
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is not running
bash-4.3$ srvctl start scan_listener -i 3
bash-4.3$ srvctl status scan_listener -i 3
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node adupgdb1
------------------------------------------------------------------

11gR2 Clusterware and Grid Home – What you need to know [ID 1053147.1] [ POST VIEWS : 15323 ]

1.

Clusterware Resource Status Check

bash-4.3# ./crsctl status resource -t

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.LISTENER.lsnr

               ONLINE  ONLINE       testdb1

               ONLINE  ONLINE       testdb2

ora.TESTARCH.dg

               ONLINE  ONLINE       testdb1

               ONLINE  ONLINE       testdb2

ora.TESTDATA.dg

               ONLINE  ONLINE       testdb1

               ONLINE  ONLINE       testdb2

ora.TESTDATA1.dg

               ONLINE  ONLINE       testdb1

               ONLINE  ONLINE       testdb2

ora.TEST_VOTE.dg

               ONLINE  ONLINE       testdb1

               ONLINE  ONLINE       testdb2

ora.asm

               ONLINE  ONLINE       testdb1                  Started

               ONLINE  ONLINE       testdb2                  Started

ora.gsd

               OFFLINE OFFLINE      testdb1

               OFFLINE OFFLINE      testdb2

ora.net1.network

               ONLINE  ONLINE       testdb1

               ONLINE  ONLINE       testdb2

ora.ons

               ONLINE  ONLINE       testdb1

               ONLINE  ONLINE       testdb2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       testdb1

ora.LISTENER_SCAN2.lsnr

      1        ONLINE  ONLINE       testdb2

ora.LISTENER_SCAN3.lsnr

      1        ONLINE  ONLINE       testdb2

ora.cvu

      1        ONLINE  ONLINE       testdb2

ora.oc4j

      1        OFFLINE OFFLINE

ora.scan1.vip

      1        ONLINE  ONLINE       testdb1

ora.scan2.vip

      1        ONLINE  ONLINE       testdb2

ora.scan3.vip

      1        ONLINE  ONLINE       testdb2

ora.test.db

      1        ONLINE  ONLINE       testdb1                  Open

      2        ONLINE  ONLINE       testdb2                  Open

ora.testdb1.vip

      1        ONLINE  ONLINE       testdb1

ora.testdb2.vip

      1        ONLINE  ONLINE       testdb2

2. STOP A and START A LISTENER

bash-4.3$ srvctl status listener -l LISTENER

Listener LISTENER is enabled

Listener LISTENER is running on node(s): testdb2,testdb1

bash-4.3$ srvctl stop listener -l LISTENER

bash-4.3$ srvctl status listener -l LISTENER

Listener LISTENER is enabled

Listener LISTENER is not running

bash-4.3$ srvctl start listener -l LISTENER

bash-4.3$ srvctl status listener -l LISTENER

Listener LISTENER is enabled

Listener LISTENER is running on node(s): testdb2,testdb1

3. stop all the clusterware services/ resources on specific node (including DB and listener) (run as root)

bash-4.3# ./crsctl stop has

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testdb1'

CRS-2673: Attempting to stop 'ora.crsd' on 'testdb1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'testdb1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'testdb1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'testdb1'

CRS-2673: Attempting to stop 'ora.TESTARCH.dg' on 'testdb1'

CRS-2673: Attempting to stop 'ora.TEST_VOTE.dg' on 'testdb1'

CRS-2673: Attempting to stop 'ora.test.db' on 'testdb1'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.testdb1.vip' on 'testdb1'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'testdb1'

CRS-2677: Stop of 'ora.testdb1.vip' on 'testdb1' succeeded

CRS-2672: Attempting to start 'ora.testdb1.vip' on 'testdb2'

CRS-2677: Stop of 'ora.scan1.vip' on 'testdb1' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'testdb2'

CRS-2677: Stop of 'ora.TESTARCH.dg' on 'testdb1' succeeded

CRS-2676: Start of 'ora.testdb1.vip' on 'testdb2' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'testdb2' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'testdb2'

CRS-2677: Stop of 'ora.test.db' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.TESTDATA.dg' on 'testdb1'

CRS-2673: Attempting to stop 'ora.TESTDATA1.dg' on 'testdb1'

CRS-2677: Stop of 'ora.TESTDATA1.dg' on 'testdb1' succeeded

CRS-2677: Stop of 'ora.TESTDATA.dg' on 'testdb1' succeeded

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'testdb2' succeeded

CRS-2677: Stop of 'ora.TEST_VOTE.dg' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'testdb1'

CRS-2677: Stop of 'ora.asm' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'testdb1'

CRS-2677: Stop of 'ora.ons' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'testdb1'

CRS-2677: Stop of 'ora.net1.network' on 'testdb1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'testdb1' has completed

CRS-2677: Stop of 'ora.crsd' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'testdb1'

CRS-2673: Attempting to stop 'ora.evmd' on 'testdb1'

CRS-2673: Attempting to stop 'ora.asm' on 'testdb1'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'testdb1'

CRS-2677: Stop of 'ora.ctssd' on 'testdb1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'testdb1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'testdb1' succeeded

CRS-2677: Stop of 'ora.asm' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'testdb1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'testdb1'

CRS-2677: Stop of 'ora.cssd' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'testdb1'

CRS-2677: Stop of 'ora.gipcd' on 'testdb1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'testdb1'

CRS-2677: Stop of 'ora.gpnpd' on 'testdb1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testdb1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

4. Start all the clusterware services/ resources on specific node (including DB and listener) (run as root)

bash-4.3# ./crsctl start has

CRS-4123: Oracle High Availability Services has been started.

5. to stop csr services on all nodes of clusterware (run as root)

bash-4.3# ./crsctl stop cluster –all

6. to start crs services on all nodes of clusterware (run as root)

bash-4.3# ./crsctl start cluster -all

7. to check if ohasd is running/ stopped (run as root)

bash-4.3# ./crsctl check has

CRS-4638: Oracle High Availability Services is online

8.  enable Oracle High Availability Services autostart (run as root)

bash-4.3# ./crsctl enable has

CRS-4622: Oracle High Availability Services autostart is enabled.

9.  disable Oracle High Availability Services autostart (run as root)

bash-4.3# ./crsctl disable has

CRS-4621: Oracle High Availability Services autostart is disabled.

10. check if Oracle High Availability Services autostart is enabled/ disabled (run

bash-4.3# ./crsctl config has

CRS-4622: Oracle High Availability Services autostart is enabled.

11.

Verifying scan status scan_listener

Run this through grid user.

bash-4.3$ ./cluvfy comp scan -verbose

Verifying scan

Checking Single Client Access Name (SCAN)...

  SCAN Name         Node          Running?      ListenerName  Port          Running?

  ----------------  ------------  ------------  ------------  ------------  ------------

  scan-test.seha.ae  testdb1       true          LISTENER_SCAN1  1521          true

  scan-test.seha.ae  testdb2       true          LISTENER_SCAN2  1521          true

  scan-test.seha.ae  testdb2       true          LISTENER_SCAN3  1521          true

Checking TCP connectivity to SCAN Listeners...

  Node          ListenerName              TCP connectivity?

  ------------  ------------------------  ------------------------

  testdb1       LISTENER_SCAN1            yes

  testdb1       LISTENER_SCAN2            yes

  testdb1       LISTENER_SCAN3            yes

TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "scan-test.seha.ae"...

  SCAN Name     IP Address                Status                    Comment

  ------------  ------------------------  ------------------------  ----------

  scan-test.seha.ae  10.105.22.133             passed

  scan-test.seha.ae  10.105.22.132             passed

  scan-test.seha.ae  10.105.22.131             passed

Verification of SCAN VIP and Listener setup passed

Verification of scan was successful.

12. Verifying scan port

bash-4.3$ srvctl config scan_listener

SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521

SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

13. To check the status of services on all nodes

bash-4.3$ srvctl status nodeapps

VIP testdb1_vip is enabled

VIP testdb1_vip is running on node: testdb1

VIP 10.104.21.56 is enabled

VIP 10.104.21.57 is running on node: testdb2

Network is enabled

Network is running on node: testdb1

Network is running on node: testdb2

GSD is disabled

GSD is not running on node: testdb1

GSD is not running on node: testdb2

ONS is enabled

ONS daemon is running on node: testdb1

ONS daemon is running on node: testdb2

14. stop all the clusterware services/ resources ON THAT NODE! (run as root)

bash-4.3# ./crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

15.  Check if Oracle High Availability Services autostart is enabled/ disabled (run as root)

bash-4.3# ./crsctl config has

CRS-4622: Oracle High Availability Services autostart is enabled.

How to Check Clusterware Version and Name

By JAGORACLE on Dec 14, 2011

To check the software versions on a single node. Software version is the lastest version installed in on an cluster.

You would use this option,When you do the rolling upgrade.

$ crsctl query crs softwareversion [hostname]

Active version is lowest version anywhere in the cluster. This is the command you would normally need to use

$ crsctl query crs activeversion

The version of oracle Clusterware must be always greater than the oracle products installed in the clusters

Permanently operating oracle Clusterware (software version vs active version) is not supported.

To check the cluster name use,

   $ORA_CRS_HOME/bin/cemutlo -n

bash-4.3# ./crsctl query crs softwareversion testdb1

Oracle Clusterware version on node [testdb1] is [11.2.0.3.0]

bash-4.3# ./crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [11.2.0.3.0]

/u02/appl/product/grid/11.2.0/bin

bash-4.3# ./cemutlo -n

scan-test

How to Start 11g R2 RAC Cluster

There are 2 methods.

Method 1:

1).Start the HAS (High Availability Service) on each node.

$ crsctl start has

2).To Start the Cluster on both the nodes.

$ crsctl start cluster –all

If OHASD is already running, you can start the Clusterware stack on any node from local node

Method 2:

1).Run the below command on each node. Starts both HAS and CRS

$ crsctl start crs

Noticed that just starting has also starts up the whole stack.

How to START and STOP Database services in Mount and Nomount mode.

oratest@srvctl start database -d TEST -o mount

oratest@srvctl start database -d TEST -o nomount

ERROR: ORA-27154: post/wait create failed ORA-27300: OS system dependent operation:semget failed with status: 28 ORA-27301: OS failure message: No space left on device ORA-27302: failure occurred at: sskgpsemsper

Hi,

Facing below error during startup the database.

ERROR:

ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper
SQL>

Solution:

Here's what's recommended for Oracle databases:

# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128

#sysctl -p

Try to startup the database again.

How to apply Opatch in Oracle Home in ERP R12

Please follow the below process to apply Opatch on Oracle Home.

[appltrn@trn ~]$ su - root
Password:
[root@trn ~]# su - appltrn
[appltrn@trn ~]$ . /u01/appltrn/TRN/inst/apps/TRN_trn/ora/10.1.3/TRN_trn.env
[appltrn@trn ~]$ echo $ORACLE_HOME
/u01/appltrn/TRN/apps/tech_st/10.1.3
[appltrn@trn ~]$ export PATH=$PATH:$PATH:/u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch
[appltrn@trn ~]$ echo $PATH
/u01/appltrn/TRN/apps/tech_st/10.1.3/bin:/u01/appltrn/TRN/inst/apps/TRN_trn/ora/                                                                                        ltrn/TRN/apps/tech_st/10.1.3/Apache/Apache/bin:/u01/appltrn/TRN/apps/tech_st/10.                                                                                        .1.3/appsutil/jdk/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/jdk/jre/bin:/u01/appl                                                                                        st/apps/TRN_trn/admin/scripts:/u01/appltrn/TRN/apps/tech_st/10.1.3/perl/bin:/u01                                                                                        u01/appltrn/TRN/apps/apps_st/appl/ad/12.0.0/bin:/u01/appltrn/TRN/apps/tech_st/10                                                                                        u01/appltrn/TRN/apps/tech_st/10.1.3/appsutil/jdk/jre/bin:/usr/kerberos/bin:/usr/                                                                                        appltrn/TRN/apps/tech_st/10.1.3/appsutil/jdk/bin:/u01/appltrn/TRN/apps/tech_st/1                                                                                        N/inst/apps/TRN_trn/ora/10.1.3/opmn/bin:/u01/appltrn/TRN/inst/apps/TRN_trn/ora/1                                                                                        trn/TRN/apps/tech_st/10.1.3/oui/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch:                                                                                        .3/jdk/jre/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/perl/bin:/usr/bin:/usr/ccs/b                                                                                        _st/10.1.3/perl/bin:/u01/appltrn/TRN/apps/tech_st/10.1.2/bin:/u01/appltrn/TRN/ap                                                                                        ltrn/TRN/apps/tech_st/10.1.3/appsutil/jdk/jre/bin::/u01/appltrn/TRN/apps/tech_st                                                                                        :/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/u01/appltrn/TRN/                                                                                        pltrn/TRN/apps/tech_st/10.1.3/ant/bin:/home/appltrn/bin:/u01/appltrn/TRN/apps/te
[appltrn@trn ~]$ cd /u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch/8411487/




[appltrn@trn 8411487]$ sh /u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch/opatch app                                                                                        ly -invPtrLoc /u01/appltrn/TRN/inst/apps/TRN_trn/admin/oraInst.loc

Oracle Interim Patch Installer version 1.0.0.0.57
Copyright (c) 2007 Oracle Corporation. All Rights Reserved..

We recommend you refer to the OPatch documentation under
OPatch/docs for usage reference. We also recommend using
the latest OPatch version. For the latest OPatch version
and other support related issues, please refer to document
293369.1 which is viewable from metalink.oracle.com

Oracle Home           : /u01/appltrn/TRN/apps/tech_st/10.1.3
Oracle Home Inventory : /u01/appltrn/TRN/apps/tech_st/10.1.3/inventory
Central Inventory     : /u01/appltrn/TRN/inst/apps/TRN_trn/admin/oraInventory
   from               : /u01/appltrn/TRN/inst/apps/TRN_trn/admin/oraInst.loc
OUI location          : /u01/appltrn/TRN/apps/tech_st/10.1.3/oui
OUI shared library    : /u01/appltrn/TRN/apps/tech_st/10.1.3/oui/lib/linux/libor                                                                                        aInstaller.so
Java location         : /u01/appltrn/TRN/apps/tech_st/10.1.3/jre/1.4.2/bin/java
Log file location     : /u01/appltrn/TRN/apps/tech_st/10.1.3/.patch_storage/<pat                                                                                        ch ID>/*.log

Creating log file "/u01/appltrn/TRN/apps/tech_st/10.1.3/.patch_storage/8411487/A                                                                                        pply_8411487_08-07-2013_19-31-04.log"

Invoking fuser to check for active processes.

Backing up comps.xml ...

OPatch detected non-cluster Oracle Home from the inventory and will patch the lo                                                                                        cal system only.


Please shut down Oracle instances running out of this ORACLE_HOME
(Oracle Home = /u01/appltrn/TRN/apps/tech_st/10.1.3)
Is this system ready for updating?
Please respond Y|N >
Y
Applying patch 8411487...

Patching jar files...



Inventory is good and does not have any dangling patches.


Updating inventory...

Verifying patch...
Backing up comps.xml ...


OPatch succeeded.
[appltrn@trn 8411487]$


Second:-2

[appltrn@trn OPatch]$ cd 8467230/
[appltrn@trn 8467230]$ . /u01/appltrn/TRN/inst/apps/TRN_trn/ora/10.1.3/TRN_trn.env
[appltrn@trn 8467230]$ echo $ORACLE_HOME
/u01/appltrn/TRN/apps/tech_st/10.1.3
[appltrn@trn 8467230]$ export PATH=$PATH:/u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch/
[appltrn@trn 8467230]$ echo $PATH
/u01/appltrn/TRN/apps/tech_st/10.1.3/bin:/u01/appltrn/TRN/inst/apps/TRN_trn/ora/10.1.3/opmn/bin:/u01/appltrn/TRN/inst/apps/TRN_trn/ora/10.1.3/Apache/Apache/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/Apache/Apache/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/oui/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch:/u01/appltrn/TRN/apps/tech_st/10.1.3/appsutil/jdk/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/jdk/jre/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/perl/bin:/usr/bin:/usr/ccs/bin:/usr/sbin:/u01/appltrn/TRN/inst/apps/TRN_trn/admin/scripts:/u01/appltrn/TRN/apps/tech_st/10.1.3/perl/bin:/u01/appltrn/TRN/apps/tech_st/10.1.2/bin:/u01/appltrn/TRN/apps/apps_st/appl/fnd/12.0.0/bin:/u01/appltrn/TRN/apps/apps_st/appl/ad/12.0.0/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/appsutil/jdk/jre/bin::/u01/appltrn/TRN/apps/tech_st/10.1.2/bin:/usr/bin:/usr/sbin:/u01/appltrn/TRN/apps/tech_st/10.1.3/appsutil/jdk/jre/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/u01/appltrn/TRN/apps/tech_st/10.1.2/OPatch:/u01/appltrn/TRN/apps/tech_st/10.1.3/appsutil/jdk/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/ant/bin:/home/appltrn/bin:/u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch/
[appltrn@trn 8467230]$ sh /u01/appltrn/TRN/apps/tech_st/10.1.3/OPatch/opatch apply -invPtrLoc /u01/appltrn/TRN/inst/apps/TRN_trn/admin/oraInst.loc

Oracle Interim Patch Installer version 1.0.0.0.57
Copyright (c) 2007 Oracle Corporation. All Rights Reserved..

We recommend you refer to the OPatch documentation under
OPatch/docs for usage reference. We also recommend using
the latest OPatch version. For the latest OPatch version
and other support related issues, please refer to document
293369.1 which is viewable from metalink.oracle.com

Oracle Home           : /u01/appltrn/TRN/apps/tech_st/10.1.3
Oracle Home Inventory : /u01/appltrn/TRN/apps/tech_st/10.1.3/inventory
Central Inventory     : /u01/appltrn/TRN/inst/apps/TRN_trn/admin/oraInventory
   from               : /u01/appltrn/TRN/inst/apps/TRN_trn/admin/oraInst.loc
OUI location          : /u01/appltrn/TRN/apps/tech_st/10.1.3/oui
OUI shared library    : /u01/appltrn/TRN/apps/tech_st/10.1.3/oui/lib/linux/liboraInstaller.so
Java location         : /u01/appltrn/TRN/apps/tech_st/10.1.3/jre/1.4.2/bin/java
Log file location     : /u01/appltrn/TRN/apps/tech_st/10.1.3/.patch_storage/<patch ID>/*.log

Creating log file "/u01/appltrn/TRN/apps/tech_st/10.1.3/.patch_storage/8467230/Apply_8467230_08-07-2013_19-44-20.log"

Invoking fuser to check for active processes.


Subset patches:  7692665,

The fixes for Patch 7692665,  are included in the patch currently
being installed (8467230).  OPatch will roll back the subset patch(es) and
install the new patch (8467230).


Backing up comps.xml ...

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.


Please shut down Oracle instances running out of this ORACLE_HOME
(Oracle Home = /u01/appltrn/TRN/apps/tech_st/10.1.3)
Is this system ready for updating?
Please respond Y|N >
Y
Rolling back patch 7692665...

Creating log file "/u01/appltrn/TRN/apps/tech_st/10.1.3/.patch_storage/7692665/RollBack_7692665_08-07-2013_19-44-27.log"

Rolling back with all-node mode.
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Oracle instances have been shut down, proceeding with auto-rollback.

Removing patch 7692665...

Restoring copied files...


Updating inventory...

Back to applying patch 8467230...

Backing up 2nd copy of comps.xml ...

Applying patch 8467230...

Patching copy files...



Inventory is good and does not have any dangling patches.


Updating inventory...

Verifying patch...
Backing up comps.xml ...


OPatch succeeded.
[appltrn@trn 8467230]$

Error: ORA-01536: Space Quota Exceeded For 'APPS_TS_TX_IDX or APPS_TS_MEDIA

Cause:

The issue is caused by a missing APPS and/or APPLSYS space quota in table dba_ts_quotas. This is justified by the fact that any of the following query returns no records, confirming that the APPS and/or APPLSYS user has no quota on the tablespace APPS_TS_TX_IDX:

select * from dba_ts_quotas
where username in ('APPS','APPLSYS')
and tablespace_name = 'APPS_TS_TX_IDX';

Solution:

1. Connect to SQL Plus

2. Execute the following command to give unlimited quota on the problematic tablespace for the user (APPS and/or APPLSYS) that returned no rows when querying dba_ts_quotas.

If APPS user returned no rows execute the following command:

alter user apps quota unlimited on APPS_TS_TX_IDX;

If APPLSYS user returned no rows execute the following command:

alter user applsys quota unlimited on APPS_TS_TX_IDX;

Concurrent Issue in ERP R12 ...Not coming up

In ERP environment concurrent is not coming up after run the CM clean. and node name is not showing as well. Please perform the below action plan.

Solution:

Step. 1 Stop the Concurrent manager.

Step.2  Take backup of the fnd_concurrent_queues table.

create table fnd_concurrent_queues_org as select * from fnd_concurrent_queues

Step.3 Check the target value.

select * from fnd_concurrent_queues

Step.4

update fnd_concurrent_queues

set TARGET_NODE='ERPTEST1'

where CONCURRENT_QUEUE_NAME='SEHA REQ MLR'  ----> Or you can set the target value for all the concurrent Queue name.

Step.5 Start the concurrent manager.

==========================================================

ORA-1031: Insufficient Privileges While Connecting As SYSDBA

I faced this issue after the Oracle Home Clone.There are many cause of this issue. Please check the below.

1. Check the entry in /etc/oratab
SID=$ORACLE_HOME:Y
2.Make sure that sqlnet.ora contains this entry: 
SQLNET.AUTHENTICATION_SERVICES = (ALL)

Issues during upgrade from R12.2.0. to R12.2.4.. When following document 1617461.1 to apply 18288881:R12.TXK.C.Delta.5 merged with Patch 19445058:R12.TXK.C

Issue:

When following document 1617461.1 to apply 18288881:R12.TXK.C.Delta.5 merged with Patch 19445058:R12.TXK.C

Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 (Doc ID 1617461.1)
  Section 3: Apply the R12.AD.C.Delta.5 and R12.TXK.C.Delta.5 Release Update Packs
     3.1: Path A - Upgrade and New Installation Customers
        6 On the run edition application tier file system, use hotpatch mode to apply Patch 18288881:R12.TXK.C.Delta.5 
           merged with Patch 19445058:R12.TXK.C (plus additional critical patches).
           $ adop phase=apply patches=18288881,19445058 hotpatch=yes merge=yes

Solution:

1. Source the Database tier environment file.

2.Unzip Patch 17537119: EBS Technology Codelevel Checker:
  unzip p17537119_R12_GENERIC.zip
  Confirm the following files are present after unzipping.
    checkDBpatch.sh
    checkDBpatch.cmd
    txk_R1220_DB_base_bugs.xml
    README.txt

3. Run checkDBpatch.sh.

4. Check if TXK_TCC_RESULTS gets installed using the following select:

SQL> select count(table_name) from all_tables where table_name='TXK_TCC_RESULTS';

5. Re-apply Patch 18288881:R12.TXK.C.Delta.5 merged with Patch 19445058:R12.TXK.C

Alert are not getting fired from concurrent programs in R12 (12.1.3)

Issue:

Alert are not getting fired from concurrent programs in R12 (12.1.3)

Alert: Version : 12.0.0

Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.

ALECDC module: Check Periodic Alert
+---------------------------------------------------------------------------+

Current system time is 15-JUL-2016 12:30:16

+---------------------------------------------------------------------------+

/u01/devap/apps/apps_st/appl/alr/12.0.0/bin/ALECDC
Program was terminated by signal 11


+---------------------------------------------------------------------------+
No completion options were requested.

Output file size: 
0


Output is not being printed because:
The print option has been disabled for this report.

+---------------------------------------------------------------------------+
Concurrent request completed
Current system time is 15-JUL-2016 12:30:16

Solution:

Please download patch 13728376 from metalink and apply it.

OPP is not getting UP and All XML report is giving Error in apps R12

Issue: OPP is not getting UP. XML Reports are going in Error.

Solutions:

This is process how to purge the FND_CP_GSM_OPP_AQTBL manually to clean up the table:

1. Shut down the concurrent managers cleanly.

2. Check the number of records in the table :

SQL> select count(*) from applsys.FND_CP_GSM_OPP_AQTBL ;

Output: Millions of rows.

3. Run the following as SYS:

DECLARE
po dbms_aqadm.aq$_purge_options_t;
BEGIN
po.block := FALSE;
DBMS_AQADM.PURGE_QUEUE_TABLE(
queue_table => 'APPLSYS.FND_CP_GSM_OPP_AQTBL',
purge_condition => NULL,
purge_options => po);
END;
/

4. Re-check again the number of records in the table:

SQL> select count(*) from applsys.FND_CP_GSM_OPP_AQTBL ;

5. Run the script $FND_TOP/patch/115/sql/afopp001.sql as the SYSTEM user.

SQL> @$FND_TOP/patch/115/sql/afopp001.sql
Enter value for 1: SYSTEM
Enter value for 2: manager
Connected.

PL/SQL procedure successfully completed.

Grant succeeded.

Commit complete.

6. Run the script $FND_TOP/patch/115/sql/afopp002.sql as the APPLSYS user.

SQL> @$FND_TOP/patch/115/sql/afopp002.sql
Enter value for 1: applsys
Enter value for 2: appssup
Connected.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

Commit complete.

7. Restart the concurrent manager and retest.

In case the purge did not complete successfully after the second time, or did not purge all the queues,
then you would have to recreate the queue. Recreate the queue using $FND_TOP/patch/115/sql/afopp002.sql file as 'APPLSYS' user.
On running the script you will be prompted for username and password. Please note that this may take a longer length of time to complete.

8. Run cmclean twice ensuring commit after each run - the script is available in note 134007.1

9. Start up the application services and retest.

*********************************************************************************

Check Locked Session and Concurrent Request in EBS R12

                  /* Use the below query to check Locked Sessions and Concurrent Request */

SQL>select
b.process " Process id ",
b.osuser,
b.machine,
b.sid,
b.serial#,
b.schemaname,
b.module,
b.action,
trunc(b.SECONDS_IN_WAIT/60/60,3) " Waiting Time in H ",
blocking_session,
'*** KILL ONLY IF BLOCKING_SESSION IS NULL, otherwise trace that blocking session till has null *** -- alter system kill session '''||b.sid||','||b.SERIAL#||''' immediate;' " SQL kill session statement" ,
'*** KILL ONLY IF BLOCKING_SESSION IS NULL, otherwise trace that blocking session till has null --- kill -9 '||b.process " OS kill process command",
b.PADDR,
p.SPID,
b.PROCESS
from v$session b , v$process p
where b.PADDR=p.ADDR and b.sid in (select BLOCKING_SESSION from v$session s where  BLOCKING_SESSION is not null);

*********************************************************************************

Check FND User Password in EBS R12

      /*Use the below query to check FND password in ERP R12*/

SQL>SELECT usr.user_name,
       get_pwd.decrypt
          ((SELECT (SELECT get_pwd.decrypt
                              (fnd_web_sec.get_guest_username_pwd,
                               usertable.encrypted_foundation_password
                              )
                      FROM DUAL) AS apps_password
              FROM fnd_user usertable
             WHERE usertable.user_name =
                      (SELECT SUBSTR
                                  (fnd_web_sec.get_guest_username_pwd,
                                   1,
                                     INSTR
                                          (fnd_web_sec.get_guest_username_pwd,
                                           '/'
                                          )
                                   - 1
                                  )
                         FROM DUAL)),
           usr.encrypted_user_password
          ) PASSWORD
  FROM fnd_user usr
 WHERE usr.user_name = '&USER_NAME';

***********************************************************************************************

How to ADD DISK GROUP in ORACLE ASM

How to ADD DISK GROUP in ORACLE ASM

Step.1  System Admin will add the disk on the server which should be visible on all the database nodes.

DBA can check using the below command on the path.

Example:  DISK PATH

/dev/oracleasmdisk

bash-4.3# ll

crwxrwxrwx   1 oragrid    oinstall    13 0x000011 Mar 29 11:31 archivdisk

crwxrwxrwx   1 oragrid    oinstall    13 0x000012 Mar 27 17:05 prddata1disk

crwxrwxrwx   1 oragrid    oinstall    13 0x000027 Mar 23 07:11 prddata2disk

crwxrwxrwx   1 oragrid    oinstall    13 0x00002f Mar 29 08:24 prddata3disk   >>> New Added Disk

crwxrwxrwx   1 oragrid    oinstall    13 0x000026 May  1  2014 prdflash

crwxrwxrwx   1 oragrid    oinstall    13 0x000014 Mar 20 21:02 prdinfradisk

crwxrwxrwx   1 oragrid    oinstall    13 0x000013 Mar 27 17:05 proddatadisk

We check the same disk on all database instance.
Step.2 Login on any database instance using grid user and connect in ASM using SQLPLUS.

Example:-

bash-4.3$ id

uid=112(oragrid) gid=107(oinstall) groups=503(asmadmin),504(asmdba),505(asmoper)

bash-4.3$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Tue Mar 29 11:56:33 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Real Application Clusters and Automatic Storage Management options              

SQL> select name, path, header_status from v$asm_disk;

NAME	PATH	HEADER_STATUS
PRODARCH_0000	/dev/oracleasmdisk/prddata3disk	CANDIDATE	After add disk Header status
PRODDATA1_0000	/dev/oracleasmdisk/archivdisk	MEMBER
PRODINFRA_0000	/dev/oracleasmdisk/prddata1disk	MEMBER
PRODDATA_0000	/dev/oracleasmdisk/prdinfradisk	MEMBER
PRODFLASH1	/dev/oracleasmdisk/prdflash	MEMBER
PRODDATA_0001	/dev/oracleasmdisk/prddata2disk	MEMBER

Step.3 Login using oragrid user and connect through sqlplus / as sysasm . Please follow the below steps.

Example:

SQL> CREATE DISKGROUP PRODDATA2 EXTERNAL REDUNDANCY DISK '/dev/oracleasmdisk/prddata3disk';

 >>>>>.. Execute it on one node

SQL>ALTER DISKGROUP PRODDATA2 mount;

<< Run the mount command in the other remaining two nodes.

Step.4 After add the disk in ASM check the header status result should be like below example.

SQL> select name, path, header_status from v$asm_disk;

NAME	PATH	HEADER_STATUS
PRODARCH_0000	/dev/oracleasmdisk/prddata3disk	MEMBER	Now header status should be member
PRODDATA1_0000	/dev/oracleasmdisk/archivdisk	MEMBER
PRODINFRA_0000	/dev/oracleasmdisk/prddata1disk	MEMBER
PRODDATA_0000	/dev/oracleasmdisk/prdinfradisk	MEMBER
PRODFLASH1	/dev/oracleasmdisk/prdflash	MEMBER
PRODDATA_0001	/dev/oracleasmdisk/prddata2disk	MEMBER

Step.5 Check the added disk in ASM is it mounted or not. You need to check on all 3 nodes.

bash-4.3$ asmcmd lsdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N        1024   4096  1048576    819200   353910                0          353910              0             N  PRODARCH/

MOUNTED  EXTERN  N        1024   4096  1048576   2099200    63706                0           63706              0             N  PRODDATA/

MOUNTED  EXTERN  N        1024   4096  1048576   1740800     5276                0            5276              0             N  PRODDATA1/

MOUNTED  EXTERN  N        1024   4096  1048576    307200   228184                0          228184              0             N  PRODDATA2/

MOUNTED  EXTERN  N        1024   4096  1048576    512000   511852                0          511852              0             N  PRODFLASH/

MOUNTED  EXTERN  N        1024   4096  4194304    204800   204248                0          204248              0             Y  PRODINFRA/

Now you can add new datafiles or tablespace according your requirements.

If your database connected with other OEM application then disk group status will be show mounted until you do not add datafiles in the added disk.

Check the FND Sessions In Oracle APPS R12

Please use the below query to check Number of sessions in Oracle APPS R12.


SQL>select icx.node_id,fnd.node_name,'Number of user sessions : ' || count( distinct icx.session_id) How_many_user_sessions
from icx_sessions icx, fnd_nodes fnd where icx.disabled_flag != 'Y'
and icx.PSEUDO_FLAG = 'N'
and icx.node_id=fnd.node_id
and (icx.last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,icx.limit_time, 0,
icx.limit_time,FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate and counter < icx.limit_connects
group by icx.node_id,fnd.node_name;

*********************************************************************************

Check Long Running Request in Oracle ERP R12

Please use the below query to check long running request in Oracle ERP R12.

SELECT a.request_id
,a.oracle_process_id "SPID"
,frt.responsibility_name
,c.concurrent_program_name || ': ' || ctl.user_concurrent_program_name
,a.description
,a.ARGUMENT_TEXT
,b.node_name
,b.db_instance
,a.logfile_name
,a.logfile_node_name
,a.outfile_name
,q.concurrent_queue_name
,a.phase_code,a.status_code, a.completion_text
, actual_start_date
, actual_completion_date
, fu.user_name
,(nvl(actual_completion_date,sysdate)-actual_start_date)*1440 mins
,(SELECT avg(nvl(a2.actual_completion_date-a2.actual_start_date,0))*1440 avg_run_time
FROM APPLSYS.fnd_Concurrent_requests a2,
APPLSYS.fnd_concurrent_programs c2
WHERE c2.concurrent_program_id = c.concurrent_program_id
AND a2.concurrent_program_id = c2.concurrent_program_id
AND a2.program_application_id = c2.application_id
AND a2.phase_code || '' = 'C') avg_mins
,round((actual_completion_date - requested_start_date),2) * 24 duration_in_hours
FROM APPLSYS.fnd_Concurrent_requests a,APPLSYS.fnd_concurrent_processes b
,applsys.fnd_concurrent_queues q
,APPLSYS.fnd_concurrent_programs c
,APPLSYS.fnd_concurrent_programs_tl ctl
,apps.fnd_user fu
,apps.FND_RESPONSIBILITY_TL frt
WHERE a.controlling_manager = b.concurrent_process_id
AND a.concurrent_program_id = c.concurrent_program_id
AND a.program_application_id = c.application_id
AND a.phase_code = 'R'
AND a.status_code = 'R'
AND b.queue_application_id = q.application_id
AND b.concurrent_queue_id = q.concurrent_queue_id
AND ctl.concurrent_program_id = c.concurrent_program_id
AND a.requested_by = fu.user_id
AND a.responsibility_id = frt.responsibility_id
ORDER BY a.actual_start_date DESC

START and STOP Local Listener and SCAN Listener in RAC Database.

How to Start and Stop Local Listener and SCAN Listener

First check which services are running from which nodes and users.

oragrid$ps -eaf|grep tns


\*-----How to set environment -----*/

bash-4.3$ which srvctl
/u02/grid/product/11.2.0/bin/srvctl
bash-4.3$ which lsnrctl


bash-4.3$ export PATH=$ORACLE_HOME/bin:$PATH


oragrid$/u02/grid/product/11.2.0/bin/lsnrctl

\*--How to STOP Local Listener----*/    :-Run this command from one by one from all nodes

oragrid$srvctl stop LISTENER -n adupgdb1
oragrid$srvctl stop LISTENER -n oracledb1
oragrid$srvctl stop LISTENER -n oracledb2

\*--How to STOP SCAN Listener----*/  :-Run from any one node

oragrid$srvctl stop scan_listener

\*----- How to START Local Listener------*/

oragrid$srvctl start LISTENER -n adupgdb1
oragrid$srvctl start LISTENER -n oracledb1
oragrid$srvctl start LISTENER -n oracledb2


\*-----How to start SCAN Listener------*/
oragrid$srvctl start scan_listener

********************************************************
\*******How to bounced Single SCAN Listener one by one on nodes*******/

Run from the grid user.

bash-4.3$ srvctl status scan_listener -i 3
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node adupgdb1

bash-4.3$ srvctl stop scan_listener -i 3

bash-4.3$ srvctl status scan_listener -i 3
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is not running

bash-4.3$ srvctl start scan_listener -i 3

bash-4.3$ srvctl status scan_listener -i 3
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node adupgdb1
*************************************************************

Listener Issue in RAC Environment (Services are not registerd in Listener)

If you are facing connection issue in RAC environment then check the both SID and service name is registered in database. Please follow the below steps.

Solution:

Node1 

alter system set local_listener='(ADDRESS=(PROTOCOL=tcp)(HOST=<vip of node1>)(PORT=1521))' scope=both SID='TEST1';
alter system register;

(Replace "<vip of node1>" with VIP of node1)

Node2

alter system set local_listener='(ADDRESS=(PROTOCOL=tcp)(HOST=<vip of node2>)(PORT=1521))' scope=both SID='TEST2';
alter system register;

(Replace "<vip of node2>" with VIP of node2)


Check the staus from one by one nodes.

Node1

$lsnrctl status LISTENER

Node2

$lsnrctl status LISTENER

**************************************************************************

Blank Page Issue In Oracle ERP R12

There are could be many reasons of the blank page issue. Recently, I faced below cause of this issue.

ERROR:1 -
Below error appeared in $LOG_HOME/ora/10.1.3/opmn/opmn/oafm_default_group_1/oafmstd.out

11/12/15 14:27:19.974 html: chain failed
javax.servlet.ServletException: java.lang.RuntimeException: ORA-04063: package body "APPS.FND_WEB_SEC" has errors
ORA-06508: PL/SQL: could not find program unit being called: "APPS.FND_WEB_SEC"
ORA-06512: at line 1

at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:899)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.EvermindPageContext.handlePageException(EvermindPageContext.java:816)
at _AppsLocalLogin._jspService(_AppsLocalLogin.java:303)
at com.orionserver[Oracle Containers for J2EE 10g (10.1.3.4.0) ].http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:462)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)

Cause: Due to FND_WEB_SEC package was corrupted.

Action Plan:1 Try to compile the package. ( It's not resolved)

SQL> ALTER PACKAGE FND_WEB_SEC COMPILE BODY;

Warning: Package Body altered with compilation errors.

SQL> SHOW ERROR
Errors for PACKAGE BODY FND_WEB_SEC:

LINE/COL ERROR
-------- -----------------------------------------------------------------
0/0 PLS-00753: malformed or corrupted wrapped unit

Solution:

Please execute the following steps:

1. Login to UNIX session of the EBS Apps Tier

2. Set the application environment file

3.  Validate the version of the package FND_WEB_SEC in database

select text from user_source
where NAME= 'FND_WEB_SEC'
and line = 2;

4. Validated the version of FND_WEB_SEC package in file system,

cd $FND_TOP/patch/115/sql

adident Header AFSCJAV8.pls
adident Header AFSCJAVB.pls
adident Header AFSCJAVS.pls

Note : FND_WEB_SEC version in DB and File system should match

5. Execute the below commands

sqlplus apps/<apps pw> $FND_TOP/patch/115/sql/AFSCJAVS.pls
sqlplus apps/<apps pw> $FND_TOP/patch/115/sql/AFSCJAVB.pls
sqlplus apps/<apps pw> $FND_TOP/patch/115/sql/AFSCJAV8.pls

4. Validate that FND_WEB_SEC the package is valid.

select OBJECT_NAME,OBJECT_TYPE,STATUS from all_objects where OBJECT_NAME='FND_WEB_SEC';


5. Retest the issue. 


Solution:2  Check your tablespace size, Might be full.

*********************************************************************************

Rename Datafiles in Oracle ASM RAC Database

I prepared below steps one by one how can rename the datafiles in ASM database with RAC environment. Below are the steps is implemented in production environment.

\*-------OLD FILES LOCATION--------*/
+PRODDATA2/prod/datafiles/a_media220.dbf => +PRODDATA2/prod/DATAFILE/APPS_TS_MEDIA.271.909047637

\*-------FILES LOCATIONS WITH NEW NAMES ------*/
+PRODDATA2/prod/datafiles/a_media231.dbf

                                  \*---------ACTION PLAN------------*/

1. SQL>Alter database datafile '+PRODDATA2/prod/datafiles/a_media220.dbf' offline;

2. RMAN>copy datafile '+PRODDATA2/prod/datafiles/a_media220.dbf' to '+PRODDATA2/prod/datafiles/a_media231.dbf';

Go to asmcmd and verify the datafile

3. SQL>Alter Database rename file '+PRODDATA2/prod/datafiles/a_media220.dbf' to '+PRODDATA2/prod/datafiles/a_media231.dbf';

4. RMAN>switch Datafile '+PRODDATA2/prod/datafiles/a_media231.dbf' to copy;

5. SQL>recover datafile '+PRODDATA2/prod/datafiles/a_media231.dbf';

6. SQL>Alter database datafile '+PRODDATA2/prod/datafiles/a_media231.dbf' online;

7. SQL>select * from dba_data_files where file_name='+PRODDATA2/prod/datafiles/a_media231.dbf';

*********************************************************************************
Note: Make sure the OLD file name before removing.

7. Remove OLD file from +PRODDATA2 Group.
Connect with Grid user in ASM to perform below activity.
ASMCMD> ls -l      -----> Check the file which exits in ASM
ASMCMD> pwd
+PRODDATA2/prod/datafiles
ASMCMD> rm a_media220.dbf              /*.... Remove OLD datafiles...*/
ASMCMD> ls

8. Bounced the RAC Database if it required.

$srvctl stop database -d PROD
*********************************************************************************