cssd resource monitoring returns UNKNOWN status and Oracle crsctl command hangs or reports failure.

book

Article ID: 100022097

calendar_today

Updated On:

Resolution

 
ERRORCODE/ MESSAGE:
VCSengine_A.log displays
2009/09/3012:20:59 VCS WARNING V-16-10001-51 (mtvv240-10) Application:cssd:monitor:Statereturned by Monitor Program(/opt/VRTSvcs/rac/bin/cssd-monitor):UNKNOWN.
 
# hares-display cssd -attribute State -sys mtvv240-10
#Resource    Attribute        System    Value
cssd        State            mtvv240-10 ONLINE|STATEUNKNOWN
 
DIAGNOSTICSTEPS:
Check if theCRS, CSSD processes are running on each node
# ps -ef| egrep "ocssd.bin|crsd.bin|evmd.bin"
 oracle22468 29008   0   Sep 03 ?           5:40/u01/oracle/crs11g/bin/evmd.bin
 oracle22164 21973   0   Sep 03 ?         143:01/u01/oracle/crs11g/bin/ocssd.bin
   root22482 29110   0   Sep 03 ?         421:28/u01/oracle/crs11g/bin/crsd.bin reboot
   root24625  5165   0 08:29:50 pts/2       0:00 egrepocssd.bin|crsd.bin|evmd.bin
 
GetORA_CRS_HOME directory from init.cssd script and use this path torun the crs commands.
# grepORA_CRS_HOME= /etc/init.d/init.cssd
ORA_CRS_HOME=/u01/oracle/crs11g
 
Check ifolsnodes command returns nodename and Private IP for each node.
#$ORA_CRS_HOME/bin/olsnodes -p
mtvv240-09mtvv240-09-priv1
mtvv240-10mtvv240-10-priv1
 
Check thecssd status by executing crsctl command on each node in the SFRAC clusterand it should return healthy.
#$ORA_CRS_HOME/bin/crsctl check cssd
ClusterSynchronization Services appears healthy
 
If the abovecommand returns "Failure 1 contacting Cluster Synchronization Services daemon"or if it hangs, then there is some problem with Oracle CRS on thatnode.
 
Check if the$ORA_CRS_HOME directory is NOT full on each nodes in the SFRAC cluster.If its full it can cause this problem.
Check numberof files in $ORA_CRS_HOME/log/`hostname`/client directory. If there arehuge number (100,000+) files then crsctl command can hang (in10.2.0.3).
 
CAUSE:
$ORA_CRS_HOMEwas 100% full on one node and that caused the crsctl check cssd commandto hang with CRS 10.2.0.4.
In 11g,crsctl check cssd did not hang but returned following error:
# crsctlcheck cssd
Failure 1contacting Cluster Synchronization Services daemon
 
SOLUTION:
Freeing upfile system space in $ORA_CRS_HOME file system resolved the crsctl hangissue and cssd-monitor script started running properly.
 
 

 

Issue/Introduction

cssd resource monitoring returns UNKNOWN status and Oracle crsctl command hangs or reports failure.
Powered by Wolken Software