CSSD daemon fails to start with oracle error - [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

book

Article ID: 100001287

calendar_today

Updated On:

Resolution

The cssd resource gets monitor timed out with error -

Mar  3 01:18:18 vcslqr1nd2c Had[5675]: [ID702911 daemon.notice] VCS ERROR V-16-1-13027 (vcslqr1nd2c) Resource(cssd) -monitor procedure did not complete within the expected time.

# hares -display cssd-attribute State -sysvcslqr1nd1c
#Resource    Attribute        System      Value
cssd        State            vcslqr1nd1c OFFLINE|MONITORTIMEDOUT

The Oracle CRS doesn't start within VCS and also outside of the VCS too with below errors -

#$CRS_HOME/bin/crsctl start crs

#$CRS_HOME/bin/crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

CRS logs show below error-
Oracle Database 10g CRS Release 10.2.0.1.0Production Copyright 1996, 2005 Oracle.  All rights reserved.
2010-03-03 02:06:43.203: [CSSCLNT][1]clsssInitNative: connect failed, rc 9

1.] No CRS process in the ps table -
# ps -ef | egrep"ocssd.bin|crsd.bin|evmd.bin"

2.] The ORA_CRS_HOME path was correct -
# grep ORA_CRS_HOME=/etc/init.d/init.cssd

3.] The oracle RAC node information was correct -
# $ORA_CRS_HOME/bin/olsnodes-p

4.] Deleted the log files in $ORA_CRS_HOME/log/`hostname`/client directory and brought below 10000 files. Still CRS failed to start.

5.] Verified that the $ORA_CRS_HOME directory is NOT full on each nodes in the SFRAC cluster.

6.] The library file -libskgxn2.so is linked correctly. For example it should be link as below-
u01/app/oracle/product/10.2.0/crs_home/lib/libskgxn2.so-> /opt/ORCLcluster/lib/libskgxn2.so

7.] Removed .oracle file or folder in /tmp or /var/tmp folders and tried starting the CRS again but it failed again.

Solution:

The below 3 lines we recommended in /etc/inittab file.
h1:3:respawn:/etc/init.d/init.evmd run>/dev/null 2>&1
h2:3:respawn:/etc/init.d/init.cssdfatal >/dev/null 2>&1
h3:3:respawn:/etc/init.d/init.crsdrun >/dev/null 2>&1

After uncommenting the above lines and running init q, CRS started as expected.

Issue/Introduction

CSSD daemon fails to start with oracle error - [ CSSCLNT ][1]clsssInitNative: connect failed, rc 9