For Oracle 10g the runInstaller from the CRS (ClusterReady Services) installation media looks for vendor clusterware by running an utility called "lsnodes" (which is temporarily copied to "/tmp/OraInstall*/*/lsnodes" ).  The "lsnodes" utility expects to find a vendor clusterware exclusively supplied library named  /opt/ORCLcluster/lib/libskgxn2.so (the full path and filename is fixed, and does not depend on LD_LIBRARY_PATH).  


If the library file is present and the vendor clusterware is running correctly, the "lsnodes" utility will successfully report the cluster node names. If the public node names in the far left-hand column are not already pre-populated in the "Cluster Configuration" dialog window of the CRS runInstaller , this means the "lsnodes" utility was unable to produce a list of vendor clusterware nodes. If the runInstaller GUI is closed, the "/tmp/OraInstall*/*/lsnodes" binary will be deleted.

To troubleshoot this problem the GUI must be left running and the "/tmp/OraInstall*/*/lsnodes"  utility should be run as the Oracle user from the commandline so as to determine why it is failing.

Note: If node names are manually supplied on the "Cluster Configuration" screen, they are written into the Oracle ClusterRegistry (OCR) with arbitrarily assigned nodeIDs which have absolutely nothing to do with the VERITAS Storage Foundation for Oracle RAC (SFRAC) vendor clusterware. As a result, running $ORA_CRS_HOME/bin/lsnodes" will deceptively report whatever nodenames have been supplied during CRS installation and CRS will fail to start correctly.

The four primary causes for "lsnodes" to fail are:

1. The /etc/llthosts file is not readable by the Oracle unix user

2. /opt/ORCLcluster/lib/libskgxn2.so is not present

3. Oracle unix user cannot read the libskgxn2.so library file due to directory or file permissions

4. The architecture of the library is wrong. SFRAC supplies two versions of this file. One is for 32-bit architecture systems and the other for 64-bit architecture systems.

For example:
# pwd
/opt/ORCLcluster/lib/9iR2

# ls-l libskgn*
lrwxrwxrwx 1 root other 19 Apr 26 01:33 libskgxn2_32.so ->./libskgxn2_32.so.1
-rwxrwxr-x 1 root sys 40004 Apr 2 2004libskgxn2_32.so.1
lrwxrwxrwx 1 root other 19 Apr 26 01:33 libskgxn2_64.so-> ./libskgxn2_64.so.1
-rwxrwxr-x 1 root sys 51776 Apr 2 2004libskgxn2_64.so.1


Note: It is possible to install32-bit Oracle on a 64-bit Operating System. Choose the library that matches the system architecture. For Oracle 9i, the runInstaller simply fails to provide the node selection window if "/tmp/OraInstall*/*/lsnodes" does not produce vendor clusterware node names.  CRS is a new component in 10g and did not exist in 9i. The same library and troubleshooting steps still apply.

Upon successfully querying the vendor clusterware (SFRAC). The runInstaller links /opt/ORCLcluster/lib/libskgxn2.so to $ORA_CRS_HOME/lib/libskgxn2*

 

For 10g:

# pwd

/u00/oracle/product/10.1.0crs/lib

# ls-la libskgxn*
lrwxrwxrwx 1 root other 33 May 2 11:04 libskgxn2.so ->/opt/ORCLcluster/lib/libskgxn2.so
-rw-rw-r-- 1 oracle dba 11816 Jan 28 2004libskgxns.so
 

The 9i runInstaller just copies the file to $ORACLE_HOME/lib:

# pwd
/u00/oracle/product/9.2.0/lib

# ls -la libskgxn*
-rwxr-xr-x 1 oracledba 51776 May 1 16:35 libskgxn9.so
-rw-r--r-- 1 oracle dba 11344 Oct 172004 libskgxns.so
 

Note: 10g uses libskgxn2.so, exactly as the source file specifies. 9i copies the file as libskgxn9.so, but the source file was libskgxn2.so. The same source file works for 9i as well as10g.

Note: for 10g $ORA_CRS_HOME must be separate from $ORACLE_HOME (home for database binaries). In 10g there is a $ORACLE_HOME/lib/libskgxn* file, but it is not altered in any way (it exclusively queries CRS for its information which in turn queries vendor clusterware whereas 9i queries the vendor clusterware directly):

# pwd
/u00/oracle/product/10.1.0bin/lib

 

# ls -la libskgxn*
-rw-r--r-- 1 oracle dba 11816 May 3 03:00libskgxns.so



After the database binary installation has completed, the cache fusion library must also be copied into $ORACLE_HOME/lib/libskgxp9.so for 9i and $ORACLE_HOME/lib/libskgxp10 for 10g. SFRAC supplies four variations to cover Oracle version and bit-ness:

# pwd
/opt/ORCLcluster/lib/9iR2

 

# ls -llibskgn*
-rwxrwxr-x 1 root sys 368436 Dec 16 15:23 libskgxp10_32.so
-rwxrwxr-x 1 root sys 438016 Dec 16 15:23 libskgxp10_64.so
-rwxrwxr-x 1root sys 368432 Dec 16 15:23 libskgxp92_32.so
-rwxrwxr-x 1 root sys 438016Dec 16 15:23 libskgxp92_64.so


The easiest way to remember which library goes where:

libskgxn2.so
^
libskgxp*.so
^
N = Node membership, automatically copied by runInstaller
P = IPC [InterProcess Communication] cache fusion library, manually copied after installation.
/----------

Note: Whilst the above information is based on the Solaris platform, it is still applicable to the other Operating Systems(HPUX, AIX and Linux), since the only difference for the other Operating Systems will be exact path and filenames for the libraries.