Sybase program "startserver" failed to start the Sybase database server on a VCS node because of the extra long PATH environment varaible

Description

Error Message

Sybase_A.log error messages:

-----------------------------

2011/09/02 13:16:06 VCS ERROR V-16-2-13066 Thread(3) Agent is calling clean for resource(sg1_syb_rs) because the resource is not up even after online completed.

2011/09/02 13:16:08 VCS ERROR V-16-2-13068 Thread(3) Resource(sg1_syb_rs) - clean completed successfully.

2011/09/02 13:16:08 VCS ERROR V-16-2-13071 Thread(3) Resource(sg1_syb_rs): reached OnlineRetryLimit(0).

-----------------------------

Sybase_A.log Debug information:

-------------------------------

2011/09/02 13:41:08 VCS DBG_1 V-16-50-0 Sybase:sg1_syb_rs:monitor:Cookie analysis finished. proc_ok is FALSE

Sybase.C:sybase_monitor[641]

2011/09/02 13:41:08 VCS DBG_1 V-16-50-0 Sybase:sg1_syb_rs:monitor:Could not find a process for data server SYB1. Returning OFFLINE.

Sybase.C:sybase_monitor[779]

2011/09/02 13:41:08 VCS DBG_AGINFO V-16-50-0 Thread(3) Resource(sg1_syb_rs) - monitor entry point exited with a confidence value 0.

VCSAgType.C:call_monitor[1409]

--------

2011/09/02 13:42:15 VCS DBG_AGDEBUG V-16-50-0 Thread(3) Calling online for resource sg1_syb_rs

VCSAgType.C:call_online[1563]

2011/09/02 13:42:46 VCS DBG_AGDEBUG V-16-50-0 Thread(3) script (/opt/VRTSagents/ha/bin/Sybase/online) exited with status (10)

VCSAgProcess.C:exec_script[1111]

2011/09/02 13:42:46 VCS DBG_AGINFO V-16-50-0 Thread(3) Resource(sg1_syb_rs) - online entry point exited with a return value 10.

VCSAgType.C:call_online[1594]

--------------------------------

---------------Use truss to check the SybaseAgent during online process ----------

root@osssvr-1 # ps -ef | grep -i sybase | grep -i agent

root 25991 1 0 Sep 08 ? 25:49 /opt/VRTSagents/ha/bin/Sybase/SybaseAgent -type Sybase -agdir /opt/VRTSagents/h

root 25993 1 0 Sep 08 ? 21:22 /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent -type SybaseBk -agdir /opt/VRTSag

root@osssvr-1 # truss -faile -o truss.out.0923 -p 25991

-------------------------------------------

The issue node truss info:

26134: 95.5537 sigaction(SIGINT, 0xFFFFFFFF7FFFCCC0, 0x00000000) = 0

26134: 95.5538 sigaction(SIGQUIT, 0xFFFFFFFF7FFFCCC0, 0x00000000) = 0

26132: 95.5544 lwp_sigmask(SIG_SETMASK, 0x00020000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]

26134: 95.5550 execve("/bin/sh", 0xFFFFFFFF7FFFCEC8, 0xFFFFFFFF7FFFE1F8) argc = 3

26134: argv: sh -c bin:/usr/sbin:/sbin <<<<<<<<<<< Sybase startserver tried to run a non-existent program and caused the online failure

The truss output from a successful Sybase resource online:

28388: 92.1887 lwp_sigmask(SIG_SETMASK, 0x00020000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]

28390: 92.1893 execve("/bin/sh", 0xFFFFFFFF7FFFD2D8, 0xFFFFFFFF7FFFE608) argc = 3

28390: argv: sh -c /export/home/sybase/ASE-15_0/install/RUN_SYB1 & <<<<<<<<< Sybase startserver should execute this program

--------------------------------------------------------------------

Cause

The HAD and Sybase Agent processes's extra long PATH environment variable was inherited from the user shell where VCS daemon was started. (hastart)

The user performed some maintenance operations and changed the PATH variable during the time and then restarted VCS. The changed PATH variable was inherited by the VCS/Agent processes. Sybase failed to online when the extra long PATH variable was inherited by the Sybase startserver program through the VCS Sybase Agent process.

Below testing shows how the user shell PATH environment can be inherited by the VCS SybaseAgent process.

---------------------------------------------------------------------------------

The environment variables are inherited by all the related VCS processes when VCS daemon (HAD) is restared. This includes the VCS Sybase agent process.

If the system boots up afresh, then the processes will have clean environment variables.

------------------------------------------------------

# echo $PATH <<<<<<

/usr/sbin:/usr/bin:/etc/vx/bin:/usr/share/webconsole/bin:/sbin:/usr/lib/fs/vxfs/bin:/usr/lib/vxvm/bin:/usr/lib/vxvm/voladm.d/bin:

/usr/openwin/bin:/usr/perl5/bin:/opt/VRTS/bin:/opt/VRTSvlic/bin:/opt/VRTSperl/bin:/opt/VRTSob/bin:/opt/VRTSob/jre/bin:

/opt/VRTSvxvm/vmcvt/bin:/opt/VRTSfspro/bin:/opt/VRTSalloc/bin:/opt/VRTSvcs/vxfen/bin:/opt/VRTSvcs/bin:

/opt/VRTSvcs/wizards/bin:/opt/VRTSweb/bin:/opt/VRTScssim/bin:/opt/VRTSvail/bin:/opt/VRTSdb2ed/bin:/etc/vx/diag.d:

/opt/VRTSspt/FS/MetaSave:/opt/VRTS/bin:/opt/VRTSspt/FS/VxBench

# pargs -e 4062

4062: /opt/VRTSvcs/bin/had <<<<<

envp[0]: EAT_CATALOG_DIR=/opt/VRTSvcs

envp[1]: EAT_HOME_DIR=/opt/VRTSvcs

envp[2]: PWD=/var/VRTSvcs/diag/had

envp[3]: VCS_DIAG=/var/VRTSvcs

envp[4]: VCS_LOG=/var/VRTSvcs

envp[5]: VCS_CONF=/etc/VRTSvcs

envp[6]: VCS_HOME=/opt/VRTSvcs

envp[7]: LD_LIBRARY_PATH=/opt/VRTSvcs/lib:

envp[8]: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin <<<<<<

envp[9]: SMF_FMRI=svc:/system/vcs:default

envp[10]: SMF_METHOD=/lib/svc/method/vcs start

envp[11]: SMF_RESTARTER=svc:/system/svc/restarter:default

envp[12]: TZ=PRC

# hastop -local –force <<<<<

# hastart

# pargs -e 9617 <<<<

9617: /opt/VRTSvcs/bin/had

envp[0]: EAT_CATALOG_DIR=/opt/VRTSvcs

envp[1]: EAT_HOME_DIR=/opt/VRTSvcs

envp[2]: VCS_DIAG=/var/VRTSvcs

envp[3]: VCS_LOG=/var/VRTSvcs

envp[4]: VCS_CONF=/etc/VRTSvcs

envp[5]: VCS_HOME=/opt/VRTSvcs

envp[6]: HOME=/

envp[7]: HZ=100

envp[8]: LD_LIBRARY_PATH=/opt/VRTSvcs/lib:

envp[9]: LOGNAME=root

envp[10]: MAIL=/var/mail/root

envp[11]: MANPATH=:/usr/share/man:/opt/VRTS/man:/opt/VRTSvlic/man:/opt/VRTSob/jre/man:/opt/VRTSdb2ed/man:/opt/VRTSd2gui/man

envp[12]: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/etc/vx/bin:/usr/share/webconsole/bin:/sbin:/usr/lib/fs/vxfs/bin:/usr/lib/vxvm/bin:

/usr/lib/vxvm/voladm.d/bin:/usr/openwin/bin:/usr/perl5/bin:/opt/VRTS/bin:/opt/VRTSvlic/bin:/opt/VRTSperl/bin:/opt/VRTSob/bin:

/opt/VRTSob/jre/bin:/opt/VRTSvxvm/vmcvt/bin:/opt/VRTSfspro/bin:/opt/VRTSalloc/bin:/opt/VRTSvcs/vxfen/bin:/opt/VRTSvcs/bin:/opt/VRTSvcs/wizards/bin:

/opt/VRTSweb/bin:/opt/VRTScssim/bin:/opt/VRTSvail/bin:/opt/VRTSdb2ed/bin:/etc/vx/diag.d:/opt/VRTSspt/FS/MetaSave:

/opt/VRTS/bin:/opt/VRTSspt/FS/VxBench <<<

----------------------------------------------------------------------------

Below testing showed the Sybase "startserver" having issue if PATH variable was too long.

If the PATH variable is more then 2045 characters, then Sybase startserver program will fail to start the Sybase database server.

---------------------------------------------------------------------

The issue is reproduced in the following way. The Sybase "startserver" fails to start the Sybase database server if the PATH variable is too long.

# su dbuser

$ . /export/home/sybase/SYBASE.sh
$ PATH=$PATH:very_long_path_variable:...... # set the PATH environment variable to have more than 2048 characters

$ echo $PATH

$ echo $PATH | wc

$ /export/home/sybase/ASE-15_0/install/startserver -f /export/home/sybase/ASE-15_0/install/RUN_SYB1

---------------------------------------------------------------------

Solution

If the PATH variable is added to /opt/VRTSvcs/bin/vcsenv, the HAD/Agent processes will inherit the PATH variable from "/opt/VRTSvcs/bin/vcsenv", not from the user shell and the problem can be fixed.

Below testing shows that the PATH environment will be inherited from "/opt/VRTSvcs/bin/vcsenv" if PATH environment variable is defined in it.

------------------------------------------------------------------------------------

# echo “PATH=/usr/sbin:/usr/bin:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSvcs/bin:/test/test” >> /opt/VRTSvcs/bin/vcsenv

# echo $PATH

/usr/sbin:/usr/bin:/etc/vx/bin:/usr/share/webconsole/bin:/sbin:/usr/lib/fs/vxfs/bin:/usr/lib/vxvm/bin:/usr/lib/vxvm/voladm.d/bin:/usr/openwin/bin:

/usr/perl5/bin:/opt/VRTS/bin:/opt/VRTSvlic/bin:/opt/VRTSperl/bin:/opt/VRTSob/bin:/opt/VRTSob/jre/bin:/opt/VRTSvxvm/vmcvt/bin:/opt/VRTSfspro/bin:

/opt/VRTSalloc/bin:/opt/VRTSvcs/vxfen/bin:/opt/VRTSvcs/bin:/opt/VRTSvcs/wizards/bin:/opt/VRTSweb/bin:/opt/VRTScssim/bin:/opt/VRTSvail/bin:

/opt/VRTSdb2ed/bin:/etc/vx/diag.d:/opt/VRTSspt/FS/MetaSave:/opt/VRTS/bin:/opt/VRTSspt/FS/VxBench:/test/test

# hastop -local -force

# hastart

# ps -ef | grep had

root 12386 11496 0 11:08:11 pts/1 0:00 grep had

root 12378 1 0 11:08:07 ? 0:00 /opt/VRTSvcs/bin/had

root 12380 1 0 11:08:07 ? 0:00 /opt/VRTSvcs/bin/hashadow

# pargs -e 12378

12378: /opt/VRTSvcs/bin/had

envp[0]: EAT_CATALOG_DIR=/opt/VRTSvcs

envp[1]: EAT_HOME_DIR=/opt/VRTSvcs

envp[2]: VCS_DIAG=/var/VRTSvcs

envp[3]: VCS_LOG=/var/VRTSvcs

envp[4]: VCS_CONF=/etc/VRTSvcs

envp[5]: VCS_HOME=/opt/VRTSvcs

envp[6]: HOME=/

envp[7]: HZ=100

envp[8]: LD_LIBRARY_PATH=/opt/VRTSvcs/lib:

envp[9]: LOGNAME=root

envp[10]: MAIL=/var/mail/root

envp[11]: MANPATH=:/usr/share/man:/opt/VRTS/man:/opt/VRTSvlic/man:/opt/VRTSob/jre/man:/opt/VRTSdb2ed/man:/opt/VRTSd2gui/man

envp[12]: OLDPWD=/

envp[13]: PATH=/usr/sbin:/usr/bin:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSvcs/bin:/test/test <<<<<

---------------------------------------------------------------------------------------

Applies To

------- Customer has an extra long PATH environment variable ---------------------------------

Use the "pargs" command to check the process environment variables.

# pargs -e 25991

25991: /opt/VRTSagents/ha/bin/Sybase/SybaseAgent -type Sybase -agdir /opt/VRTSagents/ha

envp[0]: CLUSTER_LOGDBG=DBG_1 DBG_2 DBG_3 DBG_4 DBG_5 DBG_AGINFO DBG_AGDEBUG
.....

envp[63]: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/sbin:/opt/VRTSvcs/bin:/opt/VRTS/bin:

/export/home/omc_control/osswebtools/web3/iManagerM2000V200R011C00SPC210/tools/python:/opt/OMC/3rdTools/bin:

/opt/OMC/3rdTools/python/bin:/opt/OMC/jre/bin:/opt/OMC:/opt/OMC/bin:/opt/OMC/3rdTools:/usr/local/bin:/opt/SUNWspro/bin:.:

/export/forte6/SUNWspro/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:.:/export/forte6/SUNWspro/bin:/opt/OMC/3rdTools/postgresql/bin:/opt/OMC/3rdTools/usr/local/bin:/opt/OMC/bin:/opt/OMC/3rdTools/bin:/opt/OMC/3rdTools/flex-2.5.4/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/jobscheduler/bin:/opt/sybase/RPL-15_0/bin:/opt/sybase/ASEP/bin:/opt/sybase/DBISQL/bin:/opt/sybase/UAF-2_0/bin:/opt/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/bin:

/opt/sybase/ASE-15_0/install:/opt/OSMU/omc_control/bin:/opt/OSMU/omc_control/lib/python:/usr/sbin:/sbin:/opt/VRTSvcs/bin:/opt/VRTS/bin:/usr/sbin:/sbin:

/opt/VRTSvcs/bin:/opt/VRTS/bin:/usr/sbin:/usr/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise-T5220/sbin:

/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin:

/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/lib/vxvm/bin:/opt/VRTS/bin:

/opt/VRTS/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise-T5220/sbin:

/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin:/usr/openv/netbackup/bin:

/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/lib/vxvm/bin:/opt/VRTS/bin:/opt/VRTS/bin:/usr/dt/bin:/sbin:/usr/sbin:

/usr/sfw/bin::/opt/OMC/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/OMC/3rdTools/sms/bin::/opt/OMC/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/OMC/3rdTools/sms/bin:

/usr/sbin:/sbin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise-T5220/sbin:

/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin:/usr/openv/netbackup/bin:

/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/lib/vxvm/bin:/opt/VRTS/bin:/opt/VRTS/bin
.....

---------------------------------------------------------------------------------

Resolution