Symptoms :

Following is the configuration...

Application user1 (

User =user1
StartProgram = "/apps/opt/user1/scripts/ForVCS.shstart"
StopProgram = "/apps/opt/user1/scripts/ForVCS.shstop"
MonitorProgram = "/apps/opt/user1/scripts/ForVCS.shwatch"
)


Error :

2010/02/17 00:31:03 VCS WARNINGV-16-10001-51 Application:user1:monitor:State returned by Monitor Program(/apps/opt/user1/scripts/ForVCS.sh):UNKNOWN.


Troubleshooting:

Added the Agent Debug for Application Agent as follows...

#haconf -makerw
#hatype -modify Application LogDbg -addDBG_AGDEBUG
#hatype -modify Application LogDbg -add DBG_AGINFO
#hatype-modify Application LogDbg -add DBG_4
#haconf -dump -makero


In the debug logs we found that the issue with /usr/bin/su command, which was returning the value as "0" instead of "110".

The Debug message was like this...

2010/02/17 00:17:08 VCS DBG_AGDEBUG V-16-50-0 Thread(5) script(/usr/bin/su) exited with status(0)
       VCSAgProcess.C:exec_script[1111]
2010/02/1700:17:08 VCS DBG_4 V-16-50-0 Application:user1:monitor:Status returned byMonitorProgram:0.
       Application.C:execprog[751]
2010/02/1700:17:08 VCS DBG_4 V-16-50-0 Application:user1:monitor:execprog():returnstate:STATE_UNKNOWN.


Now, we executed the Monitor Program manually(as our agent executes) in order to get the truss output...

#truss -u -f -o/var/tmp/truss.out su user1 -c "/apps/opt/user1/scripts/ForVCS.shwatch"



The truss didn't give any clues why its failing.  

By the scripting experience, we knew that trap could override the exit code so we went looking for trap statements.

when we found one, it took a little while to come to the realization that the "trap - 0"was ineffective in the child sub-shell... because its the parent that reaps theexit status of the script


So to summarize, we found that the 1 of the shell built in function is not getting cleared for the Application user. This call was referred by /etc/profile file.

#trap is the command which was causing the issue.


Solution :


1) checked/etc/profile file and in the troubleshooting we found that somehow the following line was not getting executed by OS...

trap "exit 0" 0

2) we added following line in application user's  .profile at the end......

trap - 0


3) then executed following....

#su user1 -c"/apps/opt/user1/scripts/ForVCS.sh watch" ; echo $?

110


Now we received the proper Status from Monitor Program.