Application agent falsely detects wac process as offline even when the process is running properly. Application agent is killing the wac application and restarting it.

book

Article ID: 100022610

calendar_today

Updated On:

Description

Overview:
 
Application agent falsely detects wac process as offline even when the process is running properly.  Application agent is killing the wac application and restarting it.

 
Details:
 
Application agent is detecting wac process as offline and kills it by calling clean entry point when wac process is still active
 

 
Exact Error Message from debug log:
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Total number of Processes specified:1.
 
Application.C:Application_monitor[227]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Checking for Process:/opt/VRTSvcs/bin/wac as Default User:root.
 
Application.C:Application_monitor[246]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Process:/opt/VRTSvcs/bin/wac; User:root.
 
Application.C:processExists[452]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:New process string after removing extra space:/opt/VRTSvcs/bin/wac; User:root.
 
Application.C:processExists[491]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Command prepared for getting pid is
 
/bin/ps --cols=100000 --User=root -ao pid,args | /bin/grep "/opt/VRTSvcs/bin/wac" | /bin/grep -v /bin/grep | /usr/bin/tr -s " " " " | /bin/sed -e 's/^ //' | /bin/cut -f1 -d" ".
 
Application.C:processExists[519]
 
2009/07/14 12:40:20 VCS DBG_4 V-16-50-0 Application:wac:monitor:pidstring:7473
 
2009-07-14 12:40:20, process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[908]
 
2009/07/14 12:40:20 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling getArgsOfPid for pid:7473 for process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[932]
 
2009/07/14 12:40:21 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling VCSAgExec for pid:7473
 
Application.C:getArgsOfPid[874]
 
2009/07/14 12:40:21 VCS DBG_4 V-16-50-0 Application:wac:monitor:for pid:7473, arguments are:/opt/VRTSvcs/bin/wac
 
Application.C:getArgsOfPid[879]
 
2009/07/14 12:40:21 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling getArgsOfPid for pid:2009-07-14 12:40:20 for process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[932]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling VCSAgExec for pid:2009
 
Application.C:getArgsOfPid[874]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:for pid:2009, arguments are:
 
Application.C:getArgsOfPid[879]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:pid not found for process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[950]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:process(/opt/VRTSvcs/bin/wac) does not exist.
 
Application.C:processExists[609]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:Process:/opt/VRTSvcs/bin/wac; return state: Offline.
 
Application.C:Application_monitor[257]
 
2009/07/14 12:40:22 VCS ERROR V-16-2-13067 Thread(4135934880) Agent is calling clean for resource(wac) because the resource became OFFLINE unexpectedly, on its own.
 
2009/07/14 12:40:25 VCS ERROR V-16-2-13068 Thread(4135934880) Resource(wac) - clean completed successfully.
 
2009/07/14 12:40:25 VCS ERROR V-16-2-13073 Thread(4135934880) Resource(wac) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 3) the resource.
 

 
getArgsOfPID = pid:2009-07-14 12:40:20 for process:/opt/VRTSvcs/bin/wac.
 

 
Log Files:
 
/var/VRTSvcs/lock/tmp/Application-1 file
 
Troubleshooting:
 
While calling any entry point agent framework redirects the output file descriptor to a file (thread specific file), so that output of the entry point can be captured. In case of Application agent, online entry point invokes the program mentioned in StartProgram attribute. As output file descriptor is redirected to thread specific file when invoking the online script and online script invokes the StartProgram, so output file descriptor of StartProgram will also point to the same file.
 
Problem occurs when the same thread which executed the online entry point executes monitor entry point.
 
In monitor entry point of the Application agent, we query the process id of the running process from the process table. The output will be logged into the thread specific file and if in mean time the background process write some thing to the output file descriptor then that will get written to the same file (thread specific file).

Resolution

Fixed on PC infoscale-rhel7.9_x86_64-Patch-7.4.2.1400 https://downloads.infoscale.com/infoscale/REL600675?q=infoscale-rhel7.9_x86_64-Patch-7.4.2.1400 

Fixed on PC infoscale-rhel8.3_x86_64-Patch-7.4.2.1400 https://downloads.infoscale.com/infoscale/REL600675?q=infoscale-rhel8.3_x86_64-Patch-7.4.2.1400 

Fixed on PC infoscale-sles12sp5_x86_64-Patch-7.4.2.1500 https://downloads.infoscale.com/infoscale/REL600675?q=Patch-7.4.2.1500 

Fixed on PC infoscale-aix-Patch-7.4.2.1100 https://downloads.infoscale.com/infoscale/REL600675?q=infoscale-aix-Patch-7.4.2.1100 

Fixed on PC infoscale-sol11_x64-Patch-7.4.2.1100 https://downloads.infoscale.com/infoscale/REL600675?q=infoscale-sol11_x64-Patch-7.4.2.1100 

Fixed on PC infoscale-rhel7.9_x86_64-Patch-7.3.1.3600 https://downloads.infoscale.com/infoscale/REL384171?q=infoscale-rhel7.9_x86_64-Patch-7.3.1.3600 

Fixed on PC infoscale-rhel8.4_x86_64-Patch-7.4.2.1800 https://downloads.infoscale.com/infoscale/REL600675?q=infoscale-rhel8.4_x86_64-Patch-7.4.2.1800 

Fixed on PC infoscale-sles12_x86_64-Patch-7.4.1.2900 https://downloads.infoscale.com/infoscale/REL169664?q=infoscale-sles12_x86_64-Patch-7.4.1.2900 

 

Alternatively:

Fix the StartProgram, StopProgram, MonitorProgram attribute of all the Application resources in one of the following two ways:
 
  (a) Redirect the stdout and stderr of StartProgram, StopProgram, MonitorProgram process to /dev/null, or
 
  (b) Redirect the stdout and stderr of StartProgram, StopProgram, MonitorProgram process to some file

 

Issue/Introduction

Application agent falsely detects wac process as offline even when the process is running properly. Application agent is killing the wac application and restarting it.

Additional Information

ETrack: 1837979

Powered by Wolken Software