Script Agent (/opt/VRTSvcs/bin/ScriptAgent) issues 65534 fcntl() system calls each time a child process is spawned
book
Article ID: 100020902
calendar_today
Updated On:
Resolution
Some VCS Agents implement the ScriptAgent. The following are some examples of them.
# ls -ld */* |grep ScriptAgent
lrwxrwxrwx 1 root root 28 Feb 24 23:02CFSfsckd/CFSfsckdAgent -> /opt/VRTSvcs/bin/ScriptAgent
lrwxrwxrwx 1root root 14 Feb 24 23:02ClusterConnectorConfig/ClusterConnectorConfigAgent ->../ScriptAgent
lrwxrwxrwx 1 root root 28 Feb 24 23:02CVMCluster/CVMClusterAgent ->/opt/VRTSvcs/bin/ScriptAgent
lrwxrwxrwx 1 root root 28 Feb 24 23:02 CVMVolDg/CVMVolDgAgent-> /opt/VRTSvcs/bin/ScriptAgent
lrwxrwxrwx 1 root root 28 Feb 24 23:02CVMVxconfigd/CVMVxconfigdAgent ->/opt/VRTSvcs/bin/ScriptAgent
....
An issue is found with the ScriptAgent on the AIX platform. The Script Agent needs to close the file descriptors before it spawns the child process. Currently the ScriptAgent relies on the AIX system configuration parameter _SC_OPEN_MAX to obtain the number of file descriptors to be closed and _SC_OPEN_MAX is defined as 65534. As a result the Script Agent will issue the fcntl(F_SETFD) system call 65534 times before it spawns a child process.
Please note that if you truss the ScriptAgent it will slow down the agent significantly (up to 30 seconds) and the agent may report that the monitor procedure did not complete in expected time. The following is a sample truss output. (Be cautious when using truss on the ScriptAgent on a production system.)
1523768: 1757193:6.1360: kfcntl(3, F_SETFD, 0x00000001) =0 <<< file descriptor3
...
1523768: 1757193:6.1374: kfcntl(14, F_SETFD, 0x00000001)Err#9 EBADF
1523768: 1757193:6.1376: kfcntl(15, F_SETFD, 0x00000001)Err#9 EBADF
...
1523768: 1757193:33.6473: kfcntl(65531, F_SETFD, 0x00000001)Err#9 EBADF Err#9 EBADF <<< file descriptor65531
...
1523768: 1757193:33.6497: execve("/opt/VRTSvcs/bin/CVMVolDg/monito <<< before execute the monitor script
....
This issue will also reflected in the vmstat output where the number of system calls reported will jump significantly during the time.
kthr memory page faults cpu
---------------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in *sy* cs us sy idwa <<< sy is number of systemcalls
0 0 315240 557040 0 0 0 0 0 0 27 573 749 0 1 99 0
1 0 315240 557040 0 0 0 0 0 0 32 565 763 0 0 99 0
0 0 315239 557041 0 0 0 0 0 0 31 *552* 713 0 099 0 <<<< normal number of systemcalls
.....
0 0 315298 556977 0 0 0 0 0 0 29 66213 730 1 2 97 0
2 0 315295 556980 0 0 0 0 0 0 36 99669 790 3 2 95 0
0 0 315295 556980 0 0 0 0 0 0 33 99584 755 3 2 96 0
0 0 315296 556979 0 0 0 0 0 0 31 *66216* 722 1 297 0 <<<< high number of system calls when the issue occurs
....
Please note that this issue will not cause any significant performance decline and can be safely ignored.
Issue/Introduction
Script Agent (/opt/VRTSvcs/bin/ScriptAgent) issues 65534 fcntl() system calls each time a child process is spawned
Was this article helpful?
thumb_up
Yes
thumb_down
No