Script Agent (/opt/VRTSvcs/bin/ScriptAgent) issues 65534 fcntl() system calls each time a child process is spawned

book

Article ID: 100020902

calendar_today

Updated On:

Resolution

Some VCS Agents implement the ScriptAgent.  The following are some examples of them.

# ls -ld */* |grep ScriptAgent            
lrwxrwxrwx  1 root     root          28 Feb 24 23:02CFSfsckd/CFSfsckdAgent -> /opt/VRTSvcs/bin/ScriptAgent
lrwxrwxrwx   1root     root          14 Feb 24 23:02ClusterConnectorConfig/ClusterConnectorConfigAgent ->../ScriptAgent
lrwxrwxrwx   1 root    root          28 Feb 24 23:02CVMCluster/CVMClusterAgent ->/opt/VRTSvcs/bin/ScriptAgent
lrwxrwxrwx   1 root    root          28 Feb 24 23:02 CVMVolDg/CVMVolDgAgent-> /opt/VRTSvcs/bin/ScriptAgent
lrwxrwxrwx   1 root    root          28 Feb 24 23:02CVMVxconfigd/CVMVxconfigdAgent ->/opt/VRTSvcs/bin/ScriptAgent
....


An issue is found with the ScriptAgent on the AIX platform.  The Script Agent needs to close the file descriptors before it spawns the child process.   Currently the ScriptAgent relies on the AIX system configuration parameter _SC_OPEN_MAX to obtain the number of file descriptors to be closed and _SC_OPEN_MAX is defined as 65534.    As a result the Script Agent will issue the fcntl(F_SETFD) system call 65534 times before it spawns a child process.

Please note that if you truss the ScriptAgent it will slow down the agent significantly (up to 30 seconds) and the agent may report that the monitor procedure did not complete in expected time.   The following is a sample truss output.  (Be cautious when using truss on the ScriptAgent on a production system.)

1523768:        1757193:6.1360:        kfcntl(3, F_SETFD, 0x00000001) =0               <<< file descriptor3
...
1523768:        1757193:6.1374:        kfcntl(14, F_SETFD, 0x00000001)Err#9  EBADF
1523768:        1757193:6.1376:        kfcntl(15, F_SETFD, 0x00000001)Err#9  EBADF
...
1523768:        1757193:33.6473:        kfcntl(65531, F_SETFD, 0x00000001)Err#9  EBADF Err#9  EBADF    <<< file descriptor65531
...
1523768:        1757193:33.6497:        execve("/opt/VRTSvcs/bin/CVMVolDg/monito    <<< before execute the monitor script
....


This issue will also reflected in the vmstat output where the number of system calls reported will jump significantly during the time.

kthr    memory              page              faults        cpu    
---------------- ------------------------ ------------ -----------
r  b   avm   fre  re  pi  po  fr  sr  cy  in   *sy*  cs us sy idwa           <<< sy is number of systemcalls
0  0 315240 557040   0   0   0  0    0   0  27  573 749  0  1 99  0
1  0 315240 557040   0   0   0  0    0   0  32  565 763  0  0 99  0
0  0 315239 557041   0   0   0  0    0   0  31  *552* 713  0  099  0    <<<< normal number of systemcalls
.....
0  0 315298 556977   0   0   0  0    0   0  29 66213 730  1  2 97  0
2  0 315295 556980   0   0   0  0    0   0  36 99669 790  3  2 95  0
0  0 315295 556980   0   0   0  0    0   0  33 99584 755  3  2 96  0
0  0 315296 556979   0   0   0  0    0   0  31 *66216* 722  1  297  0    <<<< high number of system calls when the issue occurs
....


Please note that this issue will not cause any significant performance decline and can be safely ignored.

 
 

 

Issue/Introduction

Script Agent (/opt/VRTSvcs/bin/ScriptAgent) issues 65534 fcntl() system calls each time a child process is spawned