Error Message

VCS engine_A.log:

2013/11/28 16:53:44 VCS WARNING V-16-1-53025 Agent Application has faulted; ipm connection was lost; restarting the agent

Stacktrace from Application agent core file shows calls from libvas.so which is from Quest's VAS authentication libraries, but it doesn't show the problematic calls:

Thread 6 (Thread 28821):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0x00bdf226 in kill () from /evidence/mtv/58/05404058/libraries/lib/libc.so.6
#2  0xf7e80a2c in VCSSignalHandler (sig_num=11) at Platform.C:2211
#3  0xf7e80a9b in VCSSegvHandler (sig_num=11) at Platform.C:2219
#4  0xf7df14f3 in vcsag_diag_handler (sig_num=11) at VCSAgMain.C:1595
#5 
#6  0xf6f97035 in vassql_init_t () from /opt/quest/lib/libvas.so.4
#7  0xf6f721d7 in libvascache_misc_db_init () from /opt/quest/lib/libvas.so.4
#8  0xf6f7b749 in ?? () from /opt/quest/lib/libvas.so.4
#9  0x098b6fe8 in ?? ()
#10 0x00000000 in ?? ()

So, created a simple program calls, UtilityTest.C (attached), which calls the getpwnam_r() in multi-threads mode.  It dumped core file in exactly same fashion, and core file shows the problematic calls clearly: 

Thread 51 (Thread 31206):
#0  0x00000039326d5887 in semop () from /lib64/libc.so.6
#1  0x00002aaaaacc4ca6 in libvascache_vpgroup_get () from /lib64/libnss_vas3.so.2
#2  0x00002aaaaacc61bd in ?? () from /lib64/libnss_vas3.so.2
#3  0x00002aaaaacc64b1 in _nss_vas3_getXXent_get_tsd () from /lib64/libnss_vas3.so.2
#4  0x00002aaaaacc99a6 in ?? () from /lib64/libnss_vas3.so.2
#5  0x00002aaaaacca364 in _nss_vas3_getpwnam_r () from /lib64/libnss_vas3.so.2
---Type  to continue, or q  to quit---
#6  0x00000039326996a5 in getpwnam_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
#7  0x0000000000400f39 in checkutility (threadargs=0x7fffe795c610) at UtilityTest.C:42
#8  0x0000003933a0673d in start_thread () from /lib64/libpthread.so.0
#9  0x00000039326d44bd in clone () from /lib64/libc.so.6

To compile UtilityTest:

$ g++ -g UtilityTest.C -lpthread -o UtilityTest.out

VCS agent uses getpwnam_r() command to fetch user information, and it fails in multi-thread environment(default) with Quest Authentication libraries.

The Quest Authentication libraries are not thread safe with getpwnam_r() command, and the issue can be resolved by changing VCS agent into single thread mode using the following command:

# hatype -modify Application NumThreads 1

Applies To

RHEL 5.6 / VCS 5.1SP1RP2 / Quest VAS Authentication