Nothing logged in any related log.
As a result of looking into the output of the strace of the _had, We made a conclusion that this issue seemed to be caused the problem of the referenced library.
$ cat had.out | grep -i error 7491 writev(2, [{"/opt/VRTSvcs/bin/had", 20}, {": ", 2}, {"relocation error", 16}, {": ", 2}, {"/lib/libdl.so.2", 15}, {": ", 2}, {"symbol _dl_catch_error, version "..., 104}, {"", 0}, {"", 0}, {"\n", 1}], 10
Thus, We suspected that the libdl.so.2 library kept the had from starting.
We suggest Customer should try to do either rename the library to the libdl.so.2.bak and then restart the had or restore OS backup image.
According to Customer's mention, We could see one of two suggestions was work and the issue was resolved. The root cause that we guessed is right.
Applies To
Two nodes cluster
VCS 5.1 SP1
Redhat Linux 2.6.18-194.el5 x86_64