Error Message

Once the file descriptor leak was hit, it became apparent in 2 ways:

a) Volume Manager commands began to fail:

Apr 16 11:12:36 ss1814 vxvm:vxconfigd: [ID 702911 daemon.error] V-5-1-1589 enable failed: License has expired or is not available for operation

Apr 16 11:12:36 ss1814 vxvm:vxconfigd: [ID 702911 daemon.error]      transactions are disabled.

 

b) Running of Solaris command "pfiles" on the vxconfigd process showed that it had 100s of file descriptors open.

e.g. in customer's case, 323 were open. Anything above 100 suggests a possible file descriptor leak and merits further investigation.

.

.

322: S_IFIFO mode:0000 dev:373,0 ino:34386222 uid:0 gid:0 size:0

      O_RDWR|O_NONBLOCK FD_CLOEXEC

323: S_IFIFO mode:0000 dev:373,0 ino:34423063 uid:0 gid:0 size:0     

O_RDWR|O_NONBLOCK FD_CLOEXEC

.

.

The dmpdr command spawns a vxnotify process every time it is run. However, the processes were not terminated, leading to a gradual file descriptor leak.

 

The issue was addressed in 2 private hot-fixes:

- for Solaris 10 and 6.2.1

- for Solaris 11 and 7.2

 

Please refer to Veritas Technical Support for further details

 

All Volume Manager commands failed with "License has expired" errors. Further analysis revealed that the dmpdr utility was leaking file descriptors via vxnotify on 6.2.1 and 7.2 on the Solaris platform. The dmpdr command was being invoked as follows: # /usr/lib/vxvm/voladm.d/bin/dmpdr -o refresh in order to clean up the DMP configuration. The repeated running of this command resulted in the hitting of a file descriptor leak, which caused all Volume Manager commands to fail.