Un-able to remove a VSCSI controller in the ODM.
Article ID: 100003323
Updated On:
Description
Error Message
Method error (/etc/methods/ucfgdevice):
0514-062 Cannot perform the requested function because the specified device is busy.
When vxconfigd uses up all the available file descripton, vxconfigd can display abnormal behaviours. For example, vx commands may complain about license problem where there is no problem according to vxlicrep output. VxVM devices may disappear when "vxdctl enable" is run because vxconfigd cannot discover the devices.
Cause
File descriptor leak in the the VSCSI ASL. The number of file descriptors used by the vxconfigd can be checked with the procfiles command. For example,
# ps -eaf | grep vxconfigd
root 14024848 1 213 Oct 23 - 3:06 vxconfigd -k
# procfiles 14024848 | head -40
14024848 : vxconfigd -k
Current rlimit: 2147483647 file descriptors
0: S_IFCHR mode:00 dev:10,4 ino:390 uid:0 gid:0 rdev:2,3
O_RDONLY
1: S_IFCHR mode:00 dev:10,4 ino:102937 uid:0 gid:0 rdev:39,0
O_RDWR
2: S_IFCHR mode:00 dev:10,4 ino:24831 uid:0 gid:0 rdev:20,0
O_RDWR
3: S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
O_RDONLY
4: S_IFREG mode:0555 dev:10,4 ino:848 uid:0 gid:0 rdev:0,0
O_RDONLY size:3738114
5: S_IFREG mode:0644 dev:10,4 ino:561 uid:0 gid:0 rdev:0,0
O_RDWR size:33048
6: S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
O_RDONLY
7: S_IFSOCK mode:00 dev:10,4 ino:103169 uid:0 gid:0 rdev:0,0
O_RDWR | O_NONBLOCK
8: S_IFSOCK mode:00 dev:10,4 ino:103201 uid:0 gid:0 rdev:0,0
O_RDWR | O_NONBLOCK
9: S_IFSOCK mode:00 dev:10,4 ino:103233 uid:0 gid:0 rdev:0,0
O_RDWR | O_NONBLOCK
13: S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
O_RDONLY
15: S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
O_RDONLY
...
524: S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
O_RDONLY
525: S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
O_RDONLY
526: S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
O_RDONLY
The number of times the files are opened can be calculated using the following command.
# procfiles 14024848 | grep mode | sed 's/^[ 0-9]*://' | uniq -c
1 S_IFCHR mode:00 dev:10,4 ino:390 uid:0 gid:0 rdev:2,3
1 S_IFCHR mode:00 dev:10,4 ino:102937 uid:0 gid:0 rdev:39,0
1 S_IFCHR mode:00 dev:10,4 ino:24831 uid:0 gid:0 rdev:20,0
1 S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
1 S_IFREG mode:0555 dev:10,4 ino:848 uid:0 gid:0 rdev:0,0
1 S_IFREG mode:0644 dev:10,4 ino:561 uid:0 gid:0 rdev:0,0
1 S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0
1 S_IFSOCK mode:00 dev:10,4 ino:103169 uid:0 gid:0 rdev:0,0
1 S_IFSOCK mode:00 dev:10,4 ino:103201 uid:0 gid:0 rdev:0,0
1 S_IFSOCK mode:00 dev:10,4 ino:103233 uid:0 gid:0 rdev:0,0
565 S_IFCHR mode:00 dev:10,4 ino:388 uid:0 gid:0 rdev:16,0 <<< inode 388 (raw device 16,0) is opened 565 times
1 S_IFIFO mode:00 dev:65535,65535 ino:1425720 uid:0 gid:0 rdev:0,0
# ls -li /dev/vscsi0
388 crw-rw-rwT 1 root system 16, 0 Aug 31 09:49 /dev/vscsi0
Resolution
Use the following procedure to remove the VSCSI Controller from the ODM database by disabling vxconfigd.
For example:
#oslevel -s
5300-08-08-0943
#lslpp -l |grep vx
VRTSvxfs 5.1.0.0 COMMITTED Veritas File System by
VRTSvxvm 5.1.0.0 COMMITTED Veritas Volume Manager by
#vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
vscsi1 IBM_VSCSI ENABLED ibm_vscsi0
vscsi2 IBM_VSCSI ENABLED ibm_vscsi0
#lsdev -Cc adapter
fcs0 Defined 01-08 FC Adapter
vsa0 Available LPAR Virtual Serial Adapter
vscsi0 Defined Virtual SCSI Client Adapter
vscsi1 Available Virtual SCSI Client Adapter
vscsi2 Available Virtual SCSI Client Adapter
#vxdisk path
SUBPATH DANAME DMNAME GROUP STATE
hdisk0 ibm_vscsi0_0 - - ENABLED
hdisk13 ibm_vscsi0_1 - - ENABLED
hdisk12 ibm_vscsi0_2 - - ENABLED
hdisk1 ibm_vscsi0_3 - - ENABLED
hdisk2 ibm_vscsi0_4 - - ENABLED
#vxdmpadm getsubpaths ctlr=vscsi2
NAME STATE[A] PATH-TYPE[M] DMPNODENAME ENCLR-TYPE ENCLR-NAME ATTRS
================================================================================
hdisk1 ENABLED(A) PRIMARY ibm_vscsi0_3 IBM_VSCSI ibm_vscsi0 -
hdisk2 ENABLED(A) PRIMARY ibm_vscsi0_4 IBM_VSCSI ibm_vscsi0 -
#vxdisk list
DEVICE TYPE DISK GROUP STATUS
ibm_vscsi0_0 auto:LVM - - LVM
ibm_vscsi0_1 auto:cdsdisk - - online
ibm_vscsi0_2 auto:cdsdisk - - online
ibm_vscsi0_3 auto:none - - online invalid
ibm_vscsi0_4 auto:none - - online invalid
#vxdmpadm -f disable ctlr=vscsi2
#vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
vscsi1 IBM_VSCSI ENABLED ibm_vscsi0
vscsi2 IBM_VSCSI DISABLED ibm_vscsi0
#vxdisk rm ibm_vscsi0_3 ibm_vscsi0_4
#vxdisk list
DEVICE TYPE DISK GROUP STATUS
ibm_vscsi0_0 auto:LVM - - LVM
ibm_vscsi0_1 auto:cdsdisk - - online
ibm_vscsi0_2 auto:cdsdisk - - online
#rmdev -Rdl vscsi2
hdisk1 deleted
hdisk2 deleted
Method error (/etc/methods/ucfgdevice):
0514-062 Cannot perform the requested function because the specified device is busy.
#vxdctl stop
#rmdev -Rdl vscsi2
vscsi2 deleted
To Prevent the file descriptor leak, please exclude the libvxvscsi.so ASL library
If you don't need DMP to manage the paths of the VSCSI devices (for example, if you have only one path to the VSCSI devices), the libvxscsi.so ASL can be excluded to prevent the file descriptor leak. Please exclude the ASL using the following command.
# vxddladm excludearray libname=llibvxvscsi.so
# vxdctl enable
# vxddladm listexclude
The Diskarrays excluded
-----------------------
Based on Library names:
libvxvscsi.so
-----------------------
Based on VID, PID Combination:
------------------------------
The VSCSI devices will be claimed by DMP as OTHER_DISKS.
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT
=======================================================================================
other_disks OTHER_DISKS OTHER_DISKS CONNECTED OTHER_DISKS 1
Solution
Please upgrade to 5.1SP1 RP1 and above to fix the issue. Currenlty there is no fix for 5.0 release yet.
Applies To
Problem affects AIX LPARs when vSCSI devices are provisioned to the LPARs. Check if any devices are claimed by the libvxvscsi.so ASL.
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT
===================================================================================
ibm_vscsi0 IBM_VSCSI VSCSI CONNECTED A/P-LB-VSCSI 1
Problem affects both VxVM 5.0 and 5.1.
Issue/Introduction
Additional Information
