Error Message

# vxdisk list hdiskpower24
Device:    hdiskpower24
devicetag: hdiskpower24
type:      auto
flags:     error private autoconfig udid_mismatch      <<< error
pubpaths:  block=/dev/vx/dmp/hdiskpower24 char=/dev/vx/rdmp/hdiskpower24
guid:      {4e84ef6c-1dd2-11b2-98f0-51ab1fc75406}
udid:      EMC%5FSYMMETRIX%5F000292601383%5F60000970000292601383533030434536
site:      -
errno:     Disk is not useable, bad format     <<<  errno
Multipathing information:
numpaths:   1
hdiskpower24    state=enabled

The problem only happens if "dmp_cache_open" is enabled.   The purpose of "dmp_cache_open" is to enhance the performance of device discovery by minimizing the overhead caused by multiple device opens.  If "dmp_cache_open" is set to on, the first open on a device is cached. The default setting is on.    Please refer to the vxdmpadm manual page and VxVM Administrator's Guide for further details.

With "dmp_cache_open" ON,  the EMC BCV LUNs controlled by EMC Powerpath return the "Permission denied" error even after the BCV LUNs are in split state. This can be observed, using the OS command "dd" to read from the split BCV LUN, where the error "Permission denied" would be returned.

# dd if=/dev/hdisk145 of=/dev/null bs=8k count=32
dd: /dev/hdisk145: Permission denied

There are some VxVM commands which trigger the DMP Device Discovery Layer (DDL) to discover the devices.  (For example, vxdctl enable, vxdisk scandisks, vxdisk online .)  The Device Discovery Layer (DDL) claims the devices controlled by Power Path under enclosure PP_EMC, using the Array Support Libraries (ASL). As part of the device discovery, a device open followed by SCSI inquiry is issued to get the details of the device. Sometimes this device open fails with “Permission Denied” on the BCV luns which are in split state. This results in marking the Disk Access record with an error “Disk is not usable”.

The following is from the vxconfigd debug log.

10/10 20:37:40:  VxVM vxconfigd DEBUG V-5-1-14870 ddl_thread_claim_disk(thread 268): disk is hdisk104
10/10 20:37:40:  VxVM vxconfigd DEBUG V-5-1-14613 ddl_claim_single_disk: disk is hdisk104
10/10 20:37:40:  VxVM vxconfigd DEBUG V-5-1-14603 ddl_claim_single_disk: disk /dev/rhdisk104 has devno 1114274
10/10 20:37:40:  VxVM vxconfigd DEBUG V-5-1-14614 ddl_claim_single_disk: do_inquiry() fails for /dev/rhdisk104 - Permission denied

During the investigation of the problem, it is found that the system opened the affected devices in Diagnostic (DIAG) mode and kept the devices in this DIAG mode. When a device is opened in DIAG mode, any subsequent access to the device will return "Permission denied" error. 

Please note that this problem only occurs if DMP Device Discovery Layer (DDL) is triggered when the BCV LUNs are in established state.    In an environment where only one VxVM diskgroup is involved, those commands which trigger the DDL Device Discovery (e.g. vxdctl enable, vxdisk scandisks, vxdisk online ) will usually be run after the BCV LUNs are split.  In such single diskgroup environment, the problem does not occur.   But if the environment involves establishing and splitting multiple diskgroups at the same time, then there will be chances that the VxVM command intended for one diskgroup will affect the BCV LUNs in another diskgroup which are still in established state and causes the problem.

** Please note that the following solution is only applicable to the specific environment described below and should not be treated as a general recommendation. **

The solution is to turn the "dmp_cache_open" tunable to off.   The following command can be used to turn dmp_cache_open off.

# vxdmpadm settune dmp_cache_open=off

This solution is only applicable to environment where there are chances that the DMP Device Discovery  (DDL) will be invoked to query the BCV LUNs which are in established state and are controlled by EMC Powerpath.   For example, if there are several BCV diskgroups on the same system and the diskgroups are established and split at different time and there is a requirement to run VxVM commands involving DDL (e.g. vxdctl enable, vxdisk scandisks, vxdisk online ) during the time.

Please note that Veritas does NOT recommend turning "dmp_cache_open" off in normal circumstances.   Turning "dmp_cache_open" off can have adverse effect on the performance of VxVM commands.   It causes longer execution time for those VxVM commands which need to access the disks (e.g. vxdctl enable, vxdisk scandisks, vxdisk -o alldgs list, vxdisk online ).

If a disk is marked as "Disk is not usable, bad format" by VxVM, the error can be fixed by first making sure that the BCV LUN is in split state, waiting a minute or so and then running "vxdctl enable".   The will let the system close the device which was opened in DIAG mode and opens it in normal mode. 
 

The state of the BCV LUNs can be verified with the following symmir commands.

The following symmir command output shows that all the LUNs in the the BCV disk group are in split state.

# symmir -g mygroup verify -split
All devices in the group 'mygroup' are in 'Split' state.

The following symmir command output shows that all the LUNs are not in split state.

# symmir -g mygroup verify -split
None of the devices in the group 'mygroup' are in 'Split' state.

The following symmir command output shows that all the LUNs are in 'Synchronized' state and are ready to be split. 

# symmir -g mygroup verify -synched
All devices in the group 'mygroup' are in 'Synchronized' state.

Applies To

Veritas Volume Manager (VxVM) on AIX platform with EMC PowerPath.   The problem doesn't affect other OS platforms or environment without EMC PowerPath.