Error Message

# zpool attach -f rpool c0t6005076300810A85B8000000000004A5d0 c0t6005076300818A6300000000000004D4d0
vdev verification failed: the following errors must be manually repaired:
/dev/dsk/c0t6005076300818A6300000000000004D4d0s0 is part of a VxVM volume.
/dev/dsk/c0t6005076300818A6300000000000004D4d0 is part of a VxVM volume.
Unable to build pool from specified devices: device already in use

 

The Solaris disk management library code spawns inuse_vxvm() which returns a bad file number (EBADF), which causes the inuse_vxvm() function to falsely report the disk is part of a VxVM volume.

Oracle have created the following Solaris BUG ID and Knowledge Article:
 

BUG 35280225 - libdiskmgt inuse_vxvm() falsely reports disk is part of a VxVM volume

Oracle Solaris 11.4 Disk Management Library Function inuse_vxvm() Falsely Reports A Disk Is Part Of A VxVM Volume (Doc ID 2941774.1)

The issue is related to Solaris 11.4 update to SRU 54 onwards.
 

Troubleshooting steps

The ZFS rpool is using /dev/dsk/c0t6005076300810A85B8000000000004A5d0:

# vxdisk list storwizev70007_0
Device:    storwizev70007_0
devicetag: storwizev70007_0
type:      auto
info:      format=ZFS
flags:    ZFS online ready private autoconfig invalid
pubpaths:  block=/dev/vx/dmp/storwizev70007_0 char=/dev/vx/rdmp/storwizev70007_0
guid:      -
udid:      IBM%5F2145%5F00c02062a16e%5F6005076300810A85B8000000000004A5
site:      -
Multipathing information:
numpaths:   1
c0t6005076300810A85B8000000000004A5d0   state=enabled


The additional disk to be attached is /dev/rdsk/c0t6005076300818A6300000000000004D4d0:
 

# vxdisk list storwizev70008_0
Device:    storwizev70008_0
devicetag: storwizev70008_0
type:      auto
info:      format=ZFS
flags:    ZFS online ready private autoconfig invalid
pubpaths:  block=/dev/vx/dmp/storwizev70008_0 char=/dev/vx/rdmp/storwizev70008_0
guid:      -
udid:      IBM%5F2145%5F00c0208298c0%5F6005076300818A6300000000000004D4
site:      -
Multipathing information:
numpaths:   1
c0t6005076300818A6300000000000004D4d0   state=enabled

Veritas does not support MPxIO in LDOM environments

# echo | format > /tmp/format

# grep 4A5 /tmp/format
      26. c0t6005076300810A85B8000000000004A5d0 
          /scsi_vhci/ssd@g6005076300810a85b8000000000004a5

# grep 4D4 /tmp/format
      27. c0t6005076300818A6300000000000004D4d0 
          /scsi_vhci/ssd@g6005076300818a6300000000000004d4

The "scsi_vhci" keyword in the format output confirms MPxIO is managing the ZFS disks.

NPIV support is only available with DMP (MPxIO must be disabled) and requires Solaris 11.4 SRU 48 or higher, along with VxVM 8.0.0.1601 Private hotfix.


The "zdb -l /dev/rdsk/c#t#d#" command can be used to show the ZFS pool information.

Sample syntax:

# zdb -l /dev/rdsk/c#t#d#


Sample output:

Disk to be attached to existing ZFS rpool:

# prvtvoc /dev/rdsk/c0t6005076300818A6300000000000004D4d0s2
* /dev/rdsk/c0t6005076300818A6300000000000004D4d0s2 partition map
*
* Dimensions:
*     512 bytes/sector
* 314572800 sectors
* 314572733 accessible sectors
*
* Label:
*   EFI/GPT
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*          34       222       255
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      4    00        256 314556127 314556382
       8     11    00  314556383     16384 314572766


The following output can be collected to assist with further troubleshooting:

1.] # /etc/vx/diag.d/vxmediadisc /dev/rdsk/c#t#d#

2.] # /etc/vx/diag.d/vxmediadisc -p /dev/rdsk/c#t#d#

3.] # /usr/lib/diskmgt/diskmgt_helper /dev/rdsk/c#t#d#

4.] # sh -x /usr/lib/diskmgt/check_vxvm /dev/rdsk/c#t#d#

 

Oracle will be providing a fix for this issue and is tentatively scheduled for release via the Solaris 11.4 SRU 60 update around mid to late August 2023.

Workaround:

Until a fix is provided by Oracle, users should use the workaround to disable in use checking by the Solaris disk management library with NOINUSE_CHECK=1.

# NOINUSE_CHECK=1; export NOINUSE_CHECK

This will essentially restore the original design behaviour prior to Solaris 11.4 SRU 54.

For sudo users:
 

$ export NOINUSE_CHECK=1
$ sudo --preserve-env=NOINUSE_CHECK ; echo $NOINUSE_CHECK
1

To unset the environment variable, type:

$ unset NOINUSE_CHECK

 

Supportability:

For Solaris LDOM related environments, MPxIO must be disabled and Veritas DMP Native Support enabled.

Note: Solaris NPIV functionality is only supported with Veritas DMP.


Steps

How to disable MPxIO

# stmsboot -d

Note: This operation will require a reboot to disable MPxIO

Run the following command to confirm if MPxIO is now disabled:

# stmsboot -L


Enable DMP Native Support

Prior to enabling DMP Native Support, all ZFS zpools must be exported first.
 

# zpool list

# vxdmpadm gettune dmp_native_support

# vxdmpadm settune dmp_native_support=on

# vxdmpadm gettune dmp_native_support


Reboot the system once DMP Native Support has been enabled.


Solaris 11.4 SRU 55 Minimum Patch Levels for InfoScale 7.4.1, 7.4.2 & 8.0
 

11.4.55.138.3    2023-03-28    infoscale-sol11_sparc-Patch-7.4.1.2300  +  cpi-Patch-7.4.1.3700

11.4.55.138.3    2023-04-04    infoscale-sol11_sparc-Patch-7.4.2.1400   + cpi-Patch-7.4.2.3100


NPIV Support only available with VxVM 8.0.0.1601 onwards

11.4.55.138.3    2023-03-27    infoscale-sol11_sparc-Patch-8.0.0.1400  +  cpi-Patch-8.0.0.1000
 

With Solaris 11.4 SRU 54 (11.4.54.138.1) users may encounter an issue when performing zpool related operations. This can occur for all ZFS pools, rpool and zpools when Veritas Volume Manager (VxVM) is configured.

Solaris 11.4 SRU 55 , 56 and 57 are also impacted. Environment: OS : Oracle Solaris 11.4 SPARC SRU 54
Infoscale Version - 7.4.1, 7.4.2 & 8.0 onwards.
# pkg info entire
Name: entire
Summary: entire incorporation including Support Repository Update
(Oracle Solaris 11.4.54.138.1).
Description: This package constrains system package versions to the same
build. WARNING: Proper system update and correct package
selection depend on the presence of this incorporation.
Removing this package will result in an unsupported system.
For more information see:
https://support.oracle.com/rs?type=doc&id=2433412.1
Category: Meta Packages/Incorporations
State: Installed
Publisher: solaris
Version: 11.4 (Oracle Solaris 11.4.54.138.1)
Branch: 11.4.54.0.1.138.1
Packaging Date: Fri Jan 27 20:40:34 2023
Last Install Time: Tue Jun 9 19:40:53 2020
Last Update Time: Fri Mar 10 13:33:04 2023
Size: 2.52 kB
FMRI: pkg://solaris/entire@11.4-11.4.54.0.1.138.1:20230127T204034Z
In this instance, the ZFS rpool consists of c0t6005076300810A85B8000000000004A5d0 (Veritas DMPNODE: storwizev70007_0).


Sample output
# zpool status rpool
pool: rpool
id: 15356636800431458597
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scan: scrub repaired 0 in 4d20h with 0 errors on Wed Mar 8 13:12:53 2023
config: NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c0t6005076300810A85B8000000000004A5d0 ONLINE 0 0 0 errors: No known data errors
# cat vxdisk_e_list | grep -iE "A5|D4"
storwizev70007_0 auto:ZFS - - ZFS c0t6005076300810A85B8000000000004A5d0 -
storwizev70008_0 auto:ZFS - - ZFS c0t6005076300818A6300000000000004D4d0 -