Error Message

Tue Oct 22 22:42:50.564: Disabled Path c11t20320002AC01ECB1d2008s2(242/12944) belonging to Dmpnode 3pardata1_85162(311/2272) due to devid mismatch

Tue Oct 22 22:42:50.565: Graceful DR steps are not followed by the user on the path c11t20320002AC01ECB1d2008s2(242/12944), The device with old serial number naa.60002AC00000000000003BB30001ECB1 is replaced with a new device with serial number naa.60002AC00000000002013BB30001ECB1

Tue Oct 22 22:42:50.566: Disabled Path c4t21620002AC01ECB1d2008s2(242/12456) belonging to Dmpnode 3pardata1_85162(311/2272) due to devid mismatch

Tue Oct 22 22:42:50.567: Graceful DR steps are not followed by the user on the path c4t21620002AC01ECB1d2008s2(242/12456), The device with old serial number naa.60002AC00000000000003BB30001ECB1 is replaced with a new device with serial number naa.60002AC00000000002013BB30001ECB1 

In this instance, DMPNODE "3pardata1_85162" has 8 paths, however, from two different physical LUNs
 

# vxdisk list 3pardata1_85162
Device: 3pardata1_85162

udid: 3PARdata%5FVV%5FECB1%5F3BB30001ECB1

tag udid_asl=3PARdata%5FVV%5FECB1%5F3BB30001ECB1
Multipathing information:
numpaths: 8
c4t21620002AC01ECB1d2101s2 state=enabled
c9t21320002AC01ECB1d2101s2 state=enabled
c9t21320002AC01ECB1d2008s2 state=disabled
c16t20620002AC01ECB1d2101s2 state=enabled
c16t20620002AC01ECB1d2008s2 state=disabled
c11t20320002AC01ECB1d2008s2 state=disabled
c11t20320002AC01ECB1d2101s2 state=enabled
c4t21620002AC01ECB1d2008s2 state=disabled
 

From ddl logs we can see that all 8 paths of 3pardata1_85162 have the same LSN, hence instead of 4 paths it is showing 8 paths.
 

Dmpnode:3pardata1_85162:0x4dc08e0:8:1

        c4t21620002AC01ECB1d2101:0x3c830d8:40000005:0x3c830d8:0:3BB30001ECB1::0x219f80:0x835
        c4t21620002AC01ECB1d2008:0x3c830a8:40000005:0x3c830a8:0:3BB30001ECB1::0x219f80:0x7d8
        c11t20320002AC01ECB1d2101:0x3c832e8:40000005:0x3c832e8:0:3BB30001ECB1::0xd8080:0x835
        c11t20320002AC01ECB1d2008:0x3c83290:40000005:0x3c83290:0:3BB30001ECB1::0xd8080:0x7d8
        c16t20620002AC01ECB1d2008:0x3c83310:40000005:0x3c83310:0:3BB30001ECB1::0x219e80:0x7d8
        c16t20620002AC01ECB1d2101:0x3c833e0:40000005:0x3c833e0:0:3BB30001ECB1::0x219e80:0x835
        c9t21320002AC01ECB1d2008:0x3c830e8:40000005:0x3c830e8:0:3BB30001ECB1::0xd8180:0x7d8
        c9t21320002AC01ECB1d2101:0x3c83260:40000005:0x3c83260:0:3BB30001ECB1::0xd8180:0x835

We are extracting the LSN with the a very short length: 3BB30001ECB1

However, if we compare the LSN for all 8 paths, we have 2 different longer LSN values

3BB30001ECB100003PAR & 3BB30001ECB102013PAR

For DMPNODE: 3pardata1_85162 the SCSI inquiry details clearly report a unique LSN for the 2 LUNs

Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2101s2, evpd 0x1, page code 0x83

Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2101s2, evpd 0x1, page code 0x83

Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2008s2, evpd 0x1, page code 0x83

Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2101s2, evpd 0x1, page code 0x83

Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2008s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2009s2, evpd 0x0, page code 0x0

Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2008s2, evpd 0x1, page code 0x83

Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2101s2, evpd 0x1, page code 0x83

Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2008s2, evpd 0x1, page code 0x83

NOTE: Out of the 8 paths , 4 paths are disabled for DMPNODE "3pardata1_85162" due to devid mismatch.

The "LUN serial number" obtained from VPD page 0, doesn't always guarantee uniqueness of a LUN. Due to this the paths from two different 3PAR LUNs have the same "LUN Serial Number". This results in DMP incorrectly merging them under a single device/disk (DMPNODE) as the LUN Serial Number was not unique enough.

The OS (luxadm display output) clearly shows two different LUNs with different sizes and paths:
 

LUN #1

DEVICE PROPERTIES for disk: 2ff70002ac01ecb1
Vendor: 3PARdata
Product ID: VV
Revision: 3312
Serial Num: CZ3732JWLV
Unformatted capacity: 525312.000 MBytes
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):

/dev/rdsk/c4t21620002AC01ECB1d2101s2
/devices/pci@400/pci@1/pci@0/pci@0/SUNW,emlxs@0,1/fp@0,0/ssd@w21620002ac01ecb1,835:c,raw
LUN path port WWN: 21620002ac01ecb1
Host controller port WWN: 10000090fa05a8f3
Path status: O.K.
/dev/rdsk/c9t21320002AC01ECB1d2101s2
/devices/pci@400/pci@1/pci@0/pci@6/SUNW,emlxs@0,1/fp@0,0/ssd@w21320002ac01ecb1,835:c,raw
LUN path port WWN: 21320002ac01ecb1
Host controller port WWN: 10000090fa05a47d
Path status: O.K.
/dev/rdsk/c11t20320002AC01ECB1d2101s2
/devices/pci@400/pci@1/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20320002ac01ecb1,835:c,raw
LUN path port WWN: 20320002ac01ecb1
Host controller port WWN: 10000090fa05b80d
Path status: O.K.
/dev/rdsk/c16t20620002AC01ECB1d2101s2
/devices/pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20620002ac01ecb1,835:c,raw
LUN path port WWN: 20620002ac01ecb1
Host controller port WWN: 10000090fa05a77f
Path status: O.K.

LUN #2

DEVICE PROPERTIES for disk: 2ff70002ac01ecb1
Vendor: 3PARdata
Product ID: VV
Revision: 3312
Serial Num: CZ3732JWLV
Unformatted capacity: 262144.000 MBytes
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):

/dev/rdsk/c4t21620002AC01ECB1d2008s2
/devices/pci@400/pci@1/pci@0/pci@0/SUNW,emlxs@0,1/fp@0,0/ssd@w21620002ac01ecb1,7d8:c,raw
LUN path port WWN: 21620002ac01ecb1
Host controller port WWN: 10000090fa05a8f3
Path status: O.K.
/dev/rdsk/c9t21320002AC01ECB1d2008s2
/devices/pci@400/pci@1/pci@0/pci@6/SUNW,emlxs@0,1/fp@0,0/ssd@w21320002ac01ecb1,7d8:c,raw
LUN path port WWN: 21320002ac01ecb1
Host controller port WWN: 10000090fa05a47d
Path status: O.K.
/dev/rdsk/c11t20320002AC01ECB1d2008s2
/devices/pci@400/pci@1/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20320002ac01ecb1,7d8:c,raw
LUN path port WWN: 20320002ac01ecb1
Host controller port WWN: 10000090fa05b80d
Path status: O.K.
/dev/rdsk/c16t20620002AC01ECB1d2008s2
/devices/pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20620002ac01ecb1,7d8:c,raw
LUN path port WWN: 20620002ac01ecb1
Host controller port WWN: 10000090fa05a77f
Path status: O.K.

3PAR arrays have been supported with DMP for a very long time. veritas has always been using page 0 offset to calculate the LSN, since 3PAR array support was provided. Changing the LSN make-up will affect all the 3PAR arrays that are deployed. The modified ASLAPM package will now take the new LSN from page 83, which will lead to an UDID mismatch being set on all the 3par disks.

LSN Examples:
 

Before ASLAPM have been deployed: 

# vxdisk -p list |grep LUN_SERIAL
LUN_SERIAL_NO : CF5E0001B096
LUN_SERIAL_NO : CF5F0001B096

After ASLAPM has been deployed:
 

# vxdisk -p list |grep LUN_SER
LUN_SERIAL_NO : 60002AC0000000003900CF5E0001B096
LUN_SERIAL_NO : 60002AC0000000003900CF5F0001B096

The ASL changes will be applied to AIX, Linux and Solaris paltforms.

The ASLAPM version updated for InfoScale 7.3.1 is 7.3.1.103 on Solaris
 

# pkg info VRTSaslapm
          Name: VRTSaslapm
       Summary: Array Support Libraries and Array Policy Modules for Veritas Volume Manager
   Description: Array Support Libraries and Array Policy Modules for Veritas
                Volume Manager
      Category: System/Administration and Configuration
         State: Installed
     Publisher: Veritas
       Version: 7.3.1.102
 Build Release: 5.11
        Branch: None
Packaging Date: February 27, 2020 03:52:26 PM
          Size: 1.95 MB
          FMRI: pkg://Veritas/VRTSaslapm@7.3.1.102,5.11:20200227T155226Z

The 3PAR ASL changes will be planned for InfoScale 7.4.1.x.

Refer to https://downloads.infoscale.com for the latest 7.3.1 ASLs for Sol, Aix and Linux and/or contact Technical Support if needed.

Newly presented 3PAR LUNs may merge with existing DMPNODEs. The VxVM related errors may imply that the required Graceful DR (Dynamic LUN Reconfiguration) steps were not followed. VxVM uses a "LUN serial number" (LSN) to uniquely identify a LUN. This "LUN serial number" is obtained by performing a SCSI inquiry on Vital-Product-Data (VPD) page 0. The VPD contains information about a device that allows the device to be administered. This typically includes the product ID, vendor ID and other information specific device values enabling the LSN to be unique. In this instance the LSN was not unique enough.