Newly presented 3PAR LUNs may merge with existing DMPNODEs stating Graceful DR steps are not followed (DCPA)
Article ID: 100047278
Updated On:
Description
Error Message
Tue Oct 22 22:42:50.564: Disabled Path c11t20320002AC01ECB1d2008s2(242/12944) belonging to Dmpnode 3pardata1_85162(311/2272) due to devid mismatch
Tue Oct 22 22:42:50.565: Graceful DR steps are not followed by the user on the path c11t20320002AC01ECB1d2008s2(242/12944), The device with old serial number naa.60002AC00000000000003BB30001ECB1 is replaced with a new device with serial number naa.60002AC00000000002013BB30001ECB1
Tue Oct 22 22:42:50.566: Disabled Path c4t21620002AC01ECB1d2008s2(242/12456) belonging to Dmpnode 3pardata1_85162(311/2272) due to devid mismatch
Tue Oct 22 22:42:50.567: Graceful DR steps are not followed by the user on the path c4t21620002AC01ECB1d2008s2(242/12456), The device with old serial number naa.60002AC00000000000003BB30001ECB1 is replaced with a new device with serial number naa.60002AC00000000002013BB30001ECB1
Cause
In this instance, DMPNODE "3pardata1_85162" has 8 paths, however, from two different physical LUNs
# vxdisk list 3pardata1_85162
Device: 3pardata1_85162
udid: 3PARdata%5FVV%5FECB1%5F3BB30001ECB1
tag udid_asl=3PARdata%5FVV%5FECB1%5F3BB30001ECB1
Multipathing information:
numpaths: 8
c4t21620002AC01ECB1d2101s2 state=enabled
c9t21320002AC01ECB1d2101s2 state=enabled
c9t21320002AC01ECB1d2008s2 state=disabled
c16t20620002AC01ECB1d2101s2 state=enabled
c16t20620002AC01ECB1d2008s2 state=disabled
c11t20320002AC01ECB1d2008s2 state=disabled
c11t20320002AC01ECB1d2101s2 state=enabled
c4t21620002AC01ECB1d2008s2 state=disabled
From ddl logs we can see that all 8 paths of 3pardata1_85162 have the same LSN, hence instead of 4 paths it is showing 8 paths.
Dmpnode:3pardata1_85162:0x4dc08e0:8:1
c4t21620002AC01ECB1d2101:0x3c830d8:40000005:0x3c830d8:0:3BB30001ECB1::0x219f80:0x835
c4t21620002AC01ECB1d2008:0x3c830a8:40000005:0x3c830a8:0:3BB30001ECB1::0x219f80:0x7d8
c11t20320002AC01ECB1d2101:0x3c832e8:40000005:0x3c832e8:0:3BB30001ECB1::0xd8080:0x835
c11t20320002AC01ECB1d2008:0x3c83290:40000005:0x3c83290:0:3BB30001ECB1::0xd8080:0x7d8
c16t20620002AC01ECB1d2008:0x3c83310:40000005:0x3c83310:0:3BB30001ECB1::0x219e80:0x7d8
c16t20620002AC01ECB1d2101:0x3c833e0:40000005:0x3c833e0:0:3BB30001ECB1::0x219e80:0x835
c9t21320002AC01ECB1d2008:0x3c830e8:40000005:0x3c830e8:0:3BB30001ECB1::0xd8180:0x7d8
c9t21320002AC01ECB1d2101:0x3c83260:40000005:0x3c83260:0:3BB30001ECB1::0xd8180:0x835
We are extracting the LSN with the a very short length: 3BB30001ECB1
However, if we compare the LSN for all 8 paths, we have 2 different longer LSN values
3BB30001ECB100003PAR & 3BB30001ECB102013PAR
For DMPNODE: 3pardata1_85162 the SCSI inquiry details clearly report a unique LSN for the 2 LUNs
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2101s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2101s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c9t21320002AC01ECB1d2008s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2101s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2008s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c16t20620002AC01ECB1d2009s2, evpd 0x0, page code 0x0
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2008s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2101s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB100003PAR
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2101s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c11t20320002AC01ECB1d2101s2, evpd 0x1, page code 0x83
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2008s2, evpd 0x0, page code 0x0
Serial Number : 3BB30001ECB102013PAR
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2008s2, evpd 0x1, page code 0x80
Inquiry for /dev/rdsk/c4t21620002AC01ECB1d2008s2, evpd 0x1, page code 0x83
NOTE: Out of the 8 paths , 4 paths are disabled for DMPNODE "3pardata1_85162" due to devid mismatch.
The "LUN serial number" obtained from VPD page 0, doesn't always guarantee uniqueness of a LUN. Due to this the paths from two different 3PAR LUNs have the same "LUN Serial Number". This results in DMP incorrectly merging them under a single device/disk (DMPNODE) as the LUN Serial Number was not unique enough.
The OS (luxadm display output) clearly shows two different LUNs with different sizes and paths:
LUN #1
DEVICE PROPERTIES for disk: 2ff70002ac01ecb1
Vendor: 3PARdata
Product ID: VV
Revision: 3312
Serial Num: CZ3732JWLV
Unformatted capacity: 525312.000 MBytes
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):
/dev/rdsk/c4t21620002AC01ECB1d2101s2
/devices/pci@400/pci@1/pci@0/pci@0/SUNW,emlxs@0,1/fp@0,0/ssd@w21620002ac01ecb1,835:c,raw
LUN path port WWN: 21620002ac01ecb1
Host controller port WWN: 10000090fa05a8f3
Path status: O.K.
/dev/rdsk/c9t21320002AC01ECB1d2101s2
/devices/pci@400/pci@1/pci@0/pci@6/SUNW,emlxs@0,1/fp@0,0/ssd@w21320002ac01ecb1,835:c,raw
LUN path port WWN: 21320002ac01ecb1
Host controller port WWN: 10000090fa05a47d
Path status: O.K.
/dev/rdsk/c11t20320002AC01ECB1d2101s2
/devices/pci@400/pci@1/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20320002ac01ecb1,835:c,raw
LUN path port WWN: 20320002ac01ecb1
Host controller port WWN: 10000090fa05b80d
Path status: O.K.
/dev/rdsk/c16t20620002AC01ECB1d2101s2
/devices/pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20620002ac01ecb1,835:c,raw
LUN path port WWN: 20620002ac01ecb1
Host controller port WWN: 10000090fa05a77f
Path status: O.K.
LUN #2
DEVICE PROPERTIES for disk: 2ff70002ac01ecb1
Vendor: 3PARdata
Product ID: VV
Revision: 3312
Serial Num: CZ3732JWLV
Unformatted capacity: 262144.000 MBytes
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):
/dev/rdsk/c4t21620002AC01ECB1d2008s2
/devices/pci@400/pci@1/pci@0/pci@0/SUNW,emlxs@0,1/fp@0,0/ssd@w21620002ac01ecb1,7d8:c,raw
LUN path port WWN: 21620002ac01ecb1
Host controller port WWN: 10000090fa05a8f3
Path status: O.K.
/dev/rdsk/c9t21320002AC01ECB1d2008s2
/devices/pci@400/pci@1/pci@0/pci@6/SUNW,emlxs@0,1/fp@0,0/ssd@w21320002ac01ecb1,7d8:c,raw
LUN path port WWN: 21320002ac01ecb1
Host controller port WWN: 10000090fa05a47d
Path status: O.K.
/dev/rdsk/c11t20320002AC01ECB1d2008s2
/devices/pci@400/pci@1/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20320002ac01ecb1,7d8:c,raw
LUN path port WWN: 20320002ac01ecb1
Host controller port WWN: 10000090fa05b80d
Path status: O.K.
/dev/rdsk/c16t20620002AC01ECB1d2008s2
/devices/pci@400/pci@2/pci@0/pci@8/SUNW,emlxs@0,1/fp@0,0/ssd@w20620002ac01ecb1,7d8:c,raw
LUN path port WWN: 20620002ac01ecb1
Host controller port WWN: 10000090fa05a77f
Path status: O.K.
Resolution
3PAR arrays have been supported with DMP for a very long time. veritas has always been using page 0 offset to calculate the LSN, since 3PAR array support was provided. Changing the LSN make-up will affect all the 3PAR arrays that are deployed. The modified ASLAPM package will now take the new LSN from page 83, which will lead to an UDID mismatch being set on all the 3par disks.
LSN Examples:
Before ASLAPM have been deployed:
# vxdisk -p list |grep LUN_SERIAL
LUN_SERIAL_NO : CF5E0001B096
LUN_SERIAL_NO : CF5F0001B096
After ASLAPM has been deployed:
# vxdisk -p list |grep LUN_SER
LUN_SERIAL_NO : 60002AC0000000003900CF5E0001B096
LUN_SERIAL_NO : 60002AC0000000003900CF5F0001B096
The ASL changes will be applied to AIX, Linux and Solaris paltforms.
The ASLAPM version updated for InfoScale 7.3.1 is 7.3.1.102 on Solaris
# pkg info VRTSaslapm
Name: VRTSaslapm
Summary: Array Support Libraries and Array Policy Modules for Veritas Volume Manager
Description: Array Support Libraries and Array Policy Modules for Veritas
Volume Manager
Category: System/Administration and Configuration
State: Installed
Publisher: Veritas
Version: 7.3.1.102
Build Release: 5.11
Branch: None
Packaging Date: February 27, 2020 03:52:26 PM
Size: 1.95 MB
FMRI: pkg://Veritas/VRTSaslapm@7.3.1.102,5.11:20200227T155226Z
The 3PAR ASL changes will be planned for InfoScale 7.4.1.x.
Solaris ASLAPM link: https://sort.veritas.com/asl/details/864
AIX and Linux ASL updates are available on SORT as follows:
https://sort.veritas.com/asl/details/867
https://sort.veritas.com/asl/details/868
Issue/Introduction
Newly presented 3PAR LUNs may merge with existing DMPNODEs. The VxVM related errors may imply that the required Graceful DR (Dynamic LUN Reconfiguration) steps were not followed. VxVM uses a "LUN serial number" (LSN) to uniquely identify a LUN. This "LUN serial number" is obtained by performing a SCSI inquiry on Vital-Product-Data (VPD) page 0. The VPD contains information about a device that allows the device to be administered. This typically includes the product ID, vendor ID and other information specific device values enabling the LSN to be unique. In this instance the LSN was not unique enough.
