InfoScale 7.4.1 is not discovering multiple NVME devices as separate enclosures

book

Article ID: 100045733

calendar_today

Updated On:

Description

Error Message

May 29 09:55:55 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] disabled controller /pci@307/pci@1/pci@0/pci@1/nvme@0 connected to disk array PHLE810101046P4BGN-
May 29 09:55:55 host vxdmp: [ID 241556 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] removed disk array PHLE810101046P4BGN-, datype = NVMe
May 29 09:55:55 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] disabled controller /pci@307/pci@1/pci@0/pci@2/nvme@0 connected to disk array PHLE810101046P4BGN-

May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] V-5-1-11551 array_da_to_disk: Cannot find active path for dmpnode host_nvme0_0
May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] 
May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] V-5-1-11551 array_da_to_disk: Cannot find active path for dmpnode host_nvme0_0
May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] 

May 29 09:55:55 host vxdmp: [ID 581508 kern.notice] NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-, datype = NVMe
May 29 09:55:55 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] enabled controller /pci@307/pci@1/pci@0/pci@2/nvme@0 connected to disk array PHLE810101046P4BGN-

May 29 09:57:50 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] enabled controller /pci@307/pci@1/pci@0/pci@1/nvme@0 connected to disk array PHLE810101046P4BGN-
May 29 09:58:16 host vxdmp: [ID 241556 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] removed disk array PHLE810101046P4BGN-, datype = NVMe 

Cause

NVMe (Non-Volatile Memory Express) is a host controller interface & storage protocol created to accelerate the transfer of data using solid-state drives (SSDs)

Uses a servers high-speed Peripheral Component Interconnect Express (PCIe) interface.

Escalation: STESC-3229

InfoScale 7.4.1 running Solaris 11.4 is unable to detect multiple NVMe devices correctly due to Cabinet (Enclosure)Serial Number (CSN) length. This can potential result in only a single NVMe device being discovered and others excluded from the DDL (Device Discovery Layer) claiming process.

The Solaris nvmeadm command can be used to display NVMe related device information:
 

# nvmeadm  list -v

SUNW-NVME-1
      PCI Vendor ID:                  0x8086
      Serial Number:                  PHLE810101046P4BGN-1
      Model Number:                   7335943:ICDPC5ED2ORA6.4T
      Firmware Revision:              QDV1RD22
      Number of Namespaces:           1

SUNW-NVME-2
       PCI Vendor ID:                  0x8086
       Serial Number:                  PHLE810101046P4BGN-2  
       Model Number:                   7335943:ICDPC5ED2ORA6.4T
       Firmware Revision:              QDV1RD22
       Number of Namespaces:           1

 

The DMP_MAX_CAB_SNO_NVME was originally defined with a length of 20 bytes. Therefore, DMP cuts the string too short, hence why, in instance only “PHLE810101046P4BGN-” is detected by DMP. DMP fails to recognize other NVMe devices (enclosures).

VxVM uses only “PHLE810101046P4BGN-” as the Enclosure (Cabinet) serial number for a NVMe devices.

May 30 14:24:22 host vxdmp: [ID 581508 kern.notice] NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-, datype = NVMe"

By increasing the size of the DMP_MAX_CAB_SNO_NVME attribute size, the product will now be able to discover multiple NVME devices.

 

Resolution

Once the updated VRTSaslapm package has been applied, unique enclosures are detected for multiple NVMe devices (enclosures).

 

NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array OTHER_DISKS, datype = OTHER_DISKS
NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-17335943:ICDP, datype = NVMe
NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array 000296700818, datype = EMC
NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-27335943:ICDP, datype = NVMe

 

By increasing the DMP_MAX_CAB_SNO_NVME attribute size, the product is now able to discover multiple NVME devices as individual enclosures.

The output from "vxdisk -p list" now clearly shows the unique CAB_SERIAL_NO for each NVMe device:

DISK                : host_nvme1_0
VID                 : NVMe
UDID                : NVMe%5F8086108E%5FPHLE810101046P4BGN-17335943%3AICDP%5F14F1F700005CD2E40000000100000001
SCSI_VERSION        : 0
PID                 : 8086108E
PHYS_CTLR_NAME      : /pci@307/pci@1/pci@0/pci@1/nvme@0
PGR_CAPABLE         : N
MEDIA_TYPE          : ssd
LUN_TYPE            : std
LUN_SNO_ORDER       : 0
LUN_SERIAL_NO       : 14F1F700005CD2E40000000100000001
LIBNAME             : libvxnvme_sol.so
DMP_DEVICE          : host_nvme1_0
DDL_DEVICE_ATTR     : ssd
CONN_TYPE           : local
CAB_SERIAL_NO       : PHLE810101046P4BGN-17335943:ICDP
ATYPE               : A/A
ANAME               : NVMe
TRANSPORT           : SCSI
ENCLOSURE_NAME      : host_nvme1
DMP_SINGLE_PATH     : /dev/rdsk/c2t1d0
LUN_SIZE            : 6251233968
NUM_PATHS           : 1
STATE               : online
DISK_TYPE           : auto
FORMAT              : cdsdisk
DA_INFO             : format=cdsdisk,privoffset=208,pubslice=2,privslice=2
PRIV_OFF            : 208
PRIV_LEN            : 65536
PUB_OFF             : 65744
PUB_LEN             : 6251168128
PRIV_UDID           : NVMe%5F8086108E%5FPHLE810101046P4BGN-17335943%3AICDP%5F14F1F700005CD2E40000000100000001
DG_NAME             : sfcache_defaultdg
DGID                : 1559309762.168.host
DG_STATE            : deported
DISKID              : 1559058982.16.host
DISK_TIMESTAMP      : Tue May 28 05:56:22 PM 2019

DISK                : host_nvme2_0
VID                 : NVMe
UDID                : NVMe%5F8086108E%5FPHLE810101046P4BGN-27335943%3AICDP%5F14F1F700005CD2E40000000100000001
SCSI_VERSION        : 0
PID                 : 8086108E
PHYS_CTLR_NAME      : /pci@307/pci@1/pci@0/pci@2/nvme@0
PGR_CAPABLE         : N
MEDIA_TYPE          : ssd
LUN_TYPE            : std
LUN_SNO_ORDER       : 0
LUN_SERIAL_NO       : 14F1F700005CD2E40000000100000001
LIBNAME             : libvxnvme_sol.so
DMP_DEVICE          : host_nvme2_0
DDL_DEVICE_ATTR     : ssd
CONN_TYPE           : local
CAB_SERIAL_NO       : PHLE810101046P4BGN-27335943:ICDP
ATYPE               : A/A
ANAME               : NVMe
TRANSPORT           : SCSI
ENCLOSURE_NAME      : host_nvme2
DMP_SINGLE_PATH     : /dev/rdsk/c3t1d0
LUN_SIZE            : 6251233968
NUM_PATHS           : 1
STATE               : online
DISK_TYPE           : auto
FORMAT              : none
DA_INFO             : format=none

 

The patch is available for download from SORT site:

https://docs.infoscale.com/

 

Issue/Introduction

InfoScale 7.4.1 is not discovering multiple NVME devices as separate enclosures.

In this example, only a single NVMe enclosure is being reported by DMP.


# vxdmpadm listctlr all output
CTLR_NAME ENCLR_TYPE STATE ENCLR_NAME PATH_COUNT
=========================================================================
c0 EMC ENABLED emc0 145
c2 NVMe ENABLED nns8a2_nvme0 1
c1 OTHER_DISKS ENABLED other_disks 2

The format command shows the NVMe devices as c2t1d0 and c3t1d0:
0. c2t1d0 <Intel-7335943:ICDPC5ED-QDV1-2.91TB> /pci@307/pci@1/pci@0/pci@1/nvme@0/disk@1 /dev/chassis/SYS/MB/PCIE2/NVME0/disk 1. c3t1d0 <Intel-7335943:ICDPC5ED-QDV1-2.91TB> /pci@307/pci@1/pci@0/pci@2/nvme@0/disk@1 /dev/chassis/SYS/MB/PCIE2/NVME1/disk In the below output, NVMe device (c3t1d0) is excluded by the DDL (Device Discovery Layer), so will not be shown in any of the VxVM outputs.

Powered by Wolken Software