LDOM SR-IOV: When using Veritas DMP unknown HBA port state will not go online until system is rebooted
Article ID: 100050786
Updated On:
Description
Error Message
The LDOM Guest will report the following syslog messages in relation to the impacted qlogic HBA:
Apr 24 07:16:52 fred pcie: [ID 286789 kern.notice] NOTICE: Live Suspend: qlc1 suspended successfully
Apr 24 07:20:23 fred pcie: [ID 914331 kern.notice] NOTICE: Live Resume: qlc1 resumed successfully
Apr 24 07:20:23 fred genunix: [ID 507916 kern.notice] NOTICE: ddihp_lsr_resume_branch: 40005599510:fp#-1 is not suspended
Apr 24 07:20:23 fred genunix: [ID 564907 kern.warning] WARNING: ddihp_lsr_resume: fp2: driver failed to resume
The "fcinfo hba-port" command displays HBA related information (model, manufacture, firmward etc) and most importantly the HBA status.
When using Veritas DMP, Solaris is unable to reactivate the "unknown" HBA state, back to an “online” operational state without rebooting the host.
# fcinfo hba-port
HBA Port WWN: 100000144ffaa515
Port Mode: Initiator
Port ID: 32bcc1
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 7023303
Firmware Version: 8.08.04
FCode/BIOS Version: BIOS: 3.43; fcode: 4.10; EFI: 6.20;
Serial Number: 463916R+1525241484
Driver Name: qlc
Driver Version: 181115-5.09
Type: N-port
State: unknown <<<<<< it should reflect online
Supported Speeds: 4Gb 8Gb 16Gb
Current Speed: 16Gb
Node WWN: 200000144ffaa515
NPIV disabled
Cause
Even with Solaris SRU 35 or higher, Veritas Dynamic Multi-pathing (DMP) is still unable to work correctly with LDOM SR-IOV enabled devices.
To displaying SR-IOV related information, type:
# dladm show-phys -V
# dladm show-vnic -V
In addition, the impacted “c3” controller is reported a cfgadm access path state of “failed”, instead of expected “failing|unusable” access path states.
Check:
# cfgadm -alo show_FCP_dev
# vxdmpadm listctlr all
CTLR_NAME ENCLR_TYPE STATE ENCLR_NAME PATH_COUNT
=========================================================================
c1 SunF80- ENABLED fred_sunf80-0 1
c1 SunF80- ENABLED fred_sunf80-1 1
c2 EMC ENABLED emc0 158
c3 EMC DISABLED emc0 158 <<<< IMPACTED controller (HBA)
c7 EMC ENABLED emc0 158
Resolution
Do not use the SR-IOV functionality with Veritas DMP.
Futher SR-IOV development efforts are required by both vendors.
See related article:
LDOM features FC-IOR (Fibre channel IO resiliency feature ) of SR-IOV are not supported with Storage Foundation (6.2.1.x and below) and InfoScale product releases 7.0 and higher
https://www.veritas.com/support/en_US/article.100051695
Issue/Introduction
When using Veritas Dynamic Multi-pathing (DMP) with SR-IOV enabled, devices will not transition from "unknown" back to "online", until the system is rebooted.
Following the loss of access to a HBA, the HBA state cannot be changed from "unknown" back to a working "online" state until the LDOM is restarted.
This is related to an on-going Solaris issue unfortunately still not addressed with Solaris 11.4 SRU 35 or higher.
Environment:
SR-IOV Solaris 11 to share LUNs to the LDOM
======================================
Solaris 11.4.29.82.3 Sparc
InfoScale 7.4.1
Service domain version :11.4
HW: oracle M6
HBA: Qlogic
Veritas Module information :
ID LOADADDR SIZE INFO REV NAMEDESC
187 11000000 615c8 4 1 vxdmp (VxVM 7.4.1.2600 Multipathing Dr)
195 7bc00000 3507f8 13 1 vxio (VxVM 7.4.1.2600 I/O driver)
202 7befa2d8 11e8 18 1 vxspec (VxVM 7.4.1.2600 control/status )
252 7bf9c000 7d20 283 1 vxlan (VXLAN tunneling driver)
256 11bafaf8 21408 308 1 vxcafs (vxcafs 7.4.1.1301 Solaris OS 5.)
257 7ac00000 3d2548 28 1 vxfs (VxFS 7.4.1.1301 Solaris OS 5.11)
258 7b4e6000 62540 309 1 vxfen (VRTS Fence 7.4.1)
259 11ae0dc8 2cb90 311 1 vxglm (VxGLM 7.4.1.0 SunOS 5.11)
260 113f9ca0 2a68 312 1 vxportal (VxFS 7.4.1.1301 Solaris OS 5.11)
262 113fc588 20d8 316 1 vxlo (Veritas Loopback Driver 0.1)
264 7afd4000 4680 318 1 vxgms (VxGMS 7.4.1.0 Solaris)
