VCS 4 node Cluster experiencing chronic scsi "reservation conflict" warnings; vxdisk-list indicates VxVM vxdisk ERROR V-5-1-539 Device tagmastore-usp0_2113: get_contents failed
Article ID: 100026829
Updated On:
Description
Error Message
Multiple nodes experiencing similar scsi errors:
Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict
Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry
Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict
Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict
Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211):
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211):
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict
Cause
---When VCS managed VxVM disk group is vxdg-imported on one node, VCS i/o fencing applies SCSI-3 PGR registration and write-exclusive reservation keys on the disk group data disks; for example:
# vxfenadm -g /dev/rdsk/c1t50060E80047E0427d71s2
Reading SCSI Registration Keys...
Device Name: /dev/rdsk/c1t50060E80047E0427d71s2
Total Number Of Keys: 2
key[0]:
Key Value [Numeric Format]: 67,86,67,83,0,0,0,0
Key Value [Character Format]: CVCS
key[1]:
Key Value [Numeric Format]: 67,86,67,83,0,0,0,0
Key Value [Character Format]: CVCS
vxfenadm -r /dev/rdsk/c1t50060E80047E0427d71s2
Reading SCSI Reservation Information...
Device Name: /dev/rdsk/c1t50060E80047E0427d71s2
Total Number Of Keys: 1
Key[0]:
Reservation Type: SCSI3_RESV_WRITEEXCLUSIVEREGISTRANTSONLY
Key Value [Numeric Format]: 67,86,67,83,0,0,0,0
Key Value [Character Format]: CVCS
---- but other nodes on the cluster experience "reservation conflict" warnings such as................
Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict
Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict
Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.w
Resolution
---- While array was placing reservation key SCSI3_RESV_WRITEEXCLUSIVEREGISTRANTSONLY the reservation key was behaving READEXCLUSIVEREGISTRANTSONLY instead. This prevented other hosts from even reading the LUNs imported to the other host. Commands like prtvtoc and dd read would fail.
----Discovered the Hitachi Tagmastore UPS600 host mode option settings were not complete. HMO2 was not set. Required settings to support SFHA for both DMP and VCS I/O Fencing are:
Host mode 09 = Solaris hosts
Host mode option 02 = VCS, CVM-CFS, SFRAC I/O fencing configured
Host mode option 07 = Auto-recognize array LUN resize;
* requires host mode option 09 and latest Solaris 10 HBA drivers
Host mode option 22 = VCS cluster with common disk groups for failover or parallel LUN I/O access; (if AIX requires host mode 0F)
Applies To
----Cluster node SFHA is in upgrade transition from 5.0MP3 to 5.1SP1RP1P1
----SF is being upgraded on nodes in phases while production service group migrated between upgraded nodes
----VCS will remain 5.0MP3 until SF is upgraded
----Solaris MPxIO is also being replaced by VxDMP
----also under VxVM 5.1SP1RP1P1 "dmp_native_support" is enabled to support ZFS LUNs
$ egrep 'SunOS' */uname_a ; cat */etc/release
SunOS nodedb01 5.10 Generic_144488-12 sun4u sparc SUNW,SPARC-Enterprise
Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Assembled 11 August 2010
$ egrep 'vx|llt|gab|emcp' */modinfo
29 1380800 50c10 317 1 vxdmp (VxVM 5.1SP1RP1P1 DMP Driver)
31 7be00000 21a918 318 1 vxio (VxVM 5.1SP1RP1P1 I/O driver)
33 13caa80 1190 319 1 vxspec (VxVM 5.1SP1RP1P1 control/status)
226 7bbf7308 d40 320 1 vxportal (VxFS 5.1_SP1RP1 portal driver)
227 7a600000 1fe098 21 1 vxfs (VxFS 5.1_SP1RP1 SunOS 5.10)
228 7ab18000 26470 323 1 llt (LLT 5.0MP3)
229 7ab38000 4a028 324 1 gab (GAB device 5.0MP3)
230 7ab6a000 51c10 325 1 vxfen (VRTS Fence 5.0MP3)
$
bash-3.00# uname -a
SunOS nodedb03 5.10 Generic_138888-06 sun4u sparc SUNW,SPARC-Enterprise
bash-3.00# modinfo | egrep 'vx|llt|gab|emcp'
36 13428a8 3e4e0 317 1 vxdmp (VxVM 5.0MP3: DMP Driver)
38 7be00000 209248 318 1 vxio (VxVM 5.0MP3 I/O driver)
40 7bfe90f0 c78 319 1 vxspec (VxVM 5.0MP3 control/status driv)
220 7af953f0 cb0 320 1 vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
221 7aa00000 1d89e0 21 1 vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
222 7abc8000 26470 322 1 llt (LLT 5.0MP3)
223 7a600000 4a028 323 1 gab (GAB device 5.0MP3)
224 7a632000 51c10 324 1 vxfen (VRTS Fence 5.0MP3)
# uname -a
SunOS nodedb25 5.10 Generic_144488-12 sun4u sparc SUNW,SPARC-Enterprise
# modinfo | egrep 'vx|llt|gab|emcp'
29 1380800 50c10 324 1 vxdmp (VxVM 5.1SP1RP1P1 DMP Driver)
31 7be00000 21a918 325 1 vxio (VxVM 5.1SP1RP1P1 I/O driver)
33 13caa80 1190 326 1 vxspec (VxVM 5.1SP1RP1P1 control/status)
227 7bbf5308 d40 327 1 vxportal (VxFS 5.1_SP1RP1 portal driver)
228 7a600000 1fe098 21 1 vxfs (VxFS 5.1_SP1RP1 SunOS 5.10)
229 7ab1e000 26470 329 1 llt (LLT 5.0MP3)
230 7ab3e000 4a028 330 1 gab (GAB device 5.0MP3)
231 7ab70000 51c10 331 1 vxfen (VRTS Fence 5.0MP3)
232 7aba2000 21ec0 332 1 vxglm (VxGLM 5.0MP3 (SunOS 5.10))
233 7ae1c000 5418 333 1 vxgms (VxGMS 5.0MP3 (SunOS))
bash-3.00# uname -a
SunOS nodedb24 5.10 Generic_138888-06 sun4u sparc SUNW,SPARC-Enterprise
bash-3.00# modinfo | egrep 'vx|llt|gab|emcp'
36 13428a8 3fdc0 322 1 vxdmp (VxVM 5.0MP3RP4: DMP Driver)
38 7be00000 211590 323 1 vxio (VxVM 5.0MP3RP4 I/O driver)
40 7bff0428 c78 324 1 vxspec (VxVM 5.0MP3RP4 control/status d)
222 7ae9d180 cb0 325 1 vxportal (VxFS 5.0_REV-5.0MP3RP4g_sol por)
223 7aa00000 1dd7d0 21 1 vxfs (VxFS 5.0_REV-5.0MP3RP4g_sol Sun)
224 7abcc000 26470 327 1 llt (LLT 5.0MP3)
225 7a600000 4a028 328 1 gab (GAB device 5.0MP3)
226 7a632000 51c10 329 1 vxfen (VRTS Fence 5.0MP3)
Issue/Introduction
