Error Message

Sample messages "FAILIO" flag set locally:
 
"New VxVM error messaging"

Jun  5 14:29:22 serverd kernel: VxVM vxdmp V-5-3-0 Reached DMP Threshold IO TimeOut (1 secs) I/O with start 4fb1d6fb25cae and end 4fb1d703abf6e time
Jun  5 14:29:22 serverd kernel:
Jun  5 14:29:22 serverd kernel: VxVM vxdmp V-5-0-0 [Error] i/o error occurred (errno=0x206) on dmpnode 201/0x30
Jun  5 14:29:22 serverd kernel:
Jun  5 14:29:22 serverd kernel: VxVM vxio V-5-3-0 voldiskiodone: VOLD_FLAG_FAIL_IO flag set on disk ibm_shark0_2
Jun  5 14:29:22 serverd kernel: VxVM vxio V-5-0-1266 Subdisk ibm_shark0_2-01 block 131048: Uncorrectable write error
Jun  5 14:29:22 serverd kernel: VxVM vxdmp V-5-3-0 I/O failed on path 8/0x50 after 1 retries for disk 201/0x30
 

In the above example, the "FAILIO" flag has been set locally against dmpnode/Veritas disk access (da) name "ibm_shark0_2" on server "d".
 

Figure 1.0


 

Sample vxassist mirror error message

VxVM vxplex ERROR V-5-1-1276 Volume datavol, plex datavol-01, block 0: Plex read:        Error: Read failure
VxVM vxplex ERROR V-5-1-1273 Volume datavol, block 0: plex copy error:        Success
VxVM vxplex ERROR V-5-1-16498 I/O error on volume datavol, plex datavol-02 attach not completed
VxVM vxassist ERROR V-5-1-502 Could not attach new mirror(s) to volume datavol

In the above example, the current attached plex has the "FAILIO" flag sent locally on the CVM master server and prevents new vxplex attach operations from completing until the "FAILIO" flag is cleared or the CVM master role is switched.

SYMPTOM: 
A Veritas Volume Manager (VxVM) volume may become inaccessible causing any read/write to fail with the following error:

# dd if=/dev/vx/dsk// of=/dev/null count=10
dd read error: No such device
0+0 records in
0+0 records out


DESCRIPTION:

If I/Os to the disks timeout due to some hardware failures like weak Storage Area Network (SAN) cable link or Host Bus Adapter (HBA) failure,
VxVM assumes that disk is bad or slow and it sets failio flag on the disk. Because of this flag, all the subsequent I/Os fail with the ‘No such device’ error.

RESOLUTION:

The code is modified so that vxdisk now provides a way to clear the ‘failio’ flag.
Use the vxkprint(1M) utility (under /etc/vx/diag.d) to check whether the ‘failio’ flag is set on the disks.
To reset this flag, execute the ‘vxdisk set <disk_name> failio=off’ command, or deport and import the disk group that holds these disks.

 

Scenario #1:

If the DMP recoveryoption for an enclosure has been defined with a timebound value of 300 seconds, and the SCSI layer takes longer than the expected 300 second (DMP timebound threshold value) window to fail the I/O, then DMP will not retry the I/O.

As a result of the DMP I/O threshold timeout being exhausted, the corresponding plex is detached and marked with the "DETACHED IOFAIL" plex state.

Scenario #2:
 

The volume consists of two plexes, plex A is attached and working fine, plex B is currently detached due to DMP I/O timeout event.

In the event, that the surviving plex is also impacted as a result of a DMP I/O threshold timeout event, the last remaining (surviving) attached plex is not detached.
The hidden "FAILIO" flag will be set against both plexes in this instance.


NOTE: The volume will only be detached when a klog write error on the volume is encountered, otherwise,  I/O error messages will continued to be reported in the syslog file solely against the impacted server.

The last surviving plex is never detached from a volume, the hidden flag is set and the plex remains attached as “Enabled Active”.
 

The enhanced VxVM functionality will be available with the release of the 6.1 MR1 patch. A series of private hot-fixes are available for Linux platforms at this time

6.0.300.204 for Linux
6.0.500.002 for Linux
 

Please contact Veritas support if you require hot-fixes or patches for other platform releases.

 

Workarounds:

Impact:

The product will not be able to properly service I/O’s through plexes which have the “ FAILIO ” flag set locally on the impacted server. Further incoming I/O ’s will experience read/write errors.

Traditionally there were 3 methods to clear the DETACHED IOFAIL “ FAILIO ” flag state:

1.] Deport the impacted disk group

2.] Recycle vxconfigd with the options –k and –r reset ( vxconfigd –k –r reset)

3.] Reboot the impacted server

Normal business operations will resume once the "FAILIO" flag has been cleared. The disk “ FAILIO ” flag is set for all cases whenever a DMP I/O timeout is experienced on a disk.

4.] Switch CVM master role

In CVM environments where the CVM master server has the "FAILIO" flag set locally for the last remaining (surviving) plex. Switching the CVM master role to a working server (where the FAILIO flag is not), will enable the attach/creation of a plex to the existing volume.

# vxclustadm setmaster

Figure 2.0
 



 

Veritas Volume Manager (VxVM) enhancement overview:

Sample Messages:

The server which encounters the DMP threshold I/O timeout event will record the setting of the "FAILIO" flag locally in the syslog file with the string " VOLD_FLAG_FAIL_IO flag set".

Jun  5 14:29:22 serverd kernel: VxVM vxio V-5-3-0 voldiskiodone : VOLD_FLAG_FAIL_IO flag set on disk ibm_shark0_2
 

Sample vxkprint :

The hidden "FAILIO" flag can be displayed using the revised "vxkprint" utlity.


# /etc/ vx / diag.d / vxkprint > kprint_out

Disk ibm_shark0_2 : dm=ibm_shark0_2 dgiid =1024.14 darid =1024.4 dmrid =0.1026
.
.
  failio flag=1


 
With the enhanced VxVM functionality, the user is now able to reset the hidden DETACHED IOFAIL “ FAILIO ” flag.

# vxdisk set ibm_shark0_2 failio =off

 

Note: The vxkprint format will change in future versions, i.e. InfoScale 7.2.x, failio flag=1 replaced with dflag=(failio).
 

New format:

# /etc/vx/diag.d/vxkprint

Disk disk_2: dm=B00A21B71027DF65100F42DB93 dgiid=1024.9 darid=1024.4 dmrid=0.1027
  kflag=(efi)
  sflag=(sdopen)
  vflag=(autoconfig|online)
  dflag=()
.
.
Disk disk_3: dm=B00A21B71027DF65130F652F3C dgiid=1024.9 darid=1024.6 dmrid=0.1026
  kflag=(failing|efi|unknown=0x40000)
  sflag=(sdopen)
  vflag=(autoconfig|online)
  dflag=(failio)

 

Applies To

Cross platform

In the event that a plex reports a state of DETACHED IOFAIL , subsequent vxplex attach operations (resync) may fail due to a hidden " FAILIO " disk related kernel flag.
The hidden flag is only visible from specific Veritas Volume Manager (VxVM) versions. With specific enhanced VxVM product releases customers are now able to see and clear the hidden DETACHED IOFAIL “ FAILIO ” flag.

Until this hidden flag is cleared, subsequent plex related operations may encounter read/write errors.