Uncorrectable write errors happens on volumes in Replicated Volume Group (RVG)

book

Article ID: 100032252

calendar_today

Updated On:

Description

Error Message

WARNING VxVM VVR vxio V-5-0-0 Volume datavol block 5037840: Uncorrectable write error
WARNING: msgcnt 2 mesg 039: V-2-39: vx_writesuper - /dev/vx/dsk/datadg/datavol file system super-block write error
WARNING: msgcnt 3 mesg 037: V-2-37: vx_metaioerr - vx_logbuf_write - /dev/vx/dsk/datadg/datavol file system meta data write error
WARNING: msgcnt 4 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/datadg/datavol file system disabled
WARNING: msgcnt 5 mesg 136: V-2-136: vx_force_unmount - /dev/vx/dsk/datadg/datavol file system has been force unmounted

Outpt from the vxtrace -lE command shows the following.

0 537027156 END reclaim vol volabc dg abcdg op 0 block 178971648 len 176 time 0 error 68 (Can't assign requested address)
47 1509054952572 ERROR write vol volabc dg abcdg time 0 op 0 block 178923344 len 512 error 6 (No such device or address)
48 1509054952572 ERROR write vol volabc dg abcdg time 0 op 0 block 178923856 len 512 error 6 (No such device or address)
49 1509054952572 ERROR write vol volabc dg abcdg time 0 op 0 block 178924368 len 512 error 6 (No such device or address)
50 1509054952572 ERROR write vol volabc dg abcdg time 0 op 0 block 178924880 len 512 error 6 (No such device or address)
0 1509054952572 ERROR write vdev volabc dg abcdg time 1509054952572 block 178923344 len 512 error 6 (No such device or address)

Cause

This issue is only applicable to UNIX/Linux systems running:
  • Veritas Volume Manager with VVR configuration
  • VxVM volumes under RVG
  • Smartsync is used in ODM
 
This issue is caused when accessing the flags in a stale pointer. IOs to VxVM with VVR needs 2 phases, SRL write and Data volume write, VxFS gets IO done after SRL write and doesn’t wait for Data Volume write completion, so if Data volume write gets started after VxFS frees IOHINT, it accesses stale one which may cause write IO error.

Resolution

A permanent fix has been identified by Veritas engineering and is currently under development/testing.
 
Below workaround can be used until a permanent fix is installed on the systems.
 
Workaround
1. Disable smartsync in odm 
  1. Stop Oracle instances
  2. Unmount ODM device (umount /dev/odm)
  3. Remount ODM device with 'nosmartsync' (mount -V vxodm -o nosmartsync /dev/odm /dev/odm)
  4. Restart Oracle instances
 
Above can be made permanent by updating the appropriate startup files.
 
(or)
 
2. Stop VVR replication
# vradmin –g stoprep
 
 
 

Issue/Introduction

Uncorrectable write errors are seen on volumes in RVG without any underlying HBA/SCSI errors when smartsync is enabled on Veritas Oracle Disk Manager (ODM). This may also cause errors in Veritas file systems (VxFS) resulting in disabled file system.

Additional Information

ETrack: 3737585
Powered by Wolken Software