Error Message

The corruption will remain undetected until the file is removed.  When the affected file is removed, the following VxFS system error messages will be logged.   (The messages will be logged by the Cluster File System (CFS) primary in a CFS environment.)

2011 Aug 18 01:29:26 sydn8k_02 kernel: vxfs: msgcnt 6 mesg 096: V-2-96: vx_setfsflags - /dev/vx/dsk/sfsdg/volalaw file system fullfsck flag set - vx_ierror
2011 Aug 18 01:29:26 sydn8k_02 kernel: vxfs: msgcnt 7 mesg 017: V-2-17: vx_te_trunc_data - /dev/vx/dsk/sfsdg/volalaw file system inode 19 marked bad incore
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000000     81a4        0        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000010        0        0 4e4ade37    3e767
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000020 4e4963c0    522e2 4e4cf796    55f8a
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000030    10300 49a93009        8        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000040        0        0        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000050        0        0   39b930        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000060    62a7b        0        0  1000000
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000070    56218        8   a7400c  1000000
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000080   aecff0        8        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x00000090        0        0        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x000000a0        0        0        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x000000b0        0        0        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x000000c0        0        0        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: ?vxfs msgcnt 7 offset 0x000000d0        0        0
2011 Aug 18 01:29:26 sydn8k_02 kernel: vxfs: msgcnt 8 mesg 079: V-2-79: vx_tranuninode - /dev/vx/dsk/sfsdg/volalaw file system inode 19 marked bad ondisk

The corruption can also be detected by fsck when a full fsck is run.

# fsck.vxfs -o full -y /dev/vx/rdsk/sfsdg/volalaw
log replay in progress
pass0 - checking structural files
pass1 - checking inode sanity and blocks
fileset 999 primary-ilist inode 19 has invalid number of blocks (3782960)
fileset 999 primary-ilist inode 19 has invalid block map
fileset 999 primary-ilist inode 19 failed validation clear? (ynq)y
rebuild structural files? (ynq)y
pass0 - checking structural files
pass1 - checking inode sanity and blocks
pass2 - checking directory linkage
fileset 999 directory 8 block devid/blknum 0/0 offset 4 references free inode
                ino 19 remove entry? (ynq)y
fileset 999 directory 8 block devid/blknum 0/0 incorrect header fix? (ynq)y
pass3 - checking reference counts
pass4 - checking resource maps
fileset 999 au 0 imap incorrect - fix (ynq)y
au 0 emap incorrect - fix? (ynq)y
au 0 summary incorrect - fix? (ynq)y
au 3 emap incorrect - fix? (ynq)y
au 3 summary incorrect - fix? (ynq)y
....
au 1599 emap incorrect - fix? (ynq)y
au 1599 summary incorrect - fix? (ynq)y
fileset 999 iau 0 summary incorrect - fix? (ynq)y
free block count incorrect 8229034 expected 12011994 fix? (ynq)y
free extent vector incorrect fix? (ynq)y
OK to clear log? (ynq)y
flush fileset headers? (ynq)y
set state to CLEAN? (ynq)y
 

The problem is caused by the incident listed in the supplemental material section.

The problem is fixed in VxFS 5.1 GA on all platforms. 

VxFS 4.1 on HP is not impacted by this incident.

The problem is fixed in the following VxFS 5.0 patch levels:

Linux Platform - Storage Foundation 5.0MP4
Solaris Platform - Storage Foundation 5.0MP3RP5
                       - Storage Foundation 5.0MP1RP4HF15
AIX Platform -  Storage Foundation 5.0 MP3 RP5
HP-UX 11.23 Platform - Storage Foundation 5.0 MP2 RP1
HP-UX 11.31 Platform - Storage Foundation 5.0.1 GA

 HP-UX 11.31 Platform - Storage Foundation 5.0GARP6

The above patches available from the Veritas Operation Readiness Tools website.

https://sort.Veritas.com/patch/matrix

Fix will be available in future Filestore patch releases.

How to confirm if corruption happened on the filesystem
----------------------------------------------------------------------------
Since VxFS will not log any system error messages until the file is deleted, the following procedure can be used to verify the filesystem.

1. Please follow the procedure in 000080853 to collect the metasave from the filesystem

2. Replay the metasave back to a file.  Please note that the name of metasave program may vary from platform to platform.  Please use the correct version of metasave.   For example, metasave_10 is for Solaris 10, assumed the metasave output is "metasave.volname.out".

# touch vxfsimage.volname
# /opt/VRTSspt/FS/MetaSave/metasave_10   -r   -f  metasave.volname.out   vxfsimage.volname

3. Run a full fsck on the restored VxFS file system image.

# fsck -F vxfs -o full -y vxfsimage.volname

4. Check if fsck reports any filesystem corruption.   If fsck reports "inode  has invalid block map", then the file has an invalid block map.  For example,

fileset 999 primary-ilist inode 19 has invalid number of blocks (3782960)
fileset 999 primary-ilist inode 19 has invalid block map            <<< invalid block map
fileset 999 primary-ilist inode 19 failed validation clear? (ynq)y
 

If "invalid block map" is detected, then please do a full file system backup immediately.   Please restore the backup data to a clean file system and verify if the data is intact.  Please make sure there is a valid backup of all the data files before proceeding to fix the problem as files will be removed during the procedure.

After a valid backup of the data is available, then please apply the required patch to fix the incident to prevent further corruption.

After the required patch is applied and the system is rebooted, then umount the filesystem and run a full fsck on the filesystem.  Please note that a full fsck will remove all inodes with invalid block map or with other inode issues that can't be fixed.    Please make sure there is a valid backup copy of the data before running full fsck.

After fsck removed the corrupt files, you can mount the filesystem and start to restore the data from the backup.

Applies To

The problem affects Veritas File System 5.0 or earlier versions without the required patch installed. 

The problem can happen on all platforms.

The problem also affects FileStore 5.5 without the required FileStore patch installed.

Files are more susceptible to the corruption if they are highly fragmented.   A file can be fragmented when it is randomly written with small data blocks.