Description

This article contains the procedure to detect and correct inode corruption associated with transient fiber link failures.

Warning: Read this article, in its entirety, before making any changes. Understand that failure to follow the proper procedures could lead to data loss. Veritas should be contacted in conjunction with using this article. We recommend doing a complete backup of the data, if possible, prior to correcting inode corruption to help prevent data loss.

Procedure

This is the procedure to help detect and correct Veritas File System inode corruption as a result of transient fiber link failures.  Under certain conditions, incore inodes can be marked bad if all paths to a device have been disabled.  If the paths are re-enabled, and the file system is still enabled, it is then possible for these incore inodes to be flushed to disk and the superblock marked as needing a full fsck.  A subsequent full fsck will clear these inodes; deleting the file.  This procedure is most relevant to inodes marked bad due to read failures, as the inode was not being updated at the time, and has the most probability of being recovered successfully.

After a link failure has been detected, the /var/adm/messages file should be analyzed for possible inode failures.  File System will print "vxfs:" messages to /var/adm/messages,  and will usually indicate which inodes have been marked bad.  

A typical message looks like this:
 
This indicates inode 10 on file system /opt/data/ora16/preprod was marked bad.  This is expected behavior, however, since the paths to the device were reset and the file system was still enabled, inode 10 was flushed to disk.  Since this was a read failure, the likelihood that inode 10 is actually corrupt is very small.

Note: Veritas Support should be contacted if file system corruption is suspected.

Procedure to Delete and Correct

1. Unmount the file system in order to attempt repairs on corrupted inodes.  The superblock can be analyzed on the failing file system to verify that it has been marked as needing a full fsck using the following command:
 
The actual device can be obtained from the vfstab file. 

The output will look something like this:
 
The key is the flags field. In this case it is "301", which breaks down to mean: VX_FULLFSCK | VX_METAIOERR | VX_DATAIOERR per the following defines:
 

2. Now that it is known that this file system has corruption, it is a good idea to perform a full backup of your data.  Also recommended is to dump the metadata with the "metasave" utility. Saving the metadata is a good idea in case there are problems with fsdb later on.

3.  Run a full fsck with the -n option to see which inodes are marked bad:
 

This indicates that inodes 2, 3, and 10 are marked bad.  

4.  Set the "aflag" field to 0x0 using fsdb.  This step must be done very carefully since it involves writing to the file system structure itself. The incorrect use of fsdb can destroy the file system. 
   
Now, clear inodes 2, 3, and 10:
 

Again, the device needs to be the raw device for the file system.

5.  The inode aflag has been cleared for the 3 inodes.  Now verify with fsck:
 
6.  Now it should be safe to run a full fsck with the -y option:
 

7.  Mount the file system and check the inodes:
 

How to detect and correct inode corruption associated with transient fiber link failures