Error Message

Full fsck needs to be run on the filesystem, but after performing full fsck on the filesystem, getting the following error.

# fsck -F vxfs -o full -y /dev/vx/rdsk/dgname/volname
UX:vxfs fsck: ERROR: V-3-20451: No valid device inodes found
UX:vxfs fsck: ERROR: V-3-20694: cannot initialize aggregate
file system check failure, aborting ...

The filesystem superblock is corrupted with the following pattern.

# fstyp -v /dev/vx/rdsk/dgname/volumename
vxfs
magic a501fcf5  version 7  ctime September  2, 2012  5:59:22 AM CDT
logstart 0  logend 0
bsize  1024 size  0 dsize  0  ninode 0  nau 0   ==> we can see that size is zero, which means that the superblock is corrupted.
defiextsize 0  ilbsize 0  immedlen 96  ndaddr 10
aufirst 0  emap 0  imap 0  iextop 0  istart 0
bstart 0  femap 0  fimap 0  fiextop 0  fistart 0  fbstart 0
nindir 2048  aulen 32768  auimlen 0  auemlen 8
auilen 0  aupad 0  aublocks 32768  maxtier 15
inopb 4  inopau 0  ndiripau 0  iaddrlen 8 bshift 10
inoshift 2  bmask fffffc00  boffmask 3ff  checksum f5527691
oltext1 338  oltext2 524  oltsize 1  checksum2 0
free 0  ifree 0
efree  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Or

Reading the super block information using fsdb

# echo '0x2000 B.p S'|/opt/VRTS/bin/fsdb /dev/vx/rdsk/dgname/volname
super-block at 00000008.0000
magic a501fcf5  version 7
ctime 1379655280 415330  (Fri Sep 20 00:34:40 2013 CDT)
log_version 0 logstart 0  logend 0
bsize  1024 size  0 dsize  0  ninode 0  nau 0
defiextsize 0  oilbsize 0  immedlen 96  ndaddr 10
aufirst 0  emap 0  imap 0  iextop 0  istart 0
bstart 0  femap 0  fimap 0  fiextop 0  fistart 0  fbstart 0
nindir 2048  aulen 32768  auimlen 0  auemlen 8
auilen 0  aupad 0  aublocks 32768  maxtier 15
inopb 4  inopau 0  ndiripau 0  iaddrlen 8   bshift 10
inoshift 2  bmask fffffc00  boffmask 3ff  checksum f7443af4
free 0  ifree 0
efree  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
flags 1 mod 0 clean 0
time 0 0  (Wed Dec 31 18:00:00 1969 CDT) 
oltext[0] 32  oltext[1] 1282  oltsize 1
iauimlen 1  iausize 4  dinosize 256
checksum2 0  checksum3 0
fsetquotaction 0
fsetquotahardlimit 0 fsetquotasoftlimit 0
log_gen 0
fs_metadevid 0 fs_metablkno 0 fs_metatype 0
fs_bsoffset 0 fs_bsdevid 0 fs_bssize 0

The issue can occur after filesystem disk layout version was upgraded using vxupgrade and metadata of multiple filesystem disk layout versions coexist in the same filesystem. 

# /opt/VRTS/bin/fsck -o full /dev/vx/rdsk/tdg/volupgrade
UX:vxfs fsck: ERROR: V-3-20012: not a valid vxfs file system
invalid super-block
search for auxiliary super-block? (ynq)y
found auxiliary super-block at offset 2362368
creation time Thu Jan 29 13:34:29 2015
version 7 blocks 0 bsize 1024                       <<< found version 7 superblock because it is in front of the version 8 one
rebuild super-block from auxiliary? (ynq)y         <<< rebuild the metadata using the incorrect version 7 superblock
UX:vxfs fsck: ERROR: V-3-20451: No valid device inodes found    <<< failed
UX:vxfs fsck: ERROR: V-3-20694: cannot initialize aggregate
file system check failure, aborting ...

The filesystem is corrupted by the above full fsck run to a state where fsck will no longer be able to handle and fix.

Before making any changes to the filesystem using full fsck (that is, before answering "yes" to any questions prompted by full fsck), run "fsck -F vxfs -n" (Solaris version) or "fsck -t vxfs -n" (Linux version) on the filesystem to check level of corruption.    For other platforms, please refer to the corresponding operation system manual page for the fsck command.  

Check the superblock info on the volume:

# fstyp -v  /dev/vx/rdsk/dgname/volname
 

or

# echo '0x2000 B.p S' | /opt/VRTS/bin/fsdb /dev/vx/rdsk/dgname/volname

Then collect dd of Ist 500MB/1GB of the volume.

# dd if=/dev/vx/rdsk/dgname/volname bs=1024k count=1024 | gzip -c > /var/tmp/volname.dd.gzip (for 500MB change count to 500)

Read the superblock address locations from the volume, we can see there are various superblocks with different layout versions.   The following command will display the byte address (first column) in decimal format.

# dd if=/dev/vx/rdsk/tdg/volupgrade bs=1024k count=1024 | od -Ad -X | grep 'a501fcf5'

0008192 a501fcf5 00000007 00000400 00000004 <<< primary superblock at 8K Bytes location

2362368 a501fcf5 00000007 00000400 00000004 <<< byte offset 2362368 has the version 7 superblock

2365440 a501fcf5 00000008 00000400 00000004 <<< byte offset 2365440 has the version 8 superblock

Now run full fsck on the volume (please don't run with "-y" option; otherwise, we will have no control on what fsck will do).

# /opt/VRTS/bin/fsck -o full /dev/vx/rdsk/dgname/volname
UX:vxfs fsck: ERROR: V-3-20012: not a valid vxfs file system
invalid super-block
search for auxiliary super-block? (ynq)y

found auxiliary super-block at offset 2362368    <<< found superblock at byte offset 2362368
creation time Thu Jan 29 13:34:29 2015
version 7 blocks 0 bsize 1024       <<< please check if the version is the correct; in the example version 7 is not the correct one
rebuild super-block from auxiliary? (ynq) n <<< answer "n" to skip it if the found superblock is not the correct one.  The correct one should be at location 2365440.
continue searching? (ynq)y

found auxiliary super-block at offset 2365440
creation time Tue Jan 30 15:00:48 2015
version 8 blocks 0 bsize 1024                       <<< found the correct version 8 superblock at location 2365440
rebuild super-block from auxiliary? (ynq)y      <<< Now can rebuild the super-block from the correct auxiliary superblock and continue the rest of fsck

If there are any concern in checking or running full fsck on the filesystem, please check with Veritas technical support for assistance.

Applies To

Veritas Filesystem on all platforms.