Description

When a directory is corrupted, those files in the corrupted directory will lose their names and will be moved (reconnected) to lost+found by fsck.   For example,

# /opt/VRTS/bin/fsck -n /dev/vx/rdsk/adg/volmanyfiles > fsck_n.out

fsck_n.out:

pass0 - checking structural files
pass1 - checking inode sanity and blocks
pass2 - checking directory linkage
fileset 999 directory 573185 block devid/blknum 0/154480 bad hash count 30840 clear block? (ynq)n
fileset 999 primary-ilist inode 573185 contains invalid directory blocks
                 clear? (ynq)n
fileset 999 primary-ilist inode 573185 has invalid alternate directory index
        (fileset 999 attribute-ilist inode 2), clear index? (ynq)n
pass3 - checking reference counts
fileset 999 primary-ilist inode 583716 unreferenced file, reconnect? (ynq)n
fileset 999 primary-ilist inode 583716 clear? (ynq)n
fileset 999 primary-ilist inode 583717 unreferenced file, reconnect? (ynq)n
fileset 999 primary-ilist inode 583717 clear? (ynq)n
....
fileset 999 primary-ilist inode 593200 clear? (ynq)n
pass4 - checking resource maps
OK to clear log? (ynq)n
sanity checks/updates have not been completed - restart? (ynq)n

According to fsck -n output, there will be 9472 files moved to lost+found. 

# grep reconnect fsck_n.out | wc -l
9472

But this number may not be accurate, the large amount of files reported as losing their names is because we answer "n" to the question in fsck pass2.  In our example, actually there is only one directory block is corrupted, the rest of the directory is still okay.  But since we answer "n" to the first question, fsck assumed the whole directory is bad, not only one single directory block.

pass2 - checking directory linkage
fileset 999 directory 573185 block devid/blknum 0/154480 bad hash count 30840 clear block? (ynq)n    <<< answer "n" here
fileset 999 primary-ilist inode 573185 contains invalid directory blocks
                 clear? (ynq)n           <<< fsck clears the whole directory, not only one block

The number of blocks used by the directory can be checked with fsdb.

rhel73vm30# echo '999fset.573185i' | /opt/VRTS/bin/fsdb /dev/vx/rdsk/adg/volmanyfiles
inode structure at 0x000258b0.0100
type IFDIR mode 40755  nlink 2  uid 0  gid 0  size 442368
atime 1539236856 626711  (Thu Oct 11 16:47:36 2018 AEDT)
mtime 1539236965 928707  (Thu Oct 11 16:49:25 2018 AEDT)
ctime 1539236965 928707  (Thu Oct 11 16:49:25 2018 AEDT)
aflags 0 orgtype 3 eopflags 0 eopdata 0
pdmode 0  rdev/reserve/dotdot/matchino 2 camode 0x00
blocks 440  gen 1649519596  version 0 31041   iattrino 0     <<< there are about 440 file system blocks; only one of these 440 blocks is corrupted
dotdot 2  inattrino 0
ext0: INDIR   boff: 0x00000000 bno:     153448 len:        8
ext1: NULL   boff: 0x00000000 bno:          0 len:        0
ext2: NULL   boff: 0x00000000 bno:          0 len:        0
ext3: NULL   boff: 0x00000000 bno:          0 len:        0
ext4: NULL   boff: 0x00000000 bno:          0 len:        0
ext5: NULL   boff: 0x00000000 bno:          0 len:        0

The actual number of files losing their names can only be determined by running full fsk and answer "y" to that question in pass2.   We can run full fsck on a restored metasave to get the accurate numbers.

We may be able to find the names of some of the affected files from the ncheck output.  Please collect ncheck output BEFORE running the full fsck.

# /opt/VRTS/bin/ncheck /dev/vx/rdsk/*/volmanyfiles > ncheck_volmanyfiles.out

Now we can use the "join" command to search the file names of the unreferenced files reported in fsck.   We need to first sort the two output files.

# sort -k 1,1 ncheck_volmanyfiles.out > ncheck_volmanyfiles.out.sorted

# grep reconnect fsck_n.out  | awk '{print $5}' | sort > unreferenced.list

9472 files are reported as losing their names, but this may not be accurate as explained previously.

# wc -l unreferenced.list
9472 unreferenced.list

Then use the following "join" syntax to find the file names.

# join -j1 1 -j2 1 unreferenced.list ncheck_volmanyfiles.out.sorted > unreferenced_file_names

rhel73vm30# more unreferenced_file_names
583763 /bigdir/file010575
583764 /bigdir/file010576
583765 /bigdir/file010577
583766 /bigdir/file010578
.....

Use the following "join" syntax to find the inodes that don't have their names in the ncheck output.

# join -j1 1 -j2 1 -v 1 unreference.list ncheck_volmanyfiles.out.sorted  > unreferenced_file_noname

# more unreferenced_file_noname
583716
583717
583718
...

There will 47 files which don't have names after full fsck.   And this 47 files should be in the corrupted directory block in our example.

# wc -l unreferenced_file_noname
47 unreferenced_file_noname

We can also check the intent log for any recent directory operations and the file names may appear in those operations.   For example,

# echo 'rfmtlog' | /opt/VRTS/bin/fsdb /dev/vx/rdsk/adg/volmanyfiles > rfmtlog.out
# echo 'fmtlog' | /opt/VRTS/bin/fsdb /dev/vx/rdsk/adg/volmanyfiles > fmtlog.out

Please note the rfmtlog output may contain stale intent log entries belongs to previous filesystem mount.  If multiple file names are found for an unreferenced inode, we should always trust the latest record from the fmtlog output.

For example,

# egrep --no-group-separator -A 2 'added  d_ino 593194' fmtlog.out
added  d_ino 593194  d_reclen 596  d_namlen 10  d_hashnext 00f8
               f  i  l  e  0  1  9  9  9  3