Error Message

The following error is reported in message file, like any normal vxfs file system full:

vxfs: [ID 702911 kern.notice] NOTICE: msgcnt 519 mesg 001: V-2-1: vx_nospace - file system full

The used space reported by df -k and du -sk will differ.

 

This can happen if the file system is NFS shared, and some files have been removed in the NFS server while they were still being used by NFS clients. This can be reproduced easily in a Lab environment by doing the following:

1. Start with a vxfs or ufs nfs shared filesystem in the NFS server:

bash-3.00# df -k /vol1
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/evac1/vol1
2097152 18136 1949213 1% /vol1

bash-3.00# dfshares
RESOURCE SERVER ACCESS TRANSPORT
sun06:/vol1 sun06 - -

2. Create a 300MB file:

bash-3.00# cd /vol1/tmp

bash-3.00# dd if=/dev/zero of=./testfile count=300 bs=1024k
300+0 records in
300+0 records out

bash-3.00# df -k /vol1
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/evac1/vol1
2097152 325344 1661205 17% /vol1

3. Mount it from a NFS client:

bash-3.00# mount sun06:/vol1 /mnt

bash-3.00# df -k /mnt
Filesystem kbytes used avail capacity Mounted on
sun06:/vol1 2097152 325344 1661205 17% /mnt

4. Run a process that keeps the file opened in the NFS client:

bash-3.00# sleep 9999 < testfile &
[1] 6907

bash-3.00# pfiles 6907
6907: sleep 9999
Current rlimit: 256 file descriptors
0: S_IFREG mode:0755 dev:349,151340 ino:48 uid:0 gid:0 size:314572800
O_RDONLY|O_LARGEFILE
/mnt/tmp/testfile
1: S_IFCHR mode:0620 dev:335,0 ino:12582918 uid:0 gid:7 rdev:24,1
O_RDWR|O_NOCTTY|O_LARGEFILE
/devices/pseudo/pts@0:1
2: S_IFCHR mode:0620 dev:335,0 ino:12582918 uid:0 gid:7 rdev:24,1
O_RDWR|O_NOCTTY|O_LARGEFILE
/devices/pseudo/pts@0:1

5. Remove the file in the NFS server. Despite the file is removed, the free space is not updated:

bash-3.00# rm testfile

bash-3.00# df -k /vol1
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/evac1/vol1
2097152 325344 1661205 17% /vol1

6. Even if we kill the process in the NFS client, the free space is not returned:

bash-3.00# jobs -l
[1]+ 6907 Running sleep 9999
bash-3.00# kill -9 6907
bash-3.00#
[1]+ Killed sleep 9999
bash-3.00# df -k /vol1
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/evac1/vol1
2097152 325344 1661205 17% /vol1


7. We need to unshare the vxfs filesystem in the server to have the free space back. It could take a moment until the free space number is updated.

bash-3.00# unshare /vol1

bash-3.00# df -k /vol1
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/evac1/vol1
2097152 325344 1661205 17% /vol1

bash-3.00# df -k /vol1
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/evac1/vol1
2097152 325344 1661205 17% /vol1

bash-3.00# df -k /vol1
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/evac1/vol1
2097152 18136 1949213 1% /vol1

 

The solution is to unshare the NFS shared resource from the NFS server, and share it back again, so the NFS server will refresh the file system information. You don't need to restart the NFS server.

Note that the problem will not go away, if we simply kill the process holding the file descriptor to the removed file at NFS client side.

If the NFS shared file system is defined as NFS Resource in a VCS cluster, we should offline and online the NFS Shared resource.

 

Applies To

At the moment, it has only been observed in Solaris 10 NFS server/client environments, for both Veritas File system and Solaris UFS.