NFSD High CPU utilization when VxFS File System is mounted with "mincache=dsync" and exported over NFS.
Article ID: 100003362
Updated On:
Description
Error Message
Name PID CPU% PgSp Owner % Used 1.1
nfsd 548940 98.7 6.1 root NFS (calls/sec) % Free 99.9
1GB untar of a file took 39 minutes when the tar was ran from the NFS Client mount.
# time tar xvf 5.0MP3.tar
real 39m1.579s
user 0m0.082s
sys 0m1.050s
Cause
-->
From the trace data we see the following sequence
1. Incoming NFS read call
2. vnop_open() of the file to see if we can use read-ahead feature (this is currently supported by GPFS but not on J2 or VxFS )
3. vnop_close() --> since the read-ahead feature cannot be used. This is where the performance issue is seen.
4. vnop_rdwr() --> issue the actual read operation to the underlying filesystem
Starting AIX 5.3 a performance enhancement was made for VNOP_OPEN read-ahead optimization. The only problem is that it is only for PFS's that key on the vinfop parameter used in VNOP_OPEN for read-ahead optimization. The function AIX uses to get this information is rfs_get_vinfop(). Note AIX does not call it for J2 since J2 does not do that kind of optimization. It is called for everything else including AIX GPFS which does use it. Symantec VxFS does not return any vinfop information. That is the reason the VNOP_CLOSE comes before the read since there is no reason to keep the file open since VxFS is not using the vinfop for anything.
Resolution
IBM has created iFix IZ86583. The iFix provides modified NFS extensions that avoid vnop_open and vnop_close calls for VxFS mounts exported over NFS. To avoid the vnop_open and vnop_close calls for VxFS mounts using "mincache=dsync" exported over NFS, customer are encouraged to install modified NFS extensions. Once the fix is applied VxFS File Systems mounted with "mincache=dsync" will no longer suffer from performance issues and NFS server NFSD daemon will return to normal CPU utilization.
Applies To
AIX 5.3, 6.1
Storage Foundation 5.0 MP3, 5.0MP3RP1, 5.0MP3RP2, 5.0MP3RP3, 5.0MP3RP4
VxFS is mounted with dsync option and then exported over NFS.
example,
mount -V vxfs -o mincache=dsync /dev/vx/dsk/testdg/testvol /vol01
exportfs -a
#On client machine mount exported NFS file system.
mount -V nfs 10.168.21.38:/vol01 /mnt1
Issue/Introduction
