Kernel virtual memory consumption by VERITAS File System (VxFS) is mainly in two areas, VxFS buffer cache and in core inodes, which are constrained by vx_bc_bufhwm and vxfs_ninode, respectively.  For large memory machines ( > 1GB of ram),VxFS currently tunes them automatically to likely numbers larger than needed.This will take up some portion of memory not needed by VxFS and better utilized elsewhere.

Therefore, if

 

1. file systems don't have many metadata changes (lots of file creation and deletion)

2. system has RAM greater than 1GB, and

3. you are experiencing kernel memory shortage or even kernel map fragmentation

Make vxfs_ninode 25% higher than ncsize and lowervx_bc_bufhwm by 20%.  Put the new parameters in /etc/system and reboot the system. This should resolve the memory shortage problems, which are believed to be caused by VxFS.
On the system side, the problem is not having a big enough dump device,  and it can be resolved by limiting the size of the dump device or increasing the size of the dump device.


KERNELTUNING:

Internal Inode Table Size
Inodes are cached in a"per file system table" known as an inode table. The following tunables areused:
 

· vx_bc_bufhwm

vxfs_ninode
The VxFS file system type uses the value of vxfs_ninode to set the number of entries in the inode table for each mounted file system, computed based on system memory size, located in /etc/system.

To set the size:
setvxfs:vxfs_ninode= new_value , where new_value is the number ofinodes

ncsize
ncsize determines the size of the directory name lookup cache (dnlc); it must be between 50-80% of vxfs_ninode to avoid excessive cpu consumption.
Located in: /etc/system file.  To set this value:
set ncsize = new_value , where new_value is the number of entries.  Every entry in the disk has to point to an entry in the inode table.

vx_bc_bufhwm
vx_bc_bufhwm or buffer cache high water mark sets the maximum amount of space for meta data buffers on a system.This value depends on:
 

· Set to not exceed 66% of the virtual memory

The value is ultimately set to the minimum of the values of the physical memory and virtual memory.

To determine VxFS Kmemusage.....

       (vx_bc_bufhwm*1k) = Kmem used

       (vxfs_ninode*1k)  = Kmem used

In this case total was ~163M.

The sun4d architecture only supports up to 256M of kernel memory, which in this case would only leave 93M of kernel memory for other system kernel memory requests.  

One approach to rectify this situation would be to tune the VxFS variables such that it does not consume so much kernel memory.

The variables to tune would be vxfs_ninode and vx_bc_bufhwm. The amounts that these should be tuned are going to be highly dependent on your estimation of what constitutes an acceptable amount of kmem for VxFS to be using.  

If the vxfs_ninode is reduced, the possible performance ramifications are that threads may have to block waiting for an in-core inode to become available.  This would be associated with how many files are open in VxFS file systems at any given time.

If the vx_bc_bufhwm is reduced,then buffered I/O would have a reduced number of buffers with which to operate,and some I/O threads may have to block waiting on a buffer to become available.

Also make sure vxfs_ninode is at least 25% larger than ncsize. If vxfs_ninode is smaller than ncsize, reduce the ncsize tunable instead of tuning up vxfs_ninode (each entry in the in-core inode table takes up 1024 bytes of kernel memory).


Based on the current kernel memory consumption and the default VxFS parameters set,  the following values should be reduced by 20% as an initial attempt to prohibit the system from hanging due to high VxFS kernel memory usage:

System defaults(currently tuned):

vx_bc_bufhwm:  36864
vxfs_ninode:    87490
ncsize:        55992

Suggested tuning (additions to/etc/system):

set vxfs:vx_bc_bufhwm=29488
setvxfs:vxfs_ninode=69992
setncsize=55992
 
****************************
How to find the default values and how to set new values

1. Get :
# echo"vx_bc_bufhwm/D" | adb -k /dev/ksyms /dev/mem
and
# echo"ncsize/D" | adb -k /dev/ksyms /dev/mem
on every system.

2. Add these values to get the total that VxFS is using
vx_bc_bufhwm * 1k = Kmemused
vxfs_ninode  * 1k = Kmem used

3. Compare this with kernel memory size that is supported on a particular platform. If it is too much, lower vx_bc_bufhwm and vxfs_ninode. Remember that vxfs_ninode should be 25% higher than ncsize.

This is a generic tuning advise and the results may differ because the systems have different applications running on them and their usage is different. vmstat, sar, iostat are Solaris tools used to monitor the system performance to tune the systems better to the applications that you are running.

Here is an example:
1. joke[root]# echo "ncsize/D" | adb -k /dev/ksyms/dev/mem
physmem 7d25
ncsize:
ncsize:        8860

2. vxfs_ninode=ncsize x 1.25= 11075
joke[root]# echo"vxfs_ninode/D" | adb -k /dev/ksyms /dev/mem
physmem7d25
vxfs_ninode:
vxfs_ninode:    12000
This looksOK

3. joke[root]# echo "vx_bc_bufhwm/D" | adb -k /dev/ksyms/dev/mem
physmem 7d25
vx_bc_bufhwm:
vx_bc_bufhwm:   24000
Youcan reduce this by 20% if needed.
24000 X .8=19200

4. Put all these new values in /etc/system:
set vxfs:vx_bc_bufhwm=19200
setvxfs:vxfs_ninode=12000
set ncsize=8860

5. Reboot