Error Message

VxFS performs slower than other filesystem types when running dd as a benchmark program.

By default VxFS is tuned to handle commercial application I/O load and pattern.   dd is simplistic program which performs sequential read or write operations which are normally not what typical commercial applications will do.  

With default filesystem mount options, those read/write data generated by dd are buffered in the kernel memory buffer.  This is called bufferred I/O.   During read operation, the data is read from the physical disk only for the first time.   As long as there are sufficient free memory, the kernel will not purge data from the memory.   If that's the case, subsequent read of the same file will actually reading the data from memory and not from the disks.    Please note that when you compare the performance of a series of dd read tests, please make sure that you are not compare the disk read speed with the memory read speed.  That can be a difference of orders of magnitude.    Similar thing is true for dd write operations.   If there are sufficient free memory, the written data will be buffered only in the memory and not written to the disks.    As a result, the first time dd write is performed, the performance is very good because there is actually no disk I/O involved.   If dd is writing to the same file again, the performance will also be very good too because the data is again buffered in memory only.    The time when actual physical I/Os are required is when the free memory drops to a certain level and requires the kernel to flush the buffered data to disks.   For example, when running dd write to a different file, it will consume additional kernel memory and dd will have to wait for the pre-existing buffered to be written on to the disk before memory is freed up to buffer the new written data.

Normally dd runs slower on VxFS filesystem because of the following default tunings of VxFS.  Those default VxFS tunings are catered for commercial application I/O load and pattern and not for a simplistic dd program.

1. The detection of the Direct I/O by VxFS (discovered_direct_iosz)

Discovered Direct I/O is where VxFS will read/write the data from/to the disks without storing the data in the kernel memory buffer.  discovered_direct_iosz is a parameter of a VxFS file system that can be tuned by the vxtunefs program to control the I/O size by which VxFS will start to perform Direct I/O.  The default value is 256KB.   If an I/O is bigger than the discovered_direct_iosz setting, VxFS will not buffer the data in the kernel memory and write the data direct to the disks instead.    This mechanism will smooth the disk utilization and prevent the large I/Os from expelling other buffered data from the kernel memory.   But when comparing the VXFS performance against other file systems that do not have similar mechanism, the comparison becomes unfair.  It is because with Direct I/O the VxFS result is a disk-based performance result while the results from other file system without Direct I/O is a memory-based performance result.   The memory-based performance result will be much better than disk-based performance.   In order to have a comparable result, the Discovered Direct I/O can be turned off by setting the discovered_direct_iosz to a value larger than the dd I/O block size.

2. Preferred I/O Block Size (read_pref_io and write_pref_io)

By default VxFS performs I/O in small block size of 64KB.    When handling the sequential I/O generated by dd, this small block size is not efficient.   Some other filesystem uses large block size, for example, Linux ext4 uses 1MB block size.    Please note that typical commercial applications will perform I/O is small I/O block size, for example, updating a record or accessing an email.    Performing I/O using large I/O block size can waste the I/O bandwidth because not all the accessed data is actually required by the application.

3. Write throttling by limiting the disk I/O queue size (max_diskq)

By default VxFS limits the number of dirty page in the disk queue to 1MB.    VxFS prevents the write operation to one particular file from monopolizing the I/O channel and narrow the bandwidth available to the other applications on the same system.   On a quiet system this may prevent dd from utilizing the whole bandwidth of the storage hardware.

Please note that the VxFS tuning suggestions for the dd program may not be applicable to the actual I/O pattern of your applications.   The following suggestions are specific to the dd program only.  The suggestions are not meant to be exhaustive.  If the dd program is run with different settings, further VxFS tuning may be required.

1. Increase the discovered_direct_iosz to force VxFS to buffer the data in memory.  The following command increases the discovered direct I/O size to 2MB.   Any I/O with block size less than or equal to 2MB will be written to memory instead of disk.

# vxtunefs -o discovered_direct_iosz=2097152  /vxfs_mount_point

2. Increase the I/O block size used by VxFS.  The following command will cause VxFS to write in 1MB block size.

# vxtunefs -o read_pref_io=1048576 /vxfs_mount_point
# vxtunefs -o write_pref_io=1048576 /vxfs_mount_point

3. Incease the max_diskq to 4MB.  The following will allow VxFS to allocate more I/O bandwidth to one particulare file.

# vxtunefs -o max_diskq=4194304 /vxfs_mount_point
 

Applies To

VxFS on supported Unix platforms.