The 'delayed allocation' (dalloc) feature of Veritas File System (VxFS) can cause silent data corruption in workloads using fsync()
Article ID: 100031907
Updated On:
Description
Error Message
There is no error message. The data corruption is silent and can only be detected by verifying the data.Cause
The issue occurs when there is a race between an allocating write using the dalloc feature and an fsync() call is issued to flush the file.
Resolution
Veritas Technologies is working on a fix for this issue. Please contact support for private HotFixes.
Public patches can be downloaded from the SORT website
https://sort.veritas.com/search?q=3866962
Workaround
A workaround also exists by disabling the 'delayed allocation' (dalloc) feature on the VxFS file systems, details are below. Disabling the dalloc feature may impact performance of asynchronous buffered writes that extend a file's size, as the extent allocations will be done in the same context of the write I/O (Pre VxFS 6.0 behavior) rather than delaying it until dirty pages are flushed to disk.
# vxtunefs -s -o dalloc_enable=0 $MOUNT_POINT
For example:
2. To make the value persistent across system reboot, add an entry to the /etc/vx/tunefstab file. This file contains tuning parameters for Veritas File Systems, which are set automatically during file system mount.
- Per file system
/dev/vx/dsk/$DISKGROUP/$VOLUME dalloc_enable=0
For example:
- System default:
For detailed explanation of the above workaround please see below:
- For a mounted file system the vxtunefs command can be used to set tuning parameters, however values that are set using the vxtunefs command are not persistent across system reboot or across file system remount (umount and mount back).
- To make the setting persistent update the /etc/vx/tunefstab file. This file contains tuning parameters for Veritas File Systems which are set automatically during filesystem mount.
Each entry in tunefstab is a line of fields in one of the following formats:
block_device tunefs options
system_default tunefs options
- block_device is the name of the device on which the file system is mounted. If there is more than one line that specifies options for a device, each line is processed and the options are set in order.
- In place of block_device, system_default specifies tunables for each device to process. If an entry for both a block_device and system_default exists then the device value takes precedence.
1. Steps for making the VxFS File system tuning values persistent per file system:
i) For the filesystem already mounted
# vxtunefs -s -o dalloc_enable=0 /$MOUNT_POINT
ii) Create the /etc/vx/tunefstab file if it does not exists and add the entry as shown in below example:
For example:
iii) To confirm the value after mount
# vxtunefs /testmnt1 | grep "dalloc_enable"
dalloc_enable = 0
Example using a single file system :
- To make it persistent add entry to /etc/vx/tunefstab
Example using multiple file systems :
2. Steps for making the VxFS File system tuning values persistent for all file system - System wide default setting
This is helpful if you need to apply the tuning system wide. If the system_default is specified instead of block_device, the tunable setting will be applied all the block devices when mounting the file systems.
i) Create the /etc/vx/tunefstab if it is not there already.
ii) Add the entry to the /etc/vx/tunefstab file as shown in below example :
system_default dalloc_enable=0
iii) This can be confirmed, after mounting the file system on the device, using the “vxtunefs –p” command.
# vxtunefs /testmnt1/ | grep "dalloc_enable"
dalloc_enable = 0
Example:
Note: The system_default setting is overridden if a block_device is also explicitly specified.
For example, if we want to change the system-wide default for dalloc_enable to 0 but do not want to set it for /dev/vx/dsk/testdg/testvol3, then we can achieve this as follows:
Public patches can be downloaded from the SORT website
https://sort.veritas.com/search?q=3866962
| Platform | VxFS Version | Public Patch | Private HotFix |
|---|---|---|---|
| Solaris 10 | 6.0.5.100 | 6.0.5.400 | - |
| 6.1 | 6.1.1.400 (in development) | - | |
| 6.2 | 6.2.1.300 (in development) | - | |
| 7.0 | 7.0.1.100 (in development) | - | |
| Solaris 11 | 6.0.5.100 | 6.0.5.400 | - |
| 6.1 | 6.1.1.400 (in development) | - | |
| 6.2 | 6.2.1.300 (in development) | - | |
| 7.0 | 7.0.1.100 (in development) | - | |
| Linux | 6.0.5.100 6.0.5.200 (RHEL6) |
6.0.5.400 | - |
| 6.1 | 6.1.1.400 (in development) | - | |
| 6.2 | 6.2.1.300 (in development) | 6.2.1.104 | |
| 7.0 | 7.0.1.100 (in development) | - |
Workaround
A workaround also exists by disabling the 'delayed allocation' (dalloc) feature on the VxFS file systems, details are below. Disabling the dalloc feature may impact performance of asynchronous buffered writes that extend a file's size, as the extent allocations will be done in the same context of the write I/O (Pre VxFS 6.0 behavior) rather than delaying it until dirty pages are flushed to disk.
Note:
- If a VxFS file system is resized or re-mounted (mount -o remount) after dalloc is disabled, the resize/remount operation may re-enable dalloc. The dalloc feature will have to be manually disabled again with vxtunefs.
- Care must be taken in a Oracle Cluster Ready Services (CRS) environment. When using CRS, as a change using vxtunefs to the CRS volume could cause a delay in heartbeat, please first disable CRS monitoring while make a vxtunefs change to /crs.
1. For a filesystem which is already mounted
# vxtunefs -s -o dalloc_enable=0 $MOUNT_POINT
For example:
# vxtunefs -s -o dalloc_enable=0 /testmnt1
2. To make the value persistent across system reboot, add an entry to the /etc/vx/tunefstab file. This file contains tuning parameters for Veritas File Systems, which are set automatically during file system mount.
The entry in tunefstab can be per file system or a system default.
- Per file system
/dev/vx/dsk/$DISKGROUP/$VOLUME dalloc_enable=0
For example:
# cat /etc/vx/tunefstab
/dev/vx/dsk/testdg/testvol1 dalloc_enable=0
/dev/vx/dsk/testdg/testvol1 dalloc_enable=0
- System default:
# cat /etc/vx/tunefstab
system_default dalloc_enable=0
system_default dalloc_enable=0
For detailed explanation of the above workaround please see below:
- For a mounted file system the vxtunefs command can be used to set tuning parameters, however values that are set using the vxtunefs command are not persistent across system reboot or across file system remount (umount and mount back).
- To make the setting persistent update the /etc/vx/tunefstab file. This file contains tuning parameters for Veritas File Systems which are set automatically during filesystem mount.
Each entry in tunefstab is a line of fields in one of the following formats:
block_device tunefs options
system_default tunefs options
- block_device is the name of the device on which the file system is mounted. If there is more than one line that specifies options for a device, each line is processed and the options are set in order.
- In place of block_device, system_default specifies tunables for each device to process. If an entry for both a block_device and system_default exists then the device value takes precedence.
1. Steps for making the VxFS File system tuning values persistent per file system:
i) For the filesystem already mounted
# vxtunefs -s -o dalloc_enable=0 /$MOUNT_POINT
For example:
# vxtunefs -s -o dalloc_enable=0 /testmnt1
ii) Create the /etc/vx/tunefstab file if it does not exists and add the entry as shown in below example:
/dev/vx/dsk/$DISKGROUP/$VOLUME dalloc_enable=0
For example:
/dev/vx/dsk/testdg/testvol1 dalloc_enable=0
iii) To confirm the value after mount
# vxtunefs /testmnt1 | grep "dalloc_enable"
dalloc_enable = 0
Example using a single file system :
- For mounted file system
# vxtunefs /testmnt1 |grep "dalloc_enable"
dalloc_enable = 1
dalloc_enable = 1
# vxtunefs -s -o dalloc_enable=0 /testmnt1
UX:vxfs vxtunefs: INFO: V-3-22525: Parameters successfully set for /testmnt1
UX:vxfs vxtunefs: INFO: V-3-22525: Parameters successfully set for /testmnt1
# vxtunefs /testmnt1 |grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
- To make it persistent add entry to /etc/vx/tunefstab
# cat /etc/vx/tunefstab
/dev/vx/dsk/testdg/testvol1 dalloc_enable=0
/dev/vx/dsk/testdg/testvol1 dalloc_enable=0
# mount -F vxfs /dev/vx/dsk/testdg/testvol1 /testmnt1/
# vxtunefs /testmnt1/ | grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
Example using multiple file systems :
# mount -F vxfs /dev/vx/dsk/testdg/testvol1 /testmnt1/
# vxtunefs /testmnt1/ | grep "dalloc_enable"
dalloc_enable = 1
dalloc_enable = 1
# mount -F vxfs /dev/vx/dsk/testdg/testvol2 /testmnt2
# vxtunefs /testmnt2/ | grep "dalloc_enable"
dalloc_enable = 1
dalloc_enable = 1
# cat /etc/vx/tunefstab
/dev/vx/dsk/testdg/testvol1 dalloc_enable=0
/dev/vx/dsk/testdg/testvol2 dalloc_enable=0
/dev/vx/dsk/testdg/testvol1 dalloc_enable=0
/dev/vx/dsk/testdg/testvol2 dalloc_enable=0
# umount /testmnt1/
# umount /testmnt2/
# umount /testmnt2/
# mount -F vxfs /dev/vx/dsk/testdg/testvol1 /testmnt1/
# mount -F vxfs /dev/vx/dsk/testdg/testvol2 /testmnt2/
# mount -F vxfs /dev/vx/dsk/testdg/testvol2 /testmnt2/
# vxtunefs /testmnt1/ | grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
# vxtunefs /testmnt2/ | grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
2. Steps for making the VxFS File system tuning values persistent for all file system - System wide default setting
This is helpful if you need to apply the tuning system wide. If the system_default is specified instead of block_device, the tunable setting will be applied all the block devices when mounting the file systems.
i) Create the /etc/vx/tunefstab if it is not there already.
# touch /etc/vx/tunefstab
# ls -l /etc/vx/tunefstab
-rw-r--r-- 1 root system 75 Aug 21 17:11 /etc/vx/tunefstab
# ls -l /etc/vx/tunefstab
-rw-r--r-- 1 root system 75 Aug 21 17:11 /etc/vx/tunefstab
ii) Add the entry to the /etc/vx/tunefstab file as shown in below example :
system_default dalloc_enable=0
iii) This can be confirmed, after mounting the file system on the device, using the “vxtunefs –p
# vxtunefs /testmnt1/ | grep "dalloc_enable"
dalloc_enable = 0
Example:
# cat /etc/vx/tunefstab
system_default dalloc_enable=0
system_default dalloc_enable=0
# mount -F vxfs /dev/vx/dsk/testdg/testvol1 /testmnt1/
# mount -F vxfs /dev/vx/dsk/testdg/testvol2 /testmnt2/
# mount -F vxfs /dev/vx/dsk/testdg/testvol2 /testmnt2/
# vxtunefs /testmnt1/ | grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
# vxtunefs /testmnt2/ | grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
Note: The system_default setting is overridden if a block_device is also explicitly specified.
For example, if we want to change the system-wide default for dalloc_enable to 0 but do not want to set it for /dev/vx/dsk/testdg/testvol3, then we can achieve this as follows:
# cat /etc/vx/tunefstab
system_default dalloc_enable=0
/dev/vx/dsk/testdg/testvol3 dalloc_enable=1
system_default dalloc_enable=0
/dev/vx/dsk/testdg/testvol3 dalloc_enable=1
# mount -F vxfs /dev/vx/dsk/testdg/testvol1 /testmnt1/
# mount -F vxfs /dev/vx/dsk/testdg/testvol2 /testmnt2/
# mount -F vxfs /dev/vx/dsk/testdg/testvol3 /testmnt3/
# mount -F vxfs /dev/vx/dsk/testdg/testvol2 /testmnt2/
# mount -F vxfs /dev/vx/dsk/testdg/testvol3 /testmnt3/
# vxtunefs /testmnt1/ | grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
# vxtunefs /testmnt2/ | grep "dalloc_enable"
dalloc_enable = 0
dalloc_enable = 0
# vxtunefs /testmnt3/ | grep "dalloc_enable"
dalloc_enable = 1
dalloc_enable = 1
Issue/Introduction
VxFS 6.x includes a new feature called 'delayed allocation' also known as 'dalloc'. This feature is enabled by default for local mounted file systems and is not supported for cluster mounted file systems.
Background Information:
Asynchronous sequential writes that extend a file’s size will create and dirty memory pages, new extents can therefore be allocated when the dirty pages are flushed to disk (via background processing) rather than allocating the extents in the same context as the write I/O. The 'dalloc' feature therefore delays the extent allocation until the dirty pages are flushed to disk. The 'dalloc' feature allows VxFS to write to a file even when the allocation has not happened.
Veritas Technologies have found an issue where silent corruption may occur when dalloc feature is used in workloads involving fsync(). The issue occurs when there is a race between an allocating write using the dalloc feature and an fsync() call is issued to flush the file.
Affected versions: 6.x and 7.x
Affected platforms: Solaris and Linux
Affected configuration: Locally mounted (not CFS) filesystem with dalloc enabled.
Affected workloads: Extending writes and fsync() call (typically used in NFS environments)
Background Information:
Asynchronous sequential writes that extend a file’s size will create and dirty memory pages, new extents can therefore be allocated when the dirty pages are flushed to disk (via background processing) rather than allocating the extents in the same context as the write I/O. The 'dalloc' feature therefore delays the extent allocation until the dirty pages are flushed to disk. The 'dalloc' feature allows VxFS to write to a file even when the allocation has not happened.
Veritas Technologies have found an issue where silent corruption may occur when dalloc feature is used in workloads involving fsync(). The issue occurs when there is a race between an allocating write using the dalloc feature and an fsync() call is issued to flush the file.
Affected versions: 6.x and 7.x
Affected platforms: Solaris and Linux
Affected configuration: Locally mounted (not CFS) filesystem with dalloc enabled.
Affected workloads: Extending writes and fsync() call (typically used in NFS environments)
Was this article helpful?
Yes
No
Powered by
