df, fuser and other commands stop responding, or "hang," when used with a Veritas File System (VxFS)
Article ID: 100031569
Updated On:
Description
Error Message
The hanging processes will not show any error messages. The thread list that was collected, either live, or via a system dump, will show threads with the stack below:Write advise thread
vx_rwsleep_lock
vx_iplock
vx_write_advise <<< write advise
vx_write_common
vx_rdwr_attr
vnop_rdwr
vno_rw
rwuio
rdwr
kewrite
syscall
dalloc thread
vx_ilock
vx_idalloc_flush
vx_idalloc_off
vx_dalloc_flush <<< dalloc
vx_workitem_process
vx_worklist_process
vx_worklist_thread
or
mutex_lock
vx_ilock
vx_dalloc_do_flush_file
vx_dalloc_flush <<< dalloc
vx_workitem_process
vx_worklist_process
vx_worklist_thread
vx_kthread_init
kernel_thread_helper
Cause
This occurs while two threads contest for two locks: ILOCK and PLOCK.
The "write advise" thread, with the above stack, owns the ILOCK, but is waiting for the PLOCK. The dalloc thread owns the PLOCK and is waiting for the ILOCK with above stack.
This deadlock is caused by an incorrect locking order between the "write advise" thread and "dalloc flusher" thread. This causes file system commands, like df, or umount, to stop responding, or "hang."
Resolution
Veritas engineering has updated the code to correct the order of locking, to avoid the deadlock between the threads.
This issue is fixed in these VxFS patches:
- 6.0.5.400
- 6.1.1.300
- 6.2.1.100
The issue is also fixed in Storage Foundation High Availability (SFHA) patch 6.1.1.100 SLES11SP4 specific release:
sfha-sles11sp4_x86_64-Patch-6.1.1.100
Patches are available from Veritas SORT:
https://sort.veritas.com/patch/finder
The following hotfixes are also available. Please contact Veritas technical support if the following hotfixes are required:
VxFS 6.1.1:
- fs-rhel6_x86_64-HotFix-6.1.1.203
- fs-rhel5_x86_64-HotFix-6.1.1.106
- fs-rhel6_x86_64-HotFix-6.1.1.105
- fs-sles11_x86_64-HotFix-6.1.1.105
- fs-rhel5_x86_64-HotFix-6.1.1.105
- fs-aix-HotFix-6.1.1.101
VxFS 6.0.5:
- fs-rhel6_x86_64-HotFix-6.0.5.204
- fs-sles11_x86_64-HotFix-6.0.5.109
- fs-sles10_x86_64-HotFix-6.0.5.109
- fs-rhel6_x86_64-HotFix-6.0.5.109
- fs-rhel5_x86_64-HotFix-6.0.5.109
Issue/Introduction
Additional Information
