The abort of a node in Cluster File System will lead to a transient hang of "df" on other remaining nodes for the Cluster Filesystem Recovery.

book

Article ID: 100009894

calendar_today

Updated On:

Cause

For the Cluster File System, CFS must maintain the data consistency clusterwide. So, it is the CFS primary node that is repsonbile for releasing the GLM(short for Group/Global Lock Manager) locks and recovering the unconsistent metadata caused by the left node, whenever node force-leave occurs.

During the procedure of the above recovering, the IO access to this Cluster File System is limited(only buffer reads are allowed usually) and some VxFS in-core meta data need to be recaped and repopulated to the kernel, which hinders the access of the VFS(Virtual File System) table and further influences the execution of commands like "df". Moreover, real data writes to the Cluster File System will be blocked until the completion of CFS recovery.

The above behaviors can by seen by checking the OS kernel threads via the OS tool. In the remaining CFS primary node or new primary node , you can find the fsckd - the VxFS daemon to conduct the recovery - that is engaged in delivering IOs to recover filesystem.

Resolution

This recovery procedure is the one to protect the data consistency, and it is done by design. So, the transient hang of "df" is inevitable in case lots of rountine jobs need to be completed by the CFS primary node.

For the applications which need to timely access the data, please increase the related TimeOut attribute.


Issue/Introduction

This recovery process can be experienced in the SFCFS or SFRAC, and the time frame of the hang can be diverse ranging from couples of seconds to several minutes for the difference of the loading on the left node and of the physical productivity of the current suvival CFS primary node or new promoted primary node. The side-effect of this recovery is that, other application processes might be influenced as well , for instance, Oracle cssd daemon might be offline unexpectedly because of the failure on accessing to the voting disk situated in the CFS, in case RAC is deployed.