vx commands hang due to fio-status executes vxdctl command during device discovery

book

Article ID: 100014057

calendar_today

Updated On:

Description

Error Message

All vx commands hang.

Cause

fio-status executes vxdctl command during device discovery in vxconfigd context, and causing dead-lock.

Resolution

Fusion IO fixed this issue in later version which does not execute vxdctl command from fio-status, so please contact Fusion IO to get the fix.

 


Applies To

RHEL 5.10 / VxVM 5.1SP1RP4 / Fusion IO 3.2.4.1086-1.0

Issue/Introduction

For device discovery, Fusion IO ASL will execute fio-status command, and it executes vxdctl command as you can see 
from below back trace:

crash> ps -p 9678
PID: 0      TASK: ffffffff8031db60  CPU: 0   COMMAND: "swapper"
PID: 1      TASK: ffff816000041040  CPU: 148  COMMAND: "init"
  PID: 27860  TASK: ffff811ffb3890c0  CPU: 144  COMMAND: "vxconfigd"
   PID: 9662   TASK: ffff8189edcc5080  CPU: 41  COMMAND: "fio-status"
    PID: 9677   TASK: ffff8189cd9f5830  CPU: 123  COMMAND: "sh"
     PID: 9678   TASK: ffff81a67cbe2040  CPU: 111  COMMAND: "vxdctl"

vxconfigd will not service any command until the discovery is completed, so this causes dead-lock situation.  
And lots of vxconfigd threads are waiting in pipe_read like below:

PID: 27860  TASK: ffff811ffb3890c0  CPU: 144  COMMAND: "vxconfigd" #0 [ffff81088d2a5d28] schedule at ffffffff80062fa0 #1 [ffff81088d2a5e00] pipe_wait at ffffffff80048228 #2 [ffff81088d2a5e50] pipe_readv at ffffffff8002ee70 #3 [ffff81088d2a5ef0] pipe_read at ffffffff800eceaa #4 [ffff81088d2a5f10] vfs_read at ffffffff8000b735 #5 [ffff81088d2a5f40] sys_read at ffffffff80011d9a #6 [ffff81088d2a5f80] sysenter_do_call at ffffffff800614b5