Volume manager commands hang/slow if BCV mirror is in established state on AIX platform

book

Article ID: 100000764

calendar_today

Updated On:

Description

e.g.# ./symmir -g bcvdg queryDevice Group (DG) Name: bcvdgDG's Type             : REGULARDG's Symmetrix ID     : 000290301414Standard Device                    BCV Device                  State-------------------------- ------------------------------------- ------------Inv.                                  Inv.Logical        Sym  Tracks Logical              Sym       Tracks STD BCV-------------------------- ------------------------------------- ------------DEV001         0076      0 BCV001               0078 *         0 SynchronizedDEV002         0077      0 BCV002               0079 *         0 SynchronizedTotal              -------                               -------Track(s)               0                                     0MB(s)                0.0                                   0.0Legend:(*): The paired BCV device is associated with this group.#vxdisk -e -o alldgs listDEVICE       TYPE      DISK         GROUP        STATUS       OS_NATIVE_NAME   ATTRdisk_0       auto      vvrdg01       vvrdg        online       hdisk1       -disk_1       auto      -             -            LVM          hdisk0       -emc1_0076    auto      d1            bcvdg        online       hdisk16      stdemc1_0077    auto      d2            bcvdg        online       hdisk17      stdemc1_0078    auto      -             -            error        hdisk18      bcvemc1_0079    auto      -             -            error        hdisk19      bcv#lsdev  -Cc disk |egrep "hdisk18|hdisk19"hdisk18    Available 1Z-08-01     EMC Symmetrix FCP Raid5hdisk19    Available 1Z-08-01     EMC Symmetrix FCP Raid5

Workaround

1. For VERITAS Volume manager devices only
 
Offline VERITAS volume manager devices before "symmir establish"   command gets called in customer's BCV script.
e.g.

  • Ofline VxVM devices 

#vxdisk offline emc1_0078#vxdisk offline emc1_0079

  •  Perform BCV establish and split operation

# ./symmir -g bcvdg split

  •  Online VxVM devices

#vxdisk online emc1_0078#vxdisk online emc1_0079

  • Import BCV diskgroup

Above work-around will resolve MonitorTimeout issue for VERITAS volume manager Diskgroups in SF-HA cluster.

2.   For VERITAS volume manager diskgroups and AIX LVM Volume groups or/only LVM Volume groups 
 
Change  BCV devices state from AVAILABLE to DEFINED before "symmir establish" command gets called in customer's BCV script.
e.g.

  • Remove VxVM Devices

#vxdisk rm emc1_0078#vxdisk rm emc1_0079

  • Change BCV device status from AVAILABLE to DEFINED using EMC rmbcv command

#/usr/lpp/EMC/Symmetrix/bin/rmbcv hdisk18#/usr/lpp/EMC/Symmetrix/bin/rmbcv hdisk19

  • Perform BCV establish and spit operation

# ./symmir -g bcvdg establish -full# ./symmir -g bcvdg split

  • Change BCV device status from DEFINED to AVAILABLE using EMC mkbcv command

# /usr/lpp/EMC/Symmetrix/bin/mkbcv -a

  • Rescan VxVM devices and import BCV diskgroup

# vxdisk scandisks

Above work-around will resolve MonitorTimeout issue for both VERITAS volume manager Diskgroups and LVM volume groups in SF-HA cluster

  •  Additional notes on Veritas Cluster (VCS) Environment

If the system is under the control of VCS and DiskGroup resource is in use and DiskGroup resource attribute PanicOnDGLoss is set to 1 (which is the default value), the above problem can cause consecutive DiskGroup monitor timeouts.   After 4 consecutive DiskGroup monitor timeouts, the DiskGroup agent will call the clean entry point to halt the system using "/usr/sbin/halt -q".   Please note that the "halt -q" command will not generate any kernel core and will just halt the system.   VCS engine log on this system may not contain the messages as listed in the Error section of this article.   Please check the VCS engine log on the other nodes in the VCS cluster to confirm if the system was halted because of PanicOnDGLoss.  

 

 

 

Resolution

When BCV devices are in established state it does not allow any ioctl from external application like VxVM/LVM which results in application command hang or system wide hang issue.

EMC is aware of this issue and they have documented issue under  their Primus(R) eServer solution (emc115907) which provides procedure of full establish including rmbcv and mkbcv commands which marks the BCV luns in a "Defined" state  as seen by lsdev -Cc disk command

On AIX plat form, customer must need to change BCV device status to "Defined" state before calling BCV establish script

How to check whether BCV devices are in established state or not

Issue/Introduction

Volume manager commands hang/slow if BCV mirror is in established state on AIX platform