Description

This documents attempts to explain how to simulate sd instance movements on Linux(RedHat) to trigger inconsistent VxVM device suppression.
 

Veritas disk access name "emc0_0289" will be excluded from veritas Volume Manager's (VxVM) view.
 

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
emc0_0288    auto:cdsdisk    -            -            online
emc0_0289    auto:cdsdisk    -            -            online                  <<< to be excluded

Note: Veritas disk access name "emc0_0289" has two paths, "sdbn" and "sdat" at this time.
 

# vxdisk path | grep emc0_0289
sdbn                        emc0_0289            -            -            ENABLED
sdat                        emc0_0289            -            -            ENABLED


# vxdmpadm exclude vxvm dmpnodename=emc0_0289


The above vxdmpadm CLI device suppression operation populates the /etc/vx/vxvm.exclude file with the following content:


# more /etc/vx/vxvm.exclude
exclude_all 0
paths
sdbn pci-0000:02:05.0-fc-0x5006048c5368e5af:0x0108000000000000 emc0_0289
sdat pci-0000:02:05.0-fc-0x5006048c5368e58f:0x011b000000000000 emc0_0289
#
controllers
#
product
#
pathgroups
#

 

The Veritas disk access name "emc0_0289" is not longer visible to VxVM and such command as "vxdisk list" or "vxdisk path" and so on.


Revised output


# vxdisk path | grep emc0_0289
 

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
emc0_0288    auto:cdsdisk    -            -            online

# vxdisk path | grep emc0_0288
sdbm                        emc0_0288            -            -            ENABLED
sdas                        emc0_0288            -            -            ENABLED

LUN reconfiguration steps

# vxdisk rm emc0_0288

# symmask list hba

Identifier        Type   Adapter           Physical Device Path      Dir:P
----------------  -----  ----------------  ------------------------  -----
210000e08b1c1df0  Fibre  210000e08b1c1df0  /dev/sdad                 16C:0
                                           /dev/sdau                 16C:1

# symmask -sid 822 remove devs 0288 -wwn 210000e08b1c1df0 -dir 16c -p 0
# symmask -sid 822 remove devs 0288 -wwn 210000e08b1c1df0 -dir 16c -p 1


# symmask -sid 822 refresh -nopr

Symmetrix FA/SE directors updated with contents of SymMask Database 000290300822


# symcfg -sid 822 discover

Attempting discovery of Symmetrix 000290300822

This operation may take up to a few minutes. Please be patient...

The system is rebooted.

Following the system restart, Veritas disk access name "emc0_0289" is visible once again, no longer suppressed.

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
emc0_0289    auto:cdsdisk    -            -            online                  <<< device is excluded

Note: The native sd paths for "emc0_0289" have changed from "sdbn" and "sdat" to "sdbl" and "sdas".
 

# vxdisk path | grep emc0_0289
sdbl                        emc0_0289            -            -            ENABLED
sdas                        emc0_0289            -            -            ENABLED
 

# more /etc/vx/vxvm.exclude
exclude_all 0
paths
sdbn pci-0000:02:05.0-fc-0x5006048c5368e5af:0x0108000000000000 emc0_0289
sdat pci-0000:02:05.0-fc-0x5006048c5368e58f:0x011b000000000000 emc0_0289
#
controllers
#
product
#
pathgroups
#
 

Workaround:

1.] udevtrigger

2.] vxdisk scandisks

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid


Note: The device suppression is active again for "emc0_0289"


Summary


Veritas Volume Manager 5.1 SP1 RP1 P1 (Linux) introduces persistent device exclusion following LUN reconfiguration events
( as per Etrack 2241750 [Product Enhancement][VxVM]Introduce persistent device exclusion following LUN reconfiguration events on Linux)

The mappings created by the udev-rule are lost during the execution of the make_devices function of the vxvm-startup script, where /dev/vx is mounted on tmpfs. As a result, during discovery, vxconfigd will not be able to determine the hardware path for the excluded devices and hence the exclusion does not work.


We therefore modified the /etc/vx/vxvm-startup script to call udevtrigger before vxconfigd starts.

 


make_devices()
{
       #
       # create devices needed under /dev/vx
       # Handle special-case for selinux enabled systems
       #
       [ -d /dev/vx ] || /bin/mkdir -m 0755 -p /dev/vx
       [ -x /usr/sbin/selinuxenabled ] && selinuxenabled || {
               # Make /dev/vx available on tmpfs
               /bin/mount -t tmpfs -o size=4k,nr_inodes=2097152,mode=0755
                       tmpfs /dev/vx
       }
      /sbin/udevtrigger                               <<<< added this new entry
       /bin/mkdir -m 0755 /dev/vx/dmp 2>/dev/null
       /bin/ln -s /dev/vx/dmp /dev/vx/rdmp 2>/dev/null
       /bin/mknod /dev/vx/config c $VXVM_MAJOR_NUMBER 0 2>/dev/null
       /bin/mknod /dev/vx/trace c $VXVM_MAJOR_NUMBER 1 2>/dev/null
       /bin/mknod /dev/vx/iod c $VXVM_MAJOR_NUMBER 2 2>/dev/null
       /bin/mknod -m 646 /dev/vx/info c $VXVM_MAJOR_NUMBER 3 2>/dev/null
       /bin/mknod /dev/vx/task c $VXVM_MAJOR_NUMBER 4 2>/dev/null
       /bin/mknod /dev/vx/taskmon c $VXVM_MAJOR_NUMBER 5 2>/dev/null
       /bin/mknod /dev/vx/clust c $VXVM_MAJOR_NUMBER 6 2>/dev/null
       /bin/mknod /dev/vx/netiod c $VXVM_MAJOR_NUMBER 7 2>/dev/null
       /bin/mknod /dev/vx/vcevent c $VXVM_MAJOR_NUMBER 8 2>/dev/null
       /bin/mknod /dev/vx/esd c $VXVM_MAJOR_NUMBER 9 2>/dev/null
       /bin/mknod /dev/vx/vsevent c $VXVM_MAJOR_NUMBER 10 2>/dev/null
       /bin/mknod /dev/vx/dmpconfig b $VXDMP_MAJOR_NUMBER 1048575 2> /dev/null
}



 
Process performed again with the "udevtrigger" entry present in /etc/vx/vxvm-startup


Note: With the above "udevtrigger" in place, the device exclusion is persistent across a reboot as shown below:

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid
emc0_0288    auto:cdsdisk    -            -            online                       <<< to be masked at the array level
emc0_0289    auto:cdsdisk    -            -            online                  <<< to be suppressed from VxVM's view

Veritas disk access name "emc0_0289" will be suppressed from VxVM's view:

# vxdisk path | grep emc0_0289
sdbl                        emc0_0289            -            -            ENABLED
sdas                        emc0_0289            -            -            ENABLED

Note: The native sd paths for "emc0_0289" are "sdbl" and "sdas".

# vxdmpadm exclude vxvm dmpnodename=emc0_0289

Veritas disk access name "emc0_0288" will be masked from the server at the array level.

# vxdisk path | grep emc0_0288
sdbp                        emc0_0288            -            -            ENABLED
sdbq                        emc0_0288            -            -            ENABLED

# vxdisk rm emc0_0288
 

# symmask -sid 822 remove devs 0288 -wwn 210000e08b1c1df0 -dir 16c -p 0

# symmask -sid 822 remove devs 0288 -wwn 210000e08b1c1df0 -dir 16c -p 1

# symmask -sid 822 refresh

Refresh Symmetrix FA/SE directors with contents of SymMask database 000290300822 (y/[n]) ? y

Symmetrix FA/SE directors updated with contents of SymMask Database 000290300822


# symcfg -sid 822 discover

Attempting discovery of Symmetrix 000290300822

This operation may take up to a few minutes. Please be patient...
#

The server is then rebooted.

Note: Following the system restart, Veritas disk access name "emc0_0289" is still suppressed from VxVM's view.
 

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:none       -            -            online invalid