How to manage EMC BCV devices on HP-UX with Veritas Volume Manager (VxVM) and Veritas DMP with VxVM 5.0.31.1
Article ID: 100038572
Updated On:
Description
Description
This document attempts to explain the recommended configuration and management steps in relation to handling EMC BCV (Business Continuance Volume) devices on HP-UX with Veritas Volume Manager (VxVM) and Veritas DMP when running VxVM 5.0.31.1.
A BCV is a hardware clone (snapshot) of another EMC Symmetrix device and is mainly used for backups and recovery related operations.
Fig 1.0
When the EMC BCV devices are being synchronized (updated), Veritas recommends offlining the Veritas disk access (da) names. Once the BCV content has been fully synchronized (made up-to-date), the BCV devices are then "split".
Following the BCV "split" operation, VxVM requires the "vxdisk scandisks" command to be performed to enable access to the BCV devices. This triggers a device discovery operation via the vxconfigd daemon and if not managed correctly can lead to vxconfigd hanging.
The BCV devices will transition from a Not-Ready (NR) to a Read-Write (RW) device state.
dmp-cache_open
VxVM 5.0 introduces a new DMP tunable called "dmp_cache_open".
If set to on, the first open of a device is performed by the Array Support Library (ASL) and is cached.
This enhances the performance of device discovery operations by minimizing the overhead caused by subsequent open operations by ASLs. If set to off, caching is not performed.
The DMP tunable "dmp_cache_open" should always be set to "on" to avoid known interoperability issues and additional overheard.
# vxdmpadm gettune dmp_cache_open
Tunable Current Value Default Value
------------------------------ ------------- -------------
dmp_cache_open on on
If the DMP tunable is set to "off", this can cause the vxconfigd daemon to hang during the device discovery activity or when the system is restarted and the BCV devices are in an established state.
Environment
# uname -a
HP-UX berty B.11.31 U 9000/800 3101432737 unlimited-user license
VxVM Patch Summary
To determine which release of VxVM that you have installed, enter the following command:
# swlist -l product VRTSvxvm
If you have the 5.0 release installed, the command output includes the following information:
VRTSvxvm 5.0.31.1 Veritas Volume Manager by Veritas
If you have the 5.0.1 release installed, the command output includes the following information:
VRTSvxvm 5.0.31.5 Veritas Volume Manager by Veritas
If you have the 5.1 SP1 release installed, the command output includes the following information:
VRTSvxvm 5.1.100.000 Veritas Volume Manager by Veritas
# swlist -l product VRTSvxvm
# Initializing...
# Contacting target "berty"...
#
# Target: berty:/
#
VRTSvxvm 5.0.31.1 Veritas Volume Manager by Veritas
EBN_AVID
Storage Foundation 5.0.1 (5.0.31.5) introduces a third DMP naming convention sometimes referred to as "EBN_AVID".
Storage Foundation goes deeper into the storage arrays and discovers the short array volume identifiers(AVID) that are used for identification and management of array volumes on the storage management console.
The device index suffix of an EBN is set to match the discovered AVID.
With the introduction of the DMP based EBN_AVID naming scheme, identifying storage devices becomes far easier with consistent device naming across multiple nodes connected to the same storage and the disk access name will never change as it is based on the name defined at the array level.
This ensures consistency of the disk access name across all nodes connected to the same SAN device.
AVIDs are usually hardware specific. On an EMC DMX array for example, the AVID value matches a device's LUN id (SYMDEV) viewable from the EMC Solution Enabler SYMCLI interface.
Whilst on Hitachi(HDS) or HP XP storage arrays, the AVID equates to a device's CU:LDEV.
This makes the life of the system and storage administrators far easier than ever before.
Sample Configuration
In this instance, the source EMC luns consist of 3 x EMC standard (STD) devices associated with 3 x EMC BCV devices for which must also be associated with the same EMC Symmetrix array (SID).
# vxdisk -eo alldgs list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
EMC0_0 auto - - online invalid c32t3d5
EMC0_1 auto - - online invalid c32t3d6
EMC0_2 auto - - online invalid c32t3d7
EMC0_3 auto - - online invalid c33t1d7
EMC0_4 auto - - online invalid c32t2d0
EMC0_5 auto - - online invalid c32t2d2
EMC0_6 auto - - online invalid c32t2d1
EMC0_7 auto - - online invalid c33t1d6
Fig 2.0
Source EMC SYMDEVs (AVIDs):
The following three source devices are managed by Veritas disk access names, EMC0_3, EMC0_6 and EMC0_7.
Note: Due to the lower VxVM product release, the value add delivered when using the recommended EBN_AVID is not shown here.
# syminq | egrep '0F7|0F8|0F9' | grep rdmp
/dev/vx/rdmp/EMC0_3 EMC SYMMETRIX 5773 22000F9000 2209920
/dev/vx/rdmp/EMC0_6 EMC SYMMETRIX 5773 22000F8000 2209920
/dev/vx/rdmp/EMC0_7 EMC SYMMETRIX 5773 22000F7000 2209920
When VxVM does not have access to the AVID for a device, it retrieves another unique LUN identifier called LUN serial number (LSN).
VxVM will sort the devices based on the LUN serial number (LSN) and uses the index to create the suffix for the device name.Where all nodes see the same set (number) of devices from the same enclosure, VxVM will have the same sorted list (indexed by LSN), leading to consistent device indices across multiple servers.
In this instance, the Veritas disk access (da) name consists of a device index value based on the sorted LSN.
Where the EMC CLI commands are potentially returning stale (out-of-date) details, the "symcfg discover" command can be issued to refresh the storage related output.
# symcfg discover
# syminq | egrep '0F7|0F8|0F9' | grep rdmp
/dev/vx/rdmp/EMC0_4 EMC SYMMETRIX 5773 22000F7000 2209920
/dev/vx/rdmp/EMC0_5 EMC SYMMETRIX 5773 22000F9000 2209920
/dev/vx/rdmp/EMC0_6 EMC SYMMETRIX 5773 22000F8000 2209920
The DMP enclosured based Veritas disk access (da) names are now correctly indexed based on the LSN.
Target EMC BCV SYMDEVs (AVIDs):
The following three source devices are managed by Veritas disk access names, EMC0_3, EMC0_6 and EMC0_7.
# syminq | egrep '0EC|0ED|0EE' | grep rdmp
/dev/vx/rdmp/EMC0_0 BCV EMC SYMMETRIX 5773 22000EC000 2209920
/dev/vx/rdmp/EMC0_1 BCV EMC SYMMETRIX 5773 22000ED000 2209920
/dev/vx/rdmp/EMC0_2 BCV EMC SYMMETRIX 5773 22000EE000 2209920
EMC BCV Configuration Steps with VxVM
1.] Prepare disk for VxVM use:
# vxdisksetup -fi EMC0_3
# vxdisksetup -fi EMC0_6
# vxdisksetup -fi EMC0_7
2.] Create VxVM DiskGroup:
# vxdg init datadg 0F7=EMC0_7 0F8=EMC0_6 0F9=EMC0_3
3.] Create EMC DeviceGroup:
# symdg create bcv_datadg
The EMC Symmetrix ID (SID) can be obtained from the "vxdmpadm listenclosure all" output.
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE
============================================================================
Disk Disk DISKS CONNECTED Disk
AMS_WMS0 AMS_WMS 75050205 CONNECTED A/A-A-HDS
EMC0 EMC 300822 CONNECTED A/A
Note: The last three digits from the ENCLR_SNO is enough to specify the EMC SID.
4.] Add the source and target (BCV) devices to the EMC DeviceGroup "bcv_datadg"
EMC Source devices:
# symld -sid 822 -g bcv_datadg add dev 00F7 DEV00F7
# symld -sid 822 -g bcv_datadg add dev 00F8 DEV00F8
# symld -sid 822 -g bcv_datadg add dev 00F9 DEV00F9
EMC BCV devices:
# symbcv -sid 822 -g bcv_datadg associate dev 00EC BCV00EC
# symbcv -sid 822 -g bcv_datadg associate dev 00ED BCV00ED
# symbcv -sid 822 -g bcv_datadg associate dev 00EE BCV00EE
5.] The EMC DeviceGroup names can be listed using "symdg list":
# symdg list
D E V I C E G R O U P S
Flags Number of
Name Type VC Symmetrix ID Devs GKs BCVs VDEVs TGTs
bcv_datadg REGULAR YN 000290300822 3 0 3 0 0
Legend:
Flags:
V(alid DG) : Y = Valid, N = Invalid, - = N/A
(In) C(g) : Y = Contained by a CG, N = Not contained by a CG, - = N/A
6.] The relationship between the STD and BCV LUNs need to be paired, before data is transferred.
Note: The first time the STD and BCV LUNs are paired, a full establish is always required to perform a full copy of the data from the source to target (clone) devices .
# symmir -g bcv_datadg -v -full establish
Execute 'Full Establish' operation for device group
'bcv_datadg' (y/[n]) ? y
'Full Establish' operation execution is in progress for
device group 'bcv_datadg'. Please wait...
SELECTING Standard devices in the group:
Device: 00F7 [SELECTED]
Device: 00F8 [SELECTED]
Device: 00F9 [SELECTED]
SELECTING BCV devices associated with the group:
Device: 00EC [SELECTED]
Device: 00ED [SELECTED]
Device: 00EE [SELECTED]
PAIRING of Standard and BCV devices:
Devices: 00F7(S) - 00EC(B) [PAIRED]
Devices: 00F8(S) - 00ED(B) [PAIRED]
Devices: 00F9(S) - 00EE(B) [PAIRED]
STARTING a BCV 'ESTABLISH' operation.
The BCV 'ESTABLISH' operation SUCCEEDED.
'Full Establish' operation successfully initiated for device group
'bcv_datadg'.
7.] The VxVM command "vxdisk scandisks" can be used to update the device characteristics.
Fig 3.0
# vxdisk -eo alldgs list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
EMC0_0 auto - - error c32t3d5
EMC0_1 auto - - error c32t3d6
EMC0_2 auto - - error c32t3d7
EMC0_3 auto - - online c33t1d7
EMC0_4 auto 0F7 datadg online c32t2d0
EMC0_5 auto 0F9 datadg online c32t2d2
EMC0_6 auto 0F8 datadg online c32t2d1
EMC0_7 auto - - online c33t1d6
Note: Whilst the EMC BCV devices are being synchronized (refreshed), the device status will be shown as "error".
8.] Veritas recommends offlining the BCV devices whilst syncing.
When BCV devices are in an established state (Not-Ready). DMP opens the path of the device in read-only mode and caches the open state.
Note: Any subsequent application open write request fail.
When the mode of the BCV devices is changed from the Not-Ready (NR) to Read-Write (RW) mode (after a BCV split operation), the OS is unable to close the device and re-open in the correct mode (because an open count is set against the device).
When the device is removed or offlined from Volume Manager control with the commands "vxdisk rm
# vxdisk offline EMC0_0 EMC0_1 EMC0_2
# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
EMC0_0 auto - - offline c32t3d5 <<< BCV (NR)
EMC0_1 auto - - offline c32t3d6 <<< BCV (NR)
EMC0_2 auto - - offline c32t3d7 <<< BCV (NR)
EMC0_3 auto - - online c33t1d7
EMC0_4 auto 0F7 datadg online c32t2d0
EMC0_5 auto 0F9 datadg online c32t2d2
EMC0_6 auto 0F8 datadg online c32t2d1
EMC0_7 auto - - online c33t1d6
9.] Once the BCV devices have been fully synchronized, they can be "split" making them available for use.
How to query the STD and BCV LUNs are fully synchronized
# symmir -g bcv_datadg query
Device Group (DG) Name: bcv_datadg
DG's Type : REGULAR
DG's Symmetrix ID : 000290300822
Standard Device BCV Device State
-------------------------- ------------------------------------- ------------
Inv. Inv.
Logical Sym Tracks Logical Sym Tracks STD <=> BCV
-------------------------- ------------------------------------- ------------
DEV00F7 00F7 0 BCV00EC 00EC * 0 Synchronized
DEV00F8 00F8 0 BCV00ED 00ED * 0 Synchronized
DEV00F9 00F9 0 BCV00EE 00EE * 0 Synchronized
Total ------- -------
Track(s) 0 0
MB(s) 0.0 0.0
Legend:
(*): The paired BCV device is associated with this group.
# symmir -g bcv_datadg split
Execute 'Split' operation for device group
'bcv_datadg' (y/[n]) ? y
'Split' operation execution is in progress for
device group 'bcv_datadg'. Please wait...
'Split' operation successfully executed for device group
'bcv_datadg'.
# symmir -g bcv_datadg -split verify
All devices in the group 'bcv_datadg' are in 'Split' state.
10.] Once the EMC BCV devices have been successfully "split", the corresponding BCV disk status should be updated by running "vxdisk scandisks".
Following the BCV split operation, the VxVM device state may automatically reflect "error" and not the previous "offline" state.
# vxdisk list
DEVICE TYPE DISK GROUP STATUS
EMC0_0 auto - - error
EMC0_1 auto - - error
EMC0_2 auto - - error
EMC0_3 auto:none - - online invalid
EMC0_4 auto:cdsdisk 0F7 datadg online
EMC0_5 auto:cdsdisk 0F9 datadg online
EMC0_6 auto:cdsdisk 0F8 datadg online
EMC0_7 auto:none - - online invalid
When attempting to online the related Veritas disk access (da) names for the BCV devices, you may see the following errors:
# vxdisk online EMC0_0 EMC0_1 EMC0_2
VxVM vxdisk ERROR V-5-1-531 Device EMC0_0: online failed:
Device path not valid
VxVM vxdisk ERROR V-5-1-531 Device EMC0_1: online failed:
Device path not valid
VxVM vxdisk ERROR V-5-1-531 Device EMC0_2: online failed:
Device path not valid
Note: It is critical that the VxVM disk attributes are refreshed following the BCV split operation, to ensure VxVM has been updated with the modified BCV device state change.
Without refreshing the VxVM structures, you may see the following error when attempting to import the BCV disk group image.
# vxdg -n bcv_datadg -o useclonedev=on -o updateid import datadg
VxVM vxdg ERROR V-5-1-10978 Disk group datadg: import failed:
No clone disks for import
Note: The "vxdisk scandisks" command will transition the devices from the VxVM device state of "error" (now split - R/W) to "online".
# vxdisk scandisks
# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
EMC0_0 auto - - online c32t3d5
EMC0_1 auto - - online c32t3d6
EMC0_2 auto - - online c32t3d7
EMC0_3 auto - - online c33t1d7
EMC0_4 auto 0F7 datadg online c32t2d0
EMC0_5 auto 0F9 datadg online c32t2d2
EMC0_6 auto 0F8 datadg online c32t2d1
EMC0_7 auto - - online c33t1d6
The BCV disk group image shows the same name as the source disk group, so needs to be made unique before it can be imported.
# vxdisk -eo alldgs list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
EMC0_0 auto - (datadg) online c32t3d5 <<< BCV disk group image
EMC0_1 auto - (datadg) online c32t3d6 <<< BCV disk group image
EMC0_2 auto - (datadg) online c32t3d7 <<< BCV disk group image
EMC0_3 auto - - online c33t1d7
EMC0_4 auto 0F7 datadg online c32t2d0
EMC0_5 auto 0F9 datadg online c32t2d2
EMC0_6 auto 0F8 datadg online c32t2d1
EMC0_7 auto - - online c33t1d6
Special VxVM disk group import options must be used to make the BCV disk group image unique (when the BCV devices are presented to the same server as the source devices).
# vxdg -n bcv_datadg -o useclonedev=on -o updateid import datadg
As a result of the above syntax, the BCV disk group image is updated and assigned an unique dgid and name. In addition, the Veritas diskid and UDID related content is updated for each BCV device associated with the successfully imported disk group.
# vxdg list
NAME STATE ID
bcv_datadg enabled,cds 1413474164.63.berty <<< BCV disk group image
datadg enabled,cds 1413367283.61.berty
# vxdisk -e list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
EMC0_0 auto 0F8 bcv_datadg online c32t3d6
EMC0_1 auto 0F7 bcv_datadg online c32t3d5
EMC0_2 auto 0F9 bcv_datadg online c32t3d7
EMC0_3 auto 0F9 datadg online c32t2d2
EMC0_4 auto - - online c33t1d6
EMC0_5 auto - - online c33t1d7
EMC0_6 auto 0F8 datadg online c32t2d1
EMC0_7 auto 0F7 datadg online c32t2d0
How to resync the BCV disk group image
The VxVM disk group for the BCV devices must be deported, prior to resyncing the BCV content.
# vxdg deport bcv_datadg
# vxdisk -eo alldgs list
EMC0_0 auto - (bcv_datadg) online c32t3d5
EMC0_1 auto - (bcv_datadg) online c32t3d6
EMC0_2 auto - (bcv_datadg) online c32t3d7
EMC0_3 auto - - online c33t1d7
EMC0_4 auto 0F7 datadg online c32t2d0
EMC0_5 auto 0F9 datadg online c32t2d2
EMC0_6 auto 0F8 datadg online c32t2d1
EMC0_7 auto - - online c33t1d6
To minimize VxVM hangs and other related activity, offline the corresponding BCV devices when resyncing.
# vxdisk offline EMC0_0 EMC0_1 EMC0_2
# symmir -g bcv_datadg establish
Execute 'Incremental Establish' operation for device group
'bcv_datadg' (y/[n]) ? y
'Incremental Establish' operation execution is in progress for
device group 'bcv_datadg'. Please wait...
'Incremental Establish' operation successfully initiated for device group
'bcv_datadg'.
# symmir -g bcv_datadg query
Device Group (DG) Name: bcv_datadg
DG's Type : REGULAR
DG's Symmetrix ID : 000290300822
Standard Device BCV Device State
-------------------------- ------------------------------------- ------------
Inv. Inv.
Logical Sym Tracks Logical Sym Tracks STD <=> BCV
-------------------------- ------------------------------------- ------------
DEV00F7 00F7 0 BCV00EC 00EC * 0 Synchronized
DEV00F8 00F8 0 BCV00ED 00ED * 0 Synchronized
DEV00F9 00F9 0 BCV00EE 00EE * 0 Synchronized
Total ------- -------
Track(s) 0 0
MB(s) 0.0 0.0
Legend:
(*): The paired BCV device is associated with this group.
# vxdisk -eo alldgs list
DEVICE TYPE DISK GROUP STATUS
EMC0_0 auto - - offline
EMC0_1 auto - - offline
EMC0_2 auto - - offline
EMC0_3 auto:none - - online invalid
EMC0_4 auto:cdsdisk 0F7 datadg online
EMC0_5 auto:cdsdisk 0F9 datadg online
EMC0_6 auto:cdsdisk 0F8 datadg online
EMC0_7 auto:none - - online invalid
Once the BCV content has been fully resynchronized, split the BCV devices.
# symmir -g bcv_datadg split
Execute 'Split' operation for device group
'bcv_datadg' (y/[n]) ? y
'Split' operation execution is in progress for
device group 'bcv_datadg'. Please wait...
'Split' operation successfully executed for device group
'bcv_datadg'.
# symmir -g bcv_datadg query
Device Group (DG) Name: bcv_datadg
DG's Type : REGULAR
DG's Symmetrix ID : 000290300822
Standard Device BCV Device State
-------------------------- ------------------------------------- ------------
Inv. Inv.
Logical Sym Tracks Logical Sym Tracks STD <=> BCV
-------------------------- ------------------------------------- ------------
DEV00F7 00F7 0 BCV00EC 00EC * 0 Split
DEV00F8 00F8 0 BCV00ED 00ED * 0 Split
DEV00F9 00F9 0 BCV00EE 00EE * 0 Split
Total ------- -------
Track(s) 0 0
MB(s) 0.0 0.0
Legend:
(*): The paired BCV device is associated with this group.
# symmir -g bcv_datadg -split verify
All devices in the group 'bcv_datadg' are in 'Split' state.
Note: When trying to online the EMC BCV disks, you may see the following errors:
# vxdisk online EMC0_0 EMC0_1 EMC0_2
VxVM vxdisk ERROR V-5-1-531 Device EMC0_0: online failed:
Device path not valid
VxVM vxdisk ERROR V-5-1-531 Device EMC0_1: online failed:
Device path not valid
VxVM vxdisk ERROR V-5-1-531 Device EMC0_2: online failed:
Device path not valid
# vxdisk scandisks
# vxdisk -eo alldgs list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME
EMC0_0 auto - (datadg) online c32t3d7
EMC0_1 auto - (datadg) online c32t3d6
EMC0_2 auto - (datadg) online c32t3d5
EMC0_3 auto - - online c33t1d7
EMC0_4 auto - - online c33t1d6
EMC0_5 auto 0F9 datadg online c32t2d2
EMC0_6 auto 0F8 datadg online c32t2d1
EMC0_7 auto 0F7 datadg online c32t2d0
Now try and import the BCV disk group image once again.
Issue/Introduction
