How to remove EMC CLARiiON devices from a FileStore (SLES) server using DMP
Article ID: 100037764
Updated On:
Description
Description
This document attempts to explain the essential steps required to remove EMC CLARiiON storage devices controlled by DMP
The following EMC CLARiiON devices will be removed from a FileStore (SLES) server using DMP as the multipathing solution.
Configuration:
# cat /etc/SuSE-release
SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL = 2
# rpm -qa | grep -i VRTSvxvm
VRTSvxvm-common-5.0.30.32-MP3RP3HF2_SLES10
VRTSvxvm-platform-5.0.30.32-MP3RP3HF2_SLES10
In this instance the EMC CLARiiON StorageGroup name is "rdg2970-03+04-02", which consists of "4" luns.
How to view the EMC CLARiiON devices associated with StorageGroup " rdg2970-03+04-02 "
Figure 1.0
Sample output
The following EMC CLI interface requires that the EMC NaviCLI related rpm (NaviCLI-Linux-64-x86-en_US-7.30.0.4.74-1.x86_64.rpm) be installed.
# /opt/Navisphere/bin/naviseccli -h 10.12.253.155 storagegroup -list -gname rdg2970-03+04-02
Storage Group Name: rdg2970-03+04-02
Storage Group UID: 5A:BD:B6:82:3D:08:E0:11:B2:F8:00:60:16:25:5F:A1
HBA/SP Pairs:
HBA UID SP Name SPPort
------- ------- ------
20:00:00:00:C9:69:36:85:10:00:00:00:C9:69:36:85 SP B 5
20:00:00:00:C9:69:76:77:10:00:00:00:C9:69:76:77 SP B 5
20:00:00:00:C9:69:36:85:10:00:00:00:C9:69:36:85 SP A 5
20:00:00:00:C9:69:76:77:10:00:00:00:C9:69:76:77 SP A 5
20:00:00:00:C9:69:36:85:10:00:00:00:C9:69:36:85 SP B 4
20:00:00:00:C9:69:76:77:10:00:00:00:C9:69:76:77 SP B 4
20:00:00:00:C9:69:36:85:10:00:00:00:C9:69:36:85 SP A 4
20:00:00:00:C9:69:76:77:10:00:00:00:C9:69:76:77 SP A 4
HLU/ALU Pairs:
HLU Number ALU Number
---------- ----------
1 25
0 24
3 23
2 22
Shareable: YES
Using the FileStore CLISH interface whilst logged in as the "master" user you can display the WWN content:
filestore1> storage hba
Node Host Initiator HBA WWNs
==== =======================
filestore1_01 10:00:00:00:c9:69:76:77
filestore1_02 10:00:00:00:c9:69:36:85
Array Volume Identifier Support
As configurations in data centers scale up and more SAN devices and nodes have to be managed, there is a greater need for consistency and predictability in device naming.
Storage Foundation(SF) support two device naming schemes - Operating System Native (OSN) naming and Enclosure Based Names (EBN)
Enclosure name consistency usually is not a problem and even when hosts are connected to multiple enclosures of the same type, the number of enclosures is small enough that consistency can be ensured through manual configurations.
Device index consistency that uniquely identifies the device within that enclosure on the other hand can be inconsistent.
To overcome this limitation SF 5.0 MP3 introduces a third naming convention sometimes referred as "EBN_AVID" to enable device name consistency
EBN = Enclosure Based Naming
AVID = Array Volume Identifier
Lun removal pre-checks
1.] Capture the Linux device ids for the luns to be removed.
Output captured whilst logged in as the "support" user
# lsscsi
[0:0:0:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sda
[0:0:1:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sdb
[5:0:0:0] disk DGC RAID 5 0326 /dev/sdc
[5:0:0:1] disk DGC RAID 5 0326 /dev/sdd
[5:0:0:2] disk DGC RAID 5 0326 /dev/sde
[5:0:0:3] disk DGC RAID 5 0326 /dev/sdf
[5:0:1:0] disk DGC RAID 5 0326 /dev/sdg
[5:0:1:1] disk DGC RAID 5 0326 /dev/sdh
[5:0:1:2] disk DGC RAID 5 0326 /dev/sdi
[5:0:1:3] disk DGC RAID 5 0326 /dev/sdj
[5:0:2:0] disk DGC RAID 5 0326 /dev/sdk
[5:0:2:1] disk DGC RAID 5 0326 /dev/sdl
[5:0:2:2] disk DGC RAID 5 0326 /dev/sdm
[5:0:2:3] disk DGC RAID 5 0326 /dev/sdn
[5:0:3:0] disk DGC RAID 5 0326 /dev/sdo
[5:0:3:1] disk DGC RAID 5 0326 /dev/sdp
[5:0:3:2] disk DGC RAID 5 0326 /dev/sdq
[5:0:3:3] disk DGC RAID 5 0326 /dev/sdr
[5:0:4:0] disk HITACHI DF600F 0000 /dev/sds
[5:0:4:1] disk HITACHI DF600F 0000 /dev/sdt
[5:0:4:2] disk HITACHI DF600F 0000 /dev/sdu
[5:0:4:3] disk HITACHI DF600F 0000 /dev/sdv
[5:0:4:4] disk HITACHI DF600F 0000 /dev/sdw
[5:0:4:5] disk HITACHI DF600F 0000 /dev/sdx
[5:0:4:6] disk HITACHI DF600F 0000 /dev/sdy
Note: In this instance the EMC CLARiiON devices to be removed are shown above in bold text.
Output captured whilst logged in as the "support" user
# vxdmpadm listenclosure all
ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT
===================================================================================
ams_wms0 AMS_WMS 75050205 CONNECTED A/A-A 7
other_disks OTHER_DISKS OTHER_DISKS CONNECTED OTHER_DISKS 2
emc_clariion0 EMC_CLARiiON CK200083201067 CONNECTED CLR-A/PF 4
Note: Confirmation that DMP has claimed 4 luns under the EMC CLARiiON enclosure.
How the introduction of the AVID feature makes life easier
With the introduction of the new EBN_AVID naming scheme, identifying storage devices becomes far easier with consistent device naming across multiple nodes connected to the same storage and the disk access name will never change as it is based on the name defined at the array level.
When SF does not have access to a device AVID, it retrieves another unique LUN identifier called LUN serial number
SF will sort the devices based on the LUN serial number (LSN) and uses the index to create the suffix for the device name. All nodes that see the same set of devices, will have the same sorted list, leading to consistent device indices across the cluster.
Finally, Storage Foundation supports a scalable framework allowing users to fully customize the device names on a host by applying a 'device naming file' that associates custom names with cabinet and LUN serial numbers.
# vxddladm get namingscheme
NAMING_SCHEME PERSISTENCE LOWERCASE USE_AVID
============================================================
Enclosure Based Yes Yes Yes
How to remove the EMC CLARiiON devices using the FileStore CLISH interface
Note: You must be connected to the FileStore "master" server and logged in as the "master" user.
In this instance the FileStore storage pool "cx-luns" consists of four luns
filestore1> storage pool list
Pool List of disks
================================== ====================================
cx-luns emc_clariion0_22 emc_clariion0_23 emc_clariion0_24 emc_clariion0_25
pool1 ams_wms0_106 ams_wms0_107
pool2 ams_wms0_131 ams_wms0_190
filestore1> storage pool destroy cx-luns
SFS pool Success V-288-1016 Pool cx-luns is destroyed
filestore1> storage pool list
Pool List of disks
================================== ====================================
pool1 ams_wms0_106 ams_wms0_107
pool2 ams_wms0_131 ams_wms0_190
filestore1>
Note: Once the storage pool has been deleted or the respective disks have been removed from the related pool, the devices need to be removed from VxVM and the O/S.
Figure 2.0
Output captured whilst logged in as the "support" user
filestore1_02:~ # vxdisk -eo alldgs list
DEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTR
ams_wms0_81 auto:simple - - online clone_disk sds std
ams_wms0_106 auto:simple ams_wms0_106 sfsdg online clone_disk shared sdx std
ams_wms0_107 auto:simple ams_wms0_107 sfsdg online clone_disk shared sdy std
ams_wms0_131 auto:simple ams_wms0_131 sfsdg online clone_disk shared sdt std
ams_wms0_190 auto:simple ams_wms0_190 sfsdg online clone_disk shared sdu std
ams_wms0_191 auto:simple - - online clone_disk sdv std
ams_wms0_192 auto:simple - - online clone_disk sdw std
emc_clariion0_22 auto:simple - - online clone_disk sdq lun
emc_clariion0_23 auto:simple - - online clone_disk sdf lun
emc_clariion0_24 auto:simple - - online clone_disk sdc lun
emc_clariion0_25 auto:simple - - online clone_disk sdp lun
sda auto:none - - online invalid sda -
sdb auto:none - - online invalid sdb -
Note: Confirmation that the EMC CLARiiON luns are no longer associated with the "sfsdg" diskgroup.
EMC CLARiiON ALU references to Veritas Disk Access Names
EMC CLARiiON ALU reference "ALU 22" refers to HLU "2", Linux device id "2", Veritas disk access name "emc_clarrion0_22", AVID "22"
EMC CLARiiON ALU reference "ALU 23" refers to HLU "3", Linux device id "3", Veritas disk access name "emc_clarrion0_23", AVID "23"
EMC CLARiiON ALU reference "ALU 24" refers to HLU "0", Linux device id "0", Veritas disk access name "emc_clarrion0_24", AVID "24"
EMC CLARiiON ALU reference "ALU 25" refers to HLU "1", Linux device id "1", Veritas disk access name "emc_clarrion0_25", AVID "25"
1.] Execute the following command on both nodes
Output captured whilst logged in as the "support" user
Master Server:
f ilestore1_02:~ # vxdisk rm emc_clariion0_22 emc_clariion0_23 emc_clariion0_24 emc_clariion0_25
Slave Server:
filestore1_01:~ # vxdisk rm emc_clariion0_22 emc_clariion0_23 emc_clariion0_24 emc_clariion0_25
Using the disk "ID" details from "storage disk list detail", delete the corresponding Linux device (sd instance) entries.
filestore1> storage disk list detail
Disk Pool Enclosure Size (Use%) ID Serial Nu
mber
==== ============================= ========= =========== == =========
=========================
ams_wms0_106 pool1 ams_wms0 20.0G 50.5% HITACHI:DF600F%5F75050205%5F0
06A:4:5 006A
ams_wms0_107 pool1 ams_wms0 20.0G 5.4% HITACHI:DF600F%5F75050205%5F0
06B:4:6 006B
ams_wms0_131 pool2 ams_wms0 20.0G 5.4% HITACHI:DF600F%5F75050205%5F0
083:4:1 0083
ams_wms0_190 pool2 ams_wms0 20.0G 0.4% HITACHI:DF600F%5F75050205%5F0
0BE:4:2 00BE
ams_wms0_191 - ams_wms0 20.0G 0.0% HITACHI:DF600F%5F75050205%5F0
0BF:4:3 00BF
ams_wms0_192 - ams_wms0 20.0G 0.0% HITACHI:DF600F%5F75050205%5F0
0C0:4:4 00C0
ams_wms0_81 - ams_wms0 20.0G 0.0% HITACHI:DF600F%5F75050205%5F0
051:4:0 0051
emc_clariion0_22 - emc_clariion0 1.0G 0.0% DGC:RAID%205:0:2 60060160C
C702100C9551D5D0D0DDF11
emc_clariion0_23 - emc_clariion0 1.0G 0.0% DGC:RAID%205:0:3 60060160C
C702100CA551D5D0D0DDF11
emc_clariion0_24 - emc_clariion0 1.0G 0.0% DGC:RAID%205:0:0 60060160C
C702100CB551D5D0D0DDF11
emc_clariion0_25 - emc_clariion0 1.0G 0.0% DGC:RAID%205:0:1 60060160C
C702100CC551D5D0D0DDF11
LINUX Device entries for disk access name "emc_clariion0_22"
Master Server:
filestore1_02:~ # lsscsi | grep ":0:2" | grep DGC | grep RAID
[1:0:0:2] disk DGC RAID 5 0326 /dev/sde
[1:0:2:0] disk DGC RAID 5 0326 /dev/sdk
[1:0:2:1] disk DGC RAID 5 0326 /dev/sdl
[1:0:2:2] disk DGC RAID 5 0326 /dev/sdm
[1:0:2:3] disk DGC RAID 5 0326 /dev/sdn
Note: Populate the echo commands with the corresponding "sd#" instance, ie /dev/sde would be referenced as "/sys/block/sde/device/delete".
filestore1_02:~ # echo 1 > /sys/block/sde/device/delete
filestore1_02:~ # echo 1 > /sys/block/sdk/device/delete
filestore1_02:~ # echo 1 > /sys/block/sdl/device/delete
filestore1_02:~ # echo 1 > /sys/block/sdm/device/delete
filestore1_02:~ # echo 1 > /sys/block/sdn/device/delete
Confirm that lsscsi no longer reports any device ids for "emc_clariion0_22"
filestore1_02:~ # lsscsi | grep ":0:2" | grep DGC | grep RAID
filestore1_02:~ #
Slave server:
Repeat LINUX device deletion on the slave server as follows:
filestore1_01:~ # lsscsi | grep ":0:2" | grep DGC | grep RAID
[5:0:0:2] disk DGC RAID 5 0326 /dev/sde
[5:0:2:0] disk DGC RAID 5 0326 /dev/sdk
[5:0:2:1] disk DGC RAID 5 0326 /dev/sdl
[5:0:2:2] disk DGC RAID 5 0326 /dev/sdm
[5:0:2:3] disk DGC RAID 5 0326 /dev/sdn
filestore1_01:~ # echo 1 > /sys/block/sde/device/delete
filestore1_01:~ # echo 1 > /sys/block/sdk/device/delete
filestore1_01:~ # echo 1 > /sys/block/sdl/device/delete
filestore1_01:~ # echo 1 > /sys/block/sdm/device/delete
filestore1_01:~ # echo 1 > /sys/block/sdn/device/delete
Confirm that lsscsi no longer reports any device ids for "emc_clariion0_22"
filestore1_01:~ # lsscsi | grep ":0:2" | grep DGC | grep RAID
filestore1_01:~ #
Repeat the process for the remaining 3 EMC CLARiiON luns.
Sample for loop
filestore1_02:~ # for disk in `lsscsi | grep DGC | awk -F"/" '{print $3}'`
> do
> echo 1 > /sys/block/$disk/device/delete
> echo "Deleted:$disk"
> done
Deleted:sdc
Deleted:sdd
Deleted:sdf
Deleted:sdg
Deleted:sdh
Deleted:sdi
Deleted:sdj
Deleted:sdo
Deleted:sdp
Deleted:sdq
Deleted:sdr
filestore1_01:~ # for disk in `lsscsi | grep DGC | awk -F"/" '{print $3}'`
> do
> echo 1 > /sys/block/$disk/device/delete
> echo "Deleted:$disk"
> done
Deleted:sdc
Deleted:sdd
Deleted:sdf
Deleted:sdg
Deleted:sdh
Deleted:sdi
Deleted:sdj
Deleted:sdo
Deleted:sdp
Deleted:sdq
Deleted:sdr
Revised "lsscsi" output from both the "master" and "slave" servers:
Master Server:
filestore1_02:~ # vxdctl -c mode
mode: enabled: cluster active - MASTER
master: filestore1_02
filestore1_02:~ # lsscsi
[0:0:0:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sda
[0:0:1:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sdb
[1:0:4:0] disk HITACHI DF600F 0000 /dev/sds
[1:0:4:1] disk HITACHI DF600F 0000 /dev/sdt
[1:0:4:2] disk HITACHI DF600F 0000 /dev/sdu
[1:0:4:3] disk HITACHI DF600F 0000 /dev/sdv
[1:0:4:4] disk HITACHI DF600F 0000 /dev/sdw
[1:0:4:5] disk HITACHI DF600F 0000 /dev/sdx
[1:0:4:6] disk HITACHI DF600F 0000 /dev/sdy
[1:0:5:0] disk EMC SYMMETRIX 5773 /dev/sdz
[1:0:6:0] disk EMC SYMMETRIX 5773 /dev/sdaa
[1:0:7:0] disk EMC SYMMETRIX 5773 /dev/sdab
[1:0:8:0] disk EMC SYMMETRIX 5773 /dev/sdac
Slave Server:
filestore1_01:~ # lsscsi
[0:0:0:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sda
[0:0:1:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sdb
[5:0:4:0] disk HITACHI DF600F 0000 /dev/sds
[5:0:4:1] disk HITACHI DF600F 0000 /dev/sdt
[5:0:4:2] disk HITACHI DF600F 0000 /dev/sdu
[5:0:4:3] disk HITACHI DF600F 0000 /dev/sdv
[5:0:4:4] disk HITACHI DF600F 0000 /dev/sdw
[5:0:4:5] disk HITACHI DF600F 0000 /dev/sdx
[5:0:4:6] disk HITACHI DF600F 0000 /dev/sdy
Note: Now both the O/S and VxVM have been cleaned. The devices can be removed at the storage layer.
EMC CLARiiON Storage commands
Master Server:
# /opt/Navisphere/bin/naviseccli -h 10.12.253.155 storagegroup -disconnecthost -o -host rdg2970-04 -gname rdg2970-03+04-02
Slave Server:
# /opt/Navisphere/bin/naviseccli -h 10.12.253.155 storagegroup -disconnecthost -o -host rdg2970-03 -gname rdg2970-03+04-02
# /opt/Navisphere/bin/naviseccli -h 10.12.253.155 storagegroup -list -gname rdg2970-03+04-02
Storage Group Name: rdg2970-03+04-02
Storage Group UID: 5A:BD:B6:82:3D:08:E0:11:B2:F8:00:60:16:25:5F:A1
HLU/ALU Pairs:
HLU Number ALU Number
---------- ----------
1 25
0 24
3 23
2 22
Shareable: YES
Note: The above output confirm that the above EMC CLARiiON luns are no longer presented to either server.
How to refresh the VxVM/DMP database and /etc/vx/disk.info file contents
Login to the FileStore "master" server as the "master" user and issue "storage scanbus" to update VxVM and DMP.
filestore1> storage scanbus
100% [#] Scanning the bus for disks
filestore1> storage disk list detail
Disk Pool Enclosure Size (Use%) ID Serial Number
==== ============================= ========= =========== == ==================================
ams_wms0_106 pool1 ams_wms0 20.0G 50.5% HITACHI:DF600F%5F75050205%5F006A:4:5 006A
ams_wms0_107 pool1 ams_wms0 20.0G 5.4% HITACHI:DF600F%5F75050205%5F006B:4:6 006B
ams_wms0_131 pool2 ams_wms0 20.0G 5.4% HITACHI:DF600F%5F75050205%5F0083:4:1 0083
ams_wms0_190 pool2 ams_wms0 20.0G 0.4% HITACHI:DF600F%5F75050205%5F00BE:4:2 00BE
ams_wms0_191 - ams_wms0 20.0G 0.0% HITACHI:DF600F%5F75050205%5F00BF:4:3 00BF
ams_wms0_192 - ams_wms0 20.0G 0.0% HITACHI:DF600F%5F75050205%5F00C0:4:4 00C0
ams_wms0_81 - ams_wms0 20.0G 0.0% HITACHI:DF600F%5F75050205%5F0051:4:0 0051
filestore1>
Master Server:
The /etc/vx/disk.info needs to be refreshed whilst signed in as the "support" user
# grep "0xff" /etc/vx/disk.info
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100CC551D5D0D0DDF11 sdn 0xffffffffffffffff 0x2 emc_clariion0_25 EMC_CLARiiON CK200083201067
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100CA551D5D0D0DDF11 sdy 0xffffffffffffffff 0x2 emc_clariion0_23 EMC_CLARiiON CK200083201067
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100C9551D5D0D0DDF11 sdx 0xffffffffffffffff 0x2 emc_clariion0_22 EMC_CLARiiON CK200083201067
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100CB551D5D0D0DDF11 sdk 0xffffffffffffffff 0x2 emc_clariion0_24 EMC_CLARiiON CK200083201067
# vxddladm assign names
# grep "0xff" /etc/vx/disk.info
Repeat the process on the slave server as performed on the master whilst signed in as the "support" user.
Slave Server:
# grep "0xff" /etc/vx/disk.info
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100C9551D5D0D0DDF11 sdm 0xffffffffffffffff 0x2 emc_clariion0_22 EMC_CLARiiON CK200083201067
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100CC551D5D0D0DDF11 sdl 0xffffffffffffffff 0x2 emc_clariion0_25 EMC_CLARiiON CK200083201067
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100CB551D5D0D0DDF11 sdk 0xffffffffffffffff 0x2 emc_clariion0_24 EMC_CLARiiON CK200083201067
DGC%5FRAID%205%5FCK200083201067%5F60060160CC702100CA551D5D0D0DDF11 sdn 0xffffffffffffffff 0x2 emc_clariion0_23 EMC_CLARiiON CK200083201067
# vxddladm assign names
# grep "0xff" /etc/vx/disk.info
Ensure the O/S no longer references the EMC CLARiiON removed luns.
Master Server:
filestore1_02:~ # lsscsi
[0:0:0:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sda
[0:0:1:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sdb
[1:0:4:0] disk HITACHI DF600F 0000 /dev/sds
[1:0:4:1] disk HITACHI DF600F 0000 /dev/sdt
[1:0:4:2] disk HITACHI DF600F 0000 /dev/sdu
[1:0:4:3] disk HITACHI DF600F 0000 /dev/sdv
[1:0:4:4] disk HITACHI DF600F 0000 /dev/sdw
[1:0:4:5] disk HITACHI DF600F 0000 /dev/sdx
[1:0:4:6] disk HITACHI DF600F 0000 /dev/sdy
[1:0:5:0] disk EMC SYMMETRIX 5773 /dev/sdz
[1:0:6:0] disk EMC SYMMETRIX 5773 /dev/sdaa
[1:0:7:0] disk EMC SYMMETRIX 5773 /dev/sdab
[1:0:8:0] disk EMC SYMMETRIX 5773 /dev/sdac
Slave Server:
filestore1_01:~ # lsscsi
[0:0:0:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sda
[0:0:1:0] disk ATA WDC WD1600YS-18S 6C07 /dev/sdb
[5:0:4:0] disk HITACHI DF600F 0000 /dev/sds
[5:0:4:1] disk HITACHI DF600F 0000 /dev/sdt
[5:0:4:2] disk HITACHI DF600F 0000 /dev/sdu
[5:0:4:3] disk HITACHI DF600F 0000 /dev/sdv
[5:0:4:4] disk HITACHI DF600F 0000 /dev/sdw
[5:0:4:5] disk HITACHI DF600F 0000 /dev/sdx
[5:0:4:6] disk HITACHI DF600F 0000 /dev/sdy
Process complete.
Issue/Introduction
