Description

This document attempts to explain the steps required to failback the EMC SRDF role following a SRDF failover event when using Veritas Volume Manager (VxVM) and DMP.

EMC SRDF logically pairs a device or a group of devices from each array and replicates data from one to the other synchronously or asynchronously. An established pair of devices can be split, so that separate hosts can access the same data independently (maybe for backup), and then resynchronized.

In this instance, the configuration consists of "4" EMC SRDF "R1" devices presented to production server "Dopey" from EMC Symmetrix id (SID) "822".
An additional "4" EMC SRDF "R2" devices have been presented to DR server "Bashful" from EMC Symmetrix ID (SID) "414".


During the SRDF failover operation the R1 devices are write-disabled, the SRDF link is set to Not-Ready (NR), then the R2 devices are write-enabled.

During the SRDF failback operation the R2 devices are write-disabled, the SRDF link is suspended, the disk tracks are then merged, the SRDF link is resumed, then the R1 devices are write-enabled.

In synchronous mode (SRDF/S), the primary array waits until the secondary array has acknowledged each write before the next write is accepted, ensuring that the replicated copy of the data is always as current as the primary. However, the latency due to propagation increases significantly with distance.

In asynchronous mode (SRDF/A) transfers the changes to the secondary array in units called delta sets, which are transferred at defined intervals. Although the remote copy of the data will never be as current as the primary copy, this method can replicate data over considerable distances and with reduced bandwidth requirements and mimimal impact on host performance.


Figure 1.0

Note: As of 5.0 MP3 onwards, the VxVM CLI command "vxdisk -e list" is capable of reporting the extended attribute based content for EMC BCV, BCV-NR, MIRROR, SRDF-R1, SRDF-R2, TDEV device states and EMC lun types.


Example


In this instance, the local EMC SYMDEVs are 008C, 008D, 008E and 008F. The remote EMC SYMDEVs are 0072, 0073, 0074 and 0075


The local VxVM Diskgroup named "DopeyDg", consists of four disks replicated to the remote host "Bashful".


1.] The VxVM disks at the local site (Dopey in this instance) are write-disabled.


Local:

Dopey # vxdisk -eo alldgs list | grep DopeyDg
emc0_008c    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E580d266s2 srdf-r1
emc0_008d    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E5A0d223s2 srdf-r1
emc0_008e    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E5A0d229s2 srdf-r1
emc0_008f    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E580d269s2 srdf-r1



2.] The VxVM disks at the remote site (Bashful in this instance) are write-enabled at this time, following the recent SRDF "failover" event.


Remote:


Bashful # vxdisk -eo alldgs list | grep -w DopeyDg
emc1_0072    auto:cdsdisk   r1-008C      DopeyDg     online               c1t5006048C536979A0d65s2 srdf-r2
emc1_0073    auto:cdsdisk   r1-008D      DopeyDg     online               c1t5006048C536979A0d66s2 srdf-r2
emc1_0074    auto:cdsdisk   r1-008E      DopeyDg     online               c1t5006048C536979A0d67s2 srdf-r2
emc1_0075    auto:cdsdisk   r1-008f      DopeyDg     online               c1t5006048C536979A0d68s2 srdf-r2


3.] Prior to performing the SRDF failback operation, umount the related filesystem(s) and deport the VxVM Diskgroup at the remote site, in this instance the remote server is "Dopey".


Note: Prior to the failover attempt, the filesystem at the local site contained the files, "host" and "vfstab".

Dopey # ls /vol01
hosts       lost+found  vfstab


In this instance to validate the resync of the R2 disk tracks back to the R1 devices, an additional file is created on the remote server.


Remote:


Bashful # cp /etc/nsswitch.conf /vol01

Bashful # ls /vol01
hosts          lost+found     nsswitch.conf  vfstab


Bashful # umount /vol01

Bashful # vxdg deport DopeyDg

Bashful # vxdisk -eo alldgs list | grep -w DopeyDg
emc1_0072    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d65s2 srdf-r2
emc1_0073    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d66s2 srdf-r2
emc1_0074    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d67s2 srdf-r2
emc1_0075    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d68s2 srdf-r2


4.] The current SRDF replicated state is shown as "Failed Over" at the remote site.


Remote:


Bashful # symrdf -g DopeyDg query


Device Group (DG) Name             : DopeyDg
DG's Type                          : RDF2
DG's Symmetrix ID                  : 000290301414    (Microcode Version: 5773)
Remote Symmetrix ID                : 000290300822    (Microcode Version: 5773)
RDF (RA) Group Number              :  11 (0A)


       Target (R2) View                 Source (R1) View     MODES
--------------------------------    ------------------------ ----- ------------
             ST                  LI      ST
Standard      A                   N       A
Logical       T  R1 Inv   R2 Inv  K       T  R1 Inv   R2 Inv       RDF Pair
Device  Dev   E  Tracks   Tracks  S Dev   E  Tracks   Tracks MDA   STATE
-------------------------------- -- ------------------------ ----- ------------

DEV072  0072 RW     440        0 NR 008C WD       0        0 S..   Failed Over
DEV073  0073 RW     435        0 NR 008D WD       0        0 S..   Failed Over
DEV074  0074 RW      61        0 NR 008E WD       0        0 S..   Failed Over
DEV075  0075 RW     435        0 NR 008F WD       0        0 S..   Failed Over

Total          -------- --------           -------- --------
  Track(s)         1371        0                  0        0
  MB(s)            85.7      0.0                0.0      0.0

Legend for MODES:

 M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
 D(omino)           : X = Enabled, . = Disabled
 A(daptive Copy)    : D = Disk Mode, W = WP Mode, . = ACp off


5.] To failback the SRDF roles, use "symrdf failback"


Remote:


Bashful # symrdf -g DopeyDg failback

Execute an RDF 'Failback' operation for device group 'DopeyDg' (y/[n]) ? y

An RDF 'Failback' operation execution is
in progress for device group 'DopeyDg'. Please wait...

    Write Disable device(s) on RA at target (R2)..............Done.
    Suspend RDF link(s).......................................Done.
    Merge device track tables between source and target.......Started.
    Devices: 008C-008F in (1414,011)......................... Merged.
    Merge device track tables between source and target.......Done.
    Resume RDF link(s)........................................Started.
    Resume RDF link(s)........................................Done.
    Read/Write Enable device(s) on SA at source (R1)..........Done.

The RDF 'Failback' operation successfully executed for device group 'DopeyDg'.


After some time, the replicated SRDF state will be shown as "Synchronized".


Bashful # symrdf -g DopeyDg query


Device Group (DG) Name             : DopeyDg
DG's Type                          : RDF2
DG's Symmetrix ID                  : 000290301414    (Microcode Version: 5773)
Remote Symmetrix ID                : 000290300822    (Microcode Version: 5773)
RDF (RA) Group Number              :  11 (0A)


       Target (R2) View                 Source (R1) View     MODES
--------------------------------    ------------------------ ----- ------------
             ST                  LI      ST
Standard      A                   N       A
Logical       T  R1 Inv   R2 Inv  K       T  R1 Inv   R2 Inv       RDF Pair
Device  Dev   E  Tracks   Tracks  S Dev   E  Tracks   Tracks MDA   STATE
-------------------------------- -- ------------------------ ----- ------------

DEV072  0072 WD       0        0 RW 008C RW       0        0 S..   Synchronized
DEV073  0073 WD       0        0 RW 008D RW       0        0 S..   Synchronized
DEV074  0074 WD       0        0 RW 008E RW       0        0 S..   Synchronized
DEV075  0075 WD       0        0 RW 008F RW       0        0 S..   Synchronized

Total          -------- --------           -------- --------
  Track(s)            0        0                  0        0
  MB(s)             0.0      0.0                0.0      0.0

Legend for MODES:

 M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
 D(omino)           : X = Enabled, . = Disabled
 A(daptive Copy)    : D = Disk Mode, W = WP Mode, . = ACp off


Note: The R2 devices have now been write-disabled. The R1 devices are now write-enabled.



6.] Refresh the VxVM content on both servers.


Remote:

Bashful # vxdisk scandisks

Bashful # vxdisk -eo alldgs list | grep -w DopeyDg
emc1_0072    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d65s2 srdf-r2
emc1_0073    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d66s2 srdf-r2
emc1_0074    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d67s2 srdf-r2
emc1_0075    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C536979A0d68s2 srdf-r2


Local:


Dopey # vxdisk scandisks

Dopey # vxdisk -eo alldgs list | grep DopeyDg
emc0_008c    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E580d266s2 srdf-r1
emc0_008d    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E5A0d223s2 srdf-r1
emc0_008e    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E5A0d229s2 srdf-r1
emc0_008f    auto:cdsdisk   -            (DopeyDg)   online               c1t5006048C5368E580d269s2 srdf-r1


7.] Import the VxVM DiskGroup on the local server.


Local:


a.] Import the diskgroup


Dopey # vxdg import DopeyDg

Dopey # vxprint -qhtg DopeyDg
dg DopeyDg      default      default  36000    1298306534.57.Dopey
dm r1-008C      emc0_008c    auto     65536    4058368  -
dm r1-008D      emc0_008d    auto     65536    4058368  -
dm r1-008E      emc0_008e    auto     65536    4058368  -
dm r1-008f      emc0_008f    auto     65536    4058368  -

v  vol01        -            DISABLED ACTIVE   2097152  SELECT    -        fsgen
pl vol01-01     vol01        DISABLED ACTIVE   2097152  CONCAT    -        RW
sd r1-008C-01   vol01-01     r1-008C  0        2097152  0         emc0_008c ENA


b.] Recover the VxVM in the DiskGroup "DopeyDg"


Dopey # vxrecover -g DopeyDg -bs


c.] Mount the filesystem for the volume "vol01" in "DopeyDg".


Dopey # mount -F vxfs /dev/vx/dsk/DopeyDg/vol01 /vol01


d.] Validate that the dummy (test) file created on the remote server, is visible on the local server.


Dopey # ls /vol01
hosts          lost+found     nsswitch.conf  vfstab

Process complete.