Issues can occur either on an Active/Active (A/A)array such as EMC Symmetrix, or an Active/Passive (A/P) arrays such as EMC CLARIION. A CLARIION array is used here with VERITAS Volume Manager 


1. Volume layout

A simple 2-column stripe volume, using two LUNS belonging to a CLARIION  array:

  vtestvol      -            ENABLED  ACTIVE  8382464  SELECT    testvol-01      fsgen
 pl testvol-01  testvol      ENABLED  ACTIVE  8382464  STRIPE    2/128            RW
 sdd2-01        testvol-01   d2      0        4191232  0/0      EMC_CLARiiON0_9  ENA
 sdd1-01        testvol-01   d1      0        4191232  1/0      EMC_CLARiiON0_18 ENA


2. File system

A VERITAS FileSystem (tm) file system resides on this volume:

  #df
 /testvol           (/dev/vx/dsk/clariiondg/testvol):3738854 blocks   467355 files


3.Multipathing

Each LUN has two paths; one LUN has the primary path on controller c2, other LUN  has the primary path on controller c3:

  # vxdisk list EMC_CLARiiON0_18
 Device:    EMC_CLARiiON0_18
 devicetag: EMC_CLARiiON0_18
 ...
  Multipathing information:
  numpaths:  2
  c2t50060169102041E8d1s2state=enabled  type=secondary
 c3t50060160102041E8d1s2 state=enabled  type=primary  

  #vxdisk list EMC_CLARiiON0_9
 Device:    EMC_CLARiiON0_9
 devicetag: EMC_CLARiiON0_9
 ...
  Multipathing information:
  numpaths:  2
 c2t50060169102041E8d10s2        state=enabled  type=primary
 c3t50060160102041E8d10s2        state=enabled  type=secondary


Examples of Failures

Two examples of path failures on the CLARIION array are illustrated here.

1.  Path failure due to transient failure such as a Service Processor (SP) going temporarily offline (for example, due to a reboot of the SP):
                                                                                                                                                   
An excerpt of some of the SCSI errors recorded in syslog :
 

 

 

 

 

A little later, you will see vxio errors:
 

 

After this, the file system gets disabled, and you will see VERITAS File System errors similar to those below :
 

 

 

 

 

 

2. Error messages from permanent path failure (such as due to SAN cable failure):

An excerpt of some of the SCSI errors recorded in syslog :
 

 

 

 

A little later, you will see vxio errors:
 

 

After this, the file system gets disabled, and you will see VERITAS File System errors similar to those below :
 

 

 

 

 

System Status after the Failure

In either of the above cases, the end result is that the volume is in a DISABLED state. If a file system resided on that volume, that file system is no longer accessible. The procedure to recover from such a situation is the same in both cases above.

1. Assume that path c3 fails (either a transient or a permanent failure). After the single-path failure  and subsequent sequence of errors shown in the above two cases, the volume goes into a DISABLED state :
 

 

2. The vxdisk list command  shows one LUN (EMC_CLARiiON0_18,  which had c3 as its PRIMARY path) is in a  FAILED state:
 

3. The df command will show that the file system is in an I/O error state:
 

 

Recovery Procedure

1. First, umount the file system:
 

  (If this fails, use the "-f" flag parameter to cause a force unmount of the file system; since the volume is in a DISABLED state, it is safe to use this option at this point).

2. Next, run the following command to force Volume Manager to rescan all paths:
 

a. If a transient path error had occurred, and the path is fully functional now, Volume Manager will rediscover this path and re-enable it. All functional paths will be in an ENABLED state.
 

b. If a permanent path failure had occurred, Volum eManager will correctly discover the loss of this path and will mark all non-functional paths as DISABLED.
 

3. Use the following command to reattach the failed LUN:
 

  Once this command completes, the disk should now be in an ONLINE state:
 

  However, the volume is now in a DISABLED/RECOVER state :
 

4. Start the volume. Use the "-f" option to force start this volume:
 

  The volume is now in an ENABLED/ACTIVE state:
 

5. If this was a raw volume, use appropriate application utilities to check the consistency of the data in that volume.

  If the file system resided on this volume, first check if the file system consistency check reports any errors (use the "-n" option to check for errors without committing any changes):
 

  If no errors, proceed to step 7

6. If the fsck utility reports errors, do the following:

  - First, capture a metasave of the file system.

  - After completing the metasave operation, run the fsck utility (with "-o full" flag ) on this filesystem
 

   If this fails with an error, contact VERITAS Technical Support for further assistance

7.  If the above fsck checks completed successfully, you can now mount the filesystem: