Description

I/O fencing is a feature that prevents data corruption in the event of a communication failure in a Veritas Cluster Server (VCS)/Clustered File System (CFS) cluster. 
I/O fencing that uses coordinator disks is referred to as disk-based I/O fencing.

Veritas supports non-SCSI3 server-based I/O fencing configurations in Virtual environments with the deployment of CP servers.

- InfoScale does not support the use of the SCSI3-PGR-based functionality for VMDKs devices, as the SCSI-3 PGR is not supported by VMware

- SCSI-3 Persistent reservations (PR) and I/O fencing support is only supported and restricted to block storage that is allocated using raw device mapping (RDM) devices in physical mode (RDM-P)

- I/O fencing is not supported with RDM logical mode (RDM-L) or with VMDK-based virtual disks

Non-SCSI3 fencing uses CP servers as coordination points with some additional configuration changes to support I/O fencing in virtual environments.

Table 1.0
 

To configure Non-SCSI3 Fencing (no disk-based or CP servers configured) Virtual Environments:

IO fencing and Virtual Machines should be configured as below;

For CPS environments

1.] Update the /etc/vxenviron file and set "data_disk_fencing=off"

# cat /etc/vxenviron
data_disk_fencing=off

Optional, following the next reboot, the following will be automated during the system startup:

2.] Disable scsi3pr

# vxdctl scsi3pr off

3.] Verify scsi3pr has been disabled at the VxVM level:

# vxdctl scsi3pr
scsi3pr: off

 

NOTE: Set "data_disk_fencing=off" when CP servers are configured and no SCSI-3 coordinator disk-based devices are configured .


If CP servers are deployed, the data_disk_fencing must be set to "off".

If the "/etc/vxenviron" file does not exist, the VxVM "vxdctl scsi3pr" setting will be enabled by default.


1.] On each node, edit the /etc/vxenviron file as follows:
data_disk_fencing=off

By design, SCSI3 reservations are disabled by scsi3pr functionality at system startup when the /etc/vxenviron file is defined as follows:
data_disk_fencing=off

# cat /etc/vxenviron
data_disk_fencing=off

The system startup file "/sbin/vxfen-startup" checks the contents of the /etc/vxenviron file, and if "data_disk_fencing=off", vxdctl scsi3pr functionality is turned off.

# grep -i scsi3pr /sbin/vxfen-startup
                        vm_setting=`${VXDCTL} scsi3pr | ${SED} 's/.*: //'`
                                ${VXDCTL} scsi3pr off

File: /sbin/vxfen-startup

#
#       Non-SCSI3 environment fencing
#       If non-scsi3 fencing has to be configured, disable scsi3 pr done by VM.
#
if [ -f /etc/vxenviron ]; then
        # Source the file to get the value of data_disk_fencing
        . /etc/vxenviron

        if [ "${data_disk_fencing}" = "off" ]; then
                # data disk fencing is off

                # set LLT up to panic if timers are stuck. Needed since
                # we do not have SCSI3 PR protection
                /sbin/lltconfig -p on

                # ensure that VM doesn't use SCSI3 PR
                if [ -x ${VXDCTL} ]; then
                        vm_setting=`${VXDCTL} scsi3pr | ${SED} 's/.*: //'`
                        if [ "${vm_setting}" = "on" ]; then
                                ${VXDCTL} scsi3pr off
                        fi
                fi
        fi
fi

I/O Fencing consists of two components, Membership Arbitration and Data Protection

Membership Arbitration allows only one of the multiple partitions of a cluster to continue operation in case of a network partition. 

The I/O fencing module uses coordination points such as SCSI3 compliant disks or Coordination Point Servers (CP Servers) for membership arbitration. 
At the time of a network partition, each partition races for the coordination points and the partition that grabs the majority of coordination points survives, whereas nodes from all other partitions panic.

Data Protection allows write access only for members of the cluster that survive after arbitration. 

It blocks non-members from accessing storage so that even a node that is accidently alive is unable to cause damage to data. 

Traditionally, I/O fencing uses SCSI3 Persistent Reservation (SCSI3-PR) to ensure that I/O operations from the losing node cannot reach a disk that the surviving partition has taken over

NOTE: I/O fencing utilizes HBA World Wide Numbers (WWNs) to create registrations on the storage; this has implications in a virtual environment where the HBA is shared between virtual servers on the same physical ESX host as the WWN used for I/O fencing ends up being the same for each virtual machine. 

Cluster File System (CFS) virtual machines (in the same CFS cluster) cannot share physical servers as the I/O fencing behavior will result in all nodes from that physical ESX host being fenced out if an event triggers the fencing functionality. 

In short, if I/O fencing is configured, the CFS nodes (in the same CFS cluster) has to be running on separate physical ESX hosts. NPIV is currently under evaluation for I/O fencing testing when using multiple Virtual Machines on the same physical ESX host.