Description

Configuration Backup-Restore overview for FSS (Flexible Storage Sharing) disk groups

The below functionality has been provided for FSS disk groups via VxVM patch 6.1.0.100 onwards.
 

Introduction/challenges of CBR for FSS(Flexible Storage Sharing)

In a FSS environment, the shared storage is allocated from internal devices present on multiple servers.

The primary function of the "vxconfigbackupd" daemon is to automatically archive the configuration details for all imported "enabled" VxVM managed disk groups. When a configuration change is detected during the current active cycle of the vxconfigbackupd process, it calls the "vxconfigbackup" script to capture the revised structural changes for the specific disk group. The disk group details are stored under the parent directory /etc/vx/cbr/bk. A sub-directory is created under the parent directory for each disk group with the naming convention of ..

A configuration change may not be stored instantly to the respective CBR dg directory, as the VxVM disk group configuration change may have occurred following the lapsed SLEEP timer interval for the vxconfigbackupd process.

The default interval for the vxconfigbackup daemon is 3600 seconds (1hour). The diskgroup configuration update is detected by comparing the current and archived dg config seqno values.
 
The vxconfigbackup utility can be used to backup up the configuration information for one or more disk groups which are imported.
The disk groups may be specified either by name or by ID.

If no disk groups are specified, all disk group configurations are backed up.

The vxconfigbackup command is provided to back up the disk group configuration structure manually at any other desired time, if the configuration has changed since the last update.

# vxconfigbackup

The following disk group related content is captured by vxconfigbackup utility script:

FILES:


/etc/vx/cbr/bk/dgname.dgid/dgid.dginfo                                                                  Default location of backup file for disk group information.
Comparison # vxdg list

/etc/vx/cbr/bk/dgname.dgid/dgid.diskinfo                                                               Default location of backup file for disk attributes.
Comparison # vxdisk list for every disk related to the diskgroup

/etc/vx/cbr/bk/dgname.dgid/dgid.binconfig                                                            Default location of backup file for binary configuration copy.

/etc/vx/cbr/bk/dgname.dgid/dgid.cfgrec                                                                  Default location of backup file for configuration records in vxprint -m format.
Comparison # vxprint -mg


Traditionally the "vxconfigbackupd" daemon triggers a disk group configuration backup whenever there is a configuration change. In a clustered environment, the backup will be triggered on all clustered nodes.

Whenever the user wants to restore the disk group configuration, one can simply use the "vxconfigrestore" utility from any node in the cluster. This will initially import the disk group in private mode on that particular node. The user can then deport it and re-import the same disk group in shared mode (vxdg -s import ).

This procedure will not work for FSS disk groups due to the following product design statements:

1. FSS disk groups cannot be imported as private disk groups.
2. Asymmetric connectivity of disks.

In order to make configuration backup-restore process work for FSS disk groups, we have made the configuration backup-restore process distributed.

The disk group configuration will be saved using vxconfigbackup on all cluster nodes. The "vxconfigbackupd" daemon must be running to trigger a backup on all cluster nodes in case of a disk group configuration change.

While restoring an FSS disk group the configuration copies will be used from all the cluster nodes. The user needs to restore the configuration on all slave nodes 1st which will create remote disks by exporting locally connected disks and lastly on the master.

The master server will actually import the disk group because all disks belonging to the disk group will be available on the master at the time of restoration (either local or remote).
 

Why vxconfigbackup failed for FSS disk groups

The  vxconfigbackup fails for a FSS disk group with the existing script because we perform a SCSI enquiry on disks and for remote disks, the SCSI enquiry will fail and hence the backup fails.

To make it work, we are not considering remote disks of the FSS disk group while backing up the disk group configuration.  The user needs to obtain vxconfigbackups across all nodes contributing storage to the FSS disk group.
   
While restoring the disk group configuration, the process will export the disks which were earlier exported as per *.diskinfo file held in the CBR backup directory. The FSS disk group being recovered will be imported in shared (read-only) mode. The read-only mode is for providing provision to user to verify whether the restored configuration is the intended one.
   
After verifying the configuration, user can commit the configuration.
 
As we know, FSS disk group can be configured in many ways as follows considering storage connectivity from master node point of view.

1.       Master contributing DAS (Directly attached storage) disks to disk group
2.       Master not contributing DAS disks to disk group but disk-group has SAN disks (completely or partially shared between cluster nodes) connected to master
3.       Master do not have any DAS or partially shared disk
 
In the 1st and 2nd scenarios, there will be a binconfig file generated on the master server as there are some disks connected to the master. 
In the 3rd scenario, there will be no binconfig file.

Therefore in the 1st and 2nd scenarios, CBR will work as designed.

If the disk group is configured as in the 3rd scenario, before starting the restoration process the user must switch the CVM master role to the server having DAS or a partially shared disk and then follow the restoration process as described below.


How to backup the diskgroup configuration

To trigger a disk group configuration backup on all cluster nodes which has connectivity to at least one disk in the disk group, type:
     
    #   /etc/vx/bin/vxconfigbackup –T


How to restore the configuration for FSS DG
 

•     Identify the master node using

      #  vxclustadm nidmap
   
     Check if master node has connectivity to at least one disk in disk group(scenario 1 and 2 as per above).
      The disk can be DAS disk, partially shared disk or completely shared disk. This can be confirmed by checking dginfo or diskinfo file in backup directory.

•     If master do not have connectivity to any disk in disk group (scenario 3 as per above), switch master to node that have connectivity to at least one DAS or partially shared disk.

      #  vxclustadm setmaster

•     Restore the configuration on slave nodes (Need to run on all slave nodes which has connectivity to at least one disk in disk group)

      #  vxconfigrestore

•     Restore the configuration on master node

      #  vxconfigrestore

•     Verify the configuration

      #  vxprint –g

•     If configuration restored looks fine, then commit the configuration.

      #  vxconfigrestore –c


Aborting(Decommit) the configuration restoration
 

•     Identify the master node using

      # vxclustadm nidmap

•     Decommit/abort the configuration on master node

      # vxconfigrestore -d

•     Decommit/abort the configuration on all slave nodes(Need to run on all slave nodes which has connectivity to at least one disk in the disk group or Need to run on all slave nodes from which precommit was triggered)

      # vxconfigrestore -d
 

Manual procedure of configuration restoration

If you are able to find the *.cfgrec file in the CBR backup directory created by vxconfigbackupd (default is /etc/vx/cbr/bk/./).

Manual way of CBR (using vxmake/vxprint) uses this cfgrec file for creation of volumes and all as follows.

# cat <*.cfgrec> | vxprint -D - -hvpsm > vol.make
# vxmake -g -d vol.make
# vxvol -g start ...
# vxvol -g start ...
# vxrecover -g -bsE

Known Issues:

1.] Volume layouts stripe-mirror and concat-mirror layouts are not recommended in a FSS environment as a node leave and join event cause loss of storage and can result in sub-volumes requiring a full resynchronization.

Workaround: Use a non stripe-mirror or non concat-mirror layout, such as a mirrored volume, which is the default in FSS.

2.]The vxconfigrestore utility is unable to restore FSS cache objects in the pre-commit stage (Etrack 3461928)

While restoring a Flexible Storage Sharing (FSS) disk group configuration that has cache objects configured, the following error messages may be displayed during the pre-commit phase of the restoration process:

VxVM vxcache ERROR V-5-1-10128 Cache object meta-data update error
VxVM vxcache ERROR V-5-1-10128 Cache object meta-data update error
VxVM vxvol WARNING V-5-1-10364 Could not start cache object
VxVM vxvol ERROR V-5-1-11802 Volume volume_name cannot be started
VxVM vxvol ERROR V-5-1-13386 Cache object on which Volume volume_name is constructed is not enabled
VxVM vxvol ERROR V-5-1-13386 Cache object on which Volume volume_name is constructed is not enabled

The error messages are harmless and do not have any impact on restoration.

After committing the disk group configuration, the cache object and the volume that is constructed on the cache object are enabled.