Node fails to join CVM cluster after panic and reboot.
Article ID: 100020604
Updated On:
Description
Error Message
2009/03/30 05:13:48 VCS ERROR V-16-10001-1005 (ssuml) CVMCluster:???:monitor:node - state: out of clusterreason: License has expired or is not available for operation: retry to add a node failed
Resolution
When CVMmaster node in a SFCFS cluster is paniced and rebooted, it
doesn't jointhe CVM cluster. Error message misleads to license error.
Cause:
The causewas found to be one shared diskgroup in disabled mode.
SCSI orI/O error on one of the shared diskgroup disks would have caused the old
CVM masternode to disable the diskgroup in all the nodes of the cluster.
When the oldCVM Master node is paniced, one of the existing slave nodes
becomes CVMmaster, but doesn't changes the diskgroup state to enabled.
Solution:
CVM clusterreconfiguration will not be successful if any of the shared diskgroups
are indisabled state. We need to manually deport and import the disabledshared
disk groupto clear the disabled state.
We need tobring down the applications that are using the filesystem's from thedisabled
shareddiskgroup before deport it.
If all theapplications are controlled by VCS, use the below procedure to fix theissue.
Elsemanually offline the application, unmount the filesystem's on all thenodes,
deport thedisk group and import it back.
1. Find theService Group which has the disabled shared DG.
#hares-display cvmvoldg1 | grep Group <=== Use the appropriate resourcename
cvmvoldg1Group global cfs <== Group name used here is cfs
cvmvoldg1CVMDiskGroup global dg1 <== Shared DG name used is dg1
2. Check ifany dependency groups for this SG, and if any, offline them as well.
#hagrp-dep cfs
#ParentChild Relationship
cfs cvmonline local firm <== There are no Parent SG for cfs, so no need to offlinethem.
3. Offlinethe SG on all the nodes.
# hagrp-offline cfs -any
4. Confirmthe SG is offline on all the nodes.
#hastatus -summary | grep cfs
5. Deportand import the shared DG on the current CVM Master node.
# vxdgdeport dg1
# vxdg -simport dg1 <== Import the DG as shared, this will import the DG on allthe active CVM nodes.
#vxrecover -g dg1 -sb <== To start all the volumes.
6. Confirmthat DG is in enabled state.
#vxdglist | grep dg1
dg1enabled,shared 1203387344.59.srini-v240
7. Onlinethe SG on all the nodes.
# hagrp-online cfs -any
8. Join CVMon rebooted node.
# hagrp-online cvm -any
Issue/Introduction
Node fails to join CVM cluster after panic and reboot.
Was this article helpful?
Yes
No
Powered by
