Error Message

VCS ERROR V-16-20006-1005 (zdzydb02) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: SCSI-3 PR operation failed: retry to add a node failed

It transpired that after checking the system log, a couple of the shared diskgroups had become disabled on the node attempting to join as a CVM slave:

05/14 20:36:21:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_033a:

 Reason: Disk write failure
05/14 20:36:21:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_036e:
 Reason: Disk write failure
05/14 20:36:21:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_040e:
 Reason: Disk write failure
05/14 20:36:25:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:36:28:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata5dg02
05/14 20:36:29:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_0226:
 Reason: Disk write failure
05/14 20:36:29:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_0252:
 Reason: Disk write failure
05/14 20:36:29:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_0302:
 Reason: Disk write failure
05/14 20:36:33:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:36:36:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata7dg01
05/14 20:36:37:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_0372:
 Reason: Disk write failure
05/14 20:36:37:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_0416:
 Reason: Disk write failure
05/14 20:36:41:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:36:43:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata8dg20
05/14 20:36:45:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_02fa:
 Reason: Disk write failure
05/14 20:36:48:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:36:51:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata2dg04
05/14 20:36:52:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_04ce:
 Reason: Disk write failure
05/14 20:36:52:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_06ae:
 Reason: Disk write failure
05/14 20:36:55:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:36:58:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata3dg02
05/14 20:37:00:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_01da:
 Reason: Disk write failure
05/14 20:37:00:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_01f6:
 Reason: Disk write failure
05/14 20:37:00:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_02ba:
 Reason: Disk write failure
05/14 20:37:03:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:37:06:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata1dg01
05/14 20:37:07:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_067e:
 Reason: Disk write failure
05/14 20:37:07:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_0146:
 Reason: Disk write failure
05/14 20:37:14:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:37:17:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata6dg02
05/14 20:37:17:  VxVM vxconfigd WARNING  V-5-1-1538 client 1 not recognized by VXVM library
05/14 20:37:18:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk ocrvotedg01
05/14 20:37:19:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_017e:
 Reason: Disk write failure
05/14 20:37:19:  VxVM vxconfigd NOTICE  V-5-1-768 Offlining config copy 1 on disk emc1_019a:
 Reason: Disk write failure
05/14 20:37:22:  VxVM vxconfigd NOTICE  V-5-1-7909 Detached disk oradata4dg32
05/14 20:38:20:  VxVM vxconfigd ERROR  V-5-1-7934 Disk group archprdg: Disabled by errors                   <<<<<< shared DG disabled

05/14 20:38:33:  VxVM vxconfigd ERROR  V-5-1-7934 Disk group datadg: Disabled by errors                   <<<<<< shared DG disabled

There had been an underlying hardware issue that rendered the paths to all the  LUNs  to be disabled (hence the disabled  diskgroup ). 
Once the disabled diskgroups had had all their paths re-enabled and a device tree rescan, the related application and filesystems had to be offlined. To remove the disabled status of the diskgroups, they had to be reimported.