Error Message

Aug 17 13:51:01 sbill1 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-13027 cfsdg03: import failed: Inconsitent PGR License settings
Aug 17 13:51:01 sbill1 vxvm:vxconfigd: [ID 702911 daemon.notice]
Aug 17 13:51:01 sbill1 vxvm:vxconfigd: [ID 702911 daemon.error] V-5-1-11092 cleanup_client: (License has expired or is not available for operation) 190
Aug 17 13:51:01 sbill1 vxvm:vxconfigd: [ID 702911 daemon.error] V-5-1-11467 kernel_fail_join() :                Reconfiguration interrupted: Reason is retry to add a node failed (13, 0)
Aug 17 13:51:01 sbill1 vxio: [ID 535583 kern.warning] WARNING: VxVM vxio V-5-0-164 Failed to join cluster sblueprint, aborting
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 0 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 0 with err 11
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 2 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 2 with err 11
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 3 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 3 with err 11
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 4 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 4 with err 11
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 5 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 5 with err 11
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 6 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 6 with err 11
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 7 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 7 with err 11
Aug 17 13:51:01 sbill1 vxio: [ID 876687 kern.notice] NOTICE: VxVM vxio V-5-3-0 joinsio_done: Node aborting, join for node 1 being failed
Aug 17 13:51:01 sbill1 vxio: [ID 317193 kern.notice] NOTICE: VxVM vxio V-5-3-0 abort_joinp: aborting joinp for node 1 with err 11
Aug 17 13:51:01 sbill1 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port v closed
Aug 17 13:51:01 sbill1 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-7901 CVM_VOLD_STOP command received
Aug 17 13:51:01 sbill1 gab: [ID 397130 kern.notice] GAB INFO V-15-1-20032 Port w closed
Aug 17 13:51:09 sbill1 Had[11228]: [ID 702911 daemon.notice] VCS ERROR V-16-1-1005 (sbill1) CVMCluster:???:monitor:node - state: out of cluster
Aug 17 13:51:09 sbill1 reason: License has expired or is not available for operation: retry to add a node failed
 

The problem is caused by inconsistent "fencekey" attribute in the diskgroup configuration.   

Normally the diskgroup configuration attribute "fencekey" should be set according to the actual PGR keys registered on the disks.  For example,

# vxprint -g racdg -m | grep fencekey
        fencekey=PGR0001

# vxfenadm -g /dev/vx/rdmp/emcpower73s2
Reading SCSI Registration Keys...
Device Name: /dev/vx/rdmp/emcpower73s2
Total Number Of Keys: 1
key[0]:
       Key value [Numeric Format]: 66,80,71,82,48,48,48,50
       Key Value [Character Format]: BPGR0001 

But if a diskgroup hit the problem, the diskgroup attribute fencekey will be empty or inconsistent with the on-disk PGR keys.  For example,

# vxprint -g racdg -m | grep fencekey
        fencekey=                    <<<< The fencekey attribute is missing !!!

One possible cause of the above situation is because the diskgroup is disabled.