"V-5-1-11092 cleanup_client: (There are minor number conflicts on a slave node)" is reportred when a CVM slave fails to join the cluster because of minor number conflict
Article ID: 100032022
Updated On:
Description
Error Message
The following messages are logged when the CVM slave tries to join the cluster.
Feb 29 18:53:21 server102 kernel: GAB INFO V-15-1-20036 Port v[GAB_LEGACY_CLIENT (refcount 0)] gen cf0652 membership 01
Feb 29 18:53:21 server102 kernel: GAB INFO V-15-1-20038 Port v[GAB_LEGACY_CLIENT (refcount 0)] gen cf0652 k_jeopardy ; 2
Feb 29 18:53:21 server102 kernel: GAB INFO V-15-1-20040 Port v[GAB_LEGACY_CLIENT (refcount 0)] gen cf0652 visible ; 2
Feb 29 18:53:21 server102 kernel: GAB INFO V-15-1-20036 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen cf0653 membership 01
Feb 29 18:53:21 server102 kernel: GAB INFO V-15-1-20038 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen cf0653 k_jeopardy ; 2
Feb 29 18:53:21 server102 kernel: GAB INFO V-15-1-20040 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen cf0653 visible ; 2
Feb 29 18:53:21 server102 vxvm:vxconfigd: V-5-1-7900 CVM_VOLD_CONFIG command received
Feb 29 18:53:21 server102 kernel: VxVM vxio V-5-3-1906 vol_gab_ms_msg: ring broadcast commit completed for join/leave reconfig cs_flags=0x580802
Feb 29 18:53:22 server102 kernel: GAB INFO V-15-1-20036 Port m[GAB_LEGACY_CLIENT (refcount 0)] gen cf0651 membership 01
Feb 29 18:53:22 server102 kernel: GAB INFO V-15-1-20038 Port m[GAB_LEGACY_CLIENT (refcount 0)] gen cf0651 k_jeopardy ; 2
Feb 29 18:53:22 server102 kernel: GAB INFO V-15-1-20040 Port m[GAB_LEGACY_CLIENT (refcount 0)] gen cf0651 visible ; 2
Feb 29 18:53:22 server102 kernel: VxVM vxio V-5-3-2015 reconfig message on port m received cf0650
Feb 29 18:53:26 server102 kernel: GAB INFO V-15-1-20036 Port w[GAB_USER_CLIENT (refcount 0)] gen cf0655 membership 01
Feb 29 18:53:26 server102 kernel: GAB INFO V-15-1-20038 Port w[GAB_USER_CLIENT (refcount 0)] gen cf0655 k_jeopardy ; 2
Feb 29 18:53:26 server102 kernel: GAB INFO V-15-1-20040 Port w[GAB_USER_CLIENT (refcount 0)] gen cf0655 visible ; 2
Feb 29 18:53:26 server102 kernel: VxVM vxio V-5-0-1910 Cleaning incomplete shared diskgroup devldg dgiid 33792.104
Feb 29 18:53:26 server102 vxvm:vxconfigd: V-5-1-11092 cleanup_client: (There are minor number conflicts on a slave node) 231
Feb 29 18:53:26 server102 vxvm:vxconfigd: V-5-1-11467 kernel_fail_join() : Reconfiguration interrupted: Reason is retry to add a node failed (13, 0)
Feb 29 18:53:26 server102 kernel: VxVM vxio V-5-0-164 Failed to join cluster clus123, aborting
Feb 29 18:53:26 server102 kernel: VxVM vxio V-5-3-1250 joinsio_done: Node aborting, join for node 0 being failed
Feb 29 18:53:26 server102 kernel: VxVM vxio V-5-3-672 abort_joinp: aborting joinp for node 0 with err 11
Feb 29 18:53:26 server102 kernel: GAB INFO V-15-1-20032 Port y closed
Feb 29 18:53:26 server102 vxvm:vxconfigd: V-5-1-7901 CVM_VOLD_STOP command received
Feb 29 18:53:26 server102 kernel: GAB INFO V-15-1-20032 Port m closed
Feb 29 18:53:26 server102 kernel: GAB INFO V-15-1-20032 Port v closed
Feb 29 18:53:26 server102 kernel: GAB INFO V-15-1-20032 Port w closed
Feb 29 18:53:26 server102 AgentFramework[11752]: VCS ERROR V-16-20006-1005 CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: There are minor number conflicts on a slave node: retry to add a node failed
Feb 29 18:55:26 server102 AgentFramework[11752]: VCS ERROR V-16-2-13066 Thread(4132432704) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Feb 29 18:55:26 server102 Had[11726]: VCS ERROR V-16-2-13066 (server102) Agent is calling clean for resource(cvm_clus) because the resource is not up even after online completed.
Feb 29 18:55:26 server102 AgentFramework[11752]: VCS ERROR V-16-2-13068 Thread(4132432704) Resource(cvm_clus) - clean completed successfully.
Feb 29 18:55:26 server102 AgentFramework[11752]: VCS ERROR V-16-2-13072 Thread(4132432704) Resource(cvm_clus): Agent is retrying online (attempt number 1 of 2).
Cause
Figure 1 - Extract from the vxtune(1M) manual page
The following tunable parameters apply for Cluster Volume Manager (CVM): autoreminor Turns on or off the automatic reminor functionality. A disk group cannot be imported if the device minor numbers of the disk group or its objects conflict with those of an existing disk group. When autoreminor is on, VxVM automatically assigns new minor numbers to a disk group if VxVM detects a conflict during an import operation. The disk group is then imported. The default value is on. Note: VxVM does not reminor a disk group that is already imported, regardless of whether autoreminor is set to on. For example, if you attempt to add a node to a cluster and the joining node has minor numbers that conflict with a disk group in the cluster. In this case, the join operation fails. You must reminor the disk group manually. In some scenarios such as with NFS file systems, assigning new minor numbers may result in issues. In this case, set the tunable parameter to off. When the autoreminor parameter is set to off, attempting to import a disk group with conflicting minor numbers will fail, even when you specify the force (-f) option. You must manually reminor the disk group before you can import the disk group.
Volume Manager (VxVM) divides the minor numbers into two sets. One set is for the private diskgroups. The other set is for the CVM shared diskgroups. The two sets of minor numbers are divided by the following vxtune parameter:
- sharedminorstart: The starting number in the range used to assign device minor numbers in shared (CVM) disk groups. The default value is 33000.
Sometimes, when a diskgroup is first initialized as a private diskgroup (running vxdg init without the -s option) VxVM will assign a base_minor of less than 33000. Later when the diskgroup is imported as shared, VxVM will not change the minor numbers and they remain below the 33000 boundary. It may then have a chance to collide with existing private diskgroups on another nodes in the cluster.
The same issue also exists the other way round. If a diskgroup iss initialized as shard at first (vxdg -s init) but later imported as private, then this private diskgroup will have minor number higher than 33000. In such situation you may want to run vxdg remnor to move the minor numbers back to the correct set to avoid conflicts.
Apart from the above possible cause, there is also one obscure cause for the CVM slave join to fail with minor number conflict. When two diskgroups are reminored to have the same base_minor while keeping some of the existing in-kernel minor numbers not changes, the CVM slave join will fail with minor number conflict.
The following is an example on how this can happen.
First two diskgroups are created and they have different base_minor, then one diskgroup is reminored to 36000.
server101# vxdg -g proddg reminor 36000 server101# vxprint -m -g proddg | grep 'minor=[0-9]' base_minor=36000 minor=36000
The CFS filesystems are unmounted and the diskgroup is deported.
server101# hagrp -offline SGprod -sys server101 # umounted the filesystem server101# vxdg deport proddg
The other diskgroup is also reminored to the same base_minor 36000 while having the CFS filesystems mounted.
server101# vxdg -g devldg reminor 36000 VxVM vxdg WARNING V-5-1-3858 Volume devlvol01: Device is open, will renumber on import
Since the filesystem are still mounted, VxVM will not change the existing in-kernel minor number.
server101# ls -lR /dev/vx/rdsk
....
/dev/vx/rdsk/devldg:
total 0
crw-------. 1 root root 199, 28000 Feb 29 18:44 devlvol01 <<< in-kernel, the volume minor number is still the old one 28000
But the on-disk configuration has already been changed to 36000.
server101# vxprint -m -g devldg | grep 'minor=[0-9]' base_minor=36000 minor=36000
Now, the previously deported diskgroup is imported.
server101# hagrp -online SGprod -sys server101 Feb 29 18:51:59 server101 vxvm:vxconfigd: V-5-1-11401 : dg import with I/O fence enabled Feb 29 18:51:59 server101 vxvm:vxconfigd: V-5-1-11401 proddg: dg import with I/O fence enabled Feb 29 18:51:59 server101 kernel: sd 3:0:0:13: reservation conflict Feb 29 18:51:59 server101 kernel: sd 5:0:0:13: reservation conflict Feb 29 18:51:59 server101 vxvm:vxconfigd: V-5-1-16765 Selecting configuration database copy from disk_6 from disks: disk_6 Feb 29 18:52:03 server101 vxvm:vxconfigd: V-5-1-16766 Trying to import the disk group proddg using configuration database copy from disk_6 Feb 29 18:52:04 server101 vxvm:vxconfigd: V-5-1-16254 Disk group import of proddg succeeded.
From the ls -lR /dev/vx/rdsk/ output, there is no minor number conflict.
server101# ls -lR /dev/vx/rdsk .... /dev/vx/rdsk/devldg: total 0 crw-------. 1 root root 199, 28000 Feb 29 18:44 devlvol01 <<< 28000 /dev/vx/rdsk/proddg: total 0 crw-------. 1 root root 199, 36000 Feb 29 18:52 prodvol01 <<< 36000
But now the slave will not join because the on-disk base_minor are the same 36000.
server101# hagrp -online cvm -sys server102 Feb 29 18:53:26 server102 kernel: VxVM vxio V-5-0-1910 Cleaning incomplete shared diskgroup devldg dgiid 33792.104 Feb 29 18:53:26 server102 vxvm:vxconfigd: V-5-1-11092 cleanup_client: (There are minor number conflicts on a slave node) 231
Resolution
For in-kernel minor numbers, you can use the following commands:
server101# ls -lR /dev/vx/rdsk .... /dev/vx/rdsk/devldg: total 0 crw-------. 1 root root 199, 28000 Feb 29 18:44 devlvol01 /dev/vx/rdsk/proddg: total 0 crw-------. 1 root root 199, 36000 Feb 29 18:52 prodvol01
For on-disk minor numbers, use vxprint:
server101# vxprint -m -g devldg | grep 'minor=[0-9]' base_minor=36000 minor=36000 server101# vxprint -m -g proddg | grep 'minor=[0-9]' base_minor=36000 minor=36000
Run the above command for all of the diskgroups that are currently imported.
If there is any base_minor conflict, please run vxdg reminor to fix the issue.
# vxdg -g proddg reminor 37000
Choose a new minor number that is not used by any diskgroups.
To have the minor numbers change immediately, unmount the filesystems first before running vxdg reminor.
Issue/Introduction
