V-5-1-11092: cleanup_client: (Memory allocation failure) 12
Article ID: 100047298
Updated On:
Description
Error Message
VxVM vxconfigd ERROR V-5-1-11092 cleanup_client: (Memory allocation failure) 12
VxVM vxconfigd ERRORV-5-1-11467 kernel_fail_join(): Reconfiguration interrupted: Reason is retry to add a node failed (13, 0)
The number of entries we are getting here is 13, but ideally we should be getting 11.
AIX: errpt error message
Detail Data
DESCRIPTION
NOTE VxVM vxio V-5-3-1250 joinsio_done: Node aborting, join for node 0 being failed
---------------------------------------------------------------------------
Detail Data
DESCRIPTION
NOTE VxVM vxio V-5-3-1251 joinsio_done: Overlapping reconfiguration, failing the join for node 0. The join will be retried.
Cause
VVR tunables vol_vvr_tcp_keepalive and vol_vvr_tcp_timeout were introduced in InfoScale 7.4.1.x for Linux only.
These tunables are not supported for Solaris and AIX platform.
From the vxconfigd Level 9 debug log, we see an error being returned from the CVM Master node when the CVM Slave node attempts to join. This returns a key while sl_state of 3.
02/20 11:22:27: VxVM vxconfigd DEBUG V-5-1-9450 TIME_JOIN 7a2636: slave_response begin do_next 2
02/20 11:22:27: VxVM vxconfigd DEBUG V-5-1-9451 TIME_JOIN 7a2636: slave_response end do_next 2
02/20 11:22:27: VxVM vxconfigd DEBUG V-5-1-9452 TIME_JOIN 7a2636: start of slave_response 3 <<<< sl_state = 3
02/20 11:22:27: VxVM vxconfigd ERROR V-5-1-11092 cleanup_client: (Memory allocation failure) 12
02/20 11:22:27: VxVM vxconfigd ERROR V-5-1-11467 kernel_fail_join() : Reconfiguration interrupted: Reason is retry to add a node failed (13, 0)
The "sl_state 3 response" means the CVM Slave node is attempting to obtain tunable information from the CVM Master node. This errors hence why the CVM slave join is aborted.
vxconfigd log snippets
//
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-900 REQUEST SLAVE_TUNABLEINFO: client 19414, pid 0, portal MASTER, size 0
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27876 entering req_slave_tunableinfo
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27886 TIME_JOIN 7a263a: req_slave_tunableinfo start
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27889 send_tunableinfo: Enter.
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27892 send_tunableinfo: num_entries = 13.
.
.
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27901 vol_get_tunableinfo: Unable to find tunable : vol_vvr_tcp_timeout
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27891 send_tunableinfo: failed to get tunable values.
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-5892 REQUEST FAILURE: client 19414: Memory allocation failure
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27893 send_tunableinfo: success.
02/21 17:38:35: VxVM vxconfigd DEBUG V-5-1-27887 send_tunableinfo failed
//
Due to a bug in code, those two tunables (vol_vvr_tcp_keepalive and vol_vvr_tcp_timeout) should only be exposed to Linux platforms. CVM could not reference the two tunables from the CVM Master (AIX) node, hence the issue.
Resolution
Code change has been made to hide vol_vvr_tcp_keepalive and vol_vvr_tcp_timeout for other platforms like Solaris and AIX.
Veritas engineering has released a VxVM Private Hot-fix for AIX (7.4.1.1402). Please contact Veritas Technical Support if you require this Private Hot-fix and mention Etracks:
INCIDENT NO:3995947 TRACKING ID:3995946
NOTE: Linux environments are not impacted by this issue.
Issue/Introduction
Memory allocation failure:
02/20 08:33:33: VxVM vxconfigd NOTICE V-5-1-8062 master: not a cluster startup
02/20 08:35:14: VxVM vxconfigd NOTICE V-5-1-18822 CVM_VOLD_STOP command received
02/20 08:38:10: VxVM vxconfigd NOTICE V-5-1-18815 CVM_VOLD_CONFIG command received
02/20 08:38:10: VxVM vxconfigd NOTICE V-5-1-18813 CVM_VOLD_CHANGE command received
02/20 08:38:17: VxVM vxconfigd ERROR V-5-1-11092 cleanup_client: (Memory allocation failure) 12
02/20 08:38:17: VxVM vxconfigd ERROR V-5-1-11467 kernel_fail_join() : Reconfiguration interrupted: Reason is retry to add a node failed (13, 0)