Remote cluster reported in 'INIT' state for a GCO environment consisting of two single node VCS 7.4.1/rhel7 clusters

book

Article ID: 100063443

calendar_today

Updated On:

Description

Error Message

Feb 10 10:44:15 site1clus kernel: Out of memory: Kill process 5020 (IcmpAgent) score 155 or sacrifice child
Feb 10 11:09:24 site1clus kernel: Out of memory: Kill process 24420 (IcmpAgent) score 154 or sacrifice child
Feb 10 12:15:48 site1clus kernel: Out of memory: Kill process 9980 (IcmpAgent) score 214 or sacrifice child


Feb 10 10:44:15 site1clus kernel: Killed process 5020 (IcmpAgent), UID 0, total-vm:5642196kB, anon-rss:1521992kB, file-rss:2816kB, shmem-rss:4kB
Feb 10 11:09:24 site1clus kernel: Killed process 24420 (IcmpAgent), UID 0, total-vm:5603216kB, anon-rss:1352832kB, file-rss:2856kB, shmem-rss:4kB
Feb 10 12:15:48 site1clus kernel: Killed process 9980 (IcmpAgent), UID 0, total-vm:7738024kB, anon-rss:2788356kB, file-rss:2836kB, shmem-rss:4kB

 

Cause

The IcmpAgent process was getting killed time and time again due to the lack of available memory on the systems and this was resulting in the 'INIT' state. This lack of memory was also affecting other processes including those of other applications running on the systems.

 

Resolution

The systems were rebooted and this resolved the issue and the remote cluster was no longer reported in 'INIT' state. The systems were going to be monitored to determine if more memory needed to be made available to them.

 

Issue/Introduction

Remote cluster reported in 'INIT' state for a GCO environment consisting of two single node VCS 7.4.1/rhel7 clusters. # hastatus -sum -- SYSTEM STATE
-- System State Frozen
A site1clus RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B ClusterService site1clus Y N ONLINE
B test04grp site1clus Y N ONLINE
B test04replgrp site1clus Y N ONLINE
-- GROUPS FROZEN
-- Group
C test04replgrp -- RESOURCES DISABLED
-- Group Type Resource
H test04replgrp DiskGroup test04vg
H test04replgrp IP test04_IP
H test04replgrp IP test04p_IP
H test04replgrp NIC test04_NIC
H test04replgrp NIC test04p_NIC
H test04replgrp RVG test04_RVG
-- WAN HEARTBEAT STATE
-- Heartbeat To State
M Icmp site2clus UNKNOWN -- REMOTE CLUSTER STATE
-- Cluster State
N site2clus INIT