VCS and Server unavailable due to error "VCS ERROR V-16-2-13469 Thread(4143811488) Agent(Application) exiting - failed to create process due to low system resources, errno(11) "

book

Article ID: 100025774

calendar_today

Updated On:

Description

Error Message

Following messages were seen in system logs:

VCS ERROR V-16-2-13067 Thread(4135545760) Agent is calling clean for resource(resource1) because the resource became OFFLINE unexpectedly, on its own.
VCS ERROR V-16-2-13067 (node) Agent is calling clean for resource(resource1) because the resource became OFFLINE unexpectedly, on its own.
VCS ERROR V-16-2-13068 Thread(4135545760) Resource(resource1) - clean completed successfully.
kernel: vxfs: msgcnt 1 mesg 097: V-2-97: vxfs failed to create new thread (//, 0xffffffff882e70d3:0x15)
VCS ERROR V-16-2-13469 Thread(4143811488) Agent(Application) exiting - failed to create process due to low system resources, errno(11)
vxvm:vxconfigd: V-5-1-14615 Failed to create process thread
last message repeated 898 times

Cause

Issue was due to low system resources.
 

Resolution

Reboot of the server resolved the resource crunch issue.

If the system is accessible can try to narrow down to the process, which is consuming the system resources using OS commands such as TOP, TOPAS etc.

However, if reboot is the only option, then it is advisable to collect system crash dump. This would get us to the root-cause of process that was eating up system resources. The crash dump can then be shared with OS vendor and Veritas for further diagnosis.

Issue/Introduction

VCS and all other Veritas process unable to spawn new threads. Also, server was observed to be hanging very much. In some cases, server is completely hung.