"V-16-2-13040 (node01) Resource(mount_res_1): Program(/opt/VRTSvcs/bin/Mount/online) was abnormally terminated with the exit code(0x9)" is reported by the VCS Mount agent

book

Article ID: 100042660

calendar_today

Updated On:

Description

Error Message

From /var/VRTSvcs/log/engine_A.log:

2017/10/16 17:37:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource mount_res_1 (Owner: Unspecified, Group: sg1) on System node01
2017/10/16 17:37:49 VCS ERROR V-16-2-13040 (node01) Resource(mount_res_1): Program(/opt/VRTSvcs/bin/Mount/online) was abnormally terminated with the exit code(0x9).
2017/10/16 17:37:50 VCS INFO V-16-1-10298 Resource ip_res_1 (Owner: Unspecified, Group: sg1) is online on node01 (VCS initiated)
2017/10/16 17:38:48 VCS ERROR V-16-2-13040 (node01) Resource(dg_res_1): Program(/opt/VRTSvcs/bin/DiskGroup/imf_register) was abnormally terminated with the exit code(0x9).
2017/10/16 17:39:48 VCS ERROR V-16-2-13040 (node01) Resource(dg_res_1): Program(/opt/VRTSvcs/bin/DiskGroup/imf_register) was abnormally terminated with the exit code(0x9).
…
…
2017/10/17 12:24:38 VCS NOTICE V-16-1-10167 Initiating manual offline of group sg1 on system node02
2017/10/17 12:24:38 VCS ERROR V-16-2-13067 (node02) Agent is calling clean for resource(mount_res_1) because the resource became OFFLINE unexpectedly, on its own.
2017/10/17 12:24:39 VCS ERROR V-16-2-13040 (node02) Resource(mount_res_1): Program(/opt/VRTSvcs/bin/Mount/clean) was abnormally terminated with the exit code(0x9).
2017/10/17 12:24:39 VCS ERROR V-16-2-13069 (node02) Resource(mount_res_1) - clean failed.
...
...
2017/11/06 12:53:06 VCS INFO V-16-1-50135 User admin fired command: hares -offline mount_res_1  node01  from ::ffff:10.217.37.71
2017/11/06 12:53:06 VCS NOTICE V-16-1-10300 Initiating Offline of Resource mount_res_1 (Owner: Unspecified, Group: sg1) on System node01
2017/11/06 12:53:07 VCS ERROR V-16-2-13040 (node01) Resource(mount_res_1): Program(/opt/VRTSvcs/bin/Mount/offline) was abnormally terminated with the exit code(0x9).
2017/11/06 12:53:07 VCS ERROR V-16-2-13064 (node01) Agent is calling clean for resource(mount_res_1) because the resource is up even after offline completed.
2017/11/06 12:53:08 VCS ERROR V-16-2-13040 (node01) Resource(mount_res_1): Program(/opt/VRTSvcs/bin/Mount/clean) was abnormally terminated with the exit code(0x9).
2017/11/06 12:53:08 VCS ERROR V-16-2-13069 (node01) Resource(mount_res_1) - clean failed.
...
...
2018/03/05 16:33:01 VCS NOTICE V-16-1-10167 Initiating manual offline of group sg1 on system node01
2018/03/05 16:33:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource app_res_1 (Owner: Unspecified, Group: sg1) on System node01

Cause

The AUDIT rules could potentially be causing the Mount resource's online, offline, or clean processes to be killed with SIGKILL (Signal#9) (i.e., kill -9 ).

For example, the following AUDIT rule will cause the scripts to terminate:
# auditctl -a exit,always -F arch=b64 -F euid=0 -F success=0 -S execve -k rootcmd

However, if you change the rule to the following (changing the value of success=1), then the scripts should no longer be terminated:
# auditctl -a exit,always -F arch=b64 -F euid=0 -F success=1 -S execve -k rootcmd

You may also consider stopping the AUDITD process or even deleting the AUDITD rules (auditctl –D).

Resolution

  • Kill the AUDITD process to see if this resolves the issue.
  • Check the following files to see if the 'success' value is set to 0 or 1:
    /etc/audit/audit.rules
    /etc/audit/rules.d/audit.rules (and any other rules under this location)

Issue/Introduction

"V-16-2-13040 (node01) Resource(mount_res_1): Program(/opt/VRTSvcs/bin/Mount/online) was abnormally terminated with the exit code(0x9)" is reported by the VCS Mount agent.
Powered by Wolken Software