Using the Asynchronous Monitor Framework (AMF) feature in Cluster Server (VCS) on Solaris 11.2 SRU 8 and above can cause agent hangs which may result in an application or system outage

book

Article ID: 100031859

calendar_today

Updated On:

Description

Error Message

/var/VRTSvcs/log/engine_A.log

2016/01/24 16:45:00 VCS WARNING V-16-1-10023 Agent Process not sending alive messages since Sun Jan 24 16:40:1820 2016

2016/01/24 16:45:00 VCS NOTICE V-16-1-53026 Agent Process ipm connection still valid

2016/01/24 16:45:00 VCS NOTICE V-16-1-53027 Waiting one more try for ipm connection for agent Process to go away

2016/01/24 16:45:19 VCS WARNING V-16-1-10023 Agent Process not sending alive messages since Sun Jan 24 16:40:18 2016

2016/01/24 16:53:19 VCS WARNING V-16-1-53025 Agent Process has faulted; ipm connection was lost; restarting the agent

2016/01/24 16:53:19 VCS ERROR V-16-1-10008 Agent Process has faulted 6 times since Sun Jan 24 16:40:18 2016

2016/01/24 16:53:19 VCS ERROR V-16-1-10009 Agent Process has faulted 6 times in less than 950 seconds -- Will not attempt to restart. Correct the problem and use haagent -start to start the agent

Cause

The issue is due to changes in Solaris 11.2 SRU 8 kernel header which causes AMF errors during process and mount registration.

Resolution

Veritas Technologies have produced patch amf-sol11_sparc-Patch-6.2.1.100 for VCS 6.2.1 to resolve the AMF issue.  
Previously the fix was available in vcs-sol11_sparc-Patch-6.2.1.100 which was recalled.

A workaround exists by disabling AMF feature
 
1) Before you upgrade to Solaris 11.2 SRU 8 or above:
 
1a) Check whether a VRTSamf patch is available. If a patch is available, apply the patch and then upgrade to Solaris 11.2 SRU 8.
1b) If a patch is not available, disable AMF.

2) If you have already upgraded to Solaris 11.2 SRU 8 or above:
2a) Disable AMF.
2b) Check whether a VRTSamf patch is available. If a patch is available, apply the patch and then enable AMF.

How to disable and enable AMF

To disable AMF in SFHA 6.2 and below:

1. Set the AMF_STOP variable to 1 in the /etc/default/amf file.
2.  Stop the AMF kernel driver:
# svcadm disable amf

To disable AMF from SFHA 6.2.1 onwards:

1.   Set the value of the AMF_DISABLE variable to 1 in the /etc/default/amf file.
2.  Stop the AMF kernel driver:
# svcadm disable amf

To enable AMF in SFHA 6.2 and below:
 
1) Set the AMT_START variable to 1 in the /etc/default/amf file
2) Start the AMF kernel driver:
# svcadm enable amf
 
To enable AMF from SFHA 6.2.1 onwards:

1) Reset the AMF_DISABLE variable in the /etc/default/amf file.
2) Start the AMF kernel driver:
# svcadm enable amf
 

Issue/Introduction

Using the Asynchronous Monitor Framework (AMF) feature in Cluster Server (VCS) on Solaris 11.2 SRU 8 and above can cause agent hangs which may result in an application or system outage

Affected OS version: Solaris 11.2 SRU 8 and above (SPARC)
Affected VCS versions: 6.0.x, 6.1.x, 6.2.x & 7.0.x

VCS is part of Storage Foundation High Availability (SFHA) solutions, InfoScale Enterprise and InfoScale Availability

Additional Information

ETrack: 3865093
Powered by Wolken Software