System panic by Asynchronous Monitoring Framework (AMF) driver after Veritas File System (VxFS) patch installation

Description

Error Message

From syslog:

panic[cpu23]/thread=30009cb1800: BAD TRAP: type=9 rp=2a100346a60 addr=0 mmu_fsr=0

MountAgent: trap type = 0x9 pid=3285, pc=0x0, sp=0x2a100346301, tstate=0x44e2001605, context=0x3 g1-g7: 0, 27, fc, 2a100346c3c, 23, 0, 30009cb1800

Panic stack:

===== panic thread: 0x30009cb1800 ==== CPU: 23 ====

==== panic user (LWP_SYS) thread: 0x30009cb1800 PID: 3285 on CPU: 23 affinity

CPU: 23 ====

cmd: /opt/VRTSvcs/bin/Mount/MountAgent -type Mount

[...]

amf:disable_vxfs_api+0x2d8(0x1, 0x60048e7a400, 0x30045efb6b0, 0x8f8c7, 0x708b6290, 0x1a)

amf:amf_ev_fsoff_delhash+0x5cc(0x60048e7a400, 0x1a, 0x0, 0x0, 0x300453669e5, 0x6004ca546bd)

amf:amf_s_event_free+0x250(0x60048e7a400, 0x7bf3141b, 0x2e8, 0x8f8c7, 0x708b6290, 0x1a)

amf:amf_s_event_release+0x560(0x60048e7a400, 0x1a, 0x0, 0x0, 0x300453669e5, 0x6004ca546bd)

amf:amf_event_release+0x254(0x60048e7a400, 0x0, 0x269, 0x8f8c7, 0x708b6040, 0x1a)

amf:amf_event_unreg_rid+0xad8(0x30045366980, 0x1a, 0x0, 0x0, 0x300453669e5, 0x6004ca546bd)

amf:amf_reaper_unreg_grp+0x1b0(0x30045366980, 0x300453669c0, 0x0, 0x2, 0x60031ccf3c0, 0x2a100347adc)

amf:amf_event_unreg+0x38(0x30045366980, 0x300453669c0, 0xffffffffffffffff, 0x0, 0x0, 0x0)

amf:amfioctl+0xe1c(0x40046105, 0x208520, 0x0, 0x2, 0x60031ccf3c0, 0x2a100347adc)

amf:amf_ioctl+0x7c(0x14900000000, 0x40046105, 0x208520, 0x100003, 0x6004af93f28, 0x2a100347adc)

specfs:spec_ioctl(0x6004bf90d00, 0x40046105, 0x208520, 0x100003, , 0x2a100347adc) - frame recycled

genunix:fop_ioctl+0x2c(0x6004bf90d00, 0x40046105, 0x208520, 0x100003, , 0x2a100347adc)

genunix:ioctl+0x184()

unix:syscall_trap32+0xcc()

-- switch to user thread's user stack --

Cause

The issue occurs when VxFS is unloaded when there are still some registrations with AMF for VxFS Mounts. The issue is tracked via Symantec internal incident # 3145047.

In a typical upgrade when using Common Product Installer (CPI) scripts, the entire SFHA stack is brought down, the packages are upgraded and then the entire SFHA stack is brought up. In this case there should not be any issue. This issue would happen only if someone unloads the VxFS module or if some patches are explicitly installed, while Mount Agent or CFSMount Agent still has some registrations with AMF for VxFS mounts.

During AMF_VXFS_API init, there is no explicit external hold that AMF takes on vxportal, as a result of which vxportal (and subsequently vxfs) gets uninstalled and reinstalled even if AMF has not called uninit. Unaware of the upgrade when AMF calls back into old function pointers which is no longer valid, the system panics.

Due to the above defect in AMF code, when a VxFS patch is installed, the existing vxfs module is unloaded, a new version of VxFS package is installed and the new vxfs module is reloaded. This causes inconsistency in the AMF resulting in the panic.

Resolution

The fix will be included in below upcoming VRTSamf releases/patches.

-5.1SP1RP4
-6.0.4
-6.1.0

Workaround

Either unregister Mount / CFSMount resources from AMF or stop VCS Mount / CFSMount Agent before carrying out VxFS upgrade.

Follow the below steps to unregister Mount / CFSMount resources from AMF:

(A) Steps to disable IMF prior to installing VxFS patch:

1) Find all resources of type Mount and CFSMount for which the IMF attribute is overridden. Note the value of Mode and then set Mode to 0 for all such resources:

# hares -display -attribute IMF -type

# hares -modify IMF –update Mode 0

Note:

If no resource has IMF attribute overridden then you will see following message:

VCS WARNING V-16-1-10554 No resource exists with attribute of IMF

2) Note the current Mode value at type level for both Mount and CFSMount. Then set Mode to 0 at type level for both:

# hatype -display -attribute IMF

# hatype -modify IMF –update Mode 0

3) Check if the monitor method changed to “Traditional” for both types:

# hares -display -attribute MonitorMethod -type

4) Confirm that no vxfs related registrations exist in amfstat output.

# /opt/VRTSamf/bin/amfstat -g | grep Mount

(This should show zero registrations)

(B) Upgrade VxFS patch.

(C) Steps to enable IMF after installing VxFS patch:

1) For all resources of type Mount and CFSMount for which IMF attribute has been overridden, set the Mode back to what it was before setting it to 0 in step A1.

# hares -modify IMF –update Mode

2) Set the type level Mode for Mount and CFSMount to what it was before setting it to 0 in step A2.

# hatype -modify IMF –update Mode

3) Gradually, the Mount and CFSMount resources will get reregistered with AMF. Check amfstat output to confirm.

# /opt/VRTSamf/bin/amfstat -g | grep Mount

Applies To

This issue is applicable to:

- Solaris

- VRTSvxfs / VRTSamf 5.1SP1 and above

- AMF is enabled

- Intelligent Monitoring Framework (IMF) is enabled for Mount and/or CFSMount agents

Issue/Introduction

System panic caused by VCS Mount agent in AMF code if VxFS module is reloaded while there exist VxFS mount or CFS mount related AMF registrations. Panic stack may resemble like:

==== panic thread: 0x30009cb1800 ==== CPU: 23 ====

==== panic user (LWP_SYS) thread: 0x30009cb1800  PID: 3285  on CPU: 23  affinity

CPU: 23 ====

cmd: /opt/VRTSvcs/bin/Mount/MountAgent -type Mount

[...]

<trap>amf:disable_vxfs_api+0x2d8(0x1, 0x60048e7a400, 0x30045efb6b0, 0x8f8c7, 0x708b6290, 0x1a)