Error Message

The following panic stack may be observed:

crash> bt
PID: 28818  TASK: ffff885fe2dfeab0  CPU: 0   COMMAND: "vxconfigd"
 #0 [ffff880202a09a58] machine_kexec at ffffffff81040f1b
 #1 [ffff880202a09ab8] crash_kexec at ffffffff810d6722
 #2 [ffff880202a09b88] panic at ffffffff81558571
 #3 [ffff880202a09c28] __perf_event_overflow at ffffffff811309aa
 #4 [ffff880202a09ca8] perf_event_overflow at ffffffff81131004
 #5 [ffff880202a09cb8] intel_pmu_handle_irq at ffffffff81025c8c
 #6 [ffff880202a09e90] perf_event_nmi_handler at ffffffff8155e83f
 #7 [ffff880202a09ea0] notifier_call_chain at ffffffff81560350
 #8 [ffff880202a09ee0] atomic_notifier_call_chain at ffffffff815603ba
 #9 [ffff880202a09ef0] notify_die at ffffffff810b12ee
#10 [ffff880202a09f20] do_nmi at ffffffff8155dea9
#11 [ffff880202a09f50] nmi at ffffffff8155d781
    [exception RIP: _spin_lock_irq+37]
    RIP: ffffffff8155c1d5  RSP: ffff885fe30bbb88  RFLAGS: 00000002
    RAX: 0000000000000000  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: 0000000000000046  RDI: ffff885febb39018
    RBP: ffff885fe30bbb88   R8: 0000000000042123   R9: 0000000000000000
    R10: 000000000000000f  R11: 000000000000000c  R12: ffff885fe30bbb98
    R13: ffff885fe2b13508  R14: ffff885febb38d00  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- ---
#12 [ffff885fe30bbb88] _spin_lock_irq at ffffffff8155c1d5
#13 [ffff885fe30bbb90] blk_throtl_drain at ffffffff81297919
#14 [ffff885fe30bbbe0] __blk_drain_queue at ffffffff81286cd1
#15 [ffff885fe30bbc10] blk_cleanup_queue at ffffffff81286fab
#16 [ffff885fe30bbc40] vxvm_put_gendisk at ffffffffa05fe418 [vxio]
#17 [ffff885fe30bbc70] volsys_unset_device at ffffffffa05fe4c2 [vxio]
#18 [ffff885fe30bbc90] vol_rmgroup_devices at ffffffffa0645322 [vxio]
#19 [ffff885fe30bbce0] voldg_delete at ffffffffa06470aa [vxio]
#20 [ffff885fe30bbd30] vol_delete_group at ffffffffa064854b [vxio]
#21 [ffff885fe30bbd60] volconfig_ioctl at ffffffffa070828f [vxio]
#22 [ffff885fe30bbda0] volsioctl_real at ffffffffa07113d8 [vxio]
#23 [ffff885fe30bbe80] vols_ioctl at ffffffffa09c9126 [vxspec]
#24 [ffff885fe30bbea0] vols_compat_ioctl at ffffffffa09c934d [vxspec]
#25 [ffff885fe30bbed0] compat_sys_ioctl at ffffffff811fb1d8
#26 [ffff885fe30bbf80] sysenter_dispatch at ffffffff81565be7
#27 [ffff885fe30bbf88] ia32_sysenter_target at ffffffff81565b34
#28 [ffff885fe30bbf98] ia32_sysenter_target at ffffffff81565b34
    RIP: 0000000000169430  RSP: 00000000ffffd4f8  RFLAGS: 00000282
    RAX: 0000000000000036  RBX: ffffffff81565be7  RCX: 00000000564f4c78
    RDX: 00000000ffffd6b0  RSI: 000000000835e210  RDI: 00000000564f4c78
    RBP: 00000000ffffd678   R8: ffffffff81565b28   R9: ffffffff81565b34
    R10: ffffffff81565b28  R11: ffffffff81565b34  R12: ffffffffffffffff
    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 0000000000000036  CS: 0023  SS: 002b

In the event that a crashdump is not available, then the following stacks may be observed in the messages files. The stacks will differ slightly due to the presence of cvm.

Example 1:
Oct 31 03:56:54 server103 kernel: Pid: 3704, comm: vxconfigd Tainted: P -- ------------ 2.6.32-754.6.3.el6.x86_64 #1
Oct 31 03:56:54 server103 kernel: Call Trace:
Oct 31 03:56:54 server103 kernel: [] ? warn_slowpath_common+0x91/0xe0
Oct 31 03:56:54 server103 kernel: [] ? warn_slowpath_null+0x1a/0x20
Oct 31 03:56:54 server103 kernel: [] ? blk_throtl_drain+0xff/0x180
Oct 31 03:56:54 server103 kernel: [] ? __blk_drain_queue+0x91/0x140
Oct 31 03:56:54 server103 kernel: [] ? blk_cleanup_queue+0xeb/0x1d0
Oct 31 03:56:54 server103 kernel: [] ? vxvm_put_gendisk+0x68/0xf0 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? volsys_unset_device+0x22/0x40 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? vol_rmgroup_devices+0x82/0x100 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? voldg_delete+0x1a/0x130 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? vol_rv_dgdelete_prepare_group+0x30/0x60 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? vol_delete_group+0x23b/0x290 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? selinux_capable+0x46/0x60
Oct 31 03:56:54 server103 kernel: [] ? volconfig_ioctl+0x603/0x6b0 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? security_capable+0x2f/0x40
Oct 31 03:56:54 server103 kernel: [] ? capable+0x2a/0x60
Oct 31 03:56:54 server103 kernel: [] ? volsioctl_real+0x412/0x550 [vxio]
Oct 31 03:56:54 server103 kernel: [] ? file_has_perm+0xd1/0xe0
Oct 31 03:56:54 server103 kernel: [] ? vols_ioctl+0x5c/0x80 [vxspec]
Oct 31 03:56:54 server103 kernel: [] ? vols_compat_ioctl+0x3d/0x60 [vxspec]
Oct 31 03:56:54 server103 kernel: [] ? compat_sys_ioctl+0xf8/0x520
Oct 31 03:56:54 server103 kernel: [] ? sysenter_dispatch+0x20/0x4b

Example 2:
Nov  3 08:56:55 systemA kernel: WARNING: at block/blk-throttle.c:1222 blk_throtl_drain+0xff/0x180() (Tainted: P           -- ------------   )
Nov  3 08:56:55 systemA kernel: Hardware name: System x3850 X5 -[7143YRW]-
Nov  3 08:56:55 systemA kernel: Modules linked in: joydev dcdbas nfs lockd fscache auth_rpcgss nfs_acl sunrpc xfs ext2 vxodm(P)(U) vxgms(P)(U) amf(P)(U) vxglm(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) rdma_cm ib_cm ib_sa ib_mad autofs4 dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) cpufreq_ondemand acpi_cpufreq freq_table mperf 8021q garNov  3 08:56:55 systemA kernel: Pid: 27235, comm: vxiod Tainted: P           -- ------------    2.6.32-754.6.3.el6.x86_64 #1
Nov  3 08:56:55 systemA kernel: Call Trace:
Nov  3 08:56:55 systemA kernel: [] ? warn_slowpath_common+0x91/0xe0
Nov  3 08:56:55 systemA kernel: [] ? warn_slowpath_null+0x1a/0x20
Nov  3 08:56:55 systemA kernel: [] ? blk_throtl_drain+0xff/0x180
Nov  3 08:56:55 systemA kernel: [] ? __blk_drain_queue+0x91/0x140
Nov  3 08:56:55 systemA kernel: [] ? blk_cleanup_queue+0xeb/0x1d0
Nov  3 08:56:55 systemA kernel: [] ? vxvm_put_gendisk+0x61/0x150 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? volsys_unset_device+0x22/0x40 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? vol_rmgroup_devices+0x82/0x100 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? voldg_delete+0x23/0x130 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? volcvm_update_connectivity_map+0xb4/0x290 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? volcvmdg_delete_msg_receive_start+0x148/0x480 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? volsync_wait+0xb8/0xd0 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? voliod_iohandle+0x128/0x250 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? voliod_loop+0xe8/0x390 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? voliod_loop+0x0/0x390 [vxio]
Nov  3 08:56:55 systemA kernel: [] ? kthread+0xa0/0xc0
Nov  3 08:56:55 systemA kernel: [] ? child_rip+0x20/0x30
Nov  3 08:56:55 systemA kernel: [] ? kthread+0x0/0xc0
Nov  3 08:56:55 systemA kernel: [] ? child_rip+0x0/0x30
Nov  3 08:56:55 systemA kernel: ---[ end trace daf93659845586a5 ]---

The panic is due to a known Redhat issue  (1624747 bugzilla).

RedHat advised that the official fix for this panic has been included in the 2.6.32-754.9.1.el6.x86_64 kernel.

If planning to use the RHEL 6.10 (2.6.32-754.6.3 ) kernel with Storage Foundation 6.2.1 and/or the InfoScale 7.x releases, then it would be recommended that the 2.6.32-754.9.1 kernel be used instead.

The SORT website has been updated to list the 2.6.32-754.9.1 kernel as qualified for InfoScale 7.x and Storage Foundation 6.2.1.