Error Message

The following was observed in the /var/log/messages files and in the crashdump.

[  642.434831] VxVM vxdmp V-5-0-1990 driver version 8.0.2.1201 installed
[  646.759633] vxio: no symbol version for storageapi_associate_blkg
[  647.049727] VxVM vxio V-5-0-1990 driver version 8.0.2.1201 installed
[  647.053317] VxVM vxspec V-5-0-1990 driver version 8.0.2.1201 installed
[  654.648362] VxVM vxdmp V-5-0-34 [Info] added disk array PZDXQ0BRHIH055, datype = NVMe
[  654.648373] VxVM vxdmp V-5-3-0 dmp_decode_add_disk da_flags: a0202
..
[  654.648643] BUG: kernel NULL pointer dereference, address: 000000000000000b
[  654.655667] #PF: supervisor read access in kernel mode
[  654.660842] #PF: error_code(0x0000) - not-present page
[  654.666019] PGD 106d03067 P4D 112e0c067 PUD 10eb81067 PMD 0 
[  654.671720] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  654.676108] CPU: 10 PID: 0 Comm: swapper/10 Kdump: loaded Tainted: P           OE    --------  ---  5.14.0-284.30.1.el9_2.x86_64 #1
[  654.688019] Hardware name: HPE ProLiant DL380 Gen11/ProLiant DL380 Gen11, BIOS 1.48 10/19/2023
[  654.696695] RIP: 0010:gendmpiodone+0xc8/0x2d0 [vxdmp]
[  654.701800] Code: e8 1d 8c 02 00 48 89 ea 44 89 e7 48 89 c6 49 89 c7 e8 1c fe 02 00 41 89 c6 48 f7 85 88 00 00 00 00 00 00 80 0f 85 c4 01 00 00 <41> f6 47 0b 20 0f 85 20 01 00 00 48 8b 5b 20 31 c0 f6 c7 21 0f 95
[  654.720710] RSP: 0018:ff69e1cfc0320e00 EFLAGS: 00010046
[  654.725972] RAX: 0000000000000000 RBX: ff4fef00803ea000 RCX: ff4fef0080226800
[  654.733160] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000500
[  654.740348] RBP: ff4fef0080226800 R08: ff4fef000a88a500 R09: ff4fef006b6b8a80
[  654.747535] R10: 0000000000000000 R11: 00000001041f4000 R12: 000000000000000a
[  654.754723] R13: ff4fef000a866000 R14: 0000000000000000 R15: 0000000000000000
[  654.761910] FS:  0000000000000000(0000) GS:ff4fef1effc80000(0000) knlGS:0000000000000000
[  654.770058] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  654.775846] CR2: 000000000000000b CR3: 000000012395e003 CR4: 0000000000771ee0
[  654.783033] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  654.790221] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  654.797409] PKRU: 55555554
[  654.800134] Call Trace:
[  654.802598]  
[  654.804625]  blk_mq_end_request_batch+0xe6/0x490
[  654.809278]  ? update_load_avg+0x7e/0x730
[  654.813318]  ? perf_event_task_tick+0x58/0xa0
[  654.817707]  nvme_irq+0x7f/0x90 [nvme]
[  654.821486]  ? nvme_pci_configure_admin_queue+0x150/0x150 [nvme]
[  654.827537]  __handle_irq_event_percpu+0x3a/0x190
[  654.832277]  handle_irq_event+0x58/0xb0
[  654.836139]  handle_edge_irq+0x93/0x240
[  654.840004]  __common_interrupt+0x3e/0xa0
[  654.844043]  common_interrupt+0x7b/0xa0
[  654.847909]  
[  654.850021]  
[  654.852132]  asm_common_interrupt+0x22/0x40
[  654.856347] RIP: 0010:cpuidle_enter_state+0xd2/0x400
[  654.861349] Code: 49 89 c5 0f 1f 44 00 00 31 ff e8 09 c3 8e ff 45 84 ff 74 12 9c 58 f6 c4 02 0f 85 12 03 00 00 31 ff e8 02 d1 94 ff fb 45 85 f6 <0f> 88 15 01 00 00 49 63 d6 4c 2b 2c 24 48 8d 04 52 48 8d 04 82 49
[  654.880258] RSP: 0018:ff69e1cfc0137e80 EFLAGS: 00000202
[  654.885521] RAX: ff4fef1effcaab80 RBX: 0000000000000002 RCX: 000000000000001f
[  654.892708] RDX: 0000000000000000 RSI: 000000002298375a RDI: 0000000000000000
[  654.899896] RBP: ff4fef1effcb6a28 R08: 000000986c197aed R09: 0000000000000018
[  654.907083] R10: 0000000000000052 R11: 0000000000000043 R12: ffffffff8c6abe60
[  654.914649] R13: 000000986c197aed R14: 0000000000000002 R15: 0000000000000000
[  654.922208]  cpuidle_enter+0x29/0x40
[  654.926176]  cpuidle_idle_call+0x12c/0x1c0
[  654.930660]  do_idle+0x7b/0xe0
[  654.934095]  cpu_startup_entry+0x19/0x20
[  654.938401]  start_secondary+0x116/0x140
[  654.942706]  secondary_startup_64_no_verify+0xe5/0xeb
[  654.948145]  

Checking the contents of the /sys/module/nvme_core/parameters/multipath file, it could be seen that native multipathing is enabled for NVME devices.

# cat /sys/module/nvme_core/parameters/multipath
Y

Hence, both the native multipathing and DMP are active for the NVME devices.
 

Using the following Redhat article as a reference point:  https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_device_mapper_multipath/enabling-multipathing-on-nvme-devices_configuring-device-mapper-multipath the following steps can be used to disable native multipathing for the NVME devices:

# grubby --update-kernel=ALL --args="nvme_core.multipath=N"

# Reboot the system.

# Once the system boots up, confirm that the native multipathing is disabled.

  - cat /sys/module/nvme_core/parameters/multipath file should return 'N'
  - cat /proc/cmdline  should contain 'nvme_core.multipath=N'

With native multipathing disabled for the NVME devices, no further panics are encountered and Veritas Volume Manager(VxVM) works fine.
 

Please note that this article references sites not owned or maintained by Veritas and, as such, Veritas is not responsible for the content portrayed on such sites, including any revisions to or deletions of content or third-party software on which this article relies. User is responsible for conducting all necessary due diligence prior to following the instructions described in this article.