Error Message

Sample stack trace:

000002a1029f4d41 vpanic(1282bb0, 600735fec08, 0, 3000012a060, 1, 18e8400)
000002a1029f4df1 kmem_error+0x4b4(7, 378, 18e8400, 0, 7, 3000012a008)
000002a1029f4ec1 dmp_free_stats_table+0x118(701f3, 16, 702f5ce8, 201, 701f3668, 15)
000002a1029f4f71 dmp_free_modules+0x24(0, 0, 1, 702f5ba0, 701f2c90, 0)
000002a1029f5021 vxdmp`_fini+0x178(0, 701f3320, 1946c00, 702f5bf8, 1289000, 70000)
000002a1029f50d1 moduninstall+0x148(1, 0, 1946c00, 0, 3000a5042a0, 7bf040bc)
000002a1029f5181 modunrload+0x6c(2b, 0, 1, 0, 0, 3000a5042a0)
000002a1029f5231 modctl+0x54(1, 2b, 2b, ffffffff7ffff248, 10010b000, 2)
000002a1029f52e1 syscall_trap+0xac(1, 2b, 2b, ffffffff7ffff248, 10010b000, 2) 

With some storage reconfiguration's the number of paths can be "8n-1".  In such situations the removal of Veritas DMP kernel driver can result in a panic.

 In this instance, the panic occurred while unloading the Solaris 5.1 SP1 RP1 P1 Veritas DMP driver, and when attempting to install VxVM hot-fix 5.1 SP1 RP1 P1 HF2 (Solaris).

The problem relates to the VXVM "P-patch" (5.1 SP1 RP1 P1), and not with the hot-fix itself.

The DMP related panic occurred due to kernel heap corruption while unloading the Veritas DMP driver, which is trying to release allocated memory.

> ::status
debugging crash dump vmcore.414 (64-bit) from dopey
operating system: 5.10 Generic_144488-07 (sun4u)
panic message: kernel heap corruption detected
dump content: kernel pages only

> ::modinfo !grep vx
43         7bed4000    50c10   1 vxdmp (VxVM 5.1SP1RP1P1 DMP Driver)
45                0        0   0 vxio (?)
47                0        0   0 vxspec (?)
229         7b7cf270      d40   1 vxportal (VxFS 5.1_SP1RP1 portal driver)
230         7a600000   1fe098   1 vxfs (VxFS 5.1_SP1RP1 SunOS 5.10) 

> *panicstr/s
0x1282bb0:      kernel heap corruption detected

SolarisCAT(vmcore.0/10U)> stack
unix:panicsys+0x48(0x1282bb0, 0x2a1029f5678, 0x18c1160, 0x1, , , 0x14001603, , , , , , , , 0x1282bb0, 0x2a1029f5678)
unix:vpanic_common+0x78(0x1282bb0, 0x2a1029f5678, 0x50, 0x60052e0faa0, 0xc8, 0x2000)
unix:panic+0x1c(0x1282bb0, 0x600735fec08, 0x0, 0x3000012a060, 0x1, 0x18e8400)
genunix:kmem_error+0x4b4(, 0x3000012a008?, 0x600735fec08?)
genunix:kmem_free(0x600735fec08, 0x16) - frame recycled                                     <<<<<<<<< NOTE: Size freed is 22 so need to look at the allocation.
vxdmp:dmp_free_stats_table+0x118()
vxdmp:dmp_free_modules+0x24(0x0, 0x0, 0x1, 0x702f5ba0, 0x701f2c90, 0x0)
vxdmp:_fini+0x178()
genunix:moduninstall+0x148(0x3000a5042a0)
genunix:modunrload+0x6c(, , 0x1)
genunix:modctl_modunload(0x2b) - frame recycled
genunix:modctl+0x54(, , 0x2b, 0xffffffff7ffff248, , 0x2)
unix:syscall_trap+0xac()
-- switch to user thread's user stack –

Summary
 

- Within the dmp_free_stats_table() routine, Veritas DMP is trying to free more than the allocated buffer size, which is leading to kernel heap corruption followed by the system panic.

 - In this instance, there are a total of 167 (8*20 + 7) paths presented to the system, and a total of 21 bytes were allocated for dmp_used_cpu_idx,  (which is used as bit mask array, maintaining the used per-cpu index by paths), however, the a total of 22 bytes are being freed.

 - The allocation was done based on the following equation:

No of bytes allocated for dmp_used_cpu_idx = total_no_of_paths/8 + 1   

          

SolarisCAT(vmcore.0/10U)> rd32 dmp_max_stats_idx
0x701f3670 vxdmp(data):dmp_max_stats_idx+0x0 = 0xa7

NOTE: The above HEX "0xa7" value, equates to a decimal value of "167".

The source code has been modified to free the kernel buffer, which is aligned to the allocation size.

The issue is addressed by applying Veritas Volumer Manager (VxVM) 5.1 SP1 RP1 P1 "HF4" on top of 5.1 SP1 RP1 P1 on Solaris.

Please contact Veritas Support if you require this hot-fix.

Applies To

Solaris
Veritas Volume Manager 5.1 SP1 RP1 P1
 

A potential DMP related panic may occur when applying a Veritas Volume Manager (VxVM) related hot-fix (ie 5.1 SP1 RP1 P1 HF2) on top of 5.1 SP1 RP1 P1. The DMP panic occurs when the total number of paths equates to “8n-1” (i.e 7, 15, 23,………,239….), and when the Veritas DMP kernel driver is unloaded.
During DMP kernel device driver unload, it frees all the allocated kernel heap memory. As part of freeing the allocated memory, Veritas DMP is trying to free more than the allocated buffer size for one of the allocated buffer's, which is leading to system panic when kernel memory audit is enabled.  

ETrack: e2318820 (2317540)