Veritas Voulme Manager (VxVM) - Race condition caused VxVM kernel driver to access freed memory and hence sysem panic

book

Article ID: 100025656

calendar_today

Updated On:

Description

Error Message

The following is the panic stack.

  000002a10247a7a1 vpanic()
  000002a10247a851 kmem_error+0x4b4()
  000002a10247a921 vol_subdisksio_done+0xe0()
  000002a10247a9d1 volkcontext_process+0x118()
  000002a10247aaa1 voldiskiodone+0x360()
  000002a10247abb1 voldmp_iodone+0xc()
  000002a10247ac61 gendmpiodone+0x1ec()
  000002a10247ad11 ssd_return_command+0x240()
  000002a10247add1 ssdintr+0x294()
  000002a10247ae81 ql_fast_fcp_post+0x184()
  000002a10247af31 ql_24xx_status_entry+0x2c8()
  000002a10247afe1 ql_response_pkt+0x29c()
  000002a10247b091 ql_isr_aif+0x76c()
  000002a10247b181 px_msiq_intr+0x200()
  000002a10247b291 intr_thread+0x168()
  000002a10240b131 cpu_halt+0x174()
  000002a10240b1e1 idle+0xd4()
  000002a10240b291 thread_start+4()

Cause

SYMPTOM:

In a Storage Foundation environment running Symantec Oracle Disk Manager (ODM), Veritas File System (VxFS) and Volume Manager (VxVM), a system panic may occur
with following the stack trace:

  000002a10247a7a1 vpanic()
  000002a10247a851 kmem_error+0x4b4()
  000002a10247a921 vol_subdisksio_done+0xe0()
  000002a10247a9d1 volkcontext_process+0x118()
  000002a10247aaa1 voldiskiodone+0x360()
  000002a10247abb1 voldmp_iodone+0xc()
  000002a10247ac61 gendmpiodone+0x1ec()
  000002a10247ad11 ssd_return_command+0x240()
  000002a10247add1 ssdintr+0x294()
  000002a10247ae81 ql_fast_fcp_post+0x184()
  000002a10247af31 ql_24xx_status_entry+0x2c8()
  000002a10247afe1 ql_response_pkt+0x29c()
  000002a10247b091 ql_isr_aif+0x76c()
  000002a10247b181 px_msiq_intr+0x200()
  000002a10247b291 intr_thread+0x168()
  000002a10240b131 cpu_halt+0x174()
  000002a10240b1e1 idle+0xd4()
  000002a10240b291 thread_start+4()

In Solaris operating system, if kmem_flag is enabled (kmem_flag=0xf), the following error message could also be seen just before system panic.

"buffer was allocated from kmem_alloc_224, caller attempting free to kmem_alloc_256"

DESCRIPTION:

A race condition exists between two IOs (specifically Volume Manager subdisk level staged I/Os) while doing 'done' processing which causes one thread to free FS-VM private information data structure before other thread accesses it.

The propensity of the race increases by increasing the number of CPUs.

RESOLUTION:

Avoid the race condition such that the slower thread doesn't access the freed FS-VM private information data structure.

 

Resolution

The problem will be fixed in the future patch releases.

 

Applies To

Storage Foundation environment running Veritas Oracle Disk Manager (ODM), Veritas File System (VxFS) and Volume Manager (VxVM).

Issue/Introduction

Race condition caused VxVM kernel driver to access freed memory and hence sysem panic.

Additional Information

ETrack: 2484685