Error Message

SolarisCAT(vmcore.0/10V)> panic
panic on cpu 204
panic string:   BAD TRAP: type=31 rp=2a10a137530 addr=90 mmu_fsr=0 occurred in module "vxio" due to a NULL pointer dereference
==== panic kernel thread: 0x2a10a137ca0  PID: 0  on CPU: 204 ====
cmd: sched
t_procp: 0x18886c0(proc_sched)
  p_as: 0x188a2e0(kas)
  zone: global
t_stk: 0x2a10a137a90  sp: 0x18b4701  t_stkbase: 0x2a10a132000
t_pri: 60(SYS)  pctcpu: 0.000000
t_lwp: 0x0  psrset: 0  last CPU: 204
idle: 1 ticks (0.01 seconds)
start: Wed Oct 20 05:07:25 2010
age: 5856666 seconds (67 days 18 hours 51 minutes 6 seconds)
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_TALLOCSTK - thread structure allocated from stk
        T_PANIC - thread initiated a system panic
tpflg:  none set
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
        TS_SIGNALLED - thread was awakened by cv_signal()
pflag:  SSYS - system resident process

pc:      unix:panicsys+0x48:   call     unix:setjmp
startpc: vxio:voliod_loop+0x0:   save   %sp, -0x110, %sp

unix:panicsys+0x48(0x10b5ca0, 0x2a10a1372d8, 0x18b50d0, 0x1, , , 0x9900001605, , , , , , , , 0x10b5ca0, 0x2a10a1372d8)
unix:vpanic_common+0x78(0x10b5ca0, 0x2a10a1372d8, 0x9, 0x1, 0x8, 0x8)
unix:panic+0x1c(0x10b5ca0, 0x31, 0x2a10a137530, 0x90, 0x0, 0x3000d7d5bd8, 0x1846768)
unix:die+0x78(0x31, 0x2a10a137530, 0x90, 0x0)
unix:trap+0x9e4(0x2a10a137530, 0x90)
unix:ktl0+0x64()
-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a10a137530  LEAF --
  addr: 0x90
pc:  0x7be60788 vxio:volkiomem_iter+0x0:   ldx  [%o1 + 0x90], %o5
npc: 0x7be6078c vxio:volkiomem_iter+0x4:   brz,pn %o5, vxio:volkiomem_iter+0x18
  global:                       %g1      0x303b0ec6d80
        %g2          0xefefbb2  %g3      0x302b85dadc8
        %g4                0x3  %g5             0x4020
        %g6               0x10  %g7      0x2a10a137ca0
  out:  %o0      0x2a10a137680  %o1                  0
        %o2                  0  %o3                  0
        %o4      0x304ff733968  %o5      0x304ff733968
        %sp      0x2a10a136dd1  %o7         0x7be60e20
  loc:  %l0             0x328e  %l1      0x3023961f030
        %l2                0x6  %l3      0x2a10a137680
        %l4      0x2a10a1376d0  %l5               0x40
        %l6                0xe  %l7             0x2000
  in:   %i0                  0  %i1                  0
        %i2                  0  %i3             0x4000
        %i4                  0  %i5                  0
        %fp      0x2a10a136f21  %i7         0x7be9bc10
vxio:volkiomem_iter+0x0(0x2a10a137680, 0x0, 0x0, 0x0)
vxio:volkio_to_kio_copy+0x44(0x302f034dcc0, , , 0x4000, 0x2000, 0x0)
vxio:vol_multistepsio_overlay_data+0x70(0x301f4eeb6c0, 0x2a10a137a30)
vxio:vol_multistepsio_start+0x77c(0x301f4eeb6c0, 0x2a10a137a30)
vxio:voliod_iohandle+0x30(0x301f4eeb6c0, 0x1, 0x2a10a137a30)
vxio:voliod_loop+0x69c(0x0, 0x0)
unix:thread_start+0x4()
-- end of kernel thread's stack --

We can see that the panic occurred in the following area:

-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a10a137530  LEAF --
  addr: 0x90
pc:  0x7be60788 vxio:volkiomem_iter+0x0:   ldx  [%o1 + 0x90], %o5
npc: 0x7be6078c vxio:volkiomem_iter+0x4:   brz,pn %o5, vxio:volkiomem_iter+0x18

%o1 is 0 which triggered the null pointer panic.

The configuration contained multi-terabyte volumes, but the panic only occurred when 6 snapshots and 5 cacheobjects were created.

Snapshots:
-----------

v  chk          -            ENABLED  ACTIVE   5368709120 SELECT  -        fsgen
pl chk-P01      chk          ENABLED  ACTIVE   5368709120 CONCAT  -        RW
sc chk-S01      chk-P01      chk_CO   0        5368709120 0       -        ENA
dc chk_dco      chk          chk_dcl    
v  chk_dcl      -            ENABLED  ACTIVE   369664   SELECT    -        gen
pl chk_dcl-01   chk_dcl      ENABLED  ACTIVE   369664   CONCAT    -        RW
sd ams_wms0_191-04 chk_dcl-01 ams_wms0_191 314572800 369664 0     ams_wms0_191 ENA
sp d02_main_snap_snp2 chk    chk_dco    

v  dev          -            ENABLED  ACTIVE   5368709120 SELECT  -        fsgen
pl dev-P01      dev          ENABLED  ACTIVE   5368709120 CONCAT  -        RW
sc dev-S01      dev-P01      dev_CO   0        5368709120 0       -        ENA
dc dev_dco      dev          dev_dcl    
v  dev_dcl      -            ENABLED  ACTIVE   369664   SELECT    -        gen
pl dev_dcl-01   dev_dcl      ENABLED  ACTIVE   369664   CONCAT    -        RW
sd ams_wms0_189-02 dev_dcl-01 ams_wms0_189 104857600 369664 0     ams_wms0_189 ENA
sp d02_main_snap_snp dev     dev_dco    

v  d02_main_snap -           ENABLED  ACTIVE   7440973824 SELECT  d02_main_snap-01 fsgen
pl d02_main_snap-01 d02_main_snap ENABLED ACTIVE 7440973824 STRIPE 4/128   RW
sd ams_wms0_172-01 d02_main_snap-01 ams_wms0_172 0 1342177280 0/0 ams_wms0_172 ENA
sd ams_wms0_60-01 d02_main_snap-01 ams_wms0_60 0 262066176 0/1342177280 ams_wms0_60 ENA
sd ams_wms0_60-03 d02_main_snap-01 ams_wms0_60 262578304 256000000 0/1604243456 ams_wms0_60 ENA
sd ams_wms0_173-01 d02_main_snap-01 ams_wms0_173 0 1342177280 1/0 ams_wms0_173 ENA
sd ams_wms0_71-01 d02_main_snap-01 ams_wms0_71 0 518066176 1/1342177280 ams_wms0_71 ENA
sd ams_wms0_174-01 d02_main_snap-01 ams_wms0_174 0 1342177280 2/0 ams_wms0_174 ENA
sd ams_wms0_72-01 d02_main_snap-01 ams_wms0_72 0 518066176 2/1342177280 ams_wms0_72 ENA
sd ams_wms0_175-01 d02_main_snap-01 ams_wms0_175 0 1342177280 3/0 ams_wms0_175 ENA
sd ams_wms0_59-02 d02_main_snap-01 ams_wms0_59 441760 518066176 3/1342177280 ams_wms0_59 ENA
dc d02_main_snap_dco d02_main_snap d02_main_snap_dcl
v  d02_main_snap_dcl -       ENABLED  ACTIVE   512128   SELECT    -        gen
pl d02_main_snap_dcl-01 d02_main_snap_dcl ENABLED ACTIVE 512128 CONCAT -   RW
sd ams_wms0_60-02 d02_main_snap_dcl-01 ams_wms0_60 262066176 512128 0 ams_wms0_60 ENA
sp chk_snp      d02_main_snap d02_main_snap_dco
sp dev_snp      d02_main_snap d02_main_snap_dco
sp stest_snp    d02_main_snap d02_main_snap_dco
sp vtest_snp    d02_main_snap d02_main_snap_dco
sp vuat_snp     d02_main_snap d02_main_snap_dco

cache objects:
---------------------

co chk_CO       cache_chk    ENABLED  ACTIVE  
v  cache_chk    chk_CO       ENABLED  ACTIVE   587202560 SELECT   cache_chk-01 fsgen
pl cache_chk-01 cache_chk    ENABLED  ACTIVE   587202560 STRIPE   4/128    RW
sd ams_wms0_191-03 cache_chk-01 ams_wms0_191 209715200 104857600 0/0 ams_wms0_191 ENA
sd ams_wms0_191-06 cache_chk-01 ams_wms0_191 335913984 20971520 0/104857600 ams_wms0_191 ENA
sd ams_wms0_191-08 cache_chk-01 ams_wms0_191 377857024 20971520 0/125829120 ams_wms0_191 ENA
sd ams_wms0_188-04 cache_chk-01 ams_wms0_188 210084864 104857600 1/0 ams_wms0_188 ENA
sd ams_wms0_188-07 cache_chk-01 ams_wms0_188 336283648 20971520 1/104857600 ams_wms0_188 ENA
sd ams_wms0_188-09 cache_chk-01 ams_wms0_188 378226688 20971520 1/125829120 ams_wms0_188 ENA
sd ams_wms0_189-04 cache_chk-01 ams_wms0_189 210084864 104857600 2/0 ams_wms0_189 ENA
sd ams_wms0_189-07 cache_chk-01 ams_wms0_189 336283648 20971520 2/104857600 ams_wms0_189 ENA
sd ams_wms0_189-09 cache_chk-01 ams_wms0_189 378226688 20971520 2/125829120 ams_wms0_189 ENA
sd ams_wms0_190-04 cache_chk-01 ams_wms0_190 210084864 104857600 3/0 ams_wms0_190 ENA
sd ams_wms0_190-07 cache_chk-01 ams_wms0_190 336283648 20971520 3/104857600 ams_wms0_190 ENA
sd ams_wms0_190-09 cache_chk-01 ams_wms0_190 378226688 20971520 3/125829120 ams_wms0_190 ENA

co dev_CO       cache_dev    ENABLED  ACTIVE  
v  cache_dev    dev_CO       ENABLED  ACTIVE   587202560 SELECT   cache_dev-01 fsgen
pl cache_dev-01 cache_dev    ENABLED  ACTIVE   587202560 STRIPE   4/128    RW
sd ams_wms0_189-01 cache_dev-01 ams_wms0_189 0 104857600 0/0      ams_wms0_189 ENA
sd ams_wms0_189-06 cache_dev-01 ams_wms0_189 315312128 20971520 0/104857600 ams_wms0_189 ENA
sd ams_wms0_189-08 cache_dev-01 ams_wms0_189 357255168 20971520 0/125829120 ams_wms0_189 ENA
sd ams_wms0_190-01 cache_dev-01 ams_wms0_190 0 104857600 1/0      ams_wms0_190 ENA
sd ams_wms0_190-05 cache_dev-01 ams_wms0_190 314942464 20971520 1/104857600 ams_wms0_190 ENA
sd ams_wms0_190-08 cache_dev-01 ams_wms0_190 357255168 20971520 1/125829120 ams_wms0_190 ENA
sd ams_wms0_191-01 cache_dev-01 ams_wms0_191 0 104857600 2/0      ams_wms0_191 ENA
sd ams_wms0_191-05 cache_dev-01 ams_wms0_191 314942464 20971520 2/104857600 ams_wms0_191 ENA
sd ams_wms0_191-07 cache_dev-01 ams_wms0_191 356885504 20971520 2/125829120 ams_wms0_191 ENA
sd ams_wms0_188-02 cache_dev-01 ams_wms0_188 369664 104857600 3/0 ams_wms0_188 ENA
sd ams_wms0_188-06 cache_dev-01 ams_wms0_188 315312128 20971520 3/104857600 ams_wms0_188 ENA
sd ams_wms0_188-08 cache_dev-01 ams_wms0_188 357255168 20971520 3/125829120 ams_wms0_188 ENA

When using multi-terabyte volumes Veritas recommends that the regionsz and volpagemod_max_memsz values are tuned.

The region size is set when a snapshot is created and Veritas would recommend that the regionsz be set to a minimum of 512k, especially if multi-terabyte volumes are used. The regionsz is the number of sectors for a dirty region logging (DRL) volume region, With a DRL, VxVM logically divides a volume into a set of consecutive regions. Larger region sizes tend to cause the cache hit-ratio for regions to improve. This improves the write performance, but it also prolongs the recovery time. 

The regionsz size cannot be changed on the fly and the snapshot will need to be recreated with the newer regionsz of 512k.

# vxassist -g datadg make SNAP-volgunnum 400m logtype=dco dcoversion=20 fastresync=on regionsz=512

Also for volpagemod_max_memsz Veritas recommends a value of 256M.

The vxtune command can be used to activate the change on the fly.
# vxtune volpagemod_max_memsz 256M

To make the above change permanent across system reboot, add the following entry into the Volume Manager kernel configuration file /kernel/drv/vxio.conf:

volpagemod_max_memsz=262144;

These changes should stop further occurrences of this panic.
 

Applies To

SF 5.0 MP3 For Solaris