vxconfigd is consuming 1.7 GB of memory
Article ID: 100005174
Updated On:
Description
Error Message
Gather vmstat information and the corresponding size of vxconfigd during vmstat gathering:
Open two terminals, run the following in terminal one:
1. create a variable x for our while loop to increment
#x=0
2. run this while loop:
while [[ $x -lt 180 ]];
do
ps -aedfklm |grep vxconfigd >>
sleep 60
done
Run the following in terminal two:
date >> vmstat.out
vmstat 15 720 >>
_______________________________________
Next step after receiving the above files
#cat
Results from command:
1732720
224
1732720
224
1732720
224
_______________________________________
---------what does vmstat sr and cy signify?------------------
http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds6/vmstat.htm
Page: Information about page faults and paging activity. These are averaged over the interval and given in units per second.
re Pager input/output list.
pi Pages paged in from paging space.
po Pages paged out to paging space.
fr Pages freed (page replacement).
sr Pages scanned by page-replacement algorithm.
cy Clock cycles by page-replacement algorithm.
memory thrashing:
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
4 2 14035838 143747 0 0 0 0 0 0 6147 113182 21062 24 7 65 4 3.50 35.0
5 1 14028843 143746 0 0 0 0 0 0 6276 148132 22700 28 7 63 2 3.90 39.0
6 1 14028828 137538 0 0 0 0 0 0 6116 143776 23757 25 6 68 1 3.49 34.9
5 5 14021421 141548 0 0 0 0 0 0 6109 174392 23573 27 7 65 1 3.81 38.1
7 2 14034123 117752 0 0 0 0 0 0 6217 162288 31557 38 13 46 3 5.51 55.1
6 1 14028930 108250 0 0 0 0 0 0 6195 156164 24083 31 7 60 2 4.23 42.3
5 1 14039925 90193 0 0 0 0 0 0 6414 128627 22209 34 6 58 2 4.46 44.6
7 1 14050774 66836 0 0 0 0 0 0 7219 122291 22273 36 7 54 4 4.66 46.6
5 2 14029613 72366 0 0 0 0 0 0 7524 114647 22185 31 7 57 5 4.27 42.7
4 3 14019597 61350 0 0 0 0 0 0 6230 143924 22806 29 7 59 5 4.11 41.1
6 4 14436715 17617 0 0 0 28476 291179 0 9625 142646 27822 36 12 41 11 5.34 53.4
8 9 14513111 17575 0 0 0 10571 147889 0 19632 133720 29090 34 20 22 24 5.91 59.1
7 8 14530789 17649 0 0 0 5418 61033 0 20119 120743 26966 32 21 22 25 5.78 57.8
7 10 14588926 18216 0 0 0 8437 80928 0 20533 139071 27088 34 22 19 25 6.10 61.0
8 10 14652568 17532 0 0 0 9761 81120 0 21319 110630 25549 33 24 21 23 6.11 61.1
11 7 14595915 17523 0 0 0 2011 17618 0 21244 97230 22076 35 25 17 23 6.42 64.2
7 10 14647067 17565 0 0 0 9037 87283 0 21614 88631 21698 32 25 21 22 6.13 61.3
11 9 14591019 17513 0 0 0 2822 24678 0 21247 121630 23487 38 25 16 21 6.72 67.2
11 9 14459990 66027 0 0 0 383 3347 0 20874 100379 22742 36 24 19 21 6.46 64.6
8 3 14048437 656467 0 0 0 0 0 0 9699 183127 32748 52 9 35 5 6.68 66.8
11 2 14068132 620146 0 0 0 0 0 0 10854 173911 36131 64 10 23 3 8.00 80.0
13 1 14059329 616576 0 0 0 0 0 0 10852 184376 37901 53 11 33 3 6.95 69.5
8 2 14073349 582520 0 0 0 0 0 0 11072 169152 36639 56 10 30 3 7.24 72.4
8 1 14082123 556059 0 0 0 0 0 0 11802 191822 34603 41 12 41 5 5.81 58.1
22 1 14105235 500345 0 0 0 0 0 0 27015 151795 31996 60 26 5 9 9.01 90.1
14 5 14227549 231947 0 0 0 0 0 0 27188 147078 33886 58 32 2 8 9.45 94.5
19 5 14271527 23373 0 0 0 23 208 0 26788 120486 31179 54 36 3 8 9.38 93.8
21 5 14273900 22671 0 0 0 10996 94338 0 27090 154932 31233 59 37 1 3 9.97 99.7
19 3 14296326 18111 0 0 0 10370 80282 0 26563 128224 29328 59 33 2 6 9.58 95.8
17 4 14306427 17456 0 0 0 8829 70781 0 25967 107154 25935 59 34 2 5 9.61 96.1
/usr/lib/objrepos:VRTSgab:5.1.0.0::COMMIT:COMPLETE:09/25/10:00;38;00
/usr/lib/objrepos:VRTSgab:5.1.0.0::APPLY:COMPLETE:09/25/10:00;38;00
/usr/lib/objrepos:VRTSgab:5.1.1.0::APPLY:COMPLETE:09/25/10:00;50;44
/usr/lib/objrepos:VRTSllt:5.1.0.0::COMMIT:COMPLETE:09/25/10:00;37;51
/usr/lib/objrepos:VRTSllt:5.1.0.0::APPLY:COMPLETE:09/25/10:00;37;51
/usr/lib/objrepos:VRTSllt:5.1.1.0::APPLY:COMPLETE:09/25/10:00;50;31
/etc/objrepos:VRTSgab:5.1.0.0::COMMIT:COMPLETE:09/25/10:00;38;02
/etc/objrepos:VRTSgab:5.1.0.0::APPLY:COMPLETE:09/25/10:00;38;02
/etc/objrepos:VRTSgab:5.1.1.0::APPLY:COMPLETE:09/25/10:00;50;48
/etc/objrepos:VRTSllt:5.1.0.0::COMMIT:COMPLETE:09/25/10:00;37;54
/etc/objrepos:VRTSllt:5.1.0.0::APPLY:COMPLETE:09/25/10:00;37;54
/etc/objrepos:VRTSllt:5.1.1.0::APPLY:COMPLETE:09/25/10:00;50;3
VRTSvxvm 5.1.1.0 APPLIED Veritas Volume Manager by Symantec 5.1RP1
PROBLEM DESCRIPTION from above:
memory thrashing occurs during high system usage
______________________________________
Below is for the vxconfigd core and libraries, and binary, and svmon -P output of vxconfigd process.
The SZ field of ps output is in pages, and vxconfigd is huge. Therefore, we will need a core of vxconfigd,
vxconfigd binary and runtime libraries. Please gather this information using the following instructions:
1. Collect a core file from the running process with the '/usr/bin/gencore' command. This command will NOT kill the process.
The format is '/usr/bin/gencore [PID] [CORE-NAME]'.
For example:
# /usr/bin/gencore 114878 core.vxconfigd
Note this command may take a few minutes to run
2. Use the '/usr/sbin/snapcore' command to collect the core, binary, and required libraries.
The format is '/usr/sbin/snapcore
# /usr/sbin/snapcore core.vxconfigd /usr/sbin/vxconfigd
Data from the gencore:
gencore of /sbin/vxconfigd.5.3:
attribute - list existing attributes
condition - list existing condition variables
barrier - list existing barrier
mutex - list existing mutexes
spinlock - list existing spinlocks
rwlock - list existing rwlocks
thread - list existing threads
tstophwp - Sets a thread-level hardware watchpoint stop
ttracehwp - Sets a thread-level hardware watchpoint trace
tstop - Sets a source-level breakpoint stop for a thread
tstopi - Sets an instruction-level breakpoint stop for a thread
ttrace - Sets a source-level trace for a thread
ttracei - Sets an instruction-level trace for a thread
tnext - Runs a thread up to the next source line
tnexti - Runs a thread up to the next machine instruction
tstep - Runs a thread one source line
tstepi - Runs a thread one machine instruction
tskip - Skips breakpoints for a thread.
(dbx) mutex
(dbx) spinlock
(dbx) rwlock
(dbx) thread
svmon -P
# ps -elf |grep vxconfigd
#svmon -P
Data from the svmon -P
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
294998 vxconfigd.5.3 523214 67825 4945 522654 N Y N
PageSize Inuse Pin Pgsp Virtual
s 4 KB 433742 81 4945 433182
m 64 KB 1496 138 0 1496
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
612c62 5 work working storage s 65536 0 825 65536
448245 4 work working storage s 65536 0 834 65536
0 0 work kernel segment (lgpg_vsid=0) L 16 16 0 16
146e544 8 work working storage s 65536 0 588 65536
209c22 6 work working storage s 65536 0 661 65536
426440 7 work working storage s 65536 0 572 65536
480249 3 work working storage s 65407 0 1055 65437
670865 9 work working storage s 39322 0 388 39322
ad70ad d work text or shared-lib code seg m 1496 138 0 1496
460247 1 clnt code,/dev/hd4:4543 s 598 0 -
-
4f024e - work s 525 78 13 531
490248 f work working storage s 140 0 9 151
4c024d c work shared memory segment s 31 0 0 31
410240 2 work process private s 30 3 0 30
220023 - clnt /dev/hd4:978 s 9 0 -
-
5c025d - clnt /dev/hd4:57372 s 0 0 -
-
540255 - clnt /dev/hd4:57364 s 0 0 -
-
c00cc1 - clnt /dev/hd9var:49176 s 0 0 -
-
570256 - clnt /dev/hd4:57365 s 0 0 -
-
5d025c - clnt /dev/hd4:57371 s 0 0 -
-
4e024f - clnt /dev/hd4:57358 s 0 0 -
-
560257 - clnt /dev/hd4:57366 s 0 0 -
-
500251 - clnt /dev/hd4:57360 s 0 0 -
-
10a030b - clnt /dev/hd9var:17 s 0 0 -
-
550254 - clnt /dev/hd4:57363 s 0 0 -
-
590258 - clnt /dev/hd4:57367 s 0 0 -
-
580259 - clnt /dev/hd4:57368 s 0 0 -
-
520253 - clnt /dev/hd4:57362 s 0 0 -
-
510250 - clnt /dev/hd4:57359 s 0 0 -
-
4a024b e mmap maps 2 source(s) s 0 0 -
-
5a025b - clnt /dev/hd4:57370 s 0 0 -
-
5b025a - clnt /dev/hd4:57369 s 0 0 -
-
530252 - clnt /dev/hd4:57361 s 0 0 -
--------------------------------------------------------------------------
size of vxconfigd on 5.1 Sol 10 server:
FFBF6000 40K rwx-- [ stack ]
total 107168K
From AIX server in lab:
# svmon -P 704618 -wf
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
704618 vxconfigd.5.3 33541 11246 0 33541 N Y N
PageSize Inuse Pin Pgsp Virtual
s 4 KB 17477 11246 0 17477
m 64 KB 1004 0 0 1004
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
330ad d work text or shared-lib code seg m 1004 0 0 1004
0 0 work kernel segment s 15233 11229 0 15233
441d5 3 work working storage s 2001 0 0 2001
4c1d7 f work working storage s 120 0 0 120
641dd - work s 68 14 0 68
601dc c work shared memory segment s 30 0 0 30
5c1d3 2 work process private s 25 3 0 25
_______________________________________
Resolution
Customer upgraded to 5.1 RP2, and the svmon -P for the process id of vxconfigd was much smaller, vxconfigd is normal in size,
Customer did not experience memory thrashing any longer.
Applies To
This issues was seen on a AIX Platform, Version 5.3 VRTSvxvm=5.1RP1
Issue/Introduction
