Error Message

Whilst troubleshooting the vxconfigd performance issue, a "gcore" of the vxconfigd process id (PID) can be captured.

You may see the following strack trace for all threads using the following syntax:

(gdb) t a a bt

Thread 3 (LWP 57864):
#0  0x00000034dbadb80d in read () from ./lib64/libc.so.6
#1  0x00000034dba71bc8 in _IO_new_file_underflow () from ./lib64/libc.so.6
#2  0x00000034dba736ce in _IO_default_uflow_internal () from ./lib64/libc.so.6
#3  0x00000034dba67daa in _IO_getline_info_internal () from ./lib64/libc.so.6
#4  0x00000034dba66c09 in fgets () from ./lib64/libc.so.6
#5  0x00000000005928b2 in blkext_build_device_map ()
#6  0x0000000000485fe7 in cds_init_internal ()
#7  0x00000000004864b9 in cds_flush ()
#8  0x0000000000481b8b in auto_sys_flush ()
#9  0x000000000047f558 in auto_flush ()
#10 0x00000000004ef906 in req_dg_flush_common ()
#11 0x00000000004efc9d in req_dg_flush_name ()
#12 0x000000000055ed7e in request_loop ()
#13 0x0000000000479f46 in main ()

Thread 2 (LWP 9575):
#0  0x00000034dbaacc5d in pause () from ./lib64/libc.so.6
#1  0x000000000055c970 in vold_timeout_handler ()
#2  0x00000034dbe07aa1 in start_thread () from ./lib64/libpthread.so.0
#3  0x00000034dbae8c4d in clone () from ./lib64/libc.so.6

Thread 1 (LWP 9574):
#0  0x00000034dbe0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0
#1  0x000000000055d35c in vold_dispatch_requests ()
#2  0x00000034dbe07aa1 in start_thread () from ./lib64/libpthread.so.0
#3  0x00000034dbae8c4d in clone () from ./lib64/libc.so.6

Sample syntax:

# pgrep vxconfigd
14551

# gcore
usage:  gcore [-a] [-o filename] pid

# gcore -o /tmp/vxconfigd.out 14551

To confirm which binary relates to the core file, type:

# file /tmp/vxconfigd.out.14551
/tmp/vxconfigd.out.14551: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'vxconfigd', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/sbin/vxconfigd', platform: 'x86_64'

# /opt/VRTSspt/vxgetcore/vxgetcore -c /tmp/vxconfigd.out.14551 -b /usr/sbin/vxconfigd

The vxgetcore utility will create a tarball which can be uploaded to Support.

This will contain the "gdb.bt.out" file capturing the backtrace (stack trace).
 

# gdb
(gdb) set solib-absolute-prefix .
(gdb) set solib-search-path .
(gdb) file vxconfigd
Reading symbols from /tmp/getcore_vxconfigd/vxconfigd...done.
(gdb) core-file core.57864


A backtrace is a summary of how your program got where it is

To display the backtrace (bt) using gdb debugger tool.

(gdb) bt
#0  0x00000034dbe0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0
#1  0x000000000055d35c in vold_dispatch_requests ()
#2  0x00000034dbe07aa1 in start_thread () from ./lib64/libpthread.so.0
#3  0x00000034dbae8c4d in clone () from ./lib64/libc.so.6
 

(gdb) thread apply all bt                                <--------- collect stacktrace for all threads

or

(gdb) t a a bt

Thread 3 (LWP 57864):
#0  0x00000034dbadb80d in read () from ./lib64/libc.so.6
#1  0x00000034dba71bc8 in _IO_new_file_underflow () from ./lib64/libc.so.6
#2  0x00000034dba736ce in _IO_default_uflow_internal () from ./lib64/libc.so.6
#3  0x00000034dba67daa in _IO_getline_info_internal () from ./lib64/libc.so.6
#4  0x00000034dba66c09 in fgets () from ./lib64/libc.so.6
#5  0x00000000005928b2 in blkext_build_device_map ()
#6  0x0000000000485fe7 in cds_init_internal ()
#7  0x00000000004864b9 in cds_flush ()
#8  0x0000000000481b8b in auto_sys_flush ()
#9  0x000000000047f558 in auto_flush ()
#10 0x00000000004ef906 in req_dg_flush_common ()
#11 0x00000000004efc9d in req_dg_flush_name ()
#12 0x000000000055ed7e in request_loop ()
#13 0x0000000000479f46 in main ()

Thread 2 (LWP 9575):
#0  0x00000034dbaacc5d in pause () from ./lib64/libc.so.6
#1  0x000000000055c970 in vold_timeout_handler ()
#2  0x00000034dbe07aa1 in start_thread () from ./lib64/libpthread.so.0
#3  0x00000034dbae8c4d in clone () from ./lib64/libc.so.6

Thread 1 (LWP 9574):
#0  0x00000034dbe0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0
#1  0x000000000055d35c in vold_dispatch_requests ()
#2  0x00000034dbe07aa1 in start_thread () from ./lib64/libpthread.so.0
#3  0x00000034dbae8c4d in clone () from ./lib64/libc.so.6
 

(gdb) quit

 

When the partition count of a device exceeds 15, the partition device gets assigned under major 259. Major no 259 is used to overcome a sd limitation (16 minors per device), which enables more partitions for a single sd device. 

Linux BLOCK_EXT_MAJOR (block major 259) is used as extended devt for block devices. 

The vxconfigd function that caused vxconfigd to respond sluggishly is called “blkext_build_device_map”.

It is responsible for parsing the contents of the /proc/partitions file line by line through the fgets() function.

The fgets()  function then finds all the partition devices with major number 259, it then stores this information for future use.

In an environment with thousands of LUNs, but none of which have more than 15 partitions according to /proc/partitions. 

# wc -l partitions-220705010302.txt
45827 partitions-220705010302.txt

# grep "^ *259" partitions-220705010302.txt
 

NOTE: Block major 259 is not present in /proc/partitions.
 

Since there are no partition numbers above 15 (no device with major 259), the blkext_build_device_map function will find nothing, however, it will consume lots of unwanted & needless cpu cycles in reading the /proc/partitions file. 

For one "vxdg flush" operation, each one of the thousands of LUNs presented to the server would have to call the blkext_build_device_map function once, which will then call fgets() 45000+ times. 


For one "vxdg flush" operation, the number of fgets() function calls varies depending on the no. of LUNs and /proc/partitions entries.

NOTE: LUN count is 2500 and /proc/partitions file has some 45000+ line entries ((2500 * 45000).

The huge amount of calls can have a negative impact on vxconfigd and result in slowness with vxconfigd when performing VxVM & DMP related commands.  
 
 

 

Engineering have optimized the code in the corresponding code path. 

In cds_init_internal function, we can check the number of partitions used by the current LUN, and only call blkext_build_device_map function when the LUN has more than 15 partitions used. 

With this optimization, the blkext_build_device_map function will never get called in this case, since none of LUNs has more than 15 partitions and extended devt is not used.
 

PATCH NAME

Veritas Volume Manager 7.4.1 Hot Fix 3408                  (vm-rhel6_x86_64-HotFix-7.4.1.3408.tar.gz)

OPERATING SYSTEMS SUPPORTED BY THE PATCH

RHEL6 x86-64

Patch ID: 7.4.1.3408

* 4081738 (Tracking ID: 4081740)

SYMPTOM:
vxdg flush command slow due to too many luns needlessly access /proc/partitions.

DESCRIPTION:
Linux BLOCK_EXT_MAJOR(block major 259) is used as extended devt for block devices. When partition number of one device is more than 15, the partition device gets assigned under major 259 to solve the sd limitations (16 minors per device), by which more partitions are allowed for one sd device. During "vxdg flush", for each lun in the disk group, vxconfigd reads file /proc/partitions line by line through fgets() to find all the partition devices with major number 259, which would cause vxconfigd to respond sluggishly if there are large amount of luns in the disk group.

RESOLUTION:
Code has been changed to remove the needless access on /proc/partitions for the luns without using extended devt.


Please contact Veritas Technical Support if you require this Private hot-fix. The code changes will be ported to other Linux platforms.
 

The Veritas Volume Manager (VxVM) vxconfigd daemon may exhibit slow response times for VxVM & DMP CLI commands, where the LUN count is above 1000 and the path count is high (i.e. 8 paths).

The more LUNs and paths presented to a system the greater the risk of vxconfigd encountering an issue in relation to clustered environments managed by VCS.

One trigger for the vxconfigd slowness is the "vxdg flush" command.

Veritas Cluster Server (VCS) calls "vxdg flush <dg-name>" when offlining the DiskGroup resource.