How to Persistently set dump memory allocation or disable vxdmpadm iostat collection across reboots

A new tuneable 'dmp_iostats_state' has been introduced at patch 7.3.1.300 and 7.4.1.1200. This will persistently disable vxdmp iostat collection.

# vxdmpadm gettune dmp_iostats_state
            Tunable               Current Value  Default Value
------------------------------    -------------  -------------
dmp_iostats_state                   enabled          enabled

# vxdmpadm settune dmp_iostats_state=disabled
Tunable value will be changed immediately

# vxdmpadm gettune dmp_iostats_state
            Tunable               Current Value  Default Value
------------------------------    -------------  -------------
dmp_iostats_state                  disabled          enabled

# vxdmpadm iostat show
VxVM vxdmpadm ERROR V-5-1-0  DMP iostat daemon is DISABLED and hence not started. Use 'vxdmpadm settune dmp_iostat_state=enabled' to first ENABLE DMP iostat daemon tunable, and then try again.

If the VxVM version is too low to use the dmp_iostats_state then a boot script can be used to disable the feature at the end of boot. 

For example on older Linux versions, add the following line to file /etc/rc.local

/sbin/vxdmpadm iostat stop

On newer Linux version which use systemd, create a service which will run after the vxvm-recover.service

Create a file /etc/systemd/system/multi-user.target.wants/vxdmpadm-iostat.service with the following content:

     [Unit]
    After=vxvm-recover.service
    
    [Service]
    ExecStart=/sbin/vxdmpadm iostat stop 

A system service can also be used to set the amount of memory instead. In this case, use the 'ExecStart' line to set the amount of memory, instead of stopping the vxdmpadm iostat subsystem. For example:

     [Unit]
    After=vxvm-recover.service
    
    [Service]
    ExecStart=/sbin/vxdmpadm start memory=4190208

Applies To

This issue is applicable to Linux systems running Volumer Manager (VxVM) versions 5.1 and above.

The product enhancement is available via the following patches:

https://downloads.infoscale.com/


infoscale-rhel6_x86_64-Patch-7.3.1.100  
infoscale-rhel7_x86_64-Patch-7.3.1.200   
infoscale-aix72-Patch-7.3.1.100          
sfha-sol11_sparc-Patch-6.2.1.200     
infoscale-rhel7_x86_64-Patch-7.3.1.300   
infoscale-sol11_sparc-Patch-7.3.1.200   
infoscale-sles12.4_x86_64-Patch-7.3.1.1100
infoscale-rhel7.7_x86_64-Patch-7.3.1.3100
sfha-rhel7.7_x86_64-Patch-6.2.1.7100 
sfha-sles12.4_x86_64-Patch-6.2.1.4100
infoscale-sles12_x86_64-Patch-7.3.1.3200 
infoscale-rhel7_x86_64-Patch-7.3.1.3300 
sfha-rhel7_x86_64-Patch-6.2.1.7200

[VxVM][181211-001424][AIR FRANCE]RHEL 6.9 VxVM 6.2.1.511 - VxVM vxdmp V-5-3-0 memory allocation failed for size =0x130000

This issue is tracked via etrack incident 3201531 mentioned in supplemental materials section.

When the vxdmp driver is loaded at boot, the vxdmp iostat subsystem is started. It will allocate memory which is used to maintain a circular buffer of i/o statistics. One table is required for each CPU. The memory requested must be contiguous, this is to ensure the best performance such that dmp statistics collection does not adversely affect i/o performance.

The size of this table (per CPU) can be seen by using 'vxdmpadm iostat show'. For example, on this server the initial size of the table is 4096 bytes.

# vxdmpadm iostat show
                       cpu usage = 1us    per cpu memory = 4096b

If large amounts of i/o complete quickly it is possible to for one of these tables to overflow. When that happens a message will be output to log file '/var/adm/vx/dmpevents.log'. For example:

Tue Jul 11 09:28:36.620: Lost 12 DMP I/O statistics records

When a table overflows, vxdmp will allocate a new, larger table (per cpu), and free the old table. This process will continue until a maximum limit is reached. This repeated process can cause memory fragmentation. The maximum limit is

( 1% of physical RAM ) / (number of cpu + 1)

If this memory allocation request fails, the below error will be logged in syslog, the size of the allocation will vary depending on the new size of the table.  The size shown is bytes (in hex).

Feb 10 21:19:02 node143 kernel: VxVM vxdmp V-5-3-0 memory allocation failed for size =0x105000
Feb 10 21:19:02 node143 kernel:
Feb 10 21:19:02 node143 kernel: VxVM vxdmp V-5-3-0 memory allocation failed for size =0x105000
Feb 10 21:19:02 node143 kernel:

A side-effect of the memory allocation failure is the kernel will take action and attempt to free space such that a similar request will succeed in the future. This can include dropping page-cache.
 

From Infoscale 7.4.1 additional enhancements are included to start with a larger table size and to increase the table size in larger steps. This will reduce the repeated freeing and allocation of memory and thus reduce any memory fragmentation it may cause. Note that this does not eliminate the dynamic allocation of the table. This enhancement has also been ported to various earlier versions as detailed below.

The vxdmpadm iostat subsystem can be started with a specified amount of memory to use for the per-cpu table. If this is done, then the table size is fixed and no dynamic increase in the table size occurs in the event of the table overflowing.

Thus, as the allocation request is never made, the failure is avoided. However, it will mean that if the table overflows some dmp I/O statistics would be lost.
 

Example: start the vxdmp iostat subsystem with a fixed table size of 256KB:

# vxdmpadm iostat stop
# vxdmpadm iostat start memory=262144

Example: start the vxdmp iostat subsystem with a fixed table size of 4190208 bytes

# vxdmpadm iostat stop
# vxdmpadm iostat start memory=4190208
 

Alternatively, the vxdmp iostat subsystem can be stopped and not used at all. As no I/O statistics are collected, the per-cpu tables can not overflow and therefore no dynamic memory allocation is made.

# vxdmpadm iostat stop

When collecting dmp statistics in an I/O intensive systems, messages similar to below may be logged in syslog. VxVM vxdmp V-5-3-0 memory allocation failed for size =0x105000 (Value of size may vary based on the size requested by the vxdmp iostat subsystem)

ETrack: 3201531