Error Message

VxVM vxdg ERROR V-5-1-10978 Disk group MSLEE_DG-0_76: import failed:
Memory allocation failure

There will be two possible causes for this error:

1. The VxVM software is not patched and setulimit is setting maxDATA to something unexpected because 10% of physmem is greater than 4GB.
2. The VxVM software is patched, and the default setting of 8 memory segments (2GB maxDATA) is not sufficient.

Limitation with setulimit() observed in VxVM 5.0 code (AIX):

setulimit() gets the current physical memory in Kilobytes. It calls ulimit() with a new value only if the current data segment resource limit is less than the new calculated limit. The new calculated limit is nothing but 10% of total physical memory.

Due to limitation/bug in setulimit function, it doesn’t check whether the new calculated limit, which is greater than the current data segment value, is a valid entity. The validity is under question if the new calculated maxData value is more than 4GB (second argument of ulimit() is unsigned long, so maximum value it can store is 4GB i.e. 4294967296 bytes). For example, if the total system memory is 51 GB, the new calculated limit is 10% of 51 GB = 5.1 GB and it is greater than 4GB.

Since setulimit() would invoke ulimit with new calculated limit as 5.1 GB, it would override the current setting (say maxDATA : 0x80000000) with unsigned long overflow value which is invalid.

Systems with less than 20GB of physmem, the default maxData of 2 GB is utilised for vxconfigd process.

Systems with less than 40GB of physmem, the 10% of physmem calculation is less than 4GB which allows the setulimit() functions to correctly set the vxconfigd maxDATA to a valid unsigned long value, which can handle values up to 4294967296 bytes.

We discuss this issue affecting servers with 40GB of Memory. Servers with 80GB and 120GB of physmem are impacted in the same way, resulting 'Memory allocation failure' errors.

This table explains the cyclical impact caused by vxconfigd unsigned int overflowing when available Physical Memory is above 40GB.

Since a vxconfigd maxdata value > 4 GB cannot be represented using 32 bits, reduced modulo 0x100000000 is applied by setulimit, resulting in a lower/unexpected value.

Examples:
4.5 GB (45G Physmem resulting in overflow condition)

overflow             = (0x120000000) % 0x100000000 = 0x20000000 = 500 MB (We were expecting 4.5GB)

For 10 GB (100GB Physmem resulting in overflow condition)

overflow             = (0x280000000) % 0x100000000 = 0x80000000 = 2 GB (We were expecting 10GB)

This limitation means that only the lowest 32 bits are used, so integer overflows cause the result to be truncated to a size that can be represented by the variable. This is often called a "wrap around", as the result appears to wrap around to 0.

On backup servers, where number of imported diskgroups at any one time may be very large, these memory allocation failure errors can manifest on systems with more than 50 diskgroups imported at the same time with maxData being set to 2GB or less (due to this setulimit() bug).

# To check the assigned value.

# dump -ov /sbin/vxconfigd

/sbin/vxconfigd:

**Object Module Header**

1. Sections Symbol Ptr # Symbols Opt Hdr Len Flags

6 0x005276da 23851 72 0x1042
Flags=( EXEC DYNLOAD DSA DEP_SYSTEM )
Timestamp = "Apr 6 08:28:12 2016"
Magic = 0x1df (32-bit XCOFF)

**Optional Header**
Tsize Dsize Bsize Tstart Dstart
0x00314550 0x0011354c 0x0005bdc4 0x100001a0 0x200006f0

SNloader SNentry SNtext SNtoc SNdata
0x0004 0x0002 0x0001 0x0002 0x0002

TXTalign DATAalign TOC vstamp entry
0x0007 0x0003 0x20112644 0x0001 0x20110098

maxSTACK maxDATA SNbss magic modtype

0x00000000 0xc0000000 0x0003 0x010b 1L

Note :

 A program can have a maximum of 13 segments or 3.25 GB of maxdata.

Applies To

- AIX 

5.2, 5.3, 6.1 systems using VERITAS 

Volume Manager 5.0 (VxVM) , 5.1 (VxVM) and above.

• This issue and solution is specific to AIX
• VxVM configurations with many diskgroups
• Systems with more than 40GB of physical memory