Error Message

 

DCPA errors:

 

Apr 5 18:05:59 fred vxvm:vxconfigd: V-5-1-8769 ddl_find_devices_in_system: ddl_reconfigure_all failed: Device or resource busy: Device or resource busy
Apr 5 18:05:59 fred vxvm:vxconfigd: V-5-1-16011 Data Corruption Protection Activated - User Corrective Action Needed:
Apr 5 18:05:59 fred vxvm:vxconfigd: To recover, first ensure that the OS device tree is up to date (requires OS specific commands).
Apr 5 18:05:59 fred vxvm:vxconfigd: Then, execute 'vxdisk rm' on the following devices before reinitiating device discovery using 'vxdisk scandisks'
Apr 5 18:05:59 fred vxvm:vxconfigd: V-5-1-0 fred_disk_1, fred_disk_3
Apr 5 18:05:59 fred vxvm:vxconfigd: V-5-1-13790 No device configuration changes have been applied to DMP kernel database.
Apr 5 18:05:59 fred vxvm:vxconfigd: V-5-1-13791 Please consult the documentation for correct procedure to replace disk/path.
 

 

The above DCPA events are related to file open failures as too many files are currently open by vxconfigd and not released correctly, resulting in FD leaks in vxconfigd.

As the devices are internal to the Netbackup Appliance, it is unexpected to see such DCPA messages which subsequently result in unintended DMPNODE failures.

Upon further investigation we can see the Lun Serial Number (LSN) changing, thus triggering the DCPA events. The impacted DMPNODEs are then disabled to protect data integrity.

The LSN changes were caused by the SCSI inquiries failing against page code 0x80 and successful inquiries from page code 0x83.  As the LSN is different from each page code location, a different LSN is reported for the same disk.

We believe the SCSI inquiries initially failed against 0x80 page code due to the FD leak with vxconfigd, which ultimately triggered the false DCPA events.

 

 

 

Veritas has since developed an Emergency Engineering Binary (EEB) hot-fix via the NBAPP_EEB_ET4034781-3.3.0.1-3.x86_64.rpm to address the Veritas Volume Manager 7.2.0.x File Descriptor (FD) leak & prevent the False DCPA events for the internal NetBackup Appliance disks.

NBA Appliance enviroment details

nbapp-release = 3.3.0.1
builddate = 3.3.0.1-20201120090116
platform-version = 3.2.705-20201028135256
model = 5240
system-product-name = NetBackup Appliance 5240
service-pack-release = 1

Please contact Veritas Technical Support to obtain the EEB eebinstaller_4034781_3_linuxR_x86 & EEB hot-fix NBAPP_EEB_ET4034781-3.3.0.1-3.x86_64.rpm

EEB Installers for: Etrack 4034781 EEB Version 3

Classic Installer(s)
eebinstaller_4034781_3_linuxR_x86

Appliance Installer(s)
NBAPP_EEB_ET4034781-3.3.0.1-3.x86_64.rpm

Once the EEB has been installed, the VRTSvxvm will list version 7.2.0.5401

/home/maintenance # rpm -qa |grep vxvm
VRTSvxvm-7.2.0.5401-RHEL7.x86_64
 

Veritas InfoScale VRTSvxvm-7.2.0.5401-RHEL7.x86_64.rpm has been ported to the Emergency Engineering Binary (EEB) hot-fix via Etrack 4034781.
 

Reference Veritas Incidents:

ET4034781
STESC-5775
 

How to install EEBs, HotFixes and Maintenance Releases on NetBackup Appliances
https://www.veritas.com/support/en_US/article.100023444

 

 

Netbackup Appliance (NBA) v3.3.0.1 running VRTSvxvm 7.2.0.5400 is reporting a File Descriptor (FD) Leaks in relation to the Veritas Volume Manager vxconfigd daemon. This is resulting in DCPA (Data Corruption Protection Activated) events causing the impacted DMPNODEs to be incorrectly disabled.

Sample FD leak messages for vxconfigd (from syslog): Apr 5 18:03:43 fred vxvm:vxconfigd: V-5-1-12223 Error in claiming /dev/sdbuz: Too many open files
Apr 5 18:03:46 fred vxvm:vxconfigd: V-5-1-14523 LUN serial number of the OS device path with device
Netbackup Appliance environment is also reporting repeated disk failures for the internal appliance disks, resulting in VxVM DCPA (Data Corruption Protection Activated) events.