Error Message

Filename: /var/adm/ras/vxconfigd.log



03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14001c: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140037: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140055: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14001b: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140036: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140054: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14001a: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140035: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14004b: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140019: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140034: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14004a: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140018: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140033: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140049: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140017: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140032: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14003d: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140016: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140031: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14003c: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140015: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x140030: A file or directory in the path name does not exist.
03/02 12:50:43:  VxVM vxconfigd ERROR V-5-1-13834 ddl_add_to_conflict_list: Could not find entry in path list for 0x14003b: A file or directory in the path name does not exist.
03/02 12:53:45:  VxVM vxconfigd ERROR V-5-1-8561 ddl_fill_dmp_info: Path 0x14000f not found in devlist 13409: Error 0
03/02 12:58:17:  VxVM vxconfigd ERROR V-5-1-8561 ddl_fill_dmp_info: Path 0x14000f not found in devlist 13409: Error 0
03/02 13:01:08:  VxVM vxconfigd ERROR V-5-1-8561 ddl_fill_dmp_info: Path 0x14000f not found in devlist 13409: Error 0



 

IMPORTANT 

The required LUN removal steps were NOT conducted in relation to deleting the OS device handles when relocating the enclosure via a NEW switch port.
It is essential that VxVM and DMP are both notified of such hardware reconfiguration events prior to disconnecting the enclosure from the original switch port and reconnecting via a different switch port.

The following steps are required to minimize the chances of vxconfigd crashing, or a potential panic of the server where the enclosure is connected to via an alternate switch port.


Device clean-up steps


1.] Inform VxVM and DMP that the devices are to be removed from the server.

# vxdisk rm

2.] Physically disconnect the cable from the existing switch port.

3.] Delete the OS device handle.

# rmdev -dl

4.] Confirm the devices do not reappear following a refresh of the OS.

# cfgmgr

# lsdev -Cc disk

5.] Refresh Veritas Volume Manager (VxVM) and Veritas DMP of the reconfiguration event.

# vxdisk scandisks

6.] Refresh the /etc/vx/disk.info and /etc/vx/array.info files.

It is possible with the introduction of 5.0 MP3 to dynamically refresh the contents of /etc/vx/disk.info without the need to delete the file and recycle vxconfigd.

# vxddladm assign names                             

With the introduction of 5.1 SP1 it is now possible to refresh the contents of the /etc/vx/array.info without the need to delete the file and recycle vxconfigd.

# vxddladm -c assign names

7.] Ensure VxVM cannot see the devices associated with the enclosure being relocated to a new switch

# vxdmpadm listenclosure all
# vxdisk -eo alldgs list

8.] If the removed devices are not seen by VxVM and DMP, reconnect the cable to the NEW switch port.

9.] Refresh the OS and VxVM using the "vxdiskconfig" command, this will execute the "cfgmgr" and "vxdctl enable" command.

# vxdiskconfig

10.] Confirm that the previous enclosure is now visible to the server
 
# vxdmpadm listenclosure all

11.] Confirm that VxVM is capable of seeing the newly presented devices via the new switch port

# vxdisk -eo alldgs list

Process complete.

 
Note: Do not delete the directory contents of /dev/vx/[r]dmp, as this was a workaround used prior to the 5.0 MP3 product release, and is no longer nor recommended to resolve DMP incore database related issues.
 

Applies To

AIX 5.3 running Veritas Volume Manager 5.0 MP3 RP3


How to use the "gdb" debug utilty on AIX_to display the stack trace for the vxconfigd (application) core dump


# /usr/local/bin/gdb -q

(gdb) set solib-absolute-prefix  /snapcore

(gdb) core-file ./core
Core was generated by `vxconfigd.5.3'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000e0a0 in ?? ()

(gdb) file ./sbin/vxconfigd.5.3
Reading symbols from /snapcore/sbin/vxconfigd.5.3...(no debugging symbols found)...done.

(gdb) info sharedlibrary
No shared libraries loaded at this time.

(gdb) bt
#0  0x0000e0a0 in ?? ()
#1  0x10015eb8 in xalloc_internal ()
#2  0x1008fb80 in ddl_delete_node ()
#3  0x1008f3b8 in ddl_migration_devlist_found ()
#4  0x1008faf8 in ddl_cmp_devlist ()
#5  0x10090730 in ddl_check_migration_of_devices ()
#6  0x10093408 in ddl_reconfigure_all ()
#7  0x1006a3a0 in ddl_find_devices_in_system ()
#8  0x1009b0e0 in find_devices_in_system ()
#9  0x1009b444 in mode_set ()
#10 0x10001794 in setup_mode ()
#11 0x10001d4c in startup ()
#12 0x10001498 in main ()




How to use the "dbx" debug utility on AIX to display the stack trace for the vxconfigd (application) core dump

# dbx -p /usr=/snapcore/usr sbin/vxconfigd.5.3 ./core
Type 'help' for help.
[using memory image in ./core]
reading symbolic information ...warning: cannot open /etc/vx/lib/discovery.d/libvxxp256.so
warning: cannot open /etc/vx/lib/discovery.d/libvxxp12k.so
warning: cannot open /etc/vx/lib/discovery.d/libvxvscsi.so
warning: cannot open /etc/vx/lib/discovery.d/libvxvpath.so
warning: cannot open /etc/vx/lib/discovery.d/libvxshark.so
warning: cannot open /etc/vx/lib/discovery.d/libvxpurple.so
warning: cannot open /etc/vx/lib/discovery.d/libvxpp.so
warning: cannot open /etc/vx/lib/discovery.d/libvxibmsvc.so
warning: cannot open /etc/vx/lib/discovery.d/libvxibmds8k.so
warning: cannot open /etc/vx/lib/discovery.d/libvxibmds6k.so
warning: cannot open /etc/vx/lib/discovery.d/libvxibmds4k.so
warning: cannot open /etc/vx/lib/discovery.d/libvxhpalua.so
warning: cannot open /etc/vx/lib/discovery.d/libvxhitachi.so
warning: cannot open /etc/vx/lib/discovery.d/libvxhdsusp.so
warning: cannot open /etc/vx/lib/discovery.d/libvxhdsalua.so
warning: cannot open /etc/vx/lib/discovery.d/libvxhds9980.so
warning: cannot open /etc/vx/lib/discovery.d/libvxhds.so
warning: cannot open /etc/vx/lib/discovery.d/libvxemc.so
warning: cannot open /etc/vx/lib/discovery.d/libvxcscovrts.so
warning: cannot open /etc/vx/lib/discovery.d/libvxCLARiiON.so
warning: cannot open /etc/vx/lib/discovery.d/libvx3par.so
warning: "/snapcore/usr/lib/libC.a(shrcore.o)" has changed since core creation
warning: "/snapcore/usr/lib/libC.a(ansicore_32.o)" has changed since core creation
warning: "/snapcore/usr/lib/libC.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libiconv.a(shr4.o)" has changed since core creation
warning: "/snapcore/usr/lib/libi18n.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/nls/loc/en_US" has changed since core creation
warning: "/snapcore/usr/lib/libc.a(pse.o)" has changed since core creation
warning: "/snapcore/usr/lib/libpthreads.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libtli.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libpthreads_compat.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libcrypt.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/liblvm.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libcfg.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libodm.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libnsl.a(shr.o)" has changed since core creation
warning: "/snapcore/usr/lib/libpthread.a(shr_xpg5.o)" has changed since core creation
warning: "/snapcore/usr/lib/libpthread.a(shr_comm.o)" has changed since core creation
warning: "/snapcore/usr/lib/libc.a(shr.o)" has changed since core creation
warning: no source compiled with -g

pthdb_session.c, 833: 0 PTHDB_INTERNAL (internal error)
pthreaded.c, 1949: PTHDB_INTERNAL (internal error)

Segmentation fault in . at 0xe0a0
0x0000e0a0 warning: Unable to access address 0xe0a0 from core

(dbx) where
pthdb_session.c, 833: 0 PTHDB_INTERNAL (internal error)
pthreaded.c, 1949: PTHDB_INTERNAL (internal error)
.() at 0xe0a0
xalloc_internal() at 0x10015eb4
ddl_delete_node() at 0x1008fb7c
ddl_migration_devlist_found() at 0x1008f3b4
ddl_cmp_devlist() at 0x1008faf4
ddl_check_migration_of_devices() at 0x1009072c
ddl_reconfigure_all() at 0x10093404
ddl_find_devices_in_system() at 0x1006a39c
find_devices_in_system() at 0x1009b0dc
mode_set() at 0x1009b440
setup_mode() at 0x10001790
startup() at 0x10001d48
main() at 0x10001494

Alternatively, run dbx with:

# dbx -p /usr=./usr:/etc=./etc ./sbin/vxconfigd* ./core
Type 'help' for help.
[using memory image in ./core]
reading symbolic information ...warning: "./usr/lib/libC.a(shrcore.o)" has changed since core creation
warning: "./usr/lib/libC.a(ansicore_32.o)" has changed since core creation
warning: "./usr/lib/libC.a(shr.o)" has changed since core creation
warning: "./usr/lib/libiconv.a(shr4.o)" has changed since core creation
warning: "./usr/lib/libi18n.a(shr.o)" has changed since core creation
warning: "./usr/lib/nls/loc/en_US" has changed since core creation
warning: "./usr/lib/libc.a(pse.o)" has changed since core creation
warning: "./usr/lib/libpthreads.a(shr.o)" has changed since core creation
warning: "./usr/lib/libtli.a(shr.o)" has changed since core creation
warning: "./usr/lib/libpthreads_compat.a(shr.o)" has changed since core creation
warning: "./usr/lib/libcrypt.a(shr.o)" has changed since core creation
warning: "./usr/lib/liblvm.a(shr.o)" has changed since core creation
warning: "./usr/lib/libcfg.a(shr.o)" has changed since core creation
warning: "./usr/lib/libodm.a(shr.o)" has changed since core creation
warning: "./usr/lib/libnsl.a(shr.o)" has changed since core creation
warning: "./usr/lib/libpthread.a(shr_xpg5.o)" has changed since core creation
warning: "./usr/lib/libpthread.a(shr_comm.o)" has changed since core creation
warning: "./usr/lib/libc.a(shr.o)" has changed since core creation
warning: no source compiled with -g
pthdb_session.c, 833: 0 PTHDB_INTERNAL (internal error)
pthreaded.c, 1949: PTHDB_INTERNAL (internal error)
Segmentation fault in . at 0xe0a0
0x0000e0a0 warning: Unable to access address 0xe0a0 from core
(dbx)