Error Message

# ./vxkextadm vxfs unload
== Checking dependant drivers: vxodm qio portal dmsample vhsm
Kernel extension /usr/lib/drivers/vxodm.ext_61 is not loaded   
Kernel extension /usr/lib/drivers/vxqio.ext_61 is not loaded
Kernel extension /usr/lib/drivers/vxportal.ext_61 is not loaded
Kernel extension /usr/lib/drivers/dmsample.ext_61 is not loaded
Kernel extension /usr/lib/drivers/vhsm.ext_61 is not loaded
== Unload VxFS
kextconfig: sysconfig: Device busy                <<<< Device busy
/etc/methods/vxcfg: failed to deconfigure kext /usr/lib/drivers/vxfs.ext_61         <<< failed

You may found similar messages in the Veritas SF installation log file (e.g. /opt/VRTS/install/logs/installmr-"date"/stop.vxfs."system name").

(Please note that if one checks the genkex output, genkex may report that the dependent drivers may still exist.   But this is not the cause of the vxkextadm command failure and can be safely ignored.   Once all mounted VxFS filesystems are unmounted, the vxfs kernel driver can be unloaded successfully.

# genkex | grep vx
f1000000c02cd000    27000 /usr/lib/drivers/vxodm.ext_61     <<< not a cause of the vxfs module unload failure
         53c0000    20000 /usr/lib/drivers/vxspec
         4f80000   440000 /usr/lib/drivers/vxio
         4ee0000    a0000 /usr/lib/drivers/vxdmp
f1000000c0286000    3d000 /usr/lib/drivers/vxgms.ext_61
         4cc0000    20000 /usr/lib/drivers/vxqio.ext_61
         4980000   330000 /usr/lib/drivers/vxfs.ext_61
f1000000c0242000    44000 /usr/lib/drivers/vxglm.ext
)

The VxFS module unload failure is caused by the kernel data structures still referencing the VxFS kernel module.

The following is a demonstration on how the problem can happen on AIX.

Mount a VxFS file system
=================== 

# mount -V vxfs /dev/vx/dsk/datadg/voltmp /voltmp

# df -k /voltmp
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/vx/dsk/datadg/voltmp      204800    188995    8%        4     1% /voltmp

# mount | grep voltmp
         /dev/vx/dsk/datadg/voltmp /voltmp          vxfs   Oct 15 17:14 rw,delaylog,suid,ioerror=mwdisable,qio,largefiles

The kernel debugger shows the mount entry in the kernel
==========================================

# kdb
.....
(0)> mount
                          GFS             DATA  TYPE    FLAGS
 22 F1000A01000D6010 028ACAF8 F1000A0201C21900  VXFS    DEVMOUNT
... /dev/vx/dsk/datadg/voltmp mounted over /voltmp

Create a process to use the file system
=============================

# sleep 10000000 > /voltmp/myfile &
[1]     23265490

Forcibly unmounts the file system
==========================

# umount /voltmp
UX:vxfs umount: ERROR: V-3-21705: /voltmp cannot unmount : Device busy

# echo $?
1

# umount -f /voltmp
forced unmount of /voltmp

# echo $?
0

# mount | grep voltmp
#

# fuser /voltmp
/voltmp:

# df -k /voltmp
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4         26214400   3944028   85%    37335     5% /

The process is still running.

# ps -eaf | grep 23265490
    root 23265490 17891542   0 17:16:33  pts/2  0:00 sleep 10000000

"umount -f" reported that it successfully unmounted the file system.  The "mount", "fuser" and "df" commands don't show the file system anymore.  Although the file system is unmounted, actually the VxFS kernel module is still being used by the above process.

Checking the kernel mount table using kdb
================================

# kdb
.....
(0)> mount
                          GFS             DATA  TYPE    FLAGS
 22 F1000A01000D6010 028ACAF8 F1000A0201C21900 VXFS    DEVMOUNT UNMOUNTING UNMOUNTED
... /dev/vx/dsk/datadg/voltmp mounted over /voltmp

Also lsof shows that the process is still using the file system.

# lsof | grep voltmp
COMMAND        PID     USER   FD   TYPE             DEVICE             SIZE/OFF       NODE NAME
sleep     23265490     root    1w  VREG       NFS,11537536 17365891166852687872 4043311618 /voltmp (/dev/vx/dsk/datadg/voltmp)

 

 

Please beware that forcibly unmounting a VxFS file system will not clean up the kernel data structures referencing the VxFS kernel module.   Forcibly umounting the VxFS file systems is not the correct way to free up the VxFS kernel module and prepare the module for unloading.    In order to unload the VxFS kernel module, please always shutdown the applications using the VxFS file systems, terminate the left behind processes (if any) and unmounts the VxFS file systems WITHOUT forcing it.

If you run into a situation where the VxFS kernel module cannot be unloaded and VxFS file systems were forcibly unmounted before, please verify if there are any processes still using those forcibly unmounted VxFS file systems.  If so, please shutdown all the relevant applications and kill the left behind processes if any.

lsof will be a very useful utility to identify the processes still using the VxFS file systems on AIX.

Run lsof and grep for /dev/vx to find all the processes that may still be using the VxFS file systems.   (Here we assume that all the VxVM volumes are with VxFS file system created.   Please confirm that before killing the processes.)  For example,

# lsof | grep /dev/vx | more
COMMAND        PID     USER   FD   TYPE             DEVICE             SIZE/OFF       NODE NAME
sleep     23265490     root    1w  VREG           37,38001                    0 4043311618 /voltmp (/dev/vx/dsk/datadg/voltmp)

Based on the above demonstration of the problem, the following process is killed.

# ps -eaf | grep 23265490
    root 23265490 17891542   0 17:16:33  pts/2  0:00 sleep 10000000

# kill -15 23265490
[1] + Terminated               sleep 10000000 > /voltmp/myfile &

Check with kdb and the mount entry (entry number 22) for /voltmp is now removed from the kernel.

# kdb 
(0)> mount
                           GFS             DATA TYPE    FLAGS
.....
 21 F1000A01000D5D10 028ACAF8 F1000A0201E39580 VXFS    DEVMOUNT
... /dev/vx/dsk/arch-bkpdg/archvol mounted over /arch
                      <<< entry 22 is gone
(0)>

Run lsof again and make sure that mount points and block devices belong to VxFS file systems don't exist.

# lsof | grep /dev/vx
#

With all the VxFS file system mount entries removed from kernel, vxkextadm will be able to unload the vxfs module successfully.

# /etc/methods/vxkextadm vxfs unload
== Checking dependant drivers: vxodm qio portal dmsample vhsm
Kernel extension /usr/lib/drivers/vxodm.ext_61 is not loaded
Kernel extension /usr/lib/drivers/vxqio.ext_61 is not loaded
Kernel extension /usr/lib/drivers/vxportal.ext_61 is not loaded
Kernel extension /usr/lib/drivers/dmsample.ext_61 is not loaded
Kernel extension /usr/lib/drivers/vhsm.ext_61 is not loaded
== Unload VxFS
/etc/methods/vxcfg: deconfigured and unloaded /usr/lib/drivers/vxfs.ext_61, kmid 0x50904000
== Unloaded

 

 

Applies To

Veritas File System on all supported platforms.

When the Veritas File System (VxFS) version needs to be upgraded or downgraded, or when the VxFS package needs to be removed, the corresponding Veritas Storage Foundation installation program (e.g. installmr, uninstallmr,installrp, uninstallrp, uninstallsf) will need to unload the VxFS kernel module. On AIX the unloading and loading of the VxFS kernel module are performed using the /etc/methods/vxkextadm program. One mandatory prerequisite for vxkextadm to successfully unload the VxFS kernel module is that all processes using the VxFS file systems must be terminated and all VxFS file systems must be unmounted. Forcibly unmounting a VxFS file system without first terminating processes using the VxFS file systems may leave kernel data structures still referencing the VxFS kernel module and prevent the VxFS kernel module from unloading.