Error Message

[root@server101 ~]# systemctl status veki.service
● veki.service - Systemd Veritas veki service
Loaded: loaded (/usr/lib/systemd/system/veki.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-03-04 12:47:32 CST; 7min ago
Process: 35428 ExecStart=/etc/vx/veki start (code=exited, status=100)

Mar 04 12:47:31 server101 systemd[1]: Starting Systemd Veritas veki service...
Mar 04 12:47:31 server101 veki[35428]: Starting veki /etc/vx/veki: line 50: [: /lib/modules/3.10.0-1160.49.1.el7.x86_64/v...xpected
Mar 04 12:47:32 server101 veki[35428]: ERROR: Old veki module is already loaded. Reboot to ensure new modules gets loaded.
Mar 04 12:47:32 server101 veki[35428]: - cannot find appropriate module.
Mar 04 12:47:32 server101 systemd[1]: veki.service: control process exited, code=exited status=100
Mar 04 12:47:32 server101 systemd[1]: Failed to start Systemd Veritas veki service.
Mar 04 12:47:32 server101 systemd[1]: Unit veki.service entered failed state.
Mar 04 12:47:32 server101 systemd[1]: veki.service fail

During system boot we can see below messages

-- Reboot --
Feb 03 22:17:12 server101 veki[15822]: Starting veki WARNING: No modules found for 3.10.0-1160.49.1.el7.x86_64, using compatible modules for 3.10.0-957.el7.x86_64.
Feb 03 22:18:39 server101 veki[21940]: Starting veki ERROR: No appropriate modules found.
Feb 03 22:18:39 server101 veki[21940]: Error in loading module "veki". See documentation.
Feb 03 22:18:39 server101 veki[21940]: - cannot find appropiate module.
Feb 03 23:02:16 server101 veki[59045]: Starting veki WARNING: No modules found for 3.10.0-1160.49.1.el7.x86_64, using compatible modules for 3.10.0-1160.el7.x86_64.
Feb 03 23:28:59 server101 veki[12814]: Stopping veki - cannot unload; veki is busy.

veki module library corruption

1. Download the Veritas software/patch which is currently installed on the server from the Veritas portal and copy it to the server having issues locally and extract it.

Download link: Veritas software download

2. Stop the cluster on the node where you are experiencing the problem

# hastop -local

3. Make sure no local vxfs filesystem also mounted on this nodes

# df -h | grep -i dsk

4. Stop the Veritas modules using the installer script to stop/unload all Veritas modules

# cd /opt/VRTS/install/
# ./installer -stop

5. Verify all modules unloaded

# lsmod | grep -i vx

6. Uninstall VRTSveki rpm

# rpm -e --nodeps

7. Switch to the directory where the software is downloaded, extract the software/patch and switch to rpms directory

8. Install the VRTSveki rpm back

# rpm -ivh --nodeps

9. Once installed, start Veritas using the installer script

# cd /opt/VRTS/install/
# ./installer -start 

10. Verify all modules came up and verify the cluster status

11. Reboot the server to verify the veki issue is resolved and the cluster starts.