Error Message

During the first reboot, a lot of Root filesystem space errors are followed by LLT link creation errors during loading of the LLT module.
 
Sep 18 20:37:56 server101 rsyslogd: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': No space left on device [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/2013 ]
..
Sep 18 20:37:58 server101 systemd: Starting VERITAS Low Latency Transport (LLT)...
Sep 18 20:37:58 server101 llt: Starting LLT:
Sep 18 20:37:58 server101 llt: LLT: loading module...
Sep 18 20:37:58 server101 llt: /bin/ln: failed to create symbolic link ‘/opt/VRTSllt/modules/llt.ko.3.10.0-1062.el7.x86_64’: No space left on device   <<<
Sep 18 20:37:58 server101 llt: /bin/ln: failed to create symbolic link ‘/opt/VRTSllt/modules/llt.ko.3.10.0-1127.el7.x86_64’: No space left on device  <<<<

In the next reboot or restarting the product, there are no space errors but the errors below are seen:

Sep 18 21:40:01 server101 llt: Starting LLT:
Sep 18 21:40:01 server101 amf: Starting AMF...
Sep 18 21:40:01 server101 llt: LLT: loading module...

Sep 18 21:40:02 server101 llt: ERROR: No appropriate modules found.
Sep 18 21:40:02 server101 llt: Error in loading module "llt". See documentation.
Sep 18 21:40:02 server101 llt: LLT:Warning: llt rdma module load failed, trying to load llt non-rdma module
Sep 18 21:40:02 server101 fsdedupschd: Starting Veritas VxFS de-duplication Scheduler:[  OK  ]r
Sep 18 21:40:02 server101 systemd: Started Systemd fsdedupschd service.
Sep 18 21:40:02 server101 llt: ERROR: No appropriate modules found.
Sep 18 21:40:02 server101 llt: Error in loading module "llt". See documentation.
Sep 18 21:40:02 server101 llt: LLT:Error: cannot find compatible module binary
Sep 18 21:40:02 server101 systemd: llt.service: main process exited, code=exited, status=1/FAILURE
Sep 18 21:40:02 server101 systemd: Failed to start VERITAS Low Latency Transport (LLT).
Sep 18 21:40:02 server101 systemd: Unit llt.service entered failed state.
Sep 18 21:40:02 server101 systemd: llt.service failed.
 

When the Root (/) filesystem is full, and either the system or the product is restarted, the LLT module attempts to create new links from existing links. However, the existing links are deleted because of /bin/ln -f -s command (used for creating new links and deleting old links at the same time.).

As the filesystem is full, it is unable to create new links. Even after sufficient space is made available, it fails to create the links as source/existing links are deleted,thereby preventing the LLT module from loading.

A supported hotfix has been made available for this issue. Please contact Veritas Technical Support to obtain this fix. This hotfix has not yet gone through any extensive Q&A testing. Consequently, if you are not adversely affected by this problem and have a satisfactory temporary workaround in place, we recommend that you wait for the public release of this hotfix.

Veritas Technologies LLC currently plans to address this issue by way of a patch or hotfix to the current version of the software. Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests. Veritas’ plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.

Please contact your Veritas Sales representative or the Veritas Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.

A workaround as below is also available for this issue: 

1) Confirm if the links are missing:

# rpm --verify VRTSllt
missing     /opt/VRTSllt/modules/llt.ko.3.10.0-1127.el7.x86_64

If the above command shows the links missing, continue further.

2) Check the files in the /opt/VRTSllt/modules/ folder.

# ls -l /opt/VRTSllt/modules/

total 80292
-rwxr-xr--. 1 root root 7573896 Mar 25  2022 llt.ko.3.10.0-1062.el7.x86_64-nonrdma
-rwxr-xr--. 1 root root 8988227 Mar 25  2022 llt.ko.3.10.0-1062.el7.x86_64-rdma
-rwxr-xr--. 1 root root 7581769 Mar 25  2022 llt.ko.3.10.0-1127.el7.x86_64-nonrdma
-rwxr-xr--. 1 root root 8996044 Mar 25  2022 llt.ko.3.10.0-1127.el7.x86_64-rdma
-rwxr-xr--. 1 root root 7585361 Mar 25  2022 llt.ko.3.10.0-1160.el7.x86_64-nonrdma
-rwxr-xr--. 1 root root 8997468 Mar 25  2022 llt.ko.3.10.0-1160.el7.x86_64-rdma
-rwxr-xr--. 1 root root 7377460 Mar 25  2022 llt.ko.3.10.0-862.el7.x86_64-nonrdma
-rwxr-xr--. 1 root root 8649336 Mar 25  2022 llt.ko.3.10.0-862.el7.x86_64-rdma
-rwxr-xr--. 1 root root 7579429 Mar 25  2022 llt.ko.3.10.0-957.el7.x86_64-nonrdma
-rwxr-xr--. 1 root root 8863946 Mar 25  2022 llt.ko.3.10.0-957.el7.x86_64-rdma

3) Create the links for the rdma or non-rdma based on the LLT configuration for the setup:

Below are links created for the rdma.
# ln -s /opt/VRTSllt/modules/llt.ko.3.10.0-1062.el7.x86_64-rdma  llt.ko.3.10.0-1062.el7.x86_64
# ln -s /opt/VRTSllt/modules/llt.ko.3.10.0-1127.el7.x86_64-rdma llt.ko.3.10.0-1127.el7.x86_64
# ln -s /opt/VRTSllt/modules/llt.ko.3.10.0-862.el7.x86_64-rdma llt.ko.3.10.0-862.el7.x86_64
# ln -s /opt/VRTSllt/modules/llt.ko.3.10.0-957.el7.x86_64-rdma llt.ko.3.10.0-957.el7.x86_64

4) Start the LLT module .

# systemctl start llt