VxVM file systems fail to mount when using Dell EMC ScaleIO storage

book

Article ID: 100062965

calendar_today

Updated On:

Description

Error Message

Although no errors are reported, it can be observed in the messages log that VxVM has already started while ScaleIO has not yet completed the startup process.

Nov 22 08:21:34 xxxxxxxxxx scini[5193]: Loading the best matching scini driver for your kernel for host RH, fallback host RH. <======== EMC ScaleIO driver starts
Nov 22 08:21:34 xxxxxxxxxx scini[5193]: The running kernel version is 4.18.0-477.27.1.el8_8.x86_64 Nov 22 08:21:34 xxxxxxxxxx scini[5193]: Didn't find a perfect match in /bin/emc/scaleio/scini_sync/driver_cache. Nov 22 08:21:34 xxxxxxxxxx scini[5193]: Attempting to fetch a perfect match to /bin/emc/scaleio/scini_sync/driver_cache.

Nov 22 08:21:35 xxxxxxxxxx automount[5229]: " Nov 22 08:21:35 xxxxxxxxxx scini[6049]: Error (sftp): File /etc/emc/scaleio/scini_sync/scini_key must exist. Nov 22 08:21:35 xxxxxxxxxx scini[6051]: driver_sync.sh END - Wed Nov 22 08:21:35 EST 2023 Nov 22 08:21:35 xxxxxxxxxx scini[5193]: driver_sync failed! Please check driver_sync.log Nov 22 08:21:35 xxxxxxxxxx scini[5193]: No match after driver_sync fetch attempt. Nov 22 08:21:35 xxxxxxxxxx scini[5193]: Attempting to find a best match... Nov 22 08:21:35 xxxxxxxxxx systemd[1]: Started Automounts filesystems on demand.

..

Nov 22 08:21:37 xxxxxxxxxx illumio-ven[5244]: - venAgentMonitor already started Nov 22 08:21:37 xxxxxxxxxx systemd[1]: Started Illumio VEN Agent top level startup. Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Started Systemd Veritas xprtld service. Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Starting Systemd Veritas veki service... Nov 22 08:21:38 xxxxxxxxxx kernel: veki: loading out-of-tree module taints kernel. Nov 22 08:21:38 xxxxxxxxxx kernel: veki: Successfully loaded Nov 22 08:21:38 xxxxxxxxxx veki[17822]: Starting veki Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Started Systemd Veritas veki service. Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Starting Veritas Asynchronous Monitoring Framework (AMF) Driver... Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Starting VERITAS Volume Manager Boot service... <==== Volume manager starting
Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Starting VERITAS Low Latency Transport (LLT)... Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Starting Systemd Veritas vxfs-boot service... Nov 22 08:21:38 xxxxxxxxxx amf[18288]: Starting AMF... Nov 22 08:21:38 xxxxxxxxxx llt[18302]: Starting LLT: Nov 22 08:21:38 xxxxxxxxxx scini[5193]: Attempting to load /bin/emc/scaleio/scini_sync/driver_cache/RH/3.6.600.113/4.18.0-348.el8.x86_64/scini.ko (version: 4.18.0-348.el8.x86_64)... <=== ScaleIO still starting
Nov 22 08:21:38 xxxxxxxxxx llt[18302]: LLT: loading module... Nov 22 08:21:38 xxxxxxxxxx kernel: vxfs: msgcnt 1 mesg 139: V-2-139: Loading VxFS version VxFS 7.4.2.4400 (Linux 2.6.x) Nov 22 08:21:38 xxxxxxxxxx kernel: vxfs : veki initialized. Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Started Systemd Veritas vxfs-boot service. <== VxFS started indicating VM has successfully started
Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Started Systemd vxfs-replication service. Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Starting Systemd fsdedupschd service... Nov 22 08:21:38 xxxxxxxxxx vxvm-boot[18297]: Starting up VxVM: Nov 22 08:21:38 xxxxxxxxxx kernel: registered amf_api with veki Nov 22 08:21:38 xxxxxxxxxx kernel: AMF Driver version 7.4.2.3300 loaded Nov 22 08:21:38 xxxxxxxxxx amf[18288]: Loaded 4.18.0-477.10.1.el8_8.x86_64 on kernel 4.18.0-477.27.1.el8_8.x86_64 Nov 22 08:21:38 xxxxxxxxxx amf[18288]: WARNING: No modules found for 4.18.0-477.27.1.el8_8.x86_64, using compatible modules for 4.18.0-477.10.1.el8_8.x86_64. Nov 22 08:21:38 xxxxxxxxxx fsdedupschd[21731]: Starting Veritas VxFS de-duplication Scheduler:[ OK ] Nov 22 08:21:38 xxxxxxxxxx systemd[1]: Started Systemd fsdedupschd service. Nov 22 08:21:38 xxxxxxxxxx kernel: VxVM vxdmp V-5-0-1990 driver version 7.4.2.4300 installed Nov 22 08:21:38 xxxxxxxxxx kernel: Nov 22 08:21:38 xxxxxxxxxx vxvm-boot[21946]: Refreshing key links for APMs Nov 22 08:21:38 xxxxxxxxxx vxfs_replication[21716]: Starting scheduler...[ OK ] Nov 22 08:21:38 xxxxxxxxxx vxfstaskd[25691]: Scheduler started with pid 25691. Nov 22 08:21:39 xxxxxxxxxx vxfs_replication[21716]: Starting replication daemon...[ OK ] Nov 22 08:21:39 xxxxxxxxxx kernel: PowerFlex drvMain_Init:61 :Driver starting load process. HZ 1000... Nov 22 08:21:39 xxxxxxxxxx kernel: PowerFlex mosCpu_Init:80 :eax=00000960, ebx=00000dac, ecx=00000064, edx=00000000 Nov 22 08:21:39 xxxxxxxxxx kernel: PowerFlex mosT10Dif_Init:933 :(T10DIF) T10DIF layer using CPU (PCLMUL) acceleration !!! Nov 22 08:21:39 xxxxxxxxxx kernel: PowerFlex drvMain_Init:63 :Load process done. Status SUCCESS Nov 22 08:21:39 xxxxxxxxxx scini[5163]: Success loading module /bin/emc/scaleio/scini.ko Nov 22 08:21:39 xxxxxxxxxx scini[5163]: Found device special file /dev/scini Nov 22 08:21:39 xxxxxxxxxx kernel: VxVM vxio V-5-0-1990 driver version 7.4.2.4300 installed Nov 22 08:21:39 xxxxxxxxxx kernel: VxVM vxspec V-5-0-1990 driver version 7.4.2.4300 installed Nov 22 08:21:39 xxxxxxxxxx vxvm-boot[30112]: cat: /proc/sys/kernel/hotplug: No such file or directory Nov 22 08:21:39 xxxxxxxxxx vxvm-boot[21946]: VxVM general startup... Nov 22 08:21:39 xxxxxxxxxx kernel: PowerFlex mapMdmLayer_SetIniGUID:6107 :No such SDC instance, allocate one.. Nov 22 08:21:39 xxxxxxxxxx kernel: PowerFlex ioctl_HandleSetIniGuid:644 :Set GUID (35908BB8-9DD7-436D-B326-150093745E95) finished . status 65 (SUCCESS) Nov 22 08:21:39 xxxxxxxxxx scini[28737]: Loading MDMs from configuration file /etc/emc/scaleio/drv_cfg.txt Nov 22 08:21:39 xxxxxxxxxx scini[28737]: Loading MDMs from configuration file /etc/emc/scaleio/drv_cfg.txt Nov 22 08:21:39 xxxxxxxxxx scini[28737]: Successfully set GUID 35908BB8-9DD7-436D-B326-150093745E95 Nov 22 08:21:39 xxxxxxxxxx scini[28737]: Successfully add mdm from cfg line 10 Nov 22 08:21:39 xxxxxxxxxx kernel: PowerFlex ioctl_HandleAddMdm:469 :Create finish for MDM. First I

...

Nov 22 08:21:44 xxxxxxxxxx scini[5163]: scinia is not ready xxxxxxxxxx... <====== ScaleIO still not started

Cause

VxVM comes online before ScaleIO storage

Resolution

This issue has been fixed in the following patches:

• For InfoScale 7.4.2 RHEL8 a private hotfix is available ( vm-rhel8_x86_64-HotFix-7.4.2.5302 ) and the fix will be included in the forthcoming update 8 patch. https://downloads.infoscale.com/infoscale/REL600675?q=5600•

For InfoScale 8.0.2 the fix is included in the update4 patch (eg https://downloads.infoscale.com/infoscale/REL333811?q=2400 ) and will be part of the all the future cumulative patches as well(eg : update 5,6,7 etc)
• For InfoScale 8.0 the fix is included in the update3 patch (eg https://downloads.infoscale.com/infoscale/REL150377?q=3300&fileCategory=Update&platform=Linux )and will be part of the all the cumulative patches released afterwards (eg : update 4,5,6 etc)

It is recommended that the above patches be installed.

If the patches are not installed, then the following workaround will need to be implemented. The workaround was developed to create a systemd service that will run at the last stage of boot, i.e. after both VxVM and sciniX devices are ready.

1) Create a systemd target /etc/systemd/system/last.target

vi /etc/systemd/system/last.target
[Unit] Description=Last Target Requires=multi-user.target After=multi-user.target AllowIsolate=yes

2) Create a systemd service that will run at the last stage of boot.

vi /etc/systemd/system/vxvm-fstab-autoimport.service
[Unit] Description=vxvm-fstab-autoimport After=multi-user.target

[Service] Type=oneshot ExecStart=/etc/vx/init.d/vxvm-fstab-autoimport

[Install] WantedBy=last.target

3) Create the script /etc/vx/init.d/vxvm-fstab-autoimport

vi /etc/vx/init.d/vxvm-fstab-autoimport
#!/bin/bash

VXTUNE=/usr/sbin/vxtune AWK=/usr/bin/awk VXRECOVER=/usr/sbin/vxrecover FSTAB=/etc/fstab BIN=/usr/bin GREP=$BIN/grep LOGFILE=/var/adm/vx/automount.log

echo "$(date)" > $LOGFILE

while read line do fs_spec=$(echo $line | awk '{print $1}') [ ${fs_spec:0:1} == "#" ] && continue [[ ${fs_spec:0:11} != "/dev/vx/dsk" ]] && continue dg=$(echo $fs_spec | sed 's/\// /g' | $AWK '{print $4}') vol=$(echo $fs_spec | sed 's/\// /g' | $AWK '{print $5}')

fs_file=$(echo $line | awk '{print $2}') fs_vfstype=$(echo $line | awk '{print $3}') fs_mntops=$(echo $line | awk '{print $4}')

noauto=$(echo ${fs_mntops} | grep -c noauto) [[ ${noauto} -eq 1 ]] && continue

mounted=$(mount | awk '{print $1}' | grep -wc ${fs_spec}) [[ ${mounted} -eq 1 ]] && continue

imported=$(vxdg -q list | grep -wc $dg); if [[ ${imported} -eq 0 ]] then /usr/sbin/vxdg import $dg fi

/usr/bin/mount -t ${fs_vfstype} -o ${fs_mntops} ${fs_spec} ${fs_file} 2>&1 mounted=$(mount | awk '{print $1}' | grep -wc ${fs_spec}) if [[ ${mounted} -eq 0 ]] then echo "VxVM: Auto-mount of ${fs_spec} on ${fs_file} failed!" >> $LOGFILE continue fi

echo "VxVM: ${fs_spec} mounted on ${fs_file}" >> $LOGFILE done < /etc/fstab

4) Enable execute permissions on the script.

chmod a+x /etc/vx/init.d/vxvm-fstab-autoimport

5) Enable the service and set /etc/systemd/system/last.target as default target.

mkdir /etc/systemd/system/last.target.wants

ln -s /etc/systemd/system/vxvm-fstab-autoimport.service /etc/systemd/system/last.target.wants/vxvm-fstab-autoimport.service

systemctl daemon-reload

systemctl set-default last.target

6) Add the following rule to /etc/udev/rules.d/40-VxVM.rules to create a soft path link to sciniX devices

KERNEL=="scini*[!0-9]", SUBSYSTEM=="block", RUN+="/lib/udev/vxpath_links %k"

7) Add the following rule to /etc/udev/rules.d/20-scini.rules to initiate VxVM device discovery.

ACTION=="add", DEVPATH=="scini*[!0-9]" RUN+="/usr/sbin/vxdisk scandisks new"

8) Reboot to verify VxVM diskgroups are automatically imported and file systems are mounted.

Jan 18 16:22:32 xxxxxxxxx systemd: Starting vxvm-fstab-autoimport... Jan 18 16:22:32 xxxxxxxxx systemd: Started Update UTMP about System Runlevel Changes. Jan 18 16:22:32 xxxxxxxxx CmdServer[7110]: VCS INFO V-16-1-11240 Command Server: running with security OFF Jan 18 16:22:32 xxxxxxxxx systemd: Started VCS Command Server. Jan 18 16:22:33 xxxxxxxxx systemd: Starting File System Check on /dev/vx/dsk/mydg/myvol... Jan 18 16:22:33 xxxxxxxxx systemd: Started File System Check on /dev/vx/dsk/mydg/myvol. Jan 18 16:22:33 xxxxxxxxx systemd: Started vxvm-fstab-autoimport. Jan 18 16:22:33 xxxxxxxxx systemd: Reached target Last Target.

Issue/Introduction

When using Dell EMC ScaleIO storage, VxFS file systems on VxVM volumes fail to mount.

Additional Information

ETrack: 4153457

Was this article helpful?

thumb_up Yes

thumb_down No

Welcome to "KB Articles"