Description

This article describes the steps involved with upgrading from InfoScale (IS) 7.3.1.x running Solaris 11.3 to InfoScale 7.4.2.x with Solaris 11.4 (SRU 42).

NOTE: Solaris zones are also presented to the Veritas Cluster Server (VCS) 2-node environment.

The process outlines the use of Alternate Boot Environments (ABEs) and the critical upgrade sequence.
 

Figure 1.0

Prior to upgrading the environment, Veritas strongly recommends performing a Full System Backup & capturing the VERITAS Explorer from all nodes to be upgraded.

In this instance, the Application will remain running on a single node in the 2-node VCS Cluster, we will refer to this node as the the Active node.

The standby node will be referred to as the Passive node.

To reduce the time taken to upgrade the Solaris version, Veritas recommends downloading the required Solaris ISO image and defining a local repository and publisher.
 

See Article
How to create a local repository (repo) for Solaris ISO image using ZFS zpool
https://supportinfoscale.cloud.com/support-home/kbsearch/article?articleNumber=100052743&articleTitle=how-to-create-a-local-repository-repo-for-solaris-iso-image-using-zfs-zpool& 

 

Solaris Boot Environments

Veritas strongly encourages customers to use the Solaris beadm utility to create and manage Alternate Boot Environments (ABEs).

Overview of Steps:

1.] Check the VCS Cluster Status before doing anything

# hastatus -sum

2.]  Confirm the Veritas packages Installed inside the Solaris Zones:
 

The below for loop uses the input from "zoneadm list -ci"

# for zone in `zoneadm list -ci | grep -v global | xargs -x`; do echo "ZONE: $zone"; pkg -R /export/zones/$zone/root/ list entire /VRTS*; done

3.] Set the solaris publisher for the locally created repository on both nodes

Summary of steps to mount Solaris ISO image located in /SOL114 named sol-11_4_42_113_1-incr-repo.iso:
:

# lofiadm -a /SOL114/sol-11_4_42_113_1-incr-repo.iso
# mkdir /cdrom114
# mount -o ro -F hsfs /dev/lofi/1 /cdrom114
 

Sample syntax:

# pkg set-publisher -g file:///Repo_Path/repo solaris

Define solaris publisher using Solaris ISO image mounted on /cdrom114
 

# pkg set-publisher -g file:///cdrom114/repo solaris

# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
help
solaris                     origin   online F file:///cdrom114/repo/

Check the required SRU version availability using the local repository:

# pkg list -af entire
NAME (PUBLISHER)                                  VERSION                    IFO
entire (solaris)                                  11.4-11.4.42.0.1.113.1     ---
entire (solaris)                                  0.5.11-0.175.3.36.0.23.0   i--
 

NOTE: In this instance, we require "SRU 11.4-11.4.42.0.1.113.1"
    

4.] Disable the solaris publisher on both nodes

NOTE: This is required to reduce the time required to install InfoScale packages

Known issue: JIRA: SDSCPE-6708
InfoScale Upgrade takes 4+ Hours to complete. After disabling solaris publisher only took 1.5 Hours
 

# pkg set-publisher --disable solaris

5.] Make VCS Configuration read-writable, freeze all VCS Service Groups persistently Read Only
 

Make the VCS Configuration read-write (rw)

# haconf -makerw
 

- Freeze all Service Groups

# for SG in `hagrp -state | grep -v "#Group" | awk '{ print $1 }' | sort | uniq | xargs -x`; do echo $SG; hagrp -freeze $SG -persistent ;done
 

Verify the Service Groups have been frozen (increase the +36 value as applicable with ggrep)

# hastatus -sum | ggrep -A +36 "GROUPS FROZEN"
 

Make the VCS Configuration read-only

# haconf -dump -makero

6.] Create a new BE with the same name on both clustered nodes

# beadm list
BE        Flags Mountpoint Space   Policy Created
--        ----- ---------- -----   ------ -------
solaris   -     -          171.23M static 2022-03-17 19:35
solaris-1 NR    /          29.27G  static 2022-03-19 16:29

NOTE: The solaris publisher must be disabled prior to creating the Alternate Boot Environment (ABE)
 

# beadm create IS742-1

# beadm list
BE        Flags Mountpoint Space   Policy Created
--        ----- ---------- -----   ------ -------
IS742-1   -     -          163.49M static 2022-03-24 18:53
solaris   -     -          171.23M static 2022-03-17 19:35
solaris-1 NR    /          29.28G  static 2022-03-19 16:29

7.] Mount newly created BE on both nodes using the same mount-point

# mkdir /IS
# beadm mount IS742-1 /IS

# beadm list
BE        Flags Mountpoint Space   Policy Created
--        ----- ---------- -----   ------ -------
IS742-1   -     /IS        163.49M static 2022-03-24 18:53
solaris   -     -          171.23M static 2022-03-17 19:35
solaris-1 NR    /          29.28G  static 2022-03-19 16:29
 

8.] Download the base 7.4.2 InfoScale software along with the latest InfoScale Solaris Update bundle

At the time of writing this article the latest update was "7.4.2 update2"

SOLARIS11 SPARC patch for 7.4.2 Update2 (infoscale-sol11_sparc-Patch-7.4.2.1300.tar.gz)

9.] Download the latest Veritas CPI patch

In this instance, we will be using "CPI_7.4.2_P20"

10.] The InfoScale s/w & patches can be placed under different directories on a single node
 

InfoScale Software has been downloaded and extracted under the directory /IS742 on Node2

# pwd
/IS742

# ls -al
total 1071148
drwxr-xr-x   5 root     root           6 Mar 24 18:22 .
drwxr-xr-x  28 root     sys           34 Mar 24 18:55 ..
drwxr-xr-x   4 root     root           7 Mar 24 17:01 7.4.2-Patch
drwxr-xr-x   2 root     root           3 Mar 24 18:22 CPI
drwxr-xr-x   3 root     root           3 May 11  2020 dvd1-sol_sparc
-rw-r--r--   1 root     root     547995422 May 11  2020 Veritas_InfoScale_7.4.2_Solaris_SPARC.tar.gz

The infoscale-sol11_sparc-Patch-7.4.2.1300 tar file is extracted to /IS742/7.4.2-Patch

# pwd
/IS742/7.4.2-Patch

# ls -al
total 631407
drwxr-xr-x   4 root     root           7 Mar 24 17:01 .
drwxr-xr-x   5 root     root           6 Mar 24 18:22 ..
-rw-r--r--   1 root     root     322958157 Mar  4 21:56 infoscale-sol11_sparc-Patch-7.4.2.1300.tar.gz
-rwxr-xr-x   1 root     root        8199 Feb 21 23:22 installVRTSinfoscale742P1300
drwxrwxr-x   3 root     root          35 Feb 21 10:51 patches
-rw-r--r--   1 root     root       59597 Feb 28 22:55 README
drwxr-xr-x   7 root     root           8 Feb 21 23:23 scripts


The CPI patch is extracted to /IS742/CPI

# ls -al
total 523
drwxr-xr-x   2 root     root           3 Mar 24 18:22 .
drwxr-xr-x   5 root     root           6 Mar 24 18:22 ..
-rwxr-xr-x   1 root     root      187944 Mar 24 16:27 CPI_7.4.2_P20.pl

 

11.] Start the CPI installer from the base 7.4.2 sol11_sparc directory on Node 2

# pwd
/IS742/dvd1-sol_sparc/sol11_sparc

# ls -al
total 49
drwxrwxr-x   9 root     root          11 May 11  2020 .
drwxr-xr-x   3 root     root           3 May 11  2020 ..
-rw-r--r--   1 root     root         934 May 11  2020 copyright
drwxrwxr-x   2 root     root           3 May 11  2020 docs
drwxr-xr-x   5 root     root           5 May 11  2020 EULA
-rwxr-xr-x   1 root     root        8850 May  5  2020 installer
drwxr-xr-x   5 root     root           5 May 11  2020 perl
drwxrwxr-x   3 root     root           4 May 11  2020 pkgs
drwxrwxr-x   7 root     root          19 May 11  2020 scripts
drwxrwxr-x   3 root     root           7 May 11  2020 tools
drwxrwxr-x   2 root     root           3 May 11  2020 windows

# date ; time ./installer -patch_path /IS742/7.4.2-Patch/ -upgrade -rootpath /IS -require /IS742/CPI/CPI_7.4.2_P20.pl node1 node2

                                                     Veritas InfoScale Storage and Availability Solutions 7.4.2 Live Upgrade Program
                                                                            node1 node2

NOTE: Keep a record of the install directory for the CPI installer
 

12.] Once the InfoScale upgrade is complete, verify that the Veritas packages have also been updated across the available Solaris Zones

NOTE: Precede the /export/zones path with the mount-point for the BE mount-point (i.e. /IS)

# for zone in `zoneadm list -ci | grep -v global | xargs -x`; do echo "ZONE: $zone"; pkg -R /IS/export/zones/$zone/root/ list entire /VRTS*; done

Sample Output

ZONE: ldom05_zone1
NAME (PUBLISHER)                                  VERSION                    IFO
VRTSodm (Veritas)                                 7.4.2.2500                 i--
VRTSperl (Veritas)                                5.30.0.0                   i--
VRTSpython (Veritas)                              3.7.4.35                   i--
VRTSvcs (Veritas)                                 7.4.2.1200                 i--
VRTSvcsag (Veritas)                               7.4.2.1200                 i--
VRTSvcsea (Veritas)                               7.4.2.1200                 i--
VRTSvlic (Veritas)                                4.1.742.300                i--
VRTSvxfs (Veritas)                                7.4.2.2500                 i--
entire (solaris)                                  0.5.11-0.175.3.36.0.23.0   i--
ZONE: ldom05_zone2
NAME (PUBLISHER)                                  VERSION                    IFO
VRTSodm (Veritas)                                 7.4.2.2500                 i--
VRTSperl (Veritas)                                5.30.0.0                   i--
VRTSpython (Veritas)                              3.7.4.35                   i--
VRTSvcs (Veritas)                                 7.4.2.1200                 i--
VRTSvcsag (Veritas)                               7.4.2.1200                 i--
VRTSvcsea (Veritas)                               7.4.2.1200                 i--
VRTSvlic (Veritas)                                4.1.742.300                i--
VRTSvxfs (Veritas)                                7.4.2.2500                 i--
entire (solaris)                                  0.5.11-0.175.3.36.0.23.0   i--


13.] Now the InfoScale upgrade is complete, now enable the solaris publisher using the local repository created earlier on both nodes and for the BE mount-point (i.e. /IS)
 

# pkg set-publisher --enable solaris

# pkg publisher
PUBLISHER                   TYPE     STATUS P LOCATION
solaris                     origin   online F file:///cdrom114/repo/

# beadm list
BE                 Flags Mountpoint Space   Policy Created
--                 ----- ---------- -----   ------ -------
IS742-1            -     /IS        2.18G   static 2022-03-24 18:54
solaris            -     -          225.95M static 2022-03-17 19:40
solaris-1          NR    /          54.32G  static 2022-03-19 16:51
solaris-1-backup-1 -     -          164.15M static 2022-03-24 01:12

NOTE: The solaris publisher also needs to be enabled within BE  mounted mount-point (i.e. /IS)
 

# date ; time pkg -R /IS set-publisher --enable solaris
Thursday, March 24, 2022 10:01:01 PM IST

real    0m7.455s
user    0m7.091s
sys     0m0.361s
 

14.] Create new BE from the IS742-1 BE on both nodes prior to starting Solaris Update

# beadm list
BE                 Flags Mountpoint Space   Policy Created
--                 ----- ---------- -----   ------ -------
IS742-1            -     /IS        2.24G   static 2022-03-24 18:54
solaris            -     -          225.95M static 2022-03-17 19:40
solaris-1          NR    /          54.32G  static 2022-03-19 16:51
solaris-1-backup-1 -     -          164.15M static 2022-03-24 01:12

# beadm create -e IS742-1 IS742-2
# beadm list
BE                 Flags Mountpoint Space   Policy Created
--                 ----- ---------- -----   ------ -------
IS742-1            -     /IS        2.24G   static 2022-03-24 18:54
IS742-2            -     -          166.63M static 2022-03-24 22:02
solaris            -     -          225.95M static 2022-03-17 19:40
solaris-1          NR    /          54.32G  static 2022-03-19 16:51
solaris-1-backup-1 -     -          164.15M static 2022-03-24 01:12
 

15.] Update Solaris to 11.4 SRU 42 using the BE "IS742-1" mount-point "/IS"

NOTE: It is possible to update the OS simultaneously on both clustered nodes 

Check you have sufficient free space on / for both nodes

# df -h /
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris-1    98G    14G        38G    28%    /

# pkg list -af entire
NAME (PUBLISHER)                                  VERSION                    IFO
entire                                            11.4-11.4.42.0.1.113.1     ---
entire                                            0.5.11-0.175.3.36.0.23.0   i--

NOTE: Specify the BE mount-point (i.e. /IS) for the Solaris 11.4 Update
 

# date ; time pkg -R /IS update --accept entire@11.4-11.4.42.0.1.113.1
Thursday, March 24, 2022 10:09:50 PM IST
Creating Plan (Solver setup): -

16:] Once the Solaris Update has completed, confirm the Solaris version update has been reflected in each of the Solaris Zones for the mounted BE (i.e. /IS)
 

# for zone in `zoneadm list -ci | grep -v global | xargs -x`; do echo "ZONE: $zone"; pkg -R /IS/export/zones/$zone/root/ list entire; done

Sample Output:

ZONE: ldom06_zone1
NAME (PUBLISHER)                                  VERSION                    IFO
entire                                            11.4-11.4.42.0.1.113.1     i--
ZONE: ldom06_zone2
NAME (PUBLISHER)                                  VERSION                    IFO
entire                                            11.4-11.4.42.0.1.113.1     i--
 

17.] Create an additional BE

# beadm create -e IS742-1 IS742-3
# beadm list
BE                 Flags Mountpoint Space   Policy Created
--                 ----- ---------- -----   ------ -------
IS742-1            -     /IS        6.62G   static 2022-03-24 18:54
IS742-2            -     -          166.63M static 2022-03-24 22:02
IS742-3            -     -          156.13M static 2022-03-25 03:02
solaris            -     -          225.95M static 2022-03-17 19:40
solaris-1          NR    /          54.65G  static 2022-03-19 16:51
solaris-1-backup-1 -     -          164.15M static 2022-03-24 01:12


The InfoScale & Solaris Upgrade is now complete within the BE created.

Sample zoneadm output
 

# zoneadm list -civ
  ID NAME             STATUS      PATH                         BRAND      IP
   0 global           running     /                            solaris    shared
   1 ldom05_zone1     running     /export/zones/ldom05_zone1   solaris    excl
   2 ldom05_zone12    running     /export/zones/ldom05_zone12  solaris    excl
   3 ldom05_zone11    running     /export/zones/ldom05_zone11  solaris    excl
   4 ldom05_zone4     running     /export/zones/ldom05_zone4   solaris    excl
   5 ldom05_zone14    running     /export/zones/ldom05_zone14  solaris    excl
   7 ldom05_zone3     running     /export/zones/ldom05_zone3   solaris    excl
   8 ldom05_zone2     running     /export/zones/ldom05_zone2   solaris    excl
   9 ldom05_zone10    running     /export/zones/ldom05_zone10  solaris    excl
  11 ldom05_zone5     running     /export/zones/ldom05_zone5   solaris    excl
  12 ldom05_zone6     running     /export/zones/ldom05_zone6   solaris    excl
  13 ldom05_zone8     running     /export/zones/ldom05_zone8   solaris    excl
  14 ldom05_zone7     running     /export/zones/ldom05_zone7   solaris    excl
  15 ldom05_zone9     running     /export/zones/ldom05_zone9   solaris    excl
  16 ldom05_zone13    running     /export/zones/ldom05_zone13  solaris    excl
  17 ldom05_zone15    running     /export/zones/ldom05_zone15  solaris    excl

 

18.] Unfreeze VCS Service Group & Stop VCS on all nodes when in maintenance window
 

Make the VCS configuration read-write

# haconf -makerw
 

Now unfreeze all the VCS Service Groups

# for SG in `hagrp -state | grep -v "#Group" | awk '{ print $1 }' | sort | uniq | xargs -x`; do echo $SG; hagrp -unfreeze $SG -persistent ;done

Dump the VCS configuration and make read-only

# haconf -dump -makero

Stop VCS on all nodes in the cluster

# hastop -all

NOTE: If VVR replication is used, detach the RLINKs prior to restarting the upgraded nodes from the Primary VVR server. Perform full autosync from the Primary RVGs, once all nodes have been upgraded to Solaris 11.4 & InfoScale 7.4.2.x
 

19.] As Solaris Zones are configured, Veritas recommends controlling the start-up sequence for the 1st reboot for each node. This allows the Solaris Zones to be checked following the InfoScale & Solaris Upgrade.

If the user wishes to control the start-up of VCS services, edit the /etc/default/llt file mount on the BE mount-point, i.e. /IS
 

# tail -f /IS/etc/default/llt
#       from /lib/svc/method/llt        for Solaris 2.10
#
# Set the two environment variables below as follows:
#
#       1 = start or stop llt
#       0 = do not start or stop llt
#

LLT_START=0              <<<< change this line to 0 (so LLT does not start at boot-time)
LLT_STOP=1
 

Make the change to the /IS/etc/default/llt file is updated for both BE's on both nodes
 

20.] Umount BE & activate BE on both systems and reboot based on your maintenance window

# beadm list
BE        Flags Mountpoint Space   Policy Created
--        ----- ---------- -----   ------ -------
IS742-1   -     /IS        6.34G   static 2022-03-24 18:53
IS742-2   -     -          166.67M static 2022-03-24 22:03
solaris   -     -          171.23M static 2022-03-17 19:35
solaris-1 NR    /          30.49G  static 2022-03-19 16:29

# beadm umount IS742-1
# beadm list
BE        Flags Mountpoint Space   Policy Created
--        ----- ---------- -----   ------ -------
IS742-1   -     -          6.34G   static 2022-03-24 18:53
IS742-2   -     -          166.67M static 2022-03-24 22:03
solaris   -     -          171.23M static 2022-03-17 19:35
solaris-1 NR    /          30.49G  static 2022-03-19 16:29

# beadm activate IS742-1
# beadm list
BE        Flags Mountpoint Space   Policy Created
--        ----- ---------- -----   ------ -------
IS742-1   R     -          35.84G  static 2022-03-24 18:53
IS742-2   -     -          166.67M static 2022-03-24 22:03
solaris   -     -          171.23M static 2022-03-17 19:35
solaris-1 N     /          534.53M static 2022-03-19 16:29
 

Repeat above steps on both nodes

Shutdown systems, stagger the times for rebooting both nodes by 5-10 minutes to avoid unwanted race conditions.
 

21.] Once the servers have successfully rebooted, edit the /etc/default/llt file on both servers. Allowing LLT to start automatically at boot-time.

To confirm if LLT is not running, type:

# lltconfig
LLT is not running
 

# tail -f /etc/default/llt
#       from /lib/svc/method/llt        for Solaris 2.10
#
# Set the two environment variables below as follows:
#
#       1 = start or stop llt
#       0 = do not start or stop llt
#

LLT_START=1              <<<< change this line to 1 (so LLT will start automatically at boot-time)
LLT_STOP=1


22.] Now start the LLT service on a single node
 

# svcadm enable llt

To confirm if LLT is running, type

# lltconfig

and

# svcs -xv llt
svc:/system/llt:default (Veritas Low Latency Transport (LLT) Init service)
 State: online since March 30, 2022 at 12:50:43 PM IST
   See: man -M /opt/VRTSllt/man/man1m/ -s 1M lltconfig
   See: /var/svc/log/system-llt:default.log
Impact: None.


NOTE: The following log file for the LLT service can also be reviewed.

# tail -20 /var/svc/log/system-llt:default.log

Now LLT is running, manually seed GAB and start VCS just on the single node to control the start-up of the VCS resources.

# gabconfig -c -x

NOTE: Port "a" will be shown if LLT & GAB are now running.

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   4cfe05 membership 0

Start vxfencing if not all ready running

# vxfenconfig -c
Log Buffer: 0x70ee6090

VXFEN vxfenconfig NOTICE Driver will use customized fencing - mechanism cps

NOTE: Port "b" shows vxfencing is running.

# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen   4cfe05 membership 0
Port b gen   4cfe09 membership 0
 

23.] Check vxodm before onlining Service Groups (post reboot) and starting VCS

if vxodm is not shown as "online", follow the below steps:

# svcs -a | grep vxodm
maintenance    16:14:23        svc:/system/vxodm:default

if vxodm is shown as "maintenance", to correct the vxodm service all Solaris local zones must be stopped.

Once all Solaris zones if applicable are stopped, proceed with disabling the vxodm service and other manual steps.

if the vxodm service is shown as "online" skip this section and proceed to step 22.

# svcadm disable svc:/system/vxodm

# svcs -l vxodm
# cat /var/svc/log/system-vxodm:default.log

In the logfile, look for:

Starting ODM...
Device busy: /dev/odm

# cd /dev
# rmdir odm
# mkdir odm

# svcadm enable vxodm


24.] Once vxodm is online then proceed to start VCS and online related Service Groups
 

# hastart

# hastatus -sum

Online the required VCS related Service Groups on the single host.
 

25.] Check the status of the Solaris local zones:
 

# zoneadm list -civ
 

Check vxodm status across the Solaris Zones:

# for zone in `zoneadm list -ci | grep -v global | xargs -x`; do echo "ZONE: $zone" ; zlogin $zone svcs -a | grep odm ; done
 

26.] Now all VCS Service Groups are online on the single node, proceed to start LLT and other related services on the remaining node.

Process is now complete.