Error Message

When offlining the following Solaris zones using the zoneadm shutdown command, the Solaris zones halt.
 

2021/11/07 16:05:21 VCS DBG_3 Zone:db-zone-RES:offline:Zone [db-zone] is still in [running] state. Halting the zone now.

2021/11/07 16:05:21 VCS DBG_3 Zone:ap-zone-RES:offline:Zone [ap-zone] is still in [running] state. Halting the zone now.

2021/11/07 16:05:21 VCS DBG_3 Zone:ora-zone-RES:offline:Zone [ora-zone] is still in [running] state. Halting the zone now.
 

The Solaris Zones can't be shutdown, so the VCS Zone agent has no option but to halt the impacted Solaris zones instead.

Once the system is restarted, the previous halted Solaris Zones cannot be onlined or booted:

2021/11/07 16:32:30 VCS DBG_3 Zone:db-zone-RES:online:Command [/usr/sbin/zoneadm -z "db-zone" boot] exited with output [zoneadm: zone db-zone: another zoneadm(8) may have an operation in progress: unable to lock /system/volatile/zones/db-zone.zoneadm.lock: Resource temporarily unavailable] and exitcode [1]

2021/11/07 16:32:31 VCS DBG_3 Zone:ap-zone-RES:online:Command [/usr/sbin/zoneadm -z "ap-zone" boot] exited with output [zoneadm: zone ap-zone: another zoneadm(8) may have an operation in progress: unable to lock /system/volatile/zones/ap-zone.zoneadm.lock: Resource temporarily unavailable] and exitcode [1]

2021/11/07 16:32:32 VCS DBG_3 Zone:ora-zone-RES:online:Command [/usr/sbin/zoneadm -z "ora-zone" boot] exited with output [zoneadm: zone ora-zone: another zoneadm(8) may have an operation in progress: unable to lock /system/volatile/zones/ora-zone.zoneadm.lock: Resource temporarily unavailable] and exitcode [1]

The Solaris zoneadm operations are performed serially instead of in parallel which results in delaying the time it takes to start all Solaris zones.

To troubleshoot the issue outside of Veritas Cluster Server (VCS) control, perform the following tests.
 

1. Start the Solaris zones outside of VCS as follows:

# zoneadm list -cv | awk '{ print $2 }' | egrep -v "NAME|global" > /tmp/zones

# for i in `cat /tmp/zones`
do echo "------$i------"
zoneadm -z $i boot
done

Note that it can take approximately 20 minutes to start 15 or more Solaris zones:
 

2. Perform the same sequence in reverse by offlining each Zone in the background, which can take approximately 20 minutes to shutdown 15 or more Solaris zones:

# for i in `cat /tmp/zones`
do echo "------$i------"
zoneadm -z $i shutdown &
done

Bug ID: 33589642 After rebooting global zone Some zones went to maintenance state

Oracle have provided an IDR patch to optimize the zoneadm related operations.

Solaris provided hidden IDR idr4945 to improve zoneadm efficency and should now be included in Solaris 11.4 SRU 42 (released 15th Feb).

Please contact Oracle Support for more details surrounding the Solaris related fix.