Error Message

Various errors can manifest including:

Booting the zone:
VCS WARNING V-16-10001-14056 (nodename) Zone:zonename:monitor:Zone is running without specified milestone [multi-user-server] online - returning offline. 

engine_A.log errors after the zone is online:
VCS WARNING V-16-10001-14070 (hostname) Zone:zoneRES:online:Run hazonesetup command from global zone to setup passwordless communication with VCS for ha commands running inside local zone [zone].

The cause of the problem is twofold.  A known issue fixed in the 5.1 SP1 RP1 Zone agent, and a procedural issue when setting up the zones.

A few elements are necessary for zones to boot successfully.

1) A minimum of VCS 5.1 SP1 RP1 is required for zones to boot and monitor smoothly.  This level is required to boot zones to "multi-user-server" milestone.

2) Make sure that the 'hostname' OS command produces the correct output for all nodes and zones.  hazonesetup command uses this output to build the Administrator names for authentication.  If this is not set, then Administrator names will be mangled.

3) Make sure that a local zone IP address is assigned through the zonecfg command.  The zone OS should be assigned a (virtual) IP address outside of VCS for proper booting, any OS ldap client authentication and other uses.

4) The standard procedure to bring zone under VCS control currently requires running hazonesetup on ALL the (global zone) nodes in the cluster. While running hazonesetup on a node, make sure that zone is in a running state on that node. (eg: examine the output of: # zoneadm list -v)

After running hazonesetup successfully on ALL the nodes in the cluster, the message
“WARNING V-16-10001-14070” should no longer occur.

If this does not solve the issue, then please provide Support following information to proceed with analysis –
a. Contents of the /.vcspwd file from the zone.
b. hazonesetup command executed with arguments and output. 
c. VRTSexplorers from all nodes.

Note that the /.vcspwd file is generated inside the zone by the halogin command executed via the hazonesetup script.

---------------

If the local zone does not have an entry for ALL global zones in the /.vcspwd file (one line per global zone), then this could be the root of the problem.  You can attempt to recreate it manually.  Essentially, this is a subset of the procedure used by the hazonesetup command.

This procedure can be used to to correct a previously failed installation.  Note that if the hazonesetup command failed to run to completion, then there may be oddly named Administrator definitions left in the main.cf file that will need to be deleted before proceeding.

1) In the global zone on first host, use the values from your system.  For example:

* z_zonename_hostname is the arbitrary name of the "Administrator" that will log in from the zone to the hostname for status.
Note "hostname" here is the short hostname of the global zone.
'zonename' is the name of your zone.
zoneSG is the VCS zone service group name.

# /opt/VRTSvcs/bin/vcsencrypt -vcs
NewEncryptedPasswdString#   (Will print a string.  Don't include any trailing # sign when you use this output later, below.)

# haconf -makerw

If there are active privileges, they must be removed first, before the old Administrator can be deleted:

# /opt/VRTSvcs/bin/hauser -display z_zonename_hostname  (for existing Administrator name)
GroupAdministrator for group(s) ( zone )

First any active privileges should be removed:
# /opt/VRTSvcs/bin/hauser -delpriv z_zone_hostname Administrator -group zoneSG

# /opt/VRTSvcs/bin/haclus -modify UserNames -delete z_zonename_hostname

# /opt/VRTSvcs/bin/haclus -modify UserNames -add z_zonename_hostname NewEncryptedPasswordString

# /opt/VRTSvcs/bin/hauser -addpriv z_zonename_hostname Administrator -group zoneSG

(These steps can be accomplished from the GUI as well.)

2) Next, when logged into the local zone, create the /.vcspwd file

# export VCS_HOST=hostname

# /opt/VRTS/bin/halogin z_zonename_hostname

# /opt/VRTSvcs/bin/hastatus -sum
(should give clean output and not prompt for password.)

If that went well, then continue:

3) Repeat the 'add' commands for remaining 'hostname' or global zone names:

# /opt/VRTSvcs/bin/haclus -modify UserNames -add z_zonename_hostname NewEncryptedPasswordString

# /opt/VRTSvcs/bin/hauser -addpriv z_zonename_hostname Administrator -group zoneSG

If that went well, continue:

4) Bring the zone up on remaining nodes, log into the zone and repeat, adding in the /.vcspwd entry:

# VCS_HOST=hostname
# export VCS_HOST

/opt/VRTS/bin/halogin z_zonename_hostname

5) haconf -dump -makero

Test offline/online of zone between hosts.  Note where any error occurred for further troubleshooting.

----- Alternate procedure using GUI assistance to create /.vcspwd file-------

This procedure can be used to create the /.vcspwd file.  This should eliminate the 'hazonesetup' warning messages in the engine_A.log.  This procedure is an alternative to the command-line only procedure above.

1) Start the GUI and go into User Manager:

-Change the password for the z_ zonename_hostname user to a known password.
-Verify this user has Group Administrator permissions for your zone  group.

 2) Log into zone on local machine:
# zlogin -C gzone
# cat /.vcspwd

Example of a working config:
100 hostname1 z_ zonename_hostname1 GnkInnNmnKnoMm
100 hostname2 z_ zonename_hostname2 fopHojOlpKppNxpJom

You can delete the entry for this node if it exists:
# cp /.vcspwd /.vcspwd.orig
# vi /.vcspwd

3) Generate a new password.  In my case, the hostname of the global zone is hostname1:
# ksh -o vi
# export VCS_HOST=hostname1
# /opt/VRTS/bin/halogin z_ zonename_hostname1
Enter Password:

Now hastatus works without a login:

# /opt/VRTSvcs/bin/hastatus -sum
-- SYSTEM STATE
-- System               State                Frozen
A  hostname1       RUNNING              0
A  hostname2       RUNNING              0
-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State
B  zoneSG         hostname1       Y          N               ONLINE
B  zoneSG         hostname2       Y          N               OFFLINE
-- RESOURCES NOT PROBED
-- Group           Type                 Resource             System
...

Verify that the /.vcspwd file was populated as expected.

Move the zone to the other node and repeat with that hostname.

Applies To

This procedure only works on Solaris Systems running VCS 5.1 SP1 RP1 or higher.