Error Message

2011/08/07 03:28:04 VCS ERROR V-16-10001-6505 MultiNICB:MNICB_Public:monitor:The mpathd process (/sbin/in.mpathd) does not exist
2011/08/07 03:28:04 VCS WARNING V-16-10001-6506 MultiNICB:MNICB_Public:monitor:Will try to restart mpathd with (/sbin/in.mpathd)
 

The VCS  MultiNICB  Agent configuration and the MultiNICB  Agent configuration for the MultiNICB  Agent Attribute of "MpathdCommand" is not set correctly to match the exact path of the Operating System's "in.mpathd" command. Since the MultiNICB resource is configured with "UseMpathd = 1", it means that the Operating System's IP Multi Pathing daemon "in.mpathd" will be actually managing the multiple paths of the Network Interfaces and VCS MultiNICB Agent depends upon the underlying OS command to report the correct status to control the failover of the IPMultiNICB resources.

To fix the problem, the VCS Agent types.cf configuration file and the main.cf configuration file needs to be updated correctly to reflect the correct path to the MpathdCommand attributes:

Verifying the current configuration:

Snippet from the VCS "/etc/VRTSvcs/conf/config/main.cf" configuration for the MultiNICB resource:
---------------------------------------------------------------------------------------------
group Network (
    SystemList = { system1 = 1, system2 = 2 }
    Parallel = 1
    AutoStartList = { system1 , system2 }
    )

    MultiNICB MNICB_Resource (
        Critical = 0
        UseMpathd = 1
        Device = { bge0 = 1, bge1 = 2 }
        MpathdCommand = "/sbin/in.mpathd"
        Failback = 1
        )
---------------------------------------------------------------------------------------------

Snippet from the VCS "/etc/VRTSvcs/conf/config/types.cf" configuration for the MultiNICB Agent type definition:
---------------------------------------------------------------------------------------------------------------------------------------
type MultiNICB (
    static int MonitorInterval = 10
    static int OfflineMonitorInterval = 60
    static str ArgList[] = { UseMpathd, MpathdCommand, ConfigCheck, MpathdRestart, Device, NetworkHosts, LinkTestRatio, IgnoreLinkStatus, NetworkTimeout, OnlineTestRepeatCount, OfflineTestRepeatCount, NoBroadcast, DefaultRouter, Failback, GroupName }
    static str Operations = None
    int UseMpathd
    str MpathdCommand = "/sbin/in.mpathd"
    int ConfigCheck = 1
    int MpathdRestart = 1
    str Device{}
    str NetworkHosts[]
    int LinkTestRatio = 1
    int IgnoreLinkStatus = 1
    int NetworkTimeout = 100
    int OnlineTestRepeatCount = 3
    int OfflineTestRepeatCount = 3
    int NoBroadcast
    str DefaultRouter = "0.0.0.0"
    int Failback
    str GroupName
)
---------------------------------------------------------------------------------------------------------------------------------------

The output of the Operating System "ps" command output:
---------------------------------------------------------------------------------------------------------------------------------------
 F S      UID   PID  PPID   C PRI NI     ADDR     SZ    WCHAN    STIME TTY         TIME CMD
 0 S     root   442     1   0  40 20        ?    344        ?   Jun 19 ?           3:21 /usr/lib/inet/in.mpathd -a
---------------------------------------------------------------------------------------------------------------------------------------

From the above, it is indicative that the "in.mpathd" command in the running system is "/usr/lib/inet/in.mpathd -a". This should reflect in our VCS Configuration.

The following steps can be performed to update the VCS types.cf and main.cf configuration to reflect the correct MpathdCommand attribute path:

A.]  Take a backup the current configuration

# cd /etc/VRTSvcs/conf/config
# cp types.cf types.cf.orig
# cp main.cf main.cf.orig

B.] Make the cluster configuration in read-write mode:
# haconf -makerw

C.] Update the MultiNICB agent's MpathdCommand attribute configuration:
# hatype -modify MultiNICB ArgList -delete MpathdCommand
# haattr -delete MultiNICB MpathdCommand
# haattr -add MultiNICB MpathdCommand -string "/usr/lib/inet/in.mpathd -a"

D.] Update the VCS main.cf cluster configuration:
# hares -modify MNICB_Public MpathdCommand "/usr/lib/inet/in.mpathd -a"
 

D.] Save and close the VCS cluster configuration:

# haconf -dump -makero

E.] Verify the changes:

# hares -display MNICB_Resource -attribute MpathdCommand
#Resource                 Attribute                                System           Value
MNICB_Resource     MpathdCommand                global     /usr/lib/inet/in.mpathd -a
 

Snippet from the VCS "/etc/VRTSvcs/conf/config/types.cf" configuration for the MultiNICB Agent type definition:
---------------------------------------------------------------------------------------------------------------------------------------
type MultiNICB (
    static int MonitorInterval = 10
    static int OfflineMonitorInterval = 60
    static str ArgList[] = { UseMpathd, MpathdCommand, ConfigCheck, MpathdRestart, Device, NetworkHosts, LinkTestRatio, IgnoreLinkStatus, NetworkTimeout, OnlineTestRepeatCount, OfflineTestRepeatCount, NoBroadcast, DefaultRouter, Failback, GroupName }
    static str Operations = None
    int UseMpathd
    str MpathdCommand = "/sbin/in.mpathd"
    int ConfigCheck = 1
    int MpathdRestart = 1
    str Device{}
    str NetworkHosts[]
    int LinkTestRatio = 1
    int IgnoreLinkStatus = 1
    int NetworkTimeout = 100
    int OnlineTestRepeatCount = 3
    int OfflineTestRepeatCount = 3
    int NoBroadcast
    str DefaultRouter = "0.0.0.0"
    int Failback
    str Protocol = IPv4
    str MpathdCommand = "/usr/lib/inet/in.mpathd -a"
)
---------------------------------------------------------------------------------------------------------------------------------------

The errors due to incorrect path for the "in.mpathd" command setting in VCS should no longer be seen after making the above changes to the VCS configuration.