Description

Details:
1) Setup multipathing for Active/Passive.  It is typically best to use active/passive instead of active/active because the routing involved is far easier, and throughput does not exceed what a current "ce" interface can handle.  You need to execute the following from the console due to the fact that you will lose connection to the network when you move the real IP from the card to the virtual.

    - Set first interface with the first test address,
     group main, deprecated and marked for no failover.

ifconfig xx0 plumb netmask + broadcast + group main
deprecated -failover up

    - Setup the IP address of the machine as a virutal address.

      ifconfig xx0 addif netmask + broadcast + failover up

    - Set the second interface with the second test address.

ifconfig xx1 plumb netmask + broadcast + group main
deprecated -failover standby up


Note: You can also put the following in the hostname files and
    reboot.  

    /etc/hostname.ce0
      netmask + broadcast + \
      group main deprecated -failover up \
      addif netmask + broadcast + failover up

   /etc/hostname.ce1
      netmask + broadcast + \
      group main deprecated -failover standby up


2) Setup VCS with a parallel service group for the NIC cards. The service group needs to be setup to run across all machines that can potentially run a particular service group.
  All machines should/need to be in the same class C address.

3) In the above service group put a MultiNICB resource with
  the settings:

NetworkHosts
ChangeTestRepeatCount
Device ce0 0
ce1 1
useMpathd 1
MpathdRestart 1
Failback 1
LinktestRatio 1
MpathdCommand /sbin/in.mpathd
OfflineTestRepeatCount 3
NetworkTimeout 100
NoBroadcast 0
ConfigCheck 1
DefaultRouter 0.0.0.0

Note: All machines in this example are configured with ce cards, and ce0 is always the active path.

You can also set the following to work around net blips and small outages.

MultiNICB MonitorInterval 60
MultiNICB ToleranceLimit 4 or 5

4) In the above service group put a phantom resource to bring the service group online.  Enable this resource.

Note: Both resources in the above SG should be set to critical.

5) In your SG that you wish to enable the MultiNICB interface you need to add a proxy to the MultiNICB resource in the above parallel service group.

    TargetResName
    TargetSysName

6) In the same SG as above (#5) you need to add an IPMultiNICB resource with the following:

    BaseResname
    Address
    NetMask
    DeviceChoice 0

7) The resource from step #5 should be set to the child of the resource from step #6 above.  The resource from #6 above will be the one that goes offline when both NIC cards fail on the system in question.

8) Test by doing the following (you can minimize the amount of testing time by tuning the monitor interval and the tolerance limit down):

    1) Unplug the xx0 interface first and make sure that IPMP fails ce0:1 to ce1:1.  VCS should not care about this, there is still a good interface available.

    2) Plug the interface back in and make sure that IPMP fails it back over.  VCS should not care about this either.

    3) Unplug both interfaces at the same time.  The system should now be off the network.  You can tune the amount of time that VCS faults by tuning the MonitorInterval and the ToleranceLimit variables.
      If you take my settings it will take 4-5 minutes to fault.

    4) Do a mixture of the above.  Try pulling ce0, then after it fails over, then pull ce1.  Then, waiting for the fail, then plug back in ce1.

Here is the complete configuration:

Group OraTest (
        SystemList = { system1 = 1 }
        AutoStartList = { system1 }
        )



        IPMultiNICB TestIP (
                Critical = 1
                BaseResName = DBMpathNIC
                Address = "10.0.1.30"
                NetMask = "255.255.255.0"
                )



        Proxy TestNICProxy (
                Critical = 1
                TargetResName = DBMpathNIC
                )



        Sqlnet TestListener (
               Critical = 0
               ResourceOwner = me
               Owner = oracle
               Home = "/oracle/product/8.1.6"
               TnsAdmin = "/var/opt/oracle"
               Listener = listener
               MonScript = "/opt/VRTSvcs/bin/Sqlnet/LsnrTest.pl"
                EnvFile = "/oracle/.mcdt"
               )



        TestIP requires TestNICProxy
        TestListener requires TestIP


group OracleMultipath (

        SystemList = { system1 = 1, system2 = 2 }
        Parallel = 1
       AutoStartList = { system1, system2 }
        )


        MultiNICB DBMpathNIC (
                UseMpathd = 1
                MpathdCommand = "/sbin/in.mpathd"
                Device @system1 = { ce0 = 0, ce1 = 1 }
                Device @system2 = { ce0 = 0, ce1 = 1 }
                DefaultRouter = "0.0.0.0"
                )

        Phantom IPMPPhantom (
                Dummy = OracleMultipath
                )

        IPMPPhantom requires DBMpathNIC