VCS cluster nodes cannot communicate with each other if MTU size of LLT private link Ethernet interface does not match. This can result in split brain situation if I/O fencing is configured in disabled mode

book

Article ID: 100023228

calendar_today

Updated On:

Resolution

VCS cluster nodes cannot communicate with each other if MTU size of LLT private link Ethernet interface does not match.

This can result in split brain situation if I/O fencing is configured in disabled mode.


LLT Link information from Node1

#/opt/VRTSllt/lltshow -l -1 | egrep "valid= 1|mtu=" | head -4
valid=1  usable= 3  tag="bge1"
sap= 0xCAFE(51966)  mtu= 9000  addrlen=6
valid= 1  usable=3  tag= "bge2"
sap= 0xCAFE(51966)  mtu= 9000  addrlen= 6

LLT Link information from Node2

#/opt/VRTSllt/lltshow -l -1 | egrep "valid= 1|mtu=" | head -4
valid=1  usable= 3  tag="bge1"
sap= 0xCAFE(51966)  mtu= 1500  addrlen=6
valid= 1  usable=3  tag= "bge2"
sap= 0xCAFE(51966)  mtu= 1500  addrlen= 6

GAB membership information from node1

node1 #gabconfig -a
GAB Port Memberships
====================================
Port a gen   d1cb04 membership01
Port b gen   d1cb06 membership 01
Port d gen  d1cb03 membership 01
Port f gen   d1cb21 membership 0
Port f gen   d1cb21    visible;1
Port h gen   d1cb1b membership 0
Port h gen  d1cb1b    visible ;1
Port o gen   d1cb07 membership 01
Port v gen   d1cb1d membership 0
Port v gen   d1cb1d    visible;1
Port w gen   d1cb1f membership 0
Port w gen  d1cb1f    visible ;1

GAB membership information from node 2

node2 # gabconfig -a
GAB Port Memberships
=====================================
Porta gen   d1cb04 membership01
Port b gen   d1cb06 membership 01
Port d gen  d1cb03 membership 01
Port f gen   d1cb17 membership ;1
Port f gen   d1cb17    visible0
Port h gen   d1cb11 membership;1
Port h gen  d1cb11    visible 0
Port o gen   d1cb07 membership 01
Port v gen   d1cb13 membership;1
Port v gen  d1cb13    visible 0
Port w gen   d1cb15 membership ;1
Port w gen   d1cb15    visible 0

Cluster status from node 1

# hastatus -sum

-- SYSTEMSTATE
--System              State                Frozen

A  node1              RUNNING              0
A  node2              UNKNOWN              0    ====<<< Note status>>>

-- GROUPSTATE
--Group          System               Probed    Auto Disabled    State

B  cvm            node1              Y          N              ONLINE
B  cvm            node2              Y          Y              OFFLINE

-- RESOURCES NOT PROBED
--Group          Type                Resource            System

D  cvm            CFSfsckd            vxfsckd              node2
D  cvm            CVMCluster          cvm_clus            node2
D  cvm            CVMVxconfigd        cvm_vxconfigd        node2


Cluster status from node 2

# hastatus -sum

-- SYSTEMSTATE
--System              State                Frozen

A  node1              UNKNOWN              0         ====<<< Note status>>>

A  node2              RUNNING              0

--GROUP STATE
--Group          System               Probed    Auto Disabled    State

B  cvm            node1              Y          Y              OFFLINE
B  cvm            node2              Y          N              ONLINE

-- RESOURCES NOT PROBED
--Group          Type                Resource            System

D  cvm            CFSfsckd            vxfsckd              node1
D  cvm            CVMCluster          cvm_clus            node1
D  cvm            CVMVxconfigd        cvm_vxconfigd        node1


If I/O Fencing is configured in one of following modes, the remaining nodes will not join cluster and will go to STALE_ADMIN_WAIT, which will prevent split brain situation.

- SCSI-DMP


Solution

1. Configure all network interface(NIC) used for LLT private links with same MTU size.

2. Configure all NICs to have same:
 
- DUPLEX MODE
- SPEED - SAP
- BROADCAST DOMAIN
 
3. Configure I/O Fencing in one of below mentioned fencing modes which will prevent possible split brain situation:
 
      - SCSI-DMP

Issue/Introduction

VCS cluster nodes cannot communicate with each other if MTU size of LLT private link Ethernet interface does not match. This can result in split brain situation if I/O fencing is configured in disabled mode
Powered by Wolken Software