LLT Considerations when using GigE and 10GigE NetworkInterface Cards

When using LLT (Low Latency Transport) Protocolacross GigE or 10GigE interfaces, it's been found that the default MTUconfigured on the NIC device driver can be set to use jumbo frame size of 9192bytes by default. When LLT is used in conjunction with a Network Switch, thelarge LLT packets are dropped by the switch if the MTU on the switch is set tohandle only regular 1500 byte MTU packets.

This has most recently beenseen on both Solaris and AIX systems where GigE and 10GigE cards are used forLLT. These devices have their default MTU size set to use JumboFrames.

---------------------------------------------

KnownSymptoms:

- The HA Daemon (had) will not properly start on more thanone node at a time, the large cluster config message exceeds 1500bytes. Thepacket is dropped by the switch during membership change.
- The I/O Fencing(vxfen) driver will not initialize on more than one node at time.
- UnderAIX, the LPAR VIO Network Interface is set to use Jumbo Frames by default.
-Under Solaris, the newer 10GigE interfaces are configured to use Jumbo Frames bydefault.

---------------------------------------------

Identification:

Thereare a few possible ways to check the size of the NIC's MTU, especially usefulwhen interfaces are not plumbed:

1. AVAILABLE IN VERITAS VXEXPLOREVCS getcomms output.

# /opt/VRTSgab/getcomms
getcomms: Gathering datafor sydsup01
getcomms: completed. results in/tmp/commslog.Aug_25_08_10_40.tar

# more lltshowl.sydsup01
=== LLTlink 0:
valid= 1  usable= 3  tag= "qfe:0"
pri= 0  muxid=8  lower= 1104cd0  heartbeat= 50
type= 0  hbinit=0x783bda2c
sap= 0xCAFE (51966)  mtu= 1500  addrlen=6     <----- MTU of 1500 which will work with ALLswitches.
saplen= 2  sapfirst= 0  dlhdrlen= 28  minlen=50
setup= 0x783b7e90  recv= 0x783b8040
xmitcnt=27395716  recvcnt= 0  errcnt= 0
===

2. Can bespecifically gathered manually from Command line with lltshow:

EXAMPLEOF A SYSTEM WITHOUT JUMBO FRAMES ALREADY SET ON INTERFACE:

#/opt/VRTSllt/lltshow -l -1 | egrep "valid= 1|mtu=" | head -6
valid=1  usable= 3  tag= "qfe:0"
sap= 0xCAFE (51966)  mtu=1500  addrlen= 6     <----- MTU of 1500 which willwork with ALL switches.
valid= 1  usable= 3  tag="qfe:1"
sap= 0xCAFE (51966)  mtu= 1500  addrlen= 6
sap=0x0000 (0)  mtu= 0  addrlen= 0
sap= 0x0000 (0)  mtu=0  addrlen= 0


EXAMPLE OF A SYSTEM WITH JUMBO FRAMES ALREADYSET ON INTERFACE:

== LLT link 0:
valid= 1  usable=3  tag= "nxge0"
pri= 0  heartbeat= 35
type= 6  hbinit=0x70801aac  lasthbsend= 0x630a21
sap= 0xCAFE (51966)  mtu=9194  addrlen= 6     <-----MTU of 9194. Needs jumboframes setting on switches.
saplen= 2  sapfirst= 0  minlen=46
setup= 0x7b2064a4  recv= 0x7b206ec0
muxid= 9  lower=0x60064fde180
xmitcnt= 24032  recvETH= 22591  recvDLPI=0  errcnt= 0
puthbcnt= 0  ln_badcksum= 0
txrate=7/1/0 (pkts),20/22/22 (KB) [per s/10s/30s]
rxrate=6/0/0 (pkts), 0/0/0 (KB) [pers/10s/30s]


3. It can be checked via the kstat command onSolaris **:

The mac_mtu value can be set to 1522 to reducethe chance of the switch dropping packets when it enforces the 1500 Byte MTUpacket size.

eg.
# kstat | grepmac_mtu
       mac_mtu                        0
       mac_mtu                        1522


If the values for mac_mtu shows 9216, then the DLPI(Layer 2) Layer is configured for Jumbo Frames already.

**Reference: Sun Documents:

Tuning Gigabit networkBlueprint:
https://www.sun.com/blueprints/0404/817-6925.pdf

How to SetJumbo-frames:
https://docs.sun.com/source/820-1606-11/jumbo_fms.html

NICManuals:
NXGE: https://dlc.sun.com/pdf/820-1606-11/820-1606-11.pdf
IPGE:https://dlc.sun.com/pdf/819-4090-11/819-4090-11.pdf

---------------------------------------------

Solutions:

1.Enable Jumbo Frames on the Network switch to allow the larger LLT packetsize tobe transferred between the VCS nodes over LLT. This will need to be performed bythe Network Administrator. Please refer to Switch Vendor documentation for thisprocedure.


2. Restrict LLT Packet size by modifying the LLTlinks in the /etc/llttab file:

From:
link nxge0 /dev/nxge:0 - ether --

To:
link nxge0 /dev/nxge:0 - ether - 1500

Note: Remember tochange all LLT Private network entries to use the 1500 value if the switchcannot be modified to use Jumbo Frames.

3. You may also be able torestrict the layer 2 MTU size in the NIC Driver. The instructions for 'How toenable/disable Jumbo frames' documentation should be obtained from theparticular O/S vendor for the machine being configured.


FORSOLARIS:

Modify NIC Driver tunable:
You may also be able torestrict the layer 2 MTU size using the mac_mtu tunable available with mostGigE/10GigE NICs.
Example: The Sun nxge NIC:
/kernel/drv/nxge.conf
name= "pciex108e,abcd" parent = "/pci@780/pci@0/pci@8/network@0" unit-address = "0"accept-jumbo=1;

For other interfaces, the specific vendor documentationshould be consulted.


FOR AIX:

ModifyTemporarily:
# chdev -l 'en0' -a mtu='1500'

ModifyPersistently:
# chdev -l 'en0' -a mtu='1500' -P


FOR HPUX11.11/11.23:

- Edit the following configuration file using an editorsuch as "vi": /etc/rc.config.d/hpietherconf.
- Set the mtusize by editingHP_IETHER_MTU[i]=mtusize,
- Insert the proper interface name:HP_IETHER_MTU[i]=HP_IETHER_INTERFACE_NAME.