Error Message

The CFS file systems/Nodes may experience poor I/O performance or it may hang.

The following LLT network configurations are required in order to use the UDP Burst feature.  If the following LLT network configuration requirements are not met, the Cluster File System (SFCFS) file systems may experience poor I/O performance or file system hangs.

For UDP Burst feature to work, the network (switch, routers, network interface cards) should support the MTU of  9000 bytes or above. 

Prerequisites for UDP Burst:
- Before starting LLT service, the following must be done.
- Receive Side Scaling (RSS) / Multi-Queue Hashing must be enabled and configured.

Each of the LLT network interface card must have the following configuration.

1. Maximum Transmission Unit size (MTU) must be 9000 or above.   MTU can be configured using ifconfig command.

# ifconfig mtu 9000 up

2. Find out the max limit values with the commands below and use the same for setting the NIC TX/RX values. 
 

# ethtool -g
Ring parameters for ens192:
Pre-set maximums:
RX:             4096
RX Mini:        2048
RX Jumbo:       4096
TX:             4096
Current hardware settings:
RX:             1024
RX Mini:        128
RX Jumbo:       512
TX:             512
 

 

3. Changing the rx/tx ring parameters of the NIC.  rx/tx ring parameters can be configured using ethtool command.

# ethtool -G rx 4096
# ethtool -G tx 4096

 

4. Change the hash options for the udp4 flow type.  Hash options can be configured using ethtool command.

# ethtool -N rx-flow-hash udp4 sdfn
# ethtool -N rx-flow-hash tcp4 sdfn

5. The kernel network option arp_ignore need to enabled for the NIC.   The parameter can be configured using sysctl command.

# sysctl -w net.ipv4.conf..arp_ignore=1

For details on the ifconfig, ethtool and sysctl commands, please refer to the Linux manual pages.

The following is a script to configured all the LLT interfaces in one go.   For example, if the LLT interfaces are eth0 to eth3, then we can run the following.

for card in eth0 eth1 eth2 eth3
do
     ifconfig $card mtu 9000 up
     ethtool -G $card rx 4096
     ethtool -G $card tx 4096
     ethtool -N $card rx-flow-hash udp4 sdnf
     ethtool -N $card rx-flow-hash tcp4 sdnf
     sysctl -w net.ipv4.conf.${card}.arp_ignore=1
done

6. Tune kernel network buffers using sysctl.

sysctl -w net.core.rmem_max=1600000000
sysctl -w net.core.wmem_max=1600000000
 
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.wmem_default=4194304
sysctl -w net.core.optmem_max=4194304
 
sysctl -w net.ipv4.udp_rmem_min=409600
sysctl -w net.ipv4.udp_wmem_min=409600
sysctl -w net.core.netdev_budget=600

7. Set IP Filters (iptables) to allow traffic on ports.  UDP Burst uses multiple UDP ports to transfer the LLT network traffic.   Consecutive UDP port numbers will be used.
For example, if given port numbers are 50000 and 50001 in /etc/llttab, then eight subsequence UDP ports will be used, that is, 50002, 50003, 50004, 50005, up to 5009.   (By default 4 sockets per link are created for each 10Gbit/s NIC.)
Total number of required UDP ports depends on total sockets are being created.

Sample /etc/llttab entries:
# link eth0 udp - udp 50000 9000 192.168.0.1  192.168.0.255
# link eth1 udp - udp 50001 9000 192.168.1.1 192.168.1.255

 

Veritas made code changes to improve the performance of LLT UDP multiport feature on 1500 MTU based networks from Infoscale 7.4.1 onwards. 

If CFS file systems experience I/O performance issues or hang on InfoScale 7.4, please check if the LLT NICs meet the above-mentioned requirements.  If MTU size is 1500, upgrade the setup to Infoscale 7.4.1 OR 7.4.2 with latest Cumulative Patch OR disable UDP Burst feature to avoid the issue.

How to Disable LLT UDP Multiport :

/etc/sysconfig/llt:
#LLT_UDP_MULTIPORT: To enable or disable UDP multiport IO-shipping.
#Default its enabled, set 0 to disable , 1 to enable.
LLT_UDP_MULTIPORT=0
 

Note : MTU size below 1500 is not supported for FSS configuration with LLT over UDP.

 

On InfoScale 7.4 there is a new performance enhancement feature called "UDP Burst". This feature helps the performance of IO Shipping in a Flexible Storage Sharing (FSS) environment with Low Latency Transport (LLT) running over UDP protocol. In a FSS environment, the Cluster Volume Manager (CVM) relies on IO Shipping through the LLT links to access the data on the remote disks. There are some network configuration requirements for the UDP Burst feature. If those network configuration requirements are not met, the Cluster File System (SFCFS) file systems may experience I/O performance issue or file system hangs.

ETrack: 4045607