Description

Veritas has observed an issue surrounding UDP buffer limits which can have a negative impact on Veritas Volume Replicator (VVR) RLINK connection issues.

- the netstat -s output shows a high number of UDP packet receive errors
- the environment is showing a high number of UDP packet drops or loss

VVR always uses the UDP protocol to send VVR heartbeats between the Primary and Secondary hosts.

VVR sends a UDP heartbeat every second and awaits acknowledgment back.
The VVR heartbeat is only 322 bytes over UDP.

If the UDP buffer size is set too low, it can result in a buffer overflow preventing UDP heartbeat messages when processed.
 

The current impact on the UDP buffer count can be seen for a specific port number (i.e. 4145 for VVR UDP port) by running:
 

# netstat -anolp | grep 4145 


When netstat -s shows 'receive buffer errors' on a RHEL7, 8 and 9 system, it increases the UDP socket buffer size to a higher value, i.e. 8MiB.

# netstat -s | grep buffer
                  receive buffer errors

The ss -ulnp and netstat -ul commands can be used to review the UDP information.

# netstat -ul
# ss -ulnp

The key string we are looking for in regards to troubleshooting VVR is "vvr-control".

# netstat -ul | grep "vvr-control"
# ss -ulnp

This can impact VVR by reporting the VVR RLINKs in an "ENABLED ACTIVE" state, preventing the RLINK returning to a "CONNECT ACTIVE" state.
The VVR RLINK(s) must be a "CONNECT ACTIVE" state for replication to be active and sending packets from the VVR Primary to the VVR Secondary nodes.

Udp: packet receive errors are increased for the following reasons: Not enough socket buffer space

Note: Until the breached UDP buffer count is reset, the RLINKs cannot be reconnected.


Communication Test

The "echo | nc -zv 4145" syntax can be used to check communication access between servers for port "4145".


System Tuning
 

The most common reason for Udp: packet receive errors .

Not enough socket buffer space

Note: If the application sets it's own maximum receive buffer size using setsockopt (SO_RCVBUF) , please check the application documentation, source code or ask the application provider and increase the buffer size in the application to match the rmem_max setting.

The following commands will set the default and maximum UDP buffer sizes to 8 MiB:

# sysctl -w net.core.rmem_max=8388608

# sysctl -w net.core.rmem_default=8388608

Note: RedHat recommends testing this in a development environment first or during a maintenance window.

After changing this value, the UDP application will need to be stopped and started for the change to take effect.

To make this change permanent after reboot, edit /etc/sysctl.conf and add the lines:

Note: RedHat doesn't use legacy /etc/sysctl.conf going forward, create a new filename under the directory path /etc/sysctl.d/.conf.

net.core.rmem_max = 8388608
net.core.rmem_default = 8388608

If drops still occur, this can be increased further. 16MiB is 16777216 , 32MiB is 33554432 .


Note: It is not ideal to have a large default buffer, as these apply to everything which is not TCP.

The best situation would be to have a large maximum but reasonable (default or 256KiB, 262144 ) default, and have the UDP application set large buffer sizes.

This depends on the application setting a large buffer size. Refer to the application vendor for assistance with this.

 

UDP checksum failure
 

Perform a packet capture of the UDP traffic whilst Udp: packet receive errors is growing:

Example

# tcpdump -i any -C 250 -s 128 -w /var/tmp/$(hostname)-$(date "+%s")-udp.pcap udp  
 

Note: Adding "udp" to the end of the tcpdump syntax will capture udp related information only

Capture the tcpdump outputs on both the VVR Primary and Secondary nodes at the same time.
 

Using RedHat doc to troubleshoot

Please note that this article references sites not owned or maintained by Veritas and, as such, Veritas is not responsible for the content portrayed on such sites, including any revisions to or deletions of content or third-party software on which this article relies. User is responsible for conducting all necessary due diligence prior to following the instructions described in this article.

Network debugging on Red Hat Enterprise Linux

https://access.redhat.com/articles/1311173