How to perform network configuration for use with Veritas Volume Replicator

With Veritas Volume Replicator (VVR), one commonly misdiagnosed problem is an issue with a operating system's network configuration that is mistaken to be an issue with VVR due to replication being slow. To avoid this misdiagnosis, you must be familiar with issues related to network configurations and methods for determining if the issue is with VVR or if the issue is instead with the network.

The following list provides some important concepts related to the health of a network:

· Lossy network A lossy network is a network that tends to lose packets during high network traffic. This loss of packets slows down data transmission due to the packets needing to be resent.

· Packet reordering Packet reordering is the inverting of packets during the tramission of data across a network, which causes the TCP receiver to believe that packets are missing. Packet reordering can occur due to multi-path routing, and slows down data transmission due to the TCP receiver requesting the sender to resend packets, which uses more bandwidth when the packets are resent.

· Latency The latency of a network is one aspect of the speed of a network, and indicates the delay from the time that a packet is sent to the time that a packet is received.

Some common replication performance issues are as follows:

· VVR does not use the full available bandwidth

· You observe variations in replication throughput

· DCM replay performance better than normal replication

· You observe a difference in replication throughput between TCP and UDP

VVR rarely uses the full bandwidth. The maximum replication throughput depends on several factors, such as write size, latency, packet reordering, and packet loss.

You can use the following tools to determine if your network has issues:

· vxrlink command See "Using vxrlink to diagnose network issues"

· Iperf tool See "About the Iperf tool"

· Appcritical tool See "About the AppCritical tool"

If you determined that the issue is with the network and not with VVR, then you must tune the operating system network parameters to improve your network performance.

See "Tuning the operating system network parameters"

Using vxrlink to diagnose network issues

The Veritas Volume Replicator (VVR) vxrlink command performs Veritas Volume Manager (VxVM) operations on RLINKs. The vxrlink -e stats command provides information about the following network stats:

Memory	Number of errors because of insufficient buffer space on the Secondary. Memory errors indicate that there is not enough system kernel memory to process the message.
Slots	Indicates that there are no message slots available.
Pool	Indicates that there is no memory available in the nmcom pool.
Timeout	Number of timeouts errors. A timeout error occurs when an acknowledgement for a message is not received from the remote host within the timeout period.
Packet	Displays the number of times that the last packet of a message was received before one or more packets of the same message had been received.
Message	Missing message errors. Displays the number of times that messages have arrived out of sequence.
Stream	Number of stream errors. Stream errors occur when the RLINK attempts to send messages faster than the network can handle. Generally, this indicates that there is a buffer or connection problem.
Checksum	Indicates that packets are getting corrupted in the network.
Transaction	The number of times that packets could not be delivered to the secondary because of transaction errors. If the secondary is busy with some kernel operations when a packet arrives at the secondary, the packet cannot be delivered until the transaction is complete.

For additional information about the errors displayed, see the vxrlink(1M) manual page.

See "Using vxrlink to diagnose packet-related network issues"

See "Using vxrlink to diagnose buffer space-related network issues"

See "Using vxrlink to diagnose packet reordering-related network issues"

See "About Veritas Volume Replicator network configuration tuning"

Using vxrlink to diagnose packet-related network issues

If you have selected the UDP transport protocol for replication, the UDP packet size used by Veritas Volume Replicator (VVR) to communicate between hosts could be an important factor in the replication performance. By default, VVR uses a UDP packet size of 8400 bytes. In certain network environments, such as those that do not support fragmented IP packets, it might be necessary to decrease the packet size.

If the network you are using loses many packets, the effective bandwidth available for replication is reduced. You can tell that this is happening if you run the vxrlink stats command on the RLINK and see many timeout errors. In this case, network performance might be improved by reducing the packet size. If the network is losing many packets, it can simply be that each time a large packet is lost, a large retransmission has to take place. In this case, try reducing the packet size until the problem is ameliorated.

If some element in the network, such as IPSEC or VPN hardware, is adding extra data to the packets, reduce the packet size so that there is space for the additional bytes in the packet, and the MTU is not exceeded. Otherwise, each packet is broken into two.

See "Using vxrlink to diagnose network issues"

Using vxrlink to diagnose buffer space-related network issues

The amount of buffer space available for requests coming in to the Secondary over the network is determined by the Veritas Volume Replicator (VVR) tunable, vol_max_nmpool_sz, which defaults to 16 megabytes. VVR allocates separate buffer space for each Secondary RVG, the size of which is equal to the value of the tunable vol_max_nmpool_sz. The buffer space on the Secondary must be large enough to prevent slowing the network transfers excessively. If the buffer space on the Secondary is too small, the replicated data is discarded and memory errors are returned back to the Primary.

While replication is active at the peak rate, run the vxrlink stats command and make sure there are no memory errors and the number of timeout errors is small:

# vxrlink -g dg1 -i5 stats rlink1

See "Using vxrlink to diagnose network issues"

Using vxrlink to diagnose packet reordering-related network issues

Veritas Volume Replicator (VVR) uses a number of memory slots on the Secondary site to store incoming network I/O. By default, number of messages slots is 1024. If the network has high packet reordering, the default number of message slots may not be enough. Packets will get dropped if they are out of order, and this will impact replication performance.

You can examine extended RLINK statistics from vxrlink -e stats command output to find packet reordering:

# vxrlink -g dg1 -i10 -e stats rlink1
Errors :
------
No memory available                             : 0
No message slots available                      : 0
No memory available in nmcom pool on Secondary : 0
Timeout                                         : 0
Missing packet                                  : 0
Missing message                                 : 0
Stream                                          : 0
Checksum                                        : 0
Unable to deliver due to transaction            : 0

Errors in the following categories indicate that packet reordering is happening in the network:

· No message slots available

· Missing message

If you determined that your network has an issue with packet reordering, you must increase the number of message slots.

See "Resolving packet reordering"

See "Using vxrlink to diagnose network issues"

About the Iperf tool

The Iperf tool is a third party tool that can create TCP and UDP streams to measure the throughput of a network. Iperf has been accredited by many major US institutions and is commonly available pre-compiled for many operating systems in their package repositories or on freeware sites. However, Iperf is not linked with nor sponsored by Symantec, and as such any defects or bugs found with Iperf should be reported directly to the Iperf developers and not to Symantec technical support.

When dealing with network connections and Veritas Volume Replicator (VVR), you must consider the following points:

· The rated speed of a piece of network hardware or connection is the theoretical maximum throughput that can be achieved in a burst in ideal conditions. As such, the real world maximum throughput that can be achieved is likely to be somewhat lower than the rated maximum throughput.

· Network throughput is commonly measured in bits per second, such as Mbps (megabits per second) or Kbps (kilobits per second). These are an order of magnitude smaller than data storage sizes such as MB (megabytes) or KB (kilobytes). As such, a network card rated at 100 Mbps can only transfer a theoretical maximum of ~12.5 MB of data per second.

· 1 Mbps equals 1000 Kbps and 1 Kbps equals 1000 bps, not 1024 bps. This is different than data storage sizes where 1 MB equals 1024 KB and 1 KB equals 1024 bytes.

· The peak throughput achievable by VVR when replicating is likely to be somewhat lower than the maximum throughput found by network testing tools such as Iperf. This is due to the design of the VVR product making it particularly sensitive to network features such as dropped or reordered packets and packet fragmentation. VVR's use of the network is inherently more complex than the tests performed by simple network test tools.

See "Using Iperf to diagnose network issues"

See "About Veritas Volume Replicator network configuration tuning"

Using Iperf to diagnose network issues

Iperf is based on a client/server model. As such, Iperf binaries must be installed on both machines that are taking part in testing. Iperf is started on one node in server mode. This node sits and listens for a connection on a specific port. Iperf is then started on the second node in client mode and is provided with the host name or IP address of the server node. The client node attempts to initiate a connection with the server node and once connected starts to push data over the connection. The characteristics of the connection can be modified by options provided in the Iperf binary. Iperf attempts to push as much data as possible over the connection in a given amount of time and once complete uses the time taken and amount of data pushed to calculate a perceived bandwidth for the connection.

When using Iperf to test network throughput between primary and secondary nodes, you must take the following actions before testing can begin:

· Replication must be stopped. Replication uses a certain proportion of the available bandwidth between the nodes and as such will cause Iperf to give an artificially small value for available bandwidth, which can be misleading. If you use Iperf to test non-Veritas Volume Replicator (VVR) ports, you can instead pause replication.

· If you use Iperf to test throughput on specific ports that are also being used by VVR, the component of VVR that is using the port must be stopped. Failure to stop the component can leave VVR bound to the port and can cause Iperf to fail since Iperf cannot bind to a given port, or Iperf can give misleading results. The component of VVR that should be stopped depends on the port being tested. For example, when using TCP for replication, data is commonly sent on TCP port 4145. As such, before you can use Iperf to test this port, you should stop vxnetd kernel threads on both nodes to ensure that the port is free.

To stop a specific daemon, you can use the daemon's specific init script. The following example procedure uses the vxstart_vvr script to stop all VVR daemons for the duration of your testing.

To use the vxstart_vvr script to stop all VVR daemons

Verify that the RLINK is CONNECT ACTIVE:
# vxprint -qtrg testdg | grep ^rl
rl to_10.12.240.33 testrvg CONNECT ACTIVE 10.12.240.33 testdg to_10.12.249.249
Stop the VVR daemons on both nodes:
# /usr/sbin/vxstart_vvr stop
Verify that the RLINK is now ENABLE ACTIVE:
# vxprint -qtrg testdg | grep ^rl
rl to_10.12.240.33 testrvg ENABLED ACTIVE 10.12.240.33 testdg to_10.12.249.249

With the RLINK in the ENABLE ACTIVE state, you now can run Iperf to test connectivity and throughput between the nodes. The Iperf options that you specify alter the way that Iperf tests the network.

See "Interpreting Iperf results"

See "Examples of using Iperf to diagnose network issues"

See "Iperf options"

After you complete your testing, kill the Iperf server process on the server node and restart the VVR daemons on both nodes.

To use the vxstart_vvr script to start all VVR daemons

Start the VVR daemons on both nodes:
# /usr/sbin/vxstart_vvr start
Verify that the RLINK is now CONNECT ACTIVE:
# vxprint -qtrg testdg | grep ^rl
rl to_10.12.240.33 testrvg CONNECT ACTIVE 10.12.240.33 testdg to_10.12.249.249

See "About the Iperf tool"

See "About Veritas Volume Replicator network configuration tuning"

Iperf options

You can specify the following options when you run Iperf:

-s	Runs Iperf locally in server mode. Iperf sits and listens for incoming connections.
-c hostname	Runs Iperf locally in client mode and attempts to initiate a connection with an existing Iperf server on the node specified by hostname. The value of hostname can be a name or an IP address.
-p port_number	Specifies the port that Iperf uses for communications. The default is port 5001.
-t time	Specifies the length in seconds for which to run Iperf testing. The defalut is 10 seconds. Specifying 30 seconds or longer generally gives more consistent results.
-u	Runs tests using the UDP protocol. The default is the TCP protocol.
-b bandwidth[KM]	Specifies the bandwidth at which the client attempts to send when using the UDP protocol. To test the throughput of a UDP connection effectively, you should specify for the client to send at a higher throughput than you believe that the network has available. For example, if testing a 100 Mbps network connection, you should specify the bandwidth at higher than 100 M, such as 250 M. Failure to do this prevents the client from sending as fast as possible and gives an artificially low result for the available network bandwidth. If you do not specify the -b option, Iperf defaults to 1 Mbps. Iperf does not want to try and flood the connection with UDP packets unless told to do so. However, you must flood the network to get a valid result. This option is not required when using the TCP protocol.
-l size	Specifies the size in bytes of datagrams to send during testing when using the UDP protocol. If not specified, Iperf defaults to 1470 byte datagrams to try and avoid packet fragmentation. This option is not required when using the TCP protocol.
-P number	Specifies the number of parallel connections to use when using the TCP protocol. If not specified, Iperf defaults to a single TCP connection. This can help simulate multiple TCP connections as used by Veritas Volume Replicator (VVR) in certain situations.
-B ip_address	Specifies a specific IP address to bind on the local node. This is useful where a machine has multiple IP addresses on the same subnet, as it can be used to bind Iperf to the same IP that would be used by VVR on each machine, ensuring that Iperf uses the same network infrastructure and routing as VVR traffic.

See "Using Iperf to diagnose network issues"

Interpreting Iperf results

Any results obtained with Iperf detail the bandwidth that was available specifically when the test was run. Available bandwidth can be influenced by many factors, such as the following examples:

· The time of day and the day of the week. Networks are expected to be busier during peak hours or when backups are running.

· Applications running on the nodes that are being tested. A high load may prevent Iperf and network stacks from getting sufficient CPU time, which skews results.

· Routing in use. Sometimes different network paths get used at different times of the day.

Because of these influencing factors, you should run Iperf a number of times across the day to get a feel for what is normally achievable bandwidth and to know if there are any periods where bandwidth is higher or lower than normal.

The available bandwidth can also be influenced by physical factors in the network, such as the following examples:

· Asymmetric routing. With asymmetric routing, data takes a different route through the network from client to server compared to server to client. Network traffic might be significantly faster in one direction.

· Data compression. Data can be compressed depending on the given ports and protocols.

You should always test the available bandwidth in both directions between the nodes to ensure that the available bandwidth is comparable depending on which node is used as the server. Otherwise, you can experience significant changes in Veritas Volume Replicator (VVR) throughput if the direction of replication is reversed after migration or fail over between sites. Likewise, you should test the available bandwidth on the same ports that is used by VVR with comparable options, such as protocol and packet size. This ensures that Iperf-generated traffic is subject to the same compression and other factors as the VVR traffic.

When using the TCP protocol, data is sent on the VVR heartbeat port, which is port 4145 by default, whereas when using the UDP protocol, data is sent on anonymous UDP ports by default. This makes it very difficult to determine the exact port numbers that are in use for data transmission. If VVR throughput is very poor, but Iperf shows significantly improved data throughput on a named UDP port or range of ports, you should configure VVR to use the same ports as a test to see if VVR shows an increase in throughput.

For further information on displaying and modifying VVR ports, see the vrport(1M) manual page.

See "Examples of using Iperf to diagnose network issues"

See "Using Iperf to diagnose network issues"

Examples of using Iperf to diagnose network issues

The following example tests a single UDP connection between two nodes on port 12345 using the default size of 1470 bytes for the datagrams. Since the network connection between nodes is believed to be ~100 Mbps, the example specifies a bandwidth value of 250 Mbps.

Note:

In UDP mode, Iperf traffic has no flow control. The Iperf client tries to send the data to the Iperf server only at a rate as specified by the -b option. Since there is no flow control, Iperf allows the network to drop the packets. As a result, the throughput reported by the Iperf client is very useful value because some packets will be dropped. The throughput reported by the Iperf server is the number of packets getting through the network without being dropped and is the actual transfer rate of the network. The bandwidth that you specify using the -b option should be close to the theoretical bandwidth of the network. If the you specifies a low bandwidth or the default bandwidth of 1 Mbit/s, the Iperf client only sends that much data and the test results are not meaningful.

Example of testing a single UDP connection between two nodes

On the server node, run Iperf in server mode:
# ./iperf -s -u -p 12345
------------------------------------------------------------
Server listening on UDP port 12345
Receiving 1470 byte datagrams
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[ 3] local 10.12.240.32 port 12345 connected with 10.12.240.33 port 46248
On the client node, run Iperf in client mode:
# ./iperf -c 10.12.240.32 -u -p 12345 -t 30 -b 250M
------------------------------------------------------------
Client connecting to 10.12.240.32, UDP port 12345
Sending 1470 byte datagrams
UDP buffer size: 8.00 KByte (default)
------------------------------------------------------------
[ 3] local 10.12.240.33 port 46248 connected with 10.12.240.32 port 12345
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-30.0 sec 298 MBytes 83.4 Mbits/sec
After connecting the client to the port, the Iperf utility tests the network and provides the throughput statistics of the network.

The following example tests a single TCP connection between two nodes on port 45678.

Example of testing a single TCP connection between two nodes

On the server node, run Iperf in server mode:
# ./iperf -s -p 45678
------------------------------------------------------------
Server listening on TCP port 45678
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37795
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-30.0 sec 189 MBytes 52.8 Mbits/sec
On the client node, run Iperf in client mode:
# ./iperf -c 10.12.240.32 -p 45678 -t 30
------------------------------------------------------------
Client connecting to 10.12.240.32, TCP port 45678
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.12.240.33 port 37795 connected with 10.12.240.32 port 45678
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-30.0 sec 189 MBytes 52.8 Mbits/sec
After connecting the client to the port, the Iperf utility tests the network and provides the throughput statistics of the network.

The following example tests 8 parallel TCP connections between two nodes on port 45678.

Example of testing 8 parallel TCP connections between two nodes

On the server node, run Iperf in server mode:
# ./iperf -s -p 45678
------------------------------------------------------------
Server listening on TCP port 45678
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37835
[ 5] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37836
[ 6] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37837
[ 7] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37838
[ 8] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37839
[ 9] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37840
[ 10] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37841
[ 11] local 10.12.240.32 port 45678 connected with 10.12.240.33 port 37842
[ ID] Interval       Transfer     Bandwidth
[ 8] 0.0-30.0 sec 41.0 MBytes 11.4 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 11] 0.0-30.0 sec 41.4 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 9] 0.0-30.0 sec 41.5 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 5] 0.0-30.0 sec 41.0 MBytes 11.5 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 7] 0.0-30.1 sec 41.5 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 10] 0.0-30.1 sec 42.6 MBytes 11.9 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 6] 0.0-30.1 sec 41.6 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 4] 0.0-30.1 sec 41.4 MBytes 11.6 Mbits/sec
[SUM] 0.0-30.1 sec    332 MBytes 92.7 Mbits/sec
On the client node, run Iperf in client mode:
# ./iperf -c 10.12.240.32 -p 45678 -t 30 -P 8
------------------------------------------------------------
Client connecting to 10.12.240.32, TCP port 45678
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 10] local 10.12.240.33 port 37842 connected with 10.12.240.32 port 45678
[ 3] local 10.12.240.33 port 37835 connected with 10.12.240.32 port 45678
[ 4] local 10.12.240.33 port 37836 connected with 10.12.240.32 port 45678
[ 5] local 10.12.240.33 port 37837 connected with 10.12.240.32 port 45678
[ 6] local 10.12.240.33 port 37838 connected with 10.12.240.32 port 45678
[ 7] local 10.12.240.33 port 37839 connected with 10.12.240.32 port 45678
[ 8] local 10.12.240.33 port 37840 connected with 10.12.240.32 port 45678
[ 9] local 10.12.240.33 port 37841 connected with 10.12.240.32 port 45678
[ ID] Interval       Transfer     Bandwidth
[ 5] 0.0-30.0 sec 41.6 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 7] 0.0-30.0 sec 41.0 MBytes 11.5 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 8] 0.0-30.0 sec 41.5 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 10] 0.0-30.0 sec 41.4 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 9] 0.0-30.0 sec 42.6 MBytes 11.9 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 4] 0.0-30.0 sec 41.0 MBytes 11.5 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 6] 0.0-30.0 sec 41.5 MBytes 11.6 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[ 3] 0.0-30.0 sec 41.4 MBytes 11.6 Mbits/sec
[SUM] 0.0-30.0 sec    332 MBytes 92.7 Mbits/sec
After connecting the client to the port, the Iperf utility tests the network and provides the throughput statistics of the network. Comparing the results of having 8 parallel TCP connections to having a single TCP connection in the previous example, you can see a significantly higher throughput with multiple connections, which is expected given the nature of the TCP protocol.

See "Interpreting Iperf results"

See "Using Iperf to diagnose network issues"

About the AppCritical tool

AppCritical™ is a third party tool that provides active-mode monitoring of your network activity. AppCritical "virtualizes" the network to create a view that is meaningful to applications. The tool looks end-to-end from the application’s perspective and produces simple, bottom line information regarding the network. Even non-network specialists can use this information to identify, assess, and diagnose issues when replication fails to complete. AppCritical requires no prior knowledge of the network, no agents, and only a few hours of training for typical general support staff with minimal skills.

AppCritical can diagnose the following faults:

· Maximum Transmission Unit (MTU) conflicts

· Full/half duplex conflicts

· Bottlenecks and congestion points

· NIC problems

· Media errors (connections and cables)

· Poorly performing routers and switches

· Mixing of TCP and legacy protocols

· Latency-bound applications

· Reordering of packets

AppCritical can also see into networks that you do not own, and can provide the following information about those networks:

· End-to-End

· Bandwidth

· Utilization

· Packet loss

· Latency

· Jitter

See "Using AppCritical to diagnose network issues"

See "About Veritas Volume Replicator network configuration tuning"

Tuning the operating system network parameters

After you have diagnosed the cause of a network issue, you can tune various network parameters to reduce or eliminate the issue.

Note:

These tunables are operating system parameters, and as such if you require additional assistance in tuning the parameters, you must contact the corresponding operating system vendor's technical support. Do not contact Symantec technical support for assistance with operating system tunables.

See "Tunable network parameters on AIX"

See "Tunable network parameters on HP-UX"

See "Tunable network parameters on Linux"

See "Tunable network parameters on Solaris"

See "Tuning error tolerance"

See "Resolving packet reordering"

See "Resolving stream errors"

See "About Veritas Volume Replicator network configuration tuning"

See "Increasing the number of TCP connections with a high latency network"

Tunable network parameters on AIX

You can tune the following AIX network parameters to resolve some or all of your network issues:

rfc1323	Enables the TCP window scaling option.
tcp_nodelayack	Enables TCP to send an immediate acknowledgement instead of waiting the default 200 ms delay.
tcprexmtthresh	Specifies how many consecutive duplicate ACK's are allowed before a TCP connection goes to fast retransmit phase. The default value is 3.
sb_max	Specifies the upper limit on the number of socket buffers queued to an individual socket. This value controls how much buffer space is used by buffers that are queued to a sender's socket or to a receiver's socket.
tcp_sendspace	Specifies how much data the sending application can buffer in the kernel before the application is blocked on a send call.
tcp_recvspace	Specifies how many bytes of data the receiving system can buffer in the kernel on the receiving sockets queue.
udp_recvspace	Specifies the amount of space for incoming data that is queued on each UDP socket. Any packets received beyond the specified value are discarded.

You can tune these parameters by using the no command.

See the no manual page.

See "Example of tuning the TCP protocol to improve replication performance on AIX"

See "Tuning the operating system network parameters"

Example of tuning the TCP protocol to improve replication performance on AIX

In this example, you are replicating your production site to disaster recovery site that is 900 km away by using TCP asynchronous replication. Even though the WAN network has more bandwidth, Veritas Volume Replicator (VVR) is not utilizing the network bandwidth, which results in a long delay with synchronizing the Replicated Volume Group (RVG). You are using a WAN optimizer, which does 10:1 TCP data packet compression, but does not support UDP.

Further analysis reveals that the round-trip network latency between the Primary and disaster recovery site is very high--up to 120 ms. As a result, the performance is not highly dependent on how big the network pipe is, but is dependent on the network latency. With TCP, you must stream packets, and TCP does the work of pushing data through the pipe. VVR is able to push around 8 mb/sec with an average round-trip time of around 100 to 120 ms using TCP and the WAN optimizer. Since the WAN optimizer does not support UDP, changing the protocol is not possible. Thus, the next available option is for you to tune the AIX TCP network settings on both the Primary and Secondary.

To tune the TCP settings on the Primary and Secondary

Enable rfc1323 so that you can set the MTU size above 8K to allow larger TCP receive space values:
# no -o rfc1323=1
Increase the MTU size to 64K for the adapter that is used for VVR replication on both the Primary and Secondary:
# ifconfig en7 mtu 65536
You should check with your networking team before tuning MTU size, as the entire network link including all the network appliances between the VVR Primary and Secondary nodes must support the new MTU size.
Confirm the new MTU size:
# pmtu display
Increase the tcp_recvspace tunable to more that 10 times the size of the MTU value on all the nodes in the Primary and Secondary site clusters. Since you increased the MTU size to 64K, you can set the tcp_recvspace value to 64K * 10 = 655360 or higher:
# no -o tcp_recvspace=655360
With a tcp_recvspace value of 655360, TCP can have 10 outstanding packets, which improves performance since TCP can stream data into the tcp_recvspace. If you set tcp_recvspace for 80 * MTU, there can be 80 un-ACK packets.
Increase the tcp_sendspace tunable to at least twice the value of tcp_recvspace on all the nodes in the Primary and Secondary site clusters. Since tcp_recvspace now has a value of 655360, set tcp_sendspace to 1310720:
# no -o tcp_sendspace=1310720
Enable the tcp_nodelayack option:
# no -o tcp_nodelayack=1
The tcp_nodelayack option forces TCP to send an immediate acknowledgement instead of delaying the default 200 ms. Normally, the TCP recipient delays before it sends an ACK in case it receives more data from sender. There is some overhead in sending more ACKs, but eliminating the ACK delay can minimize the round trip delays.

See "Tunable network parameters on AIX"

Tunable network parameters on HP-UX

You can tune the following HP-UX network parameters to resolve some or all of your network issues:

tcp_conn_request_max	Specifies the maximum number of outstanding TCP connection requests. The default value is 4096.
tcp_recv_hiwater_def	Specifies the maximum TCP receive window size. The default value is 32768 bytes.
tcp_recv_hiwater_max	Specifies the upper bound of the TCP receive buffer size. The default value is 1073725440 bytes.
tcp_xmit_hiwater_def	Specifies the amount of unsent data that triggers TCP flow control. The default value is 32768 bytes.
tcp_xmit_hiwater_max	Specifies the upper bound on TCP send buffer size. The default value is 2147483647 bytes.
tcp_cwnd_initial	Specifies the initial size of the TCP congestion window as a multiple of the Maximum Segment Size (MSS). The value is calculated with the following formula: min((4 * MSS), max(2 * MSS, 4380))
tcp_rexmit_interval_initial	Specifies the initial value for a TCP round trip time-out. The default value is 3000 ms.
udp_recv_hiwater_max	Specifies the upper bound of the UDP receive buffer size. The default value is 2147483647 (2^31) bytes.
socket_udp_rcvbuf_default	Specifies the default receive buffer size for UDP sockets.
socket_udp_sndbuf_default	Specifies the default send buffer size for UDP sockets.

You can tune these parameters by using the ndd command.

See the ndd(1M) manual page.

See "Tuning the operating system network parameters"

Tunable network parameters on Linux

You can tune the following Linux network parameters to resolve some or all of your network issues:

net.core.rmem_max	Specifies the maximum TCP receive window size.
net.core.wmem_max	Specifies the maximum TCP transmit window size.
net.ipv4.tcp_rmem	Specifies the minimum, default, and maximum TCP receive window size.
net.ipv4.tcp_wmem	Specifies the minimum, default, and maximum TCP transmit window size.
net.ipv4.tcp_window_scaling	Enables window scaling, as defined in RFC1323.
net.ipv4.tcp_timestamps	Enables timestamps, as defined in RFC1323.
net.ipv4.tcp_sack	Enables select acknowledgments (SACKS).
net.ipv4.tcp_no_metrics_save	Disables the caching of metrics on closing connections.
net.ipv4.tcp_moderate_rcvbuf	Enables TCP receive buffer auto-tuning. With receive buffer auto-tuning , TCP attempts to set the size of the buffer to match the size required by the path for full throughput. The resulting size will not be greater than the maximum specified by the net.ipv4.tcp_rmem parameter. This is enabled by default.
net.core.netdev_max_backlog	Specifies the maximum number of packets that can be queued on the input side. This queue is used when the interface receives packets faster than kernel can process them. The default value is 1000.
net.ipv4.tcp_max_syn_backlog	Specifies the maximum number of remembered connection requests that have not received an acknowledgment from connecting client. The default value is 1024 for systems with more than 128 MB of memory, and 128 for low memory machines. If server becomes overloaded with requests, then increase this number.
Ethernet interface txqueuelen	Specifies the length of the Ethernet interface transmit queue.

You can tune these parameters by using the sysctl command.

See the sysctl(8) manual page.

See "Tuning the operating system network parameters"

Tunable network parameters on Solaris

You can tune the following Solaris network parameters to resolve some or all of your network issues:

tcp_recv_hiwat	Specifies the default TCP receive window size, in bytes.
tcp_xmit_hiwat	Specifies the default TCP send window size, in bytes.
tcp_max_buf	Specifies the maximum TCP buffer size, in bytes.
tcp_cwnd_max	Specifies the maximum value of the TCP congestion window, in bytes.
tcp_rexmit_interval_min	Specifies the default TCP minimum retransmission time-out value, in milliseconds.
udp_xmit_hiwat	Specifies the default maximum UDP socket datagram size, in bytes.
udp_xmit_lowat	Specifies the default minimum UDP socket datagram size, in bytes.
udp_recv_hiwat	Specifies the default maximum UDP socket receive buffer size, in bytes.
udp_max_buf	Specifies the how large send and receive buffers can be for a UDP socket, in bytes.

You can tune these parameters by using the ndd command.

See the ndd(1M) manual page.

See "Tuning the operating system network parameters"

Tuning error tolerance

When you use UDP as the replication protocol, Veritas Volume Replicator (VVR) uses its own network flowcontrol. VVR increases or decreases the rate at which data is sent depending on the number of timeouts or memory errors it gets per second. If the number of errors is greater, VVR decreases the sending rate to avoid network congestion. If there are only a few errors or no errors, VVR continues to increase the sending rate by a fixed amount every second. For a lossy network, a large number of errors may occur, which prevents VVR from increasing the sending rate. However, these errors are not due to network congestion so VVR should continue to increase the sending rate.

You specify the error tolerance VVR uses by setting two tunables: vol_rp_increment and vol_rp_decrement. VVR increases its sending rate if timeouts or memory errors per second are not more than the vol_rp_increment value. VVR decreases its sending rate if timeouts or memory errors per second are more than vol_rp_decrement. The default value of both tunables is 8.

In the case of a lossy network, the sending rate does not increase because the number of errors per second could be more than vol_rp_increment or vol_rp_decrement, and the sending rate can decrease further. This impacts replication performance. If RLINK statistics show a higher number of errors and VVR is not using available bandwidth, you may be able to improve replication performance by tuning vol_rp_increment and vol_rp_decrement to higher value like 16 or 32. You can change both tunables using the vxtune command. This tuning is required only on the Primary host.

See the vxtune(1M) manual page.

See "Tuning the operating system network parameters"

Resolving packet reordering

If you determined that your network has an issue with packet reordering, you must increase the number of message slots. You can specify the number of message slots with the nmcom_max_msgs tunable. For a high packet reordering network, increasing default value to 2048 or 3072 may improve replication performance. You can tune this value with the vxtune command.

See the vxtune(1M) manual page.

See "Using vxrlink to diagnose packet reordering-related network issues"

See "Tuning the operating system network parameters"

Resolving stream errors

To resolve stream errors, you must increase various buffer sizes. The tunables to set depend on the operating system and transport protocol that you are using. Symantec provides some tuning recommendations, although these recommendations are only a starting point. Symantec recommends that you increase the values gradually and monitor the network performance after each tuning. Continue to increase the value of the tunables and monitor the results if the network performance is still not satisfactory.

· See "Resolving stream errors on AIX"

· See "Resolving stream errors on HP-UX"

· See "Resolving stream errors on Linux"

· See "Resolving stream errors on Solaris"

See "Tuning the operating system network parameters"

Resolving stream errors on AIX

Tuning recommendations for resolving stream errors on AIX with the UDP protocol provides tuning recommendations for resolving stream errors on AIX with the UDP protocol.

Table 1-1 Tuning recommendations for resolving stream errors on AIX with the UDP protocol

Tunable	Recommended Value
sb_max	8388608
udp_recvspace	1048576

Tuning recommendations for resolving stream errors on AIX with the TCP protocol provides tuning recommendations for resolving stream errors on AIX with the TCP protocol.

Table 1-2 Tuning recommendations for resolving stream errors on AIX with the TCP protocol

Tunable	Recommended Value
rfc1323	1
tcp_nodelayack	1
tcprexmtthresh	8
sb_max	8388608
tcp_sendspace	1048576
tcp_recvspace	524288

See "Resolving stream errors"

Resolving stream errors on HP-UX

Tuning recommendations for resolving stream errors on HP-UX with the TCP protocol provides tuning recommendations for resolving stream errors on HP-UX with the TCP protocol.

Table 1-3 Tuning recommendations for resolving stream errors on HP-UX with the TCP protocol

Tunable	Recommended Value
tcp_xmit_hiwater_def	32 KB, 64 KB, or 128 KB
tcp_xmit_lowater_def	8 KB, 16 KB, or 32 KB
tcp_recv_hiwater_def	32 KB, 64 KB, or 128 KB

See "Resolving stream errors"

Resolving stream errors on Linux

Tuning recommendations for resolving stream errors on Linux with the UDP protocol provides tuning recommendations for resolving stream errors on Linux with the UDP protocol.

Table 1-4 Tuning recommendations for resolving stream errors on Linux with the UDP protocol

Tunable	Recommended Value
net.core.rmem_max	128 KB, 1 MB, 4 MB, or 8 MB
net.core.wmem_max	128 KB, 1 MB, 4 MB, or 8 MB

Tuning recommendations for resolving stream errors on Linux with the TCP protocol provides tuning recommendations for resolving stream errors on Linux with the TCP protocol.

Table 1-5 Tuning recommendations for resolving stream errors on Linux with the TCP protocol

Tunable	Recommended Value
net.ipv4.tcp_rmem	4096, 87380, or 16777216
net.ipv4.tcp_wmem	4096, 65536, or 16777216

See "Resolving stream errors"

Resolving stream errors on Solaris

Tuning recommendations for resolving stream errors on Solaris with the UDP protocol provides tuning recommendations for resolving stream errors on Solaris with the UDP protocol.

Table 1-6 Tuning recommendations for resolving stream errors on Solaris with the UDP protocol

Tunable	Recommended Value
udp_max_buf	1 MB, 2 MB, 4 MB, or 8 MB
udp_xmit_hiwat	64 KB
udp_xmit_lowat	8 KB
udp_recv_hiwat	64 KB

Tuning recommendations for resolving stream errors on Solaris with the TCP protocol provides tuning recommendations for resolving stream errors on Solaris with the TCP protocol.

Table 1-7 Tuning recommendations for resolving stream errors on Solaris with the TCP protocol

Tunable	Recommended Value
tcp_max_buf	1 MB, 2 MB, 4 MB, or 8 MB
tcp_xmit_hiwat	64 KB
tcp_xmit_lowat	32 KB
tcp_recv_hiwat	64 KB

See "Resolving stream errors"

Increasing the number of TCP connections with a high latency network

Veritas Volume Replicator (VVR) is generally deployed in network environments where the replication network often suffers from latencies. Network latencies impact VVR replication performance. To counter latency issues, VVR in TCP mode can open multiple connections between the Primary and disaster sites so as to keep the replication at an optimum pace. VVR uses a round-robin algorithm to replicate data over the multiple TCP connections to improve replication performance.

Normally, the number of TCP connections used is automatically determined by VVR in response to network latency. However, in certain networks, such as where network compression hardware is used, the algorithm used to determine the number of connections might not be ideal. In these circumstances, you can override the automatic tuning by setting the following tunable:

nmcom_max_connections

Specifies the number of TCP connections that should be used by a Primary when using the TCP protocol for replication. This tunable only takes effect on a Primary using the TCP protocol. The nmcom_max_connections tunable is a hidden parameter, is only shown via the vxtune command if specified explicitly.

If you set this tunable manually via a kernel debugger or by editing a kernel parameters file, the change only takes effect if the vol_vvr_force_connections tunable is set to 1. If you use the vxtune command to modify nmcom_max_connections, then the vol_vvr_force_connections tunable is set to 1 automatically.

In VVR releases prior to 5.0 MP3 RP3 on AIX, Linux, and Solaris, and 5.1 on HP-UX, you can only tune nmcom_max_connections manually. You must upgrade to a later release to be able to use the vxtune command to set this tunable.

See the vxtune(1M) manual page.

The following example sets the maximum number of connections to 32 using the vxtune command:

# vxtune nmcom_max_connections 32

The following example sets the maximum number of connections to 4 by editing the /etc/system file on Solaris:

set vxio:vol_vvr_force_connections=1
set vxio:nmcom_max_connections=4

Note:

The vol_vvr_force_connections tunable is a kernel parameter and cannot be shown or changed by the vxtune command. You can check the value of this parameter only with a kernel debug command.

See "Tuning the operating system network parameters"

Issue/Introduction

How to perform network configuration for use with Veritas Volume Replicator

Was this article helpful?

thumb_up Yes

thumb_down No

Welcome to "KB Articles"