How to use llttest and lltping to test LLT heartbeat connectivity.
Article ID: 100024295
Updated On:
Resolution
Note: The tests should be conducted on an LLT port not in use by LLT to prevent any effect on a running cluster.
Determine the ports in use by LLT using the following command:
# /sbin/lltstat -p
LLT port information:
Port Usage Cookie
0 gab 0x0
opens: 0
connects: 0 1
1 gab 0x1
opens: 0
connects: 0 1
7 gab 0x7
opens: 0
connects: 0 1
31 gab 0x1F
opens: 0
connects: 0 1
In the example output above ports 0, 1, 7, and 31 are in use.
For the example tests, port 20 will be used as port 20 is available.
When using the lltping and llttest tools, a server and client model is used.
One node must be set as a server, to provide packets, and another node will be set as a client to receive the packets which will be sent.
To test LLT connectivity using lltping--
Example syntax:
# /opt/VRTSllt/lltping -[c #|s] -T -p 20 -v &
Clarification of the above parameters:
-c (client side, value is equal to the node ID # from /etc/llthosts file)
-s (server side)
-T (display packet round trip times)
-p (LLT port to be used)
-v verbose operation
& is used to background the task
The server side example syntax:
# /opt/VRTSllt/lltping -s -T -p 20 -v &
The client side example syntax:
# /opt/VRTSllt/lltping -c 1 -T -p 20 -v
An example of an successful lltping test follows:
Server side-
# /opt/VRTSllt/lltping -s -T -p20 -v &
6014
# lltping: opening LLT dev: /dev/llt port: 20
lltping: mynodeid: 0
Client side-
# /opt/VRTSllt/lltping -c 0 -T -p20 -v &
7375
# lltping: opening LLT dev: /dev/llt port: 20
lltping: mynodeid: 1
lltping: send_recv_ping to node=0, pkts=20
lltping: send pkt=0
lltping: sending a msg to node 0
lltping: pkt=0, rtt=(0s 176us), tx=(0s 93us) rx=(0s 83us)
lltping: send pkt=1
lltping: sending a msg to node 0
lltping: pkt=1, rtt=(0s 144us), tx=(0s 92us) rx=(0s 52us)
...
lltping: send pkt=19
lltping: sending a msg to node 0
lltping: pkt=19, rtt=(0s 148us), tx=(0s 125us) rx=(0s 23us)
lltping: pkts=20, msgsz=128, RTT:min/avg/max=88/112/195 usec
To test LLT connectivity using llttest--
The command "llttest" is an interactive command.
Example syntax to start an llttest interactive session:
# /opt/VRTSllt/llttest -p port [-v] [-s size]
Note: The size parameter may be relevant as some VLANs and switches may drop packets that are excluded due to size parameters or restrictions that can be manually set in the managed switch / network infrastructure.
On each node, start the interactive session-
# /opt/VRTSllt/llttest -p 20 -v
One the node that will be server, execute the following command at the '>' prompt:
> transmit -n (name of node which will receive packets) -c 1000
sending 1000 messages of size 128 to node 0
1000
On the receiving node, execute the following command:
> receive -c 1000
receiving 1000 messages of size 128
1000
If the llttest succeeds, changing the packet size may be desirable for further testing.
Additionally, a larger data output of the transmitted / received information may be desirable.
In that event, an additional debug level may be set prior to the transmit / receive test.
On the server and receiving nodes, set the debug level-
> debug 3
Initiate the test while debug is now enabled-
> transmit -n (name of node which will receive packets) -c 5
sending 5 messages of size 128 to node 0
1mp @ 0x2cd38: htonl(tmh_magic) 0x0101a10c
tmh_flags 0x00 tmh_src 0xffff tmh_dst 0xffff tmh_port 0x14
tmh_pktnum 0x0000 tmh_len 0x0080 (received 0x0080)
tmh_hdrcksum 0x8039 tmh_cksum 0x2149
0101a10c 0000ffff ffff0014 00010080 843821d9 0101a10c 0000ffff ffff0014
00020080 883b2185 0101a10c 0000ffff ffff0014 00030080 8c3a2131 0101a10c
0000ffff ffff0014 00040080 903d21bd 0101a10c 0000ffff ffff0014 00050080
943c2191 0101a10c 0000ffff
...
5mp @ 0x2cd88: htonl(tmh_magic) 0x0101a10c
tmh_flags 0x00 tmh_src 0xffff tmh_dst 0xffff tmh_port 0x14
tmh_pktnum 0x0004 tmh_len 0x0080 (received 0x0080)
tmh_hdrcksum 0x903d tmh_cksum 0x21bd
0101a10c 0000ffff ffff0014 00050080 943c2191 0101a10c 0000ffff ffff0014
00060080 983f2155 0101a10c 0000ffff ffff0014 00070080 9c3e21cd 0101a10c
0000ffff ffff0014 00080080 a03121a9 0101a10c 0000ffff ffff0014 00090080
a4302105 0101a10c 0000ffff
On the receiving node, execute the following command:
> receive -c 5
receiving 5 messages of size 128
1Received message from src 1
mp @ 0x24ac8: htonl(tmh_magic) 0x0101a10c
tmh_flags 0x00 tmh_src 0xffff tmh_dst 0xffff tmh_port 0x14
tmh_pktnum 0x0000 tmh_len 0x0080 (received 0x0080)
tmh_hdrcksum 0x8039 tmh_cksum 0x2149
0101a10c 0000ffff ffff0014 00010080 843821d9 0101a10c 0000ffff ffff0014
00020080 883b2185 0101a10c 0000ffff ffff0014 00030080 8c3a2131 0101a10c
0000ffff ffff0014 00040080 903d21bd 0101a10c 0000ffff ffff0014 00050080
943c2191 0101a10c 0000ffff
...
5Received message from src 1
mp @ 0x24ac8: htonl(tmh_magic) 0x0101a10c
tmh_flags 0x00 tmh_src 0xffff tmh_dst 0xffff tmh_port 0x14
tmh_pktnum 0x0004 tmh_len 0x0080 (received 0x0080)
tmh_hdrcksum 0x903d tmh_cksum 0x21bd
0101a10c 0000ffff ffff0014 00050080 943c2191 0101a10c 0000ffff ffff0014
00060080 983f2155 0101a10c 0000ffff ffff0014 00070080 9c3e21cd 0101a10c
0000ffff ffff0014 00080080 a03121a9 0101a10c 0000ffff ffff0014 00090080
a4302105 0101a10c 0000ffff
The packets sent and received can be compared for checksum, length, content, etc.
If the tests fail due to packets not received after sending, there still exists some network / infrastructure issue preventing proper communication between the cluster nodes.
lltconfig -E 3 to set the debug level to the most verbose before running the command below.
# /opt/VRTSllt/lltshow -b
Normal output will show information such as this:
00009290 CON N 00
00009290 UP N 00 L 02
00009295 UCN N 00 P 031 CON links 01
00009295 UCN N 00 P 000 CON links 01
00009539 UP N 00 L 00
00009539 UP N 00 L 01
00009540 UCN N 00 P 031 CON links 03
00009540 UCN N 00 P 000 CON links 03
00013817 UCN N 00 P 001 CON links 03
00014589 UCN N 00 P 007 CON links 03
00204747 UCN N 00 P 020 CON links 03
Output similar to those below indicate a disconnection or other issue:
00361318 UCN N 01 P 020 DIS rsn 04
00361318 UCN N 00 P 020 DIS rsn 04
00410871 UCN N 00 P 020 DIS rsn 04
Issue/Introduction
This can be due to network settings / infrastructure.
To determine if there is an network / infrastructure issue, the commands lltping and llttest may be used in addition to dlpiping to confirm the issue.
