What is an MTR Test?
You can think of an MTR test as a combination of traceroute and ping. It provides information about the route that the Internet traffic takes between your local system and a remote host. However, unlike traceroute, which is a snap shot in time, this can continue to give you real time information until you exit the program.
Installation and Use on Windows
You can download and install a free program called “WinMTR” – http://winmtr.net/
WinMTR provides you with a GUI interface allowing you to type in the destination host address in the box as prompted. This can be an IP address or a fully qualified domain name.
Installation and Use on Mac & Linux
Installing on Mac
You may also want to use MTR to diagnose networking issues from your local workstation. If you’re running a Mac OS X workstation, you may install MTR from MacPorts by issuing the following command:
# port install mtr
Installing on Linux
If you’re running a Linux system, you can install MTR using the commands below. On Debian and Ubuntu systems, issue the following commands to ensure that your system’s package repository is up to date, that all installed packages are up to date, and finally to install MTR itself:
# apt-get update
# apt-get upgrade
# apt-get install mtr-tiny
On CentOS and Fedora systems you will want to issue the following commands to update repositories, upgrade installed packages, and install the MTR program:
# yum update
# yum install mtr
On Arch Linux systems issue the following commands to update the package database and install MTR:
# pacman -Sy
# pacman -S mtr
Once installed you can generate MTR reports using the following syntax:
# mtr –report [destination host]
You would replace [destination host] with a domain name or IP address:
# mtr –report example.co.uk
# mtr –report 12.34.56.78
Reading an MTR Report
The MTR report contains a wealth of information
|- – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –|
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
|- – – – – – – – – – – – – – – – – – – – – — – – -|- – – – – – | – – – – – – | – – – – – – | – – – | – – – – | – – |
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| 192.168.0.254 – 0 | 10 | 10 | 0 | 0 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 0 | 10 | 10 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 15 | 15 | 19 | 15 |
| 195.66.236.125 – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
| 209.85.244.182 – 0 | 10 | 10 | 16 | 19 | 23 | 16 |
| 72.14.238.21 – 0 | 10 | 10 | 14 | 14 | 16 | 15 |
| lhr14s23-in-f5.1e100.net – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
|__________________________________________|______|______|_____|______|____|_____|
As well as showing the path between servers that packets take to reach the end host, the MTR provides a wealth of statistics which can be useful in diagnosing faults which are split up into columns.
The % (referred to as Loss% on Linux) column is where we will start, this shows the percentage of packet loss at each hop.
Next, we have Sent (Snt on Linux) which shows the number of packets sent, and Recv which is the number of packet responses received.
NOTE: WinMTR will continue to run an MTR until you tell it to stop. With Linux the –report option will send 10 packets unless specified with –report-cycles=[number-of-packets], where [number-of-packets] represents the total number of packets that you want to send to the remote host.
The next four columns Best, Avrg, Wrst and Last are all measurements of latency in milliseconds (e.g. ms). Last is the latency of the last packet sent, Avrg is average latency of all packets, while Best and Wrst display the best (shortest) and worst (longest) round trip time for a packet to this host. In most cases, the average (Avrg) column should be the focus of your attention.
In most circumstances, you can think of the MTR output in three major sections. Depending on configurations, the first 2 or 3 hops often represent the source host’s ISP, while the last 2 or 3 hops represent the destination host’s ISP. The hops in between are the routers the packet traverses to reach its destination.
When running MTR locally, if you see an abnormality in the first few hops near the source, contact your local service provider or investigate your local networking configuration. Conversely, if you see abnormalities near the destination you may want to contact the operator of the destination server or network support for that machine. Unfortunately, in cases where there are problems on the intermediate hops, both service providers will have limited ability to address those glitches.
Verifying Packet Loss
If you see a percentage of loss at any particular hop, that may be an indication that there is a problem with that particular router. However, it is common practice among some service providers to rate limit the ICMP traffic that MTR uses. This can give the illusion of packet loss when there is in fact no loss. To determine if the loss you’re seeing is real or due to rate limiting, take a look at the subsequent hop. If that hop shows a loss of 0.0%, then you can be pretty sure that you’re seeing ICMP rate limiting and not actual loss. See below for an example:
|- – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – — – – – – – – – – – – – – – – – – – – -|
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – – — – – – – – – – | – – – – – | – – – – -| – – – — | – – – – – | – – – – -| – – – -|
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| 192.168.0.254 – 0 | 10 | 10 | 0 | 0 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 0 | 10 | 10 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 15 | 15 | 19 | 15 |
| 195.66.236.125 – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
| 209.85.244.182 – 50 | 10 | 5 | 16 | 19 | 23 | 16 |
| 72.14.238.21 – 0 | 10 | 10 | 14 | 14 | 16 | 15 |
| lhr14s23-in-f5.1e100.net – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
|_________________________________________|______|______|______|______|______|____|
In this case, the loss reported between hops 7 and 8 is likely due to rate limiting on the eighth hop. Although traffic to the remaining ten hops all touch the eighth hop, there is no packet loss. If the loss continues for more than one hop, then it is possible that there is some packet loss or routing issues. Remember that rate limiting and loss can happen concurrently. In this case, take the lowest percentage of loss in a sequence as the actual loss. For instance, consider the following output:
| – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – — – – – – – – – – – – – – – – – – – – – – – – – – – – – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – – – – — -| – – – – – – | – – – – – – | – – – – – – | – – – – – – | – – – – – – | – – – – – |
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| 192.168.0.254 – 0 | 10 | 10 | 0 | 0 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 0 | 10 | 10 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 15 | 15 | 19 | 15 |
| 195.66.236.125 – 60 | 10 | 4 | 14 | 14 | 15 | 15 |
| 209.85.244.182 – 60 | 10 | 4 | 16 | 19 | 23 | 16 |
| 72.14.238.21 – 40 | 10 | 6 | 14 | 14 | 16 | 15 |
| lhr14s23-in-f5.1e100.net – 40 |10 | 6 | 14 | 14 | 15 | 15 |
|____________________________________________________|______|______|______|______|______|______|
In this case, you see 60% loss between hops 6 and 7 as well as between hops 7 and 8. You can assume that the seventh and eighth hop is likely losing some amount of traffic because no subsequent host reports zero loss. However, some of the loss is due to rate limiting as several of the final hops are only experiencing 40% loss. When different amounts of loss are reported, always trust the reports from later hops.
Some loss can also be explained by problems in the return route. Packets will reach their destination without error, but have a hard time making the return trip. This will be apparent in the report, but may be difficult to deduce from the output of MTR. For this reason it is often best to collect MTR reports in both directions when you’re experiencing an issue.
Additionally, resist the temptation to investigate or report all incidences of packet loss in your connections. The Internet protocols are designed to be resilient to some network degradation, and the routes that data takes across the Internet can fluctuate in response to load, brief maintenance events, and other routing issues. If your MTR report shows small amounts of loss in the neighborhood of 10%, there is no cause for real concern as the application layer will compensate for the loss which is likely transient.
Understanding Latency
In addition to helping you asses packet loss, MTR will also help you asses the latency of a connection between your host and the target host. By virtue of physical constraints, latency always increases with the number of hops in a route. However, the increases should be consistent and linear. Unfortunately, latency is often relative and very dependent on the quality of both host’s connections and their physical distance. When evaluating MTR reports for potentially problematic connections, consider earlier fully functional reports as context in addition to known connection speeds between other hosts in a given area.
The connection quality may also affect the amount of latency you experience for a particular route. Predictably, dial-up connections will have much higher latency than cable modem connections to the same destination. Consider the following MTR report which shows a high latency:
|- – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
|- – – – – – – — – – – – – – – – – – — – – – – – – – – | – – – – | – – – – – | – – – – -| – – – – – – | – – – – – – | – – – -|
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| 192.168.0.254 – 0 | 10 | 10 | 0 | 0 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 0 | 10 | 10 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 388 | 360 | 396 | 388 |
| 195.66.236.125 – 0 | 10 | 10 | 390 | 360 | 396 | 390 |
| 209.85.244.182 – 0 | 10 |10 | 391 | 360 | 396 | 391 |
| 72.14.238.21 – 0 | 10 | 10 | 392 | 360 | 396 | 392|
| lhr14s23-in-f5.1e100.net – 0 | 10 | 10 | 392 | 360 | 396 | 392 |
|___________________________________________|______|______|______|______|_______|______|
The amount of latency jumps significantly between hops 5 and 6 and remains high. This may point to a network latency issue as round trip times remain high after the fourth hop. From this report, it is impossible to determine the cause although a saturated peering session, a poorly configured router, or a congested link are frequent causes.
Unfortunately, high latency does not always mean a problem with the current route. A report like the one above means that despite some sort of issue with the 6th hop, traffic is still reaching the destination host and returning to the source host. Latency could be caused by a problem with the return route as well. The return route will not be seen in your MTR report, and packets can take completely different routes to and from a particular destination.
In the above example, while there is a large jump in latency between hosts 5 and 6, the latency does not increase unusually in any subsequent hops. From this it is logical to assume that there is some issue with the 6th router.
ICMP rate limiting can also create the appearance of latency, similar to the way that it can create the appearance of packet loss. Consider the following example:
| – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – — – – – – – – – – – – – – – – – – – – – – – – – – – — – – – – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – – — – – – – – – – – – – – – | – – – – | – – – – – – |- – – – | – – – – – – | – – – – | – – – |
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| 192.168.0.254 – 0 | 10 | 10 | 0 | 0 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 0 | 10 | 10 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 388 | 360 | 396 | 388 |
| 195.66.236.125 – 0 | 10 | 10 | 39 | 39 | 39 | 39 |
| 209.85.244.182 – 0 | 10 | 10 | 39 | 39 | 39 | 39 |
|__________________________________________|_______|______|______|______|______|______|
At first glance, the latency between hops 5 and 6 draws attention. However after the 7th hop, the latency drops drastically. The actual latency measured here is 39ms. In cases like this, MTR draws attention to an issue which does not affect the service. Consider the latency to the final hop when evaluating an MTR report.
Client Router Problems
Cheaper / residential routers can sometimes give misleading results. For example:
| – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – — – – – – – – – – – – – – – – – – – – – – – – – – – – – — – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – – – – — – – – – – – – – – – – – | – – – – – – | – – – – | – – – – – | – – – – – | – – – – | – – – – |
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| ??? – 100 | 10 | 0 | 1 | 1 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 0 | 10 | 10 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 15 | 15 | 19 | 15 |
| 195.66.236.125 – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
| 209.85.244.182 – 0 | 10 | 10 | 16 | 19 | 23 | 16 |
| 72.14.238.21 – 0 | 10 | 10 | 14 | 14 | 16 | 15 |
| lhr14s23-in-f5.1e100.net – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
|_____________________________________________|_______|______|______|______|___________|
The 100% loss on the second hop shouldn’t be cause for concern. As you can see no further loss is sustained on subsequent hops, therefore there is no problem.
Potential ISP Configuration Issues
Sometimes a router on the path your packet is taking can be incorrectly configured and your packets may not reach their destination. Here is an example of this happening with the question marks showing that there is no additional route information:
| – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – — – – – – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – – – – – – – | – – – – | – – – – – | – – – -| – – – – | – – – -| – – – – |
| 192.168.0.1 – 0 | 10 | 10 | 1 | 1 | 1 | 1 |
| 192.168.0.254 – 0 | 10 | 10 | 1 | 1 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
|_____________________________|_____|______|_____|_____|_____|_____|
Another example of a poorly configured router is when looping occurs:
| – – – – – – – – – – – – – – – – – – – – – – – – — – – – – – – – – – – – – – – – – – – – – – – – – – – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – – – – – – – – – | — – – | – – – – | – – – | – – – – | – – – -| – – – – |
| 192.168.0.1 – 0 | 10 | 10 | 1 | 1 | 1 | 1 |
| 192.168.0.254 – 0 | 10 | 10 | 1 | 1 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| 83.170.70.129 – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| 83.170.70.130 – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| 83.170.70.129 – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| 83.170.70.130 – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
| ??? – 0 | 10 | 0 | 0 | 0 | 0 | 0 |
|________________________________|______|______|____|______|____|______|
In both examples hop 3 appears to be the problem hop and that support needs to be sought from that network operator.
The Effects of Rate Limiting
Rate limiting can give the illusion of packet loss, however, if there is packet loss to one hop which doesn’t persist to subsequent hops, then ICMP is the likely cause. Such situations shouldn’t be cause for alarm as rate limiting is a common practice with some networks to reduce congestion and prioritise more important traffic.
| – – – – – – – – — – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – – – – – – – – – – – – | – – – – | – – – – | – – – -| – – – -| – – – -| – – – – |
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| 192.168.0.254 – 0 | 10 | 10 | 0 | 0 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 50 | 10 | 5 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 15 | 15 | 19 |15 |
| 195.66.236.125 – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
| 209.85.244.182 – 0 | 10 | 10 | 16 | 19 | 23 | 16 |
| 72.14.238.21 – 0 | 10 | 10 | 14 | 14 | 16 | 15 |
| lhr14s23-in-f5.1e100.net – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
|__________________________________|______|______|____|_____|_____|____|
Timeout Issues
There can be lots of reasons behind a timeout, but they are not necessarily an indication of packet loss. The packets ultimately reach their destination without any significant losses or latency showing, thus they can be a false positive. A few reasons include:
• Routers may discard ICMP and fail to reply causing a ??? output
• Timeouts could be attributed to routers dropping packets for QoS (quality of service)
• There may be an issue with the return route displaying itself as a timeout
| – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – |
| WinMTR statistics |
| Host – % | Sent | Recv | Best | Avrg | Wrst | Last |
| – – – – – – – – – – – – – – – – — – – – – -| – – – – – – | – – – – – – | – – – – – – | – – – – – – | – – – – – – | – – – |
| 192.168.0.1 – 0 | 10 | 10 | 0 | 0 | 0 | 0 |
| 192.168.0.254 – 0 | 10 | 10 | 0 | 0 | 2 | 1 |
| lns1.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| ge-1-1.500.core.dc5.interdsl.co.uk – 0 | 10 | 10 | 14 | 14 | 15 | 14 |
| thn.as13213.net – 0 | 10 | 10 | 15 | 15 | 17 | 15 |
| 83.170.70.129 – 0 | 10 | 10 | 15 | 15 | 19 | 15 |
| ??? – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
| ??? – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
| ??? – 0 | 10 | 10 | 14 | 14 | 15 | 15 |
| lhr14s23-in-f5.1e100.net – 0 | 10 | 10 | 14 | 14 |15 | 15 |
|________________________________|_____|______|_______|______|______|____|
Raising Support Issues
Should you have any packet loss or latency issues and have run MTR tests we kindly ask you to demonstrate this by sending us the MTR result in plain text format. This will keep the formatting of the MTR and allow us to see what you are seeing.