Chapter 6

6. Troubleshooting

Learn about common networking tools to help you diagnose and troubleshoot issues!

Subsections of 6. Troubleshooting

1. ICMP

Lesson Content

The Internet Control Message Protocol (ICMP) is part of the TCP/IP protocol suite, it used to send updates and error messages and is an extremely useful protocol used for debugging network issues such as a failed packet delivery.

Each ICMP message contains a type, code and checksum field. The type field is the type of ICMP message, the code is a sub-type and describes more information about the message and the checksum is used to detect any issues with the integrity of the message.

Let’s look at some common ICMP Types:

  • Type 0 - Echo Reply
  • Type 3 - Destination Unreachable
  • Type 8 - Echo Request
  • Type 11 - Time Exceeded

When a packet can’t get to a destination, Type 3 ICMP message is generated, within Type 3 there are 16 code values that will further describe why it can’t get to the destination:

  • Code 0 - Network Unreachable
  • Code 1 - Host Unreachable

etc..etc..

These messages will make more sense as we use some network troubleshooting tools.

Exercise

No exercises for this lesson.

Quiz Question

# What is the ICMP type for echo request? > Internet Control Message Protocol (ICMP) is one of the protocols of the TCP/IP suite. The ICMP echo request and the ICMP echo reply messages are commonly known as ping messages. 1. [ ] 1 2. [ ] 11 3. [ ] 3 4. [x] 8 5. [ ] 0

2. ping

Lesson Content

One of the most simplest networking tools ping, it’s used to test whether or not a packet can reach a host. It works by sending ICMP echo request (Type 8) packets to the destination host and waits for an ICMP echo reply (Type 0). Ping is successful when a host sends out the request packet and receives a response from the target. Let’s look at an example:

pete@icebox:~$ ping -c 3 www.google.com
PING www.google.com (74.125.239.112) 56(84) bytes of data.
64 bytes from nuq05s01-in-f16.1e100.net (74.125.239.112): icmp_seq=1 ttl=128 time=29.0 ms
64 bytes from nuq05s01-in-f16.1e100.net (74.125.239.112): icmp_seq=2 ttl=128 time=23.7 ms
64 bytes from nuq05s01-in-f16.1e100.net (74.125.239.112): icmp_seq=3 ttl=128 time=15.1 ms

In this example, we are using ping to check if we can get to www.google.com. The -c flag (count) is used to stop sending echo request packets after the count has been reached.

The first part says that we are sending 64-byte packets to 74.125.239.112 (google.com) and the rest show us the details of the trip. By default it sends a packet per second.

icmp_seq

The icmp_seq field is used to show the sequence number of packets sent, so in this case, I sent out 3 packets and we can see that 3 packets made it back. If you do a ping and you get some sequence numbers missing, that means that some connectivity issue is happening and not all your packets are getting through. If the sequence number is out of order, your connection is probably very slow as your packets are exceeding the one second default.

ttl

The Time To Live (ttl) field is used as a hop counter, as you make hops, it decrements the counter by one and once the hop counter reaches 0, our packet dies. This is meant to give the packet a lifespan, we don’t want our packets travelling around forever.

time

The roundtrip time it took from you sending the echo request packet to getting an echo reply.

Exercise

Do a ping on a website and look at the output you receive.

Quiz Question

# What is the roundtrip time unit of measurement? > Round-trip time (RTT) is the duration, measured in milliseconds, from when a browser sends a request to when it receives a response from a server. It's a key performance metric for web applications and one of the main factors, along with Time to First Byte (TTFB), when measuring page load time and network latency. 1. [ ] Ms 2. [ ] sec 3. [ ] seq 4. [x] ms

3. traceroute

Lesson Content

The traceroute command is used to see how packets are getting routed. It works by sending packets with increasing TTL values, starting with 1. So the first router gets the packet, and it decrements the TTL value by one, thus dropping the packet. The router sends back an ICMP Time Exceeded message back to us. And then the next packet gets a TTL of 2, so it makes it past the first router, but when it gets to the second router the TTL is 0 and it returns another ICMP Time Exceeded message. Traceroute works this way because as it sends and drops packets it is build a list of routers that the packets traverse, until it finally gets to its destination and gets an ICMP Echo Reply message.

Here’s a little snippet of a traceroute:

$ traceroute google.com                                                                          
traceroute to google.com (216.58.216.174), 30 hops max, 60 byte packets                          
 1  192.168.4.254 (192.168.4.254)  0.028 ms  0.009 ms  0.008 ms                                  
 2  100.64.1.113 (100.64.1.113)  1.227 ms  1.226 ms 0.920 ms
 3  100.64.0.20 (100.64.0.20)  1.501 ms 1.556 ms  0.855 ms

Each line is a router or machine that is between me and my target. It shows the name of the target and its IP address and the last three columns correspond to the round-trip time of a packet to get to that router. By default, we send three packets along the route.

Exercise

Run the traceroute command on your machine and observe the output.

Quiz Question

# What gets decremented by one when making hops across the network? > When an IP packet is sent, its TTL is usually 255 and is then decremented by 1 at each hop. If the TTL reaches 0, the packet is dropped. 1. [ ] icmp_seq 2. [ ] time 3. [ ] packet loss 4. [x] ttl

4. netstat

Lesson Content

Well Known Ports

We’ve discussed data transmission through ports on our machine, let’s look at some well known ports.

You can get a list of well-known ports by looking at the file /etc/services:

ftp             21/tcp
ssh             22/tcp
smtp            25/tcp 
domain          53/tcp  # DNS
http            80/tcp
https           443/tcp
..etc..

The first column is the name of the service, then the port number and the transport layer protocol it uses.

netstat

An extremely useful tool to get detailed information about your network is netstat. Netstat displays various network related information such network connections, routing tables, information about network interfaces and more, it’s the swiss army knife of networking tools. We will focus mostly on one feature netstat has and that’s the status of network connections. Before we look at an example, let’s talk about sockets and ports first. A socket is an interface that allows programs to send and receive data while a port is used to identify which application should send or receive data. The socket address is the combination of the IP address and port. Every connection between a host and destination requires a unique socket. For example, HTTP is a service that runs on port 80, however we can have many HTTP connections and to maintain each connection a socket gets created per connection.

pete@icebox:~$ netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 icebox:domain           *:*                     LISTEN     
tcp        0      0 localhost:ipp           *:*                     LISTEN     
tcp        0      0 icebox.lan:44468        124.28.28.50:http       TIME_WAIT  
tcp        0      0 icebox.lan:34751        124.28.29.50:http       TIME_WAIT  
tcp        0      0 icebox.lan:34604        economy.canonical.:http TIME_WAIT  
tcp6       0      0 ip6-localhost:ipp       [::]:*                  LISTEN     
tcp6       1      0 ip6-localhost:35094     ip6-localhost:ipp       CLOSE_WAIT 
tcp6       0      0 ip6-localhost:ipp       ip6-localhost:35094     FIN_WAIT2

The netstat -a command shows the listening and non-listening sockets for network connections, the -t flag shows only tcp connections.

The columns are as follows from left to right:

  • Proto: Protocol used, TCP or UDP.
  • Recv-Q: Data that is queued to be received
  • Send-Q: Data that is queued to be sent
  • Local Address: Locally connected host
  • Foreign Address: Remotely connected host
  • State: The state of the socket

See the manpage for a list of socket states, but here are a few:

  • LISTENING: The socket is listening for incoming connections, remember when we make a TCP connection our destination has to be listening for us before we can connect.
  • SYN_SENT: The socket is actively attempting to establish a connection.
  • ESTABLISHED: The socket has an established connection
  • CLOSE_WAIT: The remote host has shutdown and we’re waiting for the socket to close
  • TIME_WAIT: The socket is waiting after close to handle packets still in the network

Exercise

Look at the manpage for netstat and learn all the features it has to offer.

Quiz Question

# What port is used for HTTPS? > It is essential to know the difference between the two. HTTPS is secure and is on port 443, while HTTP is unsecured and available on port 80. Information that travels on the port 443 is encrypted using Secure Sockets Layer (SSL) or its new version, Transport Layer Security (TLS) and hence safer. 1. [ ] 6969 2. [ ] 22 3. [ ] 80 4. [x] 443

5. Packet Analysis

Lesson Content

The subject of packet analysis could fill an entire course of its own and there are many books written just on packet analysis. However, today we will just learn the basics. There are two extremely popular packet analyzers, Wireshark and tcpdump. These tools scan your network interfaces, capture the packet activity, parse the packages and output the information for us to see. They allows us to get into the nitty gritty of network analysis and get into the low level stuff. We’ll be using tcpdump since it has a simpler interface, however if you were to pick up packet analysis for your toolbelt, I would recommend looking into Wireshark.

Install tcpdump

$ sudo apt install tcpdump

Capture packet data on an interface

pete@icebox:~$ sudo tcpdump -i wlan0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlan0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:28:23.958840 IP icebox.lan > nuq04s29-in-f4.1e100.net: ICMP echo request, id 1901, seq 2, length 64
11:28:23.970928 IP nuq04s29-in-f4.1e100.net > icebox.lan: ICMP echo reply, id 1901, seq 2, length 64
11:28:24.960464 IP icebox.lan > nuq04s29-in-f4.1e100.net: ICMP echo request, id 1901, seq 3, length 64
11:28:24.979299 IP nuq04s29-in-f4.1e100.net > icebox.lan: ICMP echo reply, id 1901, seq 3, length 64
11:28:25.961869 IP icebox.lan > nuq04s29-in-f4.1e100.net: ICMP echo request, id 1901, seq 4, length 64
11:28:25.976176 IP nuq04s29-in-f4.1e100.net > icebox.lan: ICMP echo reply, id 1901, seq 4, length 64
11:28:26.963667 IP icebox.lan > nuq04s29-in-f4.1e100.net: ICMP echo request, id 1901, seq 5, length 64
11:28:26.976137 IP nuq04s29-in-f4.1e100.net > icebox.lan: ICMP echo reply, id 1901, seq 5, length 64
11:28:30.674953 ARP, Request who-has 172.254.1.0 tell ThePickleParty.lan, length 28
11:28:31.190665 IP ThePickleParty.lan.51056 > 192.168.86.255.rfe: UDP, length 306

You’ll notice a lot of stuff happening when you run a packet capture, well that’s to be expected there’s a lot of network activity happening in the background. In my above example, I’ve taken only a snippet of my capture specifically the time when I decided to ping www.google.com.

Understanding the output

11:28:23.958840 IP icebox.lan > nuq04s29-in-f4.1e100.net: ICMP echo request, id 1901, seq 2, length 64
11:28:23.970928 IP nuq04s29-in-f4.1e100.net > icebox.lan: ICMP echo reply, id 1901, seq 2, length 64
  • The first field is a timestamp of the network activity
  • IP, this contains the protocol information
  • Next, you’ll see the source and destination address: icebox.lan > nuq04s29-in-f4.1e100.net
  • seq, this is the TCP packets’s starting and ending sequence number
  • length, length in bytes

As you can see from our tcpdump output, we are sending an ICMP echo request packet to www.google.com and getting an ICMP echo reply packet in return! Also note that different packets will output different information, refer to the manpage to see what those are.

Writing tcpdump output to a file

$ sudo tcpdump -w /some/file

Some final thoughts: we only scraped the surface of the subject of packet analysis. There is so much you can look at and we haven’t even touched upon going even deeper with Hex and ASCII output. There are plenty of resources online to help you learn more about packet analyzers and I urge you to find them!

Exercise

Download and install the Wireshark tool and play around with the interface.

Quiz Question

# What is the flag to capture a specific interface with tcpdump? > - For each network interface, a number and an interface name, possibly followed by a text description of the interface, are printed. The interface name or the number can be supplied to the -i flag to specify an interface on which to capture. > - -c flag. To capture a certain number of frames and then exit, use the -c flag. Example usage: tcpdump will exit after capturing 100 frames by specifying -c 100 1. [ ] -tcpd 2. [ ] -n 3. [ ] -c 4. [x] -i