Troubleshooting

Last edited

  • One change at a time. Test each. Revert if it breaks something.
  • Document everything. Before state, every change you make.
  • Work systematically. Pick one end (client or server) and trace through components to the problem.
  • Or use the stack. Start at the top or bottom of the protocol layers and work through.

Working up the stack

Layer 1 (Physical)

Do you have a link light?

ethtool <interface>
ip link show

Is your interface configured properly?

ip link show
# Look for "state UP" vs DOWN

Do your ARP tables show other hosts?

ip neigh
arp -a

Layer 3 (Network)

Is your routing correct?

ip route show
# default via 192.168.2.1 dev enp5s0 proto dhcp src 192.168.2.11 metric 100    DEFAULT
# 10.0.0.0/24 via 192.168.2.34 dev enp5s0 proto static metric 100              ROUTE TO ANOTHER NET

Is there a firewall? Do they allow ICMP?

ping <gateway>
nc -vz 10.0.0.1 8080

Can you ping localhost, can you ping other hosts by IP?

ping 127.0.0.1
ping 192.168.2.4 -n # -n disables reverse dns lookup

Can you ping hosts on another network?

ping 8.8.8.8
traceroute 8.8.8.8 -n # no dns

Layer 7 (Application)

Is DNS working? Can you ping hosts by hostname?

# genent goes through the real system dns stack to see if the system CAN 
# resolve the name, rather than the alternatives who by bypass to test other things.
getent hosts <hostname>      # Will THIS system resolve it?
dig <hostname>               # Is DNS infrastructure working?

Do high level services work? (curl)

curl http://example.com

traceroute

Traceroute used to require setuid permisisons to craft such ICMP, but linux kernel has since changed to allow it.

# defines which Group IDs (GIDs) are allowed to create ICMP Echo sockets without root privileges
# this control can come in handy for containers
/proc/sys/net/ipv4/ping_group_range
0       2147483647
# this is a range of allowed GUIs to perform this action
# 0 - infinity
# you could do 0 - 1000

tcpdump

sudo apt install tcpdump

sudo tcpdump host google.ca # filter for host
sudo tcp src net 192.168.1.0/24 and dst port 80 # filter for http

Wireshark is the GUI version of this with some added features, it uses the same underlying libpcap library. One gotcha is display filters in wireshark only change what you’re viewing not capturing. Wireshark has a ton of CVEs always update.

Network Monitoring

Smokeping

Containerzed drop in ping monitor of a handful of sites, could be useful for quick testing. Really ugly though.

https://hub.docker.com/r/linuxserver/smokeping https://github.com/oetiker/SmokePing

services:
  smokeping:
    image: lscr.io/linuxserver/smokeping:latest
    container_name: smokeping
    hostname: smokeping #optional
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
      - CACHE_DIR=/tmp #optional
    volumes:
      - /path/to/smokeping/config:/config
      - /path/to/smokeping/data:/data
    ports:
      - 80:80
    restart: unless-stopped

iPerf

track network performance - ping based tools are good for verifiyng reachablility but they cant analyze and track network performance.

https://iperf.fr/

  1. Install iperf on both machines

  2. Start the server

    iperf -s # start the server

  3. Connect from the client

    iperf -c server.ip
Edit this page
© 2026 Monjiapawne