Here’s a bug report that has ruined more afternoons than it has any right to. Someone SSHes into a server. The login works. They type ls and it answers instantly. Then they run a command with a page of output, or start a file transfer, and the terminal freezes. Dead. Not slow — stopped. The connection is still “up” by every definition the tools agree on, and nothing in the application logs looks wrong. It just hangs the moment real data starts to move.
Or a webpage. The HTML arrives, the layout paints, and then it stalls with a spinner where a large image or a JSON payload should be. Refresh and maybe it works this time. Maybe not.
The instinct is always the same: blame the application. It’s SSH. It’s the load balancer. It’s the framework. It is almost never any of those. What you’re looking at is a path MTU black hole — a router somewhere in the middle silently throwing away your biggest packets, and a diagnostic message that would have told everyone exactly what happened getting dropped by a firewall that someone hardened in 2009. The reason small things work and big things die is the whole fingerprint, and once you can read it you’ll see this bug everywhere.
The 1500-byte number nobody chose on purpose
Every link has a maximum transmission unit — the largest packet it will carry in one piece. Ethernet’s MTU is 1500 bytes. That number isn’t a law of physics; it’s a compromise baked into Ethernet in the early 1980s, balancing efficiency against how long one station could hog the wire. It stuck, it spread, and now essentially the entire internet quietly assumes 1500 is normal. Application developers never think about it. Most sysadmins don’t either, until the day it bites.
The trouble is that 1500 is only true for plain Ethernet. The moment your traffic crosses something with encapsulation overhead, the usable MTU drops. PPPoE — the framing a lot of DSL and fiber connections run on — steals 8 bytes, leaving 1492. A VPN wraps every packet in another header: WireGuard, IPsec, and GRE tunnels all carve out their own chunk, dropping the real MTU to something like 1420, or 1400, or a number you have to look up per protocol. So the path from your laptop to a server is rarely a clean run of 1500-byte hops. Somewhere in the middle there’s usually a link that’s smaller, and packets sized for 1500 don’t fit through it.
What’s supposed to happen
The internet has a mechanism for exactly this, and on paper it’s elegant. It’s called Path MTU Discovery, standardized in RFC 1191 back in 1990.
Modern TCP sets a flag on its packets called DF — Don’t Fragment. It’s an instruction to every router along the way: do not chop this packet into smaller pieces to make it fit; if it’s too big, refuse it. So when a full-size packet hits that smaller link, the router in front of it doesn’t quietly fragment it. It drops the packet and sends back an ICMP message — type 3, code 4, whose name is refreshingly literal: “Fragmentation Needed and Don’t Fragment was Set.” Crucially, that message includes the MTU of the link that couldn’t fit the packet.
The sender receives it, reads “oh, the largest that fits is 1400,” shrinks its packets to 1400, and retransmits. From then on everything flows. The connection self-heals in a fraction of a second, and nobody ever knows it happened. That’s the design working. It’s genuinely clever: the network tells you the exact size to use, and you adapt.
The entire thing hinges on one assumption. That ICMP message has to make it back to the sender.
Why the message gets eaten
Sometime in the 2000s, a piece of folk wisdom calcified in the firewall community: ICMP is a security risk, block it. It came from a real place — ping floods, ICMP redirect abuse, tunneling data through echo packets — but it got applied with a machete instead of a scalpel. Administrators wrote rules that dropped ICMP wholesale, all types, at the network edge. And “all types” swept up type 3 code 4, the one message the entire path-MTU mechanism depends on.
Now watch what happens. Your packet sized for 1500 hits the 1400-byte link. The router drops it and dutifully sends back “fragmentation needed, MTU 1400.” That reply travels toward your sender, reaches a firewall configured to eat all ICMP, and vanishes. The sender never learns anything. As far as it knows, the packet just… didn’t get acknowledged. So TCP does the one thing it always does with an unacknowledged packet: it waits, then retransmits the exact same oversized packet. Which hits the same link, gets dropped again, generates the same ICMP message, which gets eaten again. Forever.
That is the black hole. The router is doing its job. The ICMP mechanism is doing its job. A firewall silently removed the one link in the chain that carries the bad news, and the result is a connection that establishes perfectly and then dies the instant it needs to send a full-size packet.
Which is exactly why the small stuff works. The TCP handshake — SYN, SYN-ACK, ACK — is tiny, comfortably under any MTU. Your ls, your login prompt, the first few hundred bytes of an HTTP response: all small, all fit, all fine. It’s the bulk transfer, the big output, the image, the POST body — the packets that actually push up against 1500 — that fall into the hole. The connection isn’t broken. It’s size-selective, and size-selective failure is the signature you learn to recognize on sight.
RFC 2923 wrote all of this down in the year 2000, under the wonderfully dry title “TCP Problems with Path MTU Discovery.” The problem is old enough to vote. It has not gone away, because the thing that causes it — reflexively blocking ICMP — is still treated as a best practice by people who have never watched a transfer hang because of it.
IPv6 made it mandatory, not optional
If you think you can dodge this by hoping fragmentation bails you out, IPv6 has news. In IPv4, a router is at least allowed to fragment an oversized packet itself when DF isn’t set. IPv6 removed that entirely: routers in the middle of a path are forbidden from fragmenting. If a packet is too big for the next link, the router’s only move is to drop it and send back an ICMPv6 “Packet Too Big” message (type 2). Path MTU Discovery isn’t a nice-to-have in IPv6 — it’s the only mechanism, structurally load-bearing.
So the same firewall reflex, applied to ICMPv6, is even more destructive. Block ICMPv6 Packet Too Big and you don’t get degraded IPv6; you get IPv6 that completes handshakes and then black-holes every large transfer, while IPv4 on the same box works fine and sends everyone chasing the wrong ghost. As IPv6 deployment climbs, this failure mode climbs with it.
How to catch it in thirty seconds
You don’t need fancy tooling. ping can set the DF bit and a payload size, which lets you binary-search the path MTU by hand. On Linux:
ping -M do -s 1472 example.comThe 1472 is deliberate: 1472 bytes of payload plus 8 bytes of ICMP header plus 20 bytes of IP header equals exactly 1500. If that succeeds, the full path carries 1500-byte packets. If it times out but a smaller size — say -s 1400 — goes through cleanly, you’ve found it: a link on the path can’t take full-size packets and the ICMP that should report that fact is being dropped. A working path either succeeds at 1472 or, if it can’t, tells you the real MTU. A black hole just stops, silently, exactly the way your stalled transfer did.
Two real fixes and one that isn’t
The tempting non-fix is to unblock ICMP everywhere and declare PMTUD healthy. It’s correct in principle and hopeless in practice, because you don’t control every firewall between you and the rest of the internet. Some middlebox you’ll never see will keep eating the message.
The fix that actually holds up in production is MSS clamping. TCP negotiates a Maximum Segment Size during the handshake — each side advertises the largest chunk it’s willing to receive. A router at the edge of a smaller-MTU link (your VPN concentrator, your PPPoE gateway) can rewrite that advertised MSS downward as the SYN passes through, so both ends agree on a segment size that fits the real path from the start. No oversized packet is ever sent, so no ICMP is ever needed. It sidesteps the black hole instead of depending on the broken message getting through. On Linux firewalls it’s a single rule — --clamp-mss-to-pmtu — and it is, unglamorously, how most of the internet’s tunnels quietly avoid this problem.
The more principled answer is RFC 4821, Packetization Layer Path MTU Discovery. Instead of trusting ICMP to report failures, the transport layer probes — it deliberately tries larger packets and watches whether they get acknowledged, inferring the path MTU from what survives. It needs no help from ICMP at all, which is precisely the point: it assumes the diagnostic message is missing, because in the real world it often is. It’s the right design. It’s also, twenty years on, still under-deployed relative to how thoroughly it would kill this bug — which tells you something about how much easier it is to clamp an MSS and move on than to fix the layer properly.
The claim your monitoring isn’t making
Here’s what I actually want you to take from this. “The network is up” and “the network passes full-size packets” are different statements, and almost everything that watches your infrastructure only checks the first. Your uptime monitor pings the host — small packet, gets through, green. Your health check hits an endpoint that returns a few hundred bytes — small, green. Everything is green, and meanwhile every large upload silently dies in a black hole because a 1500-byte assumption met a 1400-byte tunnel and the message that would have reconciled them got dropped at some firewall you don’t administer.
The 1500-byte MTU is one of those numbers the internet agreed on so long ago that it became invisible, and invisible assumptions are the ones that fail loudest and get diagnosed slowest. When a connection works but a big transfer hangs, don’t start with the application. Start with the size of the packets, because the packets are trying to tell you where the hole is.