regreSSHion: The Bug That Came Back

In 2006 OpenSSH killed a remote-root hole in its signal handler. In 2021 a code cleanup quietly deleted the fix, and the same bug shipped to millions of servers for three years — because nothing tested that it stayed dead.

On July 1, 2024, Qualys published the details of a remote, unauthenticated, root-level code execution bug in OpenSSH’s server. They called it regreSSHion, tracked as CVE-2024-6387, and the name is the whole story in one word. This wasn’t a new bug. It was an old one, fixed in 2006, that had crawled back into the code in 2021 and sat there — in the single most exposed network service on the internet — for three years before anyone noticed it was back.

The technical details are worth understanding, but the part that should keep you up at night isn’t the exploit. It’s how a bug that was found, understood, and patched can return years later through the front door, in a mature project run by people who are careful for a living.

The bug itself

sshd, the OpenSSH server, gives every incoming connection a grace period to authenticate. If you connect and then just… sit there, saying nothing, sshd eventually gives up on you. The default is 120 seconds — the LoginGraceTime setting. Under the hood, sshd arms a timer with alarm(), and when it fires, the operating system delivers a SIGALRM signal that yanks execution out of whatever it was doing and jumps into a signal handler to clean up the abandoned connection.

Here is the landmine. A signal can arrive at any moment, including in the middle of something delicate. So the code a signal handler is allowed to run is sharply restricted: it must be “async-signal-safe,” a specific list of functions guaranteed not to break if they interrupt themselves. The classic thing that is not on that list is memory allocation. malloc() and free() keep the heap in a consistent state using internal bookkeeping, and that bookkeeping is briefly inconsistent while they run. If a signal fires mid-malloc, and the handler calls something that also allocates, you now have two pieces of code editing the same heap structures at once. The heap corrupts.

regreSSHion’s handler called syslog(). And syslog(), on a glibc Linux system, can call malloc(). So the sequence is: a connection stalls, the alarm fires while sshd happens to be inside its own memory allocation, the handler runs syslog(), syslog() reaches for the heap that was mid-edit, and the whole thing derails into a state an attacker can steer. It is a race condition — the attacker has to make the alarm land in exactly the wrong microsecond — but a race you can retry is just a lock you haven’t picked yet.

”You can’t win this race” is not a fix

Qualys didn’t just describe the theory; they won the race. In their lab, against a 32-bit Linux target with address-space randomization turned on, it took around 10,000 connection attempts — six to eight hours of continuous hammering — to line the timing up once and land a root shell. On 64-bit the randomization is deeper and they didn’t demonstrate it, but “we didn’t finish it in the lab” is not the same sentence as “it’s safe,” and everyone in the room knew it.

Six hours sounds like a lot until you remember what the target is. This is sshd, listening on port 22, on a huge share of the servers that run the internet. An attacker isn’t racing a stopwatch; they have a botnet and infinite patience. Using Censys and Shodan, Qualys counted more than 14 million OpenSSH instances exposed to the internet running a potentially vulnerable version. The exploit was slow, and the population was enormous, and slow-times-enormous is still a catastrophe.

The CVSS score landed at 8.1 — “high,” not the maxed-out 10.0 you might expect for unauthenticated remote root. That gap is the difficulty tax: the race makes it hard, not impossible. It is exactly the kind of score that gets a bug filed under “we’ll patch it next cycle,” which is the wrong instinct when the ceiling is root on everything.

The regression

Now the part that matters. Where did this come from?

It came from 2006. The identical class of bug was reported back then as CVE-2006-5051 — an unsafe signal handler in sshd — and OpenSSH fixed it. The fix held. Versions from 4.4p1 onward were not vulnerable, for years, because the code carried the guard that kept the dangerous call out of the signal path.

Then, in OpenSSH 8.5p1, released in early 2021, that guard came out. Not through sabotage, not through a clever supply-chain attack — through ordinary code evolution. Something was refactored, a piece was moved, and the specific protection that made the 2006 bug impossible was no longer there. The 2006 vulnerability was, functionally, back. And it shipped. 8.5p1, then 8.6, 8.7, all the way through 9.7p1 in March 2024 — every release for three years carried a resurrected remote-root hole, and the project’s normal review, testing, and audit process did not catch it, because none of those processes were looking for this specific dead thing coming back to life.

That is the definition of a regression, and it is a fundamentally different failure than finding a new bug. A new bug is the cost of writing software; nobody can be blamed for not knowing what they didn’t know. A regression is the cost of forgetting — of fixing something, learning the lesson, and then not leaving behind anything that remembers the lesson on your behalf.

The fix for a bug is a test, not a patch

Here is the opinion, and I’ll state it plainly: a security patch that isn’t accompanied by a test asserting the bug can’t come back is only half a fix. Maybe less than half.

The patch closes the hole today. The test is the only thing that keeps it closed tomorrow, after you and everyone who remembers the incident has moved on and someone new is refactoring the file with the best of intentions. Code review does not catch regressions like this reliably, because a reviewer sees the diff in front of them, not the eighteen-year-old CVE the diff silently undoes. Humans are bad at noticing the absence of a thing. Tests are good at exactly that — a test is a tripwire that fires when a specific behavior changes, and it does not get bored, does not forget, and does not assume the intern knows about 2006.

The catch is that the regreSSHion class of bug is genuinely hard to write a test for. You can’t easily unit-test “under no scheduling interleaving does the SIGALRM handler ever call something that allocates.” Race conditions in signal handlers are timing-dependent and near-impossible to trigger on demand. This is real, and it’s part of why the regression survived. But “hard to test” is a reason to invest in harder tests — signal-handler audits, static analysis that flags non-async-signal-safe calls in signal paths, fuzzing the grace-timer path — not a reason to rely on the collective memory of maintainers, which is exactly the thing that failed.

OpenBSD never had it

The most quietly damning detail: OpenBSD, where OpenSSH actually comes from, was never vulnerable to regreSSHion. Not in 2006’s form, not in 2024’s. The upstream project had handled this exact condition safely since 2001, with a signal-safe logging path that doesn’t reach for the heap from inside a handler. The bug only existed in the “portable” version of OpenSSH — the one adapted to run on Linux and everything else — where the glibc syslog() behavior and the removed guard combined into the hole.

Sit with that. The same code family, split into two branches, and one branch had solved the class of problem so thoroughly that a refactor couldn’t reintroduce it, while the other had solved a specific instance in 2006 and then lost the fix. That’s the difference between fixing a bug and fixing the reason the bug was possible. One survives your own future carelessness. The other is a patch waiting to be un-applied.

What to actually take from this

Patch to OpenSSH 9.8p1 or later; that part is not interesting and you’ve probably done it. Turning LoginGraceTime to 0 was the emergency mitigation — it disables the timer, and therefore the alarm, and therefore the buggy handler — at the cost of removing a mild denial-of-service protection, which was a reasonable trade during the window.

The lasting lesson is smaller and harder than “run updates.” It’s that “we fixed that” has a shelf life. Every bug you patch and walk away from is a fact stored in exactly one place — the memory of the people who were there — and that storage medium quits, forgets, and refactors. regreSSHion is what it looks like when the fix outlives the memory of why it was there: eighteen years to find the bug the first time, a routine cleanup to bring it back, and three years shipping to fourteen million servers before someone found it again. The bug was never the interesting part. The forgetting was.

Continue the conversation

← Back to Blog