Heartbleed: The Leak That Left No Fingerprints

The worst part of Heartbleed was never the memory it leaked. It was that afterward, nobody could tell whether their memory had been leaked at all.

On April 7, 2014, two teams disclosed the same bug on the same day — Neel Mehta at Google had spotted it, and a Finnish company called Codenomicon had found it independently, named it Heartbleed, and given it a logo and a website, which is how a memory-safety bug ended up with better branding than most startups. The bug lived in OpenSSL, the library that terminates TLS for a large fraction of the web’s “secure” servers. It let anyone on the internet read up to 64 kilobytes of a server’s live memory, over and over, with a request the server treated as completely routine. Private keys, session cookies, passwords in flight, other people’s requests still sitting in the heap — whatever happened to be nearby.

And the read left no trace. That’s the sentence to hold onto. Everything else follows from it.

A feature almost nobody needed

TLS has a heartbeat extension, defined in RFC 6520. The idea is small and reasonable: to keep a connection alive without renegotiating, one side sends a little payload and says “send this back to me,” and the other side echoes it. It’s mostly useful for DTLS, the datagram version of TLS, where you want to check a peer is still there and probe path MTU. For ordinary HTTPS over TCP it does almost nothing you need. It was added anyway.

The request carries three things that matter: a type byte, a 16-bit length field, and the payload itself. The responder is supposed to allocate a buffer, copy the payload back out, and send it. Here, roughly, is what OpenSSL did:

/* p points at the incoming heartbeat message */
hbtype = *p++;
n2s(p, payload);        /* payload = the length the CLIENT claims */
pl = p;                 /* pl points at the client's actual data   */

/* ... allocate a response and copy payload bytes back ... */
buffer = OPENSSL_malloc(1 + 2 + payload + padding);
memcpy(bp, pl, payload);   /* trust the claimed length, copy that many bytes */

Read the memcpy line again. payload is a number the client sent. The code copies that many bytes starting from the client’s data — but it never checks that the client actually sent that many bytes. So you send a heartbeat with a one-byte payload and a length field that says 65,535, and OpenSSL faithfully copies 65,535 bytes back to you: your one byte, followed by 64KB of whatever was sitting next to it in the heap. It hands you memory you never sent it, because you told it the number and it believed you.

That’s the whole bug. A textbook buffer over-read: the guard that should have said “is the claimed length actually within the record we received?” simply wasn’t there.

The fix was three lines

When OpenSSL shipped 1.0.1g on the same day as the disclosure, the patch was almost anticlimactic. It added the check that had been missing:

/* Discard silently if the record is too short to hold what it claims */
if (1 + 2 + 16 > s->s3->rrec.length)
    return 0;
if (1 + 2 + payload + 16 > s->s3->rrec.length)
    return 0;

Compare the claimed length against the actual length of the record that arrived. If the message can’t possibly contain what it says it contains, drop it and don’t answer. That’s it. The difference between “a large fraction of the internet’s private keys are readable by anyone” and “not” was a comparison against a length the code already had sitting in a struct.

The bug had shipped in OpenSSL 1.0.1 on March 14, 2012, from a commit written on New Year’s Eve 2011 and merged by a maintainer who didn’t catch the missing check. It sat in production for two years. Every version from 1.0.1 through 1.0.1f was exploitable. Netcraft’s estimate at disclosure put roughly 17% of trusted HTTPS sites — about half a million certificates — at risk. Two years is a long time for a hole that size to be open, and here is the thing that made two years unbearable rather than merely bad.

You could not audit the damage

A normal exploit leaves evidence. It crashes something, it trips a log line, it sends malformed input that a filter records, it triggers an alert. Heartbleed did none of that, by design, because it wasn’t malformed at all. A malicious heartbeat is a valid heartbeat. The type byte is correct, the length field is correct as far as the protocol is concerned, the server processes it exactly the way it processes a legitimate one. There is no error, no exception, no anomaly. The attacker asks a well-formed question and the server gives a well-formed answer that happens to contain its own secrets.

OpenSSL didn’t log heartbeat contents — why would it? So for the overwhelming majority of servers, there was no record of who had sent heartbeats, how many, or how large. Which means that after April 7, every operator running a vulnerable version faced a question with no answer available: were we exploited, and if so, what walked out the door? You couldn’t check. The logs that would have told you were never written, because nothing ever thought this traffic was worth recording.

When you cannot prove you weren’t robbed, you have to assume you were. That single epistemic fact is what turned a three-line patch into weeks of global work. Patching stopped the future leaks instantly. It said nothing about the past, and the past was two years long.

The reckoning nobody wanted: revocation

If you have to assume your server’s memory was read, you have to assume your TLS private key — which lives in exactly that memory — was among what leaked. Some people initially doubted the key itself was extractable, arguing it wouldn’t be sitting in the reachable heap. Cloudflare settled the argument by standing up a deliberately vulnerable server and inviting the internet to steal its key. On April 11, 2014, a software engineer named Fedor Indutny did, and then others did too. Worst case confirmed: the key comes out.

So every operator had to do the thing certificate revocation exists for and almost never gets exercised at scale: generate new keys, get new certificates, and revoke the old ones. The reissue part worked. The revoke part is where the industry looked in the mirror and didn’t like what it saw. Revocation in 2014 rested on two broken mechanisms — CRLs, giant lists of revoked serial numbers that browsers were increasingly too impatient to download, and OCSP, a live lookup that browsers “soft-fail” on, meaning if the check times out they shrug and proceed as if the certificate were fine. An attacker holding a stolen key can just block the OCSP check and sail through. Heartbleed produced one of the largest revocation events the web had seen, the CRLs ballooned, and the uncomfortable conclusion was that revoking a certificate doesn’t reliably stop anyone from using it. That reckoning is a direct line to why the ecosystem gave up on OCSP years later and turned it off entirely, and why the real answer turned out to be making certificates live short enough that revocation barely matters. Heartbleed didn’t break revocation. It just forced everyone to use it at once, which is how they discovered it had never really worked.

The $2,000 footnote

There’s a coda that got less attention than the logo but matters more. When people went looking at who, exactly, maintained OpenSSL — this library sitting underneath the padlock on a huge share of the web — the answer was: barely anyone. Steve Marquess, who ran the OpenSSL Software Foundation, said the project took in about $2,000 a year in donations and had essentially one person able to work on it full-time. The cryptographic floor of the internet was a volunteer project running on the budget of a decent used car.

Within weeks the Linux Foundation announced the Core Infrastructure Initiative, and Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Rackspace and VMware put real money behind funding developers for exactly this kind of load-bearing, invisible, unglamorous code. It took a global panic to notice the thing everyone depended on had no one watching it — the same lesson that would land again, a decade later, when a single unpaid maintainer’s exhaustion nearly shipped a backdoor into the SSH servers of mainstream Linux distributions. We keep re-learning that “critical” and “funded” are unrelated properties.

What Heartbleed should have taught us

The tidy lesson is “use memory-safe languages,” and it’s not wrong — an over-read like this is a crash in Rust, not a silent leak, and memory-safe TLS stacks like rustls exist for exactly this reason. But the durable lesson isn’t about the language. It’s about what a read bug costs.

We spend most of our security attention on writes — code execution, injection, the attacker making the machine do something. Heartbleed did nothing. It only looked. And a bug that only looks, if it looks quietly enough, is worse to live through than one that acts, because acting leaves a mark you can investigate and looking doesn’t. The damage from Heartbleed wasn’t bounded by what any attacker actually took. It was bounded by what they might have taken, and since no one could measure that, everyone had to pay the full price of the maximum. The operators who rekeyed did it not because they were known to be compromised, but because they couldn’t be known not to be. And many never did: a month after disclosure, Netcraft found that most affected sites had neither reissued nor revoked their certificates.

Build systems that can answer “were we hit?” — that log the right thing, that make abuse look different from use. The three-line patch closed the hole in an afternoon. The two years of blindness behind it is the part we never got back.