Yes this is to be expected. I've mentioned multiple times over the years that TLS CA issuance & validation's many security holes (>=14 at last count) could be solved by changing how certificates are issued. I've never had the kind of clout to get that message wide enough that anyone would take it serious.

One of Web PKI's security holes is the fact that any CA can issue valid certs for any domain. The only official "mitigation" for that is voluntary and can be defeated.

The solution to that is to rearchitect the Web PKI ecosystem to use domain registrars as the sole source of truth for which CA is allowed to issue valid certs, in addition to cryptographic fingerprints of the source of the originator and issuer. I won't rehash it here but it's not technically difficult and would make it so only the domain owner could issue certs, and valid certs could only come from the CA the domain owner authorizes.

Maybe if this keeps happening, people will realize it's worth working on? But I doubt it, as a lot of money is at stake, and nobody wants to risk that just to stop governments and cybercriminals from spying on the occasional connection. If it was blatant and obvious then they might have to act; as long as it's kept covert and hard to prove, things stay the same.

The jabber.ru post referenced here presents clear evidence (in the section titled "Network") that the malicious actor was able to reroute traffic going to the legitimate jabber.ru server. An attacker in this position does not need an RCE to get a cert, they can just get one issued the normal way, because they do effectively control the IP address that the domain is pointing to.

We should have listened to Rachel!

>I've been aware of the ACME protocol for a while. I have tech notes going back as far as 2018, and every time I looked at it, I recoiled in horror. The whole thing amounts to "throw in every little bit of webshit tech that we can", and it makes for a real problem to try to implement this in a safe and thorough way. Many of the existing clients are also scary code, and I was not about to run any of them on my machines. They haven't earned the right to run with privileges for my private keys and/or ability to frob the web server (as root!) with their careless ways.

https://rachelbythebay.com/w/2025/05/22/ssl/

This isn't what parallel reconstruction means. This seems to be reverse engineering the attack.

Indeed. Parallel construction is when law enforcement doesn't want to burn their source or their source is unlawful, so they find another way to justify a warrant or prosecution even tho their initial justification comes from a different source.

This has been upheld as lawful, but also can unfortunately be used to disguise unlawful behavior.

Reverse engineering is not related to parallel construction.

IIRC parallel construction also covers when they reconstruct the evidence chain to avoid revealing how they knew where to look initially, not just the source. So the title might be gesturing at that usage, even if loosely applied.

I thought certificate transparency was the thing that was supposed to prevent exactly what this article is describing. What if anything is incorrect about my model of the world in this respect?

Basically, CT did indeed worked as designed, but there was no monitoring by the domain authors (which to be fair there are a dearth of solutions of the time).

On a related note, Let's Encrypt also issued the presumably-interception certificates. This can be possibly something that requires interception at the VPS level (otherwise we already detected the BGP leaks). Presumably, Hetzner was forced to do a raw interception and then redirecting all relevant ports to a middlebox for inspection and CA issuance (and since that the ACME spec is well-defined, they can simply check if the handshake contains the TLS ALPN challenge and then redirect them to special code that will reply with the correct things).

Certificate transparency worked exactly as designed in this case. Monitoring public certificate transparency logs for anomalies is a different story entirely.

By breaking the software facilitating https via ACME itself, no anomalous certificate transparency logs would have needed to have been created at all.

The front door is locked quite tightly with a watchful security camera, but the window has been left unlocked. Also no one is watching the camera feed.

CT provides a tamper-evident log of issued certificates, not of the processes that request them. Compromising ACME software means an attacker controls what gets requested in the first place - CT stays clean. This attack class, targeting the provisioning layer rather than the CA itself, seems underexamined relative to the attention CT gets as a security primitive.

Nothing, although it's more mitigate than prevent per se. They simply did not have alerting set up against the CT logs. It is one of the lessons they highlighted in their own postmortem.

If you're a CA you can just issue a cert and not publish it in the CT logs. You're not supposed to do that, but there is nothing stopping it. And the attack isn't stopped even if they do publish in CT. And you have to monitor for it anyway.

Every single mitigation for known Web PKI vulns can be worked around (if people use them, which virtually nobody does).

CT indeed worked out pretty well. At least until bots started hammering crt.sh making it unreliable, and those that want to be alerted to newly issued certificated appeared in the logs need to pay for some purpose-built service instead of just adding a relevant query to their feed reader.

The wrong part is that Let's Encrypt was willing to issue a valid cert to anyone who can temporarily redirect traffic. The authorization should have been done better, for example, sending a certificate to operator's email.

>the various ACME clients like acme.sh are run with elevated privileges

Its really not that difficult to not grant excessive privileges - at the very least for recurring ("cron") runs, once filesystem structure, cache invalidation triggers and web server configuration are in place. Its a shame this is still taught in the "just run as admin" style.

That capability should be added to acme.sh, etc so that it automatically runs with minimal privileges for the invoked task. But people seem to assume privilege management is the sole responsibility of the packager or caller, despite the tool itself being better placed to know precisely which privileges are needed for the particular task it's performing.

acme-client on OpenBSD does this, using privilege separated processes that each in turn use pledge and unveil. You wouldn't know without looking at the source code because it's entirely transparent.

We ran into this at a previous job and ended up splitting it into two scripts — one privileged that ran once to set up the cert directory with the right permissions, and a cron job that ran as a dedicated acme user with write access only to that directory. Took maybe an hour to set up, saved a lot of headaches.

> TLS wiretapping with root-CA-signed certificates is a thing that both happens and verifiably has happened. (...) This being a fact rather than a conspiracy theory tends to upset people.

Maybe what people get upset about is catchy misleading [0] summaries like this, which suggest [0] a CA - nation state collusion, despite the actual story going in a completely different [0] direction? The thing that would be actually big news [0]?

[0] in the eye of the beholder of course, as always

I could see this actually being a real parallel reconstruction for a state actor that did issue certificates from a compromised CA. If any evidence points back to them, they can just say the server was hacked with the acme RCE to generate different certs. There probably won't be a way to legally verify that such a thing never happened.

This is a reminder why you should use E2EE messengers only.

One day that'll be illegal. End to end encryption? That obviously means you're a drug trafficking money laundering pedophile terrorist. Off to jail with you despite zero evidence. Maybe they'll declare your efforts to protect yourself as being in contempt of court and then jail you indefinitely until full decryption.

I guess there is an interesting possibility here. Perhaps the targets were encrypting end to end (that is more or less the default now with XMPP clients). With the TLS over top of everything the attackers would not know that. Perhaps they went to all this trouble for nothing.

What LI vendors can break https?

The sloppy ones who want a huge headache and leave a publicly auditable trail a mile long that get analysis blogs written about their mistakes.

The ones whose sales reps can spell "compliance."

Can we get an RSS feed? Loved the post.

So it looks like Hetzner is doing the same thing OVHCloud did with EncroChat and SkyECC. But hey, we have GDPR, keep your data hosted in the EU it is very "safe" there (ironic).

https://en.wikipedia.org/wiki/Shutdown_of_Sky_Global#Communi...

EDIT: Based on German law this certainly is not a lawful interception but one made by an intelligence agency.

Worked on lawful intercept integration for a carrier-grade product years back. The MITM via BGP hijack plus fraudulent cert is exactly the playbook we worried about internally. What's underappreciated here is how short the cert validity window needs to be for CT logs to catch this in time — most operators aren't monitoring CT in near-realtime.