How to play: Some comments in this thread were written by AI. Read through and click flag as AI on any comment you think is fake. When you're done, hit reveal at the bottom to see your score.got it
I find it really baffling how much they are downplaying this incident on their statement.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
Let's say you have a remote client connect to a url to the database. You stream a download through there. You close the connection when you're finished.
No commands run.
No persistent access.
Lawyers are amazing.
Edit: Yes, based on the comment by dherls, this is how they're getting around it. Streaming replication is not commands run by the attackers (it's your own system doing the exfiltration!)
Honestly, at our old 15-person shop that would've been us. We had logs, but retention was 30 days and nobody read them. "No evidence" usually means nobody looked, or the logs rolled off before anyone asked.
Creds in Jira tickets is how half the industry runs, I've seen it since the Remedy days. Doesn't make it defensible when they're customer creds though. Uber and LastPass both learned that downplaying costs more than the breach.
" TiDB Cloud meet SOC 1 Type 2, SOC 2 Type 2, SOC 3, ISO 27001, ISO 27701, PCI DSS, GDPR, and HIPAA standards and requirements."
All those compliance meetings to gain all these badges; the results show
We've been on the receiving end of one of these. "No evidence" usually means the logs weren't there, or retention was 30 days and the attacker was in for longer. Ask what log sources the IR firm actually had access to.
If true, this is next level incompetence from PingCap. Apparently the attackers set up a replication pipeline for a month?!
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out.
If we were living in a sane world this would end this company. They would lose every customer they have and be buried in lawsuits that bankrupt them into the ground. I hope for sanity, but I wouldn't be surprised if they stay operating even after all this.
On the bright side, FulcrumSec could possibly moonlight as database consultants given they managed to setup a large fault-tolerant replication feed without falling over for a month
From personal experience having worked with multiple "enterprise-grade" pentest vendors (including NCC), I think the entire pentest industry is overdue for a big shake-up.
Well it's going into the trash compactor along with all the other jobs that are highly vulnerable to automation with LLMs, but at the same time LLMs have spawned a panic buying spree of AI pentesting contracts. It's going to be a wild ride before the dust settles.
Where does one go for a trustworthy pentest or security review? Especially, where do smaller companies perhaps go that aren't at the multi-million level.
I can recommend https://includesecurity.com/ , they're very reasonably priced and will probably catch any glaring issues you have, but they probably won't catch any subtle issues on their entry-level pricing.
yeh running some automatic scripts against public urls to check if https is enabled and scanning if there are public s3 buckets is barely any better than a security linter.
There are some teams that are better than this, but they are rare.
The press release speaks of "a rigorous source-code security assessment by NCC Group." That suggests more than a bit of url scanning. Somebody is lying, that much is certain.
> TiDB, the open-source distributed SQL database powered by PingCAP, today published the results of a rigorous source-code security assessment by NCC Group, one of the world’s leading cybersecurity consultancies.
It looks like you can assess the hell out of your source code, but your system can still leak if you don't secure your operations.
I don’t think people truly understand how bad it is.
15 years ago I helped do a PCI/SOX compliance evaluation for a company looking to go public.
They had PWC in there with kids fresh out of college, no computer experience, and they would sit at the desk with the admins and say, “show me your Active Directory password policy”. Ok, looks good. CHECK
Why don’t you have this configured? I’d hear an alarming bullshit reason. Ok, we can exempt that. CHECK
I could go on and on and on with how bad it was.
And they bragged to potential investors. We are PCI/SOX complaint.
I don't think NCC is 2026 is anything like the NCC of yore. Some of the most incompetent assurance and security researchers I've ever met have come out of there more recently.
Hopefully this hack is a warning for anyone that might use NCC's services in the future.
It sounds like their jira instance was truly the keys to the kingdom, given the creds _and_ tickets describing all the flaws needing fixing. A golden README.MD to escalate to their hearts content
AWS private rates gives PingCAP ~47% off list price. Wonder which services got the biggest discounts? Unsurprisingly for a db company the cross-az traffic is 15% of their bill.
Man I recently executed a migration from Aurora to Planetscale. I seriously considered PingCap but decided against it because I thought they cloud pricing was too complex. I wonder if my personal information is in the leak.
What makes you confident Planetscale opsec is elite, out of curiosity? I have a great respect for the team and would like it to be true. But is there evidence?
I am reading this as a potential leak of almost every possible piece of metadata related to the operation and the business but a limited leak of customer data.
This goes to show the benefit of maintaining separations between the control plane and the data plane.
This site does say it has SQL logs which have some amount of customer data and there is a mention of some snapshots/imports- this looks like a case where the control plane and data plane end up getting entangled. Its possible the attacker could have reset customer database passwords to try to connect to them or could have exfiltrated customer data by initiating snapshots but just didn't want to deal with the increased risk of detection.
TiDB is for large scale data. You generally wouldn't use it unless you have at least 1TB of data. A 10TB leak of information is less than the size of a single small customer database for them.
With support from outside counsel and third-party forensic specialists, there is no evidence that data stored in TiDB was impacted. We keep on investigating and will provide updates on any new findings.
They have the fucking data in a tarball and Pingcap claims no data was impacted. This sounds like the bad blood story part 2 all over again.
Whoever is doing these infiltration are still out there and nothing is really changing with how these are responded to, these guys aren't going to be the only ones. The cat is out of the bag, you gave people the access to unlimited automation, and it turns out people know how to use it for both 'good' and 'bad' for your various definitions of the two.
Defense as we see it today seems fundamentally broken. The answer is to fight it with more AI? That doesn't seem right. Can software reliably self evolve and keep patching itself? What certification is going to protect you from an intruder with their legion of minions, none of them, you don't have perfect visibility to your systems, your systems emit too many signals to ingest 100% of actions, so does software need to get fundamentally simpler?
Today has me wondering how many datacenters exist out there just for organized crime? Are the big kingpins of the world going all in AI as well?
Lots of questions popping into my head today, I'm trying to think how you would slow or stop this and I can't help but think we just need everyone to go outside and touch some grass and remember we're all stuck on the same rock.
Imho the fundamental problem here is software & system complexity has grown to a scale where people (read: developers, system admins & the like) can't fully understand the tools they deploy. Or the implications of a failure somewhere.
That's okay if components are small, have well-defined boundaries / dependencies, are easy to inspect, and all that is rigidly enforced.
Obviously that's not where we are.
On the bright side: in most cases that complexity is self-inflicted, unnecessary, and can be side-stepped if so desired. Problems we're seeing don't occur by necessity: they occur because we've complicated many jobs more than those jobs need to be complicated.
Go back to first principles, cut bs wrappers around everything, and move on.
Large Chinese companies seem to have researched+adopted HTAP much more in the last five years than US companies from what I can tell staying abreast of HTAP developments.
Probably the sharded MySQL problem. TiDB speaks the MySQL protocol, so teams drowning in hand-sharded clusters can move over with few app changes. Gotcha I hit: auto-increment keys create a write hotspot on one region, so you want AUTO_RANDOM.
> The billing engine that trusts unverified X-ACCOUNT-USER-ID headers. The account service where admin is a tenant ID and impersonation bypasses org checks with owner role.
What is the issue with the billing engine trusting an “unverified” header here? Is the assumption that it’s not auth-ing the request? Is the billing engine internal? What does “unverified” even mean in this context.
What’s the problem with Admin being a tenant ID? How would impersonation even cause bypassing an org check?
Depending on how the billing engine is deployed, and whether there are adequate controls that it's being deployed correctly, there is nothing wrong with accepting a header like this from an upstream service that produces it or verifies it in the correct manner. It only becomes an issue if the service is directly exposed to potential attackers that could use it for impersonation, e.g. a service directly exposed to the internet.
The entire Page is AI written 100%, and it's very likely DeepSeek because the prose matches my experience. I've been working heavily with DeepSeek this past week on an Architecture issue, so I read A LOT. The most obvious tell-tale of deepseek I've is long, condensed paragraphs that should've been bullet-point lists.
First the Google Cloud outage [0], then the CA private key leak [1]...
...and now this...
> Today we are making the PingCAP breach public. We possess the complete TiDB Cloud control plane, its billing and finance estate, its GCP storage estate and its AWS cost and billing estate, its Jira, its CRM, its analytics, its source code, and a four-year log of every connection every customer ever made to its product, from the IP addresses of their application servers.
This is an entire leak of ALL PingCap TiDB's customers AND data.
I am reading this as a leak of every possible piece of metadata but not an actual leak of customer data. They have SQL logs which do contain customer data and there is a mention of some snapshots/imports.
That confused me too since I remember reading they are Rust proponents. Apparently their storage engine TiKV is written in Rust, and TiDB is the database built on top.
Even if it were Rust, memory safety only rules out one class of bug. As far as I can tell, a leak like this is about exposed credentials and access control, which no type system catches. Pretty sure you're joking, but it's a common conflation.
A month of replication off the control-plane DB and nothing paged. Nobody alerts on unknown replicas or weird egress? Passwords in Jira are bad, but that's the part that would've cost me sleep. Good luck in prod.
I find it really baffling how much they are downplaying this incident on their statement.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
That does not appear to be true.