Live data from Hacker News

Thanksgiving 2023 security incident

blog.cloudflare.com

21–30 of 336 posts

Re: Thanksgiving 2023 security incident

#21
post #8

Earlier quoted context omitted.

The challenge being, who else could possibly handle Cloudflare's requirements? I imagine the next step is to build their own, and that's obviously not an easy pill to swallow.

They already run their own zero trust infrastructure for customers, kinda surprised they are not dogfooding it. https://www.cloudflare.com/plans/zero-trust-services/

We use our Zero Trust stuff extensively. In fact, we built it for ourselves initially.

Re: Thanksgiving 2023 security incident

#22
post #9

Great write up. > Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories > They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. > Since the Smartsheet service account had administra…

> Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though.

I think my org has on the order of 3 repositories per dev? They seem to have 3200 employees, with what I assume to be a slightly higher rate of devs, so you’d expect around 6-7 thousand?

2M Jira tickets is probably easily achieved if you create tickets using any automated process.

Re: Thanksgiving 2023 security incident

#23
post #8

Earlier quoted context omitted.

The challenge being, who else could possibly handle Cloudflare's requirements? I imagine the next step is to build their own, and that's obviously not an easy pill to swallow.

They already run their own zero trust infrastructure for customers, kinda surprised they are not dogfooding it. https://www.cloudflare.com/plans/zero-trust-services/

They are, but they don't have management for user accounts, 2fa, etc. You setup a connection to something like Okta, Google Apps, O365, SAML, etc to be your persistent user db and cloudflare just enforces it.

I wouldn't be surprised if they are working on first party IAM user support though.

Re: Thanksgiving 2023 security incident

#25

Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business. Yes, they aren’t perfect. They do some things that I disagree with. But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this. Thank you for the blog post!

[flagged]

What are they now, if not an engineering company?

Re: Thanksgiving 2023 security incident

#26
post #5

> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian produ…

> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.

Aha, the old replace-your-trusted-hardware trick.

Re: Thanksgiving 2023 security incident

#27
post #6

> we were (for the second time) the victim of a compromise of Okta’s systems I'm curious if they're rethinking being on Okta.

My company will only give us new laptops that are preinstalled with Okta’s management system.

I am grandfathered in to an old MacBook that has absolutely no management software on it, from the “Early Days” when there was no IT and we just got brand new untouched laptops.

They offered me an upgrade to an M1/M2 pro, but I refused, saying that I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.

Since that would hugely disrupt my work, I can’t upgrade. Maybe I can use incidents like this to justify my beliefs to the IT department…

Re: Thanksgiving 2023 security incident

#28
post #9

Great write up. > Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories > They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. > Since the Smartsheet service account had administra…

The number of repositories sounds really high. The number of tickets doesn't.

Re: Thanksgiving 2023 security incident

#29
post #11

Reading this 2 months after the fact feels a bit late, but I guess it’s better for your stock price if these revelations happen with remediation already in hand? Since they didn’t really have reason to believe my data was accessed, maybe that’s ok. I know from firsthand experience how hard rotating all your credentials across the whole org is.

The final security report was only released yesterday, and the amount of work they did to make sure all of their systems were secure after the incident was A Lot; two months is pretty quick for a project of that scale IMO.

Yes, but if after two months they’d found out that customer data had been compromised, that would be a little late for me to do anything about it.
Post reply on HN