Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thanksgiving 2023 security incident

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 336 posts

Re: Thanksgiving 2023 security incident

#31

Post by spenczar5 »

belltaco wrote: post #9

Great write up. > Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories > They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. > Since the Smartsheet service account had administra…

12k git repos can happen if the team uses github enterprise with forking internally.

It can also happen in franken-build systems which encourage decoupling by making separate repos: one repo that defines a service’s API, containing just proto (for example). A second repo that contains generated client code, a third with generated server code, a fourth for implementation, a fifth which supplies integration test harnesses, etc…

Sound insane? It is! But its also how an awful lot of stuff worked at AWS, just as an example.

Re: Thanksgiving 2023 security incident

#34

Post by OJFord »

> The one service token and three accounts were not rotated because mistakenly it was believed they were unused.

Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?

Re: Thanksgiving 2023 security incident

#35

Post by syncsynchalt »

sevg wrote: post #5

> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian produ…

Honestly I wish we'd had an excuse/reason to do an org-wide prod creds refresh like this at some places I've been.

You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.

Re: Thanksgiving 2023 security incident

#36

Post by tomschlick »

jedahan wrote: post #30
jrockway wrote: post #7

Which "nation state" do we think this was?

The writeup contains indicators, including IP addresses, and the location of those addresses. In this case, the IP address associated with the threat actor is currently located in Bucharest, Romania.

No nation state is going to use IPs from their own country if they don't want to be caught. They will use multiple layers of rented VPS's with fake identities to pay for those resources.

Re: Thanksgiving 2023 security incident

#37

Post by nemothekid »

Aeolun wrote: post #29
Cthulhu_ wrote: post #18

Earlier quoted context omitted.

The final security report was only released yesterday, and the amount of work they did to make sure all of their systems were secure after the incident was A Lot; two months is pretty quick for a project of that scale IMO.

Yes, but if after two months they’d found out that customer data had been compromised, that would be a little late for me to do anything about it.

What do you expect them to do? It sounds like you are complaining that they weren't able to instantly ascertain if customer data had been compromised.

Re: Thanksgiving 2023 security incident

#38

Post by margalabargala »

whalesalad wrote: post #15
Icathian wrote: post #8

Earlier quoted context omitted.

The challenge being, who else could possibly handle Cloudflare's requirements? I imagine the next step is to build their own, and that's obviously not an easy pill to swallow.

They already run their own zero trust infrastructure for customers, kinda surprised they are not dogfooding it. https://www.cloudflare.com/plans/zero-trust-services/

There are good reasons not to dogfood critical services like that; it can make recovering from unexpected issues much harder if you introduce mutual dependencies.

For example, if Slack devops team were to exclusively communicate over Slack, then a Slack outage would be much harder to resolve because the team trying to fix it would be unable to communicate.

Re: Thanksgiving 2023 security incident

#39

Post by jrockway »

tomschlick wrote: post #36
jedahan wrote: post #30

Earlier quoted context omitted.

The writeup contains indicators, including IP addresses, and the location of those addresses. In this case, the IP address associated with the threat actor is currently located in Bucharest, Romania.

No nation state is going to use IPs from their own country if they don't want to be caught. They will use multiple layers of rented VPS's with fake identities to pay for those resources.

Yeah. I've dealt with definitely-not-nation-states before, and their pattern was to sign up for free/cheap CI services (CircleCI, Github Actions, that sort of thing) and launch their attacks from there. The VPS thing also sounds very very plausible to me, I figured there was a long tail, but until I was looking up every network that was attacking us, I really had no idea how deep the long tail goes. I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there.

Re: Thanksgiving 2023 security incident

#40

Post by barkingcat »

sevg wrote: post #5

> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian produ…

I think they did have to do that far though.

Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there.

Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.

Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.

Post reply on HN