Live data from Hacker News

Thanksgiving 2023 security incident

blog.cloudflare.com

111–120 of 336 posts

Re: Thanksgiving 2023 security incident

#111
post #103

> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold. For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have t…

Not if such citizens are sanctioned. Code Red. Hint hint.

Re: Thanksgiving 2023 security incident

#112
>They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023. All threat actor access and connections were terminated on November 24 and CrowdStrike has confirmed that the last evidence of threat activity was on November 24 at 10:44.

Okta hitting everywhere

Re: Thanksgiving 2023 security incident

#113
post #5

> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian produ…

> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway. Aha, the old replace-your-trusted-hardware trick.

In a corporate environment, standard procedure when an employee's computer gets infected is to re-image it. Even if it was a stupid virus that was immediately caught, the potential risk of undetected malware running amuck is just too high.

Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.

Re: Thanksgiving 2023 security incident

#114

Earlier quoted context omitted.

I think they did have to do that far though. Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room ( https://en.wikipedia.org/wiki/Meet-me_room ) and having persistent access to key switches there. Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable…

> It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up. It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is. Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.

> Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.

The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever.

The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.

Re: Thanksgiving 2023 security incident

#115

Earlier quoted context omitted.

No nation state is going to use IPs from their own country if they don't want to be caught. They will use multiple layers of rented VPS's with fake identities to pay for those resources.

Yeah. I've dealt with definitely-not-nation-states before, and their pattern was to sign up for free/cheap CI services (CircleCI, Github Actions, that sort of thing) and launch their attacks from there. The VPS thing also sounds very very plausible to me, I figured there was a long tail, but until I was looking up every network that was attacking us, I really had no idea how deep the long tail goes. I now feel like h…

> I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there.

Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion?

Reason I ask is long-hosted small business websites are sometimes established with the intent to legitimize some future unrelated traffic.

Re: Thanksgiving 2023 security incident

#116

Earlier quoted context omitted.

A Github account, for one possible example.

This is why I use a separate Github account for work? (& then just rotate the credentials on it when you part ways with the employer.) Some of my co-workers even do a Github account per employment.

That’s me—a GitHub account per employer with employee email.

Re: Thanksgiving 2023 security incident

#118
post #34

> The one service token and three accounts were not rotated because mistakenly it was believed they were unused. Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?

blameless post mortem most likely

Great call out too

> Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.

Post reply on HN