Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thanksgiving 2023 security incident

blog.cloudflare.com

Post reply on HN
Print view View original

251–260 of 336 posts

Re: Thanksgiving 2023 security incident

#251

Post by dylan604 »

dalyons wrote: post #249
dylan604 wrote: post #238

Earlier quoted context omitted.

> All those guys wearing headphones in the office have brought in CDs to play in their laptop CD drives? I've worked for large media companies where this is exactly the only way to have music available. The production network was blocked from accessing the www. To ensure content wasn't pirated, the original media had to be used. No CD-Rs were allowed. Personal devices were kept in lockers outside the restricted areas…

In the last 10 years? Outside of govt? That sounds horrifically inefficient for 2024

This is the default knee jerk reaction, but I didn't have an issue with it. I'm not addicted to my device, so leaving it in a locker was perfectly fine with me. It was actually kind of refreshing to not approach a co-worker doom scrolling a social platform.

Re: Thanksgiving 2023 security incident

#252

Post by Bluecobra »

_cenw wrote: post #160
wepple wrote: post #156

They mention Zero Trust, yet you can gain access to applications with just a single bearer token? Am I missing something here? There’s no machine cert used? AuthN tokens aren’t cryptographically bound? This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”

these were service accounts used by third parties to provide jira integrations, not a user account

If they are using Active Directory, wouldn’t a service account be no different than a regular employee account? Both a Jira service account and the CEO of Cloudflare are still Domain Users in AD. Granted, a service account should be way more locked down and have the least amount of access possible.

Re: Thanksgiving 2023 security incident

#253

Post by cheeze »

BeefWellington wrote: post #246
bigbluedots wrote: post #222

Earlier quoted context omitted.

> They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate , after the Okta compromise of October 2023 It's fair to "punch down" imo as that's how the credentials were originally compromised. I'd agree with you if CF were trying to minimize their own mistake but that doesn't seem to be what is happening here

If a breach is disclosed and some time later your systems are compromised because you didn't bother to take appropriate action in response to that, it's not "fair" to punch down, or even reasonable to do so.

Okta was painfully negligent, with CF going as far as posting "recommendations for Okta" because it was their only way to get through to them.

I don't love CF, but IMO Okta deserves to be punched down on.

Re: Thanksgiving 2023 security incident

#254

Post by [deleted] »

eastdakota wrote: post #177
nimbius wrote: post #157

Earlier quoted context omitted.

Then, the advertisement worked. - Insist that you have better integrity than your competitors - share a few operational investigations after your latest security event what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just expl…

I was the one who made the call to bring in CrowdStrike. It had zero to do with PCI/DSS or any other compliance obligation. It was to 1) bring in a team with deep experience with a broad set of breaches; and 2) to make sure our team didn’t miss anything. The CrowdStrike team were first class and it was good to confirm they didn’t find anything significant our team hadn’t already. And, for the sake of clarity, no syst…

[deleted]

Re: Thanksgiving 2023 security incident

#255

Post by tptacek »

nimbius wrote: post #157
BytesAndGears wrote: post #13

Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business. Yes, they aren’t perfect. They do some things that I disagree with. But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this. Thank you for the blog post!

Then, the advertisement worked. - Insist that you have better integrity than your competitors - share a few operational investigations after your latest security event what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just expl…

None of this would have had anything to do with PCI (nobody gives a shit about PCI; the worst shops in the world, the proprietors of the largest breaches, have had no trouble getting PCI certified and keeping certification after their breaches). At much smaller company sizes than this, insurance requires you to retain forensics/incident response firms. There's a variety of ways they could do that cheaply. They brought in the IR firm with the best reputation in the industry (now that Google owns Mandiant), at what I'm assuming are nosebleed rates, because they want to be perceived as taking this as seriously as they seem to be.

It's a very good writeup, as these things go. Cloudflare is huge. An ancillary system of theirs got popped as sequelae to the Okta breach. They reimaged every machine in their fleet and burned all their secrets. People are going to find ways to snipe at them, because that's fun to do, but none of those people are likely to handle an incident like this better.

I am not a Cloudflare customer (technically, I am a competitor). But my estimation of them went up (I won't say by how much) after reading this writeup.

Re: Thanksgiving 2023 security incident

#256

Post by tptacek »

elitistphoenix wrote: post #232
eastdakota wrote: post #177

Earlier quoted context omitted.

I was the one who made the call to bring in CrowdStrike. It had zero to do with PCI/DSS or any other compliance obligation. It was to 1) bring in a team with deep experience with a broad set of breaches; and 2) to make sure our team didn’t miss anything. The CrowdStrike team were first class and it was good to confirm they didn’t find anything significant our team hadn’t already. And, for the sake of clarity, no syst…

Was the self hosted environment running a AV like the Crowdstrike agent? Or was it running different AV and that's why you chose to use Crowdstrike as someone different? I guess no need to specific names. I'm just using that as examples.

What's an AV going to do about the fact that Okta got popped?

Re: Thanksgiving 2023 security incident

#257

Post by BeefWellington »

cheeze wrote: post #253
BeefWellington wrote: post #246

Earlier quoted context omitted.

If a breach is disclosed and some time later your systems are compromised because you didn't bother to take appropriate action in response to that, it's not "fair" to punch down, or even reasonable to do so.

Okta was painfully negligent, with CF going as far as posting "recommendations for Okta" because it was their only way to get through to them. I don't love CF, but IMO Okta deserves to be punched down on.

Sure but for how long and in what contexts?

Is it really reasonable to come out and say your company utterly failed a pretty basic security practice when faced with a compromise but that it was really some other company's problem originally?

Of course it's not. It's still your company's failure. Own it.

Re: Thanksgiving 2023 security incident

#258

Post by sophacles »

twisteriffic wrote: post #175
marcinzm wrote: post #6

> we were (for the second time) the victim of a compromise of Okta’s systems I'm curious if they're rethinking being on Okta.

This wasn't really an additional failure at Okta. This was credentials lost during the original Okta compromise that CloudFlare failed to rotate out. Okta deserves criticism for their failure, but this feels like CloudFlare punching down to shift blame for a miss on their part.

How is Cloudflare ($384M in revenue, q3 2023) punching down at Okta ($584M in revenue, q3 2023) by stating exactly what happened.

If anything Okta is a bigger company (by revenue, by employee count) and they were founded a year earlier.

Re: Thanksgiving 2023 security incident

#259

Post by neilv »

kccqzy wrote: post #103

> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold. For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have t…

> Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.

They had government engagements with Google's consent, and all those various engagements could be disclosed to each other?

If not, what kind of drugs were flowing at this social gathering, to cause such an orgy of bad OPSEC?

Re: Thanksgiving 2023 security incident

#260

Post by mardifoufs »

tptacek wrote: post #255
nimbius wrote: post #157

Earlier quoted context omitted.

Then, the advertisement worked. - Insist that you have better integrity than your competitors - share a few operational investigations after your latest security event what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just expl…

None of this would have had anything to do with PCI (nobody gives a shit about PCI; the worst shops in the world, the proprietors of the largest breaches, have had no trouble getting PCI certified and keeping certification after their breaches). At much smaller company sizes than this, insurance requires you to retain forensics/incident response firms. There's a variety of ways they could do that cheaply. They brough…

Yeah at best PCI is somewhat hard to get at first, but after that it's basically only good, or less shady, corporations that bother keeping up compliance or make sure that they follow the guidelines at every step. Shady/troubled operators don't, and to an extent don't have to really be afraid of losing said certification unless they just go fully rogue.
Post reply on HN