Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thanksgiving 2023 security incident

blog.cloudflare.com

Post reply on HN
Print view View original

281–290 of 336 posts

Re: Thanksgiving 2023 security incident

#281

Post by fierro »

pphysch wrote: post #198
pbhjpbhj wrote: post #104

Earlier quoted context omitted.

This smells weird, surely? I'd be looking at who chose not to rotate those particular credentials. 1: "what are these accounts?" 2: "oh they're unused, they don't even appear in the logs" 1: "we should rotate them" 2: "no, let's keep those rando accounts with the old credentials, the ones we think might be compromised ... y' know, for reasons" ?

More likely: "no one has any idea what these old credentials do, so let's not touch them and potentially break everything"

this is more plausible to me

Re: Thanksgiving 2023 security incident

#282

Post by slowbdotro »

neilv wrote: post #259
kccqzy wrote: post #103

> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold. For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have t…

> Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus. They had government engagements with Google's consent , and all those various engagements could be disclosed to each other ? If not, what kind of drugs were flowing at this social gathering, to cause such an orgy of bad OPSEC?

Knowing google employees, it's coke. Lots of coke

Re: Thanksgiving 2023 security incident

#283

Post by hn_go_brrrrr »

eastdakota wrote: post #273
eep_social wrote: post #202

Earlier quoted context omitted.

> we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat. The MO screams China to me but I wouldn’t read anythin…

The name has nothing to do with where we believe the attacker came from. We borrowed it from Google. At Google they have a procedure where, in an emergency, they can declare a Code Yellow or Code Red — depending on the severity. When it happens, it becomes the top engineering priority and whoever is leading it can pull any engineer off to work on the emergency. Those may not be the exact details of Google's system bu…

Yeah, that's a pretty accurate description of the color code system. There's some additional nuance to it, but a code red is an immediate existential threat to the business.

Re: Thanksgiving 2023 security incident

#284

Post by _kb »

kccqzy wrote: post #103

> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold. For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have t…

Payroll? You guys are getting paid?

Australians get the 'opportunity' to be part of that sort of that sort of espionage as a base level condition of citizenship [0].

As an upside, I guess it helps with encouraging good practices around zero trust processes and systems dev.

[0]: https://en.wikipedia.org/wiki/Mass_surveillance_in_Australia...

Re: Thanksgiving 2023 security incident

#285

Post by nolok »

eastdakota wrote: post #247
chx wrote: post #94

Earlier quoted context omitted.

Presuming taviso is not exaggerating and why would he CF's reply to cloudbleed was ... not quite nice. https://twitter.com/taviso/status/1566077115992133634 > True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF? For those not familiar with the…

This came up before and it was super confusing to me because I had no idea what it was referring to but I also believe Tavis isn’t one to make something up. So I took some time to investigate. Turned out, no one on our management, legal, communications, or public policy team had any idea what he was talking about. Eventually I figured out that a non-executive former member of our engineering team was dating someone w…

> we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up

This is not what happened at all. What happened is that after the initial discovery, the gzero team realized it was much worse than expected AND the cloudflare team who he synced with for the disclosure started ghosting him, and yet gzero still kept to the full timeline.

If you working there and having done research can get it this wrong while it's super easy to find the event log in the open, it doesn't give a very good vibe about the attitude inside cloudflare regarding what happened and fair disclosure.

Full even log on project zero is here : https://bugs.chromium.org/p/project-zero/issues/detail?id=11...

> The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

Meanwhile link with Cloudflare went from this

> I had a call with Cloudflare, they reassured me they're planning on complete transparency and believe they can have a customer notification ready this week.

> I'm satisfied cloudflare are committed to doing the right thing, they've explained their current plan for disclosure and their rationale.

To this

> Update from Cloudflare, they're confident they can get their notification ready by EOD Tuesday (Today) or early Wednesday.

> Cloudflare told me that they couldn't make Tuesday due to more data they found that needs to be purged.

> They then told me Wednesday, but in a later reply started saying Thursday.

> I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this. If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks. https://security.googleblog.com/2013/05/disclosure-timeline-...

> If an acceptable notification is not released on Thursday, we'll decide how we want to proceed.

> I had a call with cloudflare, and explained that I was baffled why they were not sharing their notification with me.

> They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.

> Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers. They've left it too late to negotiate on the content of the notification.

So it was not project zero but cloudflare that moved the disclosure timeline around, and did so without keeping pzero in the loop, about an active in the wild exploit.

Re: Thanksgiving 2023 security incident

#286

Post by neilv »

slowbdotro wrote: post #282
neilv wrote: post #259

Earlier quoted context omitted.

> Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus. They had government engagements with Google's consent , and all those various engagements could be disclosed to each other ? If not, what kind of drugs were flowing at this social gathering, to cause such an orgy of bad OPSEC?

Knowing google employees, it's coke. Lots of coke

At last, an explanation for their fratbro interviews.

Re: Thanksgiving 2023 security incident

#287

Post by is39 »

meowface wrote: post #143
ownagefool wrote: post #140

Earlier quoted context omitted.

Cloudflare is essentially a massive mitm proxy. If you manage to pwn a key, you have access to traffic. I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext. At the very least you need to tell the server what to trust.

>If you manage to pwn a key, you have access to traffic. That's why I mentioned "Full (Strict)" SSL. If you configure this in Cloudflare then the entire user Cloudflare origin path is encrypted and attackers can't snoop on the plaintext even if they have access. They'll get some metadata, but every ISP in the world gets that at all times anyway.

While both client and origin network connections are encrypted with "Full (Strict)" SSL mode, Cloudflare proxy in the middle decrypts client traffic and then encrypts it towards the server (and vice versa). It does have access to plaintext, which is how various mitigations work. So it's indeed MITM proxy, by design.

Re: Thanksgiving 2023 security incident

#288

Post by yardstick »

mardifoufs wrote: post #280
tptacek wrote: post #262

Earlier quoted context omitted.

It's not hard to get at first, either. It's the archetypical checklist audit.

Ah I think I'm just not used to those then, I hated the whole checklist busywork that we had to do even though we were barely related to the sales infra. But yeah, it was a bit like soc2 in that regard. Is there any certification that isn't just checklist "auditing"? That involves actual monitoring or something? Not sure if that's even possible

PCI DSS does require periodical review of lots of elements, and I believe daily log reviews (which let’s face it no one does outside of very big firms with dedicated security teams and fancy SIEM tools).

Re: Thanksgiving 2023 security incident

#289

Post by yardstick »

tptacek wrote: post #255
nimbius wrote: post #157

Earlier quoted context omitted.

Then, the advertisement worked. - Insist that you have better integrity than your competitors - share a few operational investigations after your latest security event what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just expl…

None of this would have had anything to do with PCI (nobody gives a shit about PCI; the worst shops in the world, the proprietors of the largest breaches, have had no trouble getting PCI certified and keeping certification after their breaches). At much smaller company sizes than this, insurance requires you to retain forensics/incident response firms. There's a variety of ways they could do that cheaply. They brough…

> nobody gives a shit about PCI; the worst shops in the world, the proprietors of the largest breaches, have had no trouble getting PCI certified and keeping certification after their breaches

IMO it’s more a risk reward trade off. I know some companies are paying relative peanuts in non-compliance fines rather than spend money on some semblance of security which they may still not be compliant with and have to pay the fines anyway…

Re: Thanksgiving 2023 security incident

#290

Post by chx »

nolok wrote: post #285
eastdakota wrote: post #247

Earlier quoted context omitted.

This came up before and it was super confusing to me because I had no idea what it was referring to but I also believe Tavis isn’t one to make something up. So I took some time to investigate. Turned out, no one on our management, legal, communications, or public policy team had any idea what he was talking about. Eventually I figured out that a non-executive former member of our engineering team was dating someone w…

> we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up This is not what happened at all. What happened is that after the initial discovery, the gzero team realized it was much worse than expected AND the cloudflare team who he synced with for the disclosure started ghosting him, and yet gzero still kept to the full timeline. I…

> If you working there

For context: you are answering to the co-founder & CEO of Cloudflare.

Post reply on HN