Live data from Hacker News

Thanksgiving 2023 security incident

blog.cloudflare.com

321–330 of 336 posts

Re: Thanksgiving 2023 security incident

#321

Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business. Yes, they aren’t perfect. They do some things that I disagree with. But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this. Thank you for the blog post!

You actually believe that an intruder gained access to their KB/Tickets and didn't manage to get valuable information? That's not what Jira is for. If you know what Jira is for, and you're willing to run it on-prem, then you know the purpose of doing all that work is because you have something valuable to store in there.

I don't believe they didn't lose anything. That's not how this works, and most Jira/Confluence I've seen is loaded with secrets.

Re: Thanksgiving 2023 security incident

#322

Earlier quoted context omitted.

Then why not leave if there's no prospects of change in the near future and if you really care?

Because I was working with friends, on a project that was interesting, mentally valuable, and in the national interest. Just because the org that hires you is a shambles doesn’t mean you give up and quit. Thank fuck we don’t all think like that. And, again, reputation. I have a stellar reputation because I stick it out, and I care. I’ve worked with people who quit because ‘it’s shit here’. Nobody will ever work with…

Then you clearly care a lot more about these other aspects over 'tools'...

So why raise the point in the first place if it's a minor quibble relative to your top priority(s)?

Re: Thanksgiving 2023 security incident

#323

Earlier quoted context omitted.

> we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat. The MO screams China to me but I wouldn’t read anythin…

The name has nothing to do with where we believe the attacker came from. We borrowed it from Google. At Google they have a procedure where, in an emergency, they can declare a Code Yellow or Code Red — depending on the severity. When it happens, it becomes the top engineering priority and whoever is leading it can pull any engineer off to work on the emergency. Those may not be the exact details of Google's system bu…

Ha, I almost mentioned Google, having been through a code red myself, but it felt like a reach so I went elsewise with my comment! I believe it’s safe to assume google pulled it from emergency response given that their approach to incident management inherits heavily from ICS.

Re: Thanksgiving 2023 security incident

#324
post #307
post #156

They mention Zero Trust, yet you can gain access to applications with just a single bearer token? Am I missing something here? There’s no machine cert used? AuthN tokens aren’t cryptographically bound? This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”

You're not. The article makes no sense. They claim robust security controls but apparely lacked a proper accounting of service accounts with external access, especially with admin access to freakin' Jira.

Thank you! The more I think about his it makes no sense to me. If a service account needs external access why can't they also whitelist connectivity to specific public IP addresses?

Re: Thanksgiving 2023 security incident

#325
post #303

Earlier quoted context omitted.

Could you explain? Never been exposed to fraternities or Google.

Ritualistic hazing of newcomers is a large part of american "frat" culture. What the parent is likely referring to is: Things like "Wearing the noogler hat" or the hoops you jump through in interview (that have nothing to do with the job) are similar in spirit to some university fraternities admission processes, or are depicted as such in media.

Yes, also the latent class filtering, and the general "coked-up fratbro" behaviors, like enthusiastically and energetically pursuing ideas that make no sense. (No sense includes hiring for people who obsessively focus their time on memorizing and rehearsing for the rituals to get into that one frat.)

Re: Thanksgiving 2023 security incident

#326

Earlier quoted context omitted.

> we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat. The MO screams China to me but I wouldn’t read anythin…

The name has nothing to do with where we believe the attacker came from. We borrowed it from Google. At Google they have a procedure where, in an emergency, they can declare a Code Yellow or Code Red — depending on the severity. When it happens, it becomes the top engineering priority and whoever is leading it can pull any engineer off to work on the emergency. Those may not be the exact details of Google's system bu…

Did you order the code red?! I want the TRUTH.

(Sorry, so sorry, it was just too low-hanging not to to pluck: https://m.youtube.com/watch?v=W2G2sac9s34)

Re: Thanksgiving 2023 security incident

#327
I am trying to learn and understand the attack. Can someone please help me brainstorm some possibilities? (please note, I do not intend to question Cloudflare's business practices or security program; my intention is to learn and understand). --

Blog: The threat actor (TA) accessed Okta’s customer support system and viewed files uploaded by Cloudflare (CF) as support cases.

Why was the session token part of the support files uploaded to Okta support? Does Okta require it for troubleshooting?

TA hijacked a session token of a CF employee from a support ticket.

Blog: Using the token extracted from Okta, the TA accessed Cloudflare’s Okta and compromised two separate Cloudflare employee accounts within the Okta platform.

How did this happen? Was the stolen token privileged? Also, why only 2 employee accounts? Were these different employees, or was one the same one whose token was compromised? What does Okta employee account compromise mean - did TA reset the password and MFA, and how or was there no MFA?

Blog: TA used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code.

Did the stolen employee credentials only have access to Atlassian?

Blog: TA gained access to a set of credentials

Does this mean multiple credentials got uploaded to the Okta support system?

Blog: The Okta compromise was in October, but the threat actor only began targeting CF systems using stolen credentials from the Okta compromise in mid-November.

Does this mean the compromised token was long-lived?

Blog: We failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.

I didn’t get this. Does this mean that over time, CF employees had uploaded support info for 1000s of apps managed by Okta? Which credentials did CF rotate initially after the Okta compromise?

Leaked Credentials: 1. Moveworks service token that granted remote access into our Atlassian system.

Is this service token a bearer token? And without expiry. Is this like an API key?

TA accessed Atlassian Jira and Confluence using the Moveworks service token to authenticate through the gateway.

2. A service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance,

So here, the Smartsheet Saas was given access to the on-prem Atlassian Jira instance? What kind of trust is it? Is this as well managed through Okta? And how does support case filing include a service account? Here, does the Service account mean again some kind of hardcoded API key without expiry

TA used the Smartsheet service account to gain access to the Atlassian suite. They used Smartsheet credentials to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment.

Since the Smartsheet service account had administrative access to Atlassian Jira, the TA was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for the Jira plugin. This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access, the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they could not access any global networks.

3. A Bitbucket service account, which was used to access our source code management system

4. AWS environment that had no access to the global network and no customer or sensitive data.

Were these AWS access keys? Also, it looks like these keys did provide access to the AWS account. That means the access key didn’t require MFA.

The only production system the TA could access using the stolen credentials was our Atlassian environment.

Mitigations:

Blog: We decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files.

What does hardening security protocol mean here? Is it techniques for D&R or something else

Blog: We undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials)

I believe this means forced resets on employees (Okta users), right?

Re: Thanksgiving 2023 security incident

#328

Earlier quoted context omitted.

Because I was working with friends, on a project that was interesting, mentally valuable, and in the national interest. Just because the org that hires you is a shambles doesn’t mean you give up and quit. Thank fuck we don’t all think like that. And, again, reputation. I have a stellar reputation because I stick it out, and I care. I’ve worked with people who quit because ‘it’s shit here’. Nobody will ever work with…

Then you clearly care a lot more about these other aspects over 'tools'... So why raise the point in the first place if it's a minor quibble relative to your top priority(s)?

That’s not what I said.

Tools enable success. Better tools make the job easier. They make the result better.

But having bad tools isn’t a reason to give up. It’s frustrating. But you just have to get on with it.

Re: Thanksgiving 2023 security incident

#329

Earlier quoted context omitted.

Then you clearly care a lot more about these other aspects over 'tools'... So why raise the point in the first place if it's a minor quibble relative to your top priority(s)?

That’s not what I said. Tools enable success. Better tools make the job easier. They make the result better. But having bad tools isn’t a reason to give up. It’s frustrating. But you just have to get on with it.

That is what you said… tools don’t matter enough to make you quit the job, and considering the language used in the previous comment, it seems far enough from your top priority that it likely never will. Hence a minor quibble in relative terms as it has almost no impact on your final decision to stay or leave.
Post reply on HN