61–70 of 336 posts
> The one service token and three accounts were not rotated because mistakenly it was believed they were unused. Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?
Which "nation state" do we think this was?
The writeup contains indicators, including IP addresses, and the location of those addresses. In this case, the IP address associated with the threat actor is currently located in Bucharest, Romania.
Earlier quoted context omitted.
I think they did have to do that far though. Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room ( https://en.wikipedia.org/wiki/Meet-me_room ) and having persistent access to key switches there. Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable…
do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?
Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business. Yes, they aren’t perfect. They do some things that I disagree with. But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this. Thank you for the blog post!
Which "nation state" do we think this was?
> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian produ…
This is why old secops/corpsec security hands are so religious about tabletop exercises, and what's so great about BadThingsDaily† on Twitter. Being prepared to do this kind of credential rotation takes discipline and preparation and, to be frank, most teams don't make that investment, including a lot of really smart, well-resourced ones. If Cloudflare is in a position where their security team can make a call to rot…
It's good that you think you can absorb a complicated security task, it's useless if you have no way to test or verify this action.
Earlier quoted context omitted.
do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?
We have very good relations with our network vendors (in this case, Cisco, Juniper, and Arista). The CEOs of all of them 1) immediately got on a call with me late on a weekend; 2) happily RMAed the boxes at no cost; and 3) lent us their most senior forensics engineers to help with our investigation. Hat tip to all of them for first class customer service.
thanks Matthew! love the transparency and dedication to security as always. really sucks to have this be continuing fallout from Okta's breach. wish large scale key rotation was more easily automatable (or at least as a fallback, there should be a way to track key age on clientside? so that old keys stick out like a sore thumb). i guess in the absence of industry standard key rotation apis someday you might be able to "throw AI at it".
Great write up. > Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories > They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages). Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though. > Since the Smartsheet service account had administra…
12k git repos can happen if the team uses github enterprise with forking internally. It can also happen in franken-build systems which encourage decoupling by making separate repos: one repo that defines a service’s API, containing just proto (for example). A second repo that contains generated client code, a third with generated server code, a fourth for implementation, a fifth which supplies integration test harnes…
Earlier quoted context omitted.
Fair question, but I use a lot of things that are varying degrees of helpful for my work: * personal ChatGPT and copilot subscriptions, since company doesn’t pay for these * Trello account for keeping track of my todo list (following up with people, running deploys) * Obsidian for keeping notes, as a personal knowledge-base (things like technologies and reminders) * Apple account for music, copy/paste, sharing photos…
Why don't you just do those on a second, personal, laptop? Does your workplace restrict you from bringing it in?
It’s really easy to say ‘don’t use your personal stuff at work’, but when work is some locked-down behemoth whose view of productivity software is ‘just use Office’, and you’re really trying to be better at your job, using your own tools can be the only solution.
And in my situation, yeah, they didn’t want you bringing things in. I worked in a secure area.
> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian produ…
I think they did have to do that far though. Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room ( https://en.wikipedia.org/wiki/Meet-me_room ) and having persistent access to key switches there. Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable…
Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree.
(I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)