It's almost valentine
Thanksgiving 2023 security incident
181–190 of 336 posts
Re: Thanksgiving 2023 security incident
#182Thing about a data breach is once the data is out there - source code in this case - it’s out there for good and you have absolutely no control over who gets it. You can do as much post incident hardening as you want, and talk about it as much as you want, but the thing you’re trying to protect against, and blogging about how good you’re getting at preventing, has already happened. Can’t unscramble those eggs.
The source code next year is not the same as source code this year.
The customer data next year is not the same as the customer data this year.
Re: Thanksgiving 2023 security incident
#183Re: Thanksgiving 2023 security incident
#184Thing about a data breach is once the data is out there - source code in this case - it’s out there for good and you have absolutely no control over who gets it. You can do as much post incident hardening as you want, and talk about it as much as you want, but the thing you’re trying to protect against, and blogging about how good you’re getting at preventing, has already happened. Can’t unscramble those eggs.
Whats your point?
That this is messaged and received as a net win. It’s not.
Re: Thanksgiving 2023 security incident
#185Earlier quoted context omitted.
Doesn’t matter. No personal stuff on company devices. I just don’t understand any rational otherwise.
I have personal stuff on my work machine. I don’t need to say any more than that because in your eyes it’s inherently unjustifiable. So, would you care to more explicitly tell me what you think about my intelligence or ability to behave rationally compared to you? Or is there potentially some room for nuance here?
or if you take it another way... i currently don't understand any rational that has been presented that allows me that frame of thought, but i'm always wanting to learn more.
Re: Thanksgiving 2023 security incident
#186Earlier quoted context omitted.
> Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed. The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care . They only care about their golf course or backroom deal friends and about releasing PR s…
Yes, that’s literally the point being made. The point is that it isn’t a company-ending event. You are going on an unrelated rant about how those darn dumb executives aren’t as smart as God’s gift to earth, engineers.
The thing is, some events should be company ending. Something like Okta shouldn't even exist in a halfway competent world in the first place - given how many Fortune 500 companies, even governments use it, it's just a too fucking juicy target for nation states both friendly and hostile.
Instead, even the "self correcting" mechanisms of the "free market" obviously didn't work out, as the free market doesn't value technical merit, it only values financial bullshittery.
And the end result will be that once the war with China or Russia inevitably breaks out, virtually all major Western companies and governments will be out cold for weeks once Okta and Azure's AD go down, because that is where any adversary will hit first to deal immense damage.
Re: Thanksgiving 2023 security incident
#187Earlier quoted context omitted.
> I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there. Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion? Reason I ask is long-hosted small business websites are sometimes established with the intent to legitimize so…
> Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion? Probably not what's happening. I've tried to build a cloud CI service a while ago. Per their nature, you _have to_ allow arbitrary commands to be run. And you also have to allow outbound connectivity. So you don't need to 'own' anything in order to b…
That's exactly right. CI providers are good anonymizers for unsophisticated attackers because they provide an extra layer of obfuscation. But if they were doing something significantly harmful, I'd obviously be talking to those providers and asking for their own logs as part of the investigation, and then it would clearly link back to the actual culprits. So that was one popular technique to use to circumvent IP bans after abusing our service.
The whole hosting provider thing was another type of problems. I would always look at who owned the IPs that malicious sign-ups were coming from, and found a lot of ASNs owned by companies like "hosturwebsite4u.or.uk" and things like that. Those I assumed were just forgotten-about Linux boxes that the attackers used to anonymize through.
Ultimately, this was all to get a "free trial" of our cloud service, which did let you run arbitrary code. We eventually had a fairly large number of ASNs that would get a message like "contact sales for a free trial" instead of just auto-approving. That was the end of this particular brand of scammers. (They did contact sales, though! Sales was not convinced they were a legitimate customer, so didn't give them a free trial. Very fun times ;)
I should really write up the whole experience. I learned so much about crypto mining and 2020-era script-kiddie-ing in a very short period of time. My two favorite tangents were 1) I eventually wrote some automation to kill free trials that were using 100% CPU for more than 12 hours or something like that, and so they just made their miner run at 87% CPU. 2) They tried to LD_PRELOAD some code that prevented their process from showing up in the process table, but didn't realize that our tools were statically linked and that they were running in an unprivileged container, so the technique doubly didn't work. But, good old `ps` and `top` are linked against glibc, so they probably fooled a lot of people this way. They also left their code for the libc stub around, and I enjoyed reading it.
Re: Thanksgiving 2023 security incident
#188> we were (for the second time) the victim of a compromise of Okta’s systems I'm curious if they're rethinking being on Okta.
[deleted]
Re: Thanksgiving 2023 security incident
#189Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business. Yes, they aren’t perfect. They do some things that I disagree with. But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this. Thank you for the blog post!
Then, the advertisement worked. - Insist that you have better integrity than your competitors - share a few operational investigations after your latest security event what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just expl…
> Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just explain remediation without accountability.
They did, and they admitted that it was their fault. I have to give them credit for that much.
> They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023...The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This was incorrect and was how the threat actor first got into our systems and gained persistence to our Atlassian products. Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.
Re: Thanksgiving 2023 security incident
#190> we were (for the second time) the victim of a compromise of Okta’s systems I'm curious if they're rethinking being on Okta.
This wasn't really an additional failure at Okta. This was credentials lost during the original Okta compromise that CloudFlare failed to rotate out. Okta deserves criticism for their failure, but this feels like CloudFlare punching down to shift blame for a miss on their part.
Agreed, but Okta's still a $14B company.