CircleCI incident report for January 4, 2023 security incident • Hacker News Board

Re: CircleCI incident report for January 4, 2023 security incident

#11

Post by mac-chaffee » Sat, Jan 14, 2023, 4:50 AM UTC

mtlynch wrote: post #6

This is a good writeup, and I appreciate the transparency. I especially like this bit: > While one employee’s laptop was exploited through this sophisticated attack, a security incident is a systems failure. Our responsibility as an organization is to build layers of safeguards that protect against all attack vectors. I was surprised by this part: > To date, we have learned that an unauthorized third party leveraged…

Is tying session tokens to IPs actually common? I can't imagine it is given the unreliability of IP addresses causing issues.

I used to live somewhere where outbound traffic went through one of three CGNAT IPs at random, and I only had auth issues with one really old site that predates the NAT hell that is the modern internet.

Re: CircleCI incident report for January 4, 2023 security incident

#12

Post by bamboozled » Sat, Jan 14, 2023, 5:52 AM UTC

On December 29, 2022, we were alerted to suspicious GitHub OAuth activity by one of our customers. This notification kicked off a deeper review by CircleCI’s security team with GitHub.

What kind of freaks me out about this is that a customer notified Circle? If that customer hadn't of mentioned anything, where would we be now?

I have to say, it's a pretty impressive hack. I wonder who or what was behind it?

Also wondering why / how the attacker didn't get access to the runners?

Re: CircleCI incident report for January 4, 2023 security incident

#13

Post by thih9 » Sat, Jan 14, 2023, 7:19 AM UTC

I wonder if this is related to the layoffs that circleci announced on dec 7th [1].

E.g.: did a new employee get access to production systems? Were there not enough people to monitor the systems and detect the breach sooner? Etc.

[1]: https://news.ycombinator.com/item?id=33900488

Re: CircleCI incident report for January 4, 2023 security incident

#14

Post by justinludwig » Sat, Jan 14, 2023, 7:47 AM UTC

Nice writeup. Allowing customer data and secrets to be exfiltrated is a pretty big fail, and will probably make a number of customers re-think their patronage at a time when supply-chain security is top-of-mind to many.

But three things mentioned in their report do give me some confidence about the way CircleCI has engineered their internal systems:

1. They use SSO with 2FA ("an unauthorized third party leveraged malware deployed to a CircleCI engineer's laptop in order to steal a valid, 2FA-backed SSO session")

2. They maintain reasonably good audit logging (they could identify that "the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data" which had been exfiltrated)

3. They can rebuild everything from scratch ("we rotated all potentially exposed production hosts to ensure clean production machines")

A lot of companies pay lip service to best practices like these, but don't actually implement them thoroughly (or at all). The fact that CircleCI could rely on them under attack makes me think they're doing a better job than 90% of the SaaS companies out there.

Re: CircleCI incident report for January 4, 2023 security incident

#15

Post by SCdF » Sat, Jan 14, 2023, 8:18 AM UTC

> On January 4, 2023, at 6:30 PM PST / January 5, 2023, at 02:30 UTC, we sent disclosure emails, posted a...

Did they? I got an email this morning that pointed to _this_ blog post, but I never received any initial "rotate yo keys" communication from them, on any email address.

If I hadn't read HN, and none of my company had, and our use of CI was running smoothly (they eventually put up a banner in the UI), I would literally have never known until this email.

Re: CircleCI incident report for January 4, 2023 security incident

#16

Post by Dunedan » Sat, Jan 14, 2023, 9:35 AM UTC

> On January 4, 2023, at 18:30 UTC, we shut down production access to nearly all employees, limiting access to an extremely small group for operational issues.

Shouldn't that have been the case from the beginning? Why did more than a small group of employees have production access at all?

Re: CircleCI incident report for January 4, 2023 security incident

#17

Post by numbsafari » Sat, Jan 14, 2023, 1:44 PM UTC

mac-chaffee wrote: post #11

mtlynch wrote: post #6

This is a good writeup, and I appreciate the transparency. I especially like this bit: > While one employee’s laptop was exploited through this sophisticated attack, a security incident is a systems failure. Our responsibility as an organization is to build layers of safeguards that protect against all attack vectors. I was surprised by this part: > To date, we have learned that an unauthorized third party leveraged…

Is tying session tokens to IPs actually common? I can't imagine it is given the unreliability of IP addresses causing issues. I used to live somewhere where outbound traffic went through one of three CGNAT IPs at random, and I only had auth issues with one really old site that predates the NAT hell that is the modern internet.

session vs refresh… you kill the session token and require a refresh. Can be sometimes be transparent, but may cause a re-authentication using the second factor with an indicator to the user that their previous session was killed due to use by a different IP.

If you are concerned about stable IPs, use a proper VPN or bastion setup.

Re: CircleCI incident report for January 4, 2023 security incident

#18

Post by numbsafari » Sat, Jan 14, 2023, 1:45 PM UTC

mtlynch wrote: post #8

randerson wrote: post #7

Earlier quoted context omitted.

CircleCI is 100% remote. I can't imagine having to keep up with everyone's constantly changing home IPs and making exceptions while travelling.

I'd imagine they have a VPN.

I’d imagine they do now… at least for privileged account usage.

Re: CircleCI incident report for January 4, 2023 security incident

#19

Post by mtlynch » Sat, Jan 14, 2023, 2:51 PM UTC

mac-chaffee wrote: post #11

mtlynch wrote: post #6

This is a good writeup, and I appreciate the transparency. I especially like this bit: > While one employee’s laptop was exploited through this sophisticated attack, a security incident is a systems failure. Our responsibility as an organization is to build layers of safeguards that protect against all attack vectors. I was surprised by this part: > To date, we have learned that an unauthorized third party leveraged…

Is tying session tokens to IPs actually common? I can't imagine it is given the unreliability of IP addresses causing issues. I used to live somewhere where outbound traffic went through one of three CGNAT IPs at random, and I only had auth issues with one really old site that predates the NAT hell that is the modern internet.

Yeah, good point. I guess it'd be a pain to have to keep reauth'ing if your IP changed for legitimate reasons.

It would be possible to do some kind of check for "this session token was used in the US and Russia twenty minutes apart... something's fishy," but that adds in more complexity.