CircleCI incident report for January 4, 2023 security incident
1–10 of 19 posts
Re: CircleCI incident report for January 4, 2023 security incident
#2[deleted]
Re: CircleCI incident report for January 4, 2023 security incident
#3Buried in the middle of the post is this most-important tidbit:
> Though all the data exfiltrated was encrypted at rest, the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data.
Re: CircleCI incident report for January 4, 2023 security incident
#4Buried in the middle of the post is this most-important tidbit: > Though all the data exfiltrated was encrypted at rest, the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data.
Unfortunately this was already a given since there were reports of users secrets (canary tokens) being used. They got the secrets for sure. It just wasn't clear how many they got. It doesn't matter though, you have to assume if you use CircleCI then your secrets were stolen. If you haven't rotated them then likely the only reason you haven't been compromised yet is out of luck.
Re: CircleCI incident report for January 4, 2023 security incident
#5Buried in the middle of the post is this most-important tidbit: > Though all the data exfiltrated was encrypted at rest, the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data.
Unfortunately this was already a given since there were reports of users secrets (canary tokens) being used. They got the secrets for sure. It just wasn't clear how many they got. It doesn't matter though, you have to assume if you use CircleCI then your secrets were stolen. If you haven't rotated them then likely the only reason you haven't been compromised yet is out of luck.
Oh yeah, 100%. Even if they said the keys were for sure not leaked, I still would have rotated. Second I saw the disclosure on Jan 4, we went into emergency mode. Definitely not leaving that to chance.
Re: CircleCI incident report for January 4, 2023 security incident
#6This is a good writeup, and I appreciate the transparency. I especially like this bit:
>While one employee’s laptop was exploited through this sophisticated attack, a security incident is a systems failure. Our responsibility as an organization is to build layers of safeguards that protect against all attack vectors.
I was surprised by this part:
>To date, we have learned that an unauthorized third party leveraged malware deployed to a CircleCI engineer’s laptop in order to steal a valid, 2FA-backed SSO session.. the malware was able to execute session cookie theft, enabling them to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems.
I'm surprised the SSO session token isn't bound to an IP address. I'd also expect access to prod overall to be whitelisted to CircleCI-owned IP ranges.
Now some gripes:
* I never received an advisory email about this incident. I only received this follow-up to one of my Github machine accounts, not my primary billing account.
* Their secret-finding script is pretty bad. It just dumps out a bunch of metadata without helping to make it actionable. Environment variables still don't have a created_at field, so you can't verify which ones you might have missed in a broad key rotation.
Re: CircleCI incident report for January 4, 2023 security incident
#7This is a good writeup, and I appreciate the transparency. I especially like this bit: > While one employee’s laptop was exploited through this sophisticated attack, a security incident is a systems failure. Our responsibility as an organization is to build layers of safeguards that protect against all attack vectors. I was surprised by this part: > To date, we have learned that an unauthorized third party leveraged…
CircleCI is 100% remote. I can't imagine having to keep up with everyone's constantly changing home IPs and making exceptions while travelling.
Re: CircleCI incident report for January 4, 2023 security incident
#8This is a good writeup, and I appreciate the transparency. I especially like this bit: > While one employee’s laptop was exploited through this sophisticated attack, a security incident is a systems failure. Our responsibility as an organization is to build layers of safeguards that protect against all attack vectors. I was surprised by this part: > To date, we have learned that an unauthorized third party leveraged…
CircleCI is 100% remote. I can't imagine having to keep up with everyone's constantly changing home IPs and making exceptions while travelling.
I'd imagine they have a VPN.
Re: CircleCI incident report for January 4, 2023 security incident
#9Earlier quoted context omitted.
CircleCI is 100% remote. I can't imagine having to keep up with everyone's constantly changing home IPs and making exceptions while travelling.
I'd imagine they have a VPN.
But they must not be requiring it to access these admin systems. Not mentioned in the write up either, surprisingly
Re: CircleCI incident report for January 4, 2023 security incident
#10[dead]