Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CircleCI security alert: Rotate any secrets stored in CircleCI

circleci.com

Post reply on HN
Print view View original

51–60 of 87 posts

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#51

Post by bamboozled »

> We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well.

Is anyone else a little annoyed by the messaging here, I read it as, "We think something bad happened to your ultra secret data, but we don't know, so we're asking teams to spend potentially hours or days fixing things while we aren't really able to tell you if your stuff was actually compromised"?

What I find more troubling is, if they don't quite know what happened, or aren't telling us, and we do the work to change everything, how do they know it won't just happen again in the next day or so and people are still accessing our systems, where is the details?

> At this point, we are confident that there are no unauthorized actors active in our systems.

Confident isn't really a good enough word to use here in my opinion. We've just blocked Circle CI from all our systems for now until we hear more, likely start to move to another build system.

I know accidents happen but this is likely the beginning of the end for our teams relationship with Circle CI. Trust has been broken.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#52

Post by csomar »

bamboozled wrote: post #51

> We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative meas…

I can see you are bamboozled. But you should have seen the writing on the wall for CircleCI for quite some time now.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#53

Post by lrvick »

bamboozled wrote: post #50
jgaa wrote: post #44

I really don't understand why you use someones else's computer to compile and test your stuff. When their computers are compromised, by internal or external crooks, the crooks have full access to your code, and - in some cases - your data. If they wanted, they could inject their own shit into your binaries, totally ruining your reputation. As a bonus, you get to pay a premium! I still compile and test my code on my o…

It's nice you can do that, it doesn't work for large distributed teams.

Sure it does. Do engineers not compile their code locally constantly as a part of the process of writing it? Store deterministic hashes of expected binaries with signed commits in PRs. Then untrusted CI merely needs to generate and sign -matching- hashes and now we are good as long as the engineer and CI system are not compromised at the same time.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#54

Post by jgaa »

bamboozled wrote: post #50
jgaa wrote: post #44

I really don't understand why you use someones else's computer to compile and test your stuff. When their computers are compromised, by internal or external crooks, the crooks have full access to your code, and - in some cases - your data. If they wanted, they could inject their own shit into your binaries, totally ruining your reputation. As a bonus, you get to pay a premium! I still compile and test my code on my o…

It's nice you can do that, it doesn't work for large distributed teams.

I think that depends entirely on the priorities and incentives driving the company.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#55

Post by danw1979 »

Corrado wrote: post #47
rcme wrote: post #27

Earlier quoted context omitted.

This only applies if the stolen credentials can’t create roles and can’t modify existing roles.

This is a good reminder to always follow least-permission best practices.

I’d add drift detection on everything IAM / SCP / Org to this list too.

A session token with only a few minutes validity can be enough for someone to make their access permanent.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#56

Post by aequitas »

bamboozled wrote: post #51

> We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative meas…

> so we're asking teams to spend potentially hours or days fixing things

At the risk of sounding pedantic, but this is why you have everything as IaC. These kind of changes should not cost days. It should take merely minutes or an hour tops to change all your keys. It should be trivial, for cases just like this.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#57

Post by namaria »

rektide wrote: post #7

Our hodgepodge of microservices- developed over more than a decade- never got coordinated env variables, so now we've got to go through like ~50 services & libraries, one by one, updating secrets. Yuck. If you do your shit right, you can just dump most of your secrets into some Contexts- containers of env variables- and apply them. Then when this stuff roles around, it's easy to update everything centrally; change th…

Such is the tragedy of for-profit software engineering. The trade-offs we see today lead to choices that tie our hands when facing trade-offs we didn't foresee. Also why experience comes at such a premium. Seeing further down the line and knowing how to argue about it prevents whole classes of problems.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#58

Post by namaria »

__MatrixMan__ wrote: post #28
VWWHFSfQ wrote: post #26

Earlier quoted context omitted.

I don't think locking down the account will do anything. It sounds like secrets were already stolen. GitHub access tokens, etc. Locking the account won't unsteal that stuff.

Right. You'd need lock-down-all-AWS-controlled-by-the-foo-key because CircleCI got hacked and it had the foo-key. Sounds like a separate product (something about breaches and blast radii) and not a CircleCI feature.

You mean hanging whole sections of our value chain on other companies' assets was not the best idea?

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#59

Post by hacb »

lrvick wrote: post #53
bamboozled wrote: post #50

Earlier quoted context omitted.

It's nice you can do that, it doesn't work for large distributed teams.

Sure it does. Do engineers not compile their code locally constantly as a part of the process of writing it? Store deterministic hashes of expected binaries with signed commits in PRs. Then untrusted CI merely needs to generate and sign -matching- hashes and now we are good as long as the engineer and CI system are not compromised at the same time.

What about testing? In my company, before any code goes to production it has to go through hundreds if not thousands of unit tests. This can't be done on a dev laptop (see XKCD #303)

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#60

Post by taf2 »

csomar wrote: post #52
bamboozled wrote: post #51

> We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative meas…

I can see you are bamboozled. But you should have seen the writing on the wall for CircleCI for quite some time now.

Can elaborate where would I have seen the writing for this? What indicators did you see?
Post reply on HN