Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CircleCI security alert: Rotate any secrets stored in CircleCI

circleci.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#71

Post by ryanisnan »

icco wrote: post #68
taf2 wrote: post #60

Earlier quoted context omitted.

Can elaborate where would I have seen the writing for this? What indicators did you see?

Layoffs and outages at Circle. software supply chain attacks becoming more and more popular.

Outages at CircleCI have been common since the service has been launched.

Amazon is about to lay off 18k people. Circle isn't in a unique position to my understanding. Did CircleCI lay off some group or set of really important and key personnel?

Software supply chain attacks affect everyone. Is there some way that Circle is more vulnerable to this type of attack?

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#73

Post by mnahkies »

lrvick wrote: post #53
bamboozled wrote: post #50

Earlier quoted context omitted.

It's nice you can do that, it doesn't work for large distributed teams.

Sure it does. Do engineers not compile their code locally constantly as a part of the process of writing it? Store deterministic hashes of expected binaries with signed commits in PRs. Then untrusted CI merely needs to generate and sign -matching- hashes and now we are good as long as the engineer and CI system are not compromised at the same time.

You're talking about creating reproducible builds - which is a good idea, but in most cases you will still need to deliver that binary somewhere.

That typically requires authentication, whether you're deploying to kubernetes or copying the files somewhere using scp, etc

So either your laptop or the ci system needs some level of secrets present to put the artifact in the correct place

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#74

Post by aequitas »

tootie wrote: post #67
aequitas wrote: post #56

Earlier quoted context omitted.

> so we're asking teams to spend potentially hours or days fixing things At the risk of sounding pedantic, but this is why you have everything as IaC. These kind of changes should not cost days. It should take merely minutes or an hour tops to change all your keys. It should be trivial, for cases just like this.

You can't use IaC to change third-party API keys. And woe unto any service that doesn't allow multiple keys because then you're looking at outages.

You can, for circleci for example: https://registry.terraform.io/providers/mrolla/circleci/late...

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#75

Post by tootie »

aequitas wrote: post #74
tootie wrote: post #67

Earlier quoted context omitted.

You can't use IaC to change third-party API keys. And woe unto any service that doesn't allow multiple keys because then you're looking at outages.

You can, for circleci for example: https://registry.terraform.io/providers/mrolla/circleci/late...

I get that you can manage the values in Circle, but you can't actually generate the values. IE, if you have a API token to write to Salesforce, you have to go into the Salesforce admin and generate a new token. Pasting the value in the Circle UI or a terraform descriptor are not the hard part. For lots of services, you can only have one key at a time meaning that generating a new one invalidates the old one meaning you'd have to have an outage while you're pasting and deploying.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#76

Post by theogravity »

DylanL wrote: post #41
theogravity wrote: post #9

Does this also include deploy SSH keys?

Looks like yes: https://discuss.circleci.com/t/circleci-security-alert-rotat...

Thanks. Sigh, I have ~20 OSS repos with them that I'll need to generate new keys for then.

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#77

Post by chrisntb »

theogravity wrote: post #9

Does this also include deploy SSH keys?

We contacted CircleCI support and they clarified their blog post statement with the following info:

"Thank you for contacting CircleCI Support.

This does also apply to SSH Keys, as such we do recommend to rotate SSH Keys as well as to take extra caution.

If you have any other concerns please reach out."

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#78

Post by throwaway892238 »

@dang this is currently #198 off the front page, yet this is basically an emergency (literally every customer's secrets are exposed?)... either circleci has no more customers, or people are very calm about this...

we need to rotate:

  - secrets in context environment variables
  - secrets in project environment variables
  - project deploy keys
  - circleci api tokens
then we have to go back and look at all audit logs for... basically everything... and try to find something that looks weird. :/

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#79

Post by ojkelly »

jgaa wrote: post #44

I really don't understand why you use someones else's computer to compile and test your stuff. When their computers are compromised, by internal or external crooks, the crooks have full access to your code, and - in some cases - your data. If they wanted, they could inject their own shit into your binaries, totally ruining your reputation. As a bonus, you get to pay a premium! I still compile and test my code on my o…

You do need to trust someone else’s computer if they’re going to build and run your code. I think Google is doing some good work here in helping champion things like Supply-chain Levels for Software Artifacts (SLSA) [0][1]. I’d argue your build/CI/CD system should never have access to production data, but it would indirectly by being able to mutate your production environment (to deploy things). Compiling and testing on your own machine it’s necessarily safer though. Compare a typical CI/CD build instance which is usually a VM or container that has been freshly booted, or is being reused from a recent build, with your own machine that you likely also use to browse the internet and run many other apps. The (ideal of the) former is a reproducible on-demand environment with a specific toolchain, while the latter is a bespoke assortment of different toolchains, software, and unfinished projects. Not to mention your machine will not be the same as someone else on your team. I think as an industry we still have a lot of work to do around establishing trusted computing environments for CI/CD and enabling the level of auditability and observability to verify that. There are also CI/CD providers that you can run on your own infrastructure.

[0] https://cloud.google.com/blog/products/application-developme...

[1] https://slsa.dev (edit: fixed this link)

Re: CircleCI security alert: Rotate any secrets stored in CircleCI

#80

Post by bbu »

throwaway892238 wrote: post #78

@dang this is currently #198 off the front page, yet this is basically an emergency (literally every customer's secrets are exposed?)... either circleci has no more customers, or people are very calm about this... we need to rotate: - secrets in context environment variables - secrets in project environment variables - project deploy keys - circleci api tokens then we have to go back and look at all audit logs for...…

this is such a clusterfuck... and the circleci api doesn't even allow to automate most of the steps. and the ones that should work, error with "internal server error". of course, support is completely unresponsive
Post reply on HN