Toyota suffered a data breach by accidentally exposing a secret key on GitHub • Hacker News Board

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#261

Post by Too » Sat, Oct 15, 2022, 1:06 PM UTC

coenhyde wrote: post #52

wereHamster wrote: post #19

Earlier quoted context omitted.

> Production keys in source control IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.

You can put your keys in source control if you are encrypting them with another key which is not in source control. Otherwise you're doing it wrong.

This is exactly how Ansible-vault works. It's many times better than committing them plain, but I'd still vote for some external service providing the secrets runtime only.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#262

Post by Too » Sat, Oct 15, 2022, 1:28 PM UTC

crazygringo wrote: post #106

fny wrote: post #104

Is there any reason why keys don't constantly update? It seems like a service could exist where every five minutes a rotation occurs across services with decaying privileges. For example, the 5 minute old key still works, but the 10 minute woman has completely expired.

How would you update the keys every 5 min? Either you'd use an encryption algorithm that depended on a "deeper" key... or you'd fetch the new key while authenticated with, you guessed it, another "deeper" key. It's keys all the way down. Every key you use, it's your responsibility to keep it private. (Unless you want to be dealing with physical hardware dongles that generate keys, but those aren't exactly easily port…

The refresh can be done with the old key, within the time-window. You only need the deeper key for the first time authentication, when starting up or provisioning the service.

What you have to consider is that starting a service is often not just a one time fire-and-forget operations. Applications crash need auto-restart, usually via systemd, kubernetes or something. So the keys-all-the-way down knowledge need to be integrated throughout that whole stack or on the side of it.

Kubernetes ServiceAccounts are based on a very similar flow, they are temporary certificates that are mounted into each Pod, used to connect to the API server and deciding what that Pod is allowed to access.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#263

Post by kadoban » Sat, Oct 15, 2022, 3:47 PM UTC

mypalmike wrote: post #189

fiddlerwoaroof wrote: post #133

Earlier quoted context omitted.

Yeah, the issue with pre-commit hooks is you have to remember to set them up client-side. I tend to push to GitHub through a gitolite mirror, though, so I could probably put this in the hooks in my gitolite middlebox.

What do you have to set up client side? They can be committed with the project. Or do I misunderstand?

Pre-commit hooks can't be automatically set up on the client side. If they could, this would mean that any repo you clone could run arbitrary code on your machine.

It can be as simple as a script you have to run once, but it can't be automatic. Which also means you can't really trust contributors to do it, even if they're well-meaning some will forget.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#264

Post by daniel-cussen » Sat, Oct 15, 2022, 5:31 PM UTC

grepLeigh wrote: post #30

coenhyde wrote: post #13

Earlier quoted context omitted.

Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.

Surgeons have a practiced ritual ("scrubbing") to prep for surgery. Do you practice a credential-scanning ritual before saving (committing) your code or pushing your code to a remote repo? I have git hooks to lint code syntax, but nothing for scanning for leaked credentials. Looking @ TruffleHog now, mentioned by another poster.

No post body was provided.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#265

Post by fsckboy » Mon, Oct 17, 2022, 5:35 AM UTC

danielodievich wrote: post #8

Many years ago I got a trial license key for something, Aspose components of some sorts I think, and without thinking of it, checked it in into public Github repo. Well, few days later Aspose's support sends me a nicely worded note saying that they noticed that it was there and invalidated it for me. Their description and instructions were very clear about why they did it and why I shouldn't have checked it in. I tho…

or it was being abused and they noticed the abuse

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#266

Post by hnlmorg » Mon, Oct 17, 2022, 9:22 PM UTC

pabs3 wrote: post #258

hnlmorg wrote: post #185

Earlier quoted context omitted.

You cannot block what someone commits (they can block it themselves with tools like gitleaks invoked on a pre-commit hook) so the only thing you can do as a 3rd party is to scan and react when you do notice a secret published.

GitHub certainly could block push requests, at least git itself can via hooks, there are a number of hooks invoked by git-receive-pack that can influence what it does.

> GitHub certainly could block push requests

But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature.

> at least git itself can via hooks

I already said that:

> they can block it themselves with tools like gitleaks invoked on a pre-commit hook

The problem with git hooks is that they're not cloned with the repo. So you're reliant on the user installing those git hooks locally (sure, some repos will have helper scripts to install the hooks for you. But you're still reliant on the user running that script).

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#267

Post by pabs3 » Wed, Oct 19, 2022, 2:43 AM UTC

hnlmorg wrote: post #266

pabs3 wrote: post #258

Earlier quoted context omitted.

GitHub certainly could block push requests, at least git itself can via hooks, there are a number of hooks invoked by git-receive-pack that can influence what it does.

> GitHub certainly could block push requests But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature. > at least git itself can via hooks I already said that: > they can block it themselves with tools like gitleaks invoked on a pre-commit hook The problem with git hooks is that they're not cl…

> code you cannot push to origin. Definitely not a desirable feature.

If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data.

> The problem with git hooks

I was talking about GitHub's own git hooks that run on their servers, not about any local ones.

> is that they're not cloned with the repo.

It would be a terrible security issue if they were automatically enabled after cloning.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#268

Post by hnlmorg » Wed, Oct 19, 2022, 6:15 PM UTC

pabs3 wrote: post #267

hnlmorg wrote: post #266

Earlier quoted context omitted.

> GitHub certainly could block push requests But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature. > at least git itself can via hooks I already said that: > they can block it themselves with tools like gitleaks invoked on a pre-commit hook The problem with git hooks is that they're not cl…

> code you cannot push to origin. Definitely not a desirable feature. If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. > The problem with git hooks I was talking about GitHub's own git hooks that run on their servers, not about any local ones. > is that they're not cloned with the repo. It would be a terrible sec…

> If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data.

It’s already too late by that point because your secrets have already left the building. You’re not relying on upstream being honourable

> I was talking about GitHub's own git hooks that run on their servers, not about any local ones.

There’s no such thing. You can have CI tooling like GitHub Actions, but they’re a different beast to git hooks

> It would be a terrible security issue if they were automatically enabled after cloning.

It doesn’t have to be either/or. There are ways of having a sensible compromise. Like a git config that enables hooks from known safe origins. Or having the user promoted whether they want to install git hooks upon cloning.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#269

Post by pabs3 » Thu, Oct 20, 2022, 8:43 AM UTC

hnlmorg wrote: post #268

pabs3 wrote: post #267

Earlier quoted context omitted.

> code you cannot push to origin. Definitely not a desirable feature. If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. > The problem with git hooks I was talking about GitHub's own git hooks that run on their servers, not about any local ones. > is that they're not cloned with the repo. It would be a terrible sec…

> If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. It’s already too late by that point because your secrets have already left the building. You’re not relying on upstream being honourable > I was talking about GitHub's own git hooks that run on their servers, not about any local ones. There’s no such thing. You c…

> It’s already too late by that point

True, but it is better than the secrets becoming entirely public, automated bots could be harvesting them and exploiting the resources they protect.

> There’s no such thing.

I would be surprised to here that GitHub doesn't actually run git on their servers. If they receive git pushes using git, then own git hooks are involved, ones that GitHub has written for their own purposes. They could simply add one to block bad pushes.

> Like a git config that enables hooks from known safe origins.

That sounds a bit terrifying to me, but I'm not of the GitHub generation.

> Or having the user promoted whether they want to install git hooks upon cloning.

That sounds like it would enable phishing-like attacks and people just clicking "yeah sure" without verifying the safety of the hook.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#270

Post by hnlmorg » Thu, Oct 20, 2022, 6:24 PM UTC

pabs3 wrote: post #269

hnlmorg wrote: post #268

Earlier quoted context omitted.

> If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. It’s already too late by that point because your secrets have already left the building. You’re not relying on upstream being honourable > I was talking about GitHub's own git hooks that run on their servers, not about any local ones. There’s no such thing. You c…

> It’s already too late by that point True, but it is better than the secrets becoming entirely public, automated bots could be harvesting them and exploiting the resources they protect. > There’s no such thing. I would be surprised to here that GitHub doesn't actually run git on their servers. If they receive git pushes using git, then own git hooks are involved, ones that GitHub has written for their own purposes.…

> True, but it is better than the secrets becoming entirely public, automated bots could be harvesting them and exploiting the resources they protect.

True. And some popular repos do already run into this problem. So it’s not a theoretical problem.

> I would be surprised to here that GitHub doesn't actually run git on their servers.

They’ve documented about how their backend works so there’s no need to speculate. They run an implementation of git but not the standard git CLI.

> If they receive git pushes using git, then own git hooks are involved, ones that GitHub has written for their own purposes. They could simply add one to block bad pushes.

They have their automation, GitHub Actions.

Sure they “could” also implement what you’ve described but it’s not how it currently works. So a pointless argument since we could be here all year discussing the literal infinity of different things Github “could” do in theory but that their infrastructure doesn’t currently support.

> That sounds a bit terrifying to me, but I'm not of the GitHub generation.

What I posted has literally nothing to do with GitHub. In fact if your origin is private git server (as I started out using git, since GitHub didn’t exist back then) then it’s even easier to designate a trusted origin. This approach makes total sense for businesses. Doesn’t work so well for open source but it’s just one option of many.

> That sounds like it would enable phishing-like attacks and people just clicking "yeah sure" without verifying the safety of the hook.

Potentially yes. But if you’re cloning a git repo, making code changes and then committing it back, you’d hope that individual is competent enough to audit the git hook. At the very least, they’ll be running the build scripts locally to unit test their changes, so it’s not like that phishing attack isn’t already present. Feels very much like you’re looking for reasons to dismiss any suggestions here rather than have an intelligent discussion.