Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota suffered a data breach by accidentally exposing a secret key on GitHub

blog.gitguardian.com

Post reply on HN
Print view View original

31–40 of 272 posts

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#31

Post by coenhyde »

deathanatos wrote: post #24
coenhyde wrote: post #13

Earlier quoted context omitted.

Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.

And yet I see it get violated all the time. People should do a lot of things, but a lot of my coworkers are lazy and do not do quality work. Given that it happens, and that I can't prevent it, one must then ask how to guard against it. At my org, we even try to generate all secrets with a standardize prefix/suffix so as to make them very greppable. That doesn't stop "Architects", "Customer Solutions", "Analytics" typ…

Doctors used to not wash their hands too. I get it though, and i've seen the same thing. Really it comes down to education and not granting access to secrets to people who aren't capable of handling them.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#32

Post by JauntyHatAngle »

wereHamster wrote: post #19
ajross wrote: post #2

"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking mistakes that you'd think would be easy to avoid, but aren't.

> Production keys in source control IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.

Huh? No, you use an external secrets manager or leave it to run time environment vars, or leave it to your cd servers to access/supply those details.

Assuming you have been given the right tooling, there is no reason for it to be in source code.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#33

Post by moooo99 »

wereHamster wrote: post #19
ajross wrote: post #2

"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking mistakes that you'd think would be easy to avoid, but aren't.

> Production keys in source control IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.

Even with IaC you should store secrets only in your environment variables for the CI servers

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#34

Post by ajross »

coenhyde wrote: post #13
ajross wrote: post #2

"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking mistakes that you'd think would be easy to avoid, but aren't.

Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.

"Should" not be violated is the point, though. I agree, it shouldn't. But it is, all the time.

I mean, I'll bet Toyota knew this organizationally. They had security people sign off on the design who all knew how secure key management is supposed to work. They probably reviewed this github release. And it happened anyway.

Maybe they weren't supposed to be production keys. Maybe it was a development key from someone's early cut of the tooling that got mistakenly reused for production. Maybe a script that updated all the keys mixed up which was which.

The point is that the existence of a Clear And Unambiguous Right Thing to Do is, in practice, not sufficient to ensure that that thing is done. The space of ways to mess up even obvious rules is too big.

And that's surprising, which is why (1) it keeps happening and (2) people like you don't take the possibility seriously in your own work.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#36

Post by ccakes »

gw99 wrote: post #9
lumberjack24 wrote: post #3

The access model on platforms like GitHub is flawed, a single account can be used for both professional and personal projects/repositories, leading to “fat finger” errors like this one here...

Oh yes this. It's so easy to critically fuck up an invite into an organisation. If you get typo the username you are potentially compromised. I've seen a couple of near misses on this already. Note: the invite input box actually autocompletes ALL github usernames.

You can invite by email addr, so the workaround here is only invite corporate email addresses.

If the target user hasn't added their corp email to their profile then they can't be part of the org.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#37

Post by zricethezav »

dinvlad wrote: post #26
jrockway wrote: post #20

> Git is an awesome version control system, used by over 93% of developers, and is at the heart of modern CI/CD pipelines. One of the benefits of Git is that everyone has a complete copy of the project they are working on. I feel like this is copy-pasted from a pitch deck on why GitGuardian should be funded. Does anyone reading the article care about this anecdote? Like do people stop reading at "well I'm one of the…

This is definitely a marketing piece. And they charge a LOT for it, so it's not a solution for the common masses.

If using GitHub-Actions, Gitleaks offers competitive pricing for a secret scanning solution.

https://gitleaks.io/products

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#38

Post by robswc »

It can actually be comical just how _bad_ things can be at large orgs.

Anyone have details, theories, or a book on how such inefficiencies come about? I can't speak to tech-oriented large orgs but I've worked with others and its just... I'm not shocked at all. I've seen public facing API keys in HTML, private SSH keys that do god knows what in plaintext on FTP servers... I just don't understand how they seem to care so much but in reality, care so little. Just lazy?

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#39

Post by coenhyde »

grepLeigh wrote: post #30
coenhyde wrote: post #13

Earlier quoted context omitted.

Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.

Surgeons have a practiced ritual ("scrubbing") to prep for surgery. Do you practice a credential-scanning ritual before saving (committing) your code or pushing your code to a remote repo? I have git hooks to lint code syntax, but nothing for scanning for leaked credentials. Looking @ TruffleHog now, mentioned by another poster.

That's certainly a good idea. But the secrets shouldn't be in the codebase to begin with, certainly not production secrets. Production secrets should stay in production and no one has access. Whatever intends to use the production secrets should have first been developed in a dev environment and released to prod.
Post reply on HN