Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota suffered a data breach by accidentally exposing a secret key on GitHub

blog.gitguardian.com

Post reply on HN
Print view View original

251–260 of 272 posts

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#251

Post by freeqaz »

dinvlad wrote: post #57
zricethezav wrote: post #37

Earlier quoted context omitted.

If using GitHub-Actions, Gitleaks offers competitive pricing for a secret scanning solution. https://gitleaks.io/products

Thanks, I've already figured out how to run Trufflehog for free on our thousands of repos.

There is still a lot of noise with basic tools like this (I've also used trufflehog at scale).

To properly handle secret scanning requires calling live APIs to test if keys are "real". And you need to have a way to file tickets when you do have findings... if you rotate a cred from production, that's now an outage, so you need to coordinate multiple teams.

It's a lot of work and free tools only solve one part of this. I can't speak to any of the vendors in this space but I can attest that it's a harder problem than it seems!

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#252

Post by miohtama »

throwaway290 wrote: post #228
miohtama wrote: post #219

Earlier quoted context omitted.

There is already RTC 8958 Secret token scheme for this, so you do not need to invent your own prefix https://datatracker.ietf.org/doc/html/rfc8959

I see this standard linked here a lot. Did anyone read it though? It only helps with identifying whether a string is a secret, not at all the service or environment where the secret applies.

If any value does not natively support secret token sceme, you can apply secret-token: prefix and then strip it during the usage.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#253

Post by insane_dreamer »

scarface74 wrote: post #244
insane_dreamer wrote: post #242

Earlier quoted context omitted.

100% agree. We always keep all tokens (not just AWS secret keys) in a separate file that is never checked into the repo and are passed into the CloudFormation template at deployment. (The error in this case was a new repo hastily pushed and .gitignore wasn't properly updated to exclude the file with the keys.) But we've since switched to using AWS Secrets which is a much better solution.

Yeah that’s not good either. Your keys never need to be in a local file. Just put them in Parameter Store/Secrets Manager and you can reference those values in CF.

Yeah, that's what we do now

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#254

Post by prepend »

deathanatos wrote: post #249
prepend wrote: post #88

Earlier quoted context omitted.

Can you check with dns for new entries and check 443 for a couple weeks to see if there’s a tls cert there?

You can't enumerate DNS entries. (And not even privately: some of our (DNS) entries are wildcards, CNAMEs, etc. all make that hard.) We do (now) follow the CT logs for ourselves. That catches some cases, but not everything.

Why not? I can just check the zone file and go through line by line, right?

The wildcards would be tough but you could follow cnames as those would need to be in the cert as is.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#255

Post by dinvlad »

freeqaz wrote: post #251
dinvlad wrote: post #57

Earlier quoted context omitted.

Thanks, I've already figured out how to run Trufflehog for free on our thousands of repos.

There is still a lot of noise with basic tools like this (I've also used trufflehog at scale). To properly handle secret scanning requires calling live APIs to test if keys are "real". And you need to have a way to file tickets when you do have findings... if you rotate a cred from production, that's now an outage, so you need to coordinate multiple teams. It's a lot of work and free tools only solve one part of this…

Those are good points. Still, it’s fairly manageable, after certain adjustments. Also, we’re using the new (Go-based) version of TH that’s both much more performant and validates secrets against endpoints. I suspect their SaaS offering is a bit more polished and turn-key, but even the open-source one is quite decent. It doesn’t swamp us with FPs, at least.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#256

Post by dinvlad »

cwinq wrote: post #190
dinvlad wrote: post #26

Earlier quoted context omitted.

This is definitely a marketing piece. And they charge a LOT for it, so it's not a solution for the common masses.

Well, GitGuardian is free for individual developers (20 K of them use it - n°1 app on GitHub market place) and for team below 25. So I guess the masses can enjoy secrets free code! https://github.com/marketplace/gitguardian

I stand corrected on this, but what I’d argue is it’s not an affordable solution for medium-sized companies and non-profits who don’t swim in cash. It could be that our example is unusual (big non-profit), but when we evaluated GG the pricing left a sour taste..

More specifically, none of the paid security products we use cost nearly as much, and those products do much more than just detecting secrets. So from that standpoint, the pricing just seems outrageous. It’s pretty clearly aimed at big enterprises that can afford it and are vulnerable to FUD (while the “hobbyist” pricing is just free advertising). I don’t blame them for finding a way to make big money, but this business model is not what we’d pick.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#257

Post by rtev »

empyrrhicist wrote: post #216
bxtt wrote: post #204

Ah. I can’t believe this still happens in this day of age. About a decade ago, I was working for a startup and we were getting dominated in our growing space by a much larger, well funded rival. Our competitive intelligence team browsed through their git, and the rival actually exposed access to their customer, pricing and sales agent database by leaving their credentials in one of their branches. The team went to ou…

If you're in the US, that's 100% a crime. If they were responding to an unauthenticated API or web request that's one thing, but using a leaked password on a database is not legal at all.

Yeah, 3+ years in prison for that. Corporate espionage hacking is real.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#258

Post by pabs3 »

hnlmorg wrote: post #185
pabs3 wrote: post #181

Earlier quoted context omitted.

Would blocking commits containing such tokens/keys be a better option?

You cannot block what someone commits (they can block it themselves with tools like gitleaks invoked on a pre-commit hook) so the only thing you can do as a 3rd party is to scan and react when you do notice a secret published.

GitHub certainly could block push requests, at least git itself can via hooks, there are a number of hooks invoked by git-receive-pack that can influence what it does.

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#259

Post by Too »

grepLeigh wrote: post #30
coenhyde wrote: post #13

Earlier quoted context omitted.

Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.

Surgeons have a practiced ritual ("scrubbing") to prep for surgery. Do you practice a credential-scanning ritual before saving (committing) your code or pushing your code to a remote repo? I have git hooks to lint code syntax, but nothing for scanning for leaked credentials. Looking @ TruffleHog now, mentioned by another poster.

Code-reviews? Should be a ritual you do on your own code before commiting+pushing and should be a ritual that others will do in the PR before merge (arguable here a secret is already compromised).

Re: Toyota suffered a data breach by accidentally exposing a secret key on GitHub

#260

Post by Too »

D-Coder wrote: post #83
deathanatos wrote: post #29

Earlier quoted context omitted.

My company has monitoring for this, but it still seems to be a law of nature that, 1. someone adds new service/server/infra in a submarine manner 2. it goes to prod 3. the cert expires and outage begins 4. my team is asked what to do, because "we're the cert experts" 5. we add it to the monitoring So it only happens once … per service. Which isn't great. But how do you get people to slow down and do simple shit, like…

1. Clearly describe the correct process in your other process documentation. 2. Email everyone who might be involved a note about this and a link to the documentation and why it is important. 3. Next time someone ignores it, rip them and their manager a new orifice. 4. Wait for word of #3 to spread. Might help...

Trust me, half the recipients of the email would have forgotten about it within a week. You'd be lucky if they even opened it in the first place.
Post reply on HN