Is there a way (outside Github) that adversaries can get access to the "full feed" of commits? I don't understand how the attackers can find a new key from all the changes that must go into github across millions of repos, within 11 minutes.
What will happen when you commit secrets to a public Git repo?
11–20 of 68 posts
Re: What will happen when you commit secrets to a public Git repo?
#12Cool experiment! I PM the secret scanning team at GitHub and wanted to mention what GitHub did behind the scenes here. GitHub scans every commit to a public repo for secrets one of our secret scanning partners may have issued. We forward those candidate secrets to the issuing partner, and they take action. In some cases they auto-revoke the secret (AWS normally does this, I believe), in some cases they notify the use…
Why not refuse to publish a detected secret at all until the repo owner takes an action to allow it?
There are a few considerations on that one, but one very practical reason is the developer experience of dealing with false positives.
False positives are one of the big problems in secret scanning. Some partners issue credentials with patterns that make them very hard to distinguish from innocuous strings. For example, a Datadog token looks identical to a commit SHA. We would never block developers from pushing commits to GitHub just because they had 40 character hexadecimal strings in them!
GitHub's partnership approach works around the false positive problem by having the token issuer check whether a token is real and take action only if it is. However, this is a one-way communication from GitHub - the token issuer doesn't need to tell us whether the candidate secret we sent them was real or not, and in most cases we never know.
As a result, we can't replicate the zero false positive experience in a pre-receive hook (i.e., before the commit is pushed to GitHub). There would also be performance considerations from making 30+ http requests as part of a pre-receive hook.
In future, we are looking at creating a pre-receive hook solution that focuses on patterns that have a very low false positive rate. There are already some open source solutions that do this (links below) - in fact the OP linked to one from his Twitter thread. If/when GitHub offer is, it will definitely be opt-in, rather than opt-out!
Re: What will happen when you commit secrets to a public Git repo?
#13Is there a way (outside Github) that adversaries can get access to the "full feed" of commits? I don't understand how the attackers can find a new key from all the changes that must go into github across millions of repos, within 11 minutes.
There are bots (some even run by security and threat intel companies) feeding off of the firehose. For a public display of one type of scanning functionality, take a look at shhgit[0,1].
Re: What will happen when you commit secrets to a public Git repo?
#14Is there a way (outside Github) that adversaries can get access to the "full feed" of commits? I don't understand how the attackers can find a new key from all the changes that must go into github across millions of repos, within 11 minutes.
There are bots (some even run by security and threat intel companies) feeding off of the firehose. For a public display of one type of scanning functionality, take a look at shhgit[0,1]. 0: https://www.shhgit.com/ 1: https://github.com/eth0izzle/shhgit
Is the firehose public or do these companies have a relationship with github? If the latter, I assume github doesn't give the firehose feed to attackers who are only looking for AWS keys.
Re: What will happen when you commit secrets to a public Git repo?
#15Earlier quoted context omitted.
There are bots (some even run by security and threat intel companies) feeding off of the firehose. For a public display of one type of scanning functionality, take a look at shhgit[0,1]. 0: https://www.shhgit.com/ 1: https://github.com/eth0izzle/shhgit
Is the firehose public or do these companies have a relationship with github? If the latter, I assume github doesn't give the firehose feed to attackers who are only looking for AWS keys.
The author of sshgit wrote a great post on how it works using the public GitHub API: https://darkport.co.uk/blog/ahh-shhgit!/
Re: What will happen when you commit secrets to a public Git repo?
#16Just use a fucking blog, man. I'm so sick of threads like this.
Edit: I'm sorry, this came off as way more aggressive than I intended. I get why people use twitter to share stuff like this, but it's much harder to archive, find or reference in the future, not to mention it being much less readable than a simple webpage.
To anyone reading this, please consider publishing your findings on a blog as well as on twitter.
Re: What will happen when you commit secrets to a public Git repo?
#17In some cases, github will require you to remove the offending file from the commit history - or make the repo private.
e.g. https://github.com/aliostad/deep-learning-lang-detection
Re: What will happen when you commit secrets to a public Git repo?
#18More readable version:
Re: What will happen when you commit secrets to a public Git repo?
#19Just use a fucking blog, man. I'm so sick of threads like this. Edit: I'm sorry, this came off as way more aggressive than I intended. I get why people use twitter to share stuff like this, but it's much harder to archive, find or reference in the future, not to mention it being much less readable than a simple webpage. To anyone reading this, please consider publishing your findings on a blog as well as on twitter.
Or post it on Reddit or Medium or whatever if you can't be bothered with a blog.
Twitter "threads" need to die.
Re: What will happen when you commit secrets to a public Git repo?
#20Just use a fucking blog, man. I'm so sick of threads like this. Edit: I'm sorry, this came off as way more aggressive than I intended. I get why people use twitter to share stuff like this, but it's much harder to archive, find or reference in the future, not to mention it being much less readable than a simple webpage. To anyone reading this, please consider publishing your findings on a blog as well as on twitter.
You can use Nitter to make it a bit more readable and a bit less bloated. https://nitter.net/andrzejdyjak/status/1324360905237372929