Earlier quoted context omitted.
Is it really on the web service companies to make sure you (the developer) don't do anything stupid? The article author could have avoided a lot of headache if he simply checked the repo before walking away. Red flags should have gone up as soon as he received the first weird Amazon messages, but he still didn't think to check his own work. The author calls this a "Data Breach" but in reality it was a "Data Leak".
Amazon would save some money by doing this, since they usually reimburse "victims" of public AWS keys (at least on the first instance) and eat the cost.
How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
101–110 of 110 posts
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#102im impressed how fast a bot fetched the key from github, set up a trillion instances and did what exactly on them? edit: my guess, some kind of crypto currency mining?
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#103Earlier quoted context omitted.
> Don't ever check access keys into any repo, public, private or even self-hosted. So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad…
> So, how do you propose to handle version control of your exact production configuration? Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration. I personally don’t allow my team to have long-lived AWS access keys that have access to production. Production is only changed via an assumed role requiring MFA or an IAM instance role. If this k…
Why not? If your production configuration is stored as static-file in version control, then either those credentials have to live in that version control, or they have to live outside it, as a separately-backed up file of configuration variables, which are then fed to a static-file-generating tool, which means a) that one has to select a templating system and maintain those separate templates and b) that there's another step involved in developing the prototypes for those templates. This isn't a bad course of action, of course, but it does have its costs (just like storing production configuration—including credentials—in version control has its costs as well as its benefits).
> I personally don’t allow my team to have long-lived AWS access keys that have access to production.
Of course! Developers shouldn't have access to production systems, period. But operators can, and must, and I've no problem with them having access to their own version-controlled repository of information that no other team can access.
> Production is only changed via an assumed role requiring MFA or an IAM instance role.
Of course! I'm certainly not arguing for personal credentials to be stored in an operations repo.
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#104Earlier quoted context omitted.
> Don't ever check access keys into any repo, public, private or even self-hosted. So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad…
I use Ansible Vault if it's imperative to store keys in a repo, and I've been really happy with it. StackExchange has something similar called Blackbox[0]. I remember reading about a Ruby gem too, but I can't find it now. [0] https://github.com/StackExchange/blackbox
That looks pretty cool, at least at first. I'll have to take a deeper dive into it.
I like that it looks to be all shell scripts. Wish it were sh rather than bash, but c'est la vie.
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#105Earlier quoted context omitted.
Amazon would save some money by doing this, since they usually reimburse "victims" of public AWS keys (at least on the first instance) and eat the cost.
They don't "reimburse" anything. They only cancel the due amount. The only thing they lose is their wasted computing power (and some support costs) and I think the RAW computing power is very cheap to them, especially considering their scale. The cost in terms of support for the victims is probably bigger but still not significant.
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#1061) Seems like an inherent risk of closed source tools. 2) Someone other than the OP should pickup/waive the tab.
If he is using the GitHub Extension then it's open source see https://visualstudio.github.com/
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#107Earlier quoted context omitted.
There's a huge amount of value in a service that insulates you from mistakes like this. You really, really have to know what you're doing with EC2 because it is a loaded gun. Personally, I don't like using things that are covered in spring loaded booby traps that require eternal hypervigilance in order to avoid nasty financial consequences. Certainly not when deploying something that would be right at home on somethi…
> You really, really have to know what you're doing with EC2 because it is a loaded gun. That's a fair point, but keep in mind the article author was given several warnings that something was wrong, but failed to correct the problem each time (which resulted in the huge bill he ultimately received). At what point is the responsibility on the author here?
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#108This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to…
obviously someone is good at detecting it considering 6.5k in usage got racked up with data scraped within 10 minutes of commit
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#109This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to…
Is it really on the web service companies to make sure you (the developer) don't do anything stupid? The article author could have avoided a lot of headache if he simply checked the repo before walking away. Red flags should have gone up as soon as he received the first weird Amazon messages, but he still didn't think to check his own work. The author calls this a "Data Breach" but in reality it was a "Data Leak".
FTFY
It baffles me that people actually do this. Passwords and secrets DO NOT BELONG THERE. Even your 'if he simply checked the repo' statement is in my view completely crazy. No you should not have to worry about that, because that should not be there at all. Leaking your source-code should not result in a system-breach at any point.
Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
#110This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to…
You really want GitHub analyzing your code? REALLY? If the internet found out they were analyzing every push the backlash would be 10x what it is now. Even if it's public... That's still a Gmail-esque situation that would immediately raise privacy concerns. Developers should pay attention to what they're doing. Your AWS keys shouldn't even been in your private repo. You should find another way or you shouldn't be usi…