Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

humankode.com

Post reply on HN
Print view View original

61–70 of 110 posts

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#61

Post by slg »

Analemma_ wrote: post #25
vacri wrote: post #10

Earlier quoted context omitted.

AWS do do things. They have service limits and routinely waive charges from abuse events like this. I do wonder if GitHub's events firehose has benefits that outweigh all the compromised credentials, though. Not just limited to AWS keys.

> They have service limits and routinely waive charges from abuse events like this. That's almost worse though. Maybe I'm being unfair here, but the fact that they will waive charges immediately if you ask, but don't just block them pre-emptively, tells me that they're hoping at least some customers (with deep pockets and automated billing that no one pays attention to) will just pay the bill and never notice. Maybe…

Devils advocate, wouldn't you rather have illegitimate charges waived after the fact that legitimate charges declined because their algorithm threw up a red flag? Something like being posted to HN could cause a huge spike in charges for your account in a similar way as leaking your keys and it would be beyond frustrating to see that opportunity disappear because Amazon saw your bill spike.

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#62

Post by dsp1234 »

basch wrote: post #35
Analemma_ wrote: post #5

This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to…

obviously someone is good at detecting it considering 6.5k in usage got racked up with data scraped within 10 minutes of commit

The quick bitcoin mining spin-ups, and the initial email from Amazon about the key going public, are (probably) thanks to the github events firehose.

https://github.com/hootware/github-firehose-node

http://blog.scalyr.com/2013/10/exploring-the-github-firehose...

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#63

Post by Alupis »

nsxwolf wrote: post #54
Alupis wrote: post #24

Earlier quoted context omitted.

Is it really on the web service companies to make sure you (the developer) don't do anything stupid? The article author could have avoided a lot of headache if he simply checked the repo before walking away. Red flags should have gone up as soon as he received the first weird Amazon messages, but he still didn't think to check his own work. The author calls this a "Data Breach" but in reality it was a "Data Leak".

There's a huge amount of value in a service that insulates you from mistakes like this. You really, really have to know what you're doing with EC2 because it is a loaded gun. Personally, I don't like using things that are covered in spring loaded booby traps that require eternal hypervigilance in order to avoid nasty financial consequences. Certainly not when deploying something that would be right at home on somethi…

> You really, really have to know what you're doing with EC2 because it is a loaded gun.

That's a fair point, but keep in mind the article author was given several warnings that something was wrong, but failed to correct the problem each time (which resulted in the huge bill he ultimately received).

At what point is the responsibility on the author here?

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#64

Post by nailer »

merlincorey wrote: post #23
famousactress wrote: post #7

"As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach." We're only one sentence in and I'm already very, very nervous about the author's hubris.

Thankfully, the author realizes later what it they really did wrong here: > What could be done to prevent this? > Always test new version control GUIs before using them in the wild. There could be a bug that could expose your data. > Encrypt access keys in config files instead of just leaving them exposed in the config file. > Better yet, move access keys to a seperate config file, and exclude this from Git deploys w…

> Encrypt access keys in config files instead of just leaving them exposed in the config file.

How do decrypt them when you app re-launches? How do you decrypt them automatically on every instance?

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#65

Post by teuna »

Analemma_ wrote: post #5

This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to…

Natural selection at work, do not disturb.

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#66

Post by curun1r »

d2xdy2 wrote: post #21

> As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach. Ok... > Better yet, move access keys to a seperate config file, and exclude this from Git deploys with a .gitignore. No shit?

Even that's a terrible solution. AWS specifically changed their tooling to look for access keys in ~/.aws and make it more difficult to load them from inside a project directory. It's really not that hard to setup MFA and require it when using STS to assume whatever role you need to make changes. For automated changes, you can give instances the necessary an IAM role and only run those from inside EC2.

He didn't get a $6.5k bill because of a bug in VisualStudio, he got one because he didn't finish setting up his AWS account. Regardless of the public/private mixup, checking in AWS credentials to any repository is just bad practice and creates a huge risk of the exact situation that happened in this story.

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#67

Post by cptnbob »

ProAm wrote: post #26
famousactress wrote: post #7

"As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach." We're only one sentence in and I'm already very, very nervous about the author's hubris.

Why? It sounded like he is very security conscious. Thought he was creating a private repo in VS, immediately after he was notified deleted and recreated his keys. A pretty spot on course of action I think.

1. He stored credentials in source control rather than using configuration profiles as specified in the AWS SDK documentation.

2. He pushed code to a new repository without verifying the security configuration of the created repository (granted the tool made it easy to do this but you should create a repo, verify it, then push code).

3. He used his master AWS account key/secret in the code which gave global access to everything.

4. He didn't use IAM credentials with a restrictive policy set to just access the resources required.

No, clearly not security concious.

I'm a solution architect in the financial services industry and have been for 16 years. Never do I assume I know what the hell I'm doing.

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#68

Post by s73v3r »

tomphoolery wrote: post #38
Analemma_ wrote: post #5

This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to…

"letting people shoot themselves in the foot like this over and over" Stop doing it in the first place, problem solved. I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing. Also, this guy didn't just use AWS keys...he clearly used AWS master keys...which has not been the recommended course of action for AT LEAST THREE YEARS...if he had…

"I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing."

No it wouldn't, not at all. Not unless you're one of those, "REAL MEN us {$editor}" people, in which case no one cares about your opinion.

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#69

Post by cptnbob »

merlincorey wrote: post #23
famousactress wrote: post #7

"As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach." We're only one sentence in and I'm already very, very nervous about the author's hubris.

Thankfully, the author realizes later what it they really did wrong here: > What could be done to prevent this? > Always test new version control GUIs before using them in the wild. There could be a bug that could expose your data. > Encrypt access keys in config files instead of just leaving them exposed in the config file. > Better yet, move access keys to a seperate config file, and exclude this from Git deploys w…

He could also separate code and configuration as per the docs but I suspect the hubris excludes bothering to read them...:

http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/Developer...

Re: How a bug in VS2015 exposed my source code on GitHub and cost me $6,500

#70

Post by choward »

mbrutsch wrote: post #8
ocdtrekkie wrote: post #2

I still find it amazing that AWS doesn't offer any way to hardcap your billing and cut off service.

There's really no financial incentive to do so.

What about the potential customers they lose?
Post reply on HN