Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#51

Post by kelnos »

SoftTalker wrote: post #14

Don't pay it. Send them notice, by registered letter, that the charges are fradulent. If a credit card was charged, try to initiate a chargeback/fraud claim. Once you pay it, you lose all leverage. You're much less likely to ever get any money back. Probably consult with a lawyer. Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In m…

If choosing to go this route, back up any data stored on Azure, and start looking into how to migrate everything to AWS or GCP or something else, without incurring too much downtime. Refusing to pay the charge, issuing a chargeback, or even getting a lawyer involved could get OP's account terminated, or at least suspended.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#52

Post by kelnos »

poly_morphis wrote: post #31
sarlalian wrote: post #22

Please setup billing alerts, know what your daily spend should be, add a little for if things grow a little unexpectedly. But you should absolutely be getting alerts if your spend is out of the ordinary for > 2 hours.

Slightly unrelated, but I do this for everything involving my personal finances. All of my credit cards have mobile apps that instantly push notifications whenever they are used. My bank app sends me notifications for any withdrawal or deposit. I even have alerts for my investments. I guess I'm just adamant in knowing exactly where (and when) my money comes or goes.

I was really happy when all my cards/banks started offering options like this. I essentially never have to go over my statements and transaction history with a fine-toothed comb anymore, because it's so so so easy to notice a fraudulent transaction immediately as it happens.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#53

Post by kelnos »

madaxe_again wrote: post #44

I’m sorry to hear this, this is a tough situation. Microsoft might, but are unlikely, to help you out. Similar situation with your bank. Neither face a legal obligation to help you, just potential bad PR if they don’t. Your best bet may be bankruptcy. It sounds terrible, but assuming you have an LLC/Ltd company, you can clear out your coffers, wind up, pay them pennies on the dollar, if anything, and start a new busi…

> Microsoft might, but are unlikely, to help you out.

Is that true? I have no experience with Microsoft, but I've heard quite a few stories of Amazon crediting AWS accounts when customers write in to say their account was compromised. Or even cases when the customer themselves screwed up some permissions in a way that ended up costing an arm and a leg. Hard to believe this practice would be unique to AWS.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#54

Post by ownagefool »

highwaylights wrote: post #46
NorwegianDude wrote: post #13

That really sucks and is the risk of using cloud solutions with no spending limit and a lack of monitoring. You should still have someone to keep an eye on it when using cloud solutions. And when you already have someone to keep an eye on it there's a good chance you might be better off managing the infrastructure yourself.

Not really sure how this exact scenario works but if their account was legitimately hacked couldnt the hacker just remove the caps?

So it depends how and what gets hacked.

The type of company posting about getting hacked like this is probably using the root / admin accounts to do most things. Their lowest hanging fruit and biggest wins would probably MFA, then SSO.

However, IAM is generally powerful enough to allow you to configure what an account can do.

So best practice, you also want to think about how you're going to lose credentials.

- Sharing passwords across services - leak of your .dotfiles, either by having your laptop pwned, or uploading your .dotfiles to a public repo as a backup or something. - Accidently pasting into the wrong window or something.

SSO & MFA defeats all of these with exception that your sts token will be signed for 1h in those .dotfiles when you auth yourself. I'm not sure what happens if you remove it from the token from the device, but the device itself being compromised would allow someone to piggy back your session.

Ergo, you move to least privilege access, and then if your laptop, server, or ci/cd runner gets hijacked, then it's only able to do whatever it was allowed to do in the first place.

The last part is you need to detect the misuse.

When you have least privileged access, and a pretty locked down account, the hope is when a session is hijacked, the attacker will attempt to use the credentials and get an access denied. This should allow you to detect and remediate the reason for leak.

Obviously this turns your cloud install into a lot more work, and you still also need to look at maintaining and patching the actual services so they're not compromised in the first place.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#55

Post by kelnos »

schnebbau wrote: post #34
benjaminwootton wrote: post #21

Earlier quoted context omitted.

I had an incident where a SaaS product went haywire and ran up a $10k bill. I was quite shocked when AWS didn’t write it off and pursued me for the money. You can’t necessarily assume a cloud provider will have your back in these situations.

I accidentally ordered a $200 bottle of wine instead of a $20 bottle at a restaurant once, but didn't realise until after the whole bottle was gone. I was not shocked when the restaurant expected me to pay for it, because obviously I was totally responsible. Why would AWS be any different?

Because the actual cost to AWS for a customer accidentally racking up an expensive bill is minuscule compared to the cost of that bottle of wine to the restaurant.

Amazon also has an interest in keeping their customers happy, and keeping usage growing. A customer who gets a refund for accidental or fraudulent usage is much more likely to remain a loyal customer, and hopefully spend more on the platform in the future. A restaurant, while in the hospitality industry, is likely more concerned with their razor-thin margins.

Also I would expect the true motive in your example is that the restaurant-goer is trying to scam the restaurant out of an expensive bottle of wine, and I'm sure most restaurant managers would agree with me. The AWS customer is not only more likely to have made an honest mistake, but AWS support has tools available to look at the usage and make a more nuanced decision as to whether or not the customer is telling the truth.

(And I have witnessed quite a few situations where Amazon has written off usage bills in cases like this. So clearly they agree with me on this, at least some of the time. Not all of the time, of course, as the grandparent poster can attest to.)

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#56

Post by shrikant »

bearjaws wrote: post #5

AWS would reimburse this if it was the first time. Maybe some hope for MS to do the same?

GCP forgave my team's bill of approx £220,000 when we'd misconfigured an ETL job that ended up running horribly expensive pay-as-you-go queries for about 3 days before we noticed.

It was pretty much entirely our fault, and we were still able to get those charges forgiven when we owned up to the error and asked nicely.

So I'd at least recommend asking Azure politely first.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#57

Post by kelnos »

schnebbau wrote: post #40
BartjeD wrote: post #38

Earlier quoted context omitted.

... Because building and running a piece of software is a vastly more complicated process than ordering a bottle of wine and drinking it. The way to hell is paved with well intended analogies.

The complexity of the process of causing loss to another does not absolve you from it.

That doesn't really matter, though. A cloud provider would likely have different motives and apply a different calculus to the decision of whether or not to offer a refund in a case like this, than a restaurant would over a bottle of expensive wine.

Amazon is thinking about customer retention and growth, and about the goodwill lost in refusing such a request. The restaurant is thinking about the likelihood that someone is just trying to get out of paying for something for which they did actually understand the cost.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#58

Post by kelnos »

sam0x17 wrote: post #24
gigel82 wrote: post #19

This is a conundrum. On one hand, I understand how frustrating something like this can be. But on the other hand, your cloud provider did provide those services that you're being billed for. So they did incur costs, why would they just eat those costs? Unless they're somehow at fault by exposing your credentials or making it easier for hackers to log in without 2FA or something of that nature. If you're using a credi…

> why would they just eat those costs? Beacuse the public indignation directed at cloud companies who don't always eat the costs in these situations vastly outweighs the cost of simply eating these costs, at least for cloud companies at the top tier of economies of scale (AWS, GCP, Azure, etc) If AWS didn't always eat costs like this, startups might think twice before using AWS, etc, etc.

Exactly this. Cloud providers have to acknowledge that building software on their platforms is necessarily complex, and inevitably bugs can cause extremely undesirable behavior. These providers are already charging a premium for instant, on-demand provisioning and nearly limitless pay-as-you-go resources, and know that there are too few guardrails in place to prevent accidental runaway situations.

"Goodwill" has value to a corporation. Taking a hard line against legitimate mistakes that anyone (yes, anyone) can make costs them goodwill, and costs them customers.

And beyond that, while accidental/fraudulent usage doesn't cost them $0, the services are marked up to the point that they probably doesn't really lose that much by forgiving the charges.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#59

Post by jve »

xwolfi wrote: post #15

When that happened to us, we found an article showing Tesla got hacked the same week as us (was aws) and they got the money back, so why not us? We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone.

Well, a guy who learned a thing or two the hard way and won't do that ever again got fired.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#60

Post by kelnos »

cmeacham98 wrote: post #39
causi wrote: post #23

Earlier quoted context omitted.

Because it wasn't OP who was hacked? The victim of the crime is Azure, not OP. If I have an Xbox account and someone hacks it and buys a bunch of games, it is the criminal who is deceiving Microsoft into thinking they are someone else. Microsoft trying to charge me for something someone else did would just be a second incidence of fraud. I could leave my Amazon account open on my desk and, assuming I could prove it w…

I'm not a lawyer, so this is not a legal prescription (I do not know who is legally liable in this scenario, I suspect it depends a lot on the details). That said, it seems like for society to work as it does we need people to take some level of responsibility over their action and inaction related to account security. If I live in the world you describe all online services will be forced to make you upload a photo I…

The problem with this stance is that the corporation naturally has much more power in the economic relationship than the customer does. If you give the vendor too much leeway to say "the customer should have been more careful with their credentials!" then they will always say that -- and usually prevail in that opinion -- even when the customer couldn't reasonably have done better.
Post reply on HN