Live data from Hacker News

Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

news.ycombinator.com

71–80 of 81 posts

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#71
post #37

Earlier quoted context omitted.

Why was that guy running Jenkins without a password?

Years ago when I actually did any devops running services without a password was common. They would configured to only be accessible using SSH from a locked down IP range. It's far more secure than password based access (if you get it right).

Yes, but that was a design/security choice. What OP described seems like a pure negligence ("setting up authentication is hard, so let's skip it").

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#72
post #60

Earlier quoted context omitted.

I'm not a lawyer, so this is not a legal prescription (I do not know who is legally liable in this scenario, I suspect it depends a lot on the details). That said, it seems like for society to work as it does we need people to take some level of responsibility over their action and inaction related to account security. If I live in the world you describe all online services will be forced to make you upload a photo I…

The problem with this stance is that the corporation naturally has much more power in the economic relationship than the customer does. If you give the vendor too much leeway to say "the customer should have been more careful with their credentials!" then they will always say that -- and usually prevail in that opinion -- even when the customer couldn't reasonably have done better.

You seem to believe I said something like "we should always believe the company no matter what the evidence says", but if you reread my comment you'll find that I didn't.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#73

Earlier quoted context omitted.

I had an incident where a SaaS product went haywire and ran up a $10k bill. I was quite shocked when AWS didn’t write it off and pursued me for the money. You can’t necessarily assume a cloud provider will have your back in these situations.

I accidentally ordered a $200 bottle of wine instead of a $20 bottle at a restaurant once, but didn't realise until after the whole bottle was gone. I was not shocked when the restaurant expected me to pay for it, because obviously I was totally responsible. Why would AWS be any different?

Some restaurants pull a bait and switch on customers ordering wine, bringing a similar bu wildly more expensive bottle in place of the one that was ordered, hoping for exactly the outcome that you described.

I would be shocked to find a cloud provider doing that, but I wanted to point out that you might be not have made a real mistake.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#75
post #23
post #20

Earlier quoted context omitted.

To play the devil's advocate, why would Azure be liable for OP's security lapse? This would've been much easier if someone stole your credit card and bought things with it (the CC company would help with the chargeback).

Because it wasn't OP who was hacked? The victim of the crime is Azure, not OP. If I have an Xbox account and someone hacks it and buys a bunch of games, it is the criminal who is deceiving Microsoft into thinking they are someone else. Microsoft trying to charge me for something someone else did would just be a second incidence of fraud. I could leave my Amazon account open on my desk and, assuming I could prove it w…

Azure does bills on credit, IE: you spend and pay later. That's up to them, but it's far riskier than prepurchased credits.

I'd find a jury unwilling to believe that a similar real life scenario would raise no flags. It's only a flag raiser because tech companies have automated away all human interaction with billing. Imagine someone claiming to be bob, who regularly shops at the grocery store for 100 dollars a week, now wants to come in and spend, say, 10,000 dollars, on credit. This would be a red flag to any proprietor. Now imagine that proprietor going after bob, who was not there, and claim he is responsible.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#76
post #51

Don't pay it. Send them notice, by registered letter, that the charges are fradulent. If a credit card was charged, try to initiate a chargeback/fraud claim. Once you pay it, you lose all leverage. You're much less likely to ever get any money back. Probably consult with a lawyer. Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In m…

If choosing to go this route, back up any data stored on Azure, and start looking into how to migrate everything to AWS or GCP or something else, without incurring too much downtime. Refusing to pay the charge, issuing a chargeback, or even getting a lawyer involved could get OP's account terminated, or at least suspended.

Or even go back to good old VPS/metal servers. Maybe a little more work upfront but less likely to have surprises like this. Also more competition in that space and no worries about vendor lock-ins.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#77
post #4

Step 1: Read your cloud services contract with Microsoft very carefully. What does it say about your liability for fraud? Step 2: Read your business insurance policy very carefully. What does it say about fraud coverage? What are the limits and exclusions? Step 3: Unless 1 or 2 makes it real clear the business is not liable, get a lawyer.

Relevant agreements are perhaps one of:

1. OP Partner (confidential agreement) Microsoft (MPA)[1]

2. OP Microsoft (MCA)[2]

3. OP Microsoft (MOSA)[3]

The different types of agreements have different limitations of liability clauses. What OP wrote indicates a "partner" is involved and if this is the case, Microsoft have essentially shifted liability for fraud and billing non-payments onto the "partner"[4], who would then either wear the cost or try to shift this liability to the OP. It's not that straightforward though as any of the three parties could have a share of liability, and the "partner" would be very unlikely to want to get in a dispute with Microsoft as this would impact their other business. Liabilities are possibly also impacted by default spending limits and caps that are imposed by Microsoft on different services[5].

Allowing a $200,000 bill for one month (a 200x increase) has the appearance of being very poor financial management from the "partner" as they're potentially going to be stuck with unsecured $200,000+ liabilities from their customers if the customers became insolvent. I suppose it is possible the OP and "partner" have a bank guarantee in place to cover at least $200,000 but I'd hazard a guess they may just try to rely on an insurance policy instead to cover these rare events.

[1] Microsoft Partner Agreement (MPA): https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE...

[2] Microsoft Customer Agreement (MCA): https://www.microsoft.com/licensing/docs/customeragreement

[3] Microsoft Online Subscription Agreement (MOSA): https://azure.microsoft.com/en-au/support/legal/subscription...

[4] https://learn.microsoft.com/en-us/partner-center/non-payment...

[5] https://azure.microsoft.com/en-us/support/legal/offer-detail...

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#78
post #25

Don't pay it. Send them notice, by registered letter, that the charges are fradulent. If a credit card was charged, try to initiate a chargeback/fraud claim. Once you pay it, you lose all leverage. You're much less likely to ever get any money back. Probably consult with a lawyer. Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In m…

If your business is dependent on Azure, what happens when they shut down your services for lack of payment? This seems extremely risky.

They business might not be able to pay 100x their monthly cost either

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#79
post #23
post #20

Earlier quoted context omitted.

To play the devil's advocate, why would Azure be liable for OP's security lapse? This would've been much easier if someone stole your credit card and bought things with it (the CC company would help with the chargeback).

Because it wasn't OP who was hacked? The victim of the crime is Azure, not OP. If I have an Xbox account and someone hacks it and buys a bunch of games, it is the criminal who is deceiving Microsoft into thinking they are someone else. Microsoft trying to charge me for something someone else did would just be a second incidence of fraud. I could leave my Amazon account open on my desk and, assuming I could prove it w…

This is 100% OP’s fault, it’s not like Azure itself was breached. Their account got hacked due to weak security practices on their part.

Re: Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

#80
post #23

Earlier quoted context omitted.

Because it wasn't OP who was hacked? The victim of the crime is Azure, not OP. If I have an Xbox account and someone hacks it and buys a bunch of games, it is the criminal who is deceiving Microsoft into thinking they are someone else. Microsoft trying to charge me for something someone else did would just be a second incidence of fraud. I could leave my Amazon account open on my desk and, assuming I could prove it w…

Azure does bills on credit, IE: you spend and pay later. That's up to them, but it's far riskier than prepurchased credits. I'd find a jury unwilling to believe that a similar real life scenario would raise no flags. It's only a flag raiser because tech companies have automated away all human interaction with billing. Imagine someone claiming to be bob, who regularly shops at the grocery store for 100 dollars a week,…

We don't know enough about what happened but I disagree this should be an automatic red flag from the provider's side. I'm sure Azure has spending limits and alarms that one can set up (and probably should).

The attitude of "provider should eat the cost" is ripe for abuse. I can set up some expensive GPU instances to train my GPT4 clone (millions worth of compute) than -after getting my models- claim I was "hacked" and refuse to pay my bill. Or maybe -more benign- have some buyer remorse after setting up a public instance for people to test out my new AI product then get scared when it gets the HN "hug of death" and my bill skyrockets.

Post reply on HN