Earlier quoted context omitted.
Enterprise account . We got - 3trillion and change
Quick do your IPO before the books update
It's true, if you're spending that much money you must be worth a ton! Just look at SpaceX!
AWS: Inaccurate Estimated Billing Data – $1.7 billion
571–580 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#572Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
[dead]
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#573Earlier quoted context omitted.
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…
Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
Yes, I am a former AWS employee.
I got put on Focus because my "contributions were not coming through" to leadership.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#574Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error. > You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
Oh great so 2*30=60 he only owes 28.3$ million... hehe I guess you wanted to say 2^30 which makes 1.5$
But 2^30 is 28.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#575Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
I wonder if AWS billing still uses CSV files for passing data around.
IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#576[dead]
Wow, so true! Domain expertise is apparently not a requirement. At all. Wow.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#577I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.
Please do not open up emails directly always login to your account.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#578Earlier quoted context omitted.
Clearing LLMs out of our business infrastructure is going to be a massive undertaking. Though I have a tech background, I work in commercial real estate. We are recently seeing new levels of idiocy from the employees, including real estate brokers with zero tech knowledge "coding" solutions to find sites for clients and blindly trusting the output (which I came to find out was complete bullshit), as well as some who…
> Clearing LLMs out of our business infrastructure is going to be a massive undertaking. The asbestos of the future.
[deleted]
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#579Need some money for a new Launchpad
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#580Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.
This is embarrassing for Amazon, but I’d take laughably wrong over subtly wrong any day. If the bug made bills 20% higher I probably wouldn’t have queried it.