Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

531–540 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#531

Post by deltaray3 »

01284a7e wrote: post #466
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

It's just like in Superman III

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#533

Post by mendigou »

tyre wrote: post #492
qurren wrote: post #479

Earlier quoted context omitted.

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…

Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.

I am former AWS and this is pretty accurate.

The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#534

Post by pjc50 »

IAmGraydon wrote: post #377
aerhardt wrote: post #303

One can almost smell the vibes. This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making. To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.

Clearing LLMs out of our business infrastructure is going to be a massive undertaking. Though I have a tech background, I work in commercial real estate. We are recently seeing new levels of idiocy from the employees, including real estate brokers with zero tech knowledge "coding" solutions to find sites for clients and blindly trusting the output (which I came to find out was complete bullshit), as well as some who…

> Clearing LLMs out of our business infrastructure is going to be a massive undertaking.

The asbestos of the future.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#535

Post by 27183 »

mvdtnz wrote: post #511
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

Why would you think there are "no tests"?

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#536

Post by pjc50 »

positr0n wrote: post #455
drew870mitchell wrote: post #285

Earlier quoted context omitted.

AWS is basically a utility. I think it's inevitable that their carelessness around billing will end up with them being regulated like one.

I can't think of another regulated utility that doesn't provide service to (essentially) all humans directly in their homes. Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.

I wonder what fraction of homes don't load anything from AWS on a daily basis. I suspect it's way below 50%.

(Of course, they don't know they're using it, they're using a service on it)

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#537

Post by 27183 »

AlotOfReading wrote: post #484
pudgywalsh wrote: post #471

Earlier quoted context omitted.

"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.

It's not difficult to write regression tests that catch unit mistakes.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#538

Post by switchbak »

geodel wrote: post #506
tyre wrote: post #492

Earlier quoted context omitted.

Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.

"Former" seems to an important detail here.

If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else.

Or in my case, actively ignore any and all recruiting from that sesspool.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#539

Post by zapkyeskrill »

AlotOfReading wrote: post #484
pudgywalsh wrote: post #471

Earlier quoted context omitted.

"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.

Degrees, very funny.
Post reply on HN