Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

611–620 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#611

Post by dice »

thewebguyd wrote: post #588
qurren wrote: post #479

Earlier quoted context omitted.

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…

> take the 3am on-call and be the hero instead Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?" I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.

How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#613

Post by jerf »

Avicebron wrote: post #241

Nothing like generational debt to kick off a Friday morning

This is generational debt for a Kardashev Type III civilization. If there are individuals who are being charged "the whole of the GDP the human race has ever produced", then Amazon is claiming the rest of the world owes them a fairly significant portion of the galaxy if you consider the entire billing claims being made here.

There have been some big billing errors in the past but I wouldn't be surprised this is a new record. I know I've seen people charged around 92 quadrillion, which is a rounding error from underflowing a signed 64-bit integer with pennies, but those stories are usually isolated individuals. It sounds like the aggregate error here may comfortably exceed that. Hard to tell how widespread this is; as a data point my ~$2.00/month spend on S3 seems to still be billing correctly, so it must not be affecting everyone. But a few trillion here and a few trillion there and you get up to those quadrillions pretty quick.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#614

Post by Imustaskforhelp »

scrapcode wrote: post #265

Tale as old as time. When I was coming up it took a $20-40/m investment to get a "dedicated" server that you could start tinkering around on. When you couldn't afford that, you bricked the family PC trying to figure out how to configure your own LAMP stack. Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.

No not really, nobody has to be using AWS itself per se to be honest, there are still alternatives around.

I mean I own a 7$/yr vps that I pay with crypto and my stack usually involves golang/rust and oh I also self host my mail for temp-mail esque things on that server and I have some other servers floating around as well.

Its just that everyone uses AWS which is the wrong issue here to be honest. If you ask me, aside from the ram pricing caused by AI inflation, things are pretty good. The issue is that not many people know about server providers and other things and default to AWS/Azure etc. because big companies and small companies alike are using them.

Here's a global (interactive) visualization of around ~250 providers, 224 locations and 800 distinct links in 60 countries that I have made if this space interests ya: https://buyvds.net

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#615

Post by QuinnyPig »

donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

Was this the day that gp3 EBS volumes came out by any chance?

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#616

Post by 8note »

27183 wrote: post #535
mvdtnz wrote: post #511

Earlier quoted context omitted.

Why would you think there are "no tests"?

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

missing canaries more likely?

insufficient tests that dont assert on the right things?

the existence of a test doesnt mean it catches the right thing

based on the description, id bet the COE action item will be to do a migration that enforces units are passed at the billing service level

theres no good reason for the billing service to make up its own units.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#617

Post by 8note »

sscaryterry wrote: post #563
donavanm wrote: post #449

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…

This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.

its so far out as to be obviously not real

a smaller error by say, just one or two orders of magnitude are much more believable as a reader

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#618

Post by thewebguyd »

dice wrote: post #611
thewebguyd wrote: post #588

Earlier quoted context omitted.

> take the 3am on-call and be the hero instead Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?" I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.

How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?

Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#619

Post by ilamont »

You know, at one time Amazon was grouped in with all of the other big tech companies as a member of FAANG, as if the company has a culture of solid engineering at its core like Google or Apple.

It doesn't anymore, if it ever did. Read the shareholder letters over the last 5 years like this one (https://www.aboutamazon.com/news/company-news/amazon-ceo-and...). Move fast. Embrace AI in every corner of the company. Lay off thousands of people like we're a lean startup. Oh, but make sure to have a senior engineer check any AI code you want to push to production (https://news.ycombinator.com/item?id=47323017).

Amazon has made some impressive software, but there's also a ton of junk getting shipped. Customers and "partners" have to deal with it every single day. Sometimes it's "oops"-level stuff like this, at other times it's AWS regions going dark for hours.

Browse the Amazon seller forums (https://sellercentral.amazon.com/seller-forums/discussions?s...). Every day there are stories about back-end things breaking, or AIs overriding attributes or flagging random items as pesticides. The receiving errors at Amazon warehouses are incredible, and take months and multiple support tickets to fix.

Post reply on HN