Earlier quoted context omitted.
$1.7-billion isn't a mundane detail Michael!
you beat me before I refreshed the page. what would you say... you do here?
listen... i got agent skills.. im good at talking to agents!!
AWS: Inaccurate Estimated Billing Data – $1.7 billion
581–590 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#582Earlier quoted context omitted.
The way you describe requires somehow counting every bit but somehow discarding most which is obviously nonsense. This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases. Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
You have to do it when the customer list is too big to keep a counter per customer.
No you don't
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#583Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
Imagine a programming language that has physical measurement unit support so this could have never happened.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#584Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…
CSV files are widely known to be used by the most frugal companies so of course it is.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#585Has anyone received any proactive communication about this? I didn't see the email until about 9 hours after it was sent out, yet I still needed to seek out information as to a) whether this was real or phishing and b) whether the amount was correct.
Seems totally irresponsible not to send an immediate follow up email to make customers aware.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#586Earlier quoted context omitted.
One of the Mars landers famously failed due to unit conversion errors from metric to standard.
I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)
Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures.
That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#587you too with Fable huh?
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#588Earlier quoted context omitted.
Not even tests but just some basic anomaly detection lol. Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…
> take the 3am on-call and be the hero instead
Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"
I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#589Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#590Earlier quoted context omitted.
"Former" seems to an important detail here.
If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
Not the OP, but:
The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.