Test is king!
AWS: Inaccurate Estimated Billing Data – $1.7 billion
751–760 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#752Earlier quoted context omitted.
https://health.amazon.com/prime
https://health.amazon.com/prime Ugh. No. Never. Do not use Amazon Pharmacy. I had it for three months, and each month it was unable to deliver pills before I ran out, so I cancelled and switched to the brick-and-mortar pharmacy down the street. A year later, suddenly Amazon Pharmacy starts sending unauthorized prescription refill requests to my doctor. I still have the account cancellation confirmation e-mail from Am…
It wasn't a recommendation, more of a sick joke that "Amazon has that, too".
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#753[flagged]
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#754From a CFO perspective, this is catastrophic. With monthly billing automatically feeding monthly close pipelines, this would result in results that either need to be adjusted (without proper support) or tank performance until it's fixed on the vendor side. I wouldn't want to be in that situation.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#755Earlier quoted context omitted.
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
>no tests? Earlier this week my slopservant implemented several comprehensive changes to a codebase. It also wrote extensive tests to verify the correctness of the changes. A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless. The many tests it had written were just confirming the LLM's idea of correctness…
> if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!
This is a well-known issue in formal methods, often referred to as the "formalization gap". It's pretty much an intractable problem, i.e. all we can do is manage it, not solve it.
In fact this issue exists in many fields, often under the name "specification gap" or "specification problem". You'll find versions of that discussed inv economics, architecture (of buildings, not software), etc. In architecture and construction, a high percentage of mistakes that result in lawsuits turn out to be due not to construction, but to specifications that were incorrect in some way.
The bottom line is that there's no perfect formal/mathematical way to go from intent to proof.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#756Earlier quoted context omitted.
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…
> Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was. If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).
Thanks, I wasn’t familiar with XTS.
Now I’m picturing the 3am end of quarter fire drill in finance when they discover the company has accounts receivable of fourteen billion $XTS and it’s appearing in the quarterlies.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#757Earlier quoted context omitted.
This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feel…
I have bad news if you think banks are the pinnacle of cyber security practices... Maybe some rare few are, but by and large they generally stink. They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
The difference is that at bigger banks at least, they tend to throw more money at the problem. They'll have defense in depth with e.g. DMZs, proxies for ingress and egress, locked down workstations and browsers, etc., and a different team for each one of those domains.
So while the actual "on the ground" picture may look suboptimal at any given point, overall it does make for security that in practice is much better than average.
This is reflected in the actual security compromise statistics. Your money in a bank is a lot safer than, say, your credit card deals on file with many retailers.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#758Earlier quoted context omitted.
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…
the cynical take is of course that the tests were written by Claude and not validated. Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.
I love how AI means that all human-written code is now perfect, and any problems are onbviously a symptom of vibe coding. Another few years of hearing this and I may begin to believe I only wrote perfect code, when my current memory suggests otherwise.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#759AWS billing is inaccurate by $1.7B and my first thought was "maybe I do owe them that much."
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#760Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
[dead]