AI doing its "thing". Making billing software work the way it isn't supposed to.
AWS: Inaccurate Estimated Billing Data – $1.7 billion
741–750 of 793 posts
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#742Earlier quoted context omitted.
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
>no tests? Earlier this week my slopservant implemented several comprehensive changes to a codebase. It also wrote extensive tests to verify the correctness of the changes. A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless. The many tests it had written were just confirming the LLM's idea of correctness…
The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details.
Then it can write tests based on the interface, and that is usually a lot more solid.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#743Earlier quoted context omitted.
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality. You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against get…
This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption". There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job. And no one is shooting for better than OK.
Yeah but this also negates the argument that people gotta use it or they get left behind.
If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#744Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
What apology letters? Never got anything, not even a reply to my support tickets.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#745You folks are completely irresponsible with your finances. I only spent $2.4 million last night. You've got to learn to manage your money.
Yeah, buy less cappuchinos and avocado toast so you can pay your AWS bills people!
I need to drink/eat those too to feel like I'm an AWS ops guy!
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#746Earlier quoted context omitted.
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feel…
I have bad news if you think banks are the pinnacle of cyber security practices...
Maybe some rare few are, but by and large they generally stink.
They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#747I'd talk to your Amazon rep and see if you can get them to agree to $800 million as a compromise. They are usually pretty good at that stuff.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#748Just pay it and then deal with support via a ticket.
;)
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#749Earlier quoted context omitted.
To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough e…
> human-or-LLM-driven error. In a computer system, dont those categories cover pretty much everything except a meteor strike?
What I meant to say is an error from manual account management input— not a human error via code.
Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion
#750Earlier quoted context omitted.
To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough e…
Not everything is an LLM boogeyman. Sometimes a bug is just a bug.
> Not everything is an LLM boogeyman.
Indeed.
>> To me this sounds like a human-or-LLM-driven error.
In fact, half of what’s implicated in that very sentence is not an LLM.