Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: Inaccurate Estimated Billing Data – $1.7 billion

news.ycombinator.com

Post reply on HN
Print view View original

761–770 of 793 posts

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#761

Post by AjeetTester2026 »

CobrastanJorji wrote: post #589
01284a7e wrote: post #466

Earlier quoted context omitted.

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, te…

[flagged]

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#762

Post by Twirrim »

berkes wrote: post #735
Twirrim wrote: post #575

Earlier quoted context omitted.

I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…

Give that engineer a raise and an official title of "chaos monkey". On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering. Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up,…

I got known as a "Chaos Monkey" at AWS, and have sort of carried on that title. It's not exactly accurate, but tech stuff just breaks around me, and it's very, very rarely my fault. I'm pretty cautious and resist any urge to "I wonder what happens..." with anything that could possibly have an effect beyond me.

The "Chaos Monkey" effect at AWS was so pronounced you could literally see on our sev2 count graph when it was my on-call week, because I'd get dramatically more sev2s than any other engineer.

The service I worked for at AWS was amazingly stable and reliable. A large majority of the alarms that ever fired, fired because of problems with another service we depended on in some way. This billing thing is a good example. I was on-call when there was a major SQS outage in a region, through a DynamoDB outage, S3 outages, all sorts of stuff. We used to joke that it would probably be a net benefit for AWS if I wasn't on-call, just so my spooky-at-a-distance wouldn't happen.

I eventually lost my "most sev2s in a week" record toward the end of my time there when someone else was on-call and DynamoDB had a major outage. DynamoDB held all of our metadata so everything broke and every alarm we had fired over the space of a few hours.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#763

Post by NBJack »

mendigou wrote: post #533
tyre wrote: post #492

Earlier quoted context omitted.

Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.

I am former AWS and this is pretty accurate. The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).

[dead]

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#764

Post by NBJack »

akdev1l wrote: post #621
mendigou wrote: post #533

Earlier quoted context omitted.

I am former AWS and this is pretty accurate. The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).

Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways

And many of which have been shipped across the pond to be someone else's problem.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#765

Post by NBJack »

lionkor wrote: post #742
andai wrote: post #642

Earlier quoted context omitted.

>no tests? Earlier this week my slopservant implemented several comprehensive changes to a codebase. It also wrote extensive tests to verify the correctness of the changes. A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless. The many tests it had written were just confirming the LLM's idea of correctness…

The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details. Then it can write tests based on the interface, and that is usually a lot more solid.

That's yet another band aid on a hemorrhaging patient. And it still assumes the agent(s) are capable of correct comprehension of the purpose.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#766

Post by lionkor »

NBJack wrote: post #765
lionkor wrote: post #742

Earlier quoted context omitted.

The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details. Then it can write tests based on the interface, and that is usually a lot more solid.

That's yet another band aid on a hemorrhaging patient. And it still assumes the agent(s) are capable of correct comprehension of the purpose.

Yeah for sure it's a bandaid fix, but it does produce tests that fail if the interface doesn't match the implementation, more often than not doing this does (in my experience).

With a skill like this, which I have written, the LLM will end a session with "wrote 24 tests, 5 failing due to API contract mismatch" or something, and leave it at that. I can then decide if that's a bad interface or bad implementation.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#767

Post by slowdog »

9dev wrote: post #712
anvuong wrote: post #601

Earlier quoted context omitted.

That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing. Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

Eh… If I can bring prod down simply by sending CSV with a comma in a field to a prod endpoint, whoever has come up with the most naive CSV parser possible running in prod is responsible for the outage.

100% agree with this, seems weird to call an engineer “smart” for triggering an API that allows any input but can’t handle it. I also kind of doubt it was an engineer testing vs some much more subtle issue as billing is in line for everything.

If this was 10 years ago, this was also the era when any real customer could write any ARN (resource identifier) they wanted including commas due to no expected standardization across services. This could have been trivially triggered by a customer.

Just a matter of time if no one is escaping commas in billing files.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#768

Post by dawnerd »

csomar wrote: post #594
dawnerd wrote: post #269

Earlier quoted context omitted.

That’s why you always use a spend limited card with variable cost providers.

Most debit/prepaid cards will get rejected. Credit Cards technically have a limit but they really don't. It's an open line to your finances.

I’ve never had a limited card get rejected. Granted I use ones from privacy.com. That’s also what I have hooked up to my aws / amazon.

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#769

Post by lstodd »

inigyou wrote: post #496
dymk wrote: post #443

Earlier quoted context omitted.

…for emotional damage?

If you were a business maybe you could claim for the emergency on-call time spent diagnosing, but you'd probably still lose AND amazon would fire you as a customer.

"fire you as a customer"

I love this framing

Re: AWS: Inaccurate Estimated Billing Data – $1.7 billion

#770

Post by lstodd »

everforward wrote: post #419
ardacinar wrote: post #332

Earlier quoted context omitted.

Well, for my case, I was paying $0 (Exactly, I managed to hunt down and delete every last resource in my account a few months ago). It was displaying $430 million for me. I don't think that is 0*2^30.

Huh, that is odd. Working backwards, that would be ~ $0.40 originally. Wonder if that’s also flat out wrong or if they’re doing some kind of currency handling that breaks when you start dealing with huge multipliers.

using floats for currency? that would be grander even than this fuckup.
Post reply on HN