651–660 of 793 posts
Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular). Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
One can almost smell the vibes. This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making. To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
Vibes, son. Nothing else in the world smells like that ... I love the smell of Vibes in the morning.
Earlier quoted context omitted.
I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…
That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing. Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
Most engineers with more than a few years of experience know better than to directly try a stunt like that in production, but they do often get there through painful personal experience. I just wish I hadn't been one of the many that got to suffer from that engineer choosing to learn at that particular moment!
If you owe the bank $100, that's your problem. If you owe the bank $1.7 billion, that's the bank's problem.
Not if you’re Elon Musk
I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.
Wanna bet the description of this job post will be updated by the end of the day? "Software Development Engineer II, AWS Invoicing" https://www.amazon.jobs/de/jobs/10428480/software-developmen... "...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinfo…
Earlier quoted context omitted.
"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then. Worst case I've found was off by 15 orders of magnitude.
`timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`.
Unless the variable type already constrains it (e.g. Duration in Go).
I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.