711–720 of 793 posts
That’s when I logged in. That’s when I saw the 1.3 billion number.
I was pretty fast in getting that backdoor closed and hunting down that leak, I can tell you! :D
Earlier quoted context omitted.
I wonder if AWS billing still uses CSV files for passing data around. IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the product…
That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing. Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
Earlier quoted context omitted.
> Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was. If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).
You certainly don't want your service caring about a test currency. Nobody is pricing their services in test currency They are either pricing in $ or in "units" and then converting upstream This is an even bigger foot gun than what's happening here
So how is it a "foot gun" to add XTS as an additional currency, for internal use only?
I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes another exchange rate. You could even fix it as 1 XTS = 1 USD, although choosing a different exchange rate than parity is likely to surface more bugs.
Earlier quoted context omitted.
> take the 3am on-call and be the hero instead Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?" I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
Earlier quoted context omitted.
It was the mid 2010s when I sensed a lot of SaaS becoming popular. Just host your ticketing systems, your IT management planes, your security management consoles, your SOC, all off-premises. I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
One thing that you need to understand is that the usual business manager absolutely hates depending on technical expertise, and that the modern corporate world is fanatically anti-intellectual. Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares. Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!…
Earlier quoted context omitted.
Why? What's the damages? They showed you a wrong number, then later acknowledged it and fixed it. Just because the number was "very big" to you doesn't mean you were actually aggrieved in some way.
Big numbers can lead to stress which can lead to all kinds of disorders.
Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after. Services emit metering values that arent directly tied…
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators? 1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
A few hours later, I got a mail saying the estimate was wrong.
Now, I cannot create an AWS account using my email address because you cannot create a new account using the same email address…