Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody got fired for Uber's $8M ledger mistake?

news.alvaroduran.com

Post reply on HN
Print view View original

41–50 of 86 posts

Re: Nobody got fired for Uber's $8M ledger mistake?

#41

Post by Aurornis »

noodlesUK wrote: post #39
Aurornis wrote: post #27

Earlier quoted context omitted.

Letting interns carry six figure equipment, which would also be unexpectedly heavy especially if this happened some years ago, would be a weird thing for any lab I’ve worked in. There are too many things that can predictably go wrong in the hands of an inexperienced person, as happened here. Interns wouldn’t even be allowed to use $100K VNAs without a lot of supervision because so many things can go wrong. Damaging o…

I'm not sure - the level of scrutiny that usage/abusage of expensive equipment gets varies wildly from organisation to organisation. I've worked in some places where very expensive equipment is handled roughly, or even taken home in some cases. In others, there are meticulous procedures for even $1-5k pieces of equipment. It's just a cultural thing.

For this example it’s the delicacy and fragility of the instrument, the price is just a proxy for that.

Expensive VNAs are also precision, calibrated instruments with small connectors that can easily be degraded by even simple misuse. Frontends destroyed or subtly damaged in ways that break measurements by allowing the wrong signal to enter.

It’s easy to damage one in a way that will interfere with measurements for months before someone realizes what’s wrong, which is more costly than the VNA itself.

These instruments require training to handle. It’s not even about the price, it’s absurd that they’d let an intern carry one around at all (if it was allowed)

This is like the hardware equivalent of an intern accidentally dropping the production DB. My first question would be how they got to the point where an intern was in a position to be able to drop the production DB because everyone understands what can go wrong

Re: Nobody got fired for Uber's $8M ledger mistake?

#42

Post by noodlesUK »

Aurornis wrote: post #41
noodlesUK wrote: post #39

Earlier quoted context omitted.

I'm not sure - the level of scrutiny that usage/abusage of expensive equipment gets varies wildly from organisation to organisation. I've worked in some places where very expensive equipment is handled roughly, or even taken home in some cases. In others, there are meticulous procedures for even $1-5k pieces of equipment. It's just a cultural thing.

For this example it’s the delicacy and fragility of the instrument, the price is just a proxy for that. Expensive VNAs are also precision, calibrated instruments with small connectors that can easily be degraded by even simple misuse. Frontends destroyed or subtly damaged in ways that break measurements by allowing the wrong signal to enter. It’s easy to damage one in a way that will interfere with measurements for m…

Fair enough. Fragility is probably more important than price in this scenario.

Re: Nobody got fired for Uber's $8M ledger mistake?

#43

Post by kayodelycaon »

Who exactly is supposed to be fired for this?

If you don’t have price controls, it’s easy to run up a bill.

If no single person had the responsibility to check the cost, then no one actually failed at their assigned job. So you either fix the system or fire everyone involved in the decision.

What you’re doing now is looking a scapegoat to beat up. You’re angry and you’re going to make someone pay for pissing you off.

Re: Nobody got fired for Uber's $8M ledger mistake?

#44

Post by kshacker »

Can I do a mea culpa? This is more than 3 decades back. I was a junior programmer (2-3 years in industry) sent to a client site in europe. You can imagine the state of systems those days. I wrote (or rather updated) a fix which would updated the discount and tax rates on orders based on new terms. It would run every day to account for ... whatever. You pick the values from master file, update all the orders and move on.

I wiped out VAT on all orders and for the next month the paper invoices were sent without VAT. So the invoice is $100, VAT is $20, the invoice should be $120, but they were sent as $100.

100s of invoices every day would be my guess.

Nobody noticed.

For a month.

Millions of dollars of revenue and IIRC millions of dollars of VAT.

Until a customer complained to the CEO.

We had a firefight to fix it, not just technical but legal and managerial. We can send a new invoice just for tax. We can redo the invoice. We can send a debit memo. What is the right decision? But what if customers does not pay? What about returns? How will we track returns? Of course we were doing the technical solutions and the client company was front-ending how to handle it business wise.

And the managerial firefight - who did it, what are the safeguards in future? We had a company exec visit the client site to manage the issue.

I was in the hot seat but I was protected by my managers from any fallout. Just do the work. Do not screw up again. (Test every row every column even if you did not change it)

A month later the sales director at the client company got fired.

The grapevine is that this was just the tipping point, but you never know. BTW these were paper invoices printed onsite and mailed out, but I do not know if someone had the job to scrutinize them.

PS: True story, going by old memory, although such legends remain fresh in your mind, forever. Not sure it belongs here, but the mention of firing for a multi-million dollar mistake pulled this into cache memory.

Re: Nobody got fired for Uber's $8M ledger mistake?

#45

Post by blenderob »

> With each trip generating multiple ledger entries, and Uber as a whole processing 15 million trips per day, it didn’t matter that DynamoDB was great because of high throughput at global scale. The proverbial bean counter should’ve stopped this madness from happening.

> At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s blobstore, TerraBlob, for older data (i.e., cold data). TerraBlob is similar to AWS S3. For a long-term solution, we wanted to use LSG.

Honest question. Why do people go for this kind of complicated solution? Wouldn't Postgres work? Let's say each trip creates 10 ledger entries. Let's say those are 10 transactions. So 150 million transactions in a day. That's like 2000 TPS. Postgres can handle that, can't it?

If regional replication or global availability is the problem, I've to ask. Why does it matter? For something so critical like ledger, does it hurt to make the user wait a few 100 milliseconds if that means you can have a simple and robust ledger service?

I honestly want to know what others think about this.

Re: Nobody got fired for Uber's $8M ledger mistake?

#46

Post by data-ottawa »

otherme123 wrote: post #21
embedding-shape wrote: post #7

Earlier quoted context omitted.

> Firing people for bad architectural decisions is generally a terrible idea I mean, if we're considering factors that could make fire a developer, suggesting, pushing and eventually failing to implement bad designs and architectures probably ranks among some of the more reasonable reasons for firing them. It doesn't seem to have been "Oops we used MariaDB when we should have used MySQL" but more like "We made a bad…

> But on the other hand, I'd probably fire the manager/executive responsible for that move, rather than the individual developer who probably suggested it. And you just teached all your workers to be as cautious as being freezed, never be proactive, keep the status quo as much as they can, avoid being noticed, and never take a step without being forced or having someone else to take 100% blame (with paper trail) if t…

One of my favourite bosses ever was a VP who kept a bankers box at her desk and very few personal affects.

She told me she kept it there because her job was to make decisions and get fired or leave if she was wrong. She was right about so many of her choices, I would have followed her into anything. Then one day I came in and her desk was empty -- she had an apparently epic argument with the C suite and disagreed with their path so she left (never found out if that was a quit or fired). The team got a new VP, but I requested to be moved to a different team as I wasn't aligned with the new vision.

When you get to a certain level part of your job becomes owning the decisions and getting fired.

Re: Nobody got fired for Uber's $8M ledger mistake?

#47

Post by hilariously »

pc86 wrote: post #18
hilariously wrote: post #6

Earlier quoted context omitted.

Do you think that the social climbers who approved these obviously crappy projects learned anything? I have worked with all levels of engineers who come into a project glassy eyed about some technology, sure, but if you are part of the team approving a project and you cant produce a realistic budget then your management is bogus as hell. I have worked on a ton of these vanity projects, and when I voice my concerns it…

> Do you think that the social climbers who approved these obviously crappy projects learned anything? Just the framing of this question makes it seem like you simply don't like people in management / decision-makers, and you want something bad to happen to them. Maybe that's wrong, hopefully it is, but the rest of the comment doesn't do much to dissuade me of that impression either.

Something bad to happen to "them"? There's no diaphanous them, just the specific social climbing crap decision makers facing no consequences of any type.

I have worked with many hard working and caring managers, and they are generally eclipsed by said social climbers presenting at conferences every other week about know-nothing topics jumping from place to place leaving bankrupt companies and massive layoffs in their wake.

I see them posting on LI right now :)

Re: Nobody got fired for Uber's $8M ledger mistake?

#48

Post by pjc50 »

simonw wrote: post #2

Firing people for bad architectural decisions is generally a terrible idea - especially decisions that shipped and ran in production for several years. This article also doesn't make a convincing case for this being a huge mistake. Companies like Uber change their architectural decisions while they scale all the time. Provided it didn't kill the company stuff like this becomes part of the story of how they got to whe…

It probably was an unnecessary redesign that could have been avoided, but hey: at least it worked, and eight million dollars is not a huge amount for Uber.

Birmingham spent almost £150m for a system that didn't work at all:

https://www.theregister.com/2026/01/29/birmingham_oracle_lat...

While I was an undergraduate, my university also spent £9m on accounting that didn't work, also with Oracle: http://news.bbc.co.uk/1/hi/education/1634558.stm

If you've designed a system in house for your accounting, it works, makes neither financial nor software errors, is accepted by the users, and got away with it costing a relatively small fraction of your turnover? That's a big win.

Re: Nobody got fired for Uber's $8M ledger mistake?

#49

Post by jakevoytko »

alemanek wrote: post #37
simonw wrote: post #2

Firing people for bad architectural decisions is generally a terrible idea - especially decisions that shipped and ran in production for several years. This article also doesn't make a convincing case for this being a huge mistake. Companies like Uber change their architectural decisions while they scale all the time. Provided it didn't kill the company stuff like this becomes part of the story of how they got to whe…

Also the article doesn’t attempt to explore the business and resourcing constraints they were operating under at the time. I have been in situations where I was told “don’t worry about cost just get it done”. Then a few years later the business constraints shift and now we need to “worry about the cost”. It ignores that decisions made under a different set of constraints were correct, or at least reasonable, at the t…

Yeah, this is exactly what I thought when I read this post. It seemed like the author either hasn't worked in big tech, or hasn't worked in the industry very long. It's extremely likely that the engineer who designed this was standing on his desk shouting "it's going to cost THIS MUCH MONEY. I want to make sure that EVERYONE IS OK WITH THIS." and was met with shrugs.

Here's how a big tech reporting chain sees this situation when everything is smooth sailing: "We're growing 3x year-over-year? After 2 years, the cost will be an order of magnitude higher no matter what solution we pick. The constant factor doesn't matter that much. But we have such an incredible roadmap that we will book more than an order of magnitude of revenue, backed by this new ledger project. The cost will always be a nonissue because of growth."

And then 2 years go by, and this incredible product growth adds a bunch of ledger entries that weren't there 2 years ago, someone nudges your reporting chain with the question, "this is pretty expensive.. what gives?" and then someone with a good combination of social and technical skills points out that a migration to your existing storage solution would be a cost effective way to continue growing.

At every step of the way, everyone is generally happy with what's going on.

Re: Nobody got fired for Uber's $8M ledger mistake?

#50

Post by arethuza »

pjc50 wrote: post #48
simonw wrote: post #2

Firing people for bad architectural decisions is generally a terrible idea - especially decisions that shipped and ran in production for several years. This article also doesn't make a convincing case for this being a huge mistake. Companies like Uber change their architectural decisions while they scale all the time. Provided it didn't kill the company stuff like this becomes part of the story of how they got to whe…

It probably was an unnecessary redesign that could have been avoided, but hey: at least it worked, and eight million dollars is not a huge amount for Uber. Birmingham spent almost £150m for a system that didn't work at all: https://www.theregister.com/2026/01/29/birmingham_oracle_lat... While I was an undergraduate, my university also spent £9m on accounting that didn't work, also with Oracle: http://news.bbc.co.uk/1…

ERP implementations probably don't fail for those kind of architectural reasons?
Post reply on HN