Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody got fired for Uber's $8M ledger mistake?

news.alvaroduran.com

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Nobody got fired for Uber's $8M ledger mistake?

#61

Post by infecto »

Aurornis wrote: post #24

> A redesign that gets replaced 2 years later is a catastrophe. > Somebody Should Have Been Fired For This This person is not a good resource. Uber was a very fast growing company, both in terms of their product and staff. Turnover in architecture happens. Calling this a catastrophe and click baiting about firing engineers over a rounding error in Uber’s overall finances is gross. I understand this person is trying t…

It’s a terrible article. They are angry on spending $687 a day for DDb writes when they company was doing billions in revenue. It’s silly and they have a grounded truth on how much systems cost at scale.

Re: Nobody got fired for Uber's $8M ledger mistake?

#62

Post by stackskipton »

blenderob wrote: post #45

> With each trip generating multiple ledger entries, and Uber as a whole processing 15 million trips per day, it didn’t matter that DynamoDB was great because of high throughput at global scale. The proverbial bean counter should’ve stopped this madness from happening. > At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s bl…

SRE here. Most of time we see choices like this because teams are under pressure to deliver and scale would likely exceed what a database will easily handle with the out of the box settings. So tweaking is required and that takes time/knowledge that Dev team doesn't have. AI helps a bit here but it didn't exist when DynamoDB solution was chosen. However, some terraform, and boom, scalable database created, only downside is the cost which is next Product Manager problem.

Re: Nobody got fired for Uber's $8M ledger mistake?

#63

Post by vkou »

simonw wrote: post #2

Firing people for bad architectural decisions is generally a terrible idea - especially decisions that shipped and ran in production for several years. This article also doesn't make a convincing case for this being a huge mistake. Companies like Uber change their architectural decisions while they scale all the time. Provided it didn't kill the company stuff like this becomes part of the story of how they got to whe…

Meanwhile, in a sibling thread about an accounting mistake in California, everyone is screaming for blood.

Blame-free post-mortems are for me and mine, everyone else can get fucked.

Re: Nobody got fired for Uber's $8M ledger mistake?

#64

Post by stackskipton »

skizm wrote: post #16

In general is there any practical way to fix the issue of "Every rewrite was someone's promotion project"? There doesn't seem to be any incentive for employees to care about projects long term. Keeping something running smoothly is never rewarded the same as launching something new or fixing something broken.

100% change the way American Tech companies treat their workforce?

Most people don't get meaningful raises at existing jobs so if they want raises, they must job hop or internally job switch.

Companies will layoff at drop of the hat so you have to make sure your skillset is up to date so you can get next job.

So everyone is launching big splashy projects so they put on their resume to protect themselves in case of layoffs or turn into a promotion.

Re: Nobody got fired for Uber's $8M ledger mistake?

#65

Post by BeetleB »

otherme123 wrote: post #21
embedding-shape wrote: post #7

Earlier quoted context omitted.

> Firing people for bad architectural decisions is generally a terrible idea I mean, if we're considering factors that could make fire a developer, suggesting, pushing and eventually failing to implement bad designs and architectures probably ranks among some of the more reasonable reasons for firing them. It doesn't seem to have been "Oops we used MariaDB when we should have used MySQL" but more like "We made a bad…

> But on the other hand, I'd probably fire the manager/executive responsible for that move, rather than the individual developer who probably suggested it. And you just teached all your workers to be as cautious as being freezed, never be proactive, keep the status quo as much as they can, avoid being noticed, and never take a step without being forced or having someone else to take 100% blame (with paper trail) if t…

And in some workplaces, that actually is the way to go!

I once worked in a manufacturing environment where mistakes could be quite expensive. We had our annual org survey and one of the questions asked was "Risk taking is encouraged." Our team scored low on that metric, and upper management was concerned. They held a meeting to ask about it, and most of the team was confused why there was a meeting. They said they viewed it as a positive that they don't take risks.

Re: Nobody got fired for Uber's $8M ledger mistake?

#66

Post by darth_avocado »

websap wrote: post #15

> A redesign that gets replaced 2 years later is a catastrophe. People forget how quickly Uber scaled, and the user impact of not being able to track your trips could be catastrophic to retention. There's a class of tech-influencer who think they can dissect past decisions on a blog post without being in the room when the technical constraints were being laid out. This is Monday morning quaterbacking at it's most gro…

> There's a class of tech-influencer who think they can dissect past decisions on a blog post without being in the room when the technical constraints were being laid out

The cost they are laying out are not that prohibitively expensive. I’ve known corporations where people spin up test clusters that cost 5K a month and forget about it. A business critical service can definitely ignore costs in the short term if they bring in customers. The standard practice is to just ship something quickly and optimize for the cost later if it helps bring in revenue/customers.

Besides, the napkin math isn’t always true. If you’re an enterprise customer for AWS, you get massive discounts, especially in the time frame they’re talking about. And when it comes to partnerships, I remember back in the day AWS used to let you do pretty much anything for free if it meant they could parade your project to other customers.

Re: Nobody got fired for Uber's $8M ledger mistake?

#67

Post by x0x0 »

kshacker wrote: post #44

Can I do a mea culpa? This is more than 3 decades back. I was a junior programmer (2-3 years in industry) sent to a client site in europe. You can imagine the state of systems those days. I wrote (or rather updated) a fix which would updated the discount and tax rates on orders based on new terms. It would run every day to account for ... whatever. You pick the values from master file, update all the orders and move…

If a single non-malicious code change can break a thing like that with nobody noticing that's a catastrophic failure in testing, QA, and operations (nobody noticed 20% of euros transacted just stopped). It's hard to blame that on IC engineer.

Re: Nobody got fired for Uber's $8M ledger mistake?

#68

Post by Esophagus4 »

blenderob wrote: post #45

> With each trip generating multiple ledger entries, and Uber as a whole processing 15 million trips per day, it didn’t matter that DynamoDB was great because of high throughput at global scale. The proverbial bean counter should’ve stopped this madness from happening. > At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s bl…

I have been a part of over-built solutions.

It’s usually because executive management bakes hyper growth into the assumptions because they really want the biz to grow, then it becomes marching orders down the chain as it gets misinterpreted in a game of corporate telephone.

“We need to design this for 1b DAUs”

Then 1) that growth never happens and 2) you end up with a super complicated solution

Instead, someone needs to say, “Hey [boss], are you sure we need to build for 1b DAUs? Why don’t we build for 50m first, then make sure it’s extensible enough to keep improving with growth”

Re: Nobody got fired for Uber's $8M ledger mistake?

#69

Post by otherme123 »

embedding-shape wrote: post #57
pjc50 wrote: post #53

Earlier quoted context omitted.

No, he's right. Remember you can conduct only one of the two different types of postmortem, the air crash style blameless one (to find out what happened) and the blame-based one (to find out who to punish). Once you conduct the latter, everyone psychologically "lawyers up". You get a lot more meetings. A lot more paper trail. A lot more delay. You don't just pick a database, you commission a sub-committee for databas…

Right, so say we have this situation where you're choosing a SQL database. The organization made a choice that leads to lots of complications, where often times the reason for the complication is because the organization made yet another bad choice. Repeat a couple of times. We do a blameless postmortem about each one of these, where essentially we only focus on the root causes of the actual problems, but somehow it…

>was one individual who made those bad choices over and over

This was never said, or even implied, in the article. We don't even know if this was a single person choice.

You are making up "facts" like calling the person who makes mistakes "toxic", or saying that the choice was made by someone who only made bad choices.

We are talking Uber here, in 2017, which was not only playing "move fast and break things" but "move really fast while shooting an AK47 blindfolded". Not only they expected mistakes, but they encouraged them. It would be plain wrong to start firing individual people for making mistakes if that is the environment.

Re: Nobody got fired for Uber's $8M ledger mistake?

#70

Post by conductr »

alemanek wrote: post #37
simonw wrote: post #2

Firing people for bad architectural decisions is generally a terrible idea - especially decisions that shipped and ran in production for several years. This article also doesn't make a convincing case for this being a huge mistake. Companies like Uber change their architectural decisions while they scale all the time. Provided it didn't kill the company stuff like this becomes part of the story of how they got to whe…

Also the article doesn’t attempt to explore the business and resourcing constraints they were operating under at the time. I have been in situations where I was told “don’t worry about cost just get it done”. Then a few years later the business constraints shift and now we need to “worry about the cost”. It ignores that decisions made under a different set of constraints were correct, or at least reasonable, at the t…

Also totally possible that it was just an unpublished partnership of sorts between AWS and Uber. AWS wants the logo and a big case study implementation to give the product some credibility or a boost. Uber may not have been charged at all, may have even been paid to use AWS. The Uber developer may not have even known, just was given an edict to build it on dynamodb.
Post reply on HN