A former Uber engineer's disaster story • Hacker News Board

Re: A former Uber engineer's disaster story

#331

Post by saagarjha » Fri, Dec 11, 2020, 6:19 PM UTC

jimmydorry wrote: post #329

saagarjha wrote: post #153

Earlier quoted context omitted.

The Apple Maps app on iPhone, for example? If you make it fair and count the code that goes into the frameworks specifically for it, it's still tens of megabytes at the most. There is a huge part of it running server-side to support it, of course, but it is difficult to say that it is not a complicated app. Or consider the Mail app? It needs to deal with IMAP, and has custom flows for a number of named mail services…

10s of megabytes? Where did you pull that number from? Since it is a system app, its true install size is hidden away from users. The comparable google maps app is 190 MB.

I counted up the system frameworks it relies on that can be reasonably considered to exist for the purposes of serving the app.

Re: A former Uber engineer's disaster story

#332

Post by mcguire » Fri, Dec 11, 2020, 7:15 PM UTC

mewpmewp2 wrote: post #207

pfortuny wrote: post #185

Earlier quoted context omitted.

Not as per the long twitter thread.

Well... It should be.

Evaluating map data inaccuracy? More than just a payment UI? Driver/passenger matching? Data analytics? (Doesn't that require information from multiple clients?) On the client?

I mean, you could push some of that to the client, but I'd understand why your app is gigantic and doesn't do app-ly job very well with all the background processing. And I'd be surprised the app hasn't been hacked to get at the analytics data, if not the payments.

Re: A former Uber engineer's disaster story

#333

Post by Groxx » Fri, Dec 11, 2020, 8:08 PM UTC

mewpmewp2 wrote: post #309

lhorie wrote: post #241

Earlier quoted context omitted.

There's also something to be said about supporting 60+ versions of a client...

Not following that thoughtline?

By downloading dynamic code, in effect you're running N (or some combinatorial number of) different apps.

Granted, you can/should strongly isolate them so it's infeasible for one module to interfere with another. But as always, YMMV in practice. And good luck testing them all either way.

Re: A former Uber engineer's disaster story

#334

Post by Groxx » Fri, Dec 11, 2020, 8:14 PM UTC

mewpmewp2 wrote: post #308

Groxx wrote: post #234

Earlier quoted context omitted.

You're not always allowed to do that. E.g. Apple requires all code to either be in the bundle or run in a webview (and even then you risk removal if you introduce "significant" features): https://developer.apple.com/app-store/review/guidelines/#2.5... For Android purposes: maybe it's allowed in the Play store? I'm not sure tbh. There are certainly other stores with other rules (and Uber is in many of them too), or th…

Sounds like base things, common for everything should be done using native app code and everything that varies that much should be made as PWA in WebView. You can download this PWA in the background too once app has installed if you are afraid you might not be able to do this on demand. I don't see why not use PWA? The performance implications in this case are minimal imo. And you have all the other fully fledged nat…

PWA won't give you access to random device APIs that you didn't open up for a specific use before needing them, which e.g. cameras, payment systems, etc often need. All that still needs to be native, and that can be substantial in some cases.

If it does, say you made a generic shim, you're effectively downloading and executing dynamic code. Maybe not by a technical definition, but that's not the point of Apple's rule, nor will they care about technical arguments when they reject it. There are plenty of examples of this happening in practice, I'm not talking hypothetically - it's why they have a carve-out explicitly for education apps. Even if you don't do stuff like that, and stay entirely in the webview, PWAs (unless it's actually being run in the browser, of course) are not allowed to make significant changes without going back through review.

edit: you can absolutely shrink your binary size with a PWA or similar though, yeah. There will definitely be other tradeoffs you have to make though, I'm not deeply familiar with those. Performance and integration with native libraries at the very least (e.g. map rendering).

Re: A former Uber engineer's disaster story

#335

Post by rsp1984 » Fri, Dec 11, 2020, 9:21 PM UTC

StanTwinB wrote: post #293

rsp1984 wrote: post #248

The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discov…

It’s interesting to me that you came away from this thinking that I didn’t realize the eng size growth was the problem. I specifically said in the third tweet that hypergrowth was the problem. I even went on to say why it was a problem. The entire point of including the details about the org growth was specifically to draw you to that conclusion.

OK, I understand. I didn't mean to imply that you didn't see the problem of hypergrowth and I apologize if it came across this way. However, in the sequence of tweets it does sound like the conclusion / lesson is to throw ever more heroic engineering at the problem or to get better at eng. planning, as opposed to solving the organizational issues and messed up incentive structures at the root of it all.

I realize this isn't a small ask. I've worked in BigTech myself where the reward / risk ratio of trying to actually solve problems at the root is basically 0 if you're not SVP or higher. Anyway, I enjoyed the tweets nevertheless. Good luck.

Re: A former Uber engineer's disaster story

#336

Post by sigotirandolas » Fri, Dec 11, 2020, 9:36 PM UTC

lock-free wrote: post #221

dntrkv wrote: post #194

Earlier quoted context omitted.

Take look at this thread: https://news.ycombinator.com/item?id=25376346 My point is that there is 100s of screens/features in the Uber app that you don't even know exist. And you have those same 100s of screens across their other apps. Multiplied across all of the different platforms. Plus any internal tools they use. I'm curious, have you worked at a software company of Uber's scale? Not trying to be a dick here, ge…

My whole point here is I don't understand why "Uber scale" is so large. I have worked in large organizations - smaller than Uber, but we did more stuff (as in more complexity and diversity in products - shipping software and the hardware to run it, for example). "100s of screens" is as meaningless to me as "thousands of lines of code" and wholly uninteresting because it seems like the front end is the least significa…

I'm just a layman, but I often ponder this same "Why are some companies so big when a small team could do about the same thing?" and my mental theory currently is:

1. The amount of engineers scales very badly with respect to application complexity. I.e. you may need 100 or 1000 engineers to do something that "looks" about 2x as complex as something 10 engineers can do, and

2. The more complex solution tends to win in the market because on the one hand people value the application handling edge cases better, having shiny graphics, having slightly better UX, etc. and on the other hand you redeem the cost of paying those 1000 engineers thanks to software being free to copy and network effects / monopoly.

Re: A former Uber engineer's disaster story

#337

Post by StanTwinB » Sat, Dec 12, 2020, 1:53 AM UTC

rsp1984 wrote: post #335

StanTwinB wrote: post #293

Earlier quoted context omitted.

It’s interesting to me that you came away from this thinking that I didn’t realize the eng size growth was the problem. I specifically said in the third tweet that hypergrowth was the problem. I even went on to say why it was a problem. The entire point of including the details about the org growth was specifically to draw you to that conclusion.

OK, I understand. I didn't mean to imply that you didn't see the problem of hypergrowth and I apologize if it came across this way. However, in the sequence of tweets it does sound like the conclusion / lesson is to throw ever more heroic engineering at the problem or to get better at eng. planning, as opposed to solving the organizational issues and messed up incentive structures at the root of it all. I realize thi…

It’s interesting that you saw this was a tale of heroics. I view it as a tale of bad choices and organization failure.

Re: A former Uber engineer's disaster story

#338

Post by georgiecasey » Sat, Dec 12, 2020, 1:27 PM UTC

11thEarlOfMar wrote: post #69

I'm reminded of the story of Windows NT and how David Cutler commented at one point after the release that divorces and nervous breakdowns were tracked in the program metrics. I expect that's hyperbole, but there is a real cost to these death marches. I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the…

I started reading the intro to that Show Stopper book and can't stop. Great stuff, thanks

Re: A former Uber engineer's disaster story

#339

Post by georgiecasey » Sat, Dec 12, 2020, 1:59 PM UTC

Apocryphon wrote: post #166

This story at Uber is a fascinating example of the growing pains involved with transitioning to a new language. LinkedIn adopted Swift back in 1.2 . I would love to read some war stories of how things there have progressed over the years. Source: https://www.youtube.com/watch?v=X9waDi787uo (14:15-15:30, 39:26-40:37) At an interview with them a few years ago, I was told they had been moving towards rewriting the app i…

the difference is people actually need to use the Uber app /s

Re: A former Uber engineer's disaster story

#340

Post by georgiecasey » Sat, Dec 12, 2020, 2:17 PM UTC

overgard wrote: post #57

I know C++ has its warts, but almost every mobile project I've worked on has been C++ with a thin layer of objective c and java interop. IMO it's really not worth it to get so tied up in a single companies tooling even if the developer experience is superficially nicer. C++ has all sorts of awful, but it's familiar awful with mature tooling. Admittedly the things I've worked on has been games and CAD systems so the i…

The vast majority of apps are UI, CRUD, business logic and other things where native code (as in compiled C/C++) is just overkill and makes things more complicated. The few exceptions seem to be the fields you listed.