Live data from Hacker News

A former Uber engineer's disaster story

twitter.com

231–240 of 344 posts

Re: A former Uber engineer's disaster story

#231

Earlier quoted context omitted.

Former Uber engineer/EM here: I worked on the Rider app. The “there are only a few screens” is not true. The app works in 60+ countries, with features shipped in the app that often for a country, and - in rare cases - a city. The app has thousands of scenarios. It speaks to good design that each user thinks the user is there to support their 5 use cases, not showing all the other use cases (that are often regional or…

I'm now wondering why Apple doesn't let you ship different binaries for different regions?

As someone who lives in the US, but (pre-covid at least) regularly travels to the UK (where I am from), I wish regional listing was not allowed _at all_. It's incredibly frustrating to have to carry two phones with different App Store accounts to be able to switch between the UK and US versions of things - Uber is one of the (vanishingly) few that gets this right and why they have consistently been my default choice for rides.

Re: A former Uber engineer's disaster story

#232

Earlier quoted context omitted.

Android was a much easier story, Still had some binary size and method count issues but we were able to overcome them thanks to another brilliant engineer we had in the team in SF who worked on Android.

The Android apk is 56.9 MB for my device, while the iOS app is 331.3 MB. What accounts for the 6x difference?

APK is now more a delivery package format than a real executable format now.

On recent Android devices, the APK wil be recompiled to machine code on the device after a small number of runs (to gather code statistics).

So you are comparing an apple and a robot.

Re: A former Uber engineer's disaster story

#233

Earlier quoted context omitted.

A complete rewrite of a large project is risky anyway. Doing so in a language in which literally no other project of that size exists yet strikes me as insane. I love Swift. But I've been adopting it in bits and pieces over the years. It's not just that the language has changed significantly, you also have to get used to the idioms. E.g. my early Swift code often featured Pyramid of Doom patterns, and both language i…

That's why I call idioms idiots. If you have to learn idioms, it's an idiotic thing.

What do you propose as an alternative?

Re: A former Uber engineer's disaster story

#234
post #209

Earlier quoted context omitted.

Anything else seems like an even bigger mess: What happens when a user who travels unexpectedly discovers they need to download a new version of the app for where they are now? And what if they're on a low-bandwidth or capped cell plan and can't download a 100MB app easily, where 3/4 of the data is duplicates of stuff in their other 5 copies of the app? What happens when a user tries to book an Uber from one area wit…

Initial install bundle should be minimal and only have code that is required everywhere and or for downloading/installing more stuff. It's not about creating 10,000 different permutations of an app, but downloading/installing content on demand. If we are afraid that there might be problem that user later on has not enough bandwidth or issues like that, we'll start immediately downloading/installing after initial bund…

You're not always allowed to do that. E.g. Apple requires all code to either be in the bundle or run in a webview (and even then you risk removal if you introduce "significant" features): https://developer.apple.com/app-store/review/guidelines/#2.5...

For Android purposes: maybe it's allowed in the Play store? I'm not sure tbh. There are certainly other stores with other rules (and Uber is in many of them too), or they could distribute it themselves (but we can see how well that has worked for e.g. Fortnite), but you still need to handle any scenarios where it's required.

Re: A former Uber engineer's disaster story

#235

Earlier quoted context omitted.

Follow-on tweets address this somewhat: Every airport has different pickup locations, every region has different payment rules including cash, etc. Apparently all this location-specific business logic has to be in the app itself and is not downloadable on demand.

There are a lot of ways to frame these problems as interesting algorithms but it doesn't really strike me as something that is troubled by scale. I've worked on similar problems where "everything is a special case" in some vein and it takes a few very clever minds to construct the algorithms and datastructures that can come to a good solution. But that's just it, it's the work of a few engineers and not thousands wri…

A helpful way to think of it is by comparing how Uber and an OS GUI are different (since that's the comparison people are making, anyways)

With an operating system UI, the developer may reasonably know what the UI will look like, what features are present, where they are placed. With Uber, a single person may not know all the rules for a single feature, and the rules change frequently, sometimes in the most obtuse ways. Like when San Francisco decided to ban cars from one of its most important artery streets earlier this year (market st). Now not only you need to indicate to a rider that this is a thing, your UI also needs to to provide walking instructions to the nearest designated pickup spot, and the routing algorithm needs to not put drivers on market st. Oh, and don't forget Hebrew speaking tourists read right-to-left and that food delivery bicycles _are_ still allowed on market st. That's just one change in one city.

Consider that just about anything might change anywhere, any time. From how earning breakdowns are displayed for drivers in Sao Paulo (this differs by jurisdiction), to where the pickup locations are in Pearson airport in Toronto (airport rules change all the time), to what promos apply in Mexico City this week (which depends on marketing and finance analysis), to whether surges should be disabled in response to a terrorist attack, to what documents a driver signup rep should be scanning into the system as local regulations change.

Now build all the infrastructure required considering that the people on the ground reporting their local changes are not programmers, and conversely many of these changes may require making changes to the UIs that these people use, how the new data is aggregated, etc. There are literally thousands of greenlight hub / customer support employees worldwide using these internal tools, many don't speak english and timezones are all over the place. The nature of problems are often fuzzy, hyperlocal-specific (e.g. certain forms of fraud), or extremely meticulous (e.g. legalese, etc).

The trouble of scale comes with having to deal with numerous nebulously defined things from numerous stakeholders and having to distill which engineering teams are even responsible for which aspects of which thing. As the number of stakeholders grow, so does the risk of miscommunication, unclear chain of responsibility and other well-documented large organizations challenges.

Re: A former Uber engineer's disaster story

#236
post #197

Earlier quoted context omitted.

What's wrong with GCD?

It's a great library for a lot of things. When originally introduced it was also heavily oversold by Apple as a magic bullet for concurrency. Here's one account: https://tclementdev.com/posts/what_went_wrong_with_the_libdi...

This page overstates the issues. Almost all the initial GCD design is still good, we just ended up with a lot more smaller low-core and non-SMP processors than expected.

Also most of the performance issues are with excess parallelism (runnable threads), not concurrency (hiding IO latencies).

Re: A former Uber engineer's disaster story

#237
post #231

Earlier quoted context omitted.

I'm now wondering why Apple doesn't let you ship different binaries for different regions?

As someone who lives in the US, but (pre-covid at least) regularly travels to the UK (where I am from), I wish regional listing was not allowed _at all_. It's incredibly frustrating to have to carry two phones with different App Store accounts to be able to switch between the UK and US versions of things - Uber is one of the (vanishingly) few that gets this right and why they have consistently been my default choice…

Maybe iOS could keep on hand the different versions you have, and automatically switch based on your current location? Generalize the "lazy loading" so app developers don't have to deal with the problem themselves

Re: A former Uber engineer's disaster story

#238
post #77

Many engineering horror stories involve an entire system rewrite. My recommendation is to never rewrite a large system. Literally never. If the people proposing the rewrite aren’t capable of adding the features incrementally, they aren’t capable of rewriting the entire system to “easily” support those new features.

This 100%. From the tweets it sounds like the problem they were trying to solve was to make the app more modular so it would scale better to more devs. There were/are well established patterns for doing so in ObjectiveC. The Swift rewrite was not necessary and probably took a very difficult project and turned it into the death march that the OP describes.

They should have solved the architectural problem incrementally without adding a 1.0 language rewrite to the mix and then slowly rebuilt the app using Swift.

This is so obvious to anyone with experience it seems pretty evident that the decision making process there at the time was deeply flawed.

Re: A former Uber engineer's disaster story

#239
post #7

What's this "cellular limit"? Is it some apple specific oddity?

The App Store doesn’t allow downloading apps over a certain size unless connected to wifi.

Note that some of the dropping platforms advice doesn’t really help here, or only helps coincidentally, because the App Store serves you a device-specific app binary.

Re: A former Uber engineer's disaster story

#240

Posting stories as Twitter threads personally for me make me loose all respect for you as an engineer. I mean what in the hell is the abysmal way of reading? How could any person allow themselves to post something like this?

I mean, I’ve seen people post things on Twitter that would make me not hire them[1], but it was usually the actual words and not the way they’re posted.

[1] anyone remember that weird thread about Beau Brummell inventing toxic masculinity?

Post reply on HN