Live data from Hacker News

A former Uber engineer's disaster story

twitter.com

171–180 of 344 posts

Re: A former Uber engineer's disaster story

#171

I started to write a whole big thing here because I found this whole thing pretty disgusting but it's not worth it. > I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out. > With only a week left we decided eat the 8 figures and drop support for iOS 8.…

One wonders if this is common everywhere at this scale. Take Facebook's massively bloated iOS app:

https://www.reddit.com/r/iOSProgramming/comments/6upeu6/how_...

https://news.ycombinator.com/item?id=14903228

And their Android app patching Dalvik at runtime to deal with the huge number of Java methods:

https://www.facebook.com/notes/facebook-engineering/under-th...

And they're not the only ones known for bloated apps, either. Take Amazon's apps for instance:

https://news.ycombinator.com/item?id=14902875

Re: A former Uber engineer's disaster story

#172
post #60

This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens. I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system th…

Former Uber engineer/EM here: I worked on the Rider app. The “there are only a few screens” is not true. The app works in 60+ countries, with features shipped in the app that often for a country, and - in rare cases - a city. The app has thousands of scenarios. It speaks to good design that each user thinks the user is there to support their 5 use cases, not showing all the other use cases (that are often regional or…

I also worked for Uber but on backend systems in the payments flow. Every single piece of app functionality mentioned above has at least one, if not multiple, backend systems and teams supporting it.

Like in the app, "payments" sounds like it would be a simple Stripe integration but it isn't. For one, that's way too expensive at Uber's scale. They build their own payment integrations to save fractions of a point that adds up to tens of millions of dollars in saved expenses.

There's real time systems to facilitate the trip, then the order processing systems to process that raw data, hydrate it, store it in multiple data warehouses, and publish it to Kafka to be consumed by literally hundreds of other services. The main money pipeline involved many services covering things such as rider incentives, driver incentives, rider fraud, driver fraud, tax calculations, rider payments across all of those aforementioned payment methods, receipts, invoices, driver disbursements, rider support, driver support, and more.

Then take in to account that almost all of those services were written for a world designed only for taking passengers from point A to point B with Uber Black but have huge added bloat to accommodate:

* Uber X

* Uber Pool

* Uber Eats

* Uber Freight

* Uber Moto

* Lime

* Jump

and proof of concept work for all sorts of new business ventures like Uber Elevate and ATG. All of that cruft is expensive to maintain and all of those actual product lines started as experiments.

When I left in 2019, there were two versions of the vast majority of those systems in production as all were being rewritten to handle the significant changes to Uber's business model from inception to now.

Additional edit: These services also require massive numbers of server hosts. All of the old stuff was written in Python, all the new stuff is in Go. That alone is saving Uber tens of millions per year just in hosting costs.

Re: A former Uber engineer's disaster story

#173
post #60

This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens. I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system th…

From one of his replies on Twitter:

> The app looks simple from the user perspective. But on the global scale the business rules are insanely complicated. Every region has custom rules/regulations. Different products have different workflows. Some regions have cash payments. Every airport has different pick rules...

Re: A former Uber engineer's disaster story

#174
post #60

This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens. I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system th…

Follow-on tweets address this somewhat: Every airport has different pickup locations, every region has different payment rules including cash, etc. Apparently all this location-specific business logic has to be in the app itself and is not downloadable on demand.

We've previously spent a lot of resources also looking for ways to render native components through backend APIs (there was a team called screenflow that was laid off) but this was complex and wasn't able to support complex business needs. Uber does not want to remove app size at the cost of UX (ex. webviews).

Re: A former Uber engineer's disaster story

#175
post #49

Great thread. Thanks for sharing! It's interesting to me that the author feels that things were great with Uber in 2016 pre-Trump and that Trump's election was the catalyst for the negative sentiment. Susan Fowler's famous post was early 2017, and maybe _posting_ it was catalyzed by the issues surrounding Trump, but clearly the serious cultural problems she described at Uber had been going on for some time before tha…

I think the election of Trump may have made it possible for these cultural problems to come to light, the same way that (for example) women speaking up about sexual assault can empower others to do the same. People saw the issues that were placed on a national stage, then looked inwards and saw similar issues much closer to home, and these things started coming out as people were no longer OK with keeping quiet about…

> I think the election of Trump may have made it possible for these cultural problems to come to light, the same way that (for example) women speaking up about sexual assault can empower others to do the same.

> I personally think many companies have issues focusing on non-technical problems...This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others.

I agree with all that. To me, this is about taking responsibility. Our industry has shown a pattern of thinking poorly about non-technical problems while rolling out changes that affect millions of people. That's part of why so many people are angry at the tech industry -- and for good reason.

The thread is thoughtful in a lot of ways, but I get a whiff of not-taking-responsibility through some of it. The first example struck me for whatever reason. I get that people in 2016 didn't necessarily know what was going on or how bad it was. But I would expect that in 2020, we'd look back at 2016 as the time when serious issues beneath the surface were about to explode as a result of choices made at Uber, not some golden time before external forces made life hard for Uber. I don't mean to be too hard on the author or even Uber (and after all, this was a casual tweet thread). I think it's a widespread challenge for our culture in the tech industry.

Re: A former Uber engineer's disaster story

#176
post #60

This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens. I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system th…

Former Uber engineer/EM here: I worked on the Rider app. The “there are only a few screens” is not true. The app works in 60+ countries, with features shipped in the app that often for a country, and - in rare cases - a city. The app has thousands of scenarios. It speaks to good design that each user thinks the user is there to support their 5 use cases, not showing all the other use cases (that are often regional or…

thanks for sharing. I'm a former Uber engineer still reading in most Uber-related reddit threads how they could build the app in a weekend

Re: A former Uber engineer's disaster story

#177
post #152

Earlier quoted context omitted.

Eh. It doesn't seem particularly relevant to bring up Linux, with its hundreds of thousands of person-hours worth of volunteer contributions and an extremely technically focused BDFL gatekeeper. It's a relatively well known fact that multi-team corporate projects have tech debt. There was a thread recently about Google Cloud being slow due to too-many-cooks-in-the-kitchen syndrome, and many ex-amazon folks say AWS UI…

I mean, Uber has had a huge number of people working on it too, with the incentive that they are paid to do a good job. Regardless of the eventual outcome, I have no misconceptions that Uber doesn't have some engineers who are extraordinary talented. And, the project is newer, too, with fewer guarantees on stability and fewer users to boot. I think it's a valid comparison to make at least–they are both huge, complex…

I think the argument is more that even if you were to magically get rid of the cruft introduced by institutional bloat, the app will still necessarily be large given the nature of the business. Uber isn't some outlier here either; Lyft, Ola, Yandex, GoJek, Grab, etc etc all have comparable iOS app sizes.

Re: A former Uber engineer's disaster story

#178
post #3

This to me reads of Swift being pushed hard well before it was truely ready. Polynomial linker time? Wow. Ouch. No thank you.

Linkers must run in polynomial time for certain things; this is a problem independent of Swift.

That is true, but I am pretty sure this is misattributed. At the time we had two major issues which slowed down loading these sorts of dylibs.

One was attaching the code signatures (which gets attributed to the dynamic linker because it is synchronously stalling waiting for the the validation to occur).

The other was that reactive patterns tend to cause A LOT more classes and methods which result in a lot more fixups. When I am debugging apps I can often tell if they use react based one the ration of Objc/Swift metadata to __TEXT in the binary. That ended up causing dyld to stall waiting for page ins (which again are not really the linker but get attributed to it).

IOW, many of those clever patterns that make doing rapid development easier are really just a way to transfer work from the developer to the runtime, and it has a cost. Swift and SwiftUI are actually have some very clever ways to mitigate this, but reactive Swift libraries int he Swift 2.0 timeframe certainly did not.

Re: A former Uber engineer's disaster story

#179
post #60

This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens. I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system th…

Former Uber engineer/EM here: I worked on the Rider app. The “there are only a few screens” is not true. The app works in 60+ countries, with features shipped in the app that often for a country, and - in rare cases - a city. The app has thousands of scenarios. It speaks to good design that each user thinks the user is there to support their 5 use cases, not showing all the other use cases (that are often regional or…

My experience with all this is that each time I opened the Uber app, I actually did not know what I was going to find. Taking an Uber a few times a month meant finding new UX patterns every time, sometimes features would be there, sometimes they wouldn't. It was very annoying to find that useful features had suddenly disappeared. After feeling that the UX was hostile due to this, and some IRL hostile experiences with drivers with a terrible support follow-up, I just dropped Uber for good. Alternative apps were more expensive but absolutely more stable.

Re: A former Uber engineer's disaster story

#180

Earlier quoted context omitted.

Wow, I had no idea! On app load, maybe checking speed and then displaying a link to the web version for low bandwidth areas would be a good move.

Why display a link? Hybrid web/native apps are a tried-and-true strategy when you have massive complexity or dynamic content that you don't want to ship with your app. If you use it judiciously and tastefully the "native UI everything" people don't notice and come after you, either.

Not an app developer ;). But this makes sense. Thanks
Post reply on HN