Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A former Uber engineer's disaster story

twitter.com

Post reply on HN
Print view View original

291–300 of 344 posts

Re: A former Uber engineer's disaster story

#291

Post by StanTwinB »

sond813 wrote: post #243
nobunaga wrote: post #21

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it. Problem here with Swift came down to - Apple not dogfooding its own tech . Hell they did not even help us or othe…

I was an iOS engineer at Airbnb when we were battling binary size increases caused by Swift. I rewrote the watch app twice; once to migrate to watchOS 2 and once to convert it all to ObjC (which saved ~9mb). While Swift does have its problems (and had a lot more back when we worked on this), there are many factors to why the Uber app (and Airbnb) is getting so large. Airbnb is 75% executables, 9.5% assets and 8% loca…

Looking at the bundle now is not going to give you insight into the problems in the Uber app 4 years ago. Also you’re looking at the uncompressed install size, which was not the problem (over the air compressed download size was the limiting factor). Strings files are extremely compressible, encrypted binaries are not (Apple has done some work in subsequent years to make them more compressible but back then they were basically uncompressible).

Re: A former Uber engineer's disaster story

#292

Post by StanTwinB »

nwsm wrote: post #28
nobunaga wrote: post #21

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it. Problem here with Swift came down to - Apple not dogfooding its own tech . Hell they did not even help us or othe…

Appreciate your insight, and I'm sorry to hear about the AMS team. The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years". Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS deve…

If by OP you mean me, then let’s be clear. I didn’t think the rewrite was necessary.

Re: A former Uber engineer's disaster story

#293

Post by StanTwinB »

rsp1984 wrote: post #248

The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discov…

It’s interesting to me that you came away from this thinking that I didn’t realize the eng size growth was the problem. I specifically said in the third tweet that hypergrowth was the problem. I even went on to say why it was a problem. The entire point of including the details about the org growth was specifically to draw you to that conclusion.

Re: A former Uber engineer's disaster story

#294

Post by StanTwinB »

otterley wrote: post #262

"Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience. So to improve location pickup we changed the location permission to collect signal in the background so we could send the drivers to your cu…

It’s not about collecting info before you open the app. What happens with most people is they open the app, request a ride, then close the app again and do something else until the driver has arrived notification. Location services start out with a wider range and narrow in on you over time, so a longer collection window gives more precision.

Re: A former Uber engineer's disaster story

#295

Post by lhorie »

lock-free wrote: post #287
lhorie wrote: post #235

Earlier quoted context omitted.

A helpful way to think of it is by comparing how Uber and an OS GUI are different (since that's the comparison people are making, anyways) With an operating system UI, the developer may reasonably know what the UI will look like, what features are present, where they are placed. With Uber, a single person may not know all the rules for a single feature, and the rules change frequently, sometimes in the most obtuse wa…

That's a bad comparison and I'm not sure why people are making it - you're mistaking fundamentals. This isn't a UI problem, it's datastructures and algorithms. There are plenty of desktop apps with similar issues (representing a complex problem as a set of constraints and writing a solver to find a solution, and querying the solution for the data to display to a user). I don't reject that there are troubles in concer…

> there's no reason a front end developer should need to care about it

Well, yes and no. You're right that trying to express every nook and cranny of the business rules as solely a UI problem is a wrong way of trying to understand the complexity. But a frontend developer does need to care about how to surface whatever complexity needs to be surfaced, and likewise, backend developers need to know how to model the complexity, and there needs to be some level of orchestration between all the moving parts. Remember, Uber is built on a microservice architecture, and many things that would be arbitrary hardcoded values in projects elsewhere (e.g. pricing) are instead tied to complex dynamic systems in Uber.

In the pricing example, when Uber added the option for drivers to set their own multipliers, it wasn't just a new slider in the driver app. It also affected the pricing AI model, the matching algorithm, how and when prices are displayed in the rider app, geofencing rules, etc etc.

When new regulations mandate that every commit that makes production changes in your city must have an appropriate audit trail, when you need to deploy a cheat-proof face mask detection AI model in response to a global pandemic, when you need to figure out how to make customer support cost $1 per engagement instead of $10, when you need to figure out how to make your network protocol faster in Bangalore, when you need to figure out how to make GPS more accurate in Manhattan, or when any of hundreds of fuzzy problems need solving, then saying that variations of all of these problems have been seen elsewhere misses the point that going from problem statement to deployed solution isn't always a trivial task.

The original question was why so many people work at Uber. The answer is that there are a lot more fuzzy problems than meets the eye.

Re: A former Uber engineer's disaster story

#296

Post by StanTwinB »

XorNot wrote: post #37

The "curmudgeon" ending line in this story rubs me the wrong way. The ObjC people were right, basically, and it seems like the problem was essentially no one in the right place was going to drop Swift at that point. Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!" Though the 6 library limit and how that wasn't a gigantic red flag on the language is b…

Indeed the ObjC people were right (I was one of them). But there is nothing more useless than standing in the corner and saying “I told you so”. That’s the curmudgeon.

Re: A former Uber engineer's disaster story

#297

Post by whomst »

Bayart wrote: post #281
11thEarlOfMar wrote: post #69

I'm reminded of the story of Windows NT and how David Cutler commented at one point after the release that divorces and nervous breakdowns were tracked in the program metrics. I expect that's hyperbole, but there is a real cost to these death marches. I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the…

I find it fascinating that so much engineers (arguably « smart » people) are actively hurting themselves over a freaking job and acting as if it were just a fact of life.

Plenty of people throughout history have martyred themselves for what they believed in. Fascinating for sure but being smart is a completely different question.

Re: A former Uber engineer's disaster story

#298

Post by sond813 »

StanTwinB wrote: post #291
sond813 wrote: post #243

Earlier quoted context omitted.

I was an iOS engineer at Airbnb when we were battling binary size increases caused by Swift. I rewrote the watch app twice; once to migrate to watchOS 2 and once to convert it all to ObjC (which saved ~9mb). While Swift does have its problems (and had a lot more back when we worked on this), there are many factors to why the Uber app (and Airbnb) is getting so large. Airbnb is 75% executables, 9.5% assets and 8% loca…

Looking at the bundle now is not going to give you insight into the problems in the Uber app 4 years ago. Also you’re looking at the uncompressed install size, which was not the problem (over the air compressed download size was the limiting factor). Strings files are extremely compressible, encrypted binaries are not (Apple has done some work in subsequent years to make them more compressible but back then they were…

Absolutely right! I left out some details to try to keep the post concise. 4 years ago at Airbnb we had the watch app problem, but now a Swift watch app wouldn’t be an issue since the latest OSs don’t require embedded Swift standard libraries. The tweet that started this thread cited the 300+mb size of Uber’s app, which looks like the current install size (not thinned), that’s why I used it as the benchmark in my explanation.

Lately Apple has been featuring size more prominently on the App Store, and it appears to be displaying the install size. This is also what users see when they look at which apps are taking up storage on their device in Settings. I think it’s important to reduce both, so users see a good number and downloads are fast/efficient. It is a very welcome change that binary encryption doesn’t negate all the gains from compression anymore. I had noticed it in the numbers but looked around for an official statement and didn’t find any, if you happen to know where this was announced I’d appreciate a link!

Re: A former Uber engineer's disaster story

#300

Post by p_l »

trimbo wrote: post #88
staplung wrote: post #27

I've lost track of the number of "biggest engineering disaster" stories I've read that begin "so we decided to rewrite it..." The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going. Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Couple things always to mind whenever this article is referenced. Joel's example is Mozilla, but Mozilla ultimately became more successful than Netscape (in terms of userbase with Firefox). Was the problem really the rewrite, or that a browser company was doomed anyway and the rewrite exacerbated it? Also the article was published just a year before Microsoft pulled off what's maybe the most successful large rewrite…

NT wasn't a rewrite, simply.

It's a completely different system, that happens to support the same API/ABI, backed by long experience at Microsoft at making things backwards compatible (it's probably the biggest source of Microsoft's success).

Microsoft also put ~7 years into preparing the transition piece by piece, with probably the biggest impact point being Windows 2000 that provided significant host of features that compelled big (and loud in complaints) clients over, pulling a lot of software vendors along.

Post reply on HN