Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A former Uber engineer's disaster story

twitter.com

Post reply on HN
Print view View original

251–260 of 344 posts

Re: A former Uber engineer's disaster story

#251

Post by cbrauchli »

nobunaga wrote: post #21

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it. Problem here with Swift came down to - Apple not dogfooding its own tech . Hell they did not even help us or othe…

I was the person that actually managed the binary size task force and proved that it was a business problem in the first place. Not all management didn’t care about AMS :)

The Amsterdam team 100% played a critical role in solving the issue (at least until apple bumped the limit). No way we would have stayed under it without you all.

Re: A former Uber engineer's disaster story

#252

Post by firebones »

CountHackulus wrote: post #128
nwsm wrote: post #14

Great thread. > So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going. > This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs…

> > This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs > This would scare me too. For what it's worth a few (10ish) years ago we did this with the J9 compiler at IBM (technically just the backed that was shared between Java, C++, etc). Same idea with simulated annealing. It did end up finding some really strange bugs, but they were bugs that we…

It sounds almost like it is a kind of fuzz-testing then; I'm interested in what percentage of problems uncovered were problems in the spec or undefined behavior, versus problems introduced by the simulated annealing itself.

Re: A former Uber engineer's disaster story

#253

Post by xiphias2 »

Binary size should have been treated like a limited resource: set a hard limit in the test suite, and let the engineers who commit code in the organization fight it out how to delete unused code when they create new one for a new feature.

I'm following George Hotz's tinygrad that is a CPU+GPU deep learning framework under 1000 lines of code with great interest where all engineers are trying to shave lines of code while maintaining readable code (it's like a game when you set rules):

https://github.com/geohot/tinygrad

Here's the GPU ops part:

https://github.com/geohot/tinygrad/blob/master/tinygrad/ops_...

Re: A former Uber engineer's disaster story

#254

Post by nathanmock »

Fun fact: I was able to access a bunch of employee admin screens in early 2015 because of the client side architecture of the app. I noticed around a year later the architecture had changed and always wondered if my article had a role in the transition

http://nathanmock.com/archives/how-i-accessed-employee-setti...

Re: A former Uber engineer's disaster story

#255

Post by deckard1 »

_dh54 wrote: post #77

Many engineering horror stories involve an entire system rewrite. My recommendation is to never rewrite a large system. Literally never. If the people proposing the rewrite aren’t capable of adding the features incrementally, they aren’t capable of rewriting the entire system to “easily” support those new features.

I shudder at the thought of hundreds or thousands of Uber engineers banging out a rewrite together. I've been through a number of rewrites, and every single time it always meant the same thing: get your resume ready and look for the exit. It's going to be a multi-year shitshow.

Re: A former Uber engineer's disaster story

#257

Post by cactus2093 »

rsp1984 wrote: post #248

The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discov…

What you're ignoring is that some significant percentage of that "bloat" is what leads to more rides and more money. All those tweaks and a/b tests and features that solve problems for specific segments that most users won't ever see, getting shipped week after week, increasing the new user conversion rate by 0.5% here, reducing the rate of canceled trips by 0.3% there, it all adds up over time. Many of the features obviously fail too and don't improve anything they just add bloat, but you can't ever know for sure which features those will be in advance.

In aggregate, the goal is obviously that it leads to the company making more money than was lost by paying all those engineers to move mountains to get around all the issues they hit up against. In the case of Uber, now a $96 billion public company and worth nearly double their 2016 valuation, it's a little ridiculous to call it a failure.

You read this story and see an unmitigated disaster, but it's mostly just a picture of things working the way the company wants them to. The goal is to grow the company, not to have a nice, conceptually clean, tech-debt free codebase just for the sake of it.

Re: A former Uber engineer's disaster story

#258

Post by typ »

pavlov wrote: post #4

Apple's botched introduction of Swift is easily responsible for billions of dollars in lost productivity across the industry since 2014.

I’m afraid it could easily be the biggest mistake made by Apple over the last decade. The bloatedness and slow build time just aren't justified by the shiny, convenient language features.

Re: A former Uber engineer's disaster story

#259

Post by anonacct38 »

munificent wrote: post #104
11thEarlOfMar wrote: post #69

I'm reminded of the story of Windows NT and how David Cutler commented at one point after the release that divorces and nervous breakdowns were tracked in the program metrics. I expect that's hyperbole, but there is a real cost to these death marches. I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the…

> I expect that's hyperbole, but there is a real cost to these death marches. I used to be a game developer at EA. In the middle of one particularly nightmarish 60-hour-weeks-for-months crunch, a twenty-something artist with no known pre-existing conditions spontaneously died in his sleep.

I went through a pretty intense crunch time at a fortune 10.

We had one person go to the hospital with tachycardia and 3 people who were vomiting. There were only 10 people on the team.

Re: A former Uber engineer's disaster story

#260

Post by mikeyouse »

qntty wrote: post #214
didip wrote: post #126

Earlier quoted context omitted.

I have to say, this comment is quite ridiculous. Might as well say that Google is simple because it's just an app with one search field. I never work for Uber but I can easily imagine the following problems: * Map data is not a solved problem. Tons of inaccurate data especially on non-english speaking countries. * As someone mentioned about payments, different country requires different payment platform integrations.…

It seems a little strange to criticize someone for asking a question on a site where most users seem to value curiosity. Not everyone has a background that makes it as easy to imagine problems like these, and while you've given some possibilities, I would also be interested in knowing exactly what accounts for the size.

There are far better ways to ask the question than “What does the app even do, just seems like a map with a few dots” which has a heavy implication on the quality of the team that built it..
Post reply on HN