Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A former Uber engineer's disaster story

twitter.com

Post reply on HN
Print view View original

21–30 of 344 posts

Re: A former Uber engineer's disaster story

#21

Post by nobunaga »

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it.

Problem here with Swift came down to

- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)

- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)

Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.

Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.

Re: A former Uber engineer's disaster story

#22

Post by [deleted] »

app4soft wrote: post #16

> “biggest engineering disaster” I just supposed that this was a story about Uber Elevate ElectricVTOL ( eCRM ) design.[0,1] [0] https://twitter.com/TheWoracle/status/995107989886353408 [1] https://github.com/OpenVSP/OpenVSP/tree/main/examples/vspaer...

[deleted]

Re: A former Uber engineer's disaster story

#24

Post by lxe »

Making a huge decision and sticking with it -- blowing through all obstacles through workarounds and hacks -- because otherwise would be deemed to be a failure is the core issue here.

Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.

Re: A former Uber engineer's disaster story

#26

Post by curiousllama »

nwsm wrote: post #14

Great thread. > So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going. > This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs…

This isn't sunk cost; the exec chain just has different incentives. They're hired for outcomes. It's their job to make it happen, not make it make sense.

> A bunch of people got promoted. We all breathed a sigh of relief. The 90 work weeks stopped for a few weeks.

When you move up, you move on; issues from then on are the next guy's problem. Exec dynamics are often "gather successes to me and push off failures to others" - sounds like some exec won.

Re: A former Uber engineer's disaster story

#27

Post by staplung »

I've lost track of the number of "biggest engineering disaster" stories I've read that begin "so we decided to rewrite it..."

The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going.

Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Re: A former Uber engineer's disaster story

#28

Post by nwsm »

nobunaga wrote: post #21

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it. Problem here with Swift came down to - Apple not dogfooding its own tech . Hell they did not even help us or othe…

Appreciate your insight, and I'm sorry to hear about the AMS team.

The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years".

Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal.

Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?

Re: A former Uber engineer's disaster story

#29

Post by nobunaga »

lxe wrote: post #24

Making a huge decision and sticking with it -- blowing through all obstacles through workarounds and hacks -- because otherwise would be deemed to be a failure is the core issue here. Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.

Management had very different incentives while engineers had different motives. The two were not aligned, and TK culture was still very prominent back then (work hard, dont go home early, we are going to be a 1 trillion dollar company etc etc) and Thuan definitely helped him spread that culture. So definitely not great days but I think the engineers took away a lot from their time there. Lots of lessons learned

Re: A former Uber engineer's disaster story

#30

Post by jcranmer »

gambiting wrote: post #18
linuxftw wrote: post #6

The biggest engineering disaster Uber had was when it intentionally disabled the automatic safety brakes on their 'self driving' car and killed a person.

I don't know why you're getting downvoted. Making a crappy, slow app is nothing compared to conciously deciding to switch off the built-in autonomous breaking system in lieu of your own(which then doesn't work and kills a person). That's an engineering disaster, not a taxi going to the wrong city block a few times.

Considering that the engineer describes his story as "the biggest engineering disaster I’ve ever had the misfortune of being involved in," I don't think it is relevant to his story that the self-driving pedestrian collision occurred. He isn't claiming--and even from the thread title, I wouldn't assume he is claiming--that this is the biggest engineering disaster ever, or even the biggest engineering disaster Uber faced.
Post reply on HN