Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A former Uber engineer's disaster story

twitter.com

Post reply on HN
Print view View original

221–230 of 344 posts

Re: A former Uber engineer's disaster story

#221

Post by lock-free »

dntrkv wrote: post #194
lock-free wrote: post #192

Earlier quoted context omitted.

You should scratch below the surface because this isn't really explaining much - it does not sound like something that requires more than 100 people. Their legal team sounds like it should be twice that alone. Or am I crazy?

Take look at this thread: https://news.ycombinator.com/item?id=25376346 My point is that there is 100s of screens/features in the Uber app that you don't even know exist. And you have those same 100s of screens across their other apps. Multiplied across all of the different platforms. Plus any internal tools they use. I'm curious, have you worked at a software company of Uber's scale? Not trying to be a dick here, ge…

My whole point here is I don't understand why "Uber scale" is so large. I have worked in large organizations - smaller than Uber, but we did more stuff (as in more complexity and diversity in products - shipping software and the hardware to run it, for example).

"100s of screens" is as meaningless to me as "thousands of lines of code" and wholly uninteresting because it seems like the front end is the least significant reason why Uber requires so many people to operate. That's why I'm interested in seeing how many people are assigned to projects and what they actually spend their time doing, because at scale you begin to experience inefficiencies due to scale for bad reasons if your incentives aren't aligned. From the original twitter thread that sounds like it was rampant at Uber.

Not sure how long you've been around, but "Uber scale" used to be a joke a few years ago because no one could understand why they would reinvent so many wheels because "they don't work at Uber scale." The joke being that Uber didn't need to be Uber scale to begin with, they just had investor money to burn.

Re: A former Uber engineer's disaster story

#222

Post by uberiosthrow »

nwsm wrote: post #28
nobunaga wrote: post #21

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it. Problem here with Swift came down to - Apple not dogfooding its own tech . Hell they did not even help us or othe…

Appreciate your insight, and I'm sorry to hear about the AMS team. The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years". Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS deve…

As someone who was there, a rewrite was necessary, and with hindsight we should of done it with objective-c. The original obj-c app was built with under 10 iOS engineers in mind, and now Uber had 100s of mobile engineers work on one app in some form or another.

We didn't do a simple build scalability test until we were well into the project. If we did it would of revealed swift's build problems to us. Our swift experience is what slowed down our kotlin migration significantly. Today Uber android is still a majority Java app.

Even very recently the experience of working in swift compared to the old obj-c code base isn't as good. To this day there are debugger issues, xcode responsiveness issues and a slower build time.

Hiring on the other hand would be harder now, because it's getting hard to find people who know Objective-C and it's surface ugliness scares new people away. Swift is a decent language if you don't have to have a very large code base, so most people rightfully so don't have that much Objective-C, and if you were to work in Obj-C, you might as well go work at Facebook then.

Re: A former Uber engineer's disaster story

#223

Post by ehmmmmmmmm »

microtherion wrote: post #56
nwsm wrote: post #28

Earlier quoted context omitted.

Appreciate your insight, and I'm sorry to hear about the AMS team. The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years". Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS deve…

A complete rewrite of a large project is risky anyway. Doing so in a language in which literally no other project of that size exists yet strikes me as insane. I love Swift. But I've been adopting it in bits and pieces over the years. It's not just that the language has changed significantly, you also have to get used to the idioms. E.g. my early Swift code often featured Pyramid of Doom patterns, and both language i…

That's why I call idioms idiots. If you have to learn idioms, it's an idiotic thing.

Re: A former Uber engineer's disaster story

#224

Post by uberiosthrow »

sixstringtheory wrote: post #206
microtherion wrote: post #56

Earlier quoted context omitted.

A complete rewrite of a large project is risky anyway. Doing so in a language in which literally no other project of that size exists yet strikes me as insane. I love Swift. But I've been adopting it in bits and pieces over the years. It's not just that the language has changed significantly, you also have to get used to the idioms. E.g. my early Swift code often featured Pyramid of Doom patterns, and both language i…

One thing I was surprised not to see was a mention of their Swift 2 to 3 migration, which was terrible in my experience working on apps that were nowhere near the complexity of Uber. Apple should have never pushed Swift as hard as it did at the time. Between the constant API changes and issues like requiring dynamic framework linkage and copying the whole runtime into every app, it was absolutely not production ready…

It was a lot of work, but we had enough staff to make it someones job for half a year or a quarter, depending on how bad the migration is. With big codebases there are other issues that pop up with every new major Xcode release that we have to migrate, mitigate or fix. For example we are still at xcode 11 because of various bugs in xcode 12. Doing the migration was annoying, but it was not much compared to all the other issues, because the fixes were tractable.

Re: A former Uber engineer's disaster story

#226

Post by uberiosthrow »

nobunaga wrote: post #34
nwsm wrote: post #28

Earlier quoted context omitted.

Appreciate your insight, and I'm sorry to hear about the AMS team. The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years". Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS deve…

I was not part of the initial analysis team so cannot really comment if objc was more suitable as it would require to look at objc at that time and do tests etc but I suspect we would have had a more smoother experience. Though the the code at the time was bad and indeed something needed to happen to ensure future scalability of the teams and product/code, but I think there were more sane paths we could have taken. I…

We could of made it work. I think the one big benefit of the rewrite was how the architecture and nullable types basically gave us one more 9 of crash free rate, which we saw after the release with swift. With ObjC I could forsee how it would of crashed a bit more, because nullables aren't hard enforced in ObjC, but I could also see how we could of stepped around that by strong guarantees in the model & network service generation code.

Re: A former Uber engineer's disaster story

#227

Post by uberiosthrow »

WSSP wrote: post #218
offtop5 wrote: post #144

Earlier quoted context omitted.

You can enjoy coffee without grinding your own beans. I personally find lower level programing to be frustrating and difficult. I've built an amazing career with JavaScript and C# primarily. My hobbyist projects use C# , JS and Flutter. I can't wrap my head around things like pointers or memory management

post-ARC, Objective-C and Swift aren't very different in terms of memory management. retain and release are handled for you, and you only need to worry about retain cycles. structs in Swift change things a bit, but not that much

+1

With swift & obj-c your coding against the same library APIs, so in many ways it's a very similar level of programming. In many ways swift is far more complicated than Obj-C. With Obj-C iOS apps your not really writing that much C vs. the pseudo-smalltalk attachment to it.

Re: A former Uber engineer's disaster story

#228

Post by saagarjha »

aaronbrethorst wrote: post #220
saagarjha wrote: post #156

Earlier quoted context omitted.

My point was that there was no need to ask Chris Lattner back in 2016: you could just look around the landscape and see that there were not many, as they would be easy to see if they did exist.

Lots of companies don’t speak publicly about their product strategies, like certain former employers of his.

As I mentioned, there is no need for a company to publicly post details of their products for people to figure out what languages they were written in.

Re: A former Uber engineer's disaster story

#229

Post by vsareto »

GVIrish wrote: post #163
flareback wrote: post #41

> So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions. I really like this point.

I think not clearly understanding the tradeoffs was a huge factor here, but I would go one step deeper into that decision-making process. One thing that could happen when making a decision like ObjC vs Swift is that the decision makers can think they have performed a sober risk/benefit analysis, but failed to dig deep enough to uncover show stoppers. So they may have looked at developer productivity with Swift and to…

Rewriting apps (esp. as large as Uber) is essentially your plan meeting the enemy. There's many unknown-unknowns simply because technology is complex, and although it's pretty much unprovable, the brisk pace of technology also means you'll have more unknown-unknowns. Only a big industry shift in priorities will change that now.

Re: A former Uber engineer's disaster story

#230

Post by Groxx »

Apocryphon wrote: post #171
carapace wrote: post #110

I started to write a whole big thing here because I found this whole thing pretty disgusting but it's not worth it. > I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out. > With only a week left we decided eat the 8 figures and drop support for iOS 8.…

One wonders if this is common everywhere at this scale. Take Facebook's massively bloated iOS app: https://www.reddit.com/r/iOSProgramming/comments/6upeu6/how_... https://news.ycombinator.com/item?id=14903228 And their Android app patching Dalvik at runtime to deal with the huge number of Java methods: https://www.facebook.com/notes/facebook-engineering/under-th... And they're not the only ones known for bloated apps…

I suspect it is common. And mostly inevitable. When you have 10^N engineers working on something, protecting them from each other (and themselves) grows increasingly important. At some point, it becomes your main priority, as without it you have a totally non-functioning system.

Which partly implies that the only solution is to not grow to that scale. Keep a smaller, more skilled team instead. In most cases I think that'd be better, and I think most people (and companies) would agree (if only they could hire those skilled teams reliably)...

... but sometimes throwing more people at a problem really is the best (or only) option, e.g. when you have to deal with a large volume of ridiculous external constraints. For example: my first job was at a company that aggregated loan providers, which entailed consuming hundreds of random PDFs per day and dozens of weird RPC calls. We had dozens if not hundreds of hand-crafted bits of automation to detect when things changed and handle the new format. Many APIs had hand-rolled XML parsers and producers... because the company we were calling would choke if your attributes weren't in a specific order, or if X wasn't within a specific number of bytes from the start of the request, or they returned invalid XML. It was absolutely ridiculous, but there's no way a bunch of banks were going to fix their APIs for us, so there's not much of an option but to do it by hand.

Post reply on HN