Open Sourcing Peloton, Uber’s Unified Resource Scheduler • Hacker News Board

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#71

Post by jvanderbot » Wed, Mar 27, 2019, 4:46 AM UTC

theon144 wrote: post #51

warp_factor wrote: post #30

Earlier quoted context omitted.

I think you are right. I think that what is going there is also a bit political. They started to grow their Engineering department so fast that they need to justify the headcounts now. So each team is trying to invent new projects all the time. Anecdotally, this was partially confirmed to me by a friend working there. I said this before, but I still cannot understand why a service like Uber need so many engineers in…

>I said this before, but I still cannot understand why a service like Uber need so many engineers in the backend (multiple thousands). It is a complex distributed application, but nowhere near the scale or complexity of a Facebook or Google. Thank you so much, I thought I was going crazy. I understand the demands of running a service on the level Uber has, but well, for instance I can't imagine what kind of computati…

Those cars generate a lot of sensor data. (Tb per drive?). Id imagine that data needs to be made actionable and seperated into training and simulation sets pretty quickly. Mapping is a massive problem to automate.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#72

Post by theon144 » Wed, Mar 27, 2019, 7:42 AM UTC

jvanderbot wrote: post #71

theon144 wrote: post #51

Earlier quoted context omitted.

>I said this before, but I still cannot understand why a service like Uber need so many engineers in the backend (multiple thousands). It is a complex distributed application, but nowhere near the scale or complexity of a Facebook or Google. Thank you so much, I thought I was going crazy. I understand the demands of running a service on the level Uber has, but well, for instance I can't imagine what kind of computati…

Those cars generate a lot of sensor data. (Tb per drive?). Id imagine that data needs to be made actionable and seperated into training and simulation sets pretty quickly. Mapping is a massive problem to automate.

Oh right, the self-driving cars. Well that's starting to make sense now.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#73

Post by raghava » Wed, Mar 27, 2019, 8:28 AM UTC

warp_factor wrote: post #30

spyspy wrote: post #18

Is it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.

I think you are right. I think that what is going there is also a bit political. They started to grow their Engineering department so fast that they need to justify the headcounts now. So each team is trying to invent new projects all the time. Anecdotally, this was partially confirmed to me by a friend working there. I said this before, but I still cannot understand why a service like Uber need so many engineers in…

> need to justify the headcounts

Something similar happened in LinkedIn too I guess. Multiple teams building very similar tools that were on very related problems.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#74

Post by mindcrime » Wed, Mar 27, 2019, 2:23 PM UTC

spyspy wrote: post #18

Is it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.

Is it just me or is every big release from Uber just a custom rewrite of an existing technology?

I'm a little sad that this is the top comment here. I mean, maybe you're right. But so what? Some people find this useful, and some won't. Same as anything else.

At the end of the day, every line of code added to the world's pool of OSS code is a Good Thing™ as far as I'm concerned. Even if it's something I personally don't have a use for.

I think we should encourage companies to release code as open source, and give Uber at least some small measure of "props" for the stuff they release. Maybe none of their stuff is a game changer like Linux, but it doesn't need to be.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#75

Post by chillacy » Thu, Mar 28, 2019, 12:04 AM UTC

warp_factor wrote: post #60

chillacy wrote: post #54

Earlier quoted context omitted.

Maybe not at the same scale but imo their core system has a lot of tech challenges: * Lots of realtime * Resource scheduling problems * Route optimization problems (especially with shared uber or shared lyft rides)

ok, but: * Everything they do is low data (no video, image or anything high bandwith). * Their whole model can be subdivided into smaller local problems (all users//drivers in the bay area have nothing to do with the users//drivers currently in NYC). yes there is a couple of algorithms to develop for Uber Pool, and for the real time matching but everything else looks like a fairly simple app backend to me.

I'm not a network engineer so I'm unfamiliar with how problems scale by bandwidth. I do know that solving NP hard problems is difficult, so I respect Uber engineers for that at least.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#76

Post by eropple » Thu, Mar 28, 2019, 3:25 PM UTC

subway wrote: post #11

eropple wrote: post #2

My first reaction was "this just sounds like Mesos?". And it's cited in their page (which on first read I thought meant they were trying to act as a single pane of glass for Mesos/k8s/etc.): https://eng.uber.com/peloton/ In the OP blog post though, they assert "to our knowledge, there is no other open source scheduler which combines all types of workloads for web-scale companies like Uber." And then, when you dig...i…

Singularity isn't a scheduler. It's generally invoked by traditional hpc schedulers like moab or slurm. edit: clearly I'm thinking of a different Singularity than you.

Yeah - on a phone but I'm referring to the Mesos framework by HubSpot.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#77

Post by malandrew » Thu, Mar 28, 2019, 8:12 PM UTC

spyspy wrote: post #18

Is it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.

The problem with a comment like this is that it only takes into account when something was released publicly and not when the problem was first worked on and implemented internally.

Many internal projects that eventually become open-source often are not NIH projects because when the project was proposed their may have been no public open source projects or at least none that is mature enough. Even if something exists but it still in its early stages, it presents a lot of risk because your company isn't in the driver seat building and maintaining it.

Claiming something is NIH based on when it first became polished enough to be open-source ignores all the history behind the state of the world when a project was first worked on.