Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Sourcing Peloton, Uber’s Unified Resource Scheduler

eng.uber.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#51

Post by theon144 »

warp_factor wrote: post #30
spyspy wrote: post #18

Is it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.

I think you are right. I think that what is going there is also a bit political. They started to grow their Engineering department so fast that they need to justify the headcounts now. So each team is trying to invent new projects all the time. Anecdotally, this was partially confirmed to me by a friend working there. I said this before, but I still cannot understand why a service like Uber need so many engineers in…

>I said this before, but I still cannot understand why a service like Uber need so many engineers in the backend (multiple thousands). It is a complex distributed application, but nowhere near the scale or complexity of a Facebook or Google.

Thank you so much, I thought I was going crazy. I understand the demands of running a service on the level Uber has, but well, for instance I can't imagine what kind of computational workload / infrastructure requirements would make developing your own resource scheduler a reasonable option - for a Taxi app? With non-essential (to the core product) machine learning?

Forgive me if I'm ignorant, but what exactly does Uber engineering team do?

edit: On their blog I was able to find that they namely "forecast rider demand", from a relatively small [0] article - that is, comapred to the article [1] about what essentially is "just" data visualization, which doesn't help my confusion much.

0 - https://eng.uber.com/neural-networks/ 1 - https://eng.uber.com/maze/

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#53

Post by jjoonathan »

spyspy wrote: post #18

Is it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.

Does anyone know an open source Uber Michelangelo?

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#54

Post by chillacy »

warp_factor wrote: post #30
spyspy wrote: post #18

Is it just me or is every big release from Uber just a custom rewrite of an existing technology? It seems their engineering department has a large not-invented-here attitude. I could be wrong - they're certainly large enough to have custom requirements that aren't met with what's on the market but the pattern is just becoming suspect.

I think you are right. I think that what is going there is also a bit political. They started to grow their Engineering department so fast that they need to justify the headcounts now. So each team is trying to invent new projects all the time. Anecdotally, this was partially confirmed to me by a friend working there. I said this before, but I still cannot understand why a service like Uber need so many engineers in…

Maybe not at the same scale but imo their core system has a lot of tech challenges:

* Lots of realtime

* Resource scheduling problems

* Route optimization problems (especially with shared uber or shared lyft rides)

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#55

Post by platinumrad »

fenwick67 wrote: post #32
platinumrad wrote: post #31

Earlier quoted context omitted.

The term "peloton" obviously predates the company of the same name...

The term "uber" obviously predates the company of the same name...

And nobody is wringing their hands when something is given that name.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#56

Post by vimslayer »

jonny_eh wrote: post #10
r_klancer wrote: post #5

Immediate reaction: Well, now there's Peloton, the fitness tech company, Peloton, the self-driving truck caravan company, and Peloton, the cluster scheduler...

Where do all these names come from? Is it Latin or Greek or something?

In addition to what others have said here, "peloton" also means "fearless" in Finnish.

I have no idea if or why they would've used that or if they're just referring to the cycling thing, but I guess "fearless" could also be kind of fitting for this project.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#57

Post by notyourday »

theon144 wrote: post #51
warp_factor wrote: post #30

Earlier quoted context omitted.

I think you are right. I think that what is going there is also a bit political. They started to grow their Engineering department so fast that they need to justify the headcounts now. So each team is trying to invent new projects all the time. Anecdotally, this was partially confirmed to me by a friend working there. I said this before, but I still cannot understand why a service like Uber need so many engineers in…

>I said this before, but I still cannot understand why a service like Uber need so many engineers in the backend (multiple thousands). It is a complex distributed application, but nowhere near the scale or complexity of a Facebook or Google. Thank you so much, I thought I was going crazy. I understand the demands of running a service on the level Uber has, but well, for instance I can't imagine what kind of computati…

Makes it possible for me to get a ride, process payments and refunds even when the data centers are having issues or when there are temporary internet problems.

What does the facebook engineering team do?

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#58

Post by filereaper »

a-wu wrote: post #26

I’ve worked with Mesos pretty extensively before and when Uber first announced Peloton last year I was intrigued. Peloton seems to be a wrapper around Mesos that allows for running smaller, unique jobs without having to write a Mesos framework for each. Writing a Mesos framework for every small job you have can get annoying when you just want to define how your job should run and don’t really care for the resources o…

Mesosphere created dcos-commons to reduce the friction of getting frameworks up and running.

[1] https://github.com/mesosphere/dcos-commons

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#59

Post by thekhatribharat »

What exactly is Uber running in its clusters?

- Route Optimization

- Demand Forecasting

- Rider Hotspot Prediction

This post doesn't exactly tell us the true nature of their workloads (other than the crude categorization - batch, stateful, stateless), nor does it talk about the inflection points where off-the-shelf solutions don't cut it anymore and such customization is required. I mean some before & after numbers / graphs on resource utilization would have really helped.

Re: Open Sourcing Peloton, Uber’s Unified Resource Scheduler

#60

Post by warp_factor »

chillacy wrote: post #54
warp_factor wrote: post #30

Earlier quoted context omitted.

I think you are right. I think that what is going there is also a bit political. They started to grow their Engineering department so fast that they need to justify the headcounts now. So each team is trying to invent new projects all the time. Anecdotally, this was partially confirmed to me by a friend working there. I said this before, but I still cannot understand why a service like Uber need so many engineers in…

Maybe not at the same scale but imo their core system has a lot of tech challenges: * Lots of realtime * Resource scheduling problems * Route optimization problems (especially with shared uber or shared lyft rides)

ok, but:

* Everything they do is low data (no video, image or anything high bandwith).

* Their whole model can be subdivided into smaller local problems (all users//drivers in the bay area have nothing to do with the users//drivers currently in NYC).

yes there is a couple of algorithms to develop for Uber Pool, and for the real time matching but everything else looks like a fairly simple app backend to me.

Post reply on HN