Live data from Hacker News

The Uber Engineering Tech Stack, Part I: The Foundation

eng.uber.com

121–130 of 194 posts

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#121

Earlier quoted context omitted.

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

As far as I'm aware, Hailo (>3 backend devs, not quite 100s) did exactly this as well, and the ex-Hailo devs I've spoken to considered it a pretty bad move. It took them ages to refactor into a global system if I remember rightly.

[deleted]

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#122

Earlier quoted context omitted.

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

As far as I'm aware, Hailo (>3 backend devs, not quite 100s) did exactly this as well, and the ex-Hailo devs I've spoken to considered it a pretty bad move. It took them ages to refactor into a global system if I remember rightly.

Yep, I had a long chat with one of their engineers (Matt Heath) and seems like they had the exact same problems we have.

And solved it with their restructure and move to microservices.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#123

Earlier quoted context omitted.

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

Your user accounts aren't global? So does that mean if I travel to a new city I have to create a whole new account?

I've traveled to a dozen cities with Uber and didn't have to change my account, even in China. So the answer is no.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#124

Earlier quoted context omitted.

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

Your user accounts aren't global? So does that mean if I travel to a new city I have to create a whole new account?

Surprisingly although we have user accounts implemented, most of our customers don't use them.

But yes, as it stands, if you create a user account it is distinct per country.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#125

Earlier quoted context omitted.

All it takes is for one server in one datacenter to be slightly different, or perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it, now you've got a deploy that will be a lot more complicated or error-prone than a loop around deployment to one location.

> All it takes is for one server in one datacenter to be slightly different Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that. > perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it Feature flags. Default off, but flip on a new pa…

This is so weird. I hear this all the time. At my place of employment, we all have the ssh keys into our EC2 instances, but no one configures them. Ever. Period. Those ssh keys are purely for either validating changes in a test environment (like to .ebextensions) or diagnosing production issues (why did Puma fall over this time? why isn't syslog output making it to loggly?).

Of course, we lean heavily on Elastic Beanstalk; autoscaling regularly kills old instances daily since we scale from 2 to 18 and back to 2 instances in a 24 hour period across about 9 microservices.

So, if this ssh into boxes and change things is common, it means people aren't doing auto-scaling? THAT is scary.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#126

Uber is really strapped for engineering talent. Especially when it comes for SRE. Myself and many friends working SRE at various Bay Area companies get consistently hit up for free lunches and interviews. It's really weird considering that their stack doesn't NEED to be this complex....

It's really weird considering that their stack doesn't NEED to be this complex....

This is such a silly statement to make from the outside of any organization.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#127

What I'm really wondering about is their app. The UI of the app can be impacted without an app update. For example the UI during the pride parade. Or minute of silence ( http://gizmodo.com/uber-makes-riders-take-a-moment-of-silenc... ) I wonder what's the architecture of the app and the API for this.

If its updating without your consent to upgrade in the app store then its a webview youre looking at and they are just updating the webview.

Apple changed that rule a while back.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#128

Uber is really strapped for engineering talent. Especially when it comes for SRE. Myself and many friends working SRE at various Bay Area companies get consistently hit up for free lunches and interviews. It's really weird considering that their stack doesn't NEED to be this complex....

If they made it only as simple as it needs to be, then they couldn't patent very much. Investors want exclusivity, lots of convoluted tech-speak, big grants, etc. :P

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#129
post #21

I'd love to know how many people are responsible for devops/operations/app at various stages of any company's journey. Wikipedia says Uber employs 6,500 people so if even 15% of that is on the tech side of the business that's still 1,000+ people allocated to tech. I think this metric would be a useful reality check for a "modern" SaaS project with 3-10 people that's trying to emulate a backend structure similar to th…

That's all bloat. Pure and simple. At the end of the day Uber just does routing and basic allocation. It's a simple operations problem that has been solved since the 70s and no one back then needed ELK, Docker, Cassandra, etc. I've seen this bloat everywhere. It is usually a result of internal politics and posturing by management types. The kinds of people Steve Jobs would have called B and C players. Now the actual…

I disagree. Even on hacker news, people rarely express such absurd things with so little confidence. You fail to take into account many of the following:

* Extremely high volume. Uber has indicated elsewhere that they receive upwards of a few hundred thousand requests per second on just one service. Please show me the logistics stack that did this in the 70s.

* Yes, building the first version of something is extremely cheap and easy. But being able to improve it becomes harder and harder. Especially given high volume, modern companies need sophisticated analytical tools that provides reliable data to both technical and non technical staff. Please show me the analytics stack that was able to ingest, store, and analyze terabytes of business data in realtime from the 70s.

* Reliability. Modern web applications need to fail gracefully and be debugged quickly. Please show me the logistics and routing stack that was capable of extremely high uptime while being deployed constantly and serving hundreds of thousands of requests per second from the 70s.

* Extensibility. Businesses need to extend to new markets. Moves like this often invalidate past assumptions. In order to support business flexibility, modern engineers deliberately invests considerable time into building decoupled components that can be reused as platforms instead of stuffed into a monolithic codebase. Please show me the operations and routing stack that could easily be reconfigured to enable such products as Amazon Web Services, Uber's external API, the google maps API, or Uber EATS—from the 70s.

To make this more concrete, I worked on a routing stack at another company which probably works similarly to Uber's ETA systems. When considering these things, it's important to keep in mind the dependency tree of each new problem set and the work required to make those dependencies work reliably at big scale.

To give you an idea of what this area alone entails:

1. Machine learning. - wiring together of and improving algorithms: linear regression to begin with, then random forest, then neural networks. - ensuring data required for learning is reliably available and correctly computed. - tools to launch, deploy, test these models.

2. Working with map data in memory many times larger than what fits onto the smallest consumer laptop. - how do you handle updates of data? - what if you want to use different data sets in different places, because they're more accurate? - how do you debug errors in the data without visual tools (hint: it's really hard and time consuming)? - how do you optimize loading this data into memory without requiring hours to deploy your application? - where do you even store this data?

3. Requests per second in the hundreds of thousands and latency requirements (in order to ensure the app responds quickly) hovering around 10ms. - how do you profile complex distributed applications? - what optimizations are available to make graph search faster (hint: A* isn't fast enough)? - how hard is it to implement these optimizations?

4. Data science and data science tools - Visualizations! - again, reliable data pipelines

That's about what one team works on over the course of a year. Note the dependencies we have here:

1. We assume access to cloud infrastructure that doesn't require us to do all of our own devops.

2. We assume mature and automatically scaling data infrastructure: that kafka and storm have been set up and tuned to a degree that we don't have to worry about it. In reality, kafka alone requires a team of at least a dozen at linkedin to keep up with the maintenance, operations, and optimization burden of keeping up with scale.

3. We assume mature and scalable service oriented architecture tooling—if a call to another service is slow, I should be able to see on a dashboard what service is slow, how frequently, it's slow, why it's slow (if it depends on another service) etc.

and countless other things I could spend days enumerating for you but I guess it'd be wasted on you because you're pretty convinced you already solved these problems in the 70s, so why am i wasting my breath

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#130
I disagree. Even on hacker news, people rarely express such absurd things with so little confidence. You fail to take into account many of the following:

* Extremely high volume. Uber has indicated elsewhere that they receive upwards of a few hundred thousand requests per second on just one service. Please show me the logistics stack that did this in the 70s.

* Yes, building the first version of something is extremely cheap and easy. But being able to improve it becomes harder and harder. Especially given high volume, modern companies need sophisticated analytical tools that provides reliable data to both technical and non technical staff. Please show me the analytics stack that was able to ingest, store, and analyze terabytes of business data in realtime from the 70s.

* Reliability. Modern web applications need to fail gracefully and be debugged quickly. Please show me the logistics and routing stack that was capable of extremely high uptime while being deployed constantly and serving hundreds of thousands of requests per second from the 70s.

* Extensibility. Businesses need to extend to new markets. Moves like this often invalidate past assumptions. In order to support business flexibility, modern engineers deliberately invests considerable time into building decoupled components that can be reused as platforms instead of stuffed into a monolithic codebase. Please show me the operations and routing stack that could easily be reconfigured to enable such products as Amazon Web Services, Uber's external API, the google maps API, or Uber EATS—from the 70s.

To make this more concrete, I worked on a routing stack at another company which probably works similarly to Uber's ETA systems. When considering these things, it's important to keep in mind the dependency tree of each new problem set and the work required to make those dependencies work reliably at big scale.

To give you an idea of what this area alone entails:

1. Machine learning. - wiring together of and improving algorithms: linear regression to begin with, then random forest, then neural networks. - ensuring data required for learning is reliably available and correctly computed. - tools to launch, deploy, test these models.

2. Working with map data in memory many times larger than what fits onto the smallest consumer laptop. - how do you handle updates of data? - what if you want to use different data sets in different places, because they're more accurate? - how do you debug errors in the data without visual tools (hint: it's really hard and time consuming)? - how do you optimize loading this data into memory without requiring hours to deploy your application? - where do you even store this data?

3. Requests per second in the hundreds of thousands and latency requirements (in order to ensure the app responds quickly) hovering around 10ms. - how do you profile complex distributed applications? - what optimizations are available to make graph search faster (hint: A* isn't fast enough)? - how hard is it to implement these optimizations?

4. Data science and data science tools - Visualizations! - again, reliable data pipelines

That's about what one team works on over the course of a year. Note the dependencies we have here:

1. We assume access to cloud infrastructure that doesn't require us to do all of our own devops.

2. We assume mature and automatically scaling data infrastructure: that kafka and storm have been set up and tuned to a degree that we don't have to worry about it. In reality, kafka alone requires a team of at least a dozen at linkedin to keep up with the maintenance, operations, and optimization burden of keeping up with scale.

3. We assume mature and scalable service oriented architecture tooling—if a call to another service is slow, I should be able to see on a dashboard what service is slow, how frequently, it's slow, why it's slow (if it depends on another service) etc.

and countless other things I could spend days enumerating for you but I guess it'd be wasted on you because you're pretty convinced you already solved these problems in the 70s, so why am i wasting my breath

Post reply on HN