Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Uber Engineering Tech Stack, Part I: The Foundation

eng.uber.com

Post reply on HN
Print view View original

141–150 of 194 posts

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#141

Post by tupshin »

matthewrudy wrote: post #49
Animats wrote: post #25

It's interesting that they don't break the problem apart geographically. It's inherent in Uber that you're local. But their infrastructure isn't organized that way. Facebook originally tried to do that, then discovered that, as they grew, friends weren't local. Uber doesn't need to have one giant worldwide system. Most of their load is presumably positional updates. Uber wants both customers and drivers to keep their…

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

Recipe:

Create a global Cassandra cluster with regional datacenters.

Use one keyspace per region

Use per-keyspace replication to only replicate that region's data locally, and to one or more additional datacenters

Have stateless app servers colocated with Cassandra in each DC handling all local traffic

Run spark on top of Cassandra to do analytics, or to do the etl to a dedicated analytics system

Optionally have a single "master" DC, with replicas of all data from all locations, that doesn't serve end user traffic, but is to allow efficient cross region analytics.

Profit (optional step)

And yes, the company I work for (Datastax) has a product and services to help make it simple.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#142

Post by jedberg »

dominotw wrote: post #133
NotQuantum wrote: post #33

Uber is really strapped for engineering talent. Especially when it comes for SRE. Myself and many friends working SRE at various Bay Area companies get consistently hit up for free lunches and interviews. It's really weird considering that their stack doesn't NEED to be this complex....

what are some of the skills/experience needed to be an SRE? I've been having a really hard time finding a job due to being a 'jack of all trades' and having no specialty. Just an assumption. I have over a decade of experience building webapps. I've spent over 500 hrs on interviews over the past 3 months doing countless coding tests/exercises, whiteboard interviews. I just seem to never get past on-site interviews.

> I have over a decade of experience building web apps.

Were you also running those apps? SRE means you understand the intricacies of running an app too.

When I hire SREs, I look for people who have the following skills, in this order:

1. Leadership under pressure. What I mean is can you stay cool and calm and keep everyone around you cool and calm when everything is melting down

2. Experience operating a platform. Do you know basics like networking, system startup, system and OS tuning, etc. Can you diagnose a problem on a running instance?

3. Coding. Can you write decent code and can you understand good code.

The reason it is in that order is because staying cool under pressure is something I can't really teach you, it's just sort of innate for the most part.

Coding can be learned.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#143

Post by scaleout1 »

tupshin wrote: post #141
matthewrudy wrote: post #49

Earlier quoted context omitted.

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

Recipe: Create a global Cassandra cluster with regional datacenters. Use one keyspace per region Use per-keyspace replication to only replicate that region's data locally, and to one or more additional datacenters Have stateless app servers colocated with Cassandra in each DC handling all local traffic Run spark on top of Cassandra to do analytics, or to do the etl to a dedicated analytics system Optionally have a si…

Thats an interesting approach. One question though, in our use case user often travel from city to city and country to country. How do you model that if you are only using local DC and local replications?

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#144

Post by rantanplan »

dominotw wrote: post #140
rantanplan wrote: post #139

Earlier quoted context omitted.

NoSQL DBs usually target distributed environments. So... enter CAP theorem. There's no free lunch. People think we can simply throw away half a century's worth of science because JSON and schemaless are teh awesome derp derp. Implementation is surely an issue, if you take into account that the mongodb guys had to acquire another company [1] in order to overcome their abysmal write performance. And yet there were peop…

> NoSQL DBs usually target distributed environments. So... enter CAP theorem. Surely FB is not running MYSQL on a single machine. Perhaps i am misunderstanding what you are saying but saying SQL db's dont face the issues of distribution seems a little strange. Distribution comes into picture from shape and size of the data not data saving/retrieval techniques. yea?

FB and all big companies are a very bad example. They have ton of resources and usually they don't use vanilla products, since they have the engineering capacity to support their own forked versions. e.g. see their own version of PHP.

Also distributing reads is easy, writes... not so much. NoSQL systems usually offer distributed writes with the caveat of eventual consistency. RDBMS have referential integrity and other constraints which by definition cannot migrate into a distributed environment. Or at least there's not a one size fits all solution.

> Distribution comes into picture from shape and size of the data not data saving/retrieval techniques. yea?

Most definitely not. It has nothing to do with the shape and size of data. Also.. there's not such thing as "distribution" in our context. Only "distributed", from "distributed computing"[1] and it's everything to do about data saving and retrieval :)

[1] https://en.wikipedia.org/wiki/Distributed_computing

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#145

Post by dang »

candidates wrote: post #129
dkarapetyan wrote: post #23

Earlier quoted context omitted.

That's all bloat. Pure and simple. At the end of the day Uber just does routing and basic allocation. It's a simple operations problem that has been solved since the 70s and no one back then needed ELK, Docker, Cassandra, etc. I've seen this bloat everywhere. It is usually a result of internal politics and posturing by management types. The kinds of people Steve Jobs would have called B and C players. Now the actual…

I disagree. Even on hacker news, people rarely express such absurd things with so little confidence. You fail to take into account many of the following: * Extremely high volume. Uber has indicated elsewhere that they receive upwards of a few hundred thousand requests per second on just one service. Please show me the logistics stack that did this in the 70s. * Yes, building the first version of something is extremel…

It's important to counter the trivializing sort of dismissal that people often post to HN (the old "I could build Twitter in a weekend" and whatnot). We want the culture to move more toward thoughtful, substantive critique. So your detailed argument here, based on experience, is valuable. Please don't spoil it by becoming uncivil like this:

> I guess it'd be wasted on you because you're pretty convinced you already solved these problems in the 70s, so why am i wasting my breath

With that your comment does more harm than good: it poisons the atmosphere and detracts from your substantive contribution.

You're definitely not "wasting your breath" even if you fail to persuade the other person not to be snarkily dismissive, because the real audience for a comment like yours is everybody else: i.e. the rest of us who are curious about how (in this case) Uber operates and why things might be the way they are. That audience needs to see both good information about the challenges involved (as opposed to this-has-been-trivial-since-the-70s) and a good example of how to patiently respond to a trivializing comment with a thoughtful one. It's bad if, instead, you give us a reason to wince and an example of replying to a dismissive comment with a rude one.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#146

Post by dominotw »

rantanplan wrote: post #144
dominotw wrote: post #140

Earlier quoted context omitted.

> NoSQL DBs usually target distributed environments. So... enter CAP theorem. Surely FB is not running MYSQL on a single machine. Perhaps i am misunderstanding what you are saying but saying SQL db's dont face the issues of distribution seems a little strange. Distribution comes into picture from shape and size of the data not data saving/retrieval techniques. yea?

FB and all big companies are a very bad example. They have ton of resources and usually they don't use vanilla products, since they have the engineering capacity to support their own forked versions. e.g. see their own version of PHP. Also distributing reads is easy, writes... not so much. NoSQL systems usually offer distributed writes with the caveat of eventual consistency. RDBMS have referential integrity and othe…

>RDBMS have referential integrity and other constraints which by definition cannot migrate into a distributed environment.

so,

Use RDBMS if your data can be handled by a single machine( or have the resources of FB) ? '99% ppl need RDBMS' argument boils down to 99% of ppl have data that can be handled by a single machine RDBMS.

Is that a good conclusion?

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#147

Post by matthewrudy »

praneshp wrote: post #136
matthewrudy wrote: post #124

Earlier quoted context omitted.

Surprisingly although we have user accounts implemented, most of our customers don't use them. But yes, as it stands, if you create a user account it is distinct per country.

Why would having federated logins be difficult to implement?

It's just not important enough to do.

Startup resources, business priorities, customer behaviour...

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#148

Post by matthewrudy »

brianwawok wrote: post #65
matthewrudy wrote: post #49

Earlier quoted context omitted.

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

If you are doing devops, wouldn't deploying to 1 data center vs 20 be no more work? It's just a loop around a script.

There's the other aspect too, which is each country has their own cluster, so they can (and do) think of excuses to hold back updates just for them.

So software gets out of sync, the mobile app releases need to take that into account too.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#149

Post by r2dnb »

orf wrote: post #97
r2dnb wrote: post #76

Earlier quoted context omitted.

>You have no clue what SQL or ACIDity is. That's quite an attack. I trained for an Expert SQL certification from Microsoft back then, when I was writing 3000K+ long stored procedures to migrate an Access application at a fortune 40 company. So I know what it is and I know quite a good deal about RDBMS. I'm not among those who criticize what they don't know. Regarding the gist of your comment on NoSQL, I haven't been…

> 3000K+ long stored procedures You wrote a 3,000,000 lines long stored procedure in Access? That's ridiculous and it's hardly a testament to "how much you know about RDBMS".

Wow, I meant 3K thanks for pointing this out ;) And I didn't wrote the Access application, I was migrating it.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#150

Post by 10921809211 »

dang wrote: post #145
candidates wrote: post #129

Earlier quoted context omitted.

I disagree. Even on hacker news, people rarely express such absurd things with so little confidence. You fail to take into account many of the following: * Extremely high volume. Uber has indicated elsewhere that they receive upwards of a few hundred thousand requests per second on just one service. Please show me the logistics stack that did this in the 70s. * Yes, building the first version of something is extremel…

It's important to counter the trivializing sort of dismissal that people often post to HN (the old "I could build Twitter in a weekend" and whatnot). We want the culture to move more toward thoughtful, substantive critique. So your detailed argument here, based on experience, is valuable. Please don't spoil it by becoming uncivil like this: > I guess it'd be wasted on you because you're pretty convinced you already s…

This is an non-exhaustive list of technologies that the article mentions:

"Terraform, Schemaless, Riak, Cassandra, Hadoop, Redis, Twemproxy, Celery, Kafka, Elasticsearch, Logstash, Kibana, Docker, Mesos, Aurora, HAProxy, Hyperbahn, Ringpop, TChannel, Nginx, Thrift, Protobuf, Phabricator, OpenGrok, Packer, Vagrant, Boto, Unison, Jenkins, Clusto, Puppet, Grafana, Storm, Spark, React, SVG, Canvas 2D, Gufaru, DropWizard"

How are you going to discuss all that in a substantive manner? All anyone can do in the limited time frame of a HN discussion is draw parallels to previous job experiences or previous user experiences.

My own experiences are more aligned with the sentiment expressed in the "dismissal" comment.

Post reply on HN