Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Uber Engineering Tech Stack, Part I: The Foundation

eng.uber.com

Post reply on HN
Print view View original

61–70 of 194 posts

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#61

Post by r2dnb »

joeblau wrote: post #55
NotQuantum wrote: post #33

Uber is really strapped for engineering talent. Especially when it comes for SRE. Myself and many friends working SRE at various Bay Area companies get consistently hit up for free lunches and interviews. It's really weird considering that their stack doesn't NEED to be this complex....

It probably could be more simplistic. It seems like with enough engineers every company I've ever worked at eventually ends up using every technology they can because of the one thing it does well.

>because of the one thing it does well.

This "one thing it does well" business is then presented as : "using the right tool for the right job" and it's difficult to argue against that because the counterpart can easily deride you as a fanatic of some technology, someone not objective enough, etc...

It is however interesting that we used relational databases for virtually everything for decades even though SQL is suboptimal at most things if we take them in isolation. Some will argue that people are now realizing their mistake, but the truth is these companies were successful and we were all getting our paychecks. (PS: I choose to use NoSQL for virtually all my projects)

The real driver shouldn't be the one thing it does well. Many times - if not most of the time - it's preferable to use a tool optimal for the most important parts and suboptimal for the rest. I personally prefer to provision two more instances, than to add two more technology stacks.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#62

Post by stygiansonic »

golergka wrote: post #53
Animats wrote: post #25

It's interesting that they don't break the problem apart geographically. It's inherent in Uber that you're local. But their infrastructure isn't organized that way. Facebook originally tried to do that, then discovered that, as they grew, friends weren't local. Uber doesn't need to have one giant worldwide system. Most of their load is presumably positional updates. Uber wants both customers and drivers to keep their…

> Facebook originally tried to do that, then discovered that, as they grew, friends weren't local. Interesting. Do you have a link on that?

Basically, it was a way to partition users as some form of horizontal scaling. See these two Quora posts:

https://www.quora.com/How-did-Zuckerberg-code-Facebook-so-fa...

https://www.quora.com/In-what-order-did-Facebook-open-to-col...

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#63

Post by rantanplan »

r2dnb wrote: post #61
joeblau wrote: post #55

Earlier quoted context omitted.

It probably could be more simplistic. It seems like with enough engineers every company I've ever worked at eventually ends up using every technology they can because of the one thing it does well.

>because of the one thing it does well. This "one thing it does well" business is then presented as : "using the right tool for the right job" and it's difficult to argue against that because the counterpart can easily deride you as a fanatic of some technology, someone not objective enough, etc... It is however interesting that we used relational databases for virtually everything for decades even though SQL is subo…

> It is however interesting that we used relational databases for virtually everything for decades even though SQL is suboptimal at most things

You have no clue what SQL or ACIDity is. For 99% of the cases SQL/RDBMS is the right choice. You probably think you belong in that 1%, but from your comment, I suspect you do not.

> I choose to use NoSQL for virtually all my projects

That's because you have no important data to store.

When you get to store data that are important to your customers you're gonna have a big revelation.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#64

Post by wheresmypwd »

e1g wrote: post #21

I'd love to know how many people are responsible for devops/operations/app at various stages of any company's journey. Wikipedia says Uber employs 6,500 people so if even 15% of that is on the tech side of the business that's still 1,000+ people allocated to tech. I think this metric would be a useful reality check for a "modern" SaaS project with 3-10 people that's trying to emulate a backend structure similar to th…

Assume it's a team of 3-8 eng/ops/Devops managing each complex tech on the list, then 5-10 more teams of 5-20 engineers managing areas of the product, and interfacing with the infra teams to get things done.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#65

Post by brianwawok »

matthewrudy wrote: post #49
Animats wrote: post #25

It's interesting that they don't break the problem apart geographically. It's inherent in Uber that you're local. But their infrastructure isn't organized that way. Facebook originally tried to do that, then discovered that, as they grew, friends weren't local. Uber doesn't need to have one giant worldwide system. Most of their load is presumably positional updates. Uber wants both customers and drivers to keep their…

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

If you are doing devops, wouldn't deploying to 1 data center vs 20 be no more work? It's just a loop around a script.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#66

Post by anonx »

rantanplan wrote: post #63
r2dnb wrote: post #61

Earlier quoted context omitted.

>because of the one thing it does well. This "one thing it does well" business is then presented as : "using the right tool for the right job" and it's difficult to argue against that because the counterpart can easily deride you as a fanatic of some technology, someone not objective enough, etc... It is however interesting that we used relational databases for virtually everything for decades even though SQL is subo…

> It is however interesting that we used relational databases for virtually everything for decades even though SQL is suboptimal at most things You have no clue what SQL or ACIDity is. For 99% of the cases SQL/RDBMS is the right choice. You probably think you belong in that 1%, but from your comment, I suspect you do not. > I choose to use NoSQL for virtually all my projects That's because you have no important data…

>> You have no clue what SQL or ACIDity is.

"NOSQL" doesn't mean "no ACID". There are plenty of NOSQL DBs that are ACID compliant. And SQL is not the only way to write your queries. There are a lot more QLs.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#67

Post by karmelapple »

Animats wrote: post #29
dkarapetyan wrote: post #23

Earlier quoted context omitted.

That's all bloat. Pure and simple. At the end of the day Uber just does routing and basic allocation. It's a simple operations problem that has been solved since the 70s and no one back then needed ELK, Docker, Cassandra, etc. I've seen this bloat everywhere. It is usually a result of internal politics and posturing by management types. The kinds of people Steve Jobs would have called B and C players. Now the actual…

I have to admit, I could see this running for a city the size of SF on a desktop machine under the table at the taxi depot. Uber has 11,000 drivers in SF, but probably only a few thousand are on at any one time. A ride takes a few minutes, so if you figure 3,000 active drivers and 4 rides per hour, that's only about 3 ride transactions per second. You have a transaction at ordering, one at ride start, and one at ride…

I'm wondering why matching is nontrivial.

When I use these services, it doesn't always give me the closest car of the ones shown. It's all about which driver accepted my request first. Isn't it a fairly easy calculation of "all cars within X km or the nearest Y cars", and then whichever driver taps first gets matched with you?

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#68

Post by brianwawok »

rantanplan wrote: post #63
r2dnb wrote: post #61

Earlier quoted context omitted.

>because of the one thing it does well. This "one thing it does well" business is then presented as : "using the right tool for the right job" and it's difficult to argue against that because the counterpart can easily deride you as a fanatic of some technology, someone not objective enough, etc... It is however interesting that we used relational databases for virtually everything for decades even though SQL is subo…

> It is however interesting that we used relational databases for virtually everything for decades even though SQL is suboptimal at most things You have no clue what SQL or ACIDity is. For 99% of the cases SQL/RDBMS is the right choice. You probably think you belong in that 1%, but from your comment, I suspect you do not. > I choose to use NoSQL for virtually all my projects That's because you have no important data…

Why do you think NoSQL means not important data?

MongoDB eats data but I am not sure every NoSQL database does.

NoSQL does add certain kinds of complexity, but also simplifies certain problems. Depends where your hard problems are..

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#69

Post by rantanplan »

anonx wrote: post #66
rantanplan wrote: post #63

Earlier quoted context omitted.

> It is however interesting that we used relational databases for virtually everything for decades even though SQL is suboptimal at most things You have no clue what SQL or ACIDity is. For 99% of the cases SQL/RDBMS is the right choice. You probably think you belong in that 1%, but from your comment, I suspect you do not. > I choose to use NoSQL for virtually all my projects That's because you have no important data…

>> You have no clue what SQL or ACIDity is. "NOSQL" doesn't mean "no ACID". There are plenty of NOSQL DBs that are ACID compliant. And SQL is not the only way to write your queries. There are a lot more QLs.

Even though what you say it's true, my comment is still correct and relevant to the OP.

Also, of the NoSQL DBs that support ACID, I wouldn't touch them for any serious work or primary data at least. None of them are battle-tested in the same way Postgres is for example.

And again, people who really need these type of DBs fall into the 1%, and I'm being very generous.

Re: The Uber Engineering Tech Stack, Part I: The Foundation

#70

Post by rantanplan »

brianwawok wrote: post #68
rantanplan wrote: post #63

Earlier quoted context omitted.

> It is however interesting that we used relational databases for virtually everything for decades even though SQL is suboptimal at most things You have no clue what SQL or ACIDity is. For 99% of the cases SQL/RDBMS is the right choice. You probably think you belong in that 1%, but from your comment, I suspect you do not. > I choose to use NoSQL for virtually all my projects That's because you have no important data…

Why do you think NoSQL means not important data? MongoDB eats data but I am not sure every NoSQL database does. NoSQL does add certain kinds of complexity, but also simplifies certain problems. Depends where your hard problems are..

Because in the 2 decades I'm in the industry I see RDBMS make the world spin and NoSQL DBs destroying companies and families.

MongoDB and CouchDB eat data for breakfast, I know that from 1st hand experience. And all the others DBs that claim that do not keep cropping up in Aphyr's blog.

I ain't saying that all NoSQL dbs are useless. I'm just saying that proposing and choosing an RDBMS solution is going to be the right choice for 99% of the projects.

Yes, most people think that they belong in that 1% where they have the infrastructure problems and big data of Google, FB and Twitter but.... they don't.

Post reply on HN