Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

Post reply on HN
Print view View original

51–60 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#51

Post by fail2fail2ban »

esrauch wrote: post #11
tiborsaas wrote: post #7

Earlier quoted context omitted.

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you? EDIT: to downvoters: why shoot the messenger? :)

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

I have over 3 dozen private Git repos for personal amusement or utility projects for stuff that professional consumer software doesn't work as well as what I want. Then I have almost at least 10 repos on GutHub (not this userid). At my last work, 9 month contract, I had 4 I think, aside from our main team repos just to manage personal info documents, my bin folder, etc. It's easy for one developer to have several. On my Linodes I keep /etc in its own repo as well, and my VirtualBox and VMware VMs then inside those I keep several Git repos.

Edit: I remembered more personal projects, make that almost 3 dozen personal Git repos.

Re: What I Wish I Had Known Before Scaling Uber [video]

#53

Post by ramchip »

andy_ppp wrote: post #32
latch wrote: post #20

Earlier quoted context omitted.

Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent. I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of conc…

I always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free. The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.

I've given that kind of model a lot of thought too, but I did not find very satisfying solutions to these problems:

- How do you distribute the car actors on nodes, assuming the number of nodes is variable? (I think riak_core looks interesting, but it does not seem to have a way to guarantee unicity of something - it's rather built to replicate the data on multiple nodes for redundancy)

- What happens if a node fails? What mechanism is going to respawn the car actors on a different node? How do you ensure minimal downtime for the cars involved?

- What happens if there's a netsplit, e.g. how do you ensure no split brain where two nodes think they're responsible for a car?

It feels to me like the traditional erlang-process-per-request architecture coupled with a distributed store (riak or w/e) makes it possible to avoid the very difficult problem of ensuring one and only one actor per car in a distributed environment.

Re: What I Wish I Had Known Before Scaling Uber [video]

#54

Post by babyrainbow »

andrewvijay wrote: post #52

Highly recommended video. Lots of stuff he spoke are very relatable. Like having many repos , storing configs as a separate repo, politics by people, having a tracking system

Sorry, don't think so. I went through the slides and didn't find anything really interesting.

Re: What I Wish I Had Known Before Scaling Uber [video]

#57

Post by gtrubetskoy »

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to be using high level dynamic languages like Python and Ruby, and camp 2 will be more on the strongly typed side e.g. C/C++, Go, Java).

Re: What I Wish I Had Known Before Scaling Uber [video]

#58

Post by dorianm »

I got told by an Uber engineer that's because of their hyper-hyper growth their tech is basically the biggest mess possible.

That's also why that makes it an interesting place to work at and helped them achieve this growth.

Personnaly I think this should be made into a global app with no geo-fencing (e.g. Available everywhere basically).

Re: What I Wish I Had Known Before Scaling Uber [video]

#59

Post by KayEss »

gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

I still kind of think you can have great REST and JSON and make full use of a great RDBMS.

REST/JSON with strong schemas (and loose where that makes sense), using Postgres and C++.

Re: What I Wish I Had Known Before Scaling Uber [video]

#60

Post by iamnothere »

gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

Is your first example really "naive" though? In my experience, loose, flexible schemas and dynamic languages are very well suited to rapid early-stage development, much more so than rigid languages and schemas.

Sure, in the long term things should be refactored, structured, and optimized. But if you do that too soon you risk locking yourself out of potential value, as well as gold-plating things that aren't critical.

Post reply on HN