Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

Post reply on HN
Print view View original

81–90 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#81

Post by solatic »

gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

You're missing the point where you have a flexible, schemaless design while you're specing out the service and seeing if it's even necessary, viable, and will be needed for a long time for a long-lived system.

Properly encoding rigidity through type systems, SQL checks/triggers/conditions, etc is hard. It takes a really long time to really iron out all the string and integer/double/long typing out of your system, let alone do it in a way which matches up properly with your backing datastore. Once you've got it set up and nailed down with tests, then you're golden, but that's a cost that is usually not worth paying until long-term need is determined.

Re: What I Wish I Had Known Before Scaling Uber [video]

#82

Post by novaleaf »

gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

Camps 1 and 2 are not mutually exclusive (well, except your inflammatory "naively" comment).

Rest/JSON is a well understood, broadly adopted, low friction RPC format.

NoSQL is not always MongoDB (for example, Google Datastore is ACID compliant), and schema enforcement via an ORM layer I would argue is actually a good thing, as it provides schema validation at compile time.

Re: What I Wish I Had Known Before Scaling Uber [video]

#83

Post by new_hackers »

eeZah7Ux wrote: post #42
jon-wood wrote: post #25

Earlier quoted context omitted.

With 800 services and 2000 programmers you already have a deployment problem.

With 800 services and 2000 programmers you already have a problem.

With programers you already have a problem.

Re: What I Wish I Had Known Before Scaling Uber [video]

#84

Post by maxlamb »

gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

Although companies like Snowflake are bridging the gap between the two camps, with products that can natively ingest JSON and Avro, and makes it queryable via SQL allowing for joins etc.

Re: What I Wish I Had Known Before Scaling Uber [video]

#85

Post by jrochkind1 »

mranney wrote: post #76
paukiatwee wrote: post #37

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…

Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

that sounds nightmarish

Re: What I Wish I Had Known Before Scaling Uber [video]

#88

Post by fail2fail2ban »

adekok wrote: post #77
gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

> people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and Yes, and no. Yes: REST / JSON is nice. I've used them widely as a kind of cross-platform compatibility layer. i.e. instead of exposing SQL or something similar, the API is all REST / JSON. That lets everyone use tools they're familiar with, without learning about implementation details. The RES…

> SQL databases should be conservative in what they accept. Once you've inserted crap into the DB, it's hard to fix it.

You can make your schema very light and accepting almost like NoSQL which is how you get into the situation you described; the solution is to use stricter schema. That and it helps to hire a full time data engineer/administrator.

> Using NoSQL as a fast cache

I'd rather use caching technology, specifically designed for caching, like Redis or Varnish or Squid.

Re: What I Wish I Had Known Before Scaling Uber [video]

#89

Post by fail2fail2ban »

maxlamb wrote: post #84
gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

Although companies like Snowflake are bridging the gap between the two camps, with products that can natively ingest JSON and Avro, and makes it queryable via SQL allowing for joins etc.

It's possible to use SQL to retrieve data from a text file; that's not the same as using SQL backed by an actual relational database engine (and a well designed schema) that has been perfected over many decades.

Re: What I Wish I Had Known Before Scaling Uber [video]

#90

Post by dastbe »

jrochkind1 wrote: post #85
mranney wrote: post #76

Earlier quoted context omitted.

Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

that sounds nightmarish

Why? I'm assuming their highish engineer/service ratio is because their services do less individually.

Anecdotally, I've worked on services that ran tens of thousands of instances across the world. You build the tools to manage them and it works very well.

Post reply on HN