Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I Wish I Had Known Before Scaling Uber [video]

youtube.com

Post reply on HN
Print view View original

171–180 of 284 posts

Re: What I Wish I Had Known Before Scaling Uber [video]

#171

Post by cortesoft »

danek wrote: post #68
cortesoft wrote: post #49

Earlier quoted context omitted.

The breakage rate per new feature is fairly constant; if you release 7 new features once a week or 1 new feature once a day, you will have the same number of issues. The question then is; is it easier to deal with all the issues at once, or a smaller number of issues every day?

In addition to that, I would also worry about interactions between issues. I tend to lean towards spreading out the issues over time to make the eventual diagnosis easier. Occasionally we'll have a problem where we cannot deploy to production for several days (normally it's once a day). A massive inventory of ready-to-deploy features builds up. When we do finally deploy, this deluge of features and fixes creates new…

Absolutely. Confounding issues are the biggest problem, even more so than sheer volume in a small time frame.

Re: What I Wish I Had Known Before Scaling Uber [video]

#172

Post by ma2rten »

Scarblac wrote: post #139
lnanek2 wrote: post #121

Earlier quoted context omitted.

I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our…

Why would you make that a separate service when e.g. one query on a PostGIS table can do that?

You might want to add functionality such as caching or business rules.

A better question would be why not write a module or class? There a pros and cons to either, but advantages include: better monitoring and alerting, easier deployments and rollbacks, callers can timeout and use fallback logic, you can add load balancing and scale each service separately, it's easy to figure out which service uses what resources, it makes it easy to write some parts of application in other programming languages, different teams can work on different parts of the application independently as long as they agree on an API.

Re: What I Wish I Had Known Before Scaling Uber [video]

#173

Post by BinaryIdiot »

mranney wrote: post #76
paukiatwee wrote: post #37

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the sam…

Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

Whoa. That's is an insanely large amount of applications. I'm assuming that's essentially one function per microservice which is one of the huge do-not-do of microservices as it's just a complete waste of time, resources, etc.

I would love to hear a breakdown. This sounds like a nightmare to maintain and test.

Re: What I Wish I Had Known Before Scaling Uber [video]

#174

Post by rando832 »

agentultra wrote: post #128

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture: Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes. Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still…

I've never heard the term over-fill data, and google seems to have no idea of it in reference to software.

Re: What I Wish I Had Known Before Scaling Uber [video]

#175

Post by brown9-2 »

user5994461 wrote: post #168
mranney wrote: post #76

Earlier quoted context omitted.

Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

>>> Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances. Time to forbid adding more stuff and start cleaning. (12) [...] perfection has been reached not when there is nothing left to add, but when there is nothing left to take away. https://tools.ietf.org/html/rfc1925

This type of comment presumes that there is a right number of services for a company to run.

What number is that?

Re: What I Wish I Had Known Before Scaling Uber [video]

#176

Post by taeric »

user5994461 wrote: post #169
Scarblac wrote: post #139

Earlier quoted context omitted.

Why would you make that a separate service when e.g. one query on a PostGIS table can do that?

One query on a table is the exact same thing as a HTTP GET call on a service.

Strictly this is not necessarily relevant. You can easily roll that table hit into another db hit you were already making. Can't do that with services.

Re: What I Wish I Had Known Before Scaling Uber [video]

#177

Post by ChuckMcM »

hinkley wrote: post #153
ChuckMcM wrote: post #137

Earlier quoted context omitted.

That conclusion is well founded. We correlated issues at Blekko across a lot of different factors, the one that always held was code or configuration changes. Not too surprising in the large but definitely confirmed by the data.

So I should fire all my developers is what you're saying... Man, I'm gonna save so much money.

That does save a lot of money :-) The trick is managing the rate of change and the risk of disruption. If you manage it to no risk you end up changing too slowly, if you manage it to close to the risk you end up with unexpected downtime and other customer impacting events. Understanding where you are between no risk and certain doom really only comes with experience.

Re: What I Wish I Had Known Before Scaling Uber [video]

#178

Post by prodigal_erik »

novaleaf wrote: post #82
gtrubetskoy wrote: post #57

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to…

Camps 1 and 2 are not mutually exclusive (well, except your inflammatory "naively" comment). Rest/JSON is a well understood, broadly adopted, low friction RPC format. NoSQL is not always MongoDB (for example, Google Datastore is ACID compliant), and schema enforcement via an ORM layer I would argue is actually a good thing, as it provides schema validation at compile time.

The longer a database has existed, the more likely somebody in the company wrote something crucial that accesses it without your knowledge and without going through your ORM (usually because your ORM isn't implemented for the language they're using, or it emits bad queries for their use case). Sanity checks that aren't enforced by the database can't be relied on to be up to date or even happen at all.

Re: What I Wish I Had Known Before Scaling Uber [video]

#179

Post by threeseed »

agentultra wrote: post #128

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture: Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes. Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still…

I have no idea why people keep thinking microservices is all about scalability. Almost like they've never worked on a problem with them before.

Microservices is all about taking a big problem and breaking it down into smaller components, defining the contracts between the components (which an API is), testing the components in isolation and most importantly deploying and running the components independently.

It's the fact that you can make a change to say the ShippingService and provided that you maintain the same API contract with the rest of your app you can deploy it as frequently as you wish knowing full well that you won't break anything else.

It also aligns better with the trend towards smaller container based deployment methods.

Re: What I Wish I Had Known Before Scaling Uber [video]

#180

Post by agentultra »

rando832 wrote: post #174
agentultra wrote: post #128

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture: Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes. Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still…

I've never heard the term over-fill data, and google seems to have no idea of it in reference to software.

Ah... it's almost the reason for GraphQL.

Basically if you're building a hypermedia REST API you return an entity or collection of entities whose identifiers allow you to fetch them from the service like so:

    {"result": "ok!"
     "users": ["/users/123", "/users/234"]
    }
The client, if interested, can use those URLs to fetch the entities from the collection that it is interested in. This poses a problem for mobile clients where you want to minimize network traffic... so you over-fill your data collection by returning the full entity in the collection.

    {"result": "ok!"
     "users": [{"id": 123, "name": "Foo"}, {"id": 234, "name": "Bar"}]
    }
The trade off is that you have to fetch the data for every entity in the collection, the entities they own, etc; and ship one really large response. The client would receive this giant string of bytes even if the client was only interested in a subset of the properties in the collection.

GraphQL does away with this problem on the client side rather elegantly by allowing the client to query for the specific properties they are interested in. You don't end up shipping more data than is necessary to fulfill a query. Nice!

... but the tradeoff there is that you lose the domain representation in your URLs since there are none.

Post reply on HN