Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing SmartStack: Service Discovery in the Cloud

nerds.airbnb.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Introducing SmartStack: Service Discovery in the Cloud

#11

Post by igor47 »

superbaconman wrote: post #10
devonbleak wrote: post #7

Earlier quoted context omitted.

With a service-oriented architecture (SOA) you basically break your application into distinct services that run across different machines. But you also need a way to find and connect to those services (service discovery). This provides a way of doing that using ZooKeeper and local HAProxy processes.

So I register a new service (or process) to run on metal using Chef. Then Chef installs the service and updates the nerve config running on that metal device. Nerve then keeps a central Zookeeper updated with the status of its local services? Synapse checks for available services in zookeeper that your app may need to use, and then configures a local load balancer for any requests you make?

That's the idea! If you're using Chef, check out the cookbook for SmartStack; you keep a small hash of configuration information per service, and we take care of the rest. https://github.com/airbnb/smartstack-cookbook

Re: Introducing SmartStack: Service Discovery in the Cloud

#12

Post by frontendbeauty »

superbaconman wrote: post #10
devonbleak wrote: post #7

Earlier quoted context omitted.

With a service-oriented architecture (SOA) you basically break your application into distinct services that run across different machines. But you also need a way to find and connect to those services (service discovery). This provides a way of doing that using ZooKeeper and local HAProxy processes.

So I register a new service (or process) to run on metal using Chef. Then Chef installs the service and updates the nerve config running on that metal device. Nerve then keeps a central Zookeeper updated with the status of its local services? Synapse checks for available services in zookeeper that your app may need to use, and then configures a local load balancer for any requests you make?

Yep.

Re: Introducing SmartStack: Service Discovery in the Cloud

#13

Post by hkarthik »

Great post. This is interesting due to similar discussions we're having at work about moving from a monolithic Rails app architecture to an SOA.

I'm curious though, what does the local developer environment look like when you run an SOA of this complexity? Does everyone needs to run a series of Vagrant VMs/Docker containers to have a fully functional local version of the application running?

Re: Introducing SmartStack: Service Discovery in the Cloud

#14

Post by chris_va »

I'm glad to see more cluster management software getting open sourced, and this is sort of on the right track.

However, looking at the design, this still has a long way to go. There are a lot of failure modes you guys haven't encountered yet, which will result in a few design tweaks. For example, what happens if your health checkers decide to start reporting garbage data (e.g. maybe they are too overloaded to properly perform health checks)? Or when you have a query of death being issued? Also, things like traffic sloshing can very quickly build resonant failures in a system like this.

(Source: many years working on Google infrastructure, including causing outages related to load balancing code)

Re: Introducing SmartStack: Service Discovery in the Cloud

#15

Post by druiid »

Cool stuff I think though that much of this can be handled with other ways of doing things (although obviously there is never one right way of doing these kinds of things). This application kit is one way of orchestrating service/server discovery. Another way, which I have implemented personally is to use a combination of mcollective and puppet (with puppet facts enabled). This allows you to defined roles for specific systems and run tasks against servers of that specific role type, keep track of which servers are that role type, connect them to a 'central' load-balancer and many other things.

This serves to solve most of the issues that this toolkit provides for, but likely would not be the good option for everyone. Just some info on at least one other way to deal with this stuff!

Re: Introducing SmartStack: Service Discovery in the Cloud

#16

Post by inopinatus »

BaconJuice wrote: post #5

Can someone explain to me what this really is? I don't follow and yes I read the intro.

As far as I can tell, here's my summary of the architecture I just read.

For service to service communication, they've deployed client-side load balancing & discovery using per-client HAProxy instances. Rather than have every client polling every possible service for health, Zookeeper is used as an endpoint status repository. The HAProxy configs are kept up to date using a tool called "Synapse" that queries ZK. ZK is kept up-to-date by their own health check service, "Nerve".

I've developed similar myself, using LVS, for a private Australian CDN. Nice to see the model generalised, robustified and open-sourced. There are possible issues relating to work levelling and spike management, but if it's working for AirBNB, great.

If you needed to describe this in an enterprise context, I'd tell them it's a distributed SOA broker. That's a gross oversimplification but the buzzword bingo'll satisfy most project managers.

For a middle-aged IT manager I'd say "it's like the Oracle Parallel Server client reliability model, only for web services rather than databases". Again an oversimplification, but they'd get the idea.

Re: Introducing SmartStack: Service Discovery in the Cloud

#17

Post by mLewisLogic »

druiid wrote: post #15

Cool stuff I think though that much of this can be handled with other ways of doing things (although obviously there is never one right way of doing these kinds of things). This application kit is one way of orchestrating service/server discovery. Another way, which I have implemented personally is to use a combination of mcollective and puppet (with puppet facts enabled). This allows you to defined roles for specifi…

Having a central load balancer is going to turn into a nightmare once you start managing a reasonable number of servers. Hardware goes bad (especially in the cloud), and having a single point of failure leaves you at it's mercy.

Re: Introducing SmartStack: Service Discovery in the Cloud

#18

Post by schleyfox »

hkarthik wrote: post #13

Great post. This is interesting due to similar discussions we're having at work about moving from a monolithic Rails app architecture to an SOA. I'm curious though, what does the local developer environment look like when you run an SOA of this complexity? Does everyone needs to run a series of Vagrant VMs/Docker containers to have a fully functional local version of the application running?

One of the neat tricks with using the localhost HAProxy is that developer mode can just be a single Vagrant VM with all services configured to run on the port that they would have in Synapse. With HAProxy all the production servers think they are talking to localhost, and in development they actually are. The Vagrant VM can then be configured/reconfigured using Chef and the production cookbooks (with some overrides in the roles or environments). It should also be possible (though not entirely trivial) to run some of the services in the cloud for development, allowing developers to switch in and out the components that they need to actively develop.

Re: Introducing SmartStack: Service Discovery in the Cloud

#19

Post by igor47 »

hkarthik wrote: post #13

Great post. This is interesting due to similar discussions we're having at work about moving from a monolithic Rails app architecture to an SOA. I'm curious though, what does the local developer environment look like when you run an SOA of this complexity? Does everyone needs to run a series of Vagrant VMs/Docker containers to have a fully functional local version of the application running?

At Airbnb, we've moved to a single Vagrant vm for our dev environment. We configure it using the same Chef code we use to build production -- the cookbook that installs search in production also installs it in dev.

We actually usually avoid SmartStack in dev. The rule is, your service always listens on it's SmartStack port. So, for instance, search listens on port 5678 on it's backends; in prod, consumers of search will find it on localhost at port 5678 via SmartStack. In dev, consumers will just find it at localhost port 5678 natively.

Re: Introducing SmartStack: Service Discovery in the Cloud

#20

Post by igor47 »

chris_va wrote: post #14

I'm glad to see more cluster management software getting open sourced, and this is sort of on the right track. However, looking at the design, this still has a long way to go. There are a lot of failure modes you guys haven't encountered yet, which will result in a few design tweaks. For example, what happens if your health checkers decide to start reporting garbage data (e.g. maybe they are too overloaded to properl…

What is traffic sloshing?

Good point on garbage data reporting; we do basic validation in synapse, here: https://github.com/airbnb/synapse/blob/master/lib/synapse/se...

We could probably do more there to ensure valid names, IPs and ports (matching against a regex should do it). Also, because of the built-in health checking in haproxy, just the presence of some invalid name in the list of machines doesn't mean that we're going to try to start sending traffic there.

Post reply on HN