Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing SmartStack: Service Discovery in the Cloud

nerds.airbnb.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Introducing SmartStack: Service Discovery in the Cloud

#31

Post by threeseed »

eliwjones wrote: post #30

I don't know... sounds like they just shifted the 'single point of failure' to the Zookeeper cluster. Is it somehow sexier to have your SPOF be things running Zookeeper instead of things running loadbalancer or DNS software?: "The achilles heel of SmartStack is Zookeeper. Currently, the failure of our Zookeeper cluster will take out our entire infrastructure. Also, because of edge cases in the zk library we use, we m…

If your load balancer/DNS goes down then every service won't be able to find other services.

With this approach everything will still work.

Re: Introducing SmartStack: Service Discovery in the Cloud

#32

Post by eliwjones »

threeseed wrote: post #31
eliwjones wrote: post #30

I don't know... sounds like they just shifted the 'single point of failure' to the Zookeeper cluster. Is it somehow sexier to have your SPOF be things running Zookeeper instead of things running loadbalancer or DNS software?: "The achilles heel of SmartStack is Zookeeper. Currently, the failure of our Zookeeper cluster will take out our entire infrastructure. Also, because of edge cases in the zk library we use, we m…

If your load balancer/DNS goes down then every service won't be able to find other services. With this approach everything will still work.

According to the quote I provided from the article, failure of the Zookeeper cluster will bring down their entire architecture.

Re: Introducing SmartStack: Service Discovery in the Cloud

#33

Post by hkarthik »

grosskur wrote: post #26
hkarthik wrote: post #13

Great post. This is interesting due to similar discussions we're having at work about moving from a monolithic Rails app architecture to an SOA. I'm curious though, what does the local developer environment look like when you run an SOA of this complexity? Does everyone needs to run a series of Vagrant VMs/Docker containers to have a fully functional local version of the application running?

One approach is stubbing out the services you don't need. This article by a Heroku engineer describes how to do this at the Rack level: https://brandur.org/service-stubs Another approach is creating a set of shared services that developers can use rather than deploying their own instances. This article by a LinkedIn engineer describes their internal Quick Deploy system: http://engineering.linkedin.com/developer-produ…

yeah we've used stubs in the past and it's often awkward and creates silos within the team. I'm not a huge fan of this approach.

Re: Introducing SmartStack: Service Discovery in the Cloud

#34

Post by hkarthik »

igor47 wrote: post #19
hkarthik wrote: post #13

Great post. This is interesting due to similar discussions we're having at work about moving from a monolithic Rails app architecture to an SOA. I'm curious though, what does the local developer environment look like when you run an SOA of this complexity? Does everyone needs to run a series of Vagrant VMs/Docker containers to have a fully functional local version of the application running?

At Airbnb, we've moved to a single Vagrant vm for our dev environment. We configure it using the same Chef code we use to build production -- the cookbook that installs search in production also installs it in dev. We actually usually avoid SmartStack in dev. The rule is, your service always listens on it's SmartStack port. So, for instance, search listens on port 5678 on it's backends; in prod, consumers of search w…

That makes a lot of sense. I know Yammer does something similar, albeit with Puppet instead of Chef.

Sounds like this could be a whole blog post on its own. Over-complicating the local developer story is a big reason that us and many other firms have punted on SOA in the first place.

Re: Introducing SmartStack: Service Discovery in the Cloud

#35

Post by gregwebs »

igor47 wrote: post #22

Hi guys! I'm one of the primary authors of SmartStack. Happy to answer any questions that aren't covered in the blog post. We're also doing a Tech Talk on SmartStack today at Airbnb HQ; stop by if you're in SF: https://www.airbnb.com/meetups/33925h2sx-tech-talk-smartstac...

As a thought experiment, what do you think of trying to make this completely de-centralized? Each service has its own synapse, and the services form a nerve for that service. They communicate status directly to the app servers, which form their own nerve.

Boot-strapping would of course be a problem, so you would still need a centralized nervous system for that, but you could survive its failure.

Re: Introducing SmartStack: Service Discovery in the Cloud

#36

Post by vzctl »

druiid wrote: post #23
mLewisLogic wrote: post #17

Earlier quoted context omitted.

Having a central load balancer is going to turn into a nightmare once you start managing a reasonable number of servers. Hardware goes bad (especially in the cloud), and having a single point of failure leaves you at it's mercy.

A load balancer should never be a single point of failure. You should always have multiples. Also, if the response to this is then 'but it's still a central point of failure', they haven't really removed that in this solution. If the zookeeper cluster dies you lose everything. Generally if a clustered load-balancer dies and another takes over there's a half second or a couple seconds of transition, but you're back up…

by the way keepalived >= 1.2.8 also supports vrrp over unicast

Re: Introducing SmartStack: Service Discovery in the Cloud

#37

Post by nl »

threeseed wrote: post #29
nl wrote: post #28

So.. SOA. Everyone is doing it, but I never see details about how the services are wired together. Do people use ESBs, or directly wire up services to each other?

Services are generally using HTTP/REST these days.

Yes. Most (all?) ESBs support (and indeed encourage) REST.

I'm pretty sure there has never been an ESB that doesn't support HTTP.

My question is if people are actually using them to wire system together (outside corporate environments, where considerations are different).

Re: Introducing SmartStack: Service Discovery in the Cloud

#38

Post by druiid »

vzctl wrote: post #36
druiid wrote: post #23

Earlier quoted context omitted.

A load balancer should never be a single point of failure. You should always have multiples. Also, if the response to this is then 'but it's still a central point of failure', they haven't really removed that in this solution. If the zookeeper cluster dies you lose everything. Generally if a clustered load-balancer dies and another takes over there's a half second or a couple seconds of transition, but you're back up…

by the way keepalived >= 1.2.8 also supports vrrp over unicast

Indeed it does! I use it for less complicated fail-over situations myself, but if you start to need more complicated topologies (or something which has good interaction with IPVS/LVS) then Corosync!

Re: Introducing SmartStack: Service Discovery in the Cloud

#39

Post by igor47 »

gregwebs wrote: post #35
igor47 wrote: post #22

Hi guys! I'm one of the primary authors of SmartStack. Happy to answer any questions that aren't covered in the blog post. We're also doing a Tech Talk on SmartStack today at Airbnb HQ; stop by if you're in SF: https://www.airbnb.com/meetups/33925h2sx-tech-talk-smartstac...

As a thought experiment, what do you think of trying to make this completely de-centralized? Each service has its own synapse, and the services form a nerve for that service. They communicate status directly to the app servers, which form their own nerve. Boot-strapping would of course be a problem, so you would still need a centralized nervous system for that, but you could survive its failure.

i think that another project that aims to address this was announced same day as smartstack: http://www.serfdom.io/

i haven't read much beyond the splash page, but hashicorp generally does great work.

Post reply on HN