What happens to your cluster when your network experiences intermittent packet loss and your random UDP messages get lost? Nodes just start going down and up randomly? (For those of you going "So what, that's normal", this not a quality of an HA system)
Serf: A decentralized solution for service discovery and orchestration
31–40 of 48 posts
Re: Serf: A decentralized solution for service discovery and orchestration
#32This would really benefit from a "how does this relate to zookeeper". I think this is an entirely new service, with different technical insides, and trying to provide a higher level solution to what people usually cobble together with ZK.
But I'd be interested in comments from someone knowledgeable.
Edit: I see this is addressed at http://www.serfdom.io/intro/vs-zookeeper.html but it would be nice to have something more "just the facts" rather than arguing the serf is good.
Re: Serf: A decentralized solution for service discovery and orchestration
#33What happens to your cluster when your network experiences intermittent packet loss and your random UDP messages get lost? Nodes just start going down and up randomly? (For those of you going "So what, that's normal", this not a quality of an HA system)
I'd highly recommend taking a look at this page: http://www.serfdom.io/docs/internals/gossip.html. One of the great attributes of the gossip protocol is it is very robust to intermittent network failures. Under minimal packet loss conditions (tl;dr the systems is in fact designed with network errors in mind, as opposed to handling them being an afterthought.
Re: Serf: A decentralized solution for service discovery and orchestration
#34This would really benefit from a "how does this relate to zookeeper". I think this is an entirely new service, with different technical insides, and trying to provide a higher level solution to what people usually cobble together with ZK. But I'd be interested in comments from someone knowledgeable. Edit: I see this is addressed at http://www.serfdom.io/intro/vs-zookeeper.html but it would be nice to have something m…
In writing that section, we tried to provide "just the facts". If there is anything that seems wrong or misleading in any way, we'd like to know so that the page can be corrected. It is not our intention to say "Serf is good, ZooKeeper is bad". They are very different tools, and we are just trying to highlight the differences. In fact, we believe that the strongest use cases involve using those tools together.
Re: Serf: A decentralized solution for service discovery and orchestration
#35I'm jumping on a plane right now (a couple hours) but I'd be happy to answer any questions related to Serf once I land. Just leave them here and I'll give it my best shot! We've dreamt of something like Serf for quite awhile and I'm glad it is now a reality. Some recommended URLs if you're curious what the point is: "What is Serf?" http://www.serfdom.io/intro/index.html "Use Cases" http://www.serfdom.io/intro/use-cas…
A meta question if you will - I have often come across situations in work where "if only we had that tool". sometimes I have hacked something together, other times taken it further and tidied it up and released it. But this seems to have a large level of polish
so ...
When did you realise the need for surf
Did you work on it as a main project at some point or is it a side project
When and how did you decide to commit to getting this done
and the big one for me - tools are driven by a need, but often the need keeps coming while the time to build it diminishes. What strategies did you use to keep the plates spinning while building surf?
I think we can all mostly answer the questions - I just waant to know how different your answers are from say mine when I don't release two major OSS projects and you do.
cheers
Re: Serf: A decentralized solution for service discovery and orchestration
#36I'm jumping on a plane right now (a couple hours) but I'd be happy to answer any questions related to Serf once I land. Just leave them here and I'll give it my best shot! We've dreamt of something like Serf for quite awhile and I'm glad it is now a reality. Some recommended URLs if you're curious what the point is: "What is Serf?" http://www.serfdom.io/intro/index.html "Use Cases" http://www.serfdom.io/intro/use-cas…
I'm still looking for a very simple system that would let me do a live redeploy with a blocking database schema migration in between.
For that I (think I)'d need a system that will:
* start off with X nodes live in the load balancer
* trigger redeploys on half (or so) of them
* when half of the nodes have been redeployed, block and perform the database migration and trigger redeploys for the remaining server
* after the migration completed switch over the load balancer to the now updated half of servers
Is this something that you could orchestrate with Serf? Or am I looking in the wrong direction here.
Re: Serf: A decentralized solution for service discovery and orchestration
#37The package documentation for the `serf` library[1] looks really exciting. I've been wanting to make a distributed file synchronization tool, and perhaps this would be an excellent library to build it on. Question: as a relative networking idiot, how does NAT traversal fit into all of this? [1] - http://godoc.org/github.com/hashicorp/serf/serf
We designed the `serf` library to be able to be easily embedded, so hopefully it can be of some use. Unfortunately Serf does not make use of any sort of NAT traversal currently. We've open sourced the project hoping to get the community involved, and NAT traversal is something we'd gladly work with the community to get implemented.
The first building block, STUN, is implemented over here: https://github.com/ccding/go-stun
Re: Serf: A decentralized solution for service discovery and orchestration
#38I'm jumping on a plane right now (a couple hours) but I'd be happy to answer any questions related to Serf once I land. Just leave them here and I'll give it my best shot! We've dreamt of something like Serf for quite awhile and I'm glad it is now a reality. Some recommended URLs if you're curious what the point is: "What is Serf?" http://www.serfdom.io/intro/index.html "Use Cases" http://www.serfdom.io/intro/use-cas…
I'm still looking for a very simple system that would let me do a live redeploy with a blocking database schema migration in between. For that I (think I)'d need a system that will: * start off with X nodes live in the load balancer * trigger redeploys on half (or so) of them * when half of the nodes have been redeployed, block and perform the database migration and trigger redeploys for the remaining server * after…
You could build something like this on Serf, but it would take a little creativity. You might do something like this:
1) Send a "pre-deploy" event. Handler scripts use random number generator to decide which group they are in ("flip a coin" basically)
2) Half the nodes should transition to the "left" state, do the deploy and rejoin the cluster.
3) Once this is done, trigger the migration.
4) Flip the LB to the nodes that did the Join/Leave (you can potentially distinguish them using different role names, or by tracking who left and joined)
5) Run the "post-deploy". The other half of the nodes should now deploy
6) Update the LB to include everybody as the nodes leave/join
This is of course a rough sketch, it is certainly possible if tricky to build something like this.
Re: Serf: A decentralized solution for service discovery and orchestration
#39can we see serf as a kind of riak core but written in go?
Re: Serf: A decentralized solution for service discovery and orchestration
#40I'm jumping on a plane right now (a couple hours) but I'd be happy to answer any questions related to Serf once I land. Just leave them here and I'll give it my best shot! We've dreamt of something like Serf for quite awhile and I'm glad it is now a reality. Some recommended URLs if you're curious what the point is: "What is Serf?" http://www.serfdom.io/intro/index.html "Use Cases" http://www.serfdom.io/intro/use-cas…
A meta question if you will - I have often come across situations in work where "if only we had that tool". sometimes I have hacked something together, other times taken it further and tidied it up and released it. But this seems to have a large level of polish so ... When did you realise the need for surf Did you work on it as a main project at some point or is it a side project When and how did you decide to commit…
Great questions. I'll answer each in turn.
I want to mention the "polish": I personally don't believe in releasing an open source project without polish. If it is missing docs, its just not complete. If it is ugly, it is not complete. The technical aspects of Serf were done weeks ago. Getting the human side of things done took another few weeks (contracting designers and such).
> When did you realise the need for serf?
The need for something like Serf has existed since I started doing ops. Every time I hit something where I say to myself "why is this so hard/crappy" is when I write it down in my notebook for a future date. I then just think on the idea for awhile and eventually when I feel like I have a significantly better solution than what is out there already, I build it.
I decided to start building Serf when @armon started throwing gossip protocol academic papers at me. I realized he figured it out, this was clearly significantly better, so we started working on it.
> Did you work on it as a main project at some point or is it a side project?
To get it out the door we focus on it for some period of time. After it is shipped it is still what I would consider a "main project" but time is split between various projects.
> When and how did you decide to commit to getting this done?
A few weeks ago. It took about a month to build. Building it is easy. Figuring out WHAT to build... took a long time. I have to say I've had "service orchestration/membership" in my notebook for years.
> What strategies did you use to keep the plates spinning while building surf?
No good answer here, we just prioritize some things over others. Serf was our top priority this month.