Live data from Hacker News

Serf: A decentralized solution for service discovery and orchestration

serfdom.io

41–48 of 48 posts

Re: Serf: A decentralized solution for service discovery and orchestration

#41
post #11

I'm jumping on a plane right now (a couple hours) but I'd be happy to answer any questions related to Serf once I land. Just leave them here and I'll give it my best shot! We've dreamt of something like Serf for quite awhile and I'm glad it is now a reality. Some recommended URLs if you're curious what the point is: "What is Serf?" http://www.serfdom.io/intro/index.html "Use Cases" http://www.serfdom.io/intro/use-cas…

You need to do some repair on the website for iOS; the front page is white text on a white background.

Fixed. How does CSS work?

Re: Serf: A decentralized solution for service discovery and orchestration

#43
post #33

What happens to your cluster when your network experiences intermittent packet loss and your random UDP messages get lost? Nodes just start going down and up randomly? (For those of you going "So what, that's normal", this not a quality of an HA system)

I'd highly recommend taking a look at this page: http://www.serfdom.io/docs/internals/gossip.html . One of the great attributes of the gossip protocol is it is very robust to intermittent network failures. Under minimal packet loss conditions ( tl;dr the systems is in fact designed with network errors in mind, as opposed to handling them being an afterthought.

What you're saying is it's designed with the knowledge that it's going to cause false positives, and basically doesn't work well under anything more than minimal packet loss. I think this is probably an important factor to note in the description (and I still fail to see how this is considered highly available or fault tolerant, as described in Intro pages)

Re: Serf: A decentralized solution for service discovery and orchestration

#44
post #33

Earlier quoted context omitted.

I'd highly recommend taking a look at this page: http://www.serfdom.io/docs/internals/gossip.html . One of the great attributes of the gossip protocol is it is very robust to intermittent network failures. Under minimal packet loss conditions ( tl;dr the systems is in fact designed with network errors in mind, as opposed to handling them being an afterthought.

What you're saying is it's designed with the knowledge that it's going to cause false positives, and basically doesn't work well under anything more than minimal packet loss. I think this is probably an important factor to note in the description (and I still fail to see how this is considered highly available or fault tolerant, as described in Intro pages)

I think we are maybe just working with different definitions. High Availability for Serf means that it can continue to handle changes in topology and deliver user events in the face of node failures and network problems. However, it is inevitable that there will be a degradation in it's performance given network failures. If there are serious packet loss issues, Serf will mark a node as failed.

I'm not saying it "won't work well". It works as it is designed to. It will be available for operations, it will automatically heal when the partition recovers, and the state will be resynchronized with the "failed" nodes. The system will be in an eventually consistent state, which is expressly documented and is it's normal mode of operation.

If you consider 5% packet loss "minimal", I'm not sure what applications you are running. TCP degrades at over 0.1% packet loss, and most UDP streaming protocols have serious degradation over 5%.

Re: Serf: A decentralized solution for service discovery and orchestration

#45
post #39

can we see serf as a kind of riak core but written in go?

Riak Core provides a superset of the features of Serf. Riak Core uses gossip to manage membership, but it also provides quorums for coordination, and is based around the notion of a hash ring and virtual nodes. You could instead using Serf to build riak core like technology on top.

Re: Serf: A decentralized solution for service discovery and orchestration

#46
post #44

Earlier quoted context omitted.

What you're saying is it's designed with the knowledge that it's going to cause false positives, and basically doesn't work well under anything more than minimal packet loss. I think this is probably an important factor to note in the description (and I still fail to see how this is considered highly available or fault tolerant, as described in Intro pages)

I think we are maybe just working with different definitions. High Availability for Serf means that it can continue to handle changes in topology and deliver user events in the face of node failures and network problems. However, it is inevitable that there will be a degradation in it's performance given network failures. If there are serious packet loss issues, Serf will mark a node as failed. I'm not saying it "won…

I'm still confused. You mention resynchronizing when the "partition" "recovers". First, can you clarify what a partition is? Second, can you define "recovery"? I'm not worried about performance degradation, i'm worried about nodes being marked down when they aren't down.

Please correct me if i'm wrong, but it sounds like this software only works reliably when you have two sets of nodes that suddenly can't communicate at all, and are eventually connected. Sometimes that does happen on a real network, but often the cause of failures is intermittent and undetermined for hours, days, or weeks. In this case, how would this program work? Would network nodes keep appearing and disappearing, triggering floods of handler scripts, loading boxes and keeping services unavailable?

Yes, tcp performance does degrade under packet loss. It also continues to operate (at well over 50% loss) and automatically tunes itself to regain performance once degradation ends. And it does not present false positives.

It maintains its own state (ordered delivery), checks its own integrity, stands up to Byzantine events (hacking), and is supported by any platform or application. Unfortunately, due to its highly-available nature, it will eventually report a failure to an application if one exists. But if latency is more of a priority than reliability, UDP-based protocols are more useful.

If you're designing a distributed, decentralized, peer-to-peer network, that's cool! But I personally wouldn't use one to support highly-available network services (which is three out of the five suggested use cases for Serf)

Re: Serf: A decentralized solution for service discovery and orchestration

#47

Earlier quoted context omitted.

A meta question if you will - I have often come across situations in work where "if only we had that tool". sometimes I have hacked something together, other times taken it further and tidied it up and released it. But this seems to have a large level of polish so ... When did you realise the need for surf Did you work on it as a main project at some point or is it a side project When and how did you decide to commit…

Great questions. I'll answer each in turn. I want to mention the "polish": I personally don't believe in releasing an open source project without polish. If it is missing docs, its just not complete. If it is ugly, it is not complete. The technical aspects of Serf were done weeks ago. Getting the human side of things done took another few weeks (contracting designers and such). > When did you realise the need for ser…

Thank you - that "building was easy, compared to knowing what to build" put a lot into perspective. And reaching out to external people to build the polish is a surprise, but obvious in retrospect.

I am afraid that for such a helpful and clear answer, you get a mere 1 karma point from me - but thank you.

Re: Serf: A decentralized solution for service discovery and orchestration

#48
post #34

This would really benefit from a "how does this relate to zookeeper". I think this is an entirely new service, with different technical insides, and trying to provide a higher level solution to what people usually cobble together with ZK. But I'd be interested in comments from someone knowledgeable. Edit: I see this is addressed at http://www.serfdom.io/intro/vs-zookeeper.html but it would be nice to have something m…

In writing that section, we tried to provide "just the facts". If there is anything that seems wrong or misleading in any way, we'd like to know so that the page can be corrected. It is not our intention to say "Serf is good, ZooKeeper is bad". They are very different tools, and we are just trying to highlight the differences. In fact, we believe that the strongest use cases involve using those tools together.

I didn't mean to say that it sounded like a sales pitch. What I meant is that it's talking about relative strengths and weaknesses, where what I really need to understand what Serf is is more along the lines of how the API / model differs from ZKs notion of writing to or waiting on locations in the distributed space.
Post reply on HN