Etcd Clustering in AWS
engineering.monsanto.com
Etcd Clustering in AWS
1–10 of 31 posts
Re: Etcd Clustering in AWS
#2... thanks Monsanto?
In all seriousness, this is really interesting. They solved some of the problems associated with persisting a cluster and we're likely going to use that. Feels weird thanking them for anything though.
Edit: Is anyone using CoreOS in a physical DC? We're using AWS with ~1.5k VMs but have another 5-6k hosts in physical DCs. Trying to move us towards containers but struggling.
Re: Etcd Clustering in AWS
#3... thanks Monsanto? In all seriousness, this is really interesting. They solved some of the problems associated with persisting a cluster and we're likely going to use that. Feels weird thanking them for anything though. Edit: Is anyone using CoreOS in a physical DC? We're using AWS with ~1.5k VMs but have another 5-6k hosts in physical DCs. Trying to move us towards containers but struggling.
I just sent an email to some clients that I've been trying to get to blog about technical issues (for recruiting and retention purposes)--if Monsanto can do it, most anyone can.
Re: Etcd Clustering in AWS
#4... thanks Monsanto? In all seriousness, this is really interesting. They solved some of the problems associated with persisting a cluster and we're likely going to use that. Feels weird thanking them for anything though. Edit: Is anyone using CoreOS in a physical DC? We're using AWS with ~1.5k VMs but have another 5-6k hosts in physical DCs. Trying to move us towards containers but struggling.
I just sent an email to some clients that I've been trying to get to blog about technical issues (for recruiting and retention purposes)--if Monsanto can do it, most anyone can.
They're certainly known for not always knowing the difference between 'can' and 'should'. ;)
Re: Etcd Clustering in AWS
#5From my experience etcd is pretty rock solid, until you start using it across availability zones. Then if you add in SSL into the mix, the reliability drops even further if you are using the default configuration. At that point you need to start tweaking the heartbeat and timeout parameters for a the cluster to stay stable.
Re: Etcd Clustering in AWS
#6"If this happened our cluster would become unavailable and may have trouble re-clustering."
This was basically the repeated experience I had which caused me to abandon etcd for the time being.
If it can barely ever heal, what the fuck good is it? And I found that it could barely ever heal. A 3-node CoreOS cluster I ran _always_ crashed when it attempted a coordinated update, and rarely could be repaired with the help of #CoreOS over hours.
Because CoreOS pushes out updates with versions of etcd incompatible with recent versions, the etcd cluster could never survive the upgrade.
Add this to the fact that the CEO of CoreOS told me in person that he expected them to be the _only_ Operating System on the internet, and I'm generally not along for the ride with CoreOS any longer.
Consul, Mesos, and Docker are looking good.
Anyone interested in this space should check out:
https://github.com/CiscoCloud/microservices-infrastructureRe: Etcd Clustering in AWS
#7"If this happened our cluster would become unavailable and may have trouble re-clustering." This was basically the repeated experience I had which caused me to abandon etcd for the time being. If it can barely ever heal, what the fuck good is it? And I found that it could barely ever heal. A 3-node CoreOS cluster I ran _always_ crashed when it attempted a coordinated update, and rarely could be repaired with the help…
Have you tried using Mesos? We're doing a POC but ran into some issues that we're going to wait out. Also, I've spoken to Mesos and they stated that they had no intentions to make deployments easier/more stable, in favour of pushing their commercial offering.
Re: Etcd Clustering in AWS
#8I think they fixed etcd cluster problem in 2.0 release (previously this is 0.5 branch).
For example, we use CF (old version), and we hit https://github.com/coreos/etcd/issues/863.
Re: Etcd Clustering in AWS
#9"If this happened our cluster would become unavailable and may have trouble re-clustering." This was basically the repeated experience I had which caused me to abandon etcd for the time being. If it can barely ever heal, what the fuck good is it? And I found that it could barely ever heal. A 3-node CoreOS cluster I ran _always_ crashed when it attempted a coordinated update, and rarely could be repaired with the help…
Have you tried using Mesos? We're doing a POC but ran into some issues that we're going to wait out. Also, I've spoken to Mesos and they stated that they had no intentions to make deployments easier/more stable, in favour of pushing their commercial offering.
> Also, I've spoken to Mesos
Mesos is an Apache project, did you mean Mesosphere?
Re: Etcd Clustering in AWS
#10I can't get over how bad a name etcd is. Everytime I see it I think it's some sort of daemon for /etc files.