Flynn: first preview release

Post by justincormack » Mon, Apr 21, 2014, 8:48 PM UTC

nemothekid wrote: post #30

dominotw wrote: post #19

Could someone point out to me (non superficial) differences between various service discovery projects that have popped up recently like, [1]. etcd [2] skydock [3] consul [4] zookeper ( ok this is not new) [5] flynn They all seem to be doing the exact same to me. 1. https://github.com/coreos/etcd 2. https://github.com/crosbymichael/skydock 3. https://github.com/hashicorp/consul/ 5. https://flynn.io/ Edit: flynn uses…

Etcd and Zookeeper provide essentially the same functionality. They are both a strongly consistent key/value stores that support notifications to clients of changes. These two projects are limited to service discovery, there are a number of projects that depend on zookeeper (that theoretically could be ported to etcd) for various distributed state management such as Kafka, Storm, and Hadoop. Skydock (Skydns) and Cons…

Post by davidcelis » Mon, Apr 21, 2014, 8:57 PM UTC

danielsiders wrote: post #33

StavrosK wrote: post #26

Are these sorts of services good enough? I want somewhere to deploy my Django apps, ideally I should be able to get a beefy server and deploy every app as a single thing on that server, completely compartmentalized. The problem I keep hitting is that these things have a tradeoff between ease of use and power. I want to use Docker, but it has no easy way to say "take this file that contains instructions and make every…

Built-in database appliances are a big part of what Flynn will offer in the next few months (right now it's only Postgres).

Post by hendzen » Mon, Apr 21, 2014, 9:20 PM UTC

necubi wrote: post #44

hendzen wrote: post #40

Earlier quoted context omitted.

It seems to partially duplicate the functionality of Mesos, as they are writing their own task scheduling framework [0] based on Google's Omega [1]. The Omega authors claim that Mesos' system of application specific schedulers accepting resource offers from the Mesos master is well suited toward short-lived jobs (think ephemeral Map/Reduce or MPI type workloads) but is not well suited for long lived 'service' jobs (l…

I haven't read the Omega paper yet, but plenty of people are running long-running tasks in Mesos (Marathon [0] is a framework for doing just that). [0] https://github.com/mesosphere/marathon

  Mesos achieves fairness by alternately offering all 
  available cluster resources to different schedulers,
  predicated on assumptions that resources become available
  frequently and scheduler decisions are quick. As a result,
  a long scheduler decision time means that nearly all 
  cluster resources are locked down for a long time, inaccessible
  to other schedulers. The only resources available for other
  schedulers in this situation are the few becoming available
  while the slow scheduler is busy. These are often insufficient
  to schedule an above-average size batch job, meaning that
  the batch scheduler cannot make progress while the service
  scheduler holds an offer. It nonetheless keeps trying, and
  as a consequence, we find that a number of jobs are abandoned 
  because they did not finish scheduling their tasks by
  the 1,000-attempt retry limit in the Mesos case (Figure 7c).
  This pathology occurs because of Mesos’s assumption
  of quick scheduling decisions, small jobs and high re-
  source churn, which do not hold for our service jobs. Mesos
  could be extended to make only fair-share offers, although
  this would complicate the resource allocator logic, and the
  quality of the placement decisions for big or picky jobs
  would likely decrease, since each scheduler could only see
  a smaller fraction of the available resources. We have raised
  this point with the Mesos team; they agree about the 
  limitation and are considering to address it in future work.

Post by necubi » Mon, Apr 21, 2014, 9:36 PM UTC

hendzen wrote: post #45

necubi wrote: post #44

Earlier quoted context omitted.

I haven't read the Omega paper yet, but plenty of people are running long-running tasks in Mesos (Marathon [0] is a framework for doing just that). [0] https://github.com/mesosphere/marathon

From the Omega paper (section 4.2): Mesos achieves fairness by alternately offering all available cluster resources to different schedulers, predicated on assumptions that resources become available frequently and scheduler decisions are quick. As a result, a long scheduler decision time means that nearly all cluster resources are locked down for a long time, inaccessible to other schedulers. The only resources avail…

Post by dpritchett » Mon, Apr 21, 2014, 9:38 PM UTC

shykes wrote: post #37

dpritchett wrote: post #13

Earlier quoted context omitted.

PSA to anyone trying this on Ubuntu 14: the docker command seems to be "docker.io" now.

There is an ongoing conundrum in the downstream Debian/Ubuntu packaging of Docker. Our friends at Debian and Ubuntu are doing their best to solve it. In the meantime, you should either a) install the latest docker from the official docker APT repos http://docs.docker.io/installation/ or b) do something like ln /usr/bin/docker{.io,} In any case, don't leave your Docker install without a docker binary in your path. Tha…