I Want to Run Stateful Containers, Too • Hacker News Board

Re: I Want to Run Stateful Containers, Too

#51

Post by takee » Sun, Nov 22, 2015, 7:21 PM UTC

theseatoms wrote: post #5

kennu wrote: post #3

I think the author is implicitly describing the paradigm shift from software and hardware abstractions to service abstractions. We used to make computers by soldering electronics together, then by assembling cases and components, then buying premade servers, then renting cloud capacity, and now we're starting to rent everything as services. The cloud is about going from capex to opex and the "capex" now is the initia…

How does OpenStack stack up in this regard? A step in the right direction? https://en.wikipedia.org/wiki/OpenStack

Personally I think it might make sense to run your stateful apps in openstack VMs and the stateless portion in containers.

Re: I Want to Run Stateful Containers, Too

#52

Post by alrs » Sun, Nov 22, 2015, 7:38 PM UTC

takee wrote: post #51

theseatoms wrote: post #5

Earlier quoted context omitted.

How does OpenStack stack up in this regard? A step in the right direction? https://en.wikipedia.org/wiki/OpenStack

Personally I think it might make sense to run your stateful apps in openstack VMs and the stateless portion in containers.

Openstack is an equally terrible place to put stateful apps.

Re: I Want to Run Stateful Containers, Too

#53

Post by enisoc » Sun, Nov 22, 2015, 7:50 PM UTC

=== Shameful Plug ===

If you're interested in running MySQL inside Kubernetes, check out Vitess:

http://vitess.io/resources/presentations.html

We still have work to do, but we're not stopping at the easy part. We show you how to do replication, sharding, and even live re-sharding of MySQL inside Kubernetes. We're working on integrating with Outbrain's Orchestrator for automated failover, and our VTGate query routing service means those failovers will be transparent to your app.

We are admittedly running into the same limits in Kubernetes around using replication controllers for datastores. But Kubernetes is improving very quickly, and there's a reason for that. They have a cheatsheet of one way all of this can come together successfully in the form of Borg:

http://research.google.com/pubs/pub43438.html

At YouTube, we run our main MySQL databases (the ones with tables for users, videos, views, etc) inside containers with local storage. Of course, Borg has a much more mature scheduler, which gives stronger safety guarantees for replicas. My point is that we've proven this approach can work at scale for datastores in a container cluster, and through Vitess we're trying to bring the same capabilities to Kubernetes.

Re: I Want to Run Stateful Containers, Too

#54

Post by bjterry » Sun, Nov 22, 2015, 7:52 PM UTC

jacques_chester wrote: post #29

vidarh wrote: post #18

Earlier quoted context omitted.

> And so it is with PaaSes. The marker was passed years ago. We don't need to go on these spiritual quests any more. The CloudFoundry installation instructions alone makes me run away screaming. Same with OpenStack. There's no "spiritual quest" involved, but a lack of willingness to add unnecessary complexity. Of course it looks different when you've already eaten the cost of getting it up and running and/or has read…

> The CloudFoundry installation instructions alone makes me run away screaming. BOSH is getting some much-needed love right now for this reason. Incidentally, for long-running stateful services, BOSH is the right tool for the job. We use it for Cloud Foundry, MySQL and a bunch of others I don't recall right now. Anyhow. If you just want to see quickly if CF fits your needs, the place to go is Lattice[1], explicitly d…

It reads as an extended advertisement for you company's services, and your comment doesn't really respond to the statement of the person you replied to. He said CloudFoundry's installation instructions were bad, then you replied as if he said he was unsure if CF "fit his needs" and was looking for more ways to try it out, which is a pretty salesy response. I think mostly it's a question of tone and length. If you had a couple sentences acknowledging that the installation instructions sucked, and saying he should go with BOSH, you probably wouldn't have gotten downvoted.

Re: I Want to Run Stateful Containers, Too

#55

Post by vacri » Sun, Nov 22, 2015, 7:54 PM UTC

FooBarWidget wrote: post #9

jacques_chester wrote: post #7

I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…

> $1500 per month, versus weeks of engineering time spent tinkering with and upgrading and bug-fixing and trouble-shooting and security-patching a hand-made solution is a fantastic bargain. That's only true if you look at it from the point of view of an enterprise, a startup with VC funding, or generally a "rich" company. If you're a cash-strapped startup/small business with only a few thousand dollars profit per mon…

> If you're a cash-strapped startup/small business with only a few thousand dollars profit per month, $1500/mo is a huge deal

If it takes 30+ hours of engineering time per month (less than one work week), then you've hit your break-even point. This is assuming the engineer is $50/hour, including overheads.

Re: I Want to Run Stateful Containers, Too

#56

Post by davidopp_ » Sun, Nov 22, 2015, 8:06 PM UTC

cpitman wrote: post #21

rco8786 wrote: post #12

Earlier quoted context omitted.

Does kubernetes not have a mechanism for defining host/rack diversity constraints?

I know that OpenShift 3, built on kubernetes, supports both affinity and anti-affinity policies ( https://access.redhat.com/documentation/en/openshift-enterpr... ). I don't know if these are part of kubernetes or part of the extensions custom to OpenShift.

All of those scheduling policies are also available in native Kubernetes. The Kubernetes version of that documentation is here: https://github.com/kubernetes/kubernetes/blob/master/docs/de...

Re: I Want to Run Stateful Containers, Too

#57

Post by nzoschke » Sun, Nov 22, 2015, 8:15 PM UTC

vidarh wrote: post #18

jacques_chester wrote: post #7

I agree with the author that writing your own snowflake PaaS is a mistake. I work at Pivotal on the fringes of CloudFoundry; OpenShift Origin is a competing system. Either way, you should be using a full PaaS instead of rolling your own. But this is the bit that surprises me: They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. $1500 per month, versus weeks of engineer…

> And so it is with PaaSes. The marker was passed years ago. We don't need to go on these spiritual quests any more. The CloudFoundry installation instructions alone makes me run away screaming. Same with OpenStack. There's no "spiritual quest" involved, but a lack of willingness to add unnecessary complexity. Of course it looks different when you've already eaten the cost of getting it up and running and/or has read…

I am working on Convox, an open source "PaaS" that installs in your AWS account in minutes.

There is no additional layer of complexity. It configures AWS coherently with a few cloud formation templates.

I'd love to hear your opinion on this type of installer.

http://convox.com

Re: I Want to Run Stateful Containers, Too

#58

Post by jodok » Sun, Nov 22, 2015, 8:20 PM UTC

https://crate.io Co-Founder here. We're participating in this game by working hard to build a fully distributed, shared-nothing SQL database.

In our vision, a app in a container is able to access the persistence layer like in SQLite - just import it. another cluster of containers - preferably having one instance node-local - takes care of the database needs.

The database is distributed and makes sure enough replicas exist on different nodes. it's easy to scale up and down, local ressources are being utilized whenever possible (no NAS/SAN like storage).

Re: I Want to Run Stateful Containers, Too

#59

Post by mschuster91 » Sun, Nov 22, 2015, 8:40 PM UTC

Why not run good old classic bare-metal servers (or in VMs, if you like to decouple hardware from software) with the services being configured either by hand (using good documentation, something that is a lost art these days) or with a system like puppet?

The article, unfortunately, stinks of overengineering and overcomplexity for setting up a SIMPLE FUCKING SERVER ENVIRONMENT.

Or just stick to plain old Apache/Lighttpd with PHP and a standard MySQL/pgSQL database?

Don't reinvent the wheel just because it's "cool".

edit: also, I don't get why anyone would spend literally weeks reading docs, learning DSLs etc. just to get a deployment done. My personal maximum time for getting a "hello world" is two hours, if it's radically new a day. If the docs of the project are insufficient (or the quality), DO NOT RELEASE IT. Don't let your users do your work for you.

Re: I Want to Run Stateful Containers, Too

#60

Post by tinco » Sun, Nov 22, 2015, 9:07 PM UTC

paukiatwee wrote: post #6

tinco wrote: post #2

We run a sharded+replicated mongo cluster and a redundant postgresql array in docker containers. We had to write our own orchestrator on top of Etcd to get the functionality we wanted. Like the article author I feel there's some cultural divide happening. How is it possible that the 100.000 line Java project of Kubernetes lacks even basic resource management that our orchestration tool that's written in a few hundred…

* Kubernetes is not written in Java * There is no one size fit all, just like never ending Ruby vs JavaScript VS Java. * Kubernetes just hit v1.0 not long ago. * Seem like you already wrote the functionality you want, now what is the problem?

* Alright, sorry Kubernetes is in Go I must've gotten it confused with some other project.

* Well, alright, but they market it as a general purpose project. I'm just saying that to us it's strange that none of these frameworks out there deal with persistent storage, as the author of the article also observed.

* Ok.

* Well the problem is that it'd be much nicer if we could just use Kubernetes. Obviously having a homegrown orchestrator is not very nice. There's bounds to be lots of edgecases that our ops will run into that'll continuously cause us to perform maintenance on it, and maintaining an orchestration framework is not our core business.

Anyway, this is not necessarily a critique of Kubernetes, it's great software. It's just an affirmation of the point the article is making, that it's curious that there's no interest in stateful containers from the maintainers of these frameworks.