Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some questions about Docker and rkt

jvns.ca

Post reply on HN
Print view View original

61–70 of 92 posts

Re: Some questions about Docker and rkt

#61

Post by tptacek »

inopinatus wrote: post #35
justincormack wrote: post #18

In terms of the daemon model of Docker, I guess it does look a bit complicated, and is not explained very well. In production you will do docker run -d nginx, not run it in the foreground, so the client (docker) process is not really in the picture - if you run in the foreground it is just there to stream the standard IO, and so you can kill the process with ^C from the shell. The docker daemon (dockerd) is there to…

Such structures can be hard to document with both clarity and brevity. Take a look at how Wietse Venema describes the elements of Postfix, for a nuanced masterclass in the art. http://www.postfix.org/OVERVIEW.html

Both the architecture and the approach to documenting it were pioneered by Bernstein's qmail, which is the first Unix program of comparable ambition to be structured in this way --- it's crazy to think that there was a time when this implementation strategy was groundbreaking, but, it was.

http://cr.yp.to/qmail/pictures.html

(Fun fact: Venema and Bernstein had a long-running feud, and Postfix exists pretty much entirely because Venema appreciated qmail's architecture but couldn't stomach working with anything Bernstein produced.)

Re: Some questions about Docker and rkt

#62

Post by iheartmemcache »

felixgallo wrote: post #58

Julia writes: "I think "violates a lot of normal Unix assumptions about what normally happens to normal processes" is basically the whole story about containers." This is a key point. Lots and lots of standard Unix invariants are violated in the name of abstraction and simplification, and the list of those violations is not popularized; and most of the current systems have different lists. For example, in Kubernetes…

Oracle has had multi-homed master-master RDBMS setups for > 10 years. I'm pretty sure a half-competent Oracle administrator wouldn't be really 'surprised af' at functionality that's been in Oracle for at least a decade.

For things that need 'care', this has been a solved problem for decades. Banks[0] homed in the WTC on Sept 11 kept on running because OpenVMS has had NUMA clusters and multi-node replication since the DEC Alpha days. This is with 100% transactional integrity maintained and DC failovers measured within the order of 500ms to 5s. (Obviously banks don't all run on VMS.)

Platforms exist like IBM z systems let you live upgrade zOS in a test environment hosted within the mainframe to see if anything breaks, in complete isolation from production of course, revert snapshots, and do basically everything the whole ESX suite (from things like live migrations of VMotion, to newer stuff like growing raid arrays transparently / virtual storage solutions where you can add FC storage dynamically and transparently to the end user). Their stock systems let you live upgrade entire mainframes without a blip. They're built to withstand total system failure (i.e. literally processors, RAM, NICs, and PSU's could all fail on one z13 and you'd have fail-over to a hot-backup without losing any clients attached to the server). HP's Non-Stop, with which I have no experience, offers a similar comprehensive set of solutions.

[0] On Sept 11, a bunch of servers went down with those buildings. * “Because of the intense heat in our data center, all systems crashed except for our AlphaServer GS160... OpenVMS wide-area clustering and volume-shadowing technology kept our primary system running off the drives at our remote site 30 miles away.” --Werner Boensch, Executive Vice President Commerzbank, North America* http://ttk.mirrors.pdp-11.ru/_vax/ftp.hp.com/openvms/integri...

Re: Some questions about Docker and rkt

#63

Post by drdaeman »

justincormack wrote: post #52
drdaeman wrote: post #50

It seems that isolation is frequently the cause. E.g.: * Better developer environment. Actually, I'm not sure anymore. It totally makes sense for testing (all the CI/CD stuff), and - thanks to the packaging aspect - it's easy to set up external dependencies (like databases), but I just wasn't able to grasp how the actual development is better with Docker. Developers tinker with stuff, containers and images are all ab…

Developers tinker with code, but most of the time you don't tinker with the output of that code, like hot patch your binaries or whatever. Same with systems, you build a container from a Dockerfile and maybe Makefile, you don't then go and change a few things you change the source code. We are just pushing the immutability boundaries further and getting more reproducible environments as we do it.

It depends on the project, I guess. Sometimes, it's not that easy.

For scripting languages that don't have a compile-time the code is what gets executed. So with Docker there's either necessity to rebuild the container (extra delays, and quite noticeable ones) or necessity to maintain a separate Dockerfile.dev and mount-binding the code into the container a-la Vagrant.

Even for compiled stuff, it can be a nuisance with that "Sending build context to Docker daemon" phase. Like when you have a fair chunk of artwork assets next to the code. And the advantage of having the intermediate compiler results are also either lost (adding extra build time) or require extra tricks to make things smooth and nice.

And either way, it also means extra work setting up your debugger toolset jump over the isolation boundaries so you can dig into live processes' guts. One's probably going to abandon PID space isolation.

Those consequences are quite rarely mentioned when the immutability aspects of Docker are advertised. It's usually told as "you'll have a reproducible environment" (yay! great!) but never "you may lose that heartwarming experience of having a new build ready to be tested while you switch from the editor to the terminal/browser/whatever window".

Re: Some questions about Docker and rkt

#64

Post by lliamander »

> My coworker told me a very surprising thing about containers. If you run just one process in a container, then it apparently gets PID 1?

That's true for Docker (and possibly rkt, I don't know) but not for LXC. Docker is intended to provide isolation for a single service/app, so having an init process (arguably) doesn't make sense. For LXC, it's more like a separate OS, so it does need an init process.

These two operating models are referred two as "application containers" and "system containers". It seems that the former is more popular for service deployment situations, but if you want a virtual dev environment / sandbox to play in, I would think the latter is a better choice.

Re: Some questions about Docker and rkt

#65

Post by philips »

felixgallo wrote: post #58

Julia writes: "I think "violates a lot of normal Unix assumptions about what normally happens to normal processes" is basically the whole story about containers." This is a key point. Lots and lots of standard Unix invariants are violated in the name of abstraction and simplification, and the list of those violations is not popularized; and most of the current systems have different lists. For example, in Kubernetes…

I think Kubernetes does a good job creating a normal "Unix process environment".

The Pod concept allows for:

    - Container processes share localhost, mount points, etc
    - Providing a "normal" IP address that is routable
    - Ensuring a PID1 can monitor the group of processes (as done by rkt integration)
    - Allowing for normal POSIX IPC (signals, etc)
More here: http://kubernetes.io/docs/user-guide/pods/

As for PetSets I do agree that they need more work to support things that are replicated but not cluster aware. It doesn't magically solve the issues of distributed systems. Also, natively cluster aware things might be better served by controllers. See this demo of an etcd controller:

https://youtu.be/Pc9NlFEjEOc?t=18

Re: Some questions about Docker and rkt

#66

Post by felixgallo »

philips wrote: post #65
felixgallo wrote: post #58

Julia writes: "I think "violates a lot of normal Unix assumptions about what normally happens to normal processes" is basically the whole story about containers." This is a key point. Lots and lots of standard Unix invariants are violated in the name of abstraction and simplification, and the list of those violations is not popularized; and most of the current systems have different lists. For example, in Kubernetes…

I think Kubernetes does a good job creating a normal "Unix process environment". The Pod concept allows for: - Container processes share localhost, mount points, etc - Providing a "normal" IP address that is routable - Ensuring a PID1 can monitor the group of processes (as done by rkt integration) - Allowing for normal POSIX IPC (signals, etc) More here: http://kubernetes.io/docs/user-guide/pods/ As for PetSets I do…

It definitely does better than many of the rest, in my experience, and for sure it has better defaults and chooses its violations carefully and generally wisely. In fact, I wrote the first draft of a paper on this specific topic:

https://docs.google.com/document/d/1hw_0edCtZ8D4FYhc6oNRTAXm...

delineating some of the more difficult and surprising violations and some possible remediation steps.

Re: Some questions about Docker and rkt

#67

Post by felixgallo »

iheartmemcache wrote: post #62
felixgallo wrote: post #58

Julia writes: "I think "violates a lot of normal Unix assumptions about what normally happens to normal processes" is basically the whole story about containers." This is a key point. Lots and lots of standard Unix invariants are violated in the name of abstraction and simplification, and the list of those violations is not popularized; and most of the current systems have different lists. For example, in Kubernetes…

Oracle has had multi-homed master-master RDBMS setups for > 10 years. I'm pretty sure a half-competent Oracle administrator wouldn't be really 'surprised af' at functionality that's been in Oracle for at least a decade. For things that need 'care', this has been a solved problem for decades. Banks[0] homed in the WTC on Sept 11 kept on running because OpenVMS has had NUMA clusters and multi-node replication since the…

I'm saying that an arbitrary number of exact replicas of a master can magically appear on the network believing they are the one true master, identifying themselves as such, and expecting to act that way. Additionally, an arbitrary number of database masters expecting to participate in the cluster may show up or leave at any time. That is somewhat nontrivial for even modern databases to deal with.

Re: Some questions about Docker and rkt

#68

Post by theptip »

Here's my take on these questions:

1) packaging: this is the feature that's easiest to see benefits from. Having a single artifact that can be run on your CI infrastructure, development machine, and production environment is a massive win.

2) scheduling: there are big cost savings to be had by packing your application processes more efficiently onto your infrastructure. This might not be a big deal if you're a startup, and you haven't yet hit scale.

3) dev environment: It's powerful to be able to run exactly what's been deployed to prod, on your local machine. I've not found developing in a container to be great though; I still use the Django local dev server for fast code-loading. (It's possible to mount your working directory into your built container; this is just personal taste).

4) security: containers are not as robust a security boundary as hypervisors, so they are less suitable for multi-tenant architectures. The most common use-case is to run your containers in a VM, so this isn't necessarily a problem. As an additional defense-in-depth perimeter, containers are great.

5) networking: think of network namespaces as a completely isolated network stack for each container. You can run your containers in the host namespace using `--net=host`, but this is insecure [1]. Using host networking can be useful for development though. In general the port forwarding machinery allows your orchestrator to deploy multiple copies of the same container next to each other, without the deployed apps having to know about other container's port allocations. This makes it easier to pack your containers densely. (More concretely, your app just needs to listen on port 8000, even if Kubernetes is remapping one copy of it to 33112, and another copy to 33111 on the host).

6) secrets: containers force you to be more rigorous with your handling of secrets, but most of the best practices have been established for some time [2]. The general paradigm is to mount your keys/secrets as files, and consume them in the container; Kubernetes makes this easy with their "Secrets" API. You can also map secret values into env variables if you prefer.

7) container images: the Dockerfile magic is a pretty big win for building artifacts; the build process caches layers that haven't changed, which can make builds very fast when you're just updating code (leaving OS deps untouched). Having written and optimized a build pipeline that produced VMDK images, and experiencing the pain of cloning and caching those artifacts, I can attest that this a very nice 80/20 solution out of the box.

[1]: https://github.com/docker/docker/issues/14767 [2]: https://12factor.net/

Re: Some questions about Docker and rkt

#69

Post by bmurphy1976 »

felixgallo wrote: post #67
iheartmemcache wrote: post #62

Earlier quoted context omitted.

Oracle has had multi-homed master-master RDBMS setups for > 10 years. I'm pretty sure a half-competent Oracle administrator wouldn't be really 'surprised af' at functionality that's been in Oracle for at least a decade. For things that need 'care', this has been a solved problem for decades. Banks[0] homed in the WTC on Sept 11 kept on running because OpenVMS has had NUMA clusters and multi-node replication since the…

I'm saying that an arbitrary number of exact replicas of a master can magically appear on the network believing they are the one true master, identifying themselves as such, and expecting to act that way. Additionally, an arbitrary number of database masters expecting to participate in the cluster may show up or leave at any time. That is somewhat nontrivial for even modern databases to deal with.

Why run your database inside kubernetes though? We've always white gloved our database (and a few other special services). You don't have to put 100% of your infrastructure in docker/kubernetes.

Re: Some questions about Docker and rkt

#70

Post by finnh »

bmurphy1976 wrote: post #69
felixgallo wrote: post #67

Earlier quoted context omitted.

I'm saying that an arbitrary number of exact replicas of a master can magically appear on the network believing they are the one true master, identifying themselves as such, and expecting to act that way. Additionally, an arbitrary number of database masters expecting to participate in the cluster may show up or leave at any time. That is somewhat nontrivial for even modern databases to deal with.

Why run your database inside kubernetes though? We've always white gloved our database (and a few other special services). You don't have to put 100% of your infrastructure in docker/kubernetes.

That's felixgallo's point exactly.
Post reply on HN