Any time I hear about running docker inside docker (or indeed a VM host inside a VM) I'm reminded of a lesson from the great duo B&BH : https://youtu.be/PyrRVNyjlqU
Launch HN: Nestybox (YC S20) – Containers beyond microservices
61–70 of 112 posts
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#62Ex-Docker person here. I got an early peek at Sysbox and I'm really excited by it -- it's really neat. Docker is missing a bunch of features that make some software work, which is why you can't run Docker inside Docker by default. Instead of dropping from containers all the way down to hardware virtualization, Sysbox is "augmenting" containers with the missing features by simulating them in userland. That gives you a…
i'm just curious. For what kind of use cases do you need docker inside of another docker?
I use dind to build containers in gitlab ci.
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#63Seems Like a promising solution.
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#64Earlier quoted context omitted.
Ah sorry, I was unclear: I meant Amazon ECS.
Got it; the answer is no, because I believe AWS ECS (Fargate) creates the containers using the OCI runc. In order for them to offer system-containers as a service, AWS ECS would need to run Sysbox on their backend to deploy the containers.
That should be possible with EC2 nodes for your ECS cluster rather than Fargate.
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#65This is very cool. I'm trying to get a better sense of how this approach differs technically from rootless Docker / usernetes. I understand that it's not there yet , for many reasons, and I see your FAQ about it, but it's clearly working towards the same goal, right? I think what's going on is that you depend on shiftfs from Ubuntu, and SECCOMP_RET_USER_NOTIF (or something?), unprivileged user namespaces, cgroup name…
Thanks @geofft, you made a lot of great points.
I don't think rootless approach is fully aligned with what we're doing right now. True, we both rely on user-namespaces, and we both emphasize the security angle, but our goal is to expand the number of applications/functionality that can run in containers, which is something rootless approach may struggle with for some time.
Please see here if you haven't done it yet: https://github.com/nestybox/sysbox#sysbox-is-not-rootless-do...
In regards to our dependencies, we can operate with or without shiftfs. In both cases user-namespaces are always utilized. The rest of your approximation is correct: we need most of what you mentioned in your second paragraph, which btw, is already there (thanks to Canonical/LXD folks) starting in Ubuntu 5.0+ and 5.5+ for other distros. As you know, shiftfs is only present in Ubuntu at the moment, but as i said, we can live without it.
Which leads me to your question: why would you wait if the functionality you're after is already there? If having dockerd running as an unprivileged user is not a real must-have for you, then Sysbox provides a fairly secure solution while giving you all the functionality.
Sorry, i'm not familiarized with vpnkit yet, will take a look.
Correct, those are some of the benefits Sysbox-EE offers at the moment. That, plus efficiency & scalability features and hardened testing.
Thanks a lot for your detailed feedback @geofft. Please ping us on slack anytime.
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#66This is very cool. I'm trying to get a better sense of how this approach differs technically from rootless Docker / usernetes. I understand that it's not there yet , for many reasons, and I see your FAQ about it, but it's clearly working towards the same goal, right? I think what's going on is that you depend on shiftfs from Ubuntu, and SECCOMP_RET_USER_NOTIF (or something?), unprivileged user namespaces, cgroup name…
Thanks @geofft, you made a lot of great points. I don't think rootless approach is fully aligned with what we're doing right now. True, we both rely on user-namespaces, and we both emphasize the security angle, but our goal is to expand the number of applications/functionality that can run in containers, which is something rootless approach may struggle with for some time. Please see here if you haven't done it yet:…
I'm mostly meaning vpnkit in the sense of a it's thing that plugs into rootless Docker to provide networking - it seems like you could also be a plugin to upstream rootless Docker to provide sysbox-fs and your shiftfs management, at least in the long term.
Will try to remember to join the Slack next week, this is definitely a cool project :)
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#67This is very cool. I'm trying to get a better sense of how this approach differs technically from rootless Docker / usernetes. I understand that it's not there yet , for many reasons, and I see your FAQ about it, but it's clearly working towards the same goal, right? I think what's going on is that you depend on shiftfs from Ubuntu, and SECCOMP_RET_USER_NOTIF (or something?), unprivileged user namespaces, cgroup name…
Thanks @geofft, you made a lot of great points. I don't think rootless approach is fully aligned with what we're doing right now. True, we both rely on user-namespaces, and we both emphasize the security angle, but our goal is to expand the number of applications/functionality that can run in containers, which is something rootless approach may struggle with for some time. Please see here if you haven't done it yet:…
Actually - I thought unprivileged procfs and sysfs already are there, modulo the whole mount_too_revealing thing? https://github.com/opencontainers/runc/issues/1658
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#68Earlier quoted context omitted.
Got it; the answer is no, because I believe AWS ECS (Fargate) creates the containers using the OCI runc. In order for them to offer system-containers as a service, AWS ECS would need to run Sysbox on their backend to deploy the containers.
That should be possible with EC2 nodes for your ECS cluster rather than Fargate.
Yes, as long as you can install Sysbox on the EC2 nodes, you are good to go.
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#69This is completely off-topic, but why use a Github user like an organization?
Thanks @sanketdasgupta for reporting the issue and @asadlionpk for the explanation. Nestybox's github account is an organization now.
Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices
#70Earlier quoted context omitted.
Unfortunately not. Sysbox requires "true root" privileges, so it can't run inside a system container deployed by Sysbox itself (since that system container would use the user namespace). You can run Sysbox inside a privileged container however, and in fact the Sysbox test framework relies on this heavily. What use case do you envision for running sysbox in sysbox?
It's mostly curiosity. Probably docker in docker in docker.
Got it; note that inside a system container you can always run docker-in-docker using privileged containers (https://hub.docker.com/_/docker). That is, you don't need sysbox nesting in order to run more levels of docker nesting inside the system container. And those privileged containers would only be privileged within the system container, but not at host level.