Live data from Hacker News

Launch HN: Nestybox (YC S20) – Containers beyond microservices

news.ycombinator.com

11–20 of 112 posts

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#11
How is NestyBox different from LXD (and LXC), as they too can:

- Run a distro including system software, systemd etc. as a container

- Run unprivileged, using uid-mapping to provide root and other system uids inside the container?

I've been using LXC and LXD for years to run system images as containers, even migrated some real machines to containers this way.

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#12
post #4

This seems like a feature that, once the need is demonstrated, docker will add to its own product eliminating the niche you hope to fill. Is it wise to go through all the effort and risk of starting a business to prove demand for a feature that an existing established product will then add, removing the need for your company? (Note this question is not a general critique of all startup ideas, it's specific to startup…

Startups that address gaps on established products are a great acquisition target for the companies. They would not have invested such time, effort and resources as a niche startup would have. I believe chances of success for such startups are better for a good exit.

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#13
post #11

How is NestyBox different from LXD (and LXC), as they too can: - Run a distro including system software, systemd etc. as a container - Run unprivileged, using uid-mapping to provide root and other system uids inside the container? I've been using LXC and LXD for years to run system images as containers, even migrated some real machines to containers this way.

The main difference is that it's OCI-based, so works with Docker/containerd and hopefully K8s soon (we are working on the latter). Also, correct me if I am wrong, but I don't believe LXD runs K8s inside without privileged containers. Having said this, I know LXD and Sysbox use many of the same OS-virtualization techniques to do what they do. And in fact we owe much of the work we've done to the ground-work done by the good folks at Canonical/LXD.

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#14

Awesome, this will be very useful. It would be good to be able to assign physical network interfaces into the containers(using network namespaces) and also document how to create virtual network devices shared between containers. Furthermore it would be awesome to make all this numa-aware. So you could have network interfaces and cpus and memory be assigned to a single container where everything is numa local. Then y…

Thanks! Both of your suggestions sound very interesting. I personally like the idea of creating large network topologies with a very few outer containers: the real mesh would be at L2/L3 levels. You would launch your large topo with just a 'docker run'.

The numa-aware idea would take some more research on our part, but at first glance looks like something that we could definitely explore.

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#16
This is very interesting! I’m not well versed in runc and friends but I was just exploring runsc[0] and gvisor.

Is there any overlap here even though your project seems to go the path of adding more functionality? Is it fair to think of NestyBox as a type of sandbox?

https://pkg.go.dev/github.com/google/gvisor/runsc?tab=overvi...

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#17
post #11

How is NestyBox different from LXD (and LXC), as they too can: - Run a distro including system software, systemd etc. as a container - Run unprivileged, using uid-mapping to provide root and other system uids inside the container? I've been using LXC and LXD for years to run system images as containers, even migrated some real machines to containers this way.

The main difference is that it's OCI-based, so works with Docker/containerd and hopefully K8s soon (we are working on the latter). Also, correct me if I am wrong, but I don't believe LXD runs K8s inside without privileged containers. Having said this, I know LXD and Sysbox use many of the same OS-virtualization techniques to do what they do. And in fact we owe much of the work we've done to the ground-work done by th…

Here is how to think about NUMA. In a modern AMD epyc box, you have 2 CPUs. Each cpu has 350GB/s of memory bandwidth and has PCI devices connected to it. Memory speed between CPU sockets is some fraction, 70GB/s? if you run computation on cpu0 and talk to nic on cpu1, you burn a lot more cpu cycles, than if you move your computation to cpu1. So in theory, if you partition the box using containers such that everything on cpu0 runs in container0 and has own nics, and same thing on cpu1. You end up with 2 'virtual' boxes that might actually perform better within your container than outside of it.

Note on modern CPUs, each cpu is further broken down into numa nodes(numactl -H and numa stuff n lscpu)...perf degradation isn't as great going between chiplets, but it's measurable(2x reduction in ram bandwidth?).

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#18
post #15

What’s the use benefit of this over Firecracker & Ignite ( https://github.com/weaveworks/ignite )?

I've not used either, but conceptually the main difference is that those approaches use micro-VMs and thus require hardware virtualization (hypervisors). This can be a challenge if you want to run those on cloud VMs, as it would require nested virtualization. Sysbox on the other hand is a pure OS-virtualization container runtime, so it does not require hardware virtualization.

Also, I think the goal is different: I understand Firecracker is meant as a way of strengthening the isolation of containers by wrapping them in micro-VMs. Sysbox is meant as way of enabling containers to run system workloads without complex images, entrypoints, volume mounts, etc., and with proper isolation via the Linux user-namespace.

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#19

This is very interesting! I’m not well versed in runc and friends but I was just exploring runsc[0] and gvisor. Is there any overlap here even though your project seems to go the path of adding more functionality? Is it fair to think of NestyBox as a type of sandbox? https://pkg.go.dev/github.com/google/gvisor/runsc?tab=overvi...

I understand gvisor's main goal is to improve container isolation by intercepting and inspecting syscalls before they reach the kernel to reduce the attack surface. Sysbox on the other hand is meant as a way to run system software (in addition to apps/microservices) easily inside a Docker container, so its focus is on enabling this functionality. Having said this, Sysbox always enables the Linux user-namespace in containers, and thus also improves container isolation.

Re: Launch HN: Nestybox (YC S20) – Containers beyond microservices

#20

Earlier quoted context omitted.

The main difference is that it's OCI-based, so works with Docker/containerd and hopefully K8s soon (we are working on the latter). Also, correct me if I am wrong, but I don't believe LXD runs K8s inside without privileged containers. Having said this, I know LXD and Sysbox use many of the same OS-virtualization techniques to do what they do. And in fact we owe much of the work we've done to the ground-work done by th…

Here is how to think about NUMA. In a modern AMD epyc box, you have 2 CPUs. Each cpu has 350GB/s of memory bandwidth and has PCI devices connected to it. Memory speed between CPU sockets is some fraction, 70GB/s? if you run computation on cpu0 and talk to nic on cpu1, you burn a lot more cpu cycles, than if you move your computation to cpu1. So in theory, if you partition the box using containers such that everything…

Got it, thanks for the explanation. I clearly see the use-case, just need to review cgroup specs (specifically cpuset) to fully understand if what you mention is already supported (which i believe it is).
Post reply on HN