Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Server-side sandboxing: Containers and seccomp

figma.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Server-side sandboxing: Containers and seccomp

#11

Post by jagrsw »

rubenfiszel wrote: post #4

If you are looking to self-host a scalable backend that runs arbitrary code in python/typescript/bash/go with optional sandboxing using nsjail like figma, nsjail is what we use as isolation layer at https://windmill.dev (Open-source alternative to Retool/Airplane) (Our python nsjail config for instance: https://github.com/windmill-labs/windmill/blob/main/backend/... )

nsjail author here (the original one, as the tool is also maintained by others), good job!

Irrelevant nit: .proto files are protobuf definition files (like this one: https://github.com/google/nsjail/blob/master/config.proto), a text representation of a specific protobuf contents is typically called (as per man clang-format): .textpb .pb.txt or .textproto - I use .config for examples distributed with nsjail, but it's licentia poetica :)

Re: Server-side sandboxing: Containers and seccomp

#12

Post by rubenfiszel »

jagrsw wrote: post #11
rubenfiszel wrote: post #4

If you are looking to self-host a scalable backend that runs arbitrary code in python/typescript/bash/go with optional sandboxing using nsjail like figma, nsjail is what we use as isolation layer at https://windmill.dev (Open-source alternative to Retool/Airplane) (Our python nsjail config for instance: https://github.com/windmill-labs/windmill/blob/main/backend/... )

nsjail author here (the original one, as the tool is also maintained by others), good job! Irrelevant nit: .proto files are protobuf definition files (like this one: https://github.com/google/nsjail/blob/master/config.proto ), a text representation of a specific protobuf contents is typically called (as per man clang-format): .textpb .pb.txt or .textproto - I use .config for examples distributed with nsjail, but it's…

The wonders of HN strikes again. Thank you for this amazing piece of technology that is nsjail. Nsjail is very core to our security, our multitenant would be so slow without it and I think we're one of the applications that leverage it in a way that showcase nsjail to its full extent (as in, we beat containers/firecracker cold starts by a fair margin while keeping most of their benefits). That's one of the reason we're order of magniture more efficient than Airplane that uses fargate under the hood. I would love to chat if you had time, my email in my profile.

Re: Server-side sandboxing: Containers and seccomp

#13

Post by CAP_NET_ADMIN »

zxcvgm wrote: post #6

It's pretty easy to apply seccomp to a process using systemd by adding SystemCallFilter= in its unit file. There's a reasonable set of permitted syscalls for general system processes, aptly called `@system-service`, but you can tweak that to suit your needs [1]. I generally use this, among other settings, to further lock down system services [2]. [1] https://www.freedesktop.org/software/systemd/man/latest/syst... [2]…

Yep, can recommend systemd in this case, really easy to apply basic hardening to services that just works.

Re: Server-side sandboxing: Containers and seccomp

#14

Post by heroprotagonist »

Arch-TK wrote: post #10
imiric wrote: post #5

I haven't used seccomp, but have recently been playing around with the Linux pledge port[1]. It has a very friendly UI, but I still struggled with allowing some complex apps to run at all, because of the sheer amount of syscalls and devices they required. Digging through a mountain of strace output is tedious... Can someone with experience with both comment on how (the Linux port of) pledge compares to seccomp? Can i…

Pledge works well if the software developers implement it on their own application. It also works well if the software developers document what syscalls they rely on and what permissions they need. When it comes to retrofitting something like pledge (or seccomp) into an existing application when you've not developed it and/or can't easily tell what syscalls are being called then it's always a nightmare. It doesn't re…

There's no tracing tool to build policy with pledge? Seems like an obvious area to add functionality if it doesn't exist.

Commercial tools have had it for a long time.. even automatic profiling. Either explicitly profile during a test stage, which is best, or profile-on-first-observation.

In the full automatic mode, which is not optimal but is least effort, any operation performed in the first XX minutes/hours/days are considered 'allowed behavior' and anything after that is denied. Then it will either enforce or 'wait-to-enforce' where enforcement mode only turns on if there are no policy violations in the next XX configurable units of time.

Re: Server-side sandboxing: Containers and seccomp

#15

Post by tedunangst »

heroprotagonist wrote: post #14
Arch-TK wrote: post #10

Earlier quoted context omitted.

Pledge works well if the software developers implement it on their own application. It also works well if the software developers document what syscalls they rely on and what permissions they need. When it comes to retrofitting something like pledge (or seccomp) into an existing application when you've not developed it and/or can't easily tell what syscalls are being called then it's always a nightmare. It doesn't re…

There's no tracing tool to build policy with pledge? Seems like an obvious area to add functionality if it doesn't exist. Commercial tools have had it for a long time.. even automatic profiling. Either explicitly profile during a test stage, which is best, or profile-on-first-observation. In the full automatic mode, which is not optimal but is least effort, any operation performed in the first XX minutes/hours/days a…

1. You really need to understand the application more than that. Does ls need network sockets? Sure does, if you have yp enabled. But this won't appear in your trace unless you trace in such an environment. (Although pledge on openbsd transparently handles this case for you.)

2. Just because a program makes a system call doesn't mean it should. Or should at that moment. A lot of late initialization can be done earlier for tighter policies. Auto traced policies tend to be extremely broad, permitting too much stuff.

Re: Server-side sandboxing: Containers and seccomp

#16

Post by tedunangst »

imiric wrote: post #5

I haven't used seccomp, but have recently been playing around with the Linux pledge port[1]. It has a very friendly UI, but I still struggled with allowing some complex apps to run at all, because of the sheer amount of syscalls and devices they required. Digging through a mountain of strace output is tedious... Can someone with experience with both comment on how (the Linux port of) pledge compares to seccomp? Can i…

Despite the availability of linux pledge, and frequent comments mentioning its existence, I'm not aware of many people using it.

Re: Server-side sandboxing: Containers and seccomp

#17

Post by lmeyerov »

Good intro. I'd be curious how they do the syscall tracing, eg, strace logs as part of CI?

Funny enough, we've gone the reverse path for LLM AI-generated code sandboxing for louie.ai / Graphistry . We started with container isolation with careful network, volume, compute etc enablement first, and only now adding nsjail to the runners within the container as an extra defense layer.

The negative space is interesting too. We initially explored alternatives like wasm (too slow and underpowered for our generated python GPU analytics workloads) and firecracker vm (too unwieldy and unportable for our small team). As we do more k8s and enable more interactive data viz customization + web-scale static serving, would love to revisit both.

On which note, we have a bit of budget for someone to help harden the nsjail layer, if of interest!

Re: Server-side sandboxing: Containers and seccomp

#19

Post by rollcat »

Arch-TK wrote: post #10
imiric wrote: post #5

I haven't used seccomp, but have recently been playing around with the Linux pledge port[1]. It has a very friendly UI, but I still struggled with allowing some complex apps to run at all, because of the sheer amount of syscalls and devices they required. Digging through a mountain of strace output is tedious... Can someone with experience with both comment on how (the Linux port of) pledge compares to seccomp? Can i…

Pledge works well if the software developers implement it on their own application. It also works well if the software developers document what syscalls they rely on and what permissions they need. When it comes to retrofitting something like pledge (or seccomp) into an existing application when you've not developed it and/or can't easily tell what syscalls are being called then it's always a nightmare. It doesn't re…

> When it comes to retrofitting something like pledge (or seccomp) into an existing application when you've not developed it and/or can't easily tell what syscalls are being called then it's always a nightmare.

Quite the contrary. If the software in question has been written in a remotely sane way, adding some basic pledge restrictions is a matter of adding one line: pledge("stdio rpath whatever you need", NULL) - it usually goes somewhere in main, after setup() but before while(!quit).

You can usually figure out the permission set within a few attempts, even without a very good understanding of the internals, as most (sane) programs will do only a couple of things: an httpd needs to accept connections, read static files, write logs, etc; a window manager needs to talk to X11, open font files, etc; of course there are also complex beasts like Chrome but that one has been done as well.

The *real* challenge is breaking up a complex program (e.g. a streaming music player) into separate processes that are concerned with just one or two things, e.g. separate process to make requests over the network, a separate one to decode media, another to maintain an on-disk cache, and so on. Placing restrictions on these subprocesses is the easy part; figuring out where to draw these lines is what's hard.

https://man.openbsd.org/pledge.2

Post reply on HN