Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Landlock to Sandbox GNU Make

justine.lol

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Using Landlock to Sandbox GNU Make

#71

Post by chubot »

ErikCorry wrote: post #36
chubot wrote: post #20

Earlier quoted context omitted.

For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make ??? Google was started in 1998, and Bazel was created ~2006 as a replacement for Python + GNU Make ("gconfig"). I was on that team, though I only worked on Blaze a tiny bit. The "google3" build migration was sometime around 2003 or 2004. So at most there were 6 years of using Make only, i.e. "google2". Importantly, p…

I like the speed of Ninja, and the flexibility of having a stage that generates the ninja (or make) files. But it's a bit unclear to me what the best practice is for when to generate the ninja files. It feels like the programmer is expected to do this stage manually and there's no great automatic way to generate the ninja files when they need to be generated. How do good build systems solve this?

Yeah unfortunately there probably isn't a "best practice". I think there are just many types of projects and their needs vary. I think a big issue is what your dependencies are.

This person mentioned Ruby + Ninja, and I similarly used Python + Ninja (borrowing the ninja_syntax.py from the Ninja repo itself).

https://news.ycombinator.com/item?id=32303692

I'd say if you can get away with it, using a real language like that is straightforward and good. If you have a low level project that doesn't depend on anything (like a shell or VM :) ), then it will probably work.

I admitted I probably had to rewrite a small portion of the "120,000 lines of CMake" that ships with CMake, in Python. And that took some time, but it ended up working well for my needs.

If you have a big project with lots of dependencies (especially optional / detected ones), I think that is probably where CMake or Meson is better. They both use Ninja, but I haven't used them myself.

---

But I would also say that there aren't many best practices around GNU make either. The usage spans a very wide range... And lots of people generate it too, like Ninja -- kconfig for the Linux kernel generates GNU make, etc.

autotools and CMake both generate GNU make too. It's a big mess, but I think Ninja is a good foundation for at least not "starting with a mess" :)

Re: Using Landlock to Sandbox GNU Make

#72

Post by chubot »

jart wrote: post #46
chubot wrote: post #20

Earlier quoted context omitted.

For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make ??? Google was started in 1998, and Bazel was created ~2006 as a replacement for Python + GNU Make ("gconfig"). I was on that team, though I only worked on Blaze a tiny bit. The "google3" build migration was sometime around 2003 or 2004. So at most there were 6 years of using Make only, i.e. "google2". Importantly, p…

I wouldn't say it's scary. There's always been full transparency with the caveats and they're being lifted incrementally. https://twitter.com/l0kod/status/1556378406983458818 The workarounds are perfectly reasonable. Also take into consideration that Landlock is so simple as a security tool, that it really opens itself up to people like us being able to focus in on the opportunities for improvement. A lot of security…

Yeah maybe I shouldn't say "scary", but I think the shared libraries and hard-coded paths would have to be worked out / generalized for something like this to land in upstream build tools.

And obviously having a working demo like this is huge progress in that direction, so I think it's great work.

But I'm also wondering if there's a way to do it without OS-specific support patched in? That is, with a wrapper analogous to bubblewrap, sandbox-exec, etc. I know pledge() should make use of app-specific knowledge, so it can't be a CLI wrapper, but I think that isn't true for the build tool case?

It would be more portable, and reduce an O(M x N) code explosion, if there was a standard access-dropping CLI interface that every OS could implement, and that every build tool could use without custom OS-specific code. I've been meaning to experiment with that for a long time, since I agree this problem is worth solving! (for correctness, caching, distribution, etc.) And I think Ninja is a good start and you can do some creative things with it, like

https://lwn.net/Articles/821367/

I also think having the generator split is nice because you could do ./NINJA_config.py --with-sandbox on Linux, and get all the dependencies correct. And then on OS X if there is no good sandboxing tool, you can just do it without the sandbox, assuming that the deps were tested on Linux. (And I agree the most practical answer on Windows is to "use WSL" :) )

Re: Using Landlock to Sandbox GNU Make

#73

Post by chubot »

tux1968 wrote: post #28
chubot wrote: post #21

Earlier quoted context omitted.

Eh Make is not good for most problems -- see experiences here by dwheeler, me, frankohn: https://news.ycombinator.com/item?id=32301606 And this other thread I linked there, with feedback from FreeBSD engineers: https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc --- Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make

> Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make Are you suggesting it's easy to learn how to hand code Ninja files, or do you have a different easy-to-learn tool to generate them, in mind?

Yeah it's true, on the face of it, they aren't directly comparable. You do need a generator and GNU Make has more features.

There's some more color here: https://news.ycombinator.com/item?id=32382760 (I borrowed ninja_syntax.py, and it worked very well, but I don't claim that will work for everybody.)

However I also cringe when people say to write GNU makefiles from scratch -- because those ALSO don't work for everybody, and are generated more often than not (e.g. by autotools, CMake, kconfig, etc.)

So I'd say that GNU make gives you the illusion that you can just use that one tool, but you often end up needing to generate it anyway. And then you should have used Ninja with whatever generator you ended up with :)

Re: Using Landlock to Sandbox GNU Make

#74

Post by heavyset_go »

mtlmtlmtlmtl wrote: post #38

Justine is one of my fav contemporary hackers. Whenever I see her stuff posted on here, not only am I amazed that anyone did what was done, but even more amazed that one person did it all on their own. Justine, if you're reading this, keep up the amazing work!

Agreed. The word "impressed" doesn't accurately capture the way I feel about her work. Her content is exactly the kind of stuff that got me fascinated by CS at a young age, and it's nice feeling that again as an adult, as I almost invariably do whenever I come across something she's written or made.

Re: Using Landlock to Sandbox GNU Make

#75

Post by jart »

chubot wrote: post #72
jart wrote: post #46

Earlier quoted context omitted.

I wouldn't say it's scary. There's always been full transparency with the caveats and they're being lifted incrementally. https://twitter.com/l0kod/status/1556378406983458818 The workarounds are perfectly reasonable. Also take into consideration that Landlock is so simple as a security tool, that it really opens itself up to people like us being able to focus in on the opportunities for improvement. A lot of security…

Yeah maybe I shouldn't say "scary", but I think the shared libraries and hard-coded paths would have to be worked out / generalized for something like this to land in upstream build tools. And obviously having a working demo like this is huge progress in that direction, so I think it's great work. But I'm also wondering if there's a way to do it without OS-specific support patched in? That is, with a wrapper analogou…

There's an element here you may have overlooked that's important to understand. The purpose of a sandbox is to control things. The purpose of dynamic shared objects is to delegate control to other developers so you can leverage their labor at minimal cost. That means giving up control. These two concepts can't be reconciled. You can't properly sandbox dynamic shared objects existing outside your dominion because they're not yours to control.

I solve this problem for myself by using static binaries. I don't need dynamic shared objects. I ship support for it in my project releases, since I know other people do. That weakens the safety my tools can offer dso users, but it doesn't concern me, since I'm offering them incremental value; a weakened sandbox is better than no sandbox at all.

Cosmopolitan Libc is what I have for anyone wanting the stronger model. You can't use it to leverage an ecosystem of third party packages. What it can offer you is a statically linked hermetic environment with less complexity that's more conducive to control, provided you're ok with some assembly being required. Worth considering for your next greenfield project.

Re: Using Landlock to Sandbox GNU Make

#76

Post by spockz »

avar wrote: post #51
spockz wrote: post #40

The only thing I’m really missing for my usecase with make is the ability to specify goals/targets other than files. E.g., I want a docker image to be present in the daemon. Basically, calling out to a function. (Probably somebody is going to correct me and say there is something there in the form of conditionals or something..)

That's easy to do, but if you want to make it compatible with incremental builds the best way is to create a "I did it already" empty file as a dependency marker. E.g. "build" a ". build/called/apt-install-deps" or whatever.

This is exactly how I do it now but it is brittle. If either my packages are changed, or the definition of what needs to be installed/pulled changes I need either remove the file or something.

I could maybe put all these definitions, packages etc in separate files and then have make watch the time stamps. That saves the issue of changing definitions but doesn’t help when things get deleted from under your feet.

Re: Using Landlock to Sandbox GNU Make

#77

Post by kaba0 »

jart wrote: post #18
ghoward wrote: post #16

Earlier quoted context omitted.

Does Cosmopolitan's Landlocking work on anything besides Linux? By definition, that cannot be the case because Landlock is a Linux thing. Cosmopolitan is portable, yes, but this sandboxing is not.

The Landlock Make binary runs on OpenBSD too, where it also supports sandboxing. But those are the only two platforms right now where it's fully-featured. As I mentioned in the article, the next logical step is to add FreeBSD support using jails. The other platforms, I'm not so sure about. Might have to do something like run the binary in a blinkenlights emulator to intercept the system calls of programs we didn't bu…

With all due respect, then why APE, if there is a Linux either way?

Not trying to be argumentative at all, just interested.

Re: Using Landlock to Sandbox GNU Make

#78

Post by jart »

kaba0 wrote: post #77
jart wrote: post #18

Earlier quoted context omitted.

The Landlock Make binary runs on OpenBSD too, where it also supports sandboxing. But those are the only two platforms right now where it's fully-featured. As I mentioned in the article, the next logical step is to add FreeBSD support using jails. The other platforms, I'm not so sure about. Might have to do something like run the binary in a blinkenlights emulator to intercept the system calls of programs we didn't bu…

With all due respect, then why APE, if there is a Linux either way? Not trying to be argumentative at all, just interested.

I said devops not distribution. APE is build-once run-anywhere, i.e. you build once on Linux and run your binary on seven platforms. APE will stand to gain if a stronger consensus emerges about only doing coding on Linux/BSD. Then we can let Windows and Mac be more like modern teletypes that are great for ssh, browsing, and consumption. It wouldn't have been possible for me to achieve what I achieved had I focused on the traditional build-everywhere run-everywhere model. It feels liberating using one and only one toolchain, on one platform, using zero configure scripts. No operating system is beyond the reach of x86_64-pc-linux-gnu. This is the one true way for compiling code.

Re: Using Landlock to Sandbox GNU Make

#79

Post by chubot »

jart wrote: post #75
chubot wrote: post #72

Earlier quoted context omitted.

Yeah maybe I shouldn't say "scary", but I think the shared libraries and hard-coded paths would have to be worked out / generalized for something like this to land in upstream build tools. And obviously having a working demo like this is huge progress in that direction, so I think it's great work. But I'm also wondering if there's a way to do it without OS-specific support patched in? That is, with a wrapper analogou…

There's an element here you may have overlooked that's important to understand. The purpose of a sandbox is to control things. The purpose of dynamic shared objects is to delegate control to other developers so you can leverage their labor at minimal cost. That means giving up control. These two concepts can't be reconciled. You can't properly sandbox dynamic shared objects existing outside your dominion because they…

I understand that and don't really disagree with any of it, but what I'd say I'd say is that there's a third possibility of doing what containers already do -- use dynamic linking against fixed versions you control.

That is, the way I think of it is more along the lines of whether an executable is a "value" (in the Hickey sense), not whether it's statically linked. So the whole container can be a value, but it's not statically ed linked.

That has bearing on both incremental builds and distribution. This relates to my comments on the article about size optimization:

https://lobste.rs/s/rbdgr2/size_optimization_tricks#c_x6lz53

i.e. I think it makes sense for executables to retain structure for differential compression (they can be a tree, not a flat file.)

---

Also I'm wondering if the sandboxing can be done in a child process, without the cooperation of the build tool. I don't see why not, but I'll have to try it (and especially with the container model, which changes things a bit). That could actually make the dynamic library issue easier to deploy because you've punted some policy (hard-coded paths) into a separate tool, rather than having it in the build tool.

Re: Using Landlock to Sandbox GNU Make

#80

Post by dima55 »

chubot wrote: post #70
dima55 wrote: post #41

Earlier quoted context omitted.

I don't disagree with anything you said, but I think it's important to point out that your experience is unique in that you're talking about build infrastructure for a massive codebase: most of Google's internal stuff, right? In my experience plain GNU Make works great until you start working with massive projects. Similarly, the ninja speed improvement: you just don't see it unless you have a truly massive project,…

The 3 makefiles I wrote were actually for a medium-size open source project https://www.oilshell.org/ (medium measured by lines of code) I could probably write a blog post about this, but for two of the Makefiles I needed to enumerate the rules dynamically (inputs weren't fixed, they were "globbed"). And my memory is that this interacted very poorly with other GNU make features build variants Whereas that pattern is…

I'd actually be interested in seeing this case. If you dig it up or write a blog post, or something, send it my way. Thanks!
Post reply on HN