Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I shrunk a Docker image by 98.8% – featuring fanotify

blog.jtlebi.fr

Post reply on HN
Print view View original

31–40 of 51 posts

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#31

Post by birdsbolt »

dxhdr wrote: post #24
neonfreon wrote: post #9

I don't think there is any way to prove that this found all the required files. The more paths through the code, each with its own potential file accesses that can't be predicted with out run time information, the more likely one will be missed in this optimization stage.

Variation on the halting problem? Given infinite running time and arbitrary input, can you prove that a program will never access file X.

There's no general algorithm, but you could probably prove it, if you tried really hard for your given example. :D

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#32

Post by derefr »

xorcist wrote: post #30
derefr wrote: post #22

Earlier quoted context omitted.

Any Unix-ish application (i.e. one that shells out to do something at some point) will have a package dependency tree that ends up transitively closing over the "base"/"essential" package-set of the OS. "Dependency" has three meanings, to a packaging system, even though at run-time only one of them is relevant. There are: 1. "run-time dependencies" — package B needs package A installed because a binary from B actuall…

I beg to differ, but we can probably compare data points until the cows come home. Anyhow, even a large-ish application such as Oracle or a control system doesn't actually use ping or dd or troff, or most parts of what a modern unix-OS is comprised of. Most things suid are usually unnecessary, which if nothing else does decrease the attack surface. Most web apps probably needs nothing unix-ish at all. A chrooted PHP…

I agree that it's common, but it's not common enough to make this into a helpful property if you're trying to define a 100% solution. The reason Docker exists at all, apart from just nsexec(1)ing static binaries, is that a lot of things do need an environment—not of other Unix binaries per se, but of library assets like locales, charmaps, keymaps, geoip mappings, etc.—and then these asset packages think they're there to provide assets for maintenance-time functionality of a computer rather than to provide run-time functionality to an app in a container, so they pull in utilities related to themselves, which pulls in the base system.

If you can manage to get a working install of Postgres without pulling in half of Debian, I would be surprised.

But yes, on the other hand, it's perfectly possible to package some things, like the JVM, in a sort of "spread-out in a directory but equivalent to static-linked" fashion. The sort of things you see telling you up "unzip them into /opt/thispkg" because they don't really follow any Unix idioms at all, tend to be surprisingly container-friendly. They come from a world where binaries are expected to be portable across systems with different versions of OS libraries available, rather than a world where each app gets to ask the OS to install whatever OS library versions it requires.

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#33

Post by kentonv »

Sandstorm.io has baked something like this into its basic packaging tool for about a year now, except based on FUSE rather than fanotify. Really helps cut down package sizes - many are 10-20MB despite containing all userspace dependencies of the app. https://blog.sandstorm.io/news/2014-05-12-easy-port.html

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#34

Post by jcheng »

xorcist wrote: post #19

Isn't the point of running an application in a container, or any chrooted environment, to only isolate the application from the rest of the operating system? Then why would you start out with a complete extra operating system in there? Why not just put the application and its dependencies in there? To strip non-dependencies from an complete operating system sounds like a very failure prone way to accomplish almost th…

It's not only about the isolation, but also reproducibility across time and portability across machines.

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#35

Post by CMCDragonkai »

derefr wrote: post #22
xorcist wrote: post #19

Isn't the point of running an application in a container, or any chrooted environment, to only isolate the application from the rest of the operating system? Then why would you start out with a complete extra operating system in there? Why not just put the application and its dependencies in there? To strip non-dependencies from an complete operating system sounds like a very failure prone way to accomplish almost th…

Any Unix-ish application (i.e. one that shells out to do something at some point) will have a package dependency tree that ends up transitively closing over the "base"/"essential" package-set of the OS. "Dependency" has three meanings, to a packaging system, even though at run-time only one of them is relevant. There are: 1. "run-time dependencies" — package B needs package A installed because a binary from B actuall…

Nix helps with this. There is an optimisation pass that can create hardlinks between similar packages/files? There was recent talk on package deduplication. Also every package directly specifies every dependency. However it currently won't help to remove unused files in each package, that violates the immutable hashes. The solution is to create more granular packages or to leave the immutability zone and into the mutable world if you have embedded scenarios.

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#36

Post by [deleted] »

dxhdr wrote: post #24
neonfreon wrote: post #9

I don't think there is any way to prove that this found all the required files. The more paths through the code, each with its own potential file accesses that can't be predicted with out run time information, the more likely one will be missed in this optimization stage.

Variation on the halting problem? Given infinite running time and arbitrary input, can you prove that a program will never access file X.

[deleted]

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#37

Post by vacri »

neonfreon wrote: post #9

I don't think there is any way to prove that this found all the required files. The more paths through the code, each with its own potential file accesses that can't be predicted with out run time information, the more likely one will be missed in this optimization stage.

I was thinking the same thing during the article, and the author says as much in the 'Last Thought', and doesn't recommend using this for production purposes. The footnotes say that this was more of an exercise in the syscall than in docker.

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#38

Post by michaelmior »

dxhdr wrote: post #24
neonfreon wrote: post #9

I don't think there is any way to prove that this found all the required files. The more paths through the code, each with its own potential file accesses that can't be predicted with out run time information, the more likely one will be missed in this optimization stage.

Variation on the halting problem? Given infinite running time and arbitrary input, can you prove that a program will never access file X.

Agreed that this seems equivalent to me. Assume instead of the instruction "access file X", the instruction is HALT.

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#39

Post by michaelmior »

TheDong wrote: post #26
neonfreon wrote: post #9

I don't think there is any way to prove that this found all the required files. The more paths through the code, each with its own potential file accesses that can't be predicted with out run time information, the more likely one will be missed in this optimization stage.

You have runtime information though. It's true that this method will not find all things (dlopened files in strange codepaths), but just like we have tools that can verify 100% code coverage in tests, you could fuzz inputs until you find that you've hit every single branch of the executable's instructions and record all dependencies as you go. You could argue that that can still be fooled by, e.g., making the softwar…

I don't think you can ever prove that you've found the required files for an arbitrary binary. (I especially have a hard time believing that such a proof would involve fuzzing, which is random.) However, it seems reasonable that you would be able to achieve a high enough level of confidence that this technique would be viable.

Re: How I shrunk a Docker image by 98.8% – featuring fanotify

#40

Post by michaelmior »

iso8859-1 wrote: post #29
errordeveloper wrote: post #15

Earlier quoted context omitted.

They don't have to, one can run static binary without any problems. It's just that most people keep throwing in a whole distro...

You can't make a truly static binary with glibc, so almost no one has a toolchain that is able to do it.

What do you mean by "truly static"?
Post reply on HN