Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Warp – self-contained, single binary applications

github.com

Post reply on HN
Print view View original

71–80 of 157 posts

Re: Warp – self-contained, single binary applications

#71

Post by VonGallifrey »

weberc2 wrote: post #43
dharmab wrote: post #35

Earlier quoted context omitted.

For Python, this works well for my team: https://www.dharmab.com/programming/python/docker/2017/07/24...

Sorry if I'm a little dense; that doesn't seem like it solves any problems. It just says "use virtualenv and/or docker". I guess I was hoping for "how to manage dependencies in a localdev-friendly way for a Python/Docker app" or something.

Not sure if that helps, but here is what I do:

We just use a requirements.txt [1] for each service and run pip with the -r flag in the dockerfile. No Virtualenv in the container.

Most of the time we run these containers with docker-compose locally, but sometimes we want to run the service outside the container. For that we create a virtualenv outside the folder where we keep the source for the service. The reason for that is that we don't want to accidental copy the virtualenv in to the container. It wouldn't do much there but it would increase the container size.

You can do this easily with just "python3 -m venv", but there are some tools that help with that as well. I personally use pyenv-virtualenv [2] which just keeps all virtualenvs in ~/.pyenv/versions/, but there is also conda [3] and virtualenvwrapper [4] which also store the virtualenvs in a central directory.

I am not sure if there really is any more to it.

[1]: https://pip.pypa.io/en/stable/user_guide/#requirements-files

[2]: https://github.com/pyenv/pyenv-virtualenv

[3]: https://conda.io/docs/user-guide/getting-started.html#managi...

[4]: https://virtualenvwrapper.readthedocs.io/en/latest/

Re: Warp – self-contained, single binary applications

#72

Post by TheDong »

wildmusings wrote: post #9

From the README, it looks like this unpacks the application and dependencies into a local cache in the user's home directory on first run. Is it possible to instead execute the compressed application code directly and present to the resulting process a virtualized file system that decompresses the dependencies on the fly too? Then you could run the single binary without giving it any filesystem access. One way to do…

What you describe can be accomplished on linux using mount namespaces. It gives a namespaced view of the filesystem which the parent process can control via picking its root location and what is bindmounted where.

You've basically described a container where the filesystem is virtualized to a safe location on disk.

Disallowing reading/writing data entirely is pretty easy with seccomp or ptrace, though that's not required with namespacing.

Re: Warp – self-contained, single binary applications

#73

Post by haxton »

GordonS wrote: post #53
rictic wrote: post #47

Earlier quoted context omitted.

This. A .jar file could be anything. Is it a library or an executable? When I run it, do I need to make sure its dependencies are on my classpath? What're the java command line flags I need? Do I have the right version of java on this machine? There's something to be said for an entirely self-contained native binary, at least as an easy on-ramp for users.

Unfortunately, it turns out this is not a completely self contained binary - it's more of a packer. So if you pack a Java app, you're still going to need the JRE, and if you pack a .NET Framework app, you're still going the .NET Framework, etc.

So I don't know what this is doing. But I know that microsoft is developing WPF to run in a fulled contained .NET Core binary. Meaning that there is no requirement of having any preinstalled software when running it. I would assume this technology to be translated into most any .NET core app.

Re: Warp – self-contained, single binary applications

#74

Post by TimTheTinker »

ASalazarMX wrote: post #64
w8rbt wrote: post #4

Very neat. Would like to see more languages targeting single binaries. This is one reason (I believe) that has made Go so popular.

I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here). It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed…

I think this represents a fundamental (and rather annoying) difference between Linux and Windows.

On Windows, the only platform-supported global library format is COM, and that requires a very specific programming style--HRESULTs and all the rest. True, registered COM DLLs can be linked to without separate header/symbol files, and installing new programs/libraries never requires local compilation, but the extra overhead of COM itself, and the fact that it's a walled garden code-wise, makes it rather unpopular. It's also nigh on impossible to port COM code to other apps without resorting to something like WINE (which can't compete performance-wise when running highly parallel COM apps/services). Not to mention how ugly C++ COM source files are.

It would be so nice for us all if Windows shipped with compilers, straight-up C libraries with headers, a package manager, a better shell, ...

Re: Warp – self-contained, single binary applications

#75

Post by jcelerier »

open-source-ux wrote: post #11

It's fascinating how self-contained binaries are now coveted as such a desirable feature. Dozens of languages (if not more) were capable of this decades ago. In fact, some of them could generate self-contained binaries that allowed you to bundle app assets like icons and images inside the executable - no additional add-ons needed. And they were native executables, not complicated wrappers with layer upon layer of dif…

it's not really a loop, but there are really two categories of users :

* those who want applications that take the least space on the drive, and share most of their data / libraries across themselves - shared electron runtime, steam runtime shared with the os * those who want applications to conform exactly to what the developper used since the developer can guarantee a "working" configuration

Ideally it should be up to the user to choose what configuration he wants, but the problem is that it's up to the developer to make that choice. Thus, the eternal loop :

    applications released as a blob -> this takes too much disk space, we should use shared libraries ! -> applications released as shared libraries -> DLL hell -> applications released as a blob -> ...

Re: Warp – self-contained, single binary applications

#76

Post by nur0n »

ASalazarMX wrote: post #64
w8rbt wrote: post #4

Very neat. Would like to see more languages targeting single binaries. This is one reason (I believe) that has made Go so popular.

I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here). It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed…

I think the inverse argument can be made against shared libraries: if an update introduces a vulnerability, now all programs which depend on that library become vulnerable.

Re: Warp – self-contained, single binary applications

#77

Post by weberc2 »

dsnuh wrote: post #50
weberc2 wrote: post #43

Earlier quoted context omitted.

Sorry if I'm a little dense; that doesn't seem like it solves any problems. It just says "use virtualenv and/or docker". I guess I was hoping for "how to manage dependencies in a localdev-friendly way for a Python/Docker app" or something.

I'm not sure if I understand correctly. Are you saying that you run venv commands as part of the build step for your containers (i.e. a RUN line in you Dockerfile) and then pip installing modules in another build step?

No, we use pipenv instead of pip, and pipenv manages a venv. There are `--system` flags, but I guess support for installing things to the system is a little buggy or idiosyncratic or something so we use the venv behavior. I'm not sure why we tell pipenv to install the venv in the project directory though. We do the pipenv install as a RUN line in our Dockerfile. Does this answer your question?

Re: Warp – self-contained, single binary applications

#78

Post by weberc2 »

TheDong wrote: post #60
weberc2 wrote: post #23

Earlier quoted context omitted.

We're struggling using containers for our Python and Javascript applications. I think we're using venvs and node_modules and we oughtn't (or we shouldn't keep them in our project directory where they sometimes--but not always--get overwritten by source code volume mounts).

It can be a little tricky to do things right here. There are a few things of note: 1. The .dockerignore file can be used to prevent use of your `node_modules` folder during `docker build`, even if you have something like `COPY . .` in your Dockerfile.. This can let you create a new `node_modules` folder from your lockfile as part of the docker build process to create an image for testing/deployment. 2. You can mainta…

Thanks for the suggestions! I had to check, but we do both of those things. To make sure that node_modules aren't overwritten, we mount an empty named volume to the node_modules directory. I guess mounting the named volume directory somehow causes the node_modules directory from the base image to appear inside of the bind-mounted directory. We do the same thing for venvs as well.

We use venvs inside of the Docker container because we use pipenv and apparently pipenv's support for installing to the system is buggy and/or idiosyncratic.

Re: Warp – self-contained, single binary applications

#79

Post by meddlepal »

pjmlp wrote: post #39
meddlepal wrote: post #25

Earlier quoted context omitted.

JARs are fine until you need to get someone to install a JRE. Not a problem for server apps, but definitely not painless for client distributed apps.

You can bundle everything together, or use a commercial JDK with AOT compilation support.

Neither of those options are "painless".

Re: Warp – self-contained, single binary applications

#80

Post by wokwokwok »

tgtweak wrote: post #63

Latest .net core 3 apparently supports single-bibary packaging. It has been a often-requested feature for at least 2 years. https://github.com/dotnet/corefx/issues/13329 (The last comment referring to warp is indicative)

.Net Core 3 wont be out until next year.

The current .Net Core 2.x runtime has had this feature since it came out though (last year?).

(Hello world + runtime is like 100 MB and several hundred files)

Post reply on HN