Warp – self-contained, single binary applications • Hacker News Board

Re: Warp – self-contained, single binary applications

#101

Post by malkia » Sat, Oct 13, 2018, 5:10 AM UTC

virmundi wrote: post #100

icedchai wrote: post #55

Earlier quoted context omitted.

You can build "fat jars" that contain all dependencies. IMO, this is how most Java apps should be deployed.

You still have issues of needing a JVM. The JVM is not a small runtime considation. I'm in the process of converting to GO since I have more time than money. 1 JVM in Docker uses 512 MB to sit still. A Go version is around 16.

Here is what Google did internally:

- Bundle all C/C++ native libraries, and link them all together with the java launcher - e.g you produce your own java.exe that has your C/C++ libraries linked in, then you slap the .jar file - for extra bonus you can slap at the back of your .exe and make it load from there

  Voila - you have single executable (almost) with all you need, and you don't need to unpack .dll's anywhere

  But... you must compile java.exe yourself...

Re: Warp – self-contained, single binary applications

#102

Post by creatornator » Sat, Oct 13, 2018, 5:12 AM UTC

paavoova wrote: post #97

rkeene2 wrote: post #29

Earlier quoted context omitted.

Tcl is still the king of this kind of distribution with Starkits/Starpacks. Since you can virtualize the filesystem within Tcl, you can pack in all your resources as regular files and generally scripts will work without knowing that this has happened (and without hacky extracting of the archive to disk before running, or while running).

Python can do something similar albeit far simpler via packaging as an executable zipfile. Just structure everything you need in a directory and zip up with __main__.py in the root. All resources are available as an in-memory binary stream, including individual source .py files. Of course if you want to package something like a C library, you need to unpack that to disk to be able to access it with ctypes.

Are you referring to the zipapp library? I had no idea this even existed until now but that seems pretty clever. Huh. Why can't it also dynamically unzip a shared object library?

Re: Warp – self-contained, single binary applications

#103

Post by tannhaeuser » Sat, Oct 13, 2018, 5:19 AM UTC

open-source-ux wrote: post #11

It's fascinating how self-contained binaries are now coveted as such a desirable feature. Dozens of languages (if not more) were capable of this decades ago. In fact, some of them could generate self-contained binaries that allowed you to bundle app assets like icons and images inside the executable - no additional add-ons needed. And they were native executables, not complicated wrappers with layer upon layer of dif…

The root of all problems are incompatible file system layouts, library version, and package formats across Linuxes. I don't see how adding yet another package format is solving this.

Re: Warp – self-contained, single binary applications

#104

Post by finchisko » Sat, Oct 13, 2018, 5:27 AM UTC

TimTheTinker wrote: post #74

ASalazarMX wrote: post #64

Earlier quoted context omitted.

I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here). It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed…

I think this represents a fundamental (and rather annoying) difference between Linux and Windows. On Windows, the only platform-supported global library format is COM, and that requires a very specific programming style--HRESULTs and all the rest. True, registered COM DLLs can be linked to without separate header/symbol files, and installing new programs/libraries never requires local compilation, but the extra overh…

I think. You can have global DLLs in Windows. They just need to be in PATH.

Re: Warp – self-contained, single binary applications

#105

Post by therein » Sat, Oct 13, 2018, 5:40 AM UTC

creatornator wrote: post #102

paavoova wrote: post #97

Earlier quoted context omitted.

Python can do something similar albeit far simpler via packaging as an executable zipfile. Just structure everything you need in a directory and zip up with __main__.py in the root. All resources are available as an in-memory binary stream, including individual source .py files. Of course if you want to package something like a C library, you need to unpack that to disk to be able to access it with ctypes.

Are you referring to the zipapp library? I had no idea this even existed until now but that seems pretty clever. Huh. Why can't it also dynamically unzip a shared object library?

He's talking about pex. [0]

[0] https://github.com/pantsbuild/pex

Re: Warp – self-contained, single binary applications

#106

Post by therein » Sat, Oct 13, 2018, 5:42 AM UTC

nur0n wrote: post #76

ASalazarMX wrote: post #64

Earlier quoted context omitted.

I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here). It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed…

I think the inverse argument can be made against shared libraries: if an update introduces a vulnerability, now all programs which depend on that library become vulnerable.

I would like to think that projects overall tend to fix more bugs than introduce them. It's not like projects go from orderly to disorderly over time.

Re: Warp – self-contained, single binary applications

#107

Post by ComputerGuru » Sat, Oct 13, 2018, 5:51 AM UTC

TimTheTinker wrote: post #74

ASalazarMX wrote: post #64

Earlier quoted context omitted.

I'm ambivalent about this. It is cool to distribute a single binary, but years ago there was serious vulnerability in a very common open source library library whose name escapes me (it wasn't libgcc, someone help a brother here). It was used in multiple programs in both Windows and Linux. On Windows there was a rush of individual updates as each application was fixed... on Linux, simply upgrading that library fixed…

I think this represents a fundamental (and rather annoying) difference between Linux and Windows. On Windows, the only platform-supported global library format is COM, and that requires a very specific programming style--HRESULTs and all the rest. True, registered COM DLLs can be linked to without separate header/symbol files, and installing new programs/libraries never requires local compilation, but the extra overh…

You sound extremely confused. COM has not been the only global dependency format on Windows since.. longer than I can remember. Heck, even when .NET first shipped in 2000 or 2001 you could register .NET binaries in the GAC. And any DLL in system32 is automatically available for global consumption.

Re: Warp – self-contained, single binary applications

#108

Post by jordwalke » Sat, Oct 13, 2018, 5:59 AM UTC

wildmusings wrote: post #9

From the README, it looks like this unpacks the application and dependencies into a local cache in the user's home directory on first run. Is it possible to instead execute the compressed application code directly and present to the resulting process a virtualized file system that decompresses the dependencies on the fly too? Then you could run the single binary without giving it any filesystem access. One way to do…

I asked this question just yesterday. The challenge (the one I was interested in) was if this can be done in a general purpose way for arbitrary programming languages so long as they used libc to go through the file system. I actually think it might be theoretically possible using LD_PRELOAD and some equivalent on OSX. I don't know why I'm compelled to want the executable to leave no trace on disk, but it seems much cleaner. However, if you _write_ to the virtual file system, that will need to change the packed binary itself which might result in problems (the binary would grow/shrink as you pack more / less files into it at runtime by writing to the "disk").

https://twitter.com/jordwalke/status/1050277143085608960

Maybe it's just not worth it and a temp directory is not so bad. I just worry about little changes to the way `mktmp` works across OS updates etc. A totally self contained executable that leaves no trace, with virtual file system support is about as isolated and reliable as you could ever hope. I just don't think anyone has been determined enough to achieve it.

Re: Warp – self-contained, single binary applications

#109

Post by quickben » Sat, Oct 13, 2018, 6:02 AM UTC

therein wrote: post #106

nur0n wrote: post #76

Earlier quoted context omitted.

I think the inverse argument can be made against shared libraries: if an update introduces a vulnerability, now all programs which depend on that library become vulnerable.

I would like to think that projects overall tend to fix more bugs than introduce them. It's not like projects go from orderly to disorderly over time.

But some are high value targets (crypto ones, etc).

Re: Warp – self-contained, single binary applications

#110

Post by paavoova » Sat, Oct 13, 2018, 6:08 AM UTC

creatornator wrote: post #102

paavoova wrote: post #97

Earlier quoted context omitted.

Python can do something similar albeit far simpler via packaging as an executable zipfile. Just structure everything you need in a directory and zip up with __main__.py in the root. All resources are available as an in-memory binary stream, including individual source .py files. Of course if you want to package something like a C library, you need to unpack that to disk to be able to access it with ctypes.

Are you referring to the zipapp library? I had no idea this even existed until now but that seems pretty clever. Huh. Why can't it also dynamically unzip a shared object library?

You don't need to use any libraries, you can do something as simple as:

  $ zip -r mymodule.zip mymodule/
  $ echo '#!/usr/bin/env python3' > myapp
  $ cat mymodule.zip >> myapp
  $ chmod 755 myapp

CPython will automatically detect it's a zipfile, unzip it in-memory, and import the __main__.py in the root of the module and any nested modules.

The reason why you have to handle shared libraries and any other non-.py data yourself is because Python doesn't know what to do with it. You can access it as binary data via pkgutil.get_data(), but linking shared libraries is system-dependent and Python doesn't load them itself. As the dynamic linker can't find a shared library in a zip file, the only thing you can do is extract it separately. This is documented in https://docs.python.org/3/library/zipimport.html

Any files may be present in the ZIP archive, but only files .py and .pyc are available for import. ZIP import of dynamic modules (.pyd, .so) is disallowed.