Python – Create large ZIP archives without memory inflation
1–10 of 29 posts
Re: Python – Create large ZIP archives without memory inflation
#2Is there something like this for the JVM? I’m not sure whether with https://github.com/srikanth-lingala/zip4j#adding-entries-wit... will keep everything it is possible to keep it in constrained memory.
Re: Python – Create large ZIP archives without memory inflation
#3How does it compares to https://github.com/kbbdy/zipstream?
Re: Python – Create large ZIP archives without memory inflation
#4Can someone explain what it's doing? Is it using an algorithm with far superior space complexity than the usual algorithm?
Python seems a curious choice. Compression is computationally intensive.
Re: Python – Create large ZIP archives without memory inflation
#5Can someone explain what it's doing? Is it using an algorithm with far superior space complexity than the usual algorithm? Python seems a curious choice. Compression is computationally intensive.
Looks like it just splits by 16MB chunks, so just standard deflate. Actual compression is handled by the python zipfile module, which is probably C code underneath.
Re: Python – Create large ZIP archives without memory inflation
#6Does anyone know of a tar equivalent which performs deduplication?
Re: Python – Create large ZIP archives without memory inflation
#7I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/deflate is not much more than a 64k window), since it is natural that files can be larger than available RAM.
Re: Python – Create large ZIP archives without memory inflation
#8Is there something like this for the JVM? I’m not sure whether with https://github.com/srikanth-lingala/zip4j#adding-entries-wit... will keep everything it is possible to keep it in constrained memory.
The standard zip tools in Java should be fine. I regularly compress gigs of data in an aws lambda environment. Streaming from and to s3.
Re: Python – Create large ZIP archives without memory inflation
#9I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/deflate is not much more than a 64k window), since it is natural that files can be larger than available RAM.
I think it's meant for a pretty narrow use-case: serving compressed files through frameworks(as mentioned, for example Django or Flask) that expect to serve file objects, but without writing to disk.
The "usual"/naive solution (if you stay within the python ecosystem) is to compress the files and write to a BytesIO or other in-memory file like object, and then have your framework serve it. The naive solution leads to writing the whole file to memory before serving (thus memory inflation).
This library just looks like a pretty straightforward way to implement the same idea, but with chunking to bound memory usage. At the bottom, it's doing the same thing, but using generators to yield chunks at a time.
It's a useful utility for that context. Nothing groundbreaking, it's something that most intermediate and higher developers could stitch together in probably a few days (especially if they had to brush on on DEFLATE and generator protocol), but it's nice to not have to.
Re: Python – Create large ZIP archives without memory inflation
#10Does anyone know of a tar equivalent which performs deduplication?
Should not be too complicated in Python. Just calculate the sha1/sha256 on the file before adding it to the tar-archive, skip any duplicates.