Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python – Create large ZIP archives without memory inflation

github.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Python – Create large ZIP archives without memory inflation

#21

Post by mlyle »

rahimiali wrote: post #20

I have questions about the code. Why do you need to say int('0x1', 16) and int('0x2', 16)? Why not just write 0x1 and 0x2? Or just plain 1 and 2? I'm also perplexed by the goal as this seems to just call zipfile.write under the hood, which already streams to a zip file without accumulating a memory buffer? [0] https://github.com/BuzonIO/zipfly/blob/master/zipfly/zipfly....

I think the appeal is that it's a generator, so that if you need to encapsulate/cram bytes of the zip over some other transport that you can just naturally ask for a few more each time without having to accumulate it in memory.

Of course, by crafting a special file-like object you could avoid this too, but perhaps a bit less elegantly.

Re: Python – Create large ZIP archives without memory inflation

#22

Post by johndough »

ejwhite wrote: post #18
2bluesc wrote: post #17

I built a streaming zip app using nothing more then the Python stdlib zip implementation and some os primitives. It runs on a small embedded device that can stream zip archives many times larger then the disk or system ram without any issue. Example Python Falcon Proof of Concept: https://gist.github.com/kylemanna/1e22bbf31b7e5ae84bbdfa32c6... Other then what Python's zipfile buffers in memory, my implementation shou…

Interesting. I need to open a very large CSV file in Python, which is around 25GB in .zip format. Any idea how to do this in a streaming way, i.e. stopping after reading the first few thousand rows?

Works fine with Python's standard library. Files in a ZipFile can be read in a streaming manner. There is no need to store all the data in memory.

    import io, csv, zipfile

    max_lines = 10
    with zipfile.ZipFile("data.zip") as z:
        for info in z.infolist():
            with z.open(info.filename) as f:
                reader = csv.reader(io.TextIOWrapper(f))
                for i_line, line in enumerate(reader):
                    if i_line >= max_lines: break
                    print(line)

Re: Python – Create large ZIP archives without memory inflation

#23

Post by jzwinck »

da_big_ghey wrote: post #5
MaxBarraclough wrote: post #4

Can someone explain what it's doing? Is it using an algorithm with far superior space complexity than the usual algorithm? Python seems a curious choice. Compression is computationally intensive.

Looks like it just splits by 16MB chunks, so just standard deflate. Actual compression is handled by the python zipfile module, which is probably C code underneath.

zipfile uses zlib which is C. But it's even better than that: it releases the GIL, so it gives linear speedup with multiple threads. If you need to (de)compress a bunch of files, you can do them all at once quite easily using e.g. concurrent.futures.

If you want that speedup on the command line without Python, check out pigz. It's gzip with parallelism. Easy 10-20x speedup for some jobs.

Re: Python – Create large ZIP archives without memory inflation

#24

Post by CodesInChaos »

thewakalix wrote: post #13
icegreentea2 wrote: post #9

Earlier quoted context omitted.

I think it's meant for a pretty narrow use-case: serving compressed files through frameworks(as mentioned, for example Django or Flask) that expect to serve file objects, but without writing to disk. The "usual"/naive solution (if you stay within the python ecosystem) is to compress the files and write to a BytesIO or other in-memory file like object, and then have your framework serve it. The naive solution leads to…

Ah, so it's like Haskell's streaming (de)compression functions? Examples: * https://hackage.haskell.org/package/conduit-extra-1.1.7.3/do... * http://hackage.haskell.org/package/streaming-utils-0.2.0.0/d...

The functions you link look like they're for simple deflate streams (i.e. a single file), while the OP appears to be about streaming zip archives which can contain multiple files with metadata.

Re: Python – Create large ZIP archives without memory inflation

#25

Post by harel »

icegreentea2 wrote: post #9
userbinator wrote: post #7

I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/deflate is not much more than a 64k window), since it is natural that files can be larger than available RAM.

I think it's meant for a pretty narrow use-case: serving compressed files through frameworks(as mentioned, for example Django or Flask) that expect to serve file objects, but without writing to disk. The "usual"/naive solution (if you stay within the python ecosystem) is to compress the files and write to a BytesIO or other in-memory file like object, and then have your framework serve it. The naive solution leads to…

"without writing to disk"

This is a valid concern and a good enough reason to have such a library. I've written a similar thing for uploading large files to S3 (via Django) by streaming them without the file ever touching the file system (S3ChunkUploader). The reason was the large files were deemed security-sensitive and the containers were limited to 2GB in disk space. Just uploading 4 500MB files at the same time would be an attack vector.

Re: Python – Create large ZIP archives without memory inflation

#27

Post by amelius »

TheChaplain wrote: post #10
amelius wrote: post #6

Does anyone know of a tar equivalent which performs deduplication?

Should not be too complicated in Python. Just calculate the sha1/sha256 on the file before adding it to the tar-archive, skip any duplicates.

Yes, but I don't want to let it evolve into yet another side-project.

Re: Python – Create large ZIP archives without memory inflation

#28

Post by AzN1337c0d3r »

jzwinck wrote: post #23
da_big_ghey wrote: post #5

Earlier quoted context omitted.

Looks like it just splits by 16MB chunks, so just standard deflate. Actual compression is handled by the python zipfile module, which is probably C code underneath.

zipfile uses zlib which is C. But it's even better than that: it releases the GIL, so it gives linear speedup with multiple threads. If you need to (de)compress a bunch of files, you can do them all at once quite easily using e.g. concurrent.futures. If you want that speedup on the command line without Python, check out pigz. It's gzip with parallelism. Easy 10-20x speedup for some jobs.

Note that pigz only parallelizes compression, not decompression.

Re: Python – Create large ZIP archives without memory inflation

#29

Post by 2bluesc »

johndough wrote: post #22
ejwhite wrote: post #18

Earlier quoted context omitted.

Interesting. I need to open a very large CSV file in Python, which is around 25GB in .zip format. Any idea how to do this in a streaming way, i.e. stopping after reading the first few thousand rows?

Works fine with Python's standard library. Files in a ZipFile can be read in a streaming manner. There is no need to store all the data in memory. import io, csv, zipfile max_lines = 10 with zipfile.ZipFile("data.zip") as z: for info in z.infolist(): with z.open(info.filename) as f: reader = csv.reader(io.TextIOWrapper(f)) for i_line, line in enumerate(reader): if i_line >= max_lines: break print(line)

This is true when writing to a file. The goal of my PoC was to not write a file and instead to stream to the web browser.
Post reply on HN