11–20 of 29 posts
I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/deflate is not much more than a 64k window), since it is natural that files can be larger than available RAM.
I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/deflate is not much more than a 64k window), since it is natural that files can be larger than available RAM.
I think it's meant for a pretty narrow use-case: serving compressed files through frameworks(as mentioned, for example Django or Flask) that expect to serve file objects, but without writing to disk. The "usual"/naive solution (if you stay within the python ecosystem) is to compress the files and write to a BytesIO or other in-memory file like object, and then have your framework serve it. The naive solution leads to…
Examples:
* https://hackage.haskell.org/package/conduit-extra-1.1.7.3/do...
* http://hackage.haskell.org/package/streaming-utils-0.2.0.0/d...
Is there something like this for the JVM? I’m not sure whether with https://github.com/srikanth-lingala/zip4j#adding-entries-wit... will keep everything it is possible to keep it in constrained memory.
I'm a little perplexed by the "marketing" around this --- all the archivers I know of don't require more memory than the compression state (which AFAIK for ZIP/deflate is not much more than a 64k window), since it is natural that files can be larger than available RAM.
It runs on a small embedded device that can stream zip archives many times larger then the disk or system ram without any issue.
Example Python Falcon Proof of Concept:
https://gist.github.com/kylemanna/1e22bbf31b7e5ae84bbdfa32c6...
Other then what Python's zipfile buffers in memory, my implementation shouldn't use much more then a os.pipe()'s buffer (typically 64kB?).
I built a streaming zip app using nothing more then the Python stdlib zip implementation and some os primitives. It runs on a small embedded device that can stream zip archives many times larger then the disk or system ram without any issue. Example Python Falcon Proof of Concept: https://gist.github.com/kylemanna/1e22bbf31b7e5ae84bbdfa32c6... Other then what Python's zipfile buffers in memory, my implementation shou…
I need to open a very large CSV file in Python, which is around 25GB in .zip format. Any idea how to do this in a streaming way, i.e. stopping after reading the first few thousand rows?
I built a streaming zip app using nothing more then the Python stdlib zip implementation and some os primitives. It runs on a small embedded device that can stream zip archives many times larger then the disk or system ram without any issue. Example Python Falcon Proof of Concept: https://gist.github.com/kylemanna/1e22bbf31b7e5ae84bbdfa32c6... Other then what Python's zipfile buffers in memory, my implementation shou…
Interesting. I need to open a very large CSV file in Python, which is around 25GB in .zip format. Any idea how to do this in a streaming way, i.e. stopping after reading the first few thousand rows?
Replace the `file_paths` list in my proof of concept with your large file(s), delete the rest (lines 61-68, 77-79) and it should just work.
I'm also perplexed by the goal as this seems to just call zipfile.write under the hood, which already streams to a zip file without accumulating a memory buffer?
[0] https://github.com/BuzonIO/zipfly/blob/master/zipfly/zipfly....