Live data from Hacker News

Show HN: ZSV (Zip Separated Values) columnar data format

github.com

61–70 of 77 posts

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#61
post #56

This copied the superficial data layout without the key benefit of modern columnar formats: segment elimination. Most such formats support efficient querying by skipping the disk read step entirely when a chunk of data is not relevant to a query. This is done by splitting the data into segments of about 100K rows, and then calculating the min/max range for each column. That is stored separately in a header or small m…

I like your idea of storing columns as JSON arrays. I might play around with that. Thanks for giving it a look.

I have a sinking feeling like I’ve unleashed something here.

Some future programmer will be cursing my name as they try to make columnar JSON decoding performant.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#62
post #58

The core idea with these compressed columnar "big data" formats is that they minimize storage accesses. Nothing else really matters as much. If you're gonna have to load a sizable chunk of the file to get to the bits you need, the format you store it in starts mattering less. What this gets right: Part of the reason you want to store columns together is that similar values compress well, so you could reduce your IO:…

Thanks for taking a look. Regarding seekable columns, that's the reason why I use the ZIP file format. It has a central directory at the end of the ZIP file that has locations to each file inside, making it so you can seek to a specific column file to extract.

Oh interesting - I missed that tidbit! So with that and row groups and the metadata (assuming you have one metadata block per column in each row group) I think you get to full seekability, right?

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#63
I love this kind of minimalist and clever solution where one developer delivers value similar to that of projects with tens of developers involved. Unfortunately, it's still not enough to defeat the true army of complexity, which contains thousands of developers :)

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#64

Earlier quoted context omitted.

How is the data schema description language btw? I haven't used either yet.

Haven't used it directly myself. We mostly just use it for DataFrame crunching via pandas and/or polars (our usage is mixed) which tends to benefit nicely from columnar access.

Check out DFLib (https://dflib.org)

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#65
post #38

Earlier quoted context omitted.

Also known as JSONL, or JSON Lines. Basically a file of JSON objects separated by newlines. Popular format for logs these days for obvious reasons.

https://jsonlines.org/ was the first "this is trivial but let's write it down so maybe the name will stick" spec for it (from 2013ish)

Missed opportunity to just call it JSONS.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#66
post #63

I love this kind of minimalist and clever solution where one developer delivers value similar to that of projects with tens of developers involved. Unfortunately, it's still not enough to defeat the true army of complexity, which contains thousands of developers :)

Thanks for taking a look.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#67
post #57

Earlier quoted context omitted.

I was comparing it with Parquet, which is much more complex, but has features that help you access the data in less than O(n), like row groups and pages.

you mentioned NLJSON and CSV, which would require to read all columns from the disk.

Yes, but you would usually have to read at least two columns anyway. What are the datasets that are too large to be ingested completely, but too small for a proper columnar format?

If ZSV is meant to occupy the gap between CSV/NLJSON (smaller datasets) and Parquet/DuckDB (larger datasets), this niche is actually really small, if not nonexistent.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#68
post #60

It is simple, but how do you access the price in row #1234567890? If your data doesn't have this many records and can fit into RAM, a basic NLJSON or CSV will work just as well.

There's two ways to limit the number of column-rows you have to read. One is by file partitioning, that is having many ZSV files rather than one giant one, ideally organized by partitioning key field(s). The other way is mentioned as an extension to the format itself which functions much like rowgroups do in Parquet. https://github.com/Hafthor/zsvutil?tab=readme-ov-file#row-gr... Thanks for taking a look.

Oh, sorry, I must've missed the part about rowgroups and metadata. Yes, this should work to limit the scans to a reasonable amount.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#69
post #57

Earlier quoted context omitted.

you mentioned NLJSON and CSV, which would require to read all columns from the disk.

Yes, but you would usually have to read at least two columns anyway. What are the datasets that are too large to be ingested completely, but too small for a proper columnar format? If ZSV is meant to occupy the gap between CSV/NLJSON (smaller datasets) and Parquet/DuckDB (larger datasets), this niche is actually really small, if not nonexistent.

yes it's unclear to me what is the advantage over parquet with compression. And there are enough file formats flying around already.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#70
post #47
post #40

Earlier quoted context omitted.

> Like, no human is going to read 50k rows, much less 10m rows. Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in. Your bleary eyes are scaring through log files and data files, trying to find the answer.. ... believe me, you will appreciate human-readable formats for both of those. You just want to cat out the the…

> Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in. This is a classic XY problem. The issue isn't the data format, it's the fact that your organizational processes allow random code pushes at 2am that can break the whole thing. Parquet, used by basically everyone , isn't human readable (and for good reason): it's…

This is a classic I don't have a catchy term for it problem, where someone focusses on the details of a given contrived example and thinks the problem and the solution are all solved by addressing the details of that specific contrived example.

The 2am story above is not the problem. Thinking up something that would have avoided that specific story like "have 3 people in shift rotation instead of just yourself, and then you never have a 2am problem", or "don't push code at 2am" is not a solution to the problem.

The value of simple data formats that can be directly read by a human without any special tools, is that it makes the data accessible to reading, analyzing, even processing or modifying, in all of the unknown unknowable infinite possible situations.

You don't know ahead of time that you will one day solve a problem deep in the trenches by being able to read or maybe even modify a file or the stdin in a cgi before some crazy untouchable special app picks it up. You don't know ahead of time that the platform will not have a db client you can use to access the data, but you do know that everything can process text, even an obscure cpu with no gcc or git or any of your usual nice toys, has some sort of shell and some sort of text editor.

You can't touch the main app which is some legacy mainframe banking thing or something, you don't have and can't install or compile anything, but you can still read the data and see that there is some unicode character scattered all through it, and you can set up a dirty hack stream edit to convert it to a single byte ascii replacement, using nothing but plain posix sh or some equivalent no matter what the platform.

And that ugly hack is nine thousand times more useful to the bosses and to yourself than not being able to see what was wrong with the data, and then only being able to say "the other side is sending us bad data, it will be broken until they fix their end, or until we can modify the crazy untouchable thing on our end, because I'm a helpless useless twat"

You can't predict ahead of time exactly when or why or how you will end up wanting to be able to access the data without the normal proper tools or apps from the happy path. But it's a fact that it happens, and having the option is more useful than not having the option. And being the person who can solve a problem is more useful than being the person who can't do anything any other way except the normal expected way.

No one said this trumps all other considerations for all jobs for all data, just that it's very valuable, a consideration among other considerations, and you can't predict all the specific ways in which it is valuable, and so giving it up has to be necessary not thoughtless.

Post reply on HN