We need to have a come to Jesus meeting about these columnar formats.
[deleted]
Show HN: ZSV (Zip Separated Values) columnar data format
31–40 of 77 posts
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#32The only benefit this format provides is the ability to read some columns without needing to read all columns. Unfortunately it is not a seekable format. That's a pretty big miss. It also wouldn't be that hard to make it seekable. All you would have to do is make each tsv file two columns: record-id, value.
[deleted]
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#33[flagged]
[deleted]
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#34Can we just all converge on Parquet + Arrow and call it a day please? Too much effort being put into 1..N ways to solve a problem that would be better put towards a single standard. We work with Parquet + Arrow every day at $DAYJOB in a ML and Big Data context and it's been great. We don't even think we're using it to its fullest potential, but it's never been the bottleneck for us.
How is the data schema description language btw? I haven't used either yet.
Haven't used it directly myself. We mostly just use it for DataFrame crunching via pandas and/or polars (our usage is mixed) which tends to benefit nicely from columnar access.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#35I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think that nice-ness is a good reason to pick a format that forbids common ASCII characters and doesn’t have widespread support.
It’s guess there’s a sort of perma-computing angle here, this format is simple enough that you could pack a lot of almanac data into it, and given a working zlib get it back out with very limited dependencies.
But given the petabytes of parquet files out there, I feel like the format is here to stay, much like sqlite is here to stay.
EDIT: there is a great handy CLI tool for doing SQL on parquet, csv, sqlite3, and other tabular data formats called duckdb. Handy for wrangling and analyzing tabular data from 100 to 10m rows and up.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#36The only benefit this format provides is the ability to read some columns without needing to read all columns. Unfortunately it is not a seekable format. That's a pretty big miss. It also wouldn't be that hard to make it seekable. All you would have to do is make each tsv file two columns: record-id, value.
What do you mean it's not seekable?
> ZIP files are a collection of individually compressed files, with a directory as a footer to the file, which makes it easy to seek to a specific file without reading the whole file... The nature of .zip files makes it possible to seek and read just the columns required without having to read/decode the other columns.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#37The only benefit this format provides is the ability to read some columns without needing to read all columns. Unfortunately it is not a seekable format. That's a pretty big miss. It also wouldn't be that hard to make it seekable. All you would have to do is make each tsv file two columns: record-id, value.
What do you mean it's not seekable? > ZIP files are a collection of individually compressed files, with a directory as a footer to the file, which makes it easy to seek to a specific file without reading the whole file... The nature of .zip files makes it possible to seek and read just the columns required without having to read/decode the other columns.
Seeking within a column
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#38Earlier quoted context omitted.
What is NLJSON?
Also known as JSONL, or JSON Lines. Basically a file of JSON objects separated by newlines. Popular format for logs these days for obvious reasons.
https://jsonlines.org/ was the first "this is trivial but let's write it down so maybe the name will stick" spec for it (from 2013ish)
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#39I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think th…
[deleted]
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#40I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think th…
> Like, no human is going to read 50k rows, much less 10m rows.
Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in.
Your bleary eyes are scaring through log files and data files, trying to find the answer..
... believe me, you will appreciate human-readable formats for both of those. You just want to cat out the the entries in the db which the new code can't handle... the last thing you want to do is to have to invoke some other tool or write some other script to make the data human readable.
And when you find the problem, you will want to just be able to edit a text file containing test cases to verify the fix.
You don't want to write some script to generate and insert the data....at 2am, you are likely to write a buggy script which may keep you from realizing that you've already fixed the problem....or worse, indicate that you have fixed the problem when you haven't.
Fewer moving parts is always better.