Can't store tabs or newlines, odd choice.
[deleted]
Show HN: ZSV (Zip Separated Values) columnar data format
41–50 of 77 posts
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#42Earlier quoted context omitted.
Basically it's the same limitations as CSV. At least you could use something less likely to appear in data as record sepator (like 0x1E) Otherwise it's an interesting idea!
As far as I know, thanks to quoting it is possible to put basically any data you want in a CSV.
The problem is there is no uniform standard for quoting and escaping in CSV, and different software uses different variants.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#43The only benefit this format provides is the ability to read some columns without needing to read all columns. Unfortunately it is not a seekable format. That's a pretty big miss. It also wouldn't be that hard to make it seekable. All you would have to do is make each tsv file two columns: record-id, value.
Wouldn’t be too hard to add a secondary “file” in the zip with an extra index
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#44I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think th…
You normally develop the tools and scripts on a much smaller data sets. So you export 1 minute of data, examine it manually or with simple tools, write your scripts and once they work, switch to processing months worth of it.
Human-readable comes handy here.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#45Colour me out of the loop, but what is the utility of this type of approach? I can't seem to grok this.
many colums (100's) and you only need one. This approach makes it much faster.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#46Earlier quoted context omitted.
As far as I know, thanks to quoting it is possible to put basically any data you want in a CSV.
The problem is there is no uniform standard for quoting and escaping in CSV, and different software uses different variants.
There is a standard, and it is very simple and easy to use.
Different software uses different variants because we're not allowed to have nice things and devs are too lazy to use something slightly more complicated than .split(',')
Though if you're going to ban some common characters anyway like TSV, you might as well use CSV and ban commas, newlines, and quotation marks.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#47I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think th…
> Like, no human is going to read 50k rows, much less 10m rows. Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in. Your bleary eyes are scaring through log files and data files, trying to find the answer.. ... believe me, you will appreciate human-readable formats for both of those. You just want to cat out the the…
> Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in.
This is a classic XY problem. The issue isn't the data format, it's the fact that your organizational processes allow random code pushes at 2am that can break the whole thing.
Parquet, used by basically everyone, isn't human readable (and for good reason): it's for big data storage, retrieval, and processing. CSV is human readable (and for good reason): people use that data in Excel or other spreadsheeting software.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#48I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think th…
> Like, no human is going to read 50k rows, much less 10m rows. Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in. Your bleary eyes are scaring through log files and data files, trying to find the answer.. ... believe me, you will appreciate human-readable formats for both of those. You just want to cat out the the…
It’s unclear to me that this is actually fewer moving parts. There are already parquet CLI tools. If your data is in Parquet you should know how to use them or at least have them documented in your oncall runbook.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#49I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think th…
> Like, no human is going to read 50k rows, much less 10m rows. Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in. Your bleary eyes are scaring through log files and data files, trying to find the answer.. ... believe me, you will appreciate human-readable formats for both of those. You just want to cat out the the…
I’ve never been frustrated at 2am that my data in sqlite3 or Postgres isn’t in a human readable disk format.
If I’m working with parquet I’ll have duckdb on hand for fiddling parquet files. I’m much better at SQL at 2 am than I am at piping Unix tools together over N files.
I have no idea how I’d drop bad rows from this thing with a bash pipeline anyways, I need to select from one file to find the bad line numbers (grep I guess, I’ll need to look up how to cut just the line number), and then delete those lines from all the files in a zip (??). Sounds a lot harder than a single SELECT WHERE NOT or DELETE WHERE.
Re: Show HN: ZSV (Zip Separated Values) columnar data format
#50I think “human readability” isn’t a great feature for a columnar data format, because once you get data on a scale where the column oriented layout makes sense, you’re way past the scale where a human would be want to read over the stored data anyways. Like, no human is going to read 50k rows, much less 10m rows. I guess it’s nice you can spot check the rows using only zip & head -n 10 and paste, but I don’t think th…
Even more so if you store personal data in there, which ofc would be encrypted per row.