Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: ZSV (Zip Separated Values) columnar data format

github.com

Post reply on HN
Print view View original

21–30 of 77 posts

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#21

Post by 8n4vidtmkvmk »

olejorgenb wrote: post #13
nextaccountic wrote: post #9

Earlier quoted context omitted.

can't you just do quoting?

https://github.com/Hafthor/zsvutil?tab=readme-ov-file#what-a... > Any escaping or encoding of these characters would make the format less human-readable, harder to parse and could introduce ambiguity and consistency problems. Found the wording of "could introduce ambiguity and consistency problems" a bit odd, but guess they mean that even if things are specified precisely (so there's no ambiguity) not everyone would…

Please. I wrote a csv parser a couple weeks ago in an hour or two. It's not that hard to handle the quoting and edge cases. Yes, maybe different parsers will handle them differently, but just document your choices and that's that. How is ambiguity better than completely disallowing certain chars? That's a non-starter

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#22

Post by cm2187 »

orthoxerox wrote: post #16

It is simple, but how do you access the price in row #1234567890? If your data doesn't have this many records and can fit into RAM, a basic NLJSON or CSV will work just as well.

Like parquet this isn't really meant for RDBMS type of database, more like for analytics over large datasets. I work in an environment where we typically have tables with over 300 columns, 10s if not 100s millions of rows daily. When you want to do a simple sum/group by involving 2 or 3 columns, it is great to have a column store file format, where you only read the columns you need and those are compressed.

The price you pay is that it is inefficient for single record access, or for "select * " kind of queries.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#24

Post by andenacitelli »

Can we just all converge on Parquet + Arrow and call it a day please? Too much effort being put into 1..N ways to solve a problem that would be better put towards a single standard.

We work with Parquet + Arrow every day at $DAYJOB in a ML and Big Data context and it's been great. We don't even think we're using it to its fullest potential, but it's never been the bottleneck for us.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#25

Post by orthoxerox »

cm2187 wrote: post #22
orthoxerox wrote: post #16

It is simple, but how do you access the price in row #1234567890? If your data doesn't have this many records and can fit into RAM, a basic NLJSON or CSV will work just as well.

Like parquet this isn't really meant for RDBMS type of database, more like for analytics over large datasets. I work in an environment where we typically have tables with over 300 columns, 10s if not 100s millions of rows daily. When you want to do a simple sum/group by involving 2 or 3 columns, it is great to have a column store file format, where you only read the columns you need and those are compressed. The pric…

I was comparing it with Parquet, which is much more complex, but has features that help you access the data in less than O(n), like row groups and pages.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#27

Post by rzzzt »

chuckadams wrote: post #19
CapitalistCartr wrote: post #17

Earlier quoted context omitted.

What is NLJSON?

Also known as JSONL, or JSON Lines. Basically a file of JSON objects separated by newlines. Popular format for logs these days for obvious reasons.

NDJSON is the shorthand I've seen: https://github.com/ndjson/ndjson-spec

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#30

Post by pdimitar »

andenacitelli wrote: post #24

Can we just all converge on Parquet + Arrow and call it a day please? Too much effort being put into 1..N ways to solve a problem that would be better put towards a single standard. We work with Parquet + Arrow every day at $DAYJOB in a ML and Big Data context and it's been great. We don't even think we're using it to its fullest potential, but it's never been the bottleneck for us.

How is the data schema description language btw? I haven't used either yet.
Post reply on HN