Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: ZSV (Zip Separated Values) columnar data format

github.com

Post reply on HN
Print view View original

71–77 of 77 posts

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#71

Post by hafthor »

thehappypm wrote: post #37
karaterobot wrote: post #36

Earlier quoted context omitted.

What do you mean it's not seekable? > ZIP files are a collection of individually compressed files, with a directory as a footer to the file, which makes it easy to seek to a specific file without reading the whole file... The nature of .zip files makes it possible to seek and read just the columns required without having to read/decode the other columns.

Seeking within a column

There's two ways to limit the number of column-rows you have to read. One is by file partitioning, that is having many ZSV files rather than one giant one, ideally organized by partitioning key field(s). The other way is mentioned as an extension to the format itself which functions much like rowgroups do in Parquet. https://github.com/Hafthor/zsvutil?tab=readme-ov-file#row-gr...

Thanks for taking a look.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#72

Post by rhelz »

mulmen wrote: post #48
rhelz wrote: post #40

Earlier quoted context omitted.

> Like, no human is going to read 50k rows, much less 10m rows. Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in. Your bleary eyes are scaring through log files and data files, trying to find the answer.. ... believe me, you will appreciate human-readable formats for both of those. You just want to cat out the the…

It’s unclear to me that this is actually fewer moving parts. There are already parquet CLI tools. If your data is in Parquet you should know how to use them or at least have them documented in your oncall runbook.

I'm sure parquet is the bees knees, and I'm sure if its your code you'll know how to fix it.

But what if it isn't your code? And you've never heard of parquet before? And its 2AM and they can't get a hold of the guy who wrote it, so they call in you....

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#73

Post by hafthor »

jiggawatts wrote: post #61
hafthor wrote: post #56

Earlier quoted context omitted.

I like your idea of storing columns as JSON arrays. I might play around with that. Thanks for giving it a look.

I have a sinking feeling like I’ve unleashed something here. Some future programmer will be cursing my name as they try to make columnar JSON decoding performant.

hehe. I added an alternative JSON inner format spec to the readme. I need to add JSON and CSV support to the zsvutil itself next. I may actually change the spec to default to JSON. All because of you. haha.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#74

Post by mulmen »

rhelz wrote: post #72
mulmen wrote: post #48

Earlier quoted context omitted.

It’s unclear to me that this is actually fewer moving parts. There are already parquet CLI tools. If your data is in Parquet you should know how to use them or at least have them documented in your oncall runbook.

I'm sure parquet is the bees knees, and I'm sure if its your code you'll know how to fix it. But what if it isn't your code? And you've never heard of parquet before? And its 2AM and they can't get a hold of the guy who wrote it, so they call in you....

Then there’s a serious organizational failing. Parquet is the de facto standard, the chances are a random engineer knows how to interact with it and not ZSV.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#75

Post by rhelz »

jitl wrote: post #49
rhelz wrote: post #40

Earlier quoted context omitted.

> Like, no human is going to read 50k rows, much less 10m rows. Well, its 2AM, some dork has checked in code which breaks production, and it absolutely positively has to be fixed by 6:00am before the customer comes in. Your bleary eyes are scaring through log files and data files, trying to find the answer.. ... believe me, you will appreciate human-readable formats for both of those. You just want to cat out the the…

I’ve never been frustrated at 2am that my data in sqlite3 or Postgres isn’t in a human readable disk format. If I’m working with parquet I’ll have duckdb on hand for fiddling parquet files. I’m much better at SQL at 2 am than I am at piping Unix tools together over N files. I have no idea how I’d drop bad rows from this thing with a bash pipeline anyways, I need to select from one file to find the bad line numbers (g…

> I’ve never been frustrated at 2am that my data in sqlite3

What if its not your data? And you've never used sqlite3 or Postres? Its 2AM, they couldn't get a hold of the guy who wrote the code because he's on vacation, or he wrote it 20 years ago and retired....so they haul your sorry self out of bed?

You really gunna be wanting to be reading sqlite3 tutorials, while your boss and boss's boss's boss is on the video call?

I don't want to overstate my case, and sure there are plenty of reasons to use a database to store your data. I was just trying to answer the question of the person who asked why human-readable formats are preferable. They are not in all circs, of course, but all other things being equal you will appreciate it when the fit hits the shan.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#76

Post by rhelz »

mulmen wrote: post #74
rhelz wrote: post #72

Earlier quoted context omitted.

I'm sure parquet is the bees knees, and I'm sure if its your code you'll know how to fix it. But what if it isn't your code? And you've never heard of parquet before? And its 2AM and they can't get a hold of the guy who wrote it, so they call in you....

Then there’s a serious organizational failing. Parquet is the de facto standard, the chances are a random engineer knows how to interact with it and not ZSV.

> Then there’s a serious organizational failing.

Man, if your success is predicated on working at a company with with no organizational problems....

A scrappy start-up can't afford to hire multiple, redundant engineers--and what with all of the massive layoffs happening, even at the big companies a lot of engineers are going to find themselves debugging other people's code.

Re: Show HN: ZSV (Zip Separated Values) columnar data format

#77

Post by mulmen »

rhelz wrote: post #76
mulmen wrote: post #74

Earlier quoted context omitted.

Then there’s a serious organizational failing. Parquet is the de facto standard, the chances are a random engineer knows how to interact with it and not ZSV.

> Then there’s a serious organizational failing. Man, if your success is predicated on working at a company with with no organizational problems.... A scrappy start-up can't afford to hire multiple, redundant engineers--and what with all of the massive layoffs happening, even at the big companies a lot of engineers are going to find themselves debugging other people's code.

Then embrace industry standards and don't add unknowns. “There are always organizational problems” is a poor justification for creating more problems.
Post reply on HN