Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

231–240 of 711 posts

Re: A love letter to the CSV format

#231

Post by jstanley »

derriz wrote: post #195
hajile wrote: post #183

Earlier quoted context omitted.

Can you point me to a language with any significant number of users that does NOT have a JSON library? I went looking at some of the more niche languages like Prolog, COBOL, RPG, APL, Eiffel, Maple, MATLAB, tcl, and a few others. All of these and more had JSON libraries (most had one baked into the standard library). The exceptions I found (though I didn't look too far) were: Bash (use jq with it), J (an APL variant)…

I made no claim about JSON libraries. I contested the claim that "CSV libraries and tools suck". They do not.

You claimed that CSV is "easily the most widely supported data format in existence in terms of tools and language support", which is a claim that CSV is better supported than JSON, which is a claim that JSON support is lacking.

Re: A love letter to the CSV format

#233

Post by klysm »

koolba wrote: post #63
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

You serialize the keys on every row which is a bit inefficient but it’s a text format anyway

Re: A love letter to the CSV format

#234

Post by theLiminator »

sitkack wrote: post #71
polyrand wrote: post #16

As someone who likes modern formats like parquet, when in doubt, I end up using CSV or JSONL (newline-delimited JSON). Mainly because they are plain-text (fast to find things with just `grep`) and can be streamed. Most features listed in the document are also shared by JSONL, which is my favourite format. It compresses really well with gzip or zstd. Compression removes some plain-text advantages, but ripgrep can sear…

I switched to JSONL over a decade ago and I would recommend everyone else to also have switched then. This whole thread is an uninformed rehash of bad ideas.

I think that might make sense ingest side, but that's very expensive to deal with if you're doing anything remotely large.

I think sinking into something like delta-lake or iceberg probably makes sense at scale.

But yeah, I definitely agree that CSV is not great.

Re: A love letter to the CSV format

#235

Post by FridgeSeal »

derriz wrote: post #195
hajile wrote: post #183

Earlier quoted context omitted.

Can you point me to a language with any significant number of users that does NOT have a JSON library? I went looking at some of the more niche languages like Prolog, COBOL, RPG, APL, Eiffel, Maple, MATLAB, tcl, and a few others. All of these and more had JSON libraries (most had one baked into the standard library). The exceptions I found (though I didn't look too far) were: Bash (use jq with it), J (an APL variant)…

I made no claim about JSON libraries. I contested the claim that "CSV libraries and tools suck". They do not.

CSV tooling has had to invest enormous amounts of effort to make a fragile, under-specified format half-useful. I would call it ubiquitous, I would call the tooling that we’ve built around it “impressive” but I would by no means call any of it “good”.

I do not miss dealing with csv files in the slightest.

Re: A love letter to the CSV format

#237

Post by shawabawa3 »

jstanley wrote: post #231
derriz wrote: post #195

Earlier quoted context omitted.

I made no claim about JSON libraries. I contested the claim that "CSV libraries and tools suck". They do not.

You claimed that CSV is "easily the most widely supported data format in existence in terms of tools and language support", which is a claim that CSV is better supported than JSON, which is a claim that JSON support is lacking.

Can you import .jsonl files into Google sheets or excel natively?

Re: A love letter to the CSV format

#238

Post by scythe »

sundarurfriend wrote: post #88
andrepd wrote: post #24

Earlier quoted context omitted.

That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.

I remembered seeing a comment like this before, and... comment: https://news.ycombinator.com/item?id=26305052 comment: https://news.ycombinator.com/item?id=39679662 "ASCII Delimited Text – Not CSV or Tab Delimited Text" post [2014]: https://news.ycombinator.com/item?id=7474600 same post [2024]: https://news.ycombinator.com/item?id=42100499 comment: https://news.ycombinator.com/item?id=15440801 (...and many more.) "Th…

>we just need some major text editor to get the ball rolling and start making some attempts to understand these characters

Many text editors offer extensions APIs, including Vim, Emacs, Notepad++. But the ideal behavior would be to auto-align record separators and treat unit separators as a special kind of newline. That would allow the file to actually look like a table within the text editor. Input record separator as shift+space and unit separator as shift+enter.

Re: A love letter to the CSV format

#239

Post by dylan604 »

FridgeSeal wrote: post #235
derriz wrote: post #195

Earlier quoted context omitted.

I made no claim about JSON libraries. I contested the claim that "CSV libraries and tools suck". They do not.

CSV tooling has had to invest enormous amounts of effort to make a fragile, under-specified format half-useful. I would call it ubiquitous, I would call the tooling that we’ve built around it “impressive” but I would by no means call any of it “good”. I do not miss dealing with csv files in the slightest.

Microsoft Windows has had to invest enormous amounts...

Apple macOS has had to invest enormous amounts...

Pick your distro of Linux has had to invest enormous amounts...

None of them a perfect and any number of valid complaints can be said about any of them. None of the complaints make any of the things useless. Everyone has workarounds.

Hell, JSON has had to invest enormous amounts of effort...

Re: A love letter to the CSV format

#240

Post by Someone1234 »

nomel wrote: post #178
Someone1234 wrote: post #107

Earlier quoted context omitted.

For context: I have a LOT of experience of interchange formats, like "full time job, every day, all day, hundreds of formats, for 20-years" experience. Based on that experience I have come to one key, but maybe, counter-intuitive truth about interchange formats: - Too much freedom is bad . Why? Generating interchange data is cheaper than consuming it, because the creator only needs to consider the stuff they want to…

Oh, this is interesting. Are you tying different systems together? If so, do you use some preferred intermediate format? Do you have a giant library of * -> intermediate -> * converters that you sprinkle between everything? Or maybe the intermediate format is in memory? What about Parquet and the like?

It is an interchange format, so it is inter-system by virtue of that. If I am a self-creator/consumer the format I use can be literally anything even binary memory dumps.
Post reply on HN