Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

71–80 of 711 posts

Re: A love letter to the CSV format

#71

Post by sitkack »

polyrand wrote: post #16

As someone who likes modern formats like parquet, when in doubt, I end up using CSV or JSONL (newline-delimited JSON). Mainly because they are plain-text (fast to find things with just `grep`) and can be streamed. Most features listed in the document are also shared by JSONL, which is my favourite format. It compresses really well with gzip or zstd. Compression removes some plain-text advantages, but ripgrep can sear…

I switched to JSONL over a decade ago and I would recommend everyone else to also have switched then.

This whole thread is an uninformed rehash of bad ideas.

Re: A love letter to the CSV format

#73

Post by emmelaich »

primitivesuave wrote: post #40

One thing that has changed the game with how I work with CSVs is ClickHouse. It is trivially easy to run a local database, import CSV files into a table, and run blazing-fast queries on it. If you leave the data there, ClickHouse will gradually optimize the compression. It's pretty magical stuff if you work in data science.

Simon W's https://datasette.io/ is also excellent.

Re: A love letter to the CSV format

#74

Post by mort96 »

ciupicri wrote: post #57
mort96 wrote: post #38

Earlier quoted context omitted.

TSV looks incredibly ugly when opened in a text editor unless all values are 7 characters or less.

Set a bigger tab size in the editor, e.g. `:set ts=32` in vim.

Aaaand now I can fit like 3 fields next to each other before they wrap and the field with arbitrary length text still misaligns the field after it

Re: A love letter to the CSV format

#75

Post by nayuki »

mort96 wrote: post #38
nayuki wrote: post #10

I greatly prefer TSV over CSV. https://en.wikipedia.org/wiki/Tab-separated_values

TSV looks incredibly ugly when opened in a text editor unless all values are 7 characters or less.

There are some specialized text editors for editing files with tabs. https://en.wikipedia.org/wiki/Tab_stop#Dynamic_tab_stops , https://nick-gravgaard.com/elastic-tabstops/ , https://tibleiz.net/code-browser/

Re: A love letter to the CSV format

#76

Post by jandrese »

andrepd wrote: post #24
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.

If there were visible well known characters that could be printed for those and keys on a keyboard for inputting them we would probably have RSV files. Because they are buried down in the nonprintable section of the ASCII chart they are a pain for people to deal with. All it would have taken is one more key on the keyboard, maybe splitting the tab key in half.

Re: A love letter to the CSV format

#77

Post by emmelaich »

kbouck wrote: post #19
Der_Einzige wrote: post #4

I'm in on the "shit on microsoft for hard to use formats train" but as someone who did a LOT of .docx parsing - it turned into zen when I realized that I can just convert my docs into the easily parsed .html5 using something like pandoc. This is a good blog post and Xan is a really neat terminal tool.

Xan looks great. Miller is another great cli tool for transforming data among csv, tsv, json and other formats. https://miller.readthedocs/

https://miller.readthedocs.io/

Re: A love letter to the CSV format

#78

Post by croes »

hajile wrote: post #65

The argument against JSON isn't very compelling. Adding a name to every field as they do in their strawman example isn't necessary. Compare this CSV field1,field2,fieldN "value (0,0)","value (0,1)","value (0,n)" "value (1,0)","value (1,1)","value (1,n)" "value (2,0)","value (2,1)","value (2,n)" To the directly-equivalent JSON [["field1","field2","fieldN"], ["value (0,0)","value (0,1)","value (0,n)"], ["value (1,0)","…

> Because CSV is so simple, it's common for them to avoid using a parsing/encoding library.

A but unfair to compare CSV without parser library to JSON with library.

Re: A love letter to the CSV format

#79

Post by jandrese »

gibibit wrote: post #8
Qem wrote: post #2

9. Excel hates CSV It clearly means CSV must be doing something right. This is one area where LibreOffice Calc shines in comparison to Excel. Importing CSVs is much more convenient.

Excel won't import ISO 8601 timestamps either, which is crazy these days where it's the universal standard, and there's no excuse to use anything else. You have to replace the "T" separator with a space and also any trailing "Z" UTC suffix (and I think any other timezone/offset as well?) for Excel to be able to parse as a time/date.

I usually have the opposite problem. There's even a joke about it:

How is Excel like an Incel? Both of them think everything is a date.

Re: A love letter to the CSV format

#80

Post by kbouck »

emmelaich wrote: post #77
kbouck wrote: post #19

Earlier quoted context omitted.

Xan looks great. Miller is another great cli tool for transforming data among csv, tsv, json and other formats. https://miller.readthedocs/

https://miller.readthedocs.io/

trimmed the version path off and went a bit too far! thanks.
Post reply on HN