Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

361–370 of 711 posts

Re: A love letter to the CSV format

#361

Post by defrost »

kmoser wrote: post #358
susam wrote: post #263

Earlier quoted context omitted.

Clearly they should have gone with BEL as the delimiter. printf "alice\007london\007uk\nbob\007paris\007france\n" > data.bsv I'm hoping no reasonable person would ever use BEL as punctuation or decimal separator.

On the off chance you're not being facetious, why not ASCII 0 as a delimiter? (This is a rhetorical question.)

There are some decent arguments for BEL over NUL, however given you posed that as a rhetorical question I feel I can say little other than

ding! ding! ding! winner winner, chicken dinner!

Although BEL would drive me up the wall if I broke out any of my old TTY hardware.

Re: A love letter to the CSV format

#362

Post by diekhans »

sramsay64 wrote: post #360
jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

In fairness there are also several ambiguities with JSON. How do you handle multiple copies of the same key? Does the order of keys have semantic meaning? jq supports several pseudo-JSON formats that are quite useful like record separator separated JSON, newline separated JSON. These are obviously out of spec, but useful enough that I've used them and sometimes piped them into a .json file for storage. Also, encoding…

Plus the 64-bit integer problem, really 52-bit integers, due to JS not having integers.

Re: A love letter to the CSV format

#363

Post by kazinator »

[deleted] wrote: post #353
kazinator wrote: post #289

Earlier quoted context omitted.

A format consisting of newline-terminated records, each containing comma-separated JSON strings would be superior to CSV. It could use backslash escapes to denote control characters and Unicode points. Everyone would agree exactly on what the format is, in contrast to the zoo of CSV variants. It wouldn't have pitfalls in it, like spaces that defeat quotes RFC CSV JSON strings a,"b c" "a", "b c" a, "b c" "a", " \" b c…

[deleted]

"Any JSON primitive" does add a few requirements not semantically comparable to CSV, like numbers that are numbers, and keywords true, false, none.

When these syntaxes are parsed into objects, either the type info has to be retaind, or some kind of attribute tag, so they can be output back to the same form.

> make it so any consumer can parse it by splitting on newline and then ...

There is something like that called JSON-lines. It has a .org domain 'n' everything:

https://jsonlines.org/

Re: A love letter to the CSV format

#364

Post by stevage »

They're completely skipping over the complications of header rows and front matter.

"8. Reverse CSV is still valid CSV" is not true if there are header rows for instance.

But really, whether or not CSV is a good format or not comes down to how much control you have over the input you'll be reading. If you have to deal with random CSV from "in the wild", it's pretty rough. If you have some sort of supplier agreement with someone that's providing the data, or you're always parsing data from the same source, it's pretty fine.

Re: A love letter to the CSV format

#365

Post by 0cf8612b2e1e »

diekhans wrote: post #362
sramsay64 wrote: post #360

Earlier quoted context omitted.

In fairness there are also several ambiguities with JSON. How do you handle multiple copies of the same key? Does the order of keys have semantic meaning? jq supports several pseudo-JSON formats that are quite useful like record separator separated JSON, newline separated JSON. These are obviously out of spec, but useful enough that I've used them and sometimes piped them into a .json file for storage. Also, encoding…

Plus the 64-bit integer problem, really 52-bit integers, due to JS not having integers.

That’s a JavaScript problem, not JSON.

Re: A love letter to the CSV format

#366

Post by darthwalsh »

qw wrote: post #255
tfvlrue wrote: post #223

Earlier quoted context omitted.

In the past I remember that Excel not properly handling UTF-8 encoded text in a CSV. It would treat it as raw ASCII (or possibly code page 1252). So if you opened and saved a CSV, it would corrupt any Unicode text in the file. It's possible this has been fixed in newer versions, I haven't tried in a while.

It's related to how older versions of Windows/Office handled Unicode in general. From what I have heard, it's still an issue with Excel, although I assume that Windows may handle plain text better these days (I haven't used it in a while) You need to write an UTF-8 BOM at the beginning (0xEF, 0xBB, 0xBF), if you want to make sure it's recognized as UTF-8.

Ugh, UTF-8 BOM. Many apps can handle UTF-8 but will try to return those bytes as content; maybe ours in 2015 too

I was on the Power Query team when we were improving the encoding sniffing. An app can scan ahead i.e. 64kB, but ultimately the user needs to just say what the encoding is. All the Power Query data import dialogs should let you specify the encoding.

Re: A love letter to the CSV format

#367

Post by cbsmith »

polyrand wrote: post #16

As someone who likes modern formats like parquet, when in doubt, I end up using CSV or JSONL (newline-delimited JSON). Mainly because they are plain-text (fast to find things with just `grep`) and can be streamed. Most features listed in the document are also shared by JSONL, which is my favourite format. It compresses really well with gzip or zstd. Compression removes some plain-text advantages, but ripgrep can sear…

You can cat Parquet and other formats into grep just as easily.

Re: A love letter to the CSV format

#368

Post by d0mine »

jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

I lived through SOAP/WSDL horror with their numerous standards and the lack of compatibility between stacks in different programming languages. Having seen abused XML, CSV formats. CSV is preferable over XML. Human-readability matters. Relative simplicity matters.

Despite JSON may also be interpreted differently by different tools, it is a good default choice for communicating between programs

Re: A love letter to the CSV format

#370

Post by d0mine »

sramsay64 wrote: post #360
jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

In fairness there are also several ambiguities with JSON. How do you handle multiple copies of the same key? Does the order of keys have semantic meaning? jq supports several pseudo-JSON formats that are quite useful like record separator separated JSON, newline separated JSON. These are obviously out of spec, but useful enough that I've used them and sometimes piped them into a .json file for storage. Also, encoding…

JSON lines is not JSON It is built on top of it. .jsonl extension can be used to make it clear https://jsonlines.org/
Post reply on HN