Live data from Hacker News

A love letter to the CSV format

github.com

51–60 of 711 posts

Re: A love letter to the CSV format

#51
post #28
post #24

Earlier quoted context omitted.

That would be solved by using the ASCII control chars Record Separator / Unit Separator! I don't get how this is not widely used as standard.

The _entire_ point of a CSV file is that it's fully human readable and write-able. The characters you mention could be used in a custom delimiter variant of the format, but at that point it's back to a binary machine format.

Can't there be some magic sequence (like two unescaped newlines) to start a new record?

Re: A love letter to the CSV format

#52
post #8

Earlier quoted context omitted.

Excel won't import ISO 8601 timestamps either, which is crazy these days where it's the universal standard, and there's no excuse to use anything else. You have to replace the "T" separator with a space and also any trailing "Z" UTC suffix (and I think any other timezone/offset as well?) for Excel to be able to parse as a time/date.

Honestly I’m happier when Excel doesn’t try to convert anything. too much bugginess.

Especially gene names. It was so bad that the scientific community renamed the genes in question rather than suffering from the same horror endlessly.

[0] https://www.theverge.com/2020/8/6/21355674/human-genes-renam...

Re: A love letter to the CSV format

#53
post #28

Earlier quoted context omitted.

The _entire_ point of a CSV file is that it's fully human readable and write-able. The characters you mention could be used in a custom delimiter variant of the format, but at that point it's back to a binary machine format.

What do you mean? I just push the Record Separator key on my keyboard. /s in case :)

I’m pretty sure those used to exist.

But when looking for a picture to back up my (likely flawed) memory, Google helpfully told me that you can get a record separator character by hitting Ctrl-^ (caret). Who knew?

Re: A love letter to the CSV format

#54

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

I want to push Sqlite as a data interchange format! it has the benefit of being well defined, and can store binary data, like images for product pictures inside the database. not a good idea if you're trying to serve users behind a web app, but as interchange, better than a zip file with filenames that have to be "relinked".

Re: A love letter to the CSV format

#55
post #50

"the controversial ex-post RFC 4180" I looked at the RFC. What is controversial about it?

You mean aside from the fact it's ex-post ...

Doesn't it make sense to have a common document in the usual format (RFC) which every newbie can consult when in doubt?

I much prefer that to any sort of "common institutional memory" that is nevertheless only talked about on random forums. People die, other people enter the field... hello subtle incompatibilities.

Re: A love letter to the CSV format

#56
>This is so simple you might even invent it yourself without knowing it already exists while learning how to program.

This is a double-edged sword. The "you might even event it yourself" simplicity means that in practice lots of different people do end up just inventing their own version rather than standardizing to RFC-4180 or whatever when it comes to "quote values containing commas", values containing quotes, values containing newlines, etc. And the simplicity means these type of non-standard implementations can go completely undetectable until a problematic value happens to be used. Sometimes added complexity that forces paying more attention to standards and quickly surfaces a diversion from those standards is helpful.

Re: A love letter to the CSV format

#57
post #38
post #10

I greatly prefer TSV over CSV. https://en.wikipedia.org/wiki/Tab-separated_values

TSV looks incredibly ugly when opened in a text editor unless all values are 7 characters or less.

Set a bigger tab size in the editor, e.g. `:set ts=32` in vim.

Re: A love letter to the CSV format

#59
post #10

I greatly prefer TSV over CSV. https://en.wikipedia.org/wiki/Tab-separated_values

100% agree. TSV is under-rated. Tabs don't naturally occur in data nearly as often as commas so tabs are a great delimiter. Copy paste into Excel also works much better with tabs.

Code editors may convert tabs to spaces but are you really editing and saving TSV data files in your code editor?

Re: A love letter to the CSV format

#60

"the controversial ex-post RFC 4180" I looked at the RFC. What is controversial about it?

Look at how it handles escaping of special characters and particularly new lines (RFC 4180 doesn’t guarantee that a new line is a new record) and how it’s written in 2005 yet still doesn’t handle unicode other than via a comment about ”other character sets”.
Post reply on HN