Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

551–560 of 711 posts

Re: A love letter to the CSV format

#551

Post by hajile »

dietr1ch wrote: post #251
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

I don't understand why CSV became a thing when TSV, or a format using the nowadays weird ASCII control characters like start/end of text, start of heading, horizontal/vertical tab, file/group/record/unit separator. It seems many possible designs would've avoided the quoting chaos and made parsing sort of trivial.

I don't understand why CSV became a thing in the 70s when S-expressions existed since at least the 50s and are better in practically every way.

Re: A love letter to the CSV format

#552

Post by Moru »

raxxorraxor wrote: post #547
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

Try to live in a country where "," is the decimal point. Of course this causes numerous interoperability issues or hidden mistakes in various data sets. There would have been many better separators... but good idea to bring formatting into it as well...

There was a long period of my life that I thought .csv meant cemicolon separated because all I saw was cemicolon separated files and I had no idea of the pain.

Re: A love letter to the CSV format

#553

Post by a3w »

999900000999 wrote: post #109

The best part about csv, anyone can write a parser in 30 minutes meaning that I can take data from the early '90s and import it into a modern web service. The worst part about CSV, anyone can ride a parser in about 30 minutes, meaning that it's very easy to get incorrect implementations, incorrect data, and other strange undefined behaviors. But to be clear json, and yaml also have issues with everyone trying to rein…

For some side projects, I would like use XML, but cannot read the spec, as that costs money.

So I cannot trust XML in depth, and depend on using a library that bought the spec and hopefully adheres to it.

Re: A love letter to the CSV format

#554

Post by hajile »

hajile wrote: post #320
BeFlatXIII wrote: post #300

Earlier quoted context omitted.

What do you like so much about the pipe?

I think pipe is better too. Typical latin fonts divide characters into three heights: short like "e" or "m", tall like "l" or "P" and deep like "j" or "y". As you may notice, letters only use one or two of these three sections. Pipe is unique in that it uses all three at the same time from the very top to the very bottom. No matter what latin character you put next to it, it remains distinct. This makes the separator…

I forgot to add that you need a "\\" escape for when it appears before |, n, or u in text.

Re: A love letter to the CSV format

#555

Post by gpvos »

Someone wrote: post #451
gpvos wrote: post #419

Earlier quoted context omitted.

What's the problem with capital I?

https://superuser.com/questions/210027/why-does-excel-think-... says itks not capital I but “ID”. Basically, Excel uses the equivalent of ‘file’ ( https://man7.org/linux/man-pages/man1/file.1.html ), sees the magic “ID”, and decides a SYLK file, even though .csv files starting with “ID” have outnumbered .SYLK files by millions for decades.

Thanks. So I guess the easy compatible solution is to always quote the first item on the first line when writing CSV. Good to know. (Checking if the item starts with ID is more work. Possibly quote all items on the first line for simplicity.) (Reading SYLK is obviously irrelevant, so accepting unquoted ID when reading is the smarter way to go and will actually improve compatibility with writers that are not Excel. Also it takes no work.)

Re: A love letter to the CSV format

#557

Post by pwenzel »

jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

I hate CSV too. If I have to use it, I'll live with TSV or some other special-charter delimited format.

Re: A love letter to the CSV format

#558

Post by Ygg2 »

HelloNurse wrote: post #479
Ygg2 wrote: post #465

Earlier quoted context omitted.

You can. YAML 1.2 is only 16 years old. Just old enough to drive. Norway problem has been solved for only 16 years.

YAML 1.2 leaves data types ambiguous, merely making the "Norway problem" optional and at the mercy of the application rather than, in the words of https://yaml.org/type/ (which has not been marked as deprecated), "strongly recommended".

Those schemas aren't part of the core schema, and you may interpret them if you are aiming for full 1.1 compatibility. If you're aiming for 1.1 compatibility, then you accept the Norway problem.

I've been looking in the specs and I can't find the link to the https://yaml.org/type/

Re: A love letter to the CSV format

#559

Post by INTPenis »

jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

The only times I hated CSV was when it came from another system I had no control over. For example Windows and their encodings, or some other proprietary BS.

But CSV under controlled circumstances is very simple.

And speaking of Wintendo, the bonus is often that you can go straight from CSV to Excel presentation for the middle management.

Re: A love letter to the CSV format

#560

Post by Izkata »

IanCal wrote: post #505
dspillett wrote: post #497

Earlier quoted context omitted.

WRT JSON: > Headers are in each line This might be my old “space and network cost savings” reflex, which is a lot less necessary these days, kicking in, but the feels inefficient. It also gives rise to not knowing the whole schema until you read the whole dataset (which might be multiple files), unless some form of external schema definition is provided. Having said that, I accept that JSON has advantages over CSV, e…

Yes, it's not great . Space is annoying, though compression pretty much removes that as a concern (zstd is good for this, you can even have a custom dictionary). And yes, missing keys is annoying. JSONL is handy, JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers. I'm quite a fan of parquet, but never expect to receive that from a client (alas).

> JSON that's in the form {data: [...hundred megs of lines]} is annoying for various parsers.

One reason this became common was a simple protection against json hijacking: https://haacked.com/archive/2009/06/25/json-hijacking.aspx/

Post reply on HN