Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

641–650 of 711 posts

Re: A love letter to the CSV format

#641

Post by recursive »

jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

> I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec.

Have you tried RFC 4180?

https://www.ietf.org/rfc/rfc4180.txt

Re: A love letter to the CSV format

#642

Post by kragen »

gthompson512 wrote: post #638
kragen wrote: post #610

Earlier quoted context omitted.

> And many inexperienced developers seem to think that they can generate CSV without using a library because the format is supposedly so simple. Can't they? def excel_csv_of(rows): for row in rows: for i, field in enumerate(row): if i: yield ',' yield '"' for c in field: yield '""' if c == '"' else c yield '"' yield '\n' I haven't tested this, even to see if the code parses. What did I screw up?

This forces each field to be quoted, and it assumes that each row has the same fields in the same order. A library can handle the quoting issues and fields more reliably. Not sure why you went with a generator for this either. Most people expect something like `12,,213,3` instead of `"12","213","3"` which yours might give. https://en.wikipedia.org/wiki/Comma-separated_values#Basic_r...

Forcing each field to be quoted is always correct, isn't it? How could something be "more reliable" than something that is always correct?

With respect to "the same fields in the same order", no, although you may or may not feed the CSV to an application that has such an expectation. But if you apply it to data like [("Points",),(),("x","y"),("3","4"),("6","8","10")] it will successfully preserve that wonky structure in a file Excel can ingest reliably. (As reliably as Excel can ingest anything, anyway, since Excel has its own Norway problem.)

It's true that it's possible to produce more optimized output, but I didn't claim that the output was optimal, just correct.

Using generators is necessary to be able to correctly output individual fields that are many times larger than physical memory.

Re: A love letter to the CSV format

#643

Post by recursive »

mjw_byrne wrote: post #531
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

I used to be a data analyst at a Big 4 management consultancy, so I've seen an awful lot of this kind of thing. One thing I never understood is the inverse correlation between "cost of product" and "ability to do serialisation properly". Free database like Postgres? Perfect every time. Big complex 6-figure e-discovery system? Apparently written by someone who has never heard of quoting, escaping or the difference bet…

"Enterprise software" has been defined as software that is purchased based on the decisions of people that will not use it. I think that explains a lot.

Re: A love letter to the CSV format

#644

Post by joquarky »

d0mine wrote: post #370
sramsay64 wrote: post #360

Earlier quoted context omitted.

In fairness there are also several ambiguities with JSON. How do you handle multiple copies of the same key? Does the order of keys have semantic meaning? jq supports several pseudo-JSON formats that are quite useful like record separator separated JSON, newline separated JSON. These are obviously out of spec, but useful enough that I've used them and sometimes piped them into a .json file for storage. Also, encoding…

JSON lines is not JSON It is built on top of it. .jsonl extension can be used to make it clear https://jsonlines.org/

Back in my day it was called NDJSON.

The industry is so chaotic now we keep giving the same patterns different names, adding to the chaos.

Re: A love letter to the CSV format

#645

Post by recursive »

IanCal wrote: post #403
lelanthran wrote: post #396

Earlier quoted context omitted.

To be honest, I'm wondering why you are rating JSON higher than CSV. > Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, There is, actually, RFC 4180 IIRC. > there are many flavours that are incompatible with each other in the sense that a reader for one flavour would not be suitable for reading the other and vice versa. "There are many flavours that deviate from the s…

Have you had to work with csv files from the wild much? I'm not being snarky but what you're talking about is night and day to what I've experienced over the years. There aren't vast numbers of different JSON formats. There's practically one and realistically maybe two. Headers are in each line, utf8 has never been an issue for me and quoting and escaping are well defined and obeyed. This is because for datasets, alm…

> There aren't vast numbers of different JSON formats.

Independent variations I have seen:

* Trailing commas allowed or not * Comments allowed or not * Multiple kinds of date serialization conventions * Divergent conventions about distinguishing floating point types from integers * Duplicated key names tolerated or not * Different string escaping policies, such as, but not limited to "\n" vs "\x0a"

There are bazillions of JSON variations.

Re: A love letter to the CSV format

#646

Post by recursive »

flkenosad wrote: post #545
sgarland wrote: post #538

Earlier quoted context omitted.

> Waiting for someone to write a love letter to the infamous Windows INI file format... Honestly, it’s fine. TOML is better if you can use it, but otherwise for simple applications, it’s fine. PgBouncer still uses INI, though that in particular makes me twitch a bit, due to discovering that if it fails to parse its config, it logs the failed line (reasonable), which can include passwords if it’s a DSN string.

I should write a love letter to JSON.

For as good as JSON is or is not, it's definitely not under-rated.

Re: A love letter to the CSV format

#647

Post by y42 »

jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

why you hate csv, not the program that is not able to properly create csv?

Re: A love letter to the CSV format

#648

Post by recursive »

derriz wrote: post #166
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data. This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.

> Despite the lack of any sort of specification

People keep saying this but RFC 4180 exists.

Re: A love letter to the CSV format

#649

Post by recursive »

pasc1878 wrote: post #501
Kinrany wrote: post #271

Earlier quoted context omitted.

https://datatracker.ietf.org/doc/html/rfc4180 exists

And does Excel fully comply and more imprtantly tell you when the CSV file is wrong

No. Excel's fault. Not CSV. There are plenty of busted CSV parsers (and serializers) too.

Re: A love letter to the CSV format

#650

Post by recursive »

zeroimpl wrote: post #293
rcbdev wrote: post #153

Earlier quoted context omitted.

Same for JSON though. What Python considers a valid JSON might not be that if you ask a Java library.

Example? I know there's some ambiguity over whether literals like false are valid JSON, but I can't think of anything else.

Trailing commas, comments, duplicate key names, for a few examples.
Post reply on HN