Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

371–380 of 711 posts

Re: A love letter to the CSV format

#371

Post by nmz »

juliansimioni wrote: post #108
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

What happens when you need to encode the newline character in your data? That makes splitting _either_ CSV or LDJSON files difficult.

It's tricky, but simple enough, RFC states that " must be used, inserting a " is done with "". This makes knowing what a record is difficult, since you must keep a variable that keeps the entire string.

How do you do this simply? you read each line, and if there's an uneven number of ", then you have an incomplete record and you will keep all lines until there is an odd number of ". after having the string, parsing the fields correctly is harder but you can do it in regex or PEGs or a disgusting state machine.

Re: A love letter to the CSV format

#372

Post by [deleted] »

jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

[deleted]

Re: A love letter to the CSV format

#373

Post by nmz »

koolba wrote: post #63
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

This is simply not true, parsing json v csv is a difference of thousands of lines.

Re: A love letter to the CSV format

#374

Post by d0mine »

diekhans wrote: post #362
sramsay64 wrote: post #360

Earlier quoted context omitted.

In fairness there are also several ambiguities with JSON. How do you handle multiple copies of the same key? Does the order of keys have semantic meaning? jq supports several pseudo-JSON formats that are quite useful like record separator separated JSON, newline separated JSON. These are obviously out of spec, but useful enough that I've used them and sometimes piped them into a .json file for storage. Also, encoding…

Plus the 64-bit integer problem, really 52-bit integers, due to JS not having integers.

JSON itself is not limited to neither 52 nor 64-bit integers.

    integer = -? (digit | onenine digit+)
https://json.org/

Re: A love letter to the CSV format

#375

Post by akritid »

Eridrus wrote: post #355
crazygringo wrote: post #350

Earlier quoted context omitted.

I'm not clear why quotes prevent parallel processing? I mean, you don't usually parallelize reading a file in the first place, only processing what you've already read and parsed. So read each record in one process and then add it to a multiprocessing queue for multiple processes to handle. And data corruption is data corruption. If a movie I'm watching has a corrupted bit I don't mind a visual glitch and I want it t…

Doing sequential reading into a queue for workers to read is a lot more complicated than having a file format that supports parallel reading. And the fix to allow parallel reading is pretty trivial: escape new lines so that you can just keep reading until the first unescaped new line and start at that record. It is particularly helpful if you are distributing work across machines, but even in the single machine case,…

The practical solution is to generate several CSV files and distribute work at the granularity of files

Re: A love letter to the CSV format

#376

Post by nmz »

SJC_Hacker wrote: post #334
johnea wrote: post #123

I have to agree. It was pretty straightforward (although tedious) to write custom CSV data exports in embedded C, with ZERO dependencies. I know, I know, only old boomers care about removing pip from their code dev process, but, I'm an old boomer, so it was a great feature for me. Straight out of libc I was able to dump data in real-time, that everyone on the latest malware OSes was able to import and analyze. CSV is…

Yeah CSV is easy to export, because its not really a file format, but more an idea. I'm not even sure there is such a thing as "invalid" CSV The following are all valid CSV, and they should all mean the same thing, depending on your point of view: 1) foo, bar, foobar 2) "foo", "bar", "foobar" 3) "foo", bar, foobar 4) foo; bar; "foobar" 5) foo bar "foobar" 5) foo bar Have fun writing that parser!

Using makes it not csv but tsv.

Honestly if there is no comma to separate the values, then its not csv maybe Csv for character separate values or asv for anything separates values but you're right, this makes it hard how everyone is doing whatever. IMV supporting "" makes supporting anything else redundant.

Re: A love letter to the CSV format

#377

Post by pcwalton »

jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

> Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files

There is such a document: RFC 4180. It may not be a good document, but it does exist.

Re: A love letter to the CSV format

#378

Post by d0mine »

freehorse wrote: post #296
phkahler wrote: post #272

Earlier quoted context omitted.

But it handles it better than Json.

Depends on what you mean by "better". I would rather software not handle a piece of data at all, than handle it erroneously and changing the data without me realising and thus causing all sorts of issues after.

In practice, web browsers accept the tag soup that is sometimes called html and strict xml-based formats failed.

Re: A love letter to the CSV format

#379

Post by pretoriusdre »

CSV has caused me a lot of problems due to the weak type system. If I save a Dataframe to CSV and reload it, there is no guarantee that I'll end up with an identical dataframe.

I can depend on parquet. The only real disadvantages with parquet are that they aren't human-readable or mutable, but I can live with that since I can easily load and resave them.

Post reply on HN