Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

511–520 of 711 posts

Re: A love letter to the CSV format

#511

Post by mbnielsen »

otabdeveloper4 wrote: post #413
jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

People who say that CSV is "simpler" are talking about whatever format Excel exports. Also these people have only ever had to deal with the American Excel localization. So yeah, with the caveat of "only ever use Excel and only ever the American edition" CSV is pretty nice.

As someone living in a country where , is used as the decimal separator, I cannot begin to describe the number of times CSV data has caused me grief. This becomes especially common in an office environment where Excel is the de facto only data handling tool that most people can and will use. Here the behavior of loading data becomes specific to the individual machine and changes over time (e.g. when IT suddenly forces a reset of MS Office application languages to the local one).

That said, I don't really know of any alternative that won't be handled even worse by my colleagues...

Re: A love letter to the CSV format

#512

Post by achr2 »

metalliqaz wrote: post #329
achr2 wrote: post #326

Using ascii 'US' Unit Separator and 'RS' Record Separator characters would be a far better implementation of a CSV file.

and of course you can do that if you wish, as many CSV libraries allow arbitrary separators and escapes (though they usually default to the "excel compatible" format) but at least in my case, I would not like to use those characters because they are cumbersome to work with in a text editor. I like very much to be able to type out CSV columns and rows quickly, when I need to.

It’s all a pros and cons.. the benefit of those characters are they are not used anywhere else, hence you never have to worry about escaping/quoting strings. But obviously most of my csv usage is automated in/out.

Re: A love letter to the CSV format

#513

Post by _flux »

yrro wrote: post #456
sramsay64 wrote: post #360

Earlier quoted context omitted.

In fairness there are also several ambiguities with JSON. How do you handle multiple copies of the same key? Does the order of keys have semantic meaning? jq supports several pseudo-JSON formats that are quite useful like record separator separated JSON, newline separated JSON. These are obviously out of spec, but useful enough that I've used them and sometimes piped them into a .json file for storage. Also, encoding…

Internet JSON (RRC 7493) forbids objects to have members with duplicate names.

As it says:

I-JSON (short for "Internet JSON") is a restricted profile of JSON designed to maximize interoperability and increase confidence that software can process it successfully with predictable results.

So it's not JSON, but a restricted version of it.

I wonder if use of these restrictions is popular. I had never heard of I-JSON.

Re: A love letter to the CSV format

#514

Post by mjw_byrne »

lelanthran wrote: post #398
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

> So these days for serialisation of simple tabular data I prefer plain escaping, e.g. comma, newline and \ are all \-escaped. It's as easy to serialise and deserialise as CSV but without the above drawbacks. For my own parser, I made everything `\` escaped: outside of a quote or double-quote delimited string, any character prefixed with a `\` is read verbatim. There are no special exceptions resulting in `\,` produc…

I considered this but then went the other way - a \ before anything other than a \, newline or comma is treated as an error. This leaves room for adding features, e.g. \N to signify a SQL NULL.

Regarding quoting and escaping, there are two options that make sense to me - either use quoting, in which case quotes are self-escaped and that's that; or use escaping, in which case quotes aren't necessary at all.

Re: A love letter to the CSV format

#515

Post by kevincox »

thiht wrote: post #475
sbergot wrote: post #466

Earlier quoted context omitted.

interestingly other people are answering the opposite in this thread.

They're wrong. From ECMA-404[1] in section 6: > The JSON syntax does not impose any restrictions on the strings used as names, does not require that name strings be unique, and does not assign any significance to the ordering of name/value pairs. That IS unambiguous. And for more justification: > Meaningful data interchange requires agreement between a producer and consumer on the semantics attached to a particular u…

> and does not assign any significance to the ordering of name/value pairs.

I think this is outdated? I believe that the order is preserved when parsing into a JavaScript Object. (Yes, Objects have a well-defined key order. Please don't actually rely on this...)

Re: A love letter to the CSV format

#516

Post by kevincox »

pasc1878 wrote: post #503
oever wrote: post #481

Earlier quoted context omitted.

It's evitable by stating the number of bytes in a field and then the field. No escaping needed and faster parsing.

But not human editable/readable

I understand this argument in general. But basically everyone has some sort of spreatsheet application that can read CSV installed.

In some alternate worked where this "binary" format caught on it would be a very minor issue that it isn't human readable because everyone has a tool that is better at reading it than humans are. (See the above mentioned non-local property of quotes where you may think you are reading rows but are actually inside a single cell.)

Makes me also wonder if something like CBOR caught on early enough we would just be used to using something like `jq` to read it.

Re: A love letter to the CSV format

#517

Post by kokada »

nukem222 wrote: post #294
account-5 wrote: post #276

I like CSV for the same reasons I like INI files. It's simple, text based, and there's no typing encoded in the format, it's just strings. You don't need a library. They're not without their drawbacks, like no official standards etc, but they do their job well. I will be bookmarking this like I have the ini critique of toml: https://github.com/madmurphy/libconfini/wiki/An-INI-critique... I think the first line of the…

> It's simple My experience has indicated the exact opposite. CSVs are the only "structured" format nobody can claim to parse 100% (ok probably not true thinking about html etc, just take this as hyperbole.) Just use a well-specified format and save your brain-cells. Occasionally, we must work with people who can only export to csv. This does not imply csv is a reasonable way to represent data compared to other optio…

> CSVs are the only "structured" format nobody can claim to parse 100%

You don't need to though since in most cases you just need to support whatever CSV format the tool you're handling, unless of course you're trying to write the next Excel/Google Sheets competitor.

Re: A love letter to the CSV format

#519

Post by jonathanberi »

kevincox wrote: post #516
pasc1878 wrote: post #503

Earlier quoted context omitted.

But not human editable/readable

I understand this argument in general. But basically everyone has some sort of spreatsheet application that can read CSV installed. In some alternate worked where this "binary" format caught on it would be a very minor issue that it isn't human readable because everyone has a tool that is better at reading it than humans are. (See the above mentioned non-local property of quotes where you may think you are reading ro…

https://github.com/wader/fq is "jq for binary formats."

Re: A love letter to the CSV format

#520

Post by tikhonj »

lelanthran wrote: post #414
IanCal wrote: post #403

Earlier quoted context omitted.

Have you had to work with csv files from the wild much? I'm not being snarky but what you're talking about is night and day to what I've experienced over the years. There aren't vast numbers of different JSON formats. There's practically one and realistically maybe two. Headers are in each line, utf8 has never been an issue for me and quoting and escaping are well defined and obeyed. This is because for datasets, alm…

Sure, I get your arguments and we're probably mostly in agreement, but in practice I see very few problems arising with using CSV. I mean, right now , the data interchange format between multiple working systems is CSV; think payment systems, inter-bank data interchange, ERP systems, CRM systems, billing systems ... the list goes on. I just recently had a coffee with a buddy who's a salesman for some enterprise syste…

> And yet, they work.

Through a lot of often-painful manual intervention. I've seen it first-hand.

If an organization really needs something to work, it's going to work somehow—or the organization wouldn't be around any more—but that is a low bar.

In a past role, I switched some internal systems from using CSV/TSV to using Parquet and the difference was amazing both in performance and stability. But hey, the CSV version worked too! It just wasted a ton of people's time and attention. The Parquet version was far better operationally, even given the fact that you had to use parquet-tools instead of just opening files in a text editor.

Post reply on HN