Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

661–670 of 711 posts

Re: A love letter to the CSV format

#661

Post by 999900000999 »

marcosdumay wrote: post #568
999900000999 wrote: post #160

Earlier quoted context omitted.

My condolences. Any open standard runs the risk of this happening. It's not a problem I think we'll ever solve.

In principle, if you make your standard extensible enough, people should stop sneaking data into comments or strings. ... What makes the GP's problem so much more amusing. XML was the last place I'd expect to see it.

That's assuming they know how to use it properly.

Rest has this same issue.

I've seen this when trying to integrate with 3rd party apis.

Status Code 200 Body: Sorry bro, no data.

Even then, this is subject to debate. Should a 404 only be used when the endpoint doesn't exist ? When we have no data to return, etc.

Re: A love letter to the CSV format

#662

Post by int_19h »

recursive wrote: post #650
zeroimpl wrote: post #293

Earlier quoted context omitted.

Example? I know there's some ambiguity over whether literals like false are valid JSON, but I can't think of anything else.

Trailing commas, comments, duplicate key names, for a few examples.

Trailing commas and comments are plainly not standard JSON under any definition. There are standards that include them which extend JSON, sure, but I'm not aware of any JSON library that emits this kind of stuff by default.

Re: A love letter to the CSV format

#663

Post by recursive »

int_19h wrote: post #662
recursive wrote: post #650

Earlier quoted context omitted.

Trailing commas, comments, duplicate key names, for a few examples.

Trailing commas and comments are plainly not standard JSON under any definition. There are standards that include them which extend JSON, sure, but I'm not aware of any JSON library that emits this kind of stuff by default.

I'm not aware of any CSV library that doesn't follow RFC4180 by default, and yet... this whole thread.

Re: A love letter to the CSV format

#664

Post by achierius »

JadeNB wrote: post #588
Moru wrote: post #552

Earlier quoted context omitted.

There was a long period of my life that I thought .csv meant cemicolon separated because all I saw was cemicolon separated files and I had no idea of the pain.

Although it is spelled "semicolon," so that doesn't quite fit.

CMYK -- Cyan, Magenta, Yellow, blacK :)

(of course it originally stood for "key", but you don't see that much anymore)

Re: A love letter to the CSV format

#665

Post by TRiG_Ireland »

codetrotter wrote: post #261
freehorse wrote: post #245

Earlier quoted context omitted.

Importing csvs in excel can be a huge pain due to how excel handles localisation. It can basically alter your data if you are not mindful about that, and I have seen it happening too many times.

For example: Scientists rename genes because Microsoft Excel reads them as dates (2020) https://www.reddit.com/r/programming/comments/i57czq/scienti...

I was so glad of that story. It gave me something to point to to get my boss off my back.

Re: A love letter to the CSV format

#666

Post by susam »

dessimus wrote: post #478
susam wrote: post #263

Earlier quoted context omitted.

Clearly they should have gone with BEL as the delimiter. printf "alice\007london\007uk\nbob\007paris\007france\n" > data.bsv I'm hoping no reasonable person would ever use BEL as punctuation or decimal separator.

If one was going to use a non-printable character as a delimiter, why wouldn't they use the literal record separator "\030"?

Every time you cat a BSV file, your terminal beeps like it's throwing a tantrum. A record separator (RS) based file would be missing this feature! In other words, my previous comment was just a joke! :)

By the way, RS is decimal 30 (not octal '\030'). In octal, RS is '\036'. For example:

  $ printf '\036' | xxd -p
  1e
  $ printf '\x1e' | xxd -p
  1e
See also https://en.cppreference.com/w/cpp/language/ascii for confirmation.

Re: A love letter to the CSV format

#667

Post by thayne »

craftkiller wrote: post #599
SoftTalker wrote: post #596

Earlier quoted context omitted.

Not for text data. Those values are not text characters like , or " are, and have only one meaning. It would be like arguing that 0x41 isn't always the letter "A". For binary files, yeah but you don't see CSV used there anyway.

So in addition to losing human readability, we are also throwing away the ability to nest (pseudo-)CSVs? With comma delimiters, I can take an entire CSV document and put it in 1 column, but with 0x1C-0x1F delimiters and banning non-text valid utf-8 in columns I no longer can. This continues to be a step backwards.

No reason you can't escape those special characters.

Re: A love letter to the CSV format

#668

Post by thayne »

mjw_byrne wrote: post #632
saulpw wrote: post #611

Earlier quoted context omitted.

You can disallow all control characters (ASCII < 32) other than CR/LF/TAB, which is reasonable. I don't know of any data besides binary blobs which uses those. I've never heard of anyone inlining a binary file (like an image) into a "CSV" anyway.

If you disallow control characters so that you can use them as delimiters, then CSV itself becomes a "binary" data format - or to put it another way, you lose the ability to nest CSV. It isn't good enough to say "but people don't/won't/shouldn't do that", because it will just happen regardless. I've seen nested CSV in real-life data. Compare to the zero-terminated strings used by C, one legacy of which is that Postgr…

So have a way to escape those control characters.

Re: A love letter to the CSV format

#669

Post by thayne »

j45 wrote: post #628
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

CSV is a data-exchange format.

But it is terrible at that because there is no widely adhered to standard[1], the sender and receiver often disagree on the details of what exactly a CSV is.

[1]: yes, I know about RFC 4180. But csvs in the wild often don't follow it.

Re: A love letter to the CSV format

#670

Post by thayne »

realitysballs wrote: post #499
jll29 wrote: post #354

The post should at least mention in passing the major problem with CSV: it is a "no spec" family of de-facto formats, not a single thing (it is an example of "historically grown"). And omission of that meams I'm going to have to call this our for its bias (but then it is a love letter, and love makes blind...). Unlike XML or JSON, there isn't a document defining the grammar of well-formed or valid CSV files, and ther…

They do specifically mention this: “No one owns CSV. It has no real specification (yes, I know about the controversial ex-post RFC 4180), just a set of rules everyone kinda agrees to respect implicitly. It is, and will forever remain, an open and free collective idea.”

They even seem to think it is a good thing. But I don't see how not having a bunch of implementations that can't agree on the specifics of a file/interchange format is a good thing. And being free and open is completely orthogonal. There are many proprietary formats that don't have a spec, and many open formats that do have a spec (like, say, json).
Post reply on HN