Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

591–600 of 711 posts

Re: A love letter to the CSV format

#591

Post by afiori »

pasc1878 wrote: post #498
afiori wrote: post #438

Earlier quoted context omitted.

https://stackoverflow.com/questions/8695118/what-are-the-fil... I am pretty sure you shifted the meaning, the decimal separator is part of the atomic data it does not need a control character. You would use 1F instead of the comma/semicolon/tab and 1E to split lines (record means line just like in SQL). You could then use 1D to store multiple CSV tables in a single file.

Yes but then the text is not human readable or editable in a plain text editor. This would confuse most users of csvs they are not programmers they at most use text editors and Excel.

I am not proposing to do this, but if you were to use ascii separators you would do it this way

Re: A love letter to the CSV format

#592

Post by Evidlo »

There was/is CSVY [0] which attempted to put column style and separator information in a standard header. It is supported by R lang.

I also asked W3C on theirGithub if there was any spec for CSV headers and they said there isn't [1]. Kind of defeats the point of the spec in my opinion.

0: https://github.com/leeper/csvy

1: https://github.com/w3c/csvw/issues/873

Re: A love letter to the CSV format

#594

Post by mjw_byrne »

SoftTalker wrote: post #590
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

If only there were character codes specifically meant to separate fields and records.... we wouldn't have to worry so much about quoted commas or quoted quotes.

There's just no such thing as a delimiter which won't find its way into the data. Quoting and escaping really are the only robust way.

Re: A love letter to the CSV format

#595

Post by craftkiller »

SoftTalker wrote: post #590
jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

If only there were character codes specifically meant to separate fields and records.... we wouldn't have to worry so much about quoted commas or quoted quotes.

That isn't solving anything, just changing the problem. If I want to store a string containing 0x1C - 0x1F in one of the columns then we're back in the exact same situation while also losing the human readable/manually typeable aspect people seem to love about CSV. The real solution is a strict spec with mandatory escaping.

Re: A love letter to the CSV format

#596

Post by SoftTalker »

craftkiller wrote: post #595
SoftTalker wrote: post #590

Earlier quoted context omitted.

If only there were character codes specifically meant to separate fields and records.... we wouldn't have to worry so much about quoted commas or quoted quotes.

That isn't solving anything, just changing the problem. If I want to store a string containing 0x1C - 0x1F in one of the columns then we're back in the exact same situation while also losing the human readable/manually typeable aspect people seem to love about CSV. The real solution is a strict spec with mandatory escaping.

Not for text data. Those values are not text characters like , or " are, and have only one meaning. It would be like arguing that 0x41 isn't always the letter "A".

For binary files, yeah but you don't see CSV used there anyway.

Re: A love letter to the CSV format

#597

Post by Hackbraten »

SoftTalker wrote: post #596
craftkiller wrote: post #595

Earlier quoted context omitted.

That isn't solving anything, just changing the problem. If I want to store a string containing 0x1C - 0x1F in one of the columns then we're back in the exact same situation while also losing the human readable/manually typeable aspect people seem to love about CSV. The real solution is a strict spec with mandatory escaping.

Not for text data. Those values are not text characters like , or " are, and have only one meaning. It would be like arguing that 0x41 isn't always the letter "A". For binary files, yeah but you don't see CSV used there anyway.

There are several ways how a control character might inadvertently end up inside a text corpus. Given enough millions of lines, it’s bound to happen, and you absolutely don’t want it to trip up your whole export because of that one occurrence. So yes, you have to account for it in text data, too.

Re: A love letter to the CSV format

#598

Post by hermitcrab »

jwr wrote: post #522

I so hate CSV. I am on the receiving end: I have to parse CSV generated by various (very expensive, very complicated) eCAD software packages. And it's often garbage. Those expensive software packages trip on things like escaping quotes. There is no way to recover a CSV line that has an unescaped double quote. I can't point to a strict spec and say "you are doing this wrong", because there is no strict spec. Then ther…

I've written a commercial, point and click, data wrangling tool (Easy Data Transform) that can deal with a lot of these issues:

-different delimiters (comma, semi-colon, tab, pipe etc)

-different encodings (UTF8, UTF16 etc)

-different line ending (CR, LF, CR+LF)

-ragged rows

-splitting and merging columns

And much more besides.

However, if you have either:

-line feeds and/or carriage returns in data values, but no quoting

or

-quoting, but quotes in data values aren't properly handled

Then you are totally screwed and you have my sympathies!

Re: A love letter to the CSV format

#599

Post by craftkiller »

SoftTalker wrote: post #596
craftkiller wrote: post #595

Earlier quoted context omitted.

That isn't solving anything, just changing the problem. If I want to store a string containing 0x1C - 0x1F in one of the columns then we're back in the exact same situation while also losing the human readable/manually typeable aspect people seem to love about CSV. The real solution is a strict spec with mandatory escaping.

Not for text data. Those values are not text characters like , or " are, and have only one meaning. It would be like arguing that 0x41 isn't always the letter "A". For binary files, yeah but you don't see CSV used there anyway.

So in addition to losing human readability, we are also throwing away the ability to nest (pseudo-)CSVs? With comma delimiters, I can take an entire CSV document and put it in 1 column, but with 0x1C-0x1F delimiters and banning non-text valid utf-8 in columns I no longer can. This continues to be a step backwards.
Post reply on HN