Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

331–340 of 711 posts

Re: A love letter to the CSV format

#331

Post by noosphr »

niccl wrote: post #280
dietr1ch wrote: post #251

Earlier quoted context omitted.

I don't understand why CSV became a thing when TSV, or a format using the nowadays weird ASCII control characters like start/end of text, start of heading, horizontal/vertical tab, file/group/record/unit separator. It seems many possible designs would've avoided the quoting chaos and made parsing sort of trivial.

Any time you have a character with a special meaning you have to handle that character turning up in the data you're encoding. It's inevitable. No matter what obscure character you choose, you'll have to deal with it

The difference is that the coma and newline characters are much more common in text than 0x1F and 0x1E, which if you restrict your data to alphanumeric characters (which you really should) will never appear anywhere else.

Re: A love letter to the CSV format

#332

Post by bobbylarrybobby »

k_bx wrote: post #156

I've recently been developing a raspberry pi based solution which works with telemetry logs. First implementation used an SQLite database (with WAL log) – only to find it corrupted after just couple of days of extensive power on/off cycles. I've since started looking at parquet files – which turned out to not be friendly to append-only operations. I've ended up implementing writing events into ipc files which then pe…

If sqlite ends up corrupted, why wouldn't a CSV? What happens if the system dies partway through a write?

Re: A love letter to the CSV format

#333

Post by dietr1ch »

niccl wrote: post #280
dietr1ch wrote: post #251

Earlier quoted context omitted.

I don't understand why CSV became a thing when TSV, or a format using the nowadays weird ASCII control characters like start/end of text, start of heading, horizontal/vertical tab, file/group/record/unit separator. It seems many possible designs would've avoided the quoting chaos and made parsing sort of trivial.

Any time you have a character with a special meaning you have to handle that character turning up in the data you're encoding. It's inevitable. No matter what obscure character you choose, you'll have to deal with it

The characters would likely be unique, maybe even by the spec.

Even if you wanted them, we use backslashes to escape strings in most common programming languages just fine, the problem CSV is that commas aren't easy to recognize because they might be within a single or double quote string, or might just be a separator.

Can strings in CSV have newlines? I bet parsers disagree since there's no spec really.

Re: A love letter to the CSV format

#334

Post by SJC_Hacker »

johnea wrote: post #123

I have to agree. It was pretty straightforward (although tedious) to write custom CSV data exports in embedded C, with ZERO dependencies. I know, I know, only old boomers care about removing pip from their code dev process, but, I'm an old boomer, so it was a great feature for me. Straight out of libc I was able to dump data in real-time, that everyone on the latest malware OSes was able to import and analyze. CSV is…

Yeah CSV is easy to export, because its not really a file format, but more an idea. I'm not even sure there is such a thing as "invalid" CSV

The following are all valid CSV, and they should all mean the same thing, depending on your point of view:

1) foo, bar, foobar

2) "foo", "bar", "foobar"

3) "foo", bar, foobar

4) foo; bar; "foobar"

5) foobar"foobar"

5) foo bar

Have fun writing that parser!

Re: A love letter to the CSV format

#335

Post by bobmcnamara »

baumschubser wrote: post #144

Just last week I was bitten by a customer’s CSV that failed due to Windows‘ invisible BOM character that sometimes occurs at the beginning of unicode text files. The first column‘s title is not „First Title“ then but „&zwnbsp;First Title“. Imagine how long it takes before you catch that invisible character. Aside from that: Yes, if CSV would be a intentional, defined format, most of us would do something different he…

I wish the UTF8BOM was standardized. Encoding guessing usually works until it doesn't.

Re: A love letter to the CSV format

#336

Post by didgetmaster »

fsckboy wrote: post #315
didgetmaster wrote: post #313

I have written a new database system that will convert CSV, JSON, and XML files into relational tables. On of the biggest challenges to CSV files is the lack of data types on the header line that could help determine the schema for the table. For example a file containing customer data might have a column for a Zip Code. Do you make the column type a number or a string? The first thousand rows might have just 5 digit…

csv does not stop you from making the first line be column headers, with implied data types, you just have to comma separate them!

I realize that. But when reading a header you have to imply the data types which might be wrong. I always thought it would have been great if the first line read something like: name:STRING,address:STRING,zip code:INTEGER,ID:BIG_INT,...

Re: A love letter to the CSV format

#337

Post by kibwen »

nukem222 wrote: post #311
PaulHoule wrote: post #310

Earlier quoted context omitted.

The HTML 5 spec says exactly how you're supposed to deal with broken HTML files.

Yes, that is a single spec with correspondingly-small importance. Generally parsing html remains extremely difficult.

It's of quite large importance, and despite being difficult, it is well-specified, which is the point here. Importantly, there is also no competing HTML spec, either de facto or otherwise. CSV doesn't have anything of comparable authority.

Re: A love letter to the CSV format

#338

Post by wvenable »

niccl wrote: post #280
dietr1ch wrote: post #251

Earlier quoted context omitted.

I don't understand why CSV became a thing when TSV, or a format using the nowadays weird ASCII control characters like start/end of text, start of heading, horizontal/vertical tab, file/group/record/unit separator. It seems many possible designs would've avoided the quoting chaos and made parsing sort of trivial.

Any time you have a character with a special meaning you have to handle that character turning up in the data you're encoding. It's inevitable. No matter what obscure character you choose, you'll have to deal with it

Except we have all these low ASCII characters specifically for this purpose that don't turn up in the data at all. But there is, of course, also an escape character specifically for escaping them if necessary.

Re: A love letter to the CSV format

#340

Post by nh2 »

TrackerFF wrote: post #37

Excel hates CSV only if you don't use the "From text / csv" function (under the data tab). For whatever reason, it flawlessly manages to import most CSV data using that functionality. It is the only way I can reliably import data to excel with datestamps / formats. Just drag/dropping a CSV file onto a spreadsheet, or "open with excel" sucks.

Even "From Text / CSV" sucks:

It inserts an extra row at the top for its pivot table, with entries "Column1, Column2, ...".

So if you export to CSV again, you now have 2 header rows.

So Excel can't roundtrip CSVs, and the more often you roundtrip, the more header rows you get.

You need to remember to manually delete the added header row each time, otherwise software you export back to can't read it.

Post reply on HN