Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

391–400 of 594 posts

Re: Time to retire the CSV?

#391

Post by qwerty456127 »

hatmatrix wrote: post #377
qwerty456127 wrote: post #112

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

CSV is accessible to programmers and non-programmers alike. HDF5 and SQLite require some programming ability and special software to interact with the data as they are not just text files like CSV.

This is so just because no common software supports them (I don't know why, the libraries are totally free - BSD and Public Domain).

And by the way, many (if not an overwhelming majority) of the non-programmers don't even understand what does "just text files" actually mean, how do text files differ in nature from DOC files and how are CSV files different from XLS files. They can only use CSV because Excel and LibreOffice support it OOTB and consider CSV just a weird XLS cousin needed for import/export purposes.

Re: Time to retire the CSV?

#392

Post by dheera »

mivade wrote: post #387
dheera wrote: post #386

Earlier quoted context omitted.

CSV is still easier to parse because the C++ dudes still refuse to implement some kind of nice operator-overloaded interface like #include std::json myjson("{\"someArray\": [1,2,3,4,{\"a\": \"b\"}]}"); std::cout and the result is we have 50 different rogue JSON libraries instead of an STL solution. Until the STL folks wake up, boost::split can deal with the CSV.

https://github.com/nlohmann/json

ooh this is nice. STL should adopt it

Re: Time to retire the CSV?

#393

Post by ozim »

Problem is not with CSV, problem is with people.

People want to read and interpret arbitrary CSV ... well you cannot read and interpret any format that is arbitrary.

*As a consultant, I’ve written more than one internal system that attempts to reconstruct the metadata of a CSV of unknown provenance using a combination of heuristics and brute force. In short, CSV is a scourge that has followed me throughout my career.*

Ideally this should not happen because you should talk with party that you agree on common format. Someone that would explain what each field means and what should it contain, or at least some documentation for the file, not that it just is a CSV. But of course it always is more complicated than that.

Garbage in - Garbage out, even in other formats you still can have the same problem.

Re: Time to retire the CSV?

#394

Post by hermitcrab »

zabzonk wrote: post #143
javajosh wrote: post #126

Earlier quoted context omitted.

Is there something like CSVlint? Would be useful particular in the output stage, a kind of functional runtime test to make sure your program's output is valid and consumable.

Validating CSV syntax is kind of hard - in the worst case you would have to read the entire CSV file.

Even then it seems that there aren't that many checks you can do, other than:

-warning if different rows have different numbers of columns

-warning if the file ends without closing escaping

Without meta data you can't check data types.

A missing escape character might result in a valid CSV (just not the one intended).

Re: Time to retire the CSV?

#395

Post by [deleted] »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

[deleted]

Re: Time to retire the CSV?

#396

Post by proverbialbunny »

pwinnski wrote: post #215

This reads like a joke. If you think you can do better than CSV, let's see your proposal. Hint: you probably can't, and if you could, you probably couldn't get Excel to export it, so you still probably can't. "The status quo is bad, more recent popular formats aren't good enough either, but I don't actually have a specific proposal that's better than all of the above" is a lot faster to read than that article, and sa…

I wouldn't mind an updated standard to CSV that has the type information in the first line with the column labels. I feel like this is all that CSV is really missing. As a format where you can easily see and interact with the data in plain text if you need to nothing really beats CSV but I believe it could be made better, including strengthening standardization. Eg, a CSV standards version header could be put at the top of the file to minimize import export difficulty.

Re: Time to retire the CSV?

#397

Post by rendall »

The article's strongest criticism of CSV is that it's easy for someone to mangle it when manually editing. This is true. It's also true for every format.

It was weakest when it implied there is no real standard. There is, and it's robust for representing data, even data that includes any combination of commas and double-quotes.

The algorithm for creating well-formed CSV from data is straightforward and almost trivial: if the datum has no comma in it, leave it alone. It's good to go. If it has even one comma, wrap the datum in double quotes; and if it also contains double quotes, then double them.

Not complicated and covers every edge case. CSV is going nowhere. 1000 years from now, computers will still be using CSV.

The answer to his objections could be to extend the format to include metadata. Perhaps a second row that holds type data.

  fruit,price,expiration
  string,$0.00,MM/DD/YYYY
  apple,$0.45,01/24/2022
  durian,$1.34,08/20/2021
etc

Re: Time to retire the CSV?

#398

Post by mavhc »

hermitcrab wrote: post #389
divbzero wrote: post #373

Earlier quoted context omitted.

ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator to avoid conflicting with values, but they have never gained widespread adoption.

While these have clear advantages over comma and CR for parsing (no more escaping!), they aren't at all convenient for manual editing.

If they were popular they'd be supported in your editor though.

Re: Time to retire the CSV?

#399

Post by mavhc »

dtparr wrote: post #370
unnouinceput wrote: post #276

Earlier quoted context omitted.

Correction: the new xlsx is a zip file, the old xls format is true binary.

To be fair, xlsx came out with Office 2007, so it's not exactly 'new' anymore. Perhaps at this point it's reasonable for 'excel file' to mean the one that's been the default for 14 years?

So not the one the uk used to track covid infections, sigh.

Re: Time to retire the CSV?

#400

Post by magicalhippo »

turtlebits wrote: post #315
nightski wrote: post #130

Earlier quoted context omitted.

So you can constrain what type of CSV you will allow and if this happens it will bail. It's that simple. There is nothing wrong with having additional constraints on top of just saying it must be "CSV" especially in these scenarios. I'm in a similar situation, we've been using CSV for over a decade to move billions of dollars worth of product each year. It just works.

I'm pretty sure most devs are going to use whatever CSV library that comes with their language. When that breaks, it's generally not a simple fix.

We've got at least a few dozen customer integrations that parse CSV-ish files, and they all have a custom parser. Many of these have been chugging for over a decade, sending "mission critical" data back and forth.

It's dead simple to whip up, and we can easily tweak it to whatever the customers software spits out, like one field suddenly being UTF-8 encoded in an otherwise Windows-1252 file.

Post reply on HN