Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

311–320 of 594 posts

Re: Time to retire the CSV?

#311

Post by stickfigure »

derefr wrote: post #278
stickfigure wrote: post #255

Earlier quoted context omitted.

JSON/JSONL is not self-describing for many of the most common tabular data use cases. There's no date type or currency type. If you are going to displace a standard, it has to be significantly better than the old.

CSV does not have date or currency types. Those are an Excel conventional microformat overlaid on top of CSV. The semantics of CSV — how it decodes, if you don't have an application making additional configured assumptions or using heuristics to detect microformats — is just N columns, one of header names, N of raw text cells. Nothing more, nothing less. No schema defining what a column's type is.

You misunderstood my point. CSV, for all its faults (and I am intimately aware of them), is ubiquitous. You're not going to replace a widespread standard with something that only offers a tiny incremental improvement. Ubiquitous is a feature, and generally trumps all other features.

If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all it brings to the table is the ability to distinguish number vs text. That's a yawn.

Re: Time to retire the CSV?

#312

Post by SavantIdiot »

Every year this question comes up, and every year the arguments are the same.

It's an exercise in hubris because while CSV has one glaring fault, it is far fewer than other competing methods of the simplest possible exchange of textual data.

No one can decide to retire CSV because it ubiquitous. Of the three issues OP compains about: line delimiter, field delimiter, and header, only one is really a problem for anyone that has used them for any period of time.

1. Every file format suffers from Windows/NonWindows CRLF issues.

2. Metadata has been an issue since forever, and there are plenty of painful formats that support it (looking at you XML), complete with ginormous parsers and even more issues.

3. Escaping, as OP points out, was pretty clearly defined in RFC 4180.

So yes, it has a wart: escaping.

Learn what your system expects, and modify accordingly. Because using a CSV will be much faster and simpler than any other format you can try, which is why it has been so pervasive for longer than most of HN has been alive.

Re: Time to retire the CSV?

#313

Post by unnah »

mongol wrote: post #246
thibran wrote: post #206

For the start I would be happy if Excel would, in the German version, by default start to separate CSV values by comma and not semicolon. This "bug" did cost us alone hundreds of support hours. We fixed it by switching to XLS, but still CSV is out there and causing headaches (Mailjet for example can't create correct CSV files - luckily SQLite can read them). Microsoft -> The name of the format is 'comma separated val…

Problem is, many countries use comma as decimal separator. Which makes comma inconvenient as CSV separator.

Just use a decimal point instead. Switching the delimiters depending on locale makes it extremely impractical to use CSV for data interchange with continental European Excel users.

Microsoft Excel is the only program I know that insists on locale-specific interpretations of CSV. To read a standard CSV into Excel in a continental European locale, you need to either switch your locale at the Windows level, or to manually set every delimiter in the Excel text import wizard (requiring multiple mouse clicks for each column).

Re: Time to retire the CSV?

#314

Post by osacial »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

Besides, CSV can be written by a human and initially it was the way how data was entered and read, because it is a text file - database. If there is a need for something more convenient, then there are other file types. Nonissue, IMO.

Re: Time to retire the CSV?

#315

Post by turtlebits »

nightski wrote: post #130
turtlebits wrote: post #109

Earlier quoted context omitted.

It'll work until someone upstream upgrades their CSV writing library and your process breaks. But it's still writing CSV.

So you can constrain what type of CSV you will allow and if this happens it will bail. It's that simple. There is nothing wrong with having additional constraints on top of just saying it must be "CSV" especially in these scenarios. I'm in a similar situation, we've been using CSV for over a decade to move billions of dollars worth of product each year. It just works.

I'm pretty sure most devs are going to use whatever CSV library that comes with their language. When that breaks, it's generally not a simple fix.

Re: Time to retire the CSV?

#316

Post by jolmg »

jhbadger wrote: post #280
asdff wrote: post #237

Earlier quoted context omitted.

Those other things have different names like TSV

In theory, yes, to be pedantic, but for example, LibreOffice saves its exported CSVs by default as tab delimited. You actually have to manually specify you want commas to get those.

> LibreOffice saves its exported CSVs by default as tab delimited

Maybe it's actually presenting what you've selected last?

It's giving me comma as the default separator, and it's the first option in the dropdown. Tab is the 3rd option.

>> Those other things have different names like TSV

That depends on the writer. I've gotten what should be named PTVs (pipe-terminated values) as CSVs. I can understand how it happened. If the underlying software outputs PTVs, you don't want to bother converting that because you're working in a legacy language that's a pain to work with (the type where identifiers can't be longer than 4 chars), and you want the user to be able to double-click on it and have it open in a spreadsheet without prior configuration, you just push the issue to the reader of the file, since by tradition readers are already quite tolerant of format differences...

Of course, there'll always be the case where the reader is simply not tolerant enough, like when the escaping syntax differs. There doesn't seem to be a way to get LO Calc to interpret "foo|bar\|baz|" as cells "foo" and "bar|baz", for example.

Re: Time to retire the CSV?

#317

Post by kthejoker2 »

simonw wrote: post #283

One of my goals with https://datasette.io is to offer a better alternative for publishing data than sharing a link to a CSV file. The trick is that if you compile data into a SQLite file and then deploy the Datasette web application with a bundled copy of that database file, users who need CSV can still have it: every Datasette table and query offers a CSV export. But... you can also get the data out as JSON. Or you…

As an avid Datasette (and Simon Willison!) fan,

> then deploy the Datasette web application

Is a huge hurdle for non-technical folks holding on to their CSV workflows.

I think there is no barrier low enough that CSV cannot limbo beneath it.

Re: Time to retire the CSV?

#318

Post by turtlebits »

asdff wrote: post #284
turtlebits wrote: post #119

Earlier quoted context omitted.

You can write what "looks" like CSV to you, but there are no guarantees it will import correctly. The problem is 10x worse when you get CSV from one source and rely on another process to load it. I fought this problem for several days going from NetSuite to Snowflake via CSV.

Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.

Non standard delimiters. Escaping delimiters in fields - sometimes with a \, sometimes doubled (""), sometimes not at all. Double new lines.

Poor handling from standard CSV libraries. Either unable to read or unable to create for some downstream process.

Re: Time to retire the CSV?

#319

Post by simonw »

kthejoker2 wrote: post #317
simonw wrote: post #283

One of my goals with https://datasette.io is to offer a better alternative for publishing data than sharing a link to a CSV file. The trick is that if you compile data into a SQLite file and then deploy the Datasette web application with a bundled copy of that database file, users who need CSV can still have it: every Datasette table and query offers a CSV export. But... you can also get the data out as JSON. Or you…

As an avid Datasette (and Simon Willison!) fan, > then deploy the Datasette web application Is a huge hurdle for non-technical folks holding on to their CSV workflows. I think there is no barrier low enough that CSV cannot limbo beneath it.

Yeah I've put a ton of effort into that, but it's still nowhere near as easy as I would like.

Re: Time to retire the CSV?

#320

Post by derefr »

stickfigure wrote: post #311
derefr wrote: post #278

Earlier quoted context omitted.

CSV does not have date or currency types. Those are an Excel conventional microformat overlaid on top of CSV. The semantics of CSV — how it decodes, if you don't have an application making additional configured assumptions or using heuristics to detect microformats — is just N columns, one of header names, N of raw text cells. Nothing more, nothing less. No schema defining what a column's type is.

You misunderstood my point. CSV, for all its faults (and I am intimately aware of them), is ubiquitous. You're not going to replace a widespread standard with something that only offers a tiny incremental improvement. Ubiquitous is a feature, and generally trumps all other features. If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all…

JSON solves tons and tons of pain points around CSV... for implementing anything that reads/imports/normalizes CSV data. Just the fact that CSV has multiple standards around the quote/escape/separator/NULL character-sequences, with the choice not get encoded into the document, means that any format that doesn't have that problem is instantly 100x better in terms of not experiencing a disaster of data-garbling-on-import.
Post reply on HN