Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

161–170 of 594 posts

Re: Time to retire the CSV?

#161

Post by nojito »

turtlebits wrote: post #109
radmuzom wrote: post #28

Every few years an article like this pops up. I find it tiring - because they are primarily from a software engineer's viewpoint who is probably trying to write a parser and needs to handle the edge cases. As a data scientist, I receive and process around 75GB of CSV every day - of course I don't process it manually. Our processes have been running a few years now and millions of dollars of revenue rides on it. I don…

It'll work until someone upstream upgrades their CSV writing library and your process breaks. But it's still writing CSV.

That’s a good thing.

We validate on ingestion and if there are changes upstream we can immediately triage without polluting our data warehouse.

Re: Time to retire the CSV?

#162

Post by Demiurge »

Retiring CSV is as futile as retiring Unix pipes. It's a text format for text manipulations, and there is nothing simpler and more ubiquitous than a new-line and comma delimited table. Simplicity and flexibility has its drawbacks, but the advantages clearly outweigh those drawbacks. So, it will never retire, as long as we use text and blob filesystems that don't have interoperable structured metadata, across the board.

Re: Time to retire the CSV?

#164

Post by masklinn »

da_chicken wrote: post #142
cls59 wrote: post #104

Earlier quoted context omitted.

To point 1, I'd argue that a SQLite database is a great next step beyond CSV despite being a binary format.

No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does. The advantage of CSV is that it's as accurate as your plain text representation of your data can be. Since binary data can be represented by character data, that's 100% accurate. As soon as you introduce a storage format that has made assumptions about the type of data being stored, you've lost flexibility. SQ…

> No, SQLite's dynamic data types would silently coerce data just like opening a CSV directly with Excel does.

SQLite's "dynamic data types" coerce data on input, not output. Once the data is in sqlite the way you wanted it, excel has no interpretation to perform, except insofar as really really wanting dates.

> The advantage of CSV is that it's as accurate as your plain text representation of your data can be.

Yeah nah.

> SQLite is not intended for data serialization. It's intended for data storage to be read back by essentially the same application in the same environment.

That's completely, absolutely, utterly, objectively, nonsensical.

Data analysis, exchange, container, and archiving, are literally all use cases listed as "appropriate uses for sqlite" in "Situations Where SQLite Works Well" on the official sqlite website: https://www.sqlite.org/whentouse.html

Re: Time to retire the CSV?

#165

Post by CobaltFire »

qwerty456127 wrote: post #112

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

Honestly I constantly see dates argued about and people state various formats that are still confusing. 4-2-2 of any variety can be confused. Why not 2-3-4 or 4-3-2 (DD-MMM-YYYY or YYYY-MMM-DD)? I’ve never understood why that isn’t more widely used.

Re: Time to retire the CSV?

#166

Post by throwawayswede »

Meh. Author probably messed up some important deadline or emailed (or got emailed) some crappy CSV file and then threw a temper tantrum and decided to write this.

Every single one of their points are extremely subjective and very wrong. If you have an excel sheet full of equations and colorful cells and then to decide to export it as CSV and open it in Notepad, you really can't complain that that CSV is bad. I mean it's obvious that to each format a set of strengths and a set of weaknesses, and also obviously what could be a strength to someone is a weakness to someone else. The fact that CSV is so simple to parse (almost every modern language can very easily read/write a CSV) makes it a fantastic data transfer format for every single usecase I had (this doesn't mean that other formats are less important of course). Sure you'll lose your Google Sheet or Excel metadata, but this is NOT what CSV is for.

What I find fascinating though is that the author decided it's a good idea to make a blanket statement like "Time to retire?" just because they themselves have an issue with SOME use case. I mean the idea that having your personal needs not met to justify arguing that we should ALL stop using CSV is so very bizarre to me, like it's way beyond selfish.

Re: Time to retire the CSV?

#167

Post by jslaby »

zepolen wrote: post #19

CSVs are plain text, and that's huge, plus they have a few other advantages: 1. You can read them without any software 2. Streamable row by row 3. Compress well To be honest most of the points in this article could be addressed by standardizing a method of defining a schema for the CSVs. It could even be backward compatible by appending the definition as metadata on the header column or as a separate file. One thing…

I think an ideal solution would be csv with json metadata elements as the header. If a program supports the header, then you get the metadata. If not, no big deal, you just get an ugly header.

Re: Time to retire the CSV?

#168

Post by nojito »

rutthenut wrote: post #69
NBJack wrote: post #52

Earlier quoted context omitted.

Not sure what your basing that claim on. It is a natively supported format, even amidst its inconsistencies, and has been for decades. Are you referring to a particular format variant?

Excel definitely opens CSV files, and Windows file associations tend to set that up by default. However, Excel then goes ahead with changing the content to often wrongly-assumed formats. My favourite hate is how it will convert mobile numbers from a string of digits to an unusable exponential-format number. Hey, it doesn't even set column widths to make the content visible!

You should use power query to load csvs into excel that way you can control the parsing of columns.

Re: Time to retire the CSV?

#169

Post by da_chicken »

cbsmith wrote: post #123
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

1. open(2) doesn't care about the file format 2. Yes, mixing formatting with data slows down data processing, don't do it. 3. Excel is not the replacement for CSV, and CSV is not a compact format. I mean, maybe if you are used to XML it is, but otherwise, just no. Yes, there is truly a problem.

> CSV is not a compact format

That's only true if you're trying to send all your data in a single, monolithic CSV.

If you're sending multiple CSVs, you're capable of representing data as well as a relational data store. Which is to say, you're representing your data using a system of data normalization specifically designed to minimalize data duplication. A single CSV represents a single table, and in most cases with intelligent delimiter selection you can represent an entire data set with no more than one character spent between fields or records.

Yes, you do have situations where you're storing losing data density due to using plain text strings, but that's not a limitation particularly unique to CSV for data serialization formats. Additionally, it is a problem that can largely be mitigated by simple text compression. Furthermore, once you switch to a non-text representation, you're limiting yourself to whatever that data representation is. It's easy to represent an arbitrary precision decimal number in plain text. It's hard to find a binary representation that universally represents the same data regardless of the system on the other end. Again, that's not a problem unique to CSVs.

If you're working with an API, object by object, then JSON is certainly going to be better, yes, because you can use the application's object representation. If you're working with bulk data of many disparate, unrelated, complex objects, however, or where you're transferring and entire system, you're not going to do much better than CSV.

Re: Time to retire the CSV?

#170

Post by throwawayswede »

nolok wrote: post #24
podiki wrote: post #16

As long as it is text-based. At least you can actually look at a CSV and see what is going (wrong), as well as use all the text tools we have. Not that there aren't problems as the article points out. But some binary file based on the whims of a proprietary program...no thanks.

Xlsx (office format for almost a decade now) are zip and XML all the way. Not fun to look at, but totally readable by a human.

Also it's way more difficult to parse than CSV. I'm not saying that CSV is better (or worse), just that to each it's job! Wtf is this about retiring a format because it doesn't fit someone's expectation!
Post reply on HN