Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

91–100 of 594 posts

Re: Time to retire the CSV?

#91

Post by xtracto »

johnrob wrote: post #34

I find JSON “array of arrays” to be a better solution while also remaining human-readable. Especially when formatted as one array per line, analogous to CSV.

Right on. And with the inclusion of a JSON-schema header, it makes data parsing a breeze:

    { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type":"array","items": [ { "type": "number" }, { "type": "string" }, { "enum": ["Street", "Avenue", "Boulevard"] },{ "enum": ["NW", "NE", "SW", "SE"] } ]} }
    [[3,"some", "Street", "NE"],
    [4,"other", "Avenue", "SE"],
    [5,"some", "Boulevard", "SW"]]

Re: Time to retire the CSV?

#92

Post by hcarvalhoalves »

1. I wouldn't bet Avro, Parquet, Arrow, or other formats that require a library to parse are going to _entirely_ replace CSV. Those are broader in scope, but can't say are "replacements" if they don't share the simplicity of plain text data following a few conventions – much like how PDF is not a "replacement" for Markdown, but enables higher-fidelity content.

2. I would feel safer storing longer-term data in CSV than in a binary format w/ a complicated spec. Having to make sense of compressed, columnar padded data sounds worse than parsing CSV.

3. Although it's easy to point at corner cases, I don't remember the last time I couldn't figure out how to parse a file because of inconsistent quoting or exotic char encoding – and I've spent a good amount of the past 13 years exchanging CSVs and TSVs w/ 3rd parties full of horrible legacy. Asking those 3rd parties to send me an Avro/Parquet/whatever file would've made the project fail or take 10x longer.

There's a reason why CSV stuck around so long, and the alternatives make different trade-offs but miss the pros.

Re: Time to retire the CSV?

#93

Post by wanderingstan »

AdmiralAsshat wrote: post #38

CSV was a thing long before I was born, so I'm not privy to how it came about. But at least in day-to-day work, the single biggest drawback of CSV in my experience is the fact that the comma and most of the other common delimiters occur regularly in real data, forcing all of the cumbersome escape sequences. To say nothing of someone misplacing a quote somewhere and throwing off the cell count. So, question to the gre…

Some people did.

But commas can be seen, edited, and typed with ease in any text editor.

As per comment above: your CEO, SWE, or secretary can all use or contribute to a csv file. And using an easily recognizable and typeable separator has proven to be worth the downsides.

Re: Time to retire the CSV?

#94

Post by masklinn »

Sanzig wrote: post #79
masklinn wrote: post #55

Earlier quoted context omitted.

> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx…

Define "garbage." If I know what my data looks like, I can anticipate the edge cases ahead of time. Plenty of CSV exports work this way, they don't need to be general if the schema is already imposed by the system. Have you ever worked in embedded systems? Writing XML files and then zipping them on a platform with 32 kilobytes of RAM would be hell. CSV is easy, I can write the file a line at a time through a lightwei…

> Define "garbage."

Incorrect encoding, incorrect separators (record and field both), incorrect escaping / quoting, etc…

> If I know what my data looks like

If you control the entirety of the pipeline, the format you're using is basically irrelevant. You can pick whatever you want and call it however you want.

> Have you ever worked in embedded systems? Writing XML files and then zipping them on a platform with 32 kilobytes of RAM would be hell. CSV is easy, I can write the file a line at a time through a lightweight microcontroller-friendly filesystem library like FatFS.

You can pretty literally do that with XML and zip files: write the uncompressed data, keep track of the amount of data (for the bits which are not fixed-size), write the file header, done. You just need to keep track of your file sizes and offsets in order to write the central directory. And the reality's if you're replacing a CSV file the only dynamic part will be the one worksheet, everything else will be constant.

Re: Time to retire the CSV?

#95

Post by masklinn »

yrds96 wrote: post #86
masklinn wrote: post #55

Earlier quoted context omitted.

> As I mentioned down-thread, I can generate a CSV with a couple of fprintf statements and a loop. And usually generate garbage for anything but the most trivial case, which really nobody gives a shit about. That's the main reason why CSV absolutely sucks too, you have to waste month diagnosing the broken shit you're given to implement the workarounds necessary to deal with it. > I definitely can't do that with .xlsx…

People doesn't matter which format is, since it simple work in any spreadsheet software. Yeah i can generate any file with a bunch of printf, but csv i dont have to read a specification, i its possible to read with a bunch of read without have to use a xml or xlsl library.

> Yeah i can generate any file with a bunch of printf, but csv i dont have to read a specification, i its possible to read with a bunch of read

The only thing funnier than producing broken CSV is consuming broken CSV.

Re: Time to retire the CSV?

#96

Post by tzs »

JohnHaugeland wrote: post #22
MonadIsPronad wrote: post #10

Surely the successor should be SQLite. Tabular data like CSV, easy to view on different mediums, free and open source, single file databases. It has to be this, right?

yeah, why not replace a dense human readable easily generated format with a third party format that requires installation of compiled binaries and an over the wire protocol

An SQLite dump is human readable and the data in the dump actually is in a CSV format, just surrounded with information on types and relationships. Here's the first few lines for example of a dump of an SQLite DB that I have of some temperature sensor data:

  PRAGMA foreign_keys=OFF;
  BEGIN TRANSACTION;
  CREATE TABLE temperature
  (
    ts integer,
    Tc real,
    Tf real,
    src_id integer
  );
  INSERT INTO temperature VALUES(1615165342,5.5999999999999996447,42.099999999999999644,1);
  INSERT INTO temperature VALUES(1615165350,0.6,32.999999999999998223,3);
  INSERT INTO temperature VALUES(1615165404,5.5,41.899999999999995026,1);
  INSERT INTO temperature VALUES(1615165410,-17.199999999999999289,1.0,2);
  INSERT INTO temperature VALUES(1615165435,5.5,41.899999999999995026,1);
SQLite dumps would actually be a pretty good data exchange format. As mentioned there is a CSV inside there. It's not hard to extract that if you need an unadorned CSV--some grep and sed, or grep and cut, or a few lines of scripting.

Or if you have SQLite installed, it is an easy to have SQLite itself read the dump and export it in CSV, letting it deal with things like quoting that can be a pain with grep and sed/cut. This way also makes it easy to rearrange or omit columns and to do some filtering so that data you don't care about doesn't get into the CSV.

Re: Time to retire the CSV?

#97

Post by masklinn »

zabzonk wrote: post #61

As the author of a CSV munging tool (CSVfix) I think most of the problems with CSV could be fixed if people producing CSV output, and people reading CSV input obeyed the rules of the RFC. Sadly, most people don't, and any textual output or input is routinely described as CSV, when it is nothing of the sort - even to the extent of not being comma-separated!

> Sadly, most people don't

And the important think to remember is that you can not and will not make them.

Re: Time to retire the CSV?

#98

Post by Semaphor »

cogman10 wrote: post #49
podiki wrote: post #16

As long as it is text-based. At least you can actually look at a CSV and see what is going (wrong), as well as use all the text tools we have. Not that there aren't problems as the article points out. But some binary file based on the whims of a proprietary program...no thanks.

JSON and XML both make it easy to see what is going wrong and don't have near the same amount of drawbacks that CSV has.

Both are also not a good fit for columnar data at all.

Re: Time to retire the CSV?

#99

Post by osigurdson »

derefr wrote: post #74
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

Take an Excel file and change the extension to .zip, then extract the contents. You will see that it is a collection of XML files. Therefore it should be reasonable to conclude that this approach can work for Excel sized datasets.

However it is not particularly readable/diff-able if this is part of your use case.

Re: Time to retire the CSV?

#100

Post by dragontamer »

Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

> A truly open format is available and accessible.

Sqlite?

> Applications have a speed increase from using csvs.

Sqlite?

> The new format should not be grossly larger than the one it is replacing

Sqlite it is.

--------

Oh, you mean something that Excel can open? Oh yeah, I guess CSV then. But lets not pretend #1 (openness), #2 (speed), and #3 (size) are the issues.

Post reply on HN