Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

271–280 of 594 posts

Re: Time to retire the CSV?

#271

Post by turtles_ »

jerf wrote: post #194
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

"Self-describing formats like JSON Lines are big... but when you compress them, they go back to being small." For CSV file replacements, I'd expect something like "one JSON array per line, all values must be JSON scalars". In that case, it's not much larger than a CSV, especially one using quotes already for string values. But this demonstrates the problem with JSON for CSV, I suppose. Is each line an object? Is it w…

Indeed, this has already been done: http://ndjson.org/

To be fair it's not an objectionable format. Using line breaks to separate objects makes it streamable, and you don't need to enclose the whole thing in an array to make it a valid JSON document.

Re: Time to retire the CSV?

#272

Post by Existenceblinks »

xtracto wrote: post #91
johnrob wrote: post #34

I find JSON “array of arrays” to be a better solution while also remaining human-readable. Especially when formatted as one array per line, analogous to CSV.

Right on. And with the inclusion of a JSON-schema header, it makes data parsing a breeze: { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type":"array","items": [ { "type": "number" }, { "type": "string" }, { "enum": ["Street", "Avenue", "Boulevard"] },{ "enum": ["NW", "NE", "SW", "SE"] } ]} } [[3,"some", "Street", "NE"], [4,"other", "Avenue", "SE"], [5,"some", "Boulevard", "SW"]]

With draft 2020-12, you can even have module-like based on [Compound Document spec](https://json-schema.org/draft/2020-12/json-schema-core.html#...)

Now you have tables and relation. Only few people understand the usefulness of Compound Document, currently.

Re: Time to retire the CSV?

#273

Post by unnouinceput »

karteum wrote: post #264
Sunspark wrote: post #23

I don't agree with giving up csvs until the following conditions are met: 1) A truly open format is available and accessible. Csvs are textfiles. There is no system around that cannot open a textfile. If the format is binary or requires patents or whatever, then it's a non-starter. 2) Applications have a speed increase from using csvs. To wit, I loved csvs because often they finish preparing much faster than a "forma…

I personally believe that at least SQLite matches all those criteria : "1) A truly open format is available" : sqlite is open-source, MIT-licensed, and well specified (even though I am usually not so happy with its weak typing approach, yet in this case this precisely enables a 100% correspondance between CSV and sqlite since CSV has also no typing at all...) "2) Applications have a speed increase from using csvs" :…

sqlite files are binary files, so a non-starter. If I open them in Notepad I can't read the values like I can with CSV.

Re: Time to retire the CSV?

#275

Post by shezi »

This is a pet peeve of mine: csv was a bad format already when it was created, because it intermixes data with meta data, ie it uses characters that can appear as either data or control characters, leading to escaping issues. This is in addition to being under specified as to which characters are used as control characters.

It was outdated from the start because ASCII already has specific characters for file, group, record, and item separation. Using this would give broad compatibility and a wider feature set (eg. more than one table in a file) while retaining all the benefits of csv.

Re: Time to retire the CSV?

#276

Post by unnouinceput »

osigurdson wrote: post #99
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

Take an Excel file and change the extension to .zip, then extract the contents. You will see that it is a collection of XML files. Therefore it should be reasonable to conclude that this approach can work for Excel sized datasets. However it is not particularly readable/diff-able if this is part of your use case.

Correction: the new xlsx is a zip file, the old xls format is true binary.

Re: Time to retire the CSV?

#277

Post by cogman10 »

mongol wrote: post #267
cogman10 wrote: post #251

Earlier quoted context omitted.

It's just JSON. It's not an extension to the spec, it's a subset of the spec. If anything, you could say it's JSON--.

Well, to work as a CSV-like JSON, i.e greppable similar to a CSV it needs formatting with linebreaks as above. It is still completely valid JSON. ++ is the formatting convention.

Gotcha.

Yeah, it's never really come up as it's generally trivial to convert into this format from either a text editor or sed (Look for the `],[` and put a new line after the comma).

Re: Time to retire the CSV?

#278

Post by derefr »

stickfigure wrote: post #255
derefr wrote: post #74

Earlier quoted context omitted.

You're comparing CSVs to other spreadsheet document formats. But a CSV is not a spreadsheet. A CSV is raw data. (It's data that is restricted to a shape that enables it to be easily imported into a spreadsheet—but data nevertheless.) As such, it should be compared to other data formats—e.g. YAML, JSON Lines, etc. These other data formats all win on your #2 against CSV, as CSV is actually horrible at parse-time vs. ot…

JSON/JSONL is not self-describing for many of the most common tabular data use cases. There's no date type or currency type. If you are going to displace a standard, it has to be significantly better than the old.

CSV does not have date or currency types. Those are an Excel conventional microformat overlaid on top of CSV.

The semantics of CSV — how it decodes, if you don't have an application making additional configured assumptions or using heuristics to detect microformats — is just N columns, one of header names, N of raw text cells. Nothing more, nothing less. No schema defining what a column's type is.

Re: Time to retire the CSV?

#279

Post by [deleted] »

unnouinceput wrote: post #276
osigurdson wrote: post #99

Earlier quoted context omitted.

Take an Excel file and change the extension to .zip, then extract the contents. You will see that it is a collection of XML files. Therefore it should be reasonable to conclude that this approach can work for Excel sized datasets. However it is not particularly readable/diff-able if this is part of your use case.

Correction: the new xlsx is a zip file, the old xls format is true binary.

[deleted]

Re: Time to retire the CSV?

#280

Post by jhbadger »

asdff wrote: post #237
gsich wrote: post #158

Earlier quoted context omitted.

If you interpret "CSV" as purely comma seperated values then maybe. But in my bubble "CSV" means textfiles that are separated by some separator. Be it tabs, spaces, commas, or any other ASCII character. Some are more usable then others, if you have commas in your data then use tabs. If you have tabs use Form Feed or Record Separator or vertical tabs ... and so on. Of course this is not always applicable, since you so…

Those other things have different names like TSV

In theory, yes, to be pedantic, but for example, LibreOffice saves its exported CSVs by default as tab delimited. You actually have to manually specify you want commas to get those.
Post reply on HN