Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

511–520 of 594 posts

Re: Time to retire the CSV?

#511

Post by barbarbar »

masklinn wrote: post #242
cbsmith wrote: post #212

Earlier quoted context omitted.

> The entire article is about replacing CSVs for exchanging data exported from Excel... No, it's not. It's about replacing CSVs for exchanging data. It mentions that CSVs often are the product of someone exporting data from a spreadsheet or doing a table dump, and how just doing that tends to create a ton of problems, but Excel is an example, not the subject matter of the article. > The business world will laugh at y…

> The business world pays me a lot of money to teach them not to use CSVs. Could you teach them better and faster? I don't think they're getting it. You have my blessing to use violence.

This is the right suggestion.

Re: Time to retire the CSV?

#513

Post by asdff »

jacobsenscott wrote: post #450
asdff wrote: post #284

Earlier quoted context omitted.

Can you give an example? The rules for CSV files are so simple I'm struggling to imagine a case where something looks correct but in fact isn't correct.

name,position "Smith, John＂‚Manager

https://www.gnu.org/software/gawk/manual/gawk.html#Splitting...

Re: Time to retire the CSV?

#514

Post by justinator »

jrumbut wrote: post #480
7thaccount wrote: post #366

Earlier quoted context omitted.

CSV is far from perfect, but it's nice that I can easily work with them without needing any libraries. All I need is file I/O and the ability to split strings. It doesn't get much simpler. I'll admit though that "import JSON" and then being able to essentially convert the entire file into a dictionary is nice if the data has more structure to it.

The real advantage of CSV, in my mind, is that if the CSV is valid and normal then it's going to be a rectangular dataset (ignoring semantics within the dataset). If I import JSON data I have no idea what shape the result will be in, and it requires a separate standard to let me know about columns and rows and validation can get complicated.

CSV is that way too. There's nothing that says each row has to have the same number of what maps out to columns, or anything that tells me what those columns really represent (there's no schema). You could use the first line of the CSV doc to say what each tuple is named, but that's not a standard or anything. And without a schema, it certainly could be easy to lose the metadata of the info the CSV file is trying to represent. Is this column just a bunch of numbers or a date format? (for example). CSV is OK for importing and exporting data across systems that know what the format is without the help of a schema, but anything else and you run into a pile of edge cases. Even using a CSV file to import into a spreadsheet works usually but context is often lost.

Frankly, I love the format.

Re: Time to retire the CSV?

#515

Post by hnick »

rendall wrote: post #506
hnick wrote: post #448

Earlier quoted context omitted.

You didn't mention handling new lines as data. Excel for example will include these as-is and double quote the cell. It is very easy to overlook edge cases in CSV.

> It is very easy to overlook edge cases in CSV. That was not a criticism from the original article, and isn't even true. If you have newlines in your original data, and you "overlooked" this "edge case", then neither JSON nor YAML nor any other format will save you. The same fix applies to them all. This is really a very poor criticism.

The only reason I replied was that you said:

> The algorithm for creating well-formed CSV from data is straightforward and almost trivial: if the datum has no comma in it, leave it alone. It's good to go.

> Not complicated and covers every edge case

Maybe more context was implied, but I didn't want anyone to think it's that simple. I have received and had to process CSV data with unexpected new lines. It gets nearly everyone the first time when they try to process a CSV file line by line.

Re: Time to retire the CSV?

#516

Post by a9h74j »

da_chicken wrote: post #226
jerf wrote: post #194

Earlier quoted context omitted.

"Self-describing formats like JSON Lines are big... but when you compress them, they go back to being small." For CSV file replacements, I'd expect something like "one JSON array per line, all values must be JSON scalars". In that case, it's not much larger than a CSV, especially one using quotes already for string values. But this demonstrates the problem with JSON for CSV, I suppose. Is each line an object? Is it w…

> But this demonstrates the problem with JSON for CSV, I suppose. Is each line an object? How is that not a problem with every data serialization format? It does me no real good if I have an XML schema and a corresponding file. If I don't know what those elements and attributes represent I'm not really any better off. It's not like JSON or XML can meaningfully be marshaled back into objects for use generically withou…

I have wondered about a file format where a parser could be specified for [at the start of] each line. You could even have different json parsers with different well-characterized limits and relative speeds. Formats could change over time on a line-by-line basis, without being locked into a full-file IDL or similar.

Re: Time to retire the CSV?

#517

Post by slaymaker1907 »

I think a key element for any potential replacement of CSV is parser simplicity. Writing a basic CSV parser by hand is not very difficult (i.e. ignoring stuff like escaped characters and quotes) compared to writing a SQLite parser. Regardless if it is binary or not, it should be possible to print out a usable grammar on a single sheet of grammar and said grammar should be LL(1) so at most you would need to write a recursive descent parser.

Suppose someone wants to do some data processing client side in the browser with an input file. How would you do this for SQLite, even a SQLite file with a known schema such as would be the case most of the time for CSV file uploads? There is sql.js which compiles SQLite to JS or WASM, but even the WASM version weighs in around 400kb.

Re: Time to retire the CSV?

#518

Post by dolni »

anigbrowl wrote: post #410
karteum wrote: post #264

Earlier quoted context omitted.

I personally believe that at least SQLite matches all those criteria : "1) A truly open format is available" : sqlite is open-source, MIT-licensed, and well specified (even though I am usually not so happy with its weak typing approach, yet in this case this precisely enables a 100% correspondance between CSV and sqlite since CSV has also no typing at all...) "2) Applications have a speed increase from using csvs" :…

It's worth mentioning that SQLite is committed by its publishers to be backward-compatible out to 2050 and is the recommended by the Library of Congress as a safe long-term storage format (as are XML, JSON, and CSV). https://www.sqlite.org/locrsf.html For single tables a database is probably overkill, but it's nice to have around when you need something reasonably powerful without being overly complex or hard to get…

I suppose if I was to recommend some "safe long-term storage formats" I, too, would choose things that are readable in plain ASCII and/or have open source roots.

Shoutout to the Library of Congress for doing the real heavy lifting here?

Re: Time to retire the CSV?

#519

Post by zerocrates »

derefr wrote: post #427
jerf wrote: post #328

Earlier quoted context omitted.

That is not quite a CSV replacement. I use it for things with objects and stuff all the time. To be a CSV replacement you really need to add that each line must be a JSON array, and that it can only have scalars in it (no sub-arrays or objects). That would be a decent enough replacement for CSV itself. Not perfect, but the CSV "standard" is already a nightmare at the edge anyhow and honestly a lot of it can't be fixe…

> that it can only have scalars in it (no sub-arrays or objects) I see CSV files that contain JSON arrays/objects in their fields all the time. Mainly from exporting Postgres tables that contain json/jsonb-typed columns. Are you saying that these aren't valid CSVs?

They're saying that a CSV equivalent should be strictly 2-dimensional, with "flat" values.

Such a format could contain arbitrary JSON in a "cell", but simply as text, in the same way as putting the same data in a CSV.

Re: Time to retire the CSV?

#520

Post by lmm »

stickfigure wrote: post #311
derefr wrote: post #278

Earlier quoted context omitted.

CSV does not have date or currency types. Those are an Excel conventional microformat overlaid on top of CSV. The semantics of CSV — how it decodes, if you don't have an application making additional configured assumptions or using heuristics to detect microformats — is just N columns, one of header names, N of raw text cells. Nothing more, nothing less. No schema defining what a column's type is.

You misunderstood my point. CSV, for all its faults (and I am intimately aware of them), is ubiquitous. You're not going to replace a widespread standard with something that only offers a tiny incremental improvement. Ubiquitous is a feature, and generally trumps all other features. If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all…

> If you want something better than CSV to take off, at the very least it needs to solve the common pain points of CSV. JSON doesn't - all it brings to the table is the ability to distinguish number vs text. That's a yawn.

It brings the ability to parse in parallel - that's a big deal. And while number vs text might not be a huge difference in theory, in practice it eliminates what, 95% of real-world parse problems?

Post reply on HN