Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

161–170 of 711 posts

Re: A love letter to the CSV format

#161

Post by bee_rider »

mort96 wrote: post #152
bee_rider wrote: post #135

Earlier quoted context omitted.

column -t | less -S is pretty nice because you can inspect a file or dataset on a server (no X) before downloading, to see if you even want to bother. Or you can pass it along through a series of pipes to just get the rows you want.

What part of "the value of CSV is to have something that can be easily viewed and modified in any text editor" do you not understand

I’m not sure where that quote is from but it is incorrect, CSVs aren’t easily viewed or modified in text editors in general (at least not in any way that takes advantage of their tabular nature).

There’s at least a slight chance that tab separated values will look ok in a text editor (although in general, nope).

Re: A love letter to the CSV format

#162

Post by LPisGood »

0x073 wrote: post #137
LPisGood wrote: post #132

Earlier quoted context omitted.

No - that’s what I’m trying to say. If I have newlines I use something else.

Wouldn't work if csv is used as a exchange format with external companies.

Of course. I’m not saying I roll my own parser for every project that uses a CSV file, I’m just describing my criteria for using CSV vs some other format when I have the option.

Re: A love letter to the CSV format

#163

Post by hajile »

rcbdev wrote: post #153
int_19h wrote: post #134

Earlier quoted context omitted.

The problem with CSV is that there's no clear standard, so even if you do reach for a library to parse it, that doesn't ensure compatibility.

Same for JSON though. What Python considers a valid JSON might not be that if you ask a Java library.

JSON has a clearly-defined standards: ISO/IEC 21778:2017, IETF RFC 7159, and ECMA-404. Additionally, Crockford has had a spec available on json.org since it's creation in 2001.

Do you have any examples of Python, Java, or any of the other Tiobe top 40 languages breaking the JSON spec in their standard library?

In contrast, for the few of those that have CSV libraries, how many of those libraries will simply fail to parse a large number of the .csv variations out there?

Re: A love letter to the CSV format

#164

Post by evnp »

aorth wrote: post #142
Suppafly wrote: post #102

Earlier quoted context omitted.

That mostly breaks down to "excel is intentionally stupid with csv files if you don't use the import function to open them" along with the normal "don't trust customer input without stripping or escaping it" concerns you'd have with any input.

Someone filed a bug report on a project I work on, saying that it was a security vulnerability that we don't prefix cell values with a single quote (') when the cell content contains certain values like an equal sign (=). They said this can cause Excel to evaluate the content and potentially run unsafe code. I responded that this was Excel's problem, not ours, and that nobody would assign a CVE to our product for suc…

There are a lot of these sorts of bug reports running around, to the point that Google's bug bounty program has classified them as invalid: https://bughunters.google.com/learn/invalid-reports/google-p...

I agree with the characterization ("security theater") of these bug reports. The problem is that the intentions of these reports don't make the potential risk less real, depending on the setting, and I worry that the "You're just looking for attention" reaction (a very fair one!) leads to a concerning downplaying of this issue across the web.

As a library author, I agree this very well may not be something that needs to be addressed. But as someone working in a company responsible for customers, employees, and their sensitive information, disregarding this issue disregards the reality of the tools these people will invariably use, downstream of software we _are_ responsible for. Aiming to make this downstream activity as safe as possible seems like a worthy goal.

Re: A love letter to the CSV format

#165

Post by WorldMaker »

corysama wrote: post #141
kevmo314 wrote: post #33

Earlier quoted context omitted.

What do you mean? I just push the Record Separator key on my keyboard. /s in case :)

The entire argument against ASCII Delimited Text boils down to "No one bothered to support it in popular editors back in 1984. Because I grew up without it, it is impossible to imagine supporting it today." You need 4 new keyboard shortcuts. Use ctrl+, ctrl+. ctrl+[ ctrl+] You need 4 new character symbols. You need a bit of new formatting rules. Pretty much page breaks decorated with the new symbols. It's really not…

In Windows (and DOS EDIT.COM and a few other similarly ancient tools) there have existed Alt+028, Alt+029, Alt+030, and Alt+031 for a long time. I vaguely recall some file format I was working with in QBASIC used some or all of them and I was editing those files for some reason. That was not quite as far back as 1984, but sometime in the early 1990s for sure. I believe EDIT.COM had basic glyphs for them too, but I don't recall what they were, might have been random Wingdings like the playing card suits.

Having keyboard shortcuts doesn't necessarily solve why people don't want to use that format, either.

Re: A love letter to the CSV format

#166

Post by derriz »

koolba wrote: post #63
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data.

This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.

Re: A love letter to the CSV format

#167

Post by whizzter »

hajile wrote: post #163
rcbdev wrote: post #153

Earlier quoted context omitted.

Same for JSON though. What Python considers a valid JSON might not be that if you ask a Java library.

JSON has a clearly-defined standards: ISO/IEC 21778:2017, IETF RFC 7159, and ECMA-404. Additionally, Crockford has had a spec available on json.org since it's creation in 2001. Do you have any examples of Python, Java, or any of the other Tiobe top 40 languages breaking the JSON spec in their standard library? In contrast, for the few of those that have CSV libraries, how many of those libraries will simply fail to p…

Not to mention that stuff like Excel loves to export CSV files in "locale specific ways".

Sometimes commas to delimiter, sometimes semicolons, floating point values might have dots or commas to separate fraction digits.

Not to mention text encodings, Ascii, western european character sets, or maybe utf-8 or whatever...

It's a bloody mess.

Re: A love letter to the CSV format

#168

Post by koolba »

packetlost wrote: post #151
koolba wrote: post #131

Earlier quoted context omitted.

You can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

You're missing my point: basically nothing spits out data in that format because it's not ergonomic to do so. JSON is designed to represent object hierarchies, not tabular data.

JSON was designed to represent any data. There's plenty of systems that spit out data in exact that format because it's the natural way to represent tabular data using JSON serialization. And clearly if you're the one building the system you can choose to use it.

Re: A love letter to the CSV format

#169

Post by hajile »

packetlost wrote: post #151
koolba wrote: post #131

Earlier quoted context omitted.

You can easily represent it as an array: [“foo”,”bar”,123] That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

You're missing my point: basically nothing spits out data in that format because it's not ergonomic to do so. JSON is designed to represent object hierarchies, not tabular data.

CSV is lists of lists of fixed length.

JSON is lists of lists of any length and groups of key/value pairs (basically lisp S-expressions with lots of unnecessary syntax). This makes it a superset of CSV's capabilities.

JSON fundamentally IS made to represent tabular data, but it's made to represent key-value groups too.

Why make it able to represent tabular data if that's not an intended use?

Re: A love letter to the CSV format

#170

Post by liotier »

Someone1234 wrote: post #118
liotier wrote: post #93

CSV is the bane of my existence. There is no reason to use it outside of legacy use-cases, when so many alternatives are not so brittle that they require endless defensive hacks to avoid erring as soon as exposed to the universe. CSV must die.

CVS isn't brittle, and I'm not sure what "hacks" you're referring to. If you or your parser just follow RFC4180 (particularly quote every field, and double quoting to cancel-quote), that will get you 90%+ compatibility.

/me laughs in legacese

RFC4180 is a late attempt at CSV standardization, merely codifying a bunch of sane practices. It also provides a nice specification for generating CSV. But anyone taking care to code from a specification might as well use a proper file format.

The real specification for CSV is as follows: "Valid CSV is whatever is designated as CSV by its emitter". I wish I was joking.

There is literally an infinity of ways CSV can be broken. The developer will bump his head on each as he encounters them, and add a specific fix. After a while, his code will be robust against the local strains of CSV... Until the next mutation is encountered after acquiring yet another company with a bunch of ERP way past their last extended maintainance era, a history of local adaptations and CSV as a message bus.

Post reply on HN