Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

131–140 of 711 posts

Re: A love letter to the CSV format

#131

Post by koolba »

packetlost wrote: post #125
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

Eh, it really isn't. The format does not lend itself to tabular data, instead the most natural way of representing data involves duplicating the keys N times for each record.

You can easily represent it as an array:

    [“foo”,”bar”,123]
That’s as tabular as CSV but you now have optional types. You can even have lists of lists. Lists of objects. Lists of lists of objects…

Re: A love letter to the CSV format

#132

Post by LPisGood »

criddell wrote: post #111
LPisGood wrote: post #106

Earlier quoted context omitted.

I always treat CSVs as comma separated values with new line delimiters. If it’s a new line, it’s a new row.

Do you ever have CSV data that has newlines within a string?

No - that’s what I’m trying to say. If I have newlines I use something else.

Re: A love letter to the CSV format

#133

Post by deathanatos »

mitchpatin wrote: post #39

CSV still quietly powers the majority of the world’s "data plumbing." At any medium+ sized company, you’ll find huge amounts of CSVs being passed around, either stitched into ETL pipelines or sent manually between teams/departments. It’s just so damn adaptable and easy to understand.

> It's just so damn adaptable

Like a rapidly mutating virus, yes.

> and easy to understand.

Gotta disagree there.

For example, one of the CSVs my company shovels around is our Azure billing data. There are several columns that I just have absolutely no idea what the data in them is. There are several columns we discovered are essentially nullable¹ The Hard Way when we got a bill for which, e.g., included a charge that I guess Azure doesn't know what day that charge occurred on? (Or almost anything else about it.)

(If this format is documented anywhere, well, I haven't found the docs.)

Values like "1/1/25" in a "date" column. I mean, I did say it was an Azure-generated CSV, so obviously the bar wasn't exactly high, but then it never is, because anyone wanting to build something with some modicum of reliability, or discoverability, is sending data in some higher-level format, like JSON or Protobuf or almost literally anything but CSV.

If I can never see the format "JSON-in-CSV-(but-we-fucked-up-the-CSV)" ever again, that would spark joy.

(¹after parsing, as CSV obviously lacks "null"; usually, "" is a serialized null.)

Re: A love letter to the CSV format

#134

Post by int_19h »

benwilber0 wrote: post #114
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

JSON is a textual encoding no different than CSV. It's just that people tend to use specialized tools for encoding and decoding it instead of like ",".join(row) and row.split(",") I have seen people try to build up JSON strings like that too, and then you have all the same problems. So there is no problem with CSV except that maybe it's too deceptively simple. We also see people trying to build things like URLs and q…

The problem with CSV is that there's no clear standard, so even if you do reach for a library to parse it, that doesn't ensure compatibility.

Re: A love letter to the CSV format

#135

Post by bee_rider »

mort96 wrote: post #96
nayuki wrote: post #75

Earlier quoted context omitted.

There are some specialized text editors for editing files with tabs. https://en.wikipedia.org/wiki/Tab_stop#Dynamic_tab_stops , https://nick-gravgaard.com/elastic-tabstops/ , https://tibleiz.net/code-browser/

If have to use a dedicated tabular data editing program, I may as well use a spreadsheet application. What do the options you propose do better than libreoffice calc?

    column -t | less -S
is pretty nice because you can inspect a file or dataset on a server (no X) before downloading, to see if you even want to bother. Or you can pass it along through a series of pipes to just get the rows you want.

Re: A love letter to the CSV format

#136

Post by criddell »

thesuitonym wrote: post #122
criddell wrote: post #111

Earlier quoted context omitted.

Do you ever have CSV data that has newlines within a string?

I don't. If I ever have a dataset that requires newlines in a string, I use another method to store it. I don't know why so many people think every solution needs to to be a perfect fit for every problem in order to be viable. CSV is good at certain things , so use it for those things! And for anything it's not good at, use something else!

> use something else

You don't always get to pick the format in which data is provided to you.

Re: A love letter to the CSV format

#137

Post by 0x073 »

LPisGood wrote: post #132
criddell wrote: post #111

Earlier quoted context omitted.

Do you ever have CSV data that has newlines within a string?

No - that’s what I’m trying to say. If I have newlines I use something else.

Wouldn't work if csv is used as a exchange format with external companies.

Re: A love letter to the CSV format

#138

Post by evnp »

Suppafly wrote: post #102
evnp wrote: post #87

Anyone with a love of CSV hasn't been asked to deal with CSV-injection prevention in an enterprise setting, without breaking various customer data formats. There's a dearth of good resources about this around the web, this is the best I've come across: https://georgemauer.net/2017/10/07/csv-injection.html

That mostly breaks down to "excel is intentionally stupid with csv files if you don't use the import function to open them" along with the normal "don't trust customer input without stripping or escaping it" concerns you'd have with any input.

That was my initial reaction as well – it's a vulnerability in MS software, not ours, not our problem. Unfortunately, reality quickly came to bear: our customers and employees ubiquitously use excel and other similar spreadsheet software, which exposes us and them to risk regardless where the issue lies. We're inherently vulnerable because of the environment we're operating in, by using CSV.

"don't trust customer input without stripping or escaping it" feels obvious, but I don't think it stands up to scrutiny. What exactly do you strip or escape when you're trying to prevent an unknown multitude of legacy spreadsheet clients that you don't control from mishandling data in an unknown variety of ways? How do you know you're not disrupting downstream customer data flows with your escaping? The core issue, as I understand it, stems from possible unintended formula execution – which can be prevented by prefixing certain cells with a space or some invisible character (mentioned in the linked post above). This _does_ modify customer data, but hopefully in a way that unobtrusive enough to be acceptable. All in all, it seems to be a problem without a perfect solution.

Re: A love letter to the CSV format

#140

Post by Xelbair »

999900000999 wrote: post #109

The best part about csv, anyone can write a parser in 30 minutes meaning that I can take data from the early '90s and import it into a modern web service. The worst part about CSV, anyone can ride a parser in about 30 minutes, meaning that it's very easy to get incorrect implementations, incorrect data, and other strange undefined behaviors. But to be clear json, and yaml also have issues with everyone trying to rein…

until you find someone abusing XSD schemas, or someone designing a "dynamically typed" XML... or sneaks in extra data in comments - happened to me way often than it should.
Post reply on HN