Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

301–310 of 711 posts

Re: A love letter to the CSV format

#301

Post by 486sx33 »

I love CSV for a number of reasons. Not the least of which it’s super easy to write a program (code) in C to directly output all kinds of things to CSV. You can also write simple middleware to go from just about any database or just general “thing” to CSV. Very easily. Then toss CSV into excel and do literally anything you want.

It’s sort of like, the computing dream when I was growing up.

+1 to ini files. I like you can mess around with them yourself in notepad. Wish there was a general outline / structure to those though.

Re: A love letter to the CSV format

#302

Post by squeaky-clean »

niccl wrote: post #273
hajile wrote: post #183

Earlier quoted context omitted.

Can you point me to a language with any significant number of users that does NOT have a JSON library? I went looking at some of the more niche languages like Prolog, COBOL, RPG, APL, Eiffel, Maple, MATLAB, tcl, and a few others. All of these and more had JSON libraries (most had one baked into the standard library). The exceptions I found (though I didn't look too far) were: Bash (use jq with it), J (an APL variant)…

Excel. Before you dismiss it as 'not a language, people have argued that it is. And you can definitely program stuff in it, and so that surely makes it a language

Excel can import and parse JSON, it's under the "Get Data" header. It doesn't have a direct GUI way to export to JSON, but it takes just a few lines in Office Scripts. You can even use embedded TypeScript to call JSON.stringify.

Re: A love letter to the CSV format

#303

Post by autoexec »

derriz wrote: post #166
koolba wrote: post #63

Earlier quoted context omitted.

JSON serialized without extra white space with one line per record is superior to CSV. If you want CSV-ish, enforce an array of strings for each record. Or go further with actual objects and non-string types. You can even jump to an arbitrary point and then seek till you see an actual new line as it’s always a record boundary. It’s not that CSV is an invalid format. It’s that libraries and tools to parse CSV tend to…

> It’s that libraries and tools to parse CSV tend to suck. Whereas JSON is the lingua franca of data. This isn't the case. An incredible amount of effort and ingenuity has gone into CSV parsing because of its ubiquity. Despite the lack of any sort of specification, it's easily the most widely supported data format in existence in terms of tools and language support.

> it's easily the most widely supported data format in existence in terms of tools and language support.

Even better, the majority of the time I write/read CSV these days I don't need to use a library or tools at all. It'd be overkill. CSV libraries are best saved for when you're dealing with random CSV files (especially from multiple sources) since the library will handle the minor differences/issues that can pop up in the wild.

Re: A love letter to the CSV format

#304

Post by sitkack »

theLiminator wrote: post #234
sitkack wrote: post #71

Earlier quoted context omitted.

I switched to JSONL over a decade ago and I would recommend everyone else to also have switched then. This whole thread is an uninformed rehash of bad ideas.

I think that might make sense ingest side, but that's very expensive to deal with if you're doing anything remotely large. I think sinking into something like delta-lake or iceberg probably makes sense at scale. But yeah, I definitely agree that CSV is not great.

JSONL as a replacement for CSV, you shouldn't be using CSV as format for long term storage or querying, it has so many downsides and nearly zero upsides.

JSONL when compressed with zstd, most of "expensive if large" disappears as well.

Generating and consuming JSONL can easily be in the GB/s range.

Re: A love letter to the CSV format

#305

Post by bobmcnamara »

nextts wrote: post #211
taeric wrote: post #69

Earlier quoted context omitted.

Reminds me of a fatal flaw of yaml. Turns out truncating a yaml file doesn't make it invalid. Which can lead to some rather non-obvious failures.

What is the failure mode where a yaml file gets truncated? They are normally config files in Git. Or uploaded to S3 or Kubernetes etc. CSV has the same failure mode. As does HTML. (But not XML)

Bad merges.

Re: A love letter to the CSV format

#307

Post by nukem222 »

deepsun wrote: post #286
account-5 wrote: post #276

I like CSV for the same reasons I like INI files. It's simple, text based, and there's no typing encoded in the format, it's just strings. You don't need a library. They're not without their drawbacks, like no official standards etc, but they do their job well. I will be bookmarking this like I have the ini critique of toml: https://github.com/madmurphy/libconfini/wiki/An-INI-critique... I think the first line of the…

Similarly I had once loved the schemaless datastorages. They are so much simpler! Until I worked quite a bit with them and realized that there's always schema in the data, otherwise it's just random noise. The question is who maintains the schema, you or a dbms. Re. formats -- the usefulness comes from features (like format enforcing). E.g. you may skip .ini at all and just go with lines on text files, but somewhere…

Schemaless can be accomplished with well-formed formats like json, xml, yaml, toml, etc. from the producer side these are roughly equivalent interfaces. There's zero upside to using CSVs except to comfort your customer. Or maybe you have centered importing of CSVs into your actual business, in which case you should probably not exist.

Re: A love letter to the CSV format

#308

Post by ok123456 »

It's an ad hoc text format that is often abused and a last-chance format for interchange. While heuristics can frequently work at determining the structure, they can just as easily frequently fail. This is especially true when dealing with dates and times or other locale-specific formats. Then, people outright abuse it by embedding arrays or other such nonsense.

You can use CSV for interchange, but a duck db import script with the schema should accompany it.

Re: A love letter to the CSV format

#309

Post by PaulHoule »

I wish this was a joke. I'm always trying to convince data scientists with a foot in the open source world that their life will be so much better if they use parquet or Stata or Excel or any other kind of file but CSV.

On top of all the problems people mention here involving the precise definition of the format and quoting, it's outright shocking how long it takes to parse ASCII numbers into floating point. One thing that stuck with me from grad school was that you could do a huge number of FLOPS on a matrix in the time it would take to serialize and deserialize it to aSCII.

Re: A love letter to the CSV format

#310

Post by PaulHoule »

nukem222 wrote: post #294
account-5 wrote: post #276

I like CSV for the same reasons I like INI files. It's simple, text based, and there's no typing encoded in the format, it's just strings. You don't need a library. They're not without their drawbacks, like no official standards etc, but they do their job well. I will be bookmarking this like I have the ini critique of toml: https://github.com/madmurphy/libconfini/wiki/An-INI-critique... I think the first line of the…

> It's simple My experience has indicated the exact opposite. CSVs are the only "structured" format nobody can claim to parse 100% (ok probably not true thinking about html etc, just take this as hyperbole.) Just use a well-specified format and save your brain-cells. Occasionally, we must work with people who can only export to csv. This does not imply csv is a reasonable way to represent data compared to other optio…

The HTML 5 spec says exactly how you're supposed to deal with broken HTML files.
Post reply on HN