Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

61–70 of 459 posts

Re: Friends don't let friends export to CSV

#61

I never liked articles about how you should replace CSV with some other format while pulling some absolutely idiotic reasons out of their rear... 1. CSV is underspecified Okay, so specify it for your use case and you're done? E.g use rfc3339 instead of the straw-man 1-1-1970 and define how no value looks like, which is mostly an empty string. 2. CSV files have terrible compression and performance Okay, who in their r…

I agree with your other points but the first point misses the mark. Even you specify a format, you cannot use the file for exporting data between systems and organizations if they don't all agree on that format. CSV does not have a reasonable way to encode that is using a specific spec. I can open your data with my tools and silently misinterpret it. But if you are only exporting data between yourself, that's another story.

Re: Friends don't let friends export to CSV

#62
post #5

Or export to CSV correctly and test with Excel and/or LibreOffice. Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”. I’ve had far more trouble with various export to excel functions over the years, that have much more complex third-party dependencies to function. Parsing CSV correctly is not hard, you just can’t use split and be done with it. This has been my coding kata in eve…

Y'all caught me being less than rigorous in my language while posting from my phone while in the middle of making breakfast for the kids and the wife to get out the door. To the "not well defined" aspect, I disagree in part. There is an RFC for CSV and that is defined. Now the conflated part is the use of CSV as an interchange format between various systems and locales. The complexity of data exchange, in my mind, is not the fault of the simple CSV format. Nor is it the fault of the format that third party systems have decided to not implement it as it is defined. That is a different challenge.

Data exchange via CSV becomes a multiple party negotiation between software (sometimes the other side is "set in stone" such as a third party commercial system or Excel that is entrenched), users, and users' local configuration. However, while not well defined none of these are intractable from a software engineering point of view. And if your system is valuable enough and needs to ingest data from enough places it is NOT intractable to build auto detection capabilities for all the edge cases. I have done it. Is it sometimes a challenge, yes. Do project managers love it when you give 8 point estimates for what seems like it should be a simple thing, no. That is why the coding kata RFC compliance solution as a base first makes the most sense and then you troubleshoot as bug reports come in about data problems with various files.

If you are writing a commercial, enterprise solution that needs to get it right on its own every time, then that becomes a much bigger project.

But do you know what is impossible, getting the entire universe of other systems that your customers are using to support some new interchange format. Sorry, that system is no longer under active development. No one is around to make code changes. That is not compatible with the internal tools. For better or worse, CSV is the lingua franca of columns of data. As developers, we deal with it.

And yes, do support Excel XLSX format if you can. There are multiple third party libraries to do this in various languages of various quality.

As a developer, I have made a lot of my living dealing with this stuff. It can be a fun challenge, frustrating at time, but in the end as professionals we solve what we need to to get the job done.

Re: Friends don't let friends export to CSV

#64
I've written CSV exports in C from scratch, no external dependencies required.

It's "Comma Separated Variables", it doesn't really need anymore specification than that.

These files have always imported into M$ and libre office suites without issue.

Re: Friends don't let friends export to CSV

#65

> One of the infurating things about the format is that things often break in ways that tools can't pick up and tell you about This line is emblematic of the paradigm shift LLMs have brought. It’s now easier to build a better tool than change everyone’s behaviour. > You give up human readable files, but What are we even doing here.

I think this article is causing a kerfuffle because it's hitting two different audiences very differently. Honestly the article title should have been "Friends don't let friends use CSV for data pipelines".

Because when I'm wrangling data from a human - a human who is stubbornly defending their own little island of business information like their employment depended on it[1] - a CSV is about as good as I am gonna get.

I had a bear of a time just convincing people to put their data in a delimited format, instead of a table inside a powerpoint presentation, or buried in sixty levels of Access joins, or in an SVG. I need data from "what is scroll wheel" sort of users.

If I am working system to system? That's a different requirement, a requirement that is apparently where the OP author is coming from.

[1] Because it kind of does. Having a unique platform is one of those priceless keys to being skipped in the thrice-yearly layoff rituals. Unfortunately, that means anyone approaching saying words like "integration" or "API" are shot on sight.

Re: Friends don't let friends export to CSV

#66

My takeaway is that csv has some undefined behaviours, and it takes up space. I like that everyone knows about .csv files, and it's also completely human readable. So for <100mb I would still use csv.

If both parties implement RFC 4180 and use a consistent character set encoding then I don't think there are actually any undefined behaviors. But in practice a lot of implementations are simply broken, including those from major tech companies that ought to know better.

Re: Friends don't let friends export to CSV

#67
post #64

I've written CSV exports in C from scratch, no external dependencies required. It's "Comma Separated Variables", it doesn't really need anymore specification than that. These files have always imported into M$ and libre office suites without issue.

It needs more specification than that. Have you read RFC 4180?

Re: Friends don't let friends export to CSV

#70
post #69

If one could export to Parquet from Microsoft Excel, I think this would be a goer. Until such time, it seems likely many will stick with CSVs.

If you had the option to use whatever you want, and you're already sacrificing not using a text editor, why would you use parquet instead of the myriad of choices.
Post reply on HN