Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

271–280 of 459 posts

Re: Friends don't let friends export to CSV

#271

Post by askvictor »

TimTheTinker wrote: post #204
rossdavidh wrote: post #174

"You give up human readable files, but what you gain in return is..." Stop right there. You lose more than you gain. Plus, taking the data out of [proprietary software app my client's data is in] in csv is usually easy. Taking the data out in Apache Parquet is...usually impossible, but if it is possible at all you'll need to write the code for it. Loading the data into [proprietary software app my client wants data p…

I think his arguments apply more closely to SQLite databases. They're not directly human readable, but boy are there a lot of tools for working with them.

We have a use case where we effectively need to have a relational database, but in git. The database doesn't change much, but when it does, references between tables may need to be updated. But we need to easily be able to see diffs between different versions. We're trying an SQLite DB, with exports to CSV as part of CI - the CSV files are human-readable and diff'able.

It's also worth noting that SQLite can ingest CSV files into memory and perform queries on them directly - if the files are not too large, it's possible to bypass the sqlite format entirely.

Re: Friends don't let friends export to CSV

#272

Post by hnlmorg »

eviks wrote: post #247
albert_e wrote: post #179

Earlier quoted context omitted.

Everyone of us was a beginner at some point. The first time we came across CSV format we likely typed it in notepad by hand. A lot of issues with CSVs are also sometimes troubleshooted by hand-- by manually fixing a quote or a comma. There is value is the ability to do this level of editing and troubleshooting.

No we didn't, we likely typed in Excel by double clicking on our first csv

I can’t speak for everyone, but I definitely didn’t use Excel.

Re: Friends don't let friends export to CSV

#273

Post by chrisandchris »

hn_throwaway_99 wrote: post #111
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

So true. Working with imports/exports in CSV from ERP software. One can't imagine how often "Oh, this import doesn't work. I'll just fix the CSV file" occurs. Try that with some compressed, "esoteric" file, or even CML and users will break it.

Besides all the downsides CSV has, as soon as it's not only machine-machine communication and a human is involved, CSV is just simole enough.

Re: Friends don't let friends export to CSV

#275

Post by rightbyte »

wombatpm wrote: post #249
djbusby wrote: post #219

Earlier quoted context omitted.

ASCII has had field and record separators since like, forever. Wish we had kept using those.

No you don’t. It’s a holdover from when files were on tapes. The logic is all inverted too. Record separators were at the beginning of a record. Group and unit separators could then be nested. You really needed a spec for whatever document you were trying to makt

It doesn't matter if the sentinel byte is after or before each record.

Having it before is nice for syncing byte streams.

Re: Friends don't let friends export to CSV

#276

Post by likis »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Did you actually read the conclusion at the end of the article?

"Of course, we can't conclude that you should never export to CSV. If your users are just going to try to find the quickest way to turn your data into CSV anyway, there's no reason why you shouldn't deliver that. But it's a super fragile file format to use for anything serious like data integration between systems, so stick with something that at the very least has a schema and is more efficient to work with."

Re: Friends don't let friends export to CSV

#277

Post by Pikamander2 »

diogocp wrote: post #129
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

I've been amazed by how much better LibreOffice is at importing CSVs in a sane manner than Excel. Its CSV import prompt is nothing short of the gold standard and puts Excel to shame.

Also, even if the CSV format is completely valid, Excel will still find a way to misinterpret some of your cells in baffling ways, destroying the original data in the process if you don't notice it in time.

Re: Friends don't let friends export to CSV

#278

Post by rightbyte »

samatman wrote: post #152
crq-yml wrote: post #124

Earlier quoted context omitted.

There are characters from ASCII for delimiting records, which are underused because they cause confusion about whether they should be represented as a state change like a backspace character, or as a glyph. See also: "nobody can agree on the line ending character sequence". The USV proposal uses additional codepoints introduced in Unicode for the representation of the record delimiters, so they will always look and e…

> nobody is using them for some other purpose. There's a lot of tooling which uses them for their intended purpose , which is to represent the C0 control characters in a font, so they can be printed when they appear in a document. Your editor is probably one of those. Which is why I consider USV a terrible idea. If I see ␇ in a file, I don't want to be constantly wondering if it's "mention ␇" or "use ␇ to represent \…

I don't underatand why this is a question up for debate. You need eye tracking, so that there is a beep when you read the relevant part.

Re: Friends don't let friends export to CSV

#279

Post by Freak_NL »

diogocp wrote: post #129
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

Oh absolutely. Don't forget about the giant mess Microsoft made in countries like the Netherlands, where Excel will refuse to automatically open comma-separated values files (you know, CSV), unless the separator is a semicolon — because someone in the past thought that was how the Dutch did it.

You want people to be able to open your nice and sensible CSV files in Excel? First figure out which arcane locale specific settings you need, then generate those monstrosities and annoy anyone who genuinely expected Unicode comma-separated values.

My solution was to just write a generic spreadsheet writer component in our API and have it generate CSV (normal, comma-separated, UTF-8), XLSX, or ODS. Anyone using Excel just grabs the XLSX output. Generating XLSX and ODS was just a matter of saving a minimal example file for each and figuring out where to write cells and rows. No extra library needed (both file formats are just zipped up XML files), and everybody is happy.

Re: Friends don't let friends export to CSV

#280

Post by likis »

hn_throwaway_99 wrote: post #111
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

It seems like you missed the conclusion in the article. If users want CSV exports, let them have it.

If you have important data being shuffled around systems, pick something with a specification instead.

Post reply on HN