Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

121–130 of 459 posts

Re: Friends don't let friends export to CSV

#121

Post by hiccuphippo »

ryncewynd wrote: post #116
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

ASCII has characters for unit, record, group and file separator. And a some days ago there was a story here about using the unicode printable representation of these for an editor friendly format.

https://news.ycombinator.com/item?id=39679378

Re: Friends don't let friends export to CSV

#122

Post by msla »

breadwinner wrote: post #98

The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Pa…

> there are no data tools that don't support CSV format.

They support CSV but not your CSV.

For example, how does quoting work? Does quoting work?

Re: Friends don't let friends export to CSV

#124

Post by crq-yml »

ryncewynd wrote: post #116
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

There are characters from ASCII for delimiting records, which are underused because they cause confusion about whether they should be represented as a state change like a backspace character, or as a glyph. See also: "nobody can agree on the line ending character sequence".

The USV proposal uses additional codepoints introduced in Unicode for the representation of the record delimiters, so they will always look and edit like character glyphs, and nobody is using them for some other purpose. The standardized look of these glyphs is unappealing, and they aren't easy to type, but it's fixable with a font and some editing functions.

Most of the issue hinges on Excel support.

Re: Friends don't let friends export to CSV

#125

Post by msla »

johnea wrote: post #64

I've written CSV exports in C from scratch, no external dependencies required. It's "Comma Separated Variables", it doesn't really need anymore specification than that. These files have always imported into M$ and libre office suites without issue.

Comma Separated Values.

Normally, I wouldn't nitpick (hold for laughter) but this is just a perfect example of how CSV is not CSV and how it's probably impossible to support everything people call CSV, simply because of the tiny (or not so tiny) differences between formats.

Re: Friends don't let friends export to CSV

#126

Post by th0ma5 »

jitl wrote: post #57
JumpCrisscross wrote: post #4

> One of the infurating things about the format is that things often break in ways that tools can't pick up and tell you about This line is emblematic of the paradigm shift LLMs have brought. It’s now easier to build a better tool than change everyone’s behaviour. > You give up human readable files, but What are we even doing here.

We’re doing data pipelines. I would rather my data pipeline go 10x faster with Parquet than be able to human read a 30gb CSV file.

The vast majority of the thread is missing this point. The CSV abuse is out of control! Lol

Re: Friends don't let friends export to CSV

#128

Post by hn_throwaway_99 »

ryncewynd wrote: post #116
hn_throwaway_99 wrote: post #111

Earlier quoted context omitted.

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties

Yes, this recent discussion has lots of good info and links in the comments: https://news.ycombinator.com/item?id=39679378

Re: Friends don't let friends export to CSV

#129

Post by diogocp »

hn_throwaway_99 wrote: post #111
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

> in the real world the spec is essentially "Can you import it to Excel?"

And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

Re: Friends don't let friends export to CSV

#130

Post by whakim »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them CSV.

I also think we shouldn't blindly dismiss the performance drawbacks of CSV when working with data pipelines. At even modest scales it becomes hard to work with very large CSVs because the data often doesn't fit into memory, a problem easily solved by Parquet and other formats assuming you only need a subset.

Post reply on HN