"I'm a big fan of Apache Parquet as a good default. You give up human readable files, but..."
Lost me right there. It has to be human readable.
Friends don't let friends export to CSV
151–160 of 459 posts
Re: Friends don't let friends export to CSV
#152Earlier quoted context omitted.
Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties
There are characters from ASCII for delimiting records, which are underused because they cause confusion about whether they should be represented as a state change like a backspace character, or as a glyph. See also: "nobody can agree on the line ending character sequence". The USV proposal uses additional codepoints introduced in Unicode for the representation of the record delimiters, so they will always look and e…
> nobody is using them for some other purpose.
There's a lot of tooling which uses them for their intended purpose, which is to represent the C0 control characters in a font, so they can be printed when they appear in a document. Your editor is probably one of those.
Which is why I consider USV a terrible idea. If I see ␇ in a file, I don't want to be constantly wondering if it's "mention ␇" or "use ␇ to represent \x07". That's why the control pictures block exists: to provide pictographs of invisible control characters. Not to make a cutesy hack "look! it's a picture of a control character pretending to be what it isn't!!" format.
Re: Friends don't let friends export to CSV
#153I like how their alternative is an instant non-starter.
Yes... if i need a better format, i'll just use sqlite.
Re: Friends don't let friends export to CSV
#154This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…
Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…
>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters
Commas can be typed by anyone on any keyboard and readable by anyone.
Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone.
Convenience is a virtue.
Re: Friends don't let friends export to CSV
#155As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…
> In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator Heh. Ah, HN. Always good for a laugh line.
Yeah, hon hon hon and all, but one of my (US) bank statements exports a CSV which uses commas in numbers in the US fashion, so $1,500 and the like. Writing a custom CSV munger to intake that into ledger-csv was... fun, but then again, only had to do it once.
Re: Friends don't let friends export to CSV
#156Trying to do business without using CSV is like trying to weld without using a torch. Might be possible but you aren't likely to have success at it.
Re: Friends don't let friends export to CSV
#157Tell me you've never worked a real job without telling me. This is a technologists solution in search of a problem. Do you also argue that "email is dead"?
Re: Friends don't let friends export to CSV
#158Earlier quoted context omitted.
Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…
>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters Commas can be typed by anyone on any keyboard and readable by anyone. Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. Convenience is a virtue.
I can't remember the last time I, or anyone I've ever worked with for that matter, ever typed up a CSV from scratch. The whole point of USV is that the delimiters can't normally be typed so you don't have to worry about escaping.
USV supports displayable delimiters (see https://github.com/SixArm/usv), so for the much more common case of editing an existing CSV in a text editor, you can just copy and paste.
Re: Friends don't let friends export to CSV
#159The reason CSV is popular is because it is (1) super simple, and (2) the simplicity leads to ubiquity. It is extremely easy to add CSV export and import capability to a data tool, and that has come to mean that there are no data tools that don't support CSV format. Parquet is the opposite of simple. Even when good libraries are available (which it usually isn't), it is painful to read a Parquet file. Try reading a Pa…
Avro is a terrible serialization format. Well, not necessarily the spec but all tooling around it is. It should never be picked over multitude of other, better options unless the company is, maybe, a Java shop.
Re: Friends don't let friends export to CSV
#160CSV is a superb, incredibly useful data format.. but not perfect or complete.
Instead of breaking CSV by adding to it .. I recommend augmenting it :
It would be useful to have a good standardized / canonical json format for things like encoding, delimiter, schema and metadata, to accompany a zipped csv file, perhaps packaged in the same archive.
Gradually datasets would become more self-documenting and machine-usable without wrangling.