there needs to be some pandoc (panbin?) for binary formats to convert between parquet, hdf5, fits, netcdf, grib, ROOT, sqlite, etc. (Ok these are not all equivalent in capability...).
Friends don't let friends export to CSV
171–180 of 459 posts
Re: Friends don't let friends export to CSV
#172CSV is a superb, incredibly useful data format.. but not perfect or complete. Instead of breaking CSV by adding to it .. I recommend augmenting it : It would be useful to have a good standardized / canonical json format for things like encoding, delimiter, schema and metadata, to accompany a zipped csv file, perhaps packaged in the same archive. Gradually datasets would become more self-documenting and machine-usable…
> It would be useful to have a good standardized / canonical json format for things like encoding, delimiter, schema and metadata
We already have that. Dan Brickley and others put a lot of thoughtful effort into it https://www.w3.org/TR/tabular-data-primer/#dialects>:
> A lot of what's called "CSV" that's published on the web isn't actually CSV. It might use something other than commas (such as tabs or semi-colons) as separators between values, or might have multiple header lines. [...] You can provide guidance to processors that are trying to parse those files through the `dialect` property
As is usually the case with standards, it's not that the standard doesn't exist but that people just don't even bother checking (much less caring about what it says or actually trying to follow it).
Re: Friends don't let friends export to CSV
#173CSV is very durable. If I want it read in 20 years, csv is the way to go until it’s just too big to matter. Of course there are better formats. But for many use cases friends encourage friends to export to CSV.
Eh. I much prefer to produce and consume line delimited JSON. (Or just raw JSON). Its easy to parse, self descriptive and doesn't have any of CSV's ambiguity around delimiters and escape characters. Its a little harder to load into a spreadsheet, but in my experience, way easier to reliably parse in any programming language.
JSON(newline delimited or full file) is significantly larger than csv. With csv the field name is mentioned once in the header row. In JSON every single line repeats the field names. It adds up fast, and is more of a difference than between csv to parquet.
Re: Friends don't let friends export to CSV
#174"You give up human readable files, but what you gain in return is..."
Stop right there. You lose more than you gain.
Plus, taking the data out of [proprietary software app my client's data is in] in csv is usually easy. Taking the data out in Apache Parquet is...usually impossible, but if it is possible at all you'll need to write the code for it.
Loading the data into [proprietary software app my client wants data put into] using a csv is usually already a feature it has. If it doesn't, I can manipulate csv to put it into their import format with any language's basic tools.
And if it doesn't work, I can look at the csv myself, because it's human readable, to see what the problem is.
90% of real world coding is taking data from a source you don't control, and somehow getting it to a destination you don't control, possibly doing things with it along the way. Your choices are usually csv, xlsx, json, or [shudder] xml. Looking at the pros and cons of those is a reasonable discussion to have.
Re: Friends don't let friends export to CSV
#175Parquet is a columnar format. Which might be what you want, but it also might not, like if you want to process one row at a time in a stream. Maybe avro would be a better format in that case?
Re: Friends don't let friends export to CSV
#176what about just exporting to sqlite files?
Re: Friends don't let friends export to CSV
#177Earlier quoted context omitted.
And the application doesn't try to convert the cells into non-string data types like numbers, dates, etc.
Converting strings into other data types is out of scope for CSV, not really undefined behavior. The type conversions happen at a later stage of the import process.
It's out of scope for the RFC, but it could still be undefined behavior for the import/export process.
Re: Friends don't let friends export to CSV
#178Earlier quoted context omitted.
I think IFS=$'\a' works too.
Only in bash and possibly other shells that extend the POSIX syntax, not in the basic POSIX standard.
I seem to remember something about dash adding that functionality.
...I found it - the question is under review by the Austin group for inclusion in POSIX.
Re: Friends don't let friends export to CSV
#179Earlier quoted context omitted.
>I do hope "Unicode Separated Values" takes off. It's essentially just a slight tweak to CSV where the delimiters are special unicode characters Commas can be typed by anyone on any keyboard and readable by anyone. Special Unicode Characters(tm) can't be typed by anyone on any keyboard and readable by noone. Convenience is a virtue.
I can't remember the last time I, or anyone I've ever worked with for that matter, ever typed up a CSV from scratch. The whole point of USV is that the delimiters can't normally be typed so you don't have to worry about escaping. USV supports displayable delimiters (see https://github.com/SixArm/usv ), so for the much more common case of editing an existing CSV in a text editor, you can just copy and paste.
Everyone of us was a beginner at some point. The first time we came across CSV format we likely typed it in notepad by hand. A lot of issues with CSVs are also sometimes troubleshooted by hand-- by manually fixing a quote or a comma.
There is value is the ability to do this level of editing and troubleshooting.
Re: Friends don't let friends export to CSV
#180Earlier quoted context omitted.
Are there unicode characters specifically for delimiters? If Excel had a standardised "Save as USV" option it would solve so many issues for me. I get so many broken CSVs from third-parties
> Are there unicode characters specifically for delimiters? We could use the HL7 pipe ‘|’ and all enjoy that hell.
God, please no.
For those unfamiliar with the atrocity that is HL7v2, the format is essentially CSV, but with the record separator set to a lone CR, and the field separator usually set to |. Usually, because the format lets the consumer of the format redefine it, for whatever reasons. (The first use of the field separator it determines whatever character it will be. Thankfully, the first use is in a fixed spot, so it's determinable, but still. Oh, but we don't know the character encoding until like the 18th field in … and it doesn't necessarily have to be an ASCII superset. So I have no idea what an HL7v2 message in a non-ASCII superset even looks like, or how a parser is even supposed to reasonably parse such a thing. I presume attempt a decoding in all possible decodings, and then see which one matches the embedded character set, and pray nobody can create a polyglot?)
There's also further separators, delimiting within a field.
It also has its own escape sequences, to deal with the above.
… and it is what carries an unfortunate amount of medical data, and is generally how providers interoperate, despite the existence of more civilized standards like FHIR