Surprised no one has mentioned sqlite even once in these comments.
I regret forgetting about it in the article. sqlite is a great solution.
Friends don't let friends export to CSV
281–290 of 459 posts
Re: Friends don't let friends export to CSV
#282This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…
Agree. Not saying csv doesn’t have its issues, but I don’t think the author made a convincing argument. A lot of the issues the author brought up didn’t sound that bad and/or it sounds like he never looked at the source data first. If you’re doing work with large datasets, I think it’s a good practice to at least go and look at the source data briefly to see what to expect. This will give you a good idea of the forma…
But the point is that you don't have to look at the source data if you have an actual specification and defined format, right?
Re: Friends don't let friends export to CSV
#283Using parquet in python requires installing pyarrow and numpy, whereas CSV comes with stdlib.
Also, the csv has a very pythonic interface vis-a-vis parquet, in most cases if I can fit the file in memory I would go with CSV.
Re: Friends don't let friends export to CSV
#284Every single use I've ever seen of CSV would be improved by the very simple change to TSV. Even Excel can handle it. It is far safer to munge data containing tabs (convert to spaces, etc), than commas (remove? convert to dots? escape?). The better answer is to use ASCII separators as Lyndon Johnson intended, but that turns out to be asking a lot of data producers. Generating TSV is usually easier than generating CSV.
And as a practical matter, Google sheets can export tsv. It didn't always have csv as an export (it does now). I've used Google sheets as a business friendly way to edit data for many years. With a little bit of instruction, this can work pretty well. Usually it just involves me telling people to not get creative with the column names and leave them alone.
This may shock some people but Excell and other MS Office products aren't that common any more to have around for developers. Through the nineties, MS had a defacto monopoly on desktop back in the day. But these days, a lot of developers use macs and they are also popular with business people. Google docs seems popular with startups. All companies I've been in for the last 12 years default to that.
Anyway, I've done this on a few teams where it just short cuts the whole discussion about needing a bespoke UI to edit some table of stuff for managers. I've even replaced a failed project to build such a UI with simple stuff like this. All you need is a bit of validation when importing and you can keep the last know good export in git and do pull requests to update the file. There's a weird dynamic where giving this level of control to some managers actually makes them feel more involved and engaged.
My preferred format to work with is actually ndjson (newline delimited json). Mainly because it's easier to deal with nested objects and lists in that. Whenever people start putting lists in table cells or start adding dots or other separators to their column names to indicate some kind of hierarcy, ndjson is the better solution. I've seen all sorts of attempts by business people to stuff non tabular data into a table.
Re: Friends don't let friends export to CSV
#285Earlier quoted context omitted.
Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.
A manager: I wish I could have a CSV inside my CSV. Any sane person: NO!
Well you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field".
As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.
Re: Friends don't let friends export to CSV
#286There is CSVY, which lets you set a delimiter, schema, column types, etc. and has libraries in many languages and is natively supported in R.
Also is backwards-compatible with most CSV parsers.
Re: Friends don't let friends export to CSV
#287Earlier quoted context omitted.
To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them…
I deal with gig size csvs all the time and don’t have any performance issues. These aren’t huge files, but decent sized. And most are just a few megs and only thousands to millions of records. Csv is not very performant, but it doesn’t matter for these use cases. I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my…
If I run a simulation workload it's pretty easy to generate gigabytes of data per second. CSV encoding adds a huge overhead space and time wise, so saving trajectories to disc for later analysis can easily become the bottleneck.
I have had many other situations where CSV was the bottleneck.
I still would default to CSV first in many situations because it's robust and easily inspected by hand.
Re: Friends don't let friends export to CSV
#288Earlier quoted context omitted.
It kind of does. See `man ascii` * FS (0x1C) file separator * GS (0x1D) group separator * RS (0x1E) record separator * US (0x1F) unit separator I've never seen these in the wild though.
They‘re used a lot in barcodes, e.g. for delimiting the different fields of a driving license.
That sounds like the premise for an utterly fascinating deep dive.
Re: Friends don't let friends export to CSV
#289Earlier quoted context omitted.
> Esperanto is a superior language to English. Not really. You can say it's more regular but that's because it sees barely any actual use; if it ever gained popularity it wouldn't stay regular. (And given that pilots speak in set phrases anyway, irregularity isn't really an issue). It's not a great language by any stretch, it's an awkward mismash of four european languages; sure it sounds kind of nice in an Italianat…
It would stay regular if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". English really is a disaster of a language. There was a(nother) great XKCD about it just a few days ago. https://xkcd.com/2907/
> if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct".
There is no way it can work.
People don't care about governing bodies when they speak a language.
Re: Friends don't let friends export to CSV
#290As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…
Isn't this exactly what quoting solves?
i.e.: ``` "1,20","2,3",hello "2,40","4,6",goodbye ```
If your tool reads CSV by doing `string_split(',', line);`, your tool is doing it wrong. There's a bunch of nuance and shit, which can make CSVs interesting to work with, but storing a comma in a field is a pretty solved issue if the tool in question has more than 5 minutes thought put into it.