[deleted]
Friends don't let friends export to CSV
411–420 of 459 posts
Re: Friends don't let friends export to CSV
#412The poor performance argument is not true even for Python ecosystem that the author discusses. Try saving geospatial data in GeoPackage, GeoJson, FlatGeobuf. They are saved slower than in plain CSV (the only inconvenience is that you must convert geometries into WKT strings). GeoPackage was "the Format of the Future" 8 years ago, but it's utterly slow when saving, because it's an SQLite database and indexes all the d…
> for geospatial data... GeoPackage was "the Format of the Future" 8 years ago What's the current consensus? Can you link to a summary article? (Some still say GeoPackage is: https://mapscaping.com/shapefiles-vs-geopackage/ )
I'd say compared to Shapefile, it is indeed better in every aspect (to begin with, shp has 8-character column names limit). For some kinds of data and operations GPKG is superior to other geo-formats. Like 1) store a lot of data, but retreive within an area (you can set an arbitrary polygon as a filter with GDAL driver, IIRC), 2) append/delete/modify and have the data indexed -- with CSV here you'll have to just reprocess and rewrite the entire file.
The problem is that in data science you want whole datasets to be atomic, to have reproducible results. So you don't care much of these sub-dataset operations.
Another sudden issue with GPKG and atomicity is that sqlite changes DB modification time every time you just read. So if you use Makefile, which checks for updates by modification time, you either have to let it re-run some updates, or manually touch other files downstream, or rely on separate files that you `touch` (unix tool that updates file's modification time).
I read a Russian OSM blogger Ilya Zverev evangelize for GPKG back in 2016 in his blog: https://shtosm.ru. I guess he was referring to GPKG vs ShapeFile too, not CSV. I think he's totally correct in this. But look above at my other comment with a benchmark: CSV turns out far easier on resources if you have lots of points.
Back in 2017 I've made a tool that could read and write CSV, Fiona-supported formats (GeoJson, GPKG, CSV, Postgres DB), and our proprietary MongoDB. (Here's the tool, without the Mongo feature https://github.com/culebron/erde/ ) And I tried all easily available formats, and every single one has some favorable cases, and sucks at some other (well, Shapefile is outdated, so it's out of competition). Among them, FGB is kinda like better GPKG if you don't need mutations.
Re: Friends don't let friends export to CSV
#413Earlier quoted context omitted.
But now all our strings might be numbers! We now have to parse every quoted string, and we can no longer represent numbers as text. unquoted input: 0, 10, "Text", "123" unambiguous output: (Num) 0, (Num) 10, (Text) Text, (Text) 123 quoted input: "0", "10", "Text", "123" output : (Num) 0, (Num) 10, (Text) Text, (Num) 123
Why would you quote Text in the first example? That makes no sense. Text does not contain any delimiters or special characters. Unquoted input should look like this: unquoted input: 0,10,Text,123 Note the absence of spaces as well. Not sure what flavour of CSV you are using, but there usually aren't spaces after the delimiter.
Fair point, but that doesn't really resolve the issue. Here's a cleaned up example showing the same problem:
unquoted input:
0,10,"Text,","123"
output:
(Num) 0
(Num) 10
(Text) Text,
(Text) 123
quoted input:
"0","10","Text,","123"
output:
(Num) 0
(Num) 10
(Text) Text,
(Num) 123Re: Friends don't let friends export to CSV
#414Earlier quoted context omitted.
They kind of do. In my language there exists a central "governing" body that decides what is correct, and some "incorrect" regionalisms are disappearing because of it.
Are you referring to french? Because, if anything, french in France has an insane quantity of slangs and has an extremely emergent vocabulary. Much more so than any English speaking country I can think of. Quebec isn't really influenced by the Académie française yet has a much more "correct" usage of the language generally speaking. Maybe it's a totally different language but still it goes to show that even a very pr…
But maybe that's beside the point. If someone wants to "learn French" they can learn by the official rules and communicate with other french-speaking people regardless of how many slang variants exist in France. They can also probably watch French television and understand it.
The point of esperonto was to make it easier to learn. French is regular, but extremely complicated. English is complicated and has a million special cases. Both languages are hard enough to master that society starts to judge a person's intelligence by how well they know the rules and special cases.
Re: Friends don't let friends export to CSV
#415Earlier quoted context omitted.
It would stay regular if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". English really is a disaster of a language. There was a(nother) great XKCD about it just a few days ago. https://xkcd.com/2907/
Correction it's a disaster of at least 4 languages and this is probably why English is so hard to dethrone as it have no strict ownership so everyone is kind of equal in speaking it incorrectly. Sometimes lack of rigidity is actually an feature that allows for things to sort of work that would be politically impossible if thinks had to be specified formally before being used.
I'm pretty sure the reason English is hard to dethrone is because Britain ~helped~ forced the various colonies to join world commerce using English, so they started teaching it to entire generations as the national second-language, and then because the USA dominated world commerce after that in a sort of "we'll let you in on the game if you speak our language and use our money" sort of way.
Re: Friends don't let friends export to CSV
#416Earlier quoted context omitted.
I deal with gig size csvs all the time and don’t have any performance issues. These aren’t huge files, but decent sized. And most are just a few megs and only thousands to millions of records. Csv is not very performant, but it doesn’t matter for these use cases. I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my…
That may be your experience, but certainly not a universal experience (and apparently not the author's, either). In my experience, it's pretty easy to have CSVs (or Parquet files, or whatever) that are tens or hundreds of GBs in size. The space savings from a more modern file format are significant, as is the convenience of being able to specify and download/open only a subset of rows or columns over the network. Mos…
The author is giving universal advice to all friends.
If the title was “friends in certain circumstances shouldn’t let friends in certain circumstances export to csv.”
Even a laptop with 8gb ram can open a gig csv.
Of course the internals of your etl will use some efficient data structure, but you’d still want to export as csv at some point to get data to other people. Or you want your friends to export csv to get data to you.
Re: Friends don't let friends export to CSV
#417Earlier quoted context omitted.
I deal with gig size csvs all the time and don’t have any performance issues. These aren’t huge files, but decent sized. And most are just a few megs and only thousands to millions of records. Csv is not very performant, but it doesn’t matter for these use cases. I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my…
> That being said my workstation is pretty big and can handle 10s of gigs of csv before I care. How much RAM do you have? What's the ratio of [smallest CSV file which bottlenecks]/[your RAM]?
My dev workstation has 96gb. I don’t work with massive data files so I’ve never really hit my limit. I think the biggest raw data file I’ve opened was 10-20gb.
Re: Friends don't let friends export to CSV
#418Earlier quoted context omitted.
I got a little snarky but I think the analogy holds. Esperanto is a superior language to English. And English has many flaws. Theoretically it would be better to have all pilots and airports learn an efficient language. But it would be stupid and immature to seriously write a blog post about that, especially without talking about all the flaws in that plan.
> Esperanto is a superior language to English. And English has many flaws. At least one one perspective, English is superior. That perspective is that you can actually use it in almost any modern situation because it has been tried and tested globally.
This is the point of my analogy.
English:Esperanto::csv:parquet
(Although I think parquet is much more useful than Esperanto and may eventually end up dethroning csv)
Re: Friends don't let friends export to CSV
#419This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…
> all these sources aren’t able to agree on some successful format. But the same is true for csv, and they are not readable by everyone since you don't always know how to read them, there is not enough info for that Also it's not a good reflection on "deep experience" if it leads to reflexive defense of common stupid things people do with wrong analogies (e.g, flac is less efficient, so more like csv)
In my experience csv has the fewest problems. Not that it has zero problems.
Re: Friends don't let friends export to CSV
#420"You give up human readable files, but what you gain in return is..." Stop right there. You lose more than you gain. Plus, taking the data out of [proprietary software app my client's data is in] in csv is usually easy. Taking the data out in Apache Parquet is...usually impossible, but if it is possible at all you'll need to write the code for it. Loading the data into [proprietary software app my client wants data p…
Real world example of this that we just experienced: I work with a provider who offers CSV exports as the only way to access data. Recently, we found they were including unsanitized user input directly in fields. They weren't even quoting these fields. The platform "notified their quality assurance team ASAP" (like every other issue, we never heard back), but we had a deadline. This, of course, was a mess, but being…
This sounds like a problem that wouldn’t have existed in the first place if following a binary protocol with a standard format and using a proper serialization library.
The issue comes from CSV files looking easy to generate by hand, when it in fact is not.