Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB is probably the most important geospatial software of the last decade

dbreunig.com

Post reply on HN
Print view View original

91–100 of 177 posts

Re: DuckDB is probably the most important geospatial software of the last decade

#91

Post by indeyets »

wenc wrote: post #83
vasco wrote: post #76

Earlier quoted context omitted.

I haven't yet understood this pattern (and I tried using duckdb). Unless you're only ever going to query those files once or twice in your life, importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB. Also as a side note, is everyone just using DuckDB in memory? Because as soon as you want some multiple session stuff I'd assume you'd use DuckDB on top of a local data…

> importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB. Usually new data is generated regularly and would require creating a separate ETL process to ingest into Postgres. With DuckDB, no ETL is needed. New Parquet files are just read off the disk. > Also as a side note, is everyone just using DuckDB in memory? DuckDB is generally used as a single-user, and yes in-m…

> Usually new data is generated regularly

This part was not obvious. In a lot of cases geodata is mostly stable and reads/searches dominate over appends. And that’s why we keep this in DB (usually postgis, yes).

So DuckDB is optimised for very different use case and it is not always obvious when it’s mentioned

Re: DuckDB is probably the most important geospatial software of the last decade

#92

Post by Amadiro »

maxxen wrote: post #28
larsiusprime wrote: post #6

“import geopandas” also exists and has for some time. Snark aside, WHAT is special about duckDB? I wish the author had actually shown some practical examples so I could understand their claims better.

I replied to another comment, but I think a big part is that duckdbs spatial extension provides a SQL interface to a whole suite of standard foss gis packages by statically bundling everything (including inlining the default PROJ database of coordinate projection systems into the binary) and providing it for multiple platforms (including WASM). I.E there are no transitive dependencies except libc. Yes, DuckDB does a…

Is there any strong reason to use GeoParquet instead of straight up parquet if all I'm interested in is storing and operating on lat/lons?

I'm curious if it compresses them better or something like that. I see lots of people online saying it compresses well (but mostly compared to .shp or similar) but normal parquet (.gz.parquet or .snappy.parquet) already does that really well. So it's not clear to me if I should spend time investigating it...

I mostly process normal parquet with spark and sometimes clickhouse right now.

Re: DuckDB is probably the most important geospatial software of the last decade

#93

Post by debugnik »

Demiurge wrote: post #45
dbreunig wrote: post #22

Earlier quoted context omitted.

Yes. The difference between provisioning a server and running 'install spatial' in a CLI is night and day. Docker has been a big improvement (when I was first learning PostGIS, the amount of time I had to hunt for proj directories or compile software just to install the plugin was a major hurdle), but it's many steps away from: ``` $ duckdb D install spatial; ```

What do you mean by "provisioning a server"? That's a strange requirement. You can install Postgis on a macbook in one command, or actually on all 3 major OS's in one command: "brew install postgis", "apt-get install postgresql-postgis, and "choco install postgis-9.3". Does DuckDB not require a "server" or a "computer"? What does Docker have to do with anything? This is a very confusing train of thought.

> Does DuckDB not require a "server"

No, it works like SQLite.

Re: DuckDB is probably the most important geospatial software of the last decade

#94

Post by willtemperley »

I have some concerns regarding licensing of DuckDB and GEOS which DuckDB spatial depends on. The former is MIT licensed and the latter LGPL 2.1.

This leads to some complex situations where some builds would contravene LGPL 2.1 e.g static linking with a closed source application.

Re: DuckDB is probably the most important geospatial software of the last decade

#95

Post by vasco »

wenc wrote: post #84
vasco wrote: post #77

Earlier quoted context omitted.

There's no point in learning any much deeper SQL anymore, AI assistants have largely solved SQL querying. Just ask for what you want with natural language.

I don't quite agree. SQL queries are one area where correctness matters a lot, because downstream applications rely on them to be correct. If you get a query wrong, especially in an ETL process, you can generate a lots of garbage data for a very long time (I'm speaking from lived experience). It might take a long time to correct (via backfill) and sometimes the original data might no longer be available. I use LLMs t…

Testing SQL for correctness is besides the point of how the SQL was generated. It's important of course to review and test anything. My point is SQL writing is particularly good already with current state of the art of assistants for it to be worth it to spend much time hand crafting complex queries.

Re: DuckDB is probably the most important geospatial software of the last decade

#96

Post by simlevesque »

indeyets wrote: post #91
wenc wrote: post #83

Earlier quoted context omitted.

> importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB. Usually new data is generated regularly and would require creating a separate ETL process to ingest into Postgres. With DuckDB, no ETL is needed. New Parquet files are just read off the disk. > Also as a side note, is everyone just using DuckDB in memory? DuckDB is generally used as a single-user, and yes in-m…

> Usually new data is generated regularly This part was not obvious. In a lot of cases geodata is mostly stable and reads/searches dominate over appends. And that’s why we keep this in DB (usually postgis, yes). So DuckDB is optimised for very different use case and it is not always obvious when it’s mentioned

But DuckDB works just as well with static data.

Re: DuckDB is probably the most important geospatial software of the last decade

#97

Post by kriro »

That's a pretty grandiose statement and frankly the kind of advertisement I'm not a fan of at all.

If you want to import something and work with it GeoPandas exists. If you want something integrated with a SQL database, PostGIS exists.

On the application side of the spectrum, GRASS GIS, QGIS etc. say hi. They are being used in the agriculture industry and by government agencies (at least I know that's the case in Germany and Brazil).

Re: DuckDB is probably the most important geospatial software of the last decade

#98

Post by touisteur »

wenc wrote: post #47
dopidopHN wrote: post #40

Earlier quoted context omitted.

I’m very familiar with Postgres and spinning one with postgis seems easy enough. Do I get more with duckdb? Most of the time I store locations and compute distance to them. Would that being faster to implement with duckdb

Probably no difference for your use-case (ST_Distance). If you already have data in Postgres, you should continue using Postgis. In my use case, I use DuckDB because of speed at scale. I have 600GBs of lat-longs in Parquet files on disk. If I wanted to use Postgis, I would have to ingest all this data into Postgres first. With DuckDB, I can literally drop into a Jupyter notebook, and do this in under 10 seconds, and…

And now I'm curious whether there's a way to actually index external files (make these queries over 600GB faster) and have this index (or many indices) be persistent. I might have missed that when I looked at the docs...

Re: DuckDB is probably the most important geospatial software of the last decade

#99

Post by kashifr »

jparishy wrote: post #14

I work on geospatial apps and the software I think I am most excited about is https://felt.com/ . I want to see them expand their tooling such that maps and data source authentication/authorization was controllable by the developer, to enable tenant isolation with proprietary data access. They could really disrupt how geospatial tech gets integrated into consumer apps. This article doesn't acknowledge how niche this…

Have you tried https://geobase.app/ they recently also had a post about duckdb integration: https://geobase.app/blog/duckdb-1-1-3

Re: DuckDB is probably the most important geospatial software of the last decade

#100

Post by jokoon »

Demiurge wrote: post #44

> Prior to this, getting up and running from a cold-start might’ve required installing or even compiling severall OSS packages, carefully noting path locations, standing up a specialized database… Enough work that a data generalist might not have bothered, or their IT department might not have supported it. I've been able to "CREATE EXTENSION postgis;" for more than a decade. There have been spatial extensions for PG…

I tested spatialite, it works okay, but the setup is a bit tedious when inserting data.
Post reply on HN