Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB is probably the most important geospatial software of the last decade

dbreunig.com

Post reply on HN
Print view View original

131–140 of 177 posts

Re: DuckDB is probably the most important geospatial software of the last decade

#131

Post by mettamage »

serjester wrote: post #127

I think this is part of a broader trend of geospatial data just becoming easier to work with. DuckDB is great for quick ad hoc stuff, but I find polars to be easier to maintain. Personally, I'm really excited for polars to (eventually) add true geospatial support. In the meantime, creating a custom h3 plugin only took a couple days and it simplified massive parts of our old geo pandas / duckdb code. The faster we can…

Is it possible with polars to store data more efficiently than a database though?

I work with polars, but I haven't delved too deep into the performance characteristics versus postgres or anything like that.

Re: DuckDB is probably the most important geospatial software of the last decade

#134

Post by Groxx »

quasarj wrote: post #132

I've been out of the loop on DuckDB, where can I get a real overview of what the excitement is about? It just looks like a new sqlite from the CLI...

It's essentially a column-oriented sqlite, so it's much faster for some kinds of queries. There's a big market for that category of queries, and nobody wants to run their own hadoop cluster to do single-machine-scale stuff.

Re: DuckDB is probably the most important geospatial software of the last decade

#135

Post by perrygeo »

mpalmer wrote: post #124
perrygeo wrote: post #122

Why? The article is light on details. Yes, having spatial analysis combined with SQL is awesome and very natural. There's nothing special about 2D geometries that makes them significantly different from floats and strings in an RDBMS perspective - geometry is just another column type, albeit with some special operators and indexes. We've been doing it with PostGIS, Spatialite, etc for two decades at this point. What…

The argument seemed pretty clear to me; more accessible tooling means more users and contributors. But as an apparent SME I could see how you'd feel clickbaited by the title.

Mainly a point of clarification - I don't think DuckDB represents anything new in geospatial. It represents a new paradigm for data management and data architecture. Understanding the difference isn't optional.

DuckDB handles floating point numbers too - is DuckDB the most important thing in floating point data? Of course not, the data types and operators haven't changed. The underlying compute environment has. That's where the innovation is. I'd simply appreciate if technical writers took two seconds to make this vital distinction - being precise isn't hard and I don't know what we gain by publishing intentionally muddled technical articles.

Re: DuckDB is probably the most important geospatial software of the last decade

#136

Post by wenc »

tmpz22 wrote: post #108
wenc wrote: post #46

Earlier quoted context omitted.

How is the adoption among DevOps folks? I'm still getting feedback that many devs are not too comfortable with reading and writing SQL. They learned simple SELECT statements in school, but get confused by JOINs and GROUP BYs.

It’s not a good database for many storage workloads because it only allows a single process to write to it. For example you wouldn’t want 20 app servers writing their logs or metrics or traces to a single duck db instance. You would instead have them write rotating log files that are somehow aggregated then queried through duck db via something like grafana. So the operational savings are more for using it as a light…

DuckDB can be thought of as an analytics engine rather than storage engine. It’s OLAP rather than OLTP.

Re: DuckDB is probably the most important geospatial software of the last decade

#137

Post by wenc »

wodenokoto wrote: post #85
wenc wrote: post #8

I'm a big fan of DuckDB and I do geospatial analysis, mostly around partitioning geographies (into Uber H3 hexagons), calculating Haversine distances, calculating areas of geometries, figuring out which geometry a point falls in, etc. Many of these features have existed in some form or other in geopandas or postgis, so DuckDB's spatial extensions bring nothing new. But what DuckDB as an engine does is it lets me work…

Why do you use haver-sine over geodesic or reprojection? I’ve been doing the reprojection thing, projecting coordinates to a “local” CRS, for previous projects mainly because that’s what geopandas recommend and is built around, but I am reaching a stage where I’d like to calculate distance for objects all over the globe, and I’m genuinely interested to learn what’s a good choice here.

Reprojection is accurate locally but inaccurate at scale.

Geodesics are the most accurate (Vincenty etc) but are computationally heavy.

Haversine is a nice middle ground.

Re: DuckDB is probably the most important geospatial software of the last decade

#138

Post by paradox460 »

Demiurge wrote: post #44

> Prior to this, getting up and running from a cold-start might’ve required installing or even compiling severall OSS packages, carefully noting path locations, standing up a specialized database… Enough work that a data generalist might not have bothered, or their IT department might not have supported it. I've been able to "CREATE EXTENSION postgis;" for more than a decade. There have been spatial extensions for PG…

The original article feels a tremendous amount like another piece of DuckDB marketing, from the breathless admiration to the baseless claims like the title

Re: DuckDB is probably the most important geospatial software of the last decade

#140

Post by kianN »

elchief wrote: post #139

I wish to god they would allow 2 connections. One read-only for my BI tool, and one read-write for dbt/sqlmesh

I think using a pyarrow dataset as an intermediary would allow for a zero copy read/write from one connection into a second connection.
Post reply on HN