Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SedonaDB: A new geospatial DataFrame library written in Rust

sedona.apache.org

Post reply on HN
Print view View original

41–50 of 50 posts

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#41

Post by vlovich123 »

ratmice wrote: post #39
cbzbc wrote: post #25

Earlier quoted context omitted.

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

I doubt this hypothesis, because duckdb written in c++ should be able to tolerate memory failure, while this written in rust has to deal with rusts memory allocation failures are panic's behavior. That is to say that if the issue is duckdb running out of memory, it is most likely because the rust implementation is using memory more efficiently for whatever query is crashing duckdb, rather than graceful handling of me…

I’ve never seen anyone try to catch allocation failures in C++ code and in many cases doing so correctly is very difficult, not least of which is that writing exception-safe code is the exception, not the rule.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#42

Post by paleolimbot »

0x9e3779b6 wrote: post #34

There is another great lib built on Apache Arrow - polars dataframe, which has amazing DSL. It comes a disappointment for me that SedonaDB hasn’t adopted a similar approach. Apache stack provides everything needed, but for small things I would not prefer SQL exactly

Agreed that the polars interface is far superior to SQL! There are a few ways to do this if there's interest...polars wasn't an option because we needed Arrow extension types (https://github.com/pola-rs/polars/issues/9112).

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#43

Post by deschutes »

ratmice wrote: post #39
cbzbc wrote: post #25

Earlier quoted context omitted.

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

I doubt this hypothesis, because duckdb written in c++ should be able to tolerate memory failure, while this written in rust has to deal with rusts memory allocation failures are panic's behavior. That is to say that if the issue is duckdb running out of memory, it is most likely because the rust implementation is using memory more efficiently for whatever query is crashing duckdb, rather than graceful handling of me…

You cannot use the rust standard library in environments where arbitrary allocations may fail but neither can you use the STL. The difference is the rust standard library doesn't pretend that it has some reasonable way to deal with allocation failure. std::bad_alloc is mainly a parlor trick used to manufacture the idea that copy and move fallibility are reasonable things.

I wouldn't wager a nickel on someone's life if it depended on embedded STL usage.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#44

Post by dr-jia-yu »

itsthecourier wrote: post #38

PostGIS not being included in benchmarks got me suspicious

You’re absolutely asking the right question. As we noted in Future Work section of the SpatialBench result (https://sedona.apache.org/spatialbench/single-node-benchmark...), this benchmark is focused on geospatial analytical queries. For these workloads, features like columnar layout, vectorized execution, zero-copy, and zero SerDe provide huge performance benefits.

While PostGIS is often used for spatial analytics because of its rich spatial function coverage, it is fundamentally a transactional database. This design makes it less suited for analytical query performance, and including it directly in SpatialBench would risk claims of being an “apples-to-oranges” comparison. That’s why we exclude PostGIS from the published benchmark results.

That said, we do continuously validate against PostGIS. For every single function in SedonaDB, we maintain an automated PyTest benchmark framework (https://github.com/apache/sedona-db/tree/main/benchmarks) that compares both speed and correctness against DuckDB and PostGIS. This ensures we catch regressions early and guarantees correctness. You can even run these benchmarks yourself to see how SedonaDB performs. It is often extremely fast in practice.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#45

Post by jinjin2 »

ZeroCool2u wrote: post #22

Everyone asking why this exists when DuckDB or PostGIS or the JVM based Sedona already exists, clearly has not run into the painful experience of working on these large geospatial workloads when the legacy options are either not viable or not an option for other reasons, which happens more often than you might expect! And the CRS awareness!!! Incredible! This is such a huge source of error when you throw folks that a…

I usually start with PostGIS for single-node workloads and then switch to Exasol when I get to truly massive datasets (Exasol has a more limited set of spatial operators, but scales effortlessly across multiple nodes).

It will be great with some more options in this space, especially if it makes a smooth transition from single-node/local interactions to multi-node scale-out.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#48

Post by wichert »

larodi wrote: post #9

Somehow I dont see this applicable for 90% of all current spatial needs, where PostGIS does just right, and same IMHO goes for DuckDB. There perhaps exists 10% of business where data is so immense you want to hit it with Rust & whatnot, but all others do just fine im Postgre. My bet is most of actually useful spatial ST_ functions are not implemented in this one, as they are not in the DuckDB offering.

Not having a dependency on a running service can be an excellent reason to use tools like DuckDB.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#49

Post by larodi »

mattforrest wrote: post #14
larodi wrote: post #9

Somehow I dont see this applicable for 90% of all current spatial needs, where PostGIS does just right, and same IMHO goes for DuckDB. There perhaps exists 10% of business where data is so immense you want to hit it with Rust & whatnot, but all others do just fine im Postgre. My bet is most of actually useful spatial ST_ functions are not implemented in this one, as they are not in the DuckDB offering.

I wrote a book on PostGIS and used it for years and these single node analytical tools make sense when PostGIS performance starts to break down. For many tasks PostGIS works great, but again you are limited by the fact that your tables have to live in the DB and can only scale as much as the computing resources you have allocated. In terms of number of functions PostGIS is still the leader, but for analytical functio…

Not saying these shouldn't be used together, but even then, increased complexity will pay only in very limited scenarios. The generic SQLite can perhaps handle 80% of all wordpress needs.

Postgres made gigantic leaps in recent years - both in performance and feature-set. I don't think ever comparing the new contenders with daddy is fair. But then there are the DuckDB advocates who claim it pioneered spatial, which is so much not true.

Postgres is amazing system, which is also available free. We don;t have too many of these, and too many aging that well.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#50

Post by eptcyka »

ratmice wrote: post #39
cbzbc wrote: post #25

Earlier quoted context omitted.

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

I doubt this hypothesis, because duckdb written in c++ should be able to tolerate memory failure, while this written in rust has to deal with rusts memory allocation failures are panic's behavior. That is to say that if the issue is duckdb running out of memory, it is most likely because the rust implementation is using memory more efficiently for whatever query is crashing duckdb, rather than graceful handling of me…

There's an effort to expose allocation errors in the standard library for the Linux kernel. Pretty sure it is well under way.
Post reply on HN