Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SedonaDB: A new geospatial DataFrame library written in Rust

sedona.apache.org

Post reply on HN
Print view View original

31–40 of 50 posts

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#31

Post by MrPowers »

cbzbc wrote: post #25
throwmeaway222 wrote: post #24

Earlier quoted context omitted.

well for one, it's not crashing at some larger use-cases when duckdb does. according to the graph unless I'm mis-reading

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

You can generate the dataset with the instructions in this readme: https://github.com/apache/sedona-spatialbench/tree/main

Here are the queries: https://github.com/apache/sedona-spatialbench/blob/main/prin...

They should be fairly easy to replicate!

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#32

Post by cinntaile »

cbzbc wrote: post #25
throwmeaway222 wrote: post #24

Earlier quoted context omitted.

well for one, it's not crashing at some larger use-cases when duckdb does. according to the graph unless I'm mis-reading

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

Crashing when running out of memory is not acceptable software behavior in my opinion.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#33

Post by MrPowers »

dzonga wrote: post #30

is written in Rust a value add ? vs say Accessible not only via Python but first party support in Ruby, C#, Javascript ?

Rust is a good language for performant computing in general, but especially for data projects because there are so many great OSS data libraries like DataFusion and Arrow.

SedonaDB currently supports SQL, Python, R, and Rust APIs. We can support APIs for other languages in the future. That's another nice part about Rust. There are lots of libraries to expose other language bindings to Rust projects.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#34

Post by 0x9e3779b6 »

There is another great lib built on Apache Arrow - polars dataframe, which has amazing DSL.

It comes a disappointment for me that SedonaDB hasn’t adopted a similar approach.

Apache stack provides everything needed, but for small things I would not prefer SQL exactly

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#35

Post by cbzbc »

cinntaile wrote: post #32
cbzbc wrote: post #25

Earlier quoted context omitted.

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

Crashing when running out of memory is not acceptable software behavior in my opinion.

Right, but all it says is that an error was thrown.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#36

Post by cyanydeez »

cbzbc wrote: post #25
throwmeaway222 wrote: post #24

Earlier quoted context omitted.

well for one, it's not crashing at some larger use-cases when duckdb does. according to the graph unless I'm mis-reading

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

OOM are still something a DB can "avoid" so it's not like that class of bugs is some special issue that nullifies thing.

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#39

Post by ratmice »

cbzbc wrote: post #25
throwmeaway222 wrote: post #24

Earlier quoted context omitted.

well for one, it's not crashing at some larger use-cases when duckdb does. according to the graph unless I'm mis-reading

I'd like to know the details of the errors -- because it could have been as simple as running out of memory.

I doubt this hypothesis, because duckdb written in c++ should be able to tolerate memory failure, while this written in rust has to deal with rusts memory allocation failures are panic's behavior.

That is to say that if the issue is duckdb running out of memory, it is most likely because the rust implementation is using memory more efficiently for whatever query is crashing duckdb, rather than graceful handling of memory allocation failure.

Where it is possible in c++ to gracefully handle memory allocation failure, it is not really a thing in rust I'm not even sure whether it is possible to catch_unwind it. I say this as a rust person who doesn't fancy c++ in the slightest...

Re: SedonaDB: A new geospatial DataFrame library written in Rust

#40

Post by paleolimbot »

drewda wrote: post #2

Interesting, but why share the Sedona name? I thought Apache Sedona is implemented in Java/Scala for distributed runtimes like Spark and Flink. Wouldn't Rust tooling for interactive use be built atop a completely different stack?

It's built on a separate stack but conceptually it's very similar (DataFusion shares a number of idioms with Spark and has a number of projects implementing various Spark compatibility)...I think the idea was to bring the successful pieces of Sedona Spark to a wider audience.
Post reply on HN