Viewing profile — MrPowers

Rust is a good language for performant computing in general, but especially for data projects because there are so many great OSS data libraries like DataFusion and Arrow. SedonaDB…

You can generate the dataset with the instructions in this readme: https://github.com/apache/sedona-spatialbench/tree/main Here are the queries: https://github.com/apache/sedona-sp…

The "DuckDB is probably the most important geospatial software of the last decade" post has a nice related discussion: https://news.ycombinator.com/item?id=43881468

There is a project called GeoPolars: https://github.com/geopolars/geopolars From the README: > Update (August 2024): GeoPolars is blocked on Polars supporting Arrow extension types…

SedonaDB builds on libraries in the Rust ecosystem, like Apache DataFusion, to provide users with a nice geospatial DataFrame experience. It has functions like ST_Intersects that a…

IMO, it would have been better to donate the repos to a shared org and motivate the community to continue maintaining them. But pretty awesome this individual is retiring from prog…

> A Data Lakehouse is fine but what benefit does it give you over a much more simple solution of ETL/ELTing the data in batches (weekly, daily, hourly, etc) and letting it sit in s…

Lots of Spark workloads are executed with the C++ Photon engine on the Databricks platform, so we ironically have partially moved back to C++. Disclosure: I work for Databricks.

The OP is the original creator of Ballista, so he's well aware of the project. Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark fa…

I love Medellin and lived there for many years, but the air quality is terrible and getting worse. You can talk with any locals and they say that the climate is noticeably differen…

I work at Databricks, but am pretty much just an OSS nerd, mainly focusing on Delta Rust recently: https://github.com/delta-io/delta-rs I did some keyword research and wrote this p…

Delta Live Tables are a Databricks feature and aren't related to Delta Lake. Can't you just setup a cron job to vacuum periodically?

Yea, comparing Delta Lake to Iceberg is more apt, but I've been shying away from that content cause I don't wanna flamewar. Another poster is asking for this post tho, so maybe I s…

Lots of Parquet files in the same directory are typically referred to as a "Parquet table". Yes, Parquet can be compressed with zip, but snappy is much more common because it's spl…

Yea, that's exactly what Delta Lake does. All the table metadata is stored in a Parquet file (it's initially stored in JSON files, but eventually compacted into Parquet files). The…

Yea, it is fair feedback. I respect the Iceberg team & their work. I've been shying away from that post cause I don't wanna start a flamewar, but I will reflect on this and reconsi…