How does this handle sort ordering / collations?
Sort ordering is handled automatically by Datafusion.
Collations at the column/operation level are not yet supported but we're working on it.
Transforming Postgres into a Fast OLAP Database
21–30 of 62 posts
Re: Transforming Postgres into a Fast OLAP Database
#22Nice benchmark with amazing results! Congrats!
Re: Transforming Postgres into a Fast OLAP Database
#23Interesting stuff keeps happening in PostgresLand. This uses Apache DataFusion, first think i hear about it, but from reading the docs it seems like it already offers build in SQL interface atop of Apache parquet and Apache Arrow. What is the difference between this two versions of SQL. Can I still use my favourite pl/sql function together with a deltalake table and does this play nice with other kind of tables?
Re: Transforming Postgres into a Fast OLAP Database
#24If I combined ParadeDB with a common postgres extension like PostGIS, would I get the benefits of both?
Yes and no, depending on what you mean.
The custom types/indexes introduced by PostGIS won't work with deltalake tables. Even if it were possible, the benefits of using deltalake tables to execute geospatical queries are unclear, since PostGIS indexes are already optimized for this task.
But, what ParadeDB enables is for geospatial tables and deltalake tables to exist within the same database.
Re: Transforming Postgres into a Fast OLAP Database
#25Interesting stuff keeps happening in PostgresLand. This uses Apache DataFusion, first think i hear about it, but from reading the docs it seems like it already offers build in SQL interface atop of Apache parquet and Apache Arrow. What is the difference between this two versions of SQL. Can I still use my favourite pl/sql function together with a deltalake table and does this play nice with other kind of tables?
Datafusion's SQL dialect has some slight quirks that pertain to Datafusion-specific functionality. For instance, the ability to create an external table.
With regards to ParadeDB, we rely on the Datafusion SQL parser, which can transform the Postgres SQL dialect into a Datafusion logical plan that can be executed by Datafusion. We actually have an open PR that adds support for user-defined functions...it will likely get merged within a few days.
Re: Transforming Postgres into a Fast OLAP Database
#26Slightly related but it's so interesting to me all those new small tech companies that basically provides value on top of Postgres! They usually blog about some obscure stuff about Postgres and one of their articles gets on the frontpage of HN. Good marketing and very cool stuff!
Re: Transforming Postgres into a Fast OLAP Database
#27Interesting stuff keeps happening in PostgresLand. This uses Apache DataFusion, first think i hear about it, but from reading the docs it seems like it already offers build in SQL interface atop of Apache parquet and Apache Arrow. What is the difference between this two versions of SQL. Can I still use my favourite pl/sql function together with a deltalake table and does this play nice with other kind of tables?
Datafusion's SQL dialect has some slight quirks that pertain to Datafusion-specific functionality. For instance, the ability to create an external table. With regards to ParadeDB, we rely on the Datafusion SQL parser, which can transform the Postgres SQL dialect into a Datafusion logical plan that can be executed by Datafusion. We actually have an open PR that adds support for user-defined functions...it will likely…
thx for your reply, so postgres is taking care of the parquet files in the background in some way i guess, would it be possible to combine this with something like neon?
Re: Transforming Postgres into a Fast OLAP Database
#28Excited to see ParadeDB make progress in this area :)
Re: Transforming Postgres into a Fast OLAP Database
#29OLAP was my favorite character in Frozen
Re: Transforming Postgres into a Fast OLAP Database
#30Thanks for sharing, I really like Postgres. However, I generally use Postgres for OLTP work. I would like to point out two things: 1. Based on Clickbench results, pg_analytics is still far from top-tier performance. If you're looking for a high-performance OLAP database, you should consider the top-ranked products. 2. The queries tested by Clickbench are really simple, far from real data analysis scenarios. You must…
Clickbench is really nice because it is so easy to compare and contribute benchmarks. Is there anything out there like Clickbench, but for the TPC-DS and TPC-H benchmarks?