I hope the poor devs that built this wernt subjected to the brutal 996 culture (9am-9pm, 6 days per week)
AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
21–30 of 58 posts
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#22On a drive-by-glance it looks like if you had a tighter integrated version of PSQL FDW for DuckDB and Vector Storage - meets Vespa. I find it interesting they went with extending MySQL instead of FDW route on PSQL?
probably they had millions of lines of code already using mysql
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#23having an embedded column database for analytics in your traditional db is a massive win for productivity + operations simplicity. at the moment I use PG + Tiger Data - couldn't find a mysql equivalent so this as one.
Mariadb has a columnar engine already (though I did not use it myself) https://mariadb.com/docs/analytics/mariadb-columnstore/colum... and is mostly mysql compatible. For about a year releases include a vector storage type, so it will be interesting to see it compared in performance with what Alibaba did. Just wanted to plug that out. Given how often Postgres is plugged on HN, I think people ignore how versatile mari…
This ColumnStore is very simple and just do table scans sequentially on every query. It doesn't support indexes and unique constraints. It is almost an append-only serialization file format, but with some columnar concepts.
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#24having an embedded column database for analytics in your traditional db is a massive win for productivity + operations simplicity. at the moment I use PG + Tiger Data - couldn't find a mysql equivalent so this as one.
MariaDB has supported columnar tables for a bit https://mariadb.com/resources/blog/see-columnar-storage-for-...
I don't think MariaDB ColumnStore has any kind of advantage. It is just an append-only storage format with some columnar concepts.
https://vettabase.com/mariadb-columnstore-sql-limitations/#I...
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#25I hope the poor devs that built this wernt subjected to the brutal 996 culture (9am-9pm, 6 days per week)
[deleted]
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#26[flagged]
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#27[dead]
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#28[dead]
On this page, we introduce how to implement a read-only Columnar Store (DuckDB) node leveraging the MySQL binlog mechanism. https://github.com/alibaba/AliSQL/blob/master/wiki/duckdb/du... In this implementation, we have performed extensive optimizations for binlog batch transmission, write operations, and more.
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#29Curious how it stacks up to pg_duckdb. (pg_duckdb seems pretty clean, due to Postres' powerful extension mechanisms)
Here is the professional English translation of your analysis, optimized for a technical audience or a blog post:
Why I Believe MySQL is More Suited than PostgreSQL for DuckDB Integration Currently, there are three mainstream solutions in the ecosystem: pg_duckdb, pg_mooncake, and pg_lake. However, they face several critical hurdles. First, PostgreSQL's logical replication is not mature enough—falling far behind the robustness of its physical replication—making it difficult to reliably connect a PG primary node to a DuckDB read-only replica via logical streams.
Furthermore, PostgreSQL lacks a truly mature pluggable storage engine architecture. While it provides the Table Access Method as an interface, it does not offer standardized support for primary-replica replication or Crash Recovery at the interface level. This makes it challenging to guarantee data consistency in many production scenarios.
MySQL, however, solves these issues elegantly:
Native Pluggable Architecture: MySQL was born with a pluggable storage engine design. Historically, MySQL pivoted from MyISAM to InnoDB as the default engine specifically to leverage InnoDB's row-level MVCC. While previous columnar attempts like InfoBright existed, they didn't reach mass adoption. Adding DuckDB as a native columnar engine in MySQL is a natural progression. It eliminates the need for "workaround" architectures seen in PostgreSQL, where data must first be written to a row-store before being converted into a columnar format.
The Power of the Binlog Ecosystem: MySQL’s "dual-log" mechanism (Binlog and Redo Log) is a double-edged sword; while it impacts raw write performance, the Binlog provides unparalleled support for the broader data ecosystem. By providing a clean stream of data changes, it facilitates seamless replication to downstream systems. This is precisely why OLAP solutions like ClickHouse, StarRocks, and SelectDB have flourished within the MySQL ecosystem.
Seamless HTAP Integration: When using DuckDB as a MySQL storage engine, the Binlog ecosystem remains fully compatible and intact. This allows the system to function as a data warehouse node that can still "egress" its own Binlog. In an HTAP (Hybrid Transactional/Analytical Processing) scenario, a primary MySQL node using InnoDB can stream Binlog directly to a downstream MySQL node using the DuckDB engine, achieving a perfectly compatible and fluid data pipeline.
Re: AliSQL: Alibaba's open-source MySQL with vector and DuckDB engines
#30[dead]
Nice question! We did spend a lot of time considering the issue of data consistency.
In the MySQL replication, GTID is crucial for ensuring that no transaction is missed or replayed repeatedly. We handle this in two scenarios (depending on whether binlog is enabled):
- log_bin is OFF: We ensure that transaction in DuckDB are committed before the GTID is written to disk (in the mysql.gtid_executed table). Furthermore, after a crash recovery, we perform idempotent writes to DuckDB for a period of time (the principle is similar to upsert or delete+insert). Therefore, at any given moment after a crash recovery, we can guarantee that the data in DuckDB is consistent with the primary database.
- log_bin is ON: Unlike the previous scenario, we no longer rely on the `mysql.gtid_executed` table; we directly use the Binlog for GTID persistence. However, a new problem arises: Binlog persistence occurs before the Storage Engine commits. Therefore, we created a table in DuckDB to record the valid Binlog position. If the DuckDB transaction fails to commit, the Binlog will be truncated to the last valid position. This ensures that the data in DuckDB is consistent with the contents of the Binlog.