Live data from Hacker News

HTAP is Dead

mooncake.dev

21–30 of 95 posts

Re: HTAP is Dead

#21
post #18

Earlier quoted context omitted.

Oh God people are still using Mongo in production? Why? Kafka exists but is deeply obsolete and mostly marginalized outside of things with dependencies on the weird way it works (Debezium, etc) I've always liked Redis but choosing it as a core tech on a new product in the last, say, 6 years is basically malpractice? 10 if you're uncharitable. The thing these all have in common is having their economics and ergonomics…

What's the better kafka/redis? Mongo I know you can just use your favorite relational tool with JSON support if needed (PG/MYSQL)

> What's the better kafka/redis?

If you are going to leverage caching I’d use the OSS Valkey over Redis. Based on the company’s past behavior, Redis is dead to me now.

Re: HTAP is Dead

#22

From a modern startup’s POV - fast pivots, fast feedback - it’s fair to say HTAP is “dead.” The market is sticky and slow-moving. But I’d argue that’s precisely why it’s still interesting: fewer teams can survive the long game, but the payoff can be disproportionate.

I agree the opportunity is still there, although the long game keeps getting longer.

Prof. Viktor Leis suggested [0] that SQL itself - being so complex to implement and so ineffectively standardized - may be the biggest inhibitor to faster experimentation in the field of database startups. It's a shame there's no clear path to solving that problem directly.

[0] https://www.juxt.pro/blog/sane-query-languages-podcast/

Re: HTAP is Dead

#23
post #18

Earlier quoted context omitted.

Oh God people are still using Mongo in production? Why? Kafka exists but is deeply obsolete and mostly marginalized outside of things with dependencies on the weird way it works (Debezium, etc) I've always liked Redis but choosing it as a core tech on a new product in the last, say, 6 years is basically malpractice? 10 if you're uncharitable. The thing these all have in common is having their economics and ergonomics…

What's the better kafka/redis? Mongo I know you can just use your favorite relational tool with JSON support if needed (PG/MYSQL)

If you're already built around Redis I'd just keep using it, but if you're doing new development there's not so much a single drop in replacement as a substantially better alternative for any given feature (and not particularly any advantage to having all your data in the "same Redis instance"). That said, 90+% of the time the answer is probably "transactional SQL database" or "message queue"

For Kafka, the answer is probably an object store, a message queue, a specialized logging system, an ordinary transactional database table, or whatever mechanism your chosen analytics DB uses for bulk input (probably S3 or equivalent these days). Or maybe just a REST interface in front of a filesystem. Unless of course you truly need to interface with a Kafka consumer/producer in which case you’re stuck with it (the actual reason I've seen for every Kafka deployment I've personally witnessed in recent history)

Re: HTAP is Dead

#24
I would say compute and storage separation is the way to go, especially for hyperscaler offering ala aurora db/cosmos/alloy. And later more opensource alternatives will catch up.

Re: HTAP is Dead

#25
On the data warehousing side, I think the story looks like this:

1) Cloud data warehouses like Redshift, Snowflake, and BigQuery proved to be quite good at handling very large datasets (petabytes) with very fast querying.

2) Customers of these proprietary solutions didn't want to be locked in. So many are drifting toward Iceberg tables on top of Parquet (columnar) data files.

Another "hidden" motive here is that Cloud object stores give you regional (multi-zonal) redundancy without having to pay extra inter-zonal fees. An OLTP database would likely have to pay this cost, as it likely won't be based purely on object stores - it'll need a fast durable medium (disk), if at least for the WAL or the hot pages. So here we see the topology of Cloud object stores being another reason forcing the split between OLTP and OLAP.

But how does this new world of open OLTP/OLAP technologies look like? Pretty complicated.

1) You'd probably run PostGres as your OLTP DB, as it's the default these days and scales quite well.

2) You'd set up an Iceberg/Parquet system for OLAP, probably on Cloud object stores.

3) Now you need to stream the changes from PostGres to Iceberg/Parquet. The canonical OSS way to do this is to set up a Kafka cluster with Kafka Connect. You use the Debezium CDC connector for Postgres to pull deltas, then write to Iceberg/Parquet using the Iceberg sink connector. This incurs extra compute, memory, network, and disk.

There's so many moving parts here. The ideal is likely a direct Postgres->Iceberg write flow built-into PostGres. The pg_mooncake this company is offering also adds DuckDB-based querying, but that's likely not necessary if you plan to use Iceberg-compatible querying engines anyway.

Ideally, you have one plugin for purely streaming PostGres writes to Iceberg with some defined lag. That would cut out the third bullet above.

Re: HTAP is Dead

#26
post #18

Earlier quoted context omitted.

wait we're not in the nosql era anymore? dynamo and mongo are huge, redis and kafka (and their clones) are ubiquitous, etc etc

Oh God people are still using Mongo in production? Why? Kafka exists but is deeply obsolete and mostly marginalized outside of things with dependencies on the weird way it works (Debezium, etc) I've always liked Redis but choosing it as a core tech on a new product in the last, say, 6 years is basically malpractice? 10 if you're uncharitable. The thing these all have in common is having their economics and ergonomics…

I work for a database company and of my ~100 customer meetings last year, only one of the notes mentions Mongo as software they use in production. Maybe it’s a different world or something, idk, but I don’t understand the use case.

If I’m ingesting unstructured data for search or “parse it later” purposes, I’ll choose OpenSearch (elastic). Otherwise I’m going PG by default and if I need analytics I’ll use Parquet or Delta and pick the query engine based on requirements.

I honestly cannot think of a use case where Mongo is the appropriate solution.

Re: HTAP is Dead

#27
post #25

On the data warehousing side, I think the story looks like this: 1) Cloud data warehouses like Redshift, Snowflake, and BigQuery proved to be quite good at handling very large datasets (petabytes) with very fast querying. 2) Customers of these proprietary solutions didn't want to be locked in. So many are drifting toward Iceberg tables on top of Parquet (columnar) data files. Another "hidden" motive here is that Clou…

totally agreed on 3. You're also missing the challenges of dealing with updates/deletes; and managing the many small files.

CDC from OLTP to Iceberg is extremely non-trivial.

Re: HTAP is Dead

#28
post #10

Earlier quoted context omitted.

In the nosql era the idea that you could run even the basics for a >1m user SaaS platform on an ordinary, free, single-node transactional SQL database would have been considered nuts.

wait we're not in the nosql era anymore? dynamo and mongo are huge, redis and kafka (and their clones) are ubiquitous, etc etc

We’re not in the no-sql era anymore, because the prevailing marketing and “thought leadership” isn’t peaking these things _instead of_ a sql database. They’re now _parts_ of a system, of which SQL DB’s are still a very big part.

Re: HTAP is Dead

#29
post #25

On the data warehousing side, I think the story looks like this: 1) Cloud data warehouses like Redshift, Snowflake, and BigQuery proved to be quite good at handling very large datasets (petabytes) with very fast querying. 2) Customers of these proprietary solutions didn't want to be locked in. So many are drifting toward Iceberg tables on top of Parquet (columnar) data files. Another "hidden" motive here is that Clou…

totally agreed on 3. You're also missing the challenges of dealing with updates/deletes; and managing the many small files. CDC from OLTP to Iceberg is extremely non-trivial.

The small writes problem that Iceberg has is totally silly. They spend so much effort requiring a tree of metadata files, but you still need an ACID DB to manage the pointer to the latest tree. At that point, why not just move all that metadata to the DB itself? It’s not sooo massive in scale.

The current Iceberg architecture requires table reads to do so many small reads, of the files in the metadata tree.

The brand new DuckLake post makes all this clear.

https://duckdb.org/2025/05/27/ducklake.html

Still Iceberg will probably do just fine because every data warehousing vendor is adding support for it. Worse is better.

Re: HTAP is Dead

#30
post #24

I would say compute and storage separation is the way to go, especially for hyperscaler offering ala aurora db/cosmos/alloy. And later more opensource alternatives will catch up.

Most analytics workloads are bandwidth-bound if you are optimizing them at all. The major issue with disaggregated storage is that the storage bandwidth is terrible in the cloud. I can buy a server from Dell with 10x the usable storage bandwidth of the fastest environments in AWS and that will be reflected in workload performance. The lack of usable bandwidth even on huge instance types means most of that compute and memory is not doing much — you are forced to buy compute you don’t need to access mediocre bandwidth of which there is never enough. The economics are poor as a result.

This is an architectural decision of the cloud providers to some extent. Linux can drive well over 1 Tbps of direct-attached storage bandwidth on a modern server but that bandwidth is largely beyond the limits of cheap off-the-shelf networking that disaggregated storage is often running over.

Post reply on HN