My takeaway about all this is that nobody really cares much about consistency or the cost to build and run lambda-like architectures.
HTAP is Dead
41–50 of 95 posts
Re: HTAP is Dead
#42The 2nd last line is the summary -
"The HTAP challenge of our time comes down to making the lakehouse real-time ready."
We are building this platform as well. There are 2 aspects to it - the "enterprise way" and the "greenfield way". The greenfield way will win out in 10-15 years, but unless you have capital to last that long, as a startup we need to go the Enterprise way first until we are big enough to go the unified HTAP-style way. The Lakehouse - open columnar data - is here to stay. It needs a better connection to OLTP than Kafka, but it will take time between A and B.
Re: HTAP is Dead
#43Earlier quoted context omitted.
I thought it was the weakest point. The need for a distributed DB is rarely performance, it's availability and durability.
But you can get more availability and more durability with much easier alternatives: - Availability: spin up more read replicas. - Durability: spin up more read replicas and also write to S3 asynchronously. With Postgres on Neon, you can have both of these very easily. Same with Aurora. (Disclaimer: I work at Neon)
This doesn’t seem to provide higher write availability, and if the read replicas are consistent with the write replica this design must surely degrade write availability as it improves read availability, since the write replica must update all the read replicas.
This also doesn’t appear to describe a higher durability design at all by normal definitions (in the context of databases at least) if it’s async…?
Re: HTAP is Dead
#44I found the title amusing. This died.. right after inception.
Clearly, the objectives and limitations of OLAP and OLTP differ so much that merging the two domains in a fantasy.
It's like asking two people to view through the same lens.
Re: HTAP is Dead
#45Earlier quoted context omitted.
But you can get more availability and more durability with much easier alternatives: - Availability: spin up more read replicas. - Durability: spin up more read replicas and also write to S3 asynchronously. With Postgres on Neon, you can have both of these very easily. Same with Aurora. (Disclaimer: I work at Neon)
This doesn’t seem to provide higher write availability, and if the read replicas are consistent with the write replica this design must surely degrade write availability as it improves read availability, since the write replica must update all the read replicas. This also doesn’t appear to describe a higher durability design at all by normal definitions (in the context of databases at least) if it’s async…?
Yeah, this is not about write availability, but as the OP/author points out, scaling that is not the bottleneck for most apps.
Re: HTAP is Dead
#46Earlier quoted context omitted.
This doesn’t seem to provide higher write availability, and if the read replicas are consistent with the write replica this design must surely degrade write availability as it improves read availability, since the write replica must update all the read replicas. This also doesn’t appear to describe a higher durability design at all by normal definitions (in the context of databases at least) if it’s async…?
Yeah, this is not about write availability, but as the OP/author points out, scaling that is not the bottleneck for most apps.
I think you may have misunderstood the GP and are perhaps misusing terminology. You cannot meaningfully scale vertically to improve write availability, and if you care about availability a single machine (and often a primary/secondary setup) is insufficient.
Even if you only care about scaling reads, eventually the 1:N write:read replica ratio will become too costly to maintain, and long before you reach that point you likely sacrifice real-time isolation guarantees to maintain your write availability and throughput.
Re: HTAP is Dead
#47You cannot say HTAP is dead when the alternative is so much complexity and so many moving parts. Most enterprises are burning huge amounts of resources literally just shuffling data around for zero business value.
The dream is a single data mesh presenting an SQL userland where I can write and join data from across the business with high throughput and low latency. With that, I can kill off basically every microservice that exists, and work on stuff that matters at pace, instead of half of all projects being infrastructure churn. We are close but we are not there yet and I will be furious if people stop trying to reach this endgame.
Re: HTAP is Dead
#48On the data warehousing side, I think the story looks like this: 1) Cloud data warehouses like Redshift, Snowflake, and BigQuery proved to be quite good at handling very large datasets (petabytes) with very fast querying. 2) Customers of these proprietary solutions didn't want to be locked in. So many are drifting toward Iceberg tables on top of Parquet (columnar) data files. Another "hidden" motive here is that Clou…
> There's so many moving parts here. Yep. At the scope of a single table, append-only history is nice but you're often after a clone of your source table within Iceberg, materialized from insert/update/delete events with bounded latency. There are also nuances like Postgres REPLICA IDENTITY and TOAST columns. Enabling REPLICA IDENTITY FULL amplifies you source DB WAL volume, but not having it means your CDC updates w…
can you explain this please "not having it means your CDC updates will clobber your unchanged TOAST values" ?
Re: HTAP is Dead
#49You cannot say HTAP is dead when the alternative is so much complexity and so many moving parts. Most enterprises are burning huge amounts of resources literally just shuffling data around for zero business value. The dream is a single data mesh presenting an SQL userland where I can write and join data from across the business with high throughput and low latency. With that, I can kill off basically every microservi…
Maybe GraphQL can be your savior?
Re: HTAP is Dead
#50Never heard of it. Maybe it's a good thing it's being considered dead... /s