Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HTAP is Dead

mooncake.dev

Post reply on HN
Print view View original

81–90 of 95 posts

Re: HTAP is Dead

#81

Post by layer8 »

jandrewrogers wrote: post #79
layer8 wrote: post #77

Earlier quoted context omitted.

What are these used for, to all have to be in a single unified database?

A large part of those workloads is stitching together a single derived model of operational reality and how different entities interact over time from the samples you get from each individual source. You need a running log of all entity behavior and interactions over time to look back on in order to contextualize what you see at the current point in time. Most of this is not pre-computable because the combinatorial s…

Thanks. It would be interesting to talk about the business specifics, but that would move into confidential territory I guess.

Re: HTAP is Dead

#82

Post by dehrmann »

thom wrote: post #47

You cannot say HTAP is dead when the alternative is so much complexity and so many moving parts. Most enterprises are burning huge amounts of resources literally just shuffling data around for zero business value. The dream is a single data mesh presenting an SQL userland where I can write and join data from across the business with high throughput and low latency. With that, I can kill off basically every microservi…

A sibling mentioned GraphQL. That works, but it was really built for clients interacting with Meta's Ent framework. The web layer is largely a monolith, and user objects are modeled as "ents," linked to each other, and stored in heavily cached MySQL. GraphQL exposes access to them.

Re: HTAP is Dead

#83

Post by sgarland »

_benedict wrote: post #69
sgarland wrote: post #67

Earlier quoted context omitted.

> You cannot meaningfully scale vertically to improve write availability Disagree. Even if you limit yourself to the cloud, r7i/r8g.48xl gets you 192 vCPU / 1.5 TiB RAM. If you really want to get silly, x2iedn.32xl is 128 vCPU / 4 TiB RAM, and you get 3.8 TiB of local NVMe storage for temp tablespace. The money you’ll pay ($16.5K - $44K month, depending on specific class) would pay for a similarly spec’d server in th…

> If you really want to get silly, x2iedn.32xl is 128 vCPU / 4 TiB RAM, and you get 3.8 TiB of local NVMe This doesn't affect availability - except insofar as unavailability might be caused by insufficient capacity, which is not the typical definition. > Depending on your availability SLOs, of course Yes, exactly. Which is the point the GP was making. You generally make the trade-off in question not for performance,…

> This doesn't affect availability - except insofar as unavailability might be caused by insufficient capacity, which is not the typical definition.

I agree, but it seemed to me that GP was using it as such: "You cannot meaningfully scale vertically to improve write availability"

Re: HTAP is Dead

#84

Post by sgarland »

ghc wrote: post #70
sgarland wrote: post #61

Earlier quoted context omitted.

> but for some reason it doesn't apply to most people's databases. It’s because RDBMS effectively hasn’t changed in decades, and so requires fundamental knowledge of how computers work, and the ability to read dense technical docs. If those two clauses don’t seem related, go read the docs for HAProxy, or Linux man pages, or anything else ancient in the tech world. It used to be assumed that if you were operating comp…

> Django does neither, for example. Django is "ancient" just like HAProxy. I deployed my first Django app at the end of 2005.

Fair enough, I didn't know it was that old.

Re: HTAP is Dead

#85

Post by sgarland »

jandrewrogers wrote: post #72
sgarland wrote: post #62

Earlier quoted context omitted.

> The dream is a single data mesh presenting an SQL userland where I can write and join data from across the business with high throughput and low latency. That exists, and has for years: an extremely large DB loaded to the gills with RAM and local NVMe drives. Add some read replicas if you need them, similarly configured. Dedicate one for OLAP.

This doesn’t work quite as well as people assume. The first limit is simply size, you can only cram a few petabytes of NVMe in a server (before any redundancy) and many operational analytic workloads are quite a bit larger these days. One of the major advantages of disaggregated storage in theory is that it allows you to completely remove size limits. Many operational analytic workloads don’t need a lot of compute, j…

It works to a certain point, yes, but I daresay that the overwhelming majority of OLTP needs are in the <= TB range, not PB. OLAP is its own beast, though I'll also say that most modern tech companies' schema is hot garbage, full of denormalized tables for no good reason, JSON everywhere, etc. and thus the entire thing could be much, much smaller if RDMBS was used as it was intended: relationally.

Re: HTAP is Dead

#87

Post by roncesvalles »

davidgomes wrote: post #36
roncesvalles wrote: post #35

Earlier quoted context omitted.

I thought it was the weakest point. The need for a distributed DB is rarely performance, it's availability and durability.

But you can get more availability and more durability with much easier alternatives: - Availability: spin up more read replicas. - Durability: spin up more read replicas and also write to S3 asynchronously. With Postgres on Neon, you can have both of these very easily. Same with Aurora. (Disclaimer: I work at Neon)

No loss of committed transactions is acceptable to any serious business.

>I work at Neon

In my opinion, distributed DB solutions without synchronous write replication are DOA. Apparently a good number of people don't share this opinion because there's a whole cottage industry around such solutions, but I would never touch them with a 10 foot stick.

Re: HTAP is Dead

#88

Post by physix »

thom wrote: post #47

You cannot say HTAP is dead when the alternative is so much complexity and so many moving parts. Most enterprises are burning huge amounts of resources literally just shuffling data around for zero business value. The dream is a single data mesh presenting an SQL userland where I can write and join data from across the business with high throughput and low latency. With that, I can kill off basically every microservi…

> You cannot say HTAP is dead when the alternative is so much complexity and so many moving parts. Most enterprises are burning huge amounts of resources literally just shuffling data around for zero business value.

We built an HTAP platform as a layer over Cassandra for precisely that reason round about when Gartner invented the term.

In finance and fintech, there are ample use cases where the need for transactional consistency and horizontal scalability to process and report on large volumes come together, and where the banks really struggle to meet requirements.

I dug out an old description of our platform, updated it a bit, and put it on Medium, in case anyone is interested: https://medium.com/@paul_42036/a-technical-description-of-th...

Re: HTAP is Dead

#89

Post by pradn »

BewareTheYiga wrote: post #78
pradn wrote: post #75

Earlier quoted context omitted.

Their product looks promising. It looks like the PostGres schema and writes have to be "Iceberg-aware": special work to get around the fact that a small write results in a new, small Parquet file. That's not the end of the world - but perhaps ideally, you wouldn't be aware of Iceberg much at all when using PostGres. That might be a dream though. Fully using PostGres without awareness of Iceberg would require full dec…

I'd argue the bigger value is keeping the data in one storage place and bringing the compute to it. Works especially well for Big Corp use cases where entire divisions of the corp go their own way. Throw in M&A activity and it is a good hedge for the unknown (I.e you might be an Databricks and Azure shop and you just bought a Snowflake & AWS company). Keep the data in an open table format, and let everyone query usin…

There's two problems being discussed in this article and thread:

1) Combining OLTP and OLAP databases into one system

2) Using an open data format to be able to read/write from many system (OLTP/PostGres, analytics engine/Spark)

> I'd argue the bigger value is keeping the data in one storage place and bringing the compute to it.

Yes, I agree with you. This observation is the idea behind #2, and why Iceberg has so much momentum now.

Post reply on HN