Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HTAP is Dead

mooncake.dev

Post reply on HN
Print view View original

31–40 of 95 posts

Re: HTAP is Dead

#31

Post by api »

bcoates wrote: post #10
charcircuit wrote: post #3

>Cursor is powered by a single-box Postgres instance Why wouldn't it? The resources needed to run the backend of Cursor come from the compute for the AI models. Updating someone's quota in a database every few minutes is not going to be causing issues.

In the nosql era the idea that you could run even the basics for a >1m user SaaS platform on an ordinary, free, single-node transactional SQL database would have been considered nuts.

It may have been back then, though I'd argue that you could have done it back then with efficient code and a very well-tuned DB.

Today big boxes are big. Really big. Stuff like 128 cores, 1TB RAM, and dozens of terabytes of incredibly fast RAIDed flash storage is available out there.

They're also more reliable than they used to be. Hardware still fails of course, but it doesn't fail as often as OG spinning disk did.

Re: HTAP is Dead

#32

Post by apwell23 »

Article is really messing up my browser so couldnt read on my phone. But htap never made sense to me be because in my experience its very rare that you'd need analytics on a single database. Its often a confluence of multiple datasources- streams, databases, csvs, vendor provided data .

Re: HTAP is Dead

#33

Post by hn_throwaway_99 »

> Most workloads don’t need distributed OLTP. Hardware got faster and cheaper. A single beefy machine can handle the majority of transactional workloads. Cursor and OpenAI are powered by a single-box Postgres instance. You’ll be just fine.

I thought this was such an important point. Sooooo many dev hours were spent figuring out how to do distributed writes, and for a lot of companies that work was never needed.

Re: HTAP is Dead

#34

Post by cyberax »

LAC-Tech wrote: post #12
bcoates wrote: post #10

Earlier quoted context omitted.

In the nosql era the idea that you could run even the basics for a >1m user SaaS platform on an ordinary, free, single-node transactional SQL database would have been considered nuts.

wait we're not in the nosql era anymore? dynamo and mongo are huge, redis and kafka (and their clones) are ubiquitous, etc etc

> wait we're not in the nosql era anymore?

Kinda. It turned out, that for the vast majority of users, a single Postgres instance on a reasonably large host is more than enough. Perhaps with a read replica as a hot standby.

You can easily get 1 million transactions per second from it (simple ones, granted). So why bother with NoSQL?

> redis and kafka (and their clones) are ubiquitous, etc etc

That's a bit different. Kafka is a message queue, and Redis is mostly used as a cache.

Re: HTAP is Dead

#35

Post by roncesvalles »

hn_throwaway_99 wrote: post #33

> Most workloads don’t need distributed OLTP. Hardware got faster and cheaper. A single beefy machine can handle the majority of transactional workloads. Cursor and OpenAI are powered by a single-box Postgres instance. You’ll be just fine. I thought this was such an important point. Sooooo many dev hours were spent figuring out how to do distributed writes, and for a lot of companies that work was never needed.

I thought it was the weakest point. The need for a distributed DB is rarely performance, it's availability and durability.

Re: HTAP is Dead

#36

Post by davidgomes »

roncesvalles wrote: post #35
hn_throwaway_99 wrote: post #33

> Most workloads don’t need distributed OLTP. Hardware got faster and cheaper. A single beefy machine can handle the majority of transactional workloads. Cursor and OpenAI are powered by a single-box Postgres instance. You’ll be just fine. I thought this was such an important point. Sooooo many dev hours were spent figuring out how to do distributed writes, and for a lot of companies that work was never needed.

I thought it was the weakest point. The need for a distributed DB is rarely performance, it's availability and durability.

But you can get more availability and more durability with much easier alternatives:

- Availability: spin up more read replicas.

- Durability: spin up more read replicas and also write to S3 asynchronously.

With Postgres on Neon, you can have both of these very easily. Same with Aurora.

(Disclaimer: I work at Neon)

Re: HTAP is Dead

#37

Post by hn_throwaway_99 »

roncesvalles wrote: post #35
hn_throwaway_99 wrote: post #33

> Most workloads don’t need distributed OLTP. Hardware got faster and cheaper. A single beefy machine can handle the majority of transactional workloads. Cursor and OpenAI are powered by a single-box Postgres instance. You’ll be just fine. I thought this was such an important point. Sooooo many dev hours were spent figuring out how to do distributed writes, and for a lot of companies that work was never needed.

I thought it was the weakest point. The need for a distributed DB is rarely performance, it's availability and durability.

I think you misunderstood his point (and mine). There are usually much better ways to support availability and durability than to have multiple simultaneous write servers. On the contrary, having multiple write servers is usually worse for availability and durability because of the complexity.

For example, look at how Google Cloud SQL's aptly name "High Availability" configuration supports high availability: 1 primary and 1 standby. The standby is synced to the primary, and the roles are switched if a failover occurs.

Re: HTAP is Dead

#38

Post by growlNark »

hn_throwaway_99 wrote: post #33

> Most workloads don’t need distributed OLTP. Hardware got faster and cheaper. A single beefy machine can handle the majority of transactional workloads. Cursor and OpenAI are powered by a single-box Postgres instance. You’ll be just fine. I thought this was such an important point. Sooooo many dev hours were spent figuring out how to do distributed writes, and for a lot of companies that work was never needed.

Something tells me neither cursor nor openai need write workloads, so they would probably do just as fine using a flat file. I'm honestly curious what use either would have for queries that you couldn't get with a filesystem.

Certainly neither products have much obvious need for OLTP workloads. Hell, neither have any need for transactions at all. You're just paying them for raw CPU.

Re: HTAP is Dead

#39

Post by pragmatic »

apwell23 wrote: post #32

Article is really messing up my browser so couldnt read on my phone. But htap never made sense to me be because in my experience its very rare that you'd need analytics on a single database. Its often a confluence of multiple datasources- streams, databases, csvs, vendor provided data .

Analytics on your hot OLTP data.

Like realtime dashboards/reports as the transactions are coming in.

Think of a SaaS with high usage.

The analytics you're referring to use the more slow moving "ETL all the source data together" and then analyze it.

Different use cases.

Re: HTAP is Dead

#40

Post by jgraettinger1 »

pradn wrote: post #25

On the data warehousing side, I think the story looks like this: 1) Cloud data warehouses like Redshift, Snowflake, and BigQuery proved to be quite good at handling very large datasets (petabytes) with very fast querying. 2) Customers of these proprietary solutions didn't want to be locked in. So many are drifting toward Iceberg tables on top of Parquet (columnar) data files. Another "hidden" motive here is that Clou…

> There's so many moving parts here.

Yep. At the scope of a single table, append-only history is nice but you're often after a clone of your source table within Iceberg, materialized from insert/update/delete events with bounded latency.

There are also nuances like Postgres REPLICA IDENTITY and TOAST columns. Enabling REPLICA IDENTITY FULL amplifies you source DB WAL volume, but not having it means your CDC updates will clobber your unchanged TOAST values.

If you're moving multiple tables, ideally your multi-table source transactions map into corresponding Iceberg transactions.

Zooming out, there's the orchestration concern of propagating changes to table schema over time, or handling tables that come and go at the source DB, or adding new data sources, or handling sources without trivially mapped schema (legacy lakes / NoSQL / SaaS).

As an on-topic plug, my company tackles this problem. Postgres => Iceberg is a common use case.

[0] https://docs.estuary.dev/reference/Connectors/materializatio...

Post reply on HN