51–60 of 95 posts
You cannot say HTAP is dead when the alternative is so much complexity and so many moving parts. Most enterprises are burning huge amounts of resources literally just shuffling data around for zero business value. The dream is a single data mesh presenting an SQL userland where I can write and join data from across the business with high throughput and low latency. With that, I can kill off basically every microservi…
Maybe GraphQL can be your savior?
I would say compute and storage separation is the way to go, especially for hyperscaler offering ala aurora db/cosmos/alloy. And later more opensource alternatives will catch up.
Most analytics workloads are bandwidth-bound if you are optimizing them at all. The major issue with disaggregated storage is that the storage bandwidth is terrible in the cloud. I can buy a server from Dell with 10x the usable storage bandwidth of the fastest environments in AWS and that will be reflected in workload performance. The lack of usable bandwidth even on huge instance types means most of that compute and…
You cannot say HTAP is dead when the alternative is so much complexity and so many moving parts. Most enterprises are burning huge amounts of resources literally just shuffling data around for zero business value. The dream is a single data mesh presenting an SQL userland where I can write and join data from across the business with high throughput and low latency. With that, I can kill off basically every microservi…
Maybe GraphQL can be your savior?
It is, of course, possible that SQL is too complex a language for this dream.
Well no, not according to your own source:
This setup consists of one primary database and dozens of replicas.
There have been several instances in the past where issues related to PostgreSQL have led to outages of ChatGPT.
However, “write requests” have become a major bottleneck. OpenAI has implemented numerous optimizations in this area, such as offloading write loads wherever possible and avoiding the addition of new services to the primary database.
It's been less than a week since someone in the company posted in Slack "we tried scaling up the db (Azure mssql) but it didn't fix the performance issues."
Earlier quoted context omitted.
Yeah, this is not about write availability, but as the OP/author points out, scaling that is not the bottleneck for most apps.
I think you may have misunderstood the GP and are perhaps misusing terminology. You cannot meaningfully scale vertically to improve write availability, and if you care about availability a single machine (and often a primary/secondary setup) is insufficient. Even if you only care about scaling reads, eventually the 1:N write:read replica ratio will become too costly to maintain, and long before you reach that point y…
No worries there, in all likelihood isolation has probably been killed twice already. Once by running the DB on READ COMMITTED, and a second time by using an ORM like EF to read data into your application, fiddle with it in-RAM, and write the new (unrelated-to-what-was-read) data back to the DB.
In other words, we throw out all that performant 2010-2020 NoSQL & eventual consistency tech, and go back to good old fashioned SQL & ACID, because everyone knows SQL, and ACID is amazing. Then we use LINQ/EF instead because it turns out that no-one actually wants to touch SQL, and full isolation is too slow so that gets axed too.
> Cursor and OpenAI are powered by a single-box Postgres instance. You’ll be just fine. Well no, not according to your own source: This setup consists of one primary database and dozens of replicas. Are they just fine? There have been several instances in the past where issues related to PostgreSQL have led to outages of ChatGPT. OK but let's pretend it's acceptable to have outages. It's fine apart from that? However…
Network round trip? Scaling the instance aint gonna help. Row by agonizing row? Maybe some linear speedups as you get more IO, but cloud storage is pretty fucking slow. Terrible plan/table/indexing/statistics? Still gonna be bad with more grunt. Blocking and locking and deadlocking the problem? Speeding up might make it worse :)
If people have exponential problems they don't think "let's just get more machines" they think "lets measure and fix the damn thing" but for some reason it doesn't apply to most people's databases.
On the data warehousing side, I think the story looks like this: 1) Cloud data warehouses like Redshift, Snowflake, and BigQuery proved to be quite good at handling very large datasets (petabytes) with very fast querying. 2) Customers of these proprietary solutions didn't want to be locked in. So many are drifting toward Iceberg tables on top of Parquet (columnar) data files. Another "hidden" motive here is that Clou…
Earlier quoted context omitted.
GraphQL is just a language much like SQL.
Yes but it provides a standardized way to deliver a unified interface to query your data, which is what OP is after?