Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling PostgreSQL to power 800M ChatGPT users

openai.com

Post reply on HN
Print view View original

51–60 of 145 posts

Re: Scaling PostgreSQL to power 800M ChatGPT users

#51

Post by jiggawatts »

antonkochubey wrote: post #43
jiggawatts wrote: post #36

Earlier quoted context omitted.

The major hyperscalers all offer a plethora of virtual machines SKUs that are essentially one entire two-socket box with many-core CPUs. For example, Azure Standard_E192ibds_v6 is 96 cores with 1.8 TB of memory and 10 TB of local SSD storage with 3 million IOPS. Past those "general purpose" VMs you get the enormous machines with 8, 16, or even 32 sockets.[1] These are almost exclusively used for SAP HANA in-memory da…

> Something like this will set you back $30K-$60K per year lol, no, cloud is nowhere near that good value. It’s $3.5M annually. > The Standard_HX176rs HPC VM size gives you 176 cores and 1.4 TB of memory This one is $124k per year.

Thanks for the correction, fixed.

I noticed that the M896i is so obscure and rarely used that there are typos associated with it everywhere including the official docs! In once place is says it has 23 TB of memory when it actually has 32 TB.

Re: Scaling PostgreSQL to power 800M ChatGPT users

#52

Post by jiggawatts »

manquer wrote: post #40
jiggawatts wrote: post #36

Earlier quoted context omitted.

The major hyperscalers all offer a plethora of virtual machines SKUs that are essentially one entire two-socket box with many-core CPUs. For example, Azure Standard_E192ibds_v6 is 96 cores with 1.8 TB of memory and 10 TB of local SSD storage with 3 million IOPS. Past those "general purpose" VMs you get the enormous machines with 8, 16, or even 32 sockets.[1] These are almost exclusively used for SAP HANA in-memory da…

On the AWS side there are "HANA certified" instances that max out at 1920 cores and 32 TB RAM - u7inh-32tb.480xlarge https://docs.aws.amazon.com/sap/latest/general/sap-hana-aws-...

I'm pretty sure both Azure and AWS are merely reselling the same HPE Compute Scale-up Server 3200 chassis with some variations. Azure seems to have only the 16-socket model, but AWS has the 32-socket model.

That AWS instance uses these 60-core processors: https://www.intel.com/content/www/us/en/products/sku/231747/...

To anyone wondering about these huge memory systems: avoid them if at all possible! Only ever use these if you absolutely must.

For one, these systems have specialised parts that are more expensive per unit compute: $283 per CPU core instead of something like $85 for a current-gen AMD EPYC, which are also about 2x as fast as the older Intel Scalable Xeons that need to go into this chassis! So the cost efficiency ratio is something like 6:1 in favour of AMD processors. (The cost of the single large host system vs multiple smaller ones can get complicated.)

The second effect is that 32-way systems have huge inter-processor cache synchronisation overheads. Only very carefully coded software can scale to use thousands of cores without absolutely drowning in cache line invalidations.

At these scales you're almost always better off scaling out "medium" sized boxes. A single writer and multiple read-only secondary replicas will take you very far, up to hundreds of gigabits of aggregate database traffic.

Re: Scaling PostgreSQL to power 800M ChatGPT users

#53

Post by trhway »

"However, some read queries must remain on the primary because they’re part of write transactions. "

if there is a read replica that has reached required snapshot - it is usually enough (depends on your task of course) for it to be the snapshot that was at the start of your transaction - and if the read query doesn't need to read your transaction uncommitted data, then that replica can serve the read query.

Re: Scaling PostgreSQL to power 800M ChatGPT users

#54

Post by ramraj07 »

sbstp wrote: post #37
dbuser99 wrote: post #17

I don’t get it. This whole thing says single writer does not scale, so we stopped writing as much and removed reads away from it, so it works ok and we decided that’s enough. I guess thats great.

This article has very little useful information... There's nothing novel about optimizing queries, sharding and using read replicas.

It has one piece of useful info: their main data store even for 800M users is a single instance of postgres (for writes) without sharding.

Re: Scaling PostgreSQL to power 800M ChatGPT users

#56

Post by gloflo »

simonw wrote: post #20
dmix wrote: post #13

Earlier quoted context omitted.

This is specifically what they said about sharding > The primary rationale is that sharding existing application workloads would be highly complex and time-consuming, requiring changes to hundreds of application endpoints and potentially taking months or even years

Genuinely sounds like the kind of challenge that could be solved with a swarm of Codex coding agents. I'm surprised they aren't treating this as an ideal use-case to show off their stack!

Oh snap! Maybe it's all a great deception for making money?

Re: Scaling PostgreSQL to power 800M ChatGPT users

#58

Post by 0xdeafbeef »

anonzzzies wrote: post #7
0xdeafbeef wrote: post #5

Earlier quoted context omitted.

Tidb should handle it nice. I've wrote 200к inserts / sec for hour in peak. Underlying lsm works better for writes

That would mean it improved somewhat. We always got better write performance from mysql vs postgres, however that is a while ago; we then tried tidb to go further but it was basically rather slow. Again, a while ago. When did you get your results, might be time to re-evaluate.

It was 1 year ago. Around 15 tikv serves, 32 cpu, 128 ram each, 4 tb nvme. In this case latency matters a lot. When i had load server in different region with ping of 3ms I got 70k inserts, when moved to the same region with sub ms ping it went to thousands

Re: Scaling PostgreSQL to power 800M ChatGPT users

#59

Post by freakynit »

CodeCompost wrote: post #46
huksley wrote: post #45

"... If a new feature requires additional tables, they must be in alternative sharded systems such as Azure CosmosDB rather than PostgreSQL...." So it is not really scaling too much now, rather maintaining current state of things and new features go to a different DB?

Azure CosmosDB is insanely expensive. I can't imagine anybody using it unless you have OpenAI money.

*Microsoft's money

Re: Scaling PostgreSQL to power 800M ChatGPT users

#60

Post by bostik »

tomnipotent wrote: post #30
n_u wrote: post #24

Cool! I'd love to know a bit more about the replication setup. I'm guessing they are doing async replication. > We added nearly 50 read replicas, while keeping replication lag near zero I wonder what those replication lag numbers are exactly and how they deal with stragglers. It seems likely that at any given moment at least one of the 50 read replicas may be lagging cuz CPU/mem usage spike. Then presumably that woul…

> would slow down the primary since it has to wait for the TCP acks Other than keeping around more WAL segments not sure why it would slow down the primary?

If you use streaming replication (ie. WAL shipping over the replication connection), a single replica getting really far behind can eventually cause the primary to block writes. Some time back I commented on the behaviour: https://news.ycombinator.com/item?id=45758543

You could use asynchronous WAL shipping, where the WAL files are uploaded to an object store (S3 / Azure Blob) and the streaming connections are only used to signal the position of WAL head to the replicas. The replicas will then fetch the WAL files from the object store and replay them independently. This is what wall-g does, for a real life example.

The tradeoffs when using that mechanism are pretty funky, though. For one, the strategy imposes a hard lower bound to replication delay because even the happy path is now "primary writes WAL file; primary updates WAL head position; primary uploads WAL file to object store; replica downloads WAL file from object store; replica replays WAL file". In case of unhappy write bursts the delay can go up significantly. You are also subject to any object store and/or API rate limits. The setup makes replication delays slightly more complex to monitor for, but for a competent engineering team that shouldn't be an issue.

But it is rather hilarious (in retrospect only) when an object store performance degdaration takes all your replicas effectively offline and the readers fail over to getting their up-to-date data from the single primary.

Post reply on HN