I like the way of thinking. Instead of migrating to another database, they keep that awesome one running and found smart workaround to push limits.
Scaling PostgreSQL to power 800M ChatGPT users
11–20 of 145 posts
Re: Scaling PostgreSQL to power 800M ChatGPT users
#12Someone ask Microsoft what does it feel to be bested by an open source project on their very own cloud platform!!! Lol.
Azure offers Postgres “DBaaS”, so I’m pretty sure they are no where near that stage. It’s more likely that we should watch out for the Microsoft E-E-E strategy.
Re: Scaling PostgreSQL to power 800M ChatGPT users
#13This is why I love Postgres. It can get you to being one of the largest websites before you need to reconsider your architecture just by throwing CPU and disk at it. At that point you can well afford to hire people who are deep experts at sharding etc.
PostgreSQL actually supports sharding out of the box, it's just a matter of setting up the right table partitioning and using Foreign Data Wrapper (FDW) to forward queries to remote databases. I'm not sure what the post is referencing when they say that sharding requires leaving Postgres altogether.
This is specifically what they said about sharding
> The primary rationale is that sharding existing application workloads would be highly complex and time-consuming, requiring changes to hundreds of application endpoints and potentially taking months or even years
Re: Scaling PostgreSQL to power 800M ChatGPT users
#14Earlier quoted context omitted.
PostgreSQL actually supports sharding out of the box, it's just a matter of setting up the right table partitioning and using Foreign Data Wrapper (FDW) to forward queries to remote databases. I'm not sure what the post is referencing when they say that sharding requires leaving Postgres altogether.
This is specifically what they said about sharding > The primary rationale is that sharding existing application workloads would be highly complex and time-consuming, requiring changes to hundreds of application endpoints and potentially taking months or even years
I know they said that, but in fact sharding is entirely a database-level concern. The application need not be aware of it at all.
Re: Scaling PostgreSQL to power 800M ChatGPT users
#15Someone ask Microsoft what does it feel to be bested by an open source project on their very own cloud platform!!! Lol.
And the same for Linux boxes on Azure - they dominate Windows servers by a huge margin.
Re: Scaling PostgreSQL to power 800M ChatGPT users
#16Earlier quoted context omitted.
This is specifically what they said about sharding > The primary rationale is that sharding existing application workloads would be highly complex and time-consuming, requiring changes to hundreds of application endpoints and potentially taking months or even years
I know they said that, but in fact sharding is entirely a database-level concern. The application need not be aware of it at all.
Sharding can be made mostly transparent, but it's not purely a DB-level concern in practice. Once data is split across nodes, join patterns, cross-shard transactions, global uniqueness, certain keys hit with a lot of traffic, etc matter a lot. Even if partitioning handles routing, the application's query patterns and its consistency/latency requirements can still force application-level changes.
Re: Scaling PostgreSQL to power 800M ChatGPT users
#17I don’t get it. This whole thing says single writer does not scale, so we stopped writing as much and removed reads away from it, so it works ok and we decided that’s enough. I guess thats great.
Re: Scaling PostgreSQL to power 800M ChatGPT users
#18They could’ve used mongodb which is web scale NoSQL database because SQL is 1990’s era legacy technology.
/s
Re: Scaling PostgreSQL to power 800M ChatGPT users
#19First OpenAI Engineering blog? I'm definitely interested in seeing more and how they handled the rapid growth.
Re: Scaling PostgreSQL to power 800M ChatGPT users
#20Earlier quoted context omitted.
PostgreSQL actually supports sharding out of the box, it's just a matter of setting up the right table partitioning and using Foreign Data Wrapper (FDW) to forward queries to remote databases. I'm not sure what the post is referencing when they say that sharding requires leaving Postgres altogether.
This is specifically what they said about sharding > The primary rationale is that sharding existing application workloads would be highly complex and time-consuming, requiring changes to hundreds of application endpoints and potentially taking months or even years
Genuinely sounds like the kind of challenge that could be solved with a swarm of Codex coding agents. I'm surprised they aren't treating this as an ideal use-case to show off their stack!