Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating to Postgres

engineering.usemotion.com

Post reply on HN
Print view View original

131–140 of 278 posts

Re: Migrating to Postgres

#131

Post by evanelias »

etler wrote: post #40

I've lost count of how many "Migrating from X to Postgres" articles I've seen. I don't think I've once seen a migrating away from Postgres article.

Not an article, and I have no direct knowledge of this either way, but I would strongly suspect that Instagram migrated off Postgres a while back. Probably to fb-mysql + myrocks, or some other RocksDB based solution.

The compression level is vastly superior to any available Postgres-based solution, and at Instagram’s scale it amounts to extremely compelling hardware cost savings.

Also if they were still primarily on pg, it would be one of the largest pg deployments in existence, and there would be obvious signs of the eng impact of that (conference talks, FOSS contributions, etc).

Bigger-picture: Postgres is an amazing database, and it’s often the right choice, but nothing in tech is always the best choice 100% of the time. There’s always trade-offs somewhere.

Re: Migrating to Postgres

#132

Post by theshrike79 »

pier25 wrote: post #44
VWWHFSfQ wrote: post #38

Earlier quoted context omitted.

Every ORM is bad. Especially the "any DB" ORMs. Because they trick you into thinking about your data patterns in terms of writing application code, instead of writing code for the database. And most of the time their features and APIs are abstracted in a way that basically means you can only use the least-common-denominator of all the database backends that they can support. I've sworn off ORMs entirely. My applicati…

I use PG with Entity Framework in .NET and at least 90% of my queries don't need any PG-specific features. When I need something PG specific I have options like writing raw SQL queries. Having most of my data layer in C# is fantastic for productivity and in most cases the performance compared to SQL is negligible.

Coming from Javaland to C#, Entity Framework is a breath of fresh air.

The Npgsql driver automatically applies PG-specific tricks without me having to do anything special

The only path I had to do myself is the data ingress point that had some race condition issues, everything else seems to perform pretty well out of the box.

Re: Migrating to Postgres

#133

Post by ringeryless »

luhn wrote: post #55

> By Jan 2024, our largest table had roughly 100 million rows. I did a double take at this. At the onset of the article, the fact they're using a distributed database and the mention of a "mid 6 figure" DB bill made me assume they have some obscenely large database that's far beyond what a single node could do. They don't detail the Postgres setup that replaced it, so I assume it's a pretty standard single primary an…

OTOH they are admittedly using an ORM (Prisma, known for its weight)

Re: Migrating to Postgres

#134

Post by bigfatkitten »

gbear605 wrote: post #79
thomasfromcdnjs wrote: post #66

Earlier quoted context omitted.

Does mid six figure mean ~$500k? That sounds insane for a crud app with one million users. What am I missing?

I’ve seen startups with a thousand active users paying $50k/month (though that’s overall costs, not just db). It’s really easy to waste a lot of money doing nothing.

It’s especially easy to waste money on databases.

People just throw more compute power (ie money) at performance problems, rather than fixing their queries or making better use of indices.

Re: Migrating to Postgres

#135

Post by graealex »

from-nibly wrote: post #33

Feels like postgres is always the answer. I mean like there's gotta be some edge case somewhere where postgres just can't begin to compete with other more specialized database but I'd think that going from postgres to something else is much easier than the other way around.

It's not like PostgreSQL hasn't been in development for close to 30 years, covering basically every use case imaginable just through millions of deployments.

In addition, SQL in itself is a proven technology. The reality is that most problems you might think about solving with specialized databases (Big Data TM etc) could probably easily be solved with your run-of-the-mill RDBMS anyway, if more than five minutes are spent on designing the schema. It's extremely versatile, despite just being one layer above key-value storage.

Re: Migrating to Postgres

#136

Post by graealex »

999900000999 wrote: post #112
SchemaLoad wrote: post #110

Earlier quoted context omitted.

What situations do you encounter where you don't care about the structure of the data? The only ones I've ever encountered have been logging, where it's only purpose is to be manually text searchable, and something like OpenStreetMap where everything is just a key value store and the structure is loosely community defined. As soon as you have a loosely defined object you can't access any specific keys which makes it…

You define the data schema client side. That's the entire idea behind Firebase. It makes prototyping much faster. I don't know how well it scales, but it works for most smaller projects.

Wait until you hear about ORMs.

Re: Migrating to Postgres

#137

Post by graealex »

jacobsenscott wrote: post #102
from-nibly wrote: post #33

Feels like postgres is always the answer. I mean like there's gotta be some edge case somewhere where postgres just can't begin to compete with other more specialized database but I'd think that going from postgres to something else is much easier than the other way around.

PG requires a lot of expertise to keep running when you get to a billion rows or massive ingest. It can do it, but it doesn't just do it out of box running the defaults.

Hopefully at 1B records, you have a business model that allows you to spend some money on either hardware or talent to solve this problem.

Re: Migrating to Postgres

#138

Post by imhoguy »

thomasfromcdnjs wrote: post #66
luhn wrote: post #55

> By Jan 2024, our largest table had roughly 100 million rows. I did a double take at this. At the onset of the article, the fact they're using a distributed database and the mention of a "mid 6 figure" DB bill made me assume they have some obscenely large database that's far beyond what a single node could do. They don't detail the Postgres setup that replaced it, so I assume it's a pretty standard single primary an…

Does mid six figure mean ~$500k? That sounds insane for a crud app with one million users. What am I missing?

I bet it is cost of query processing (CPU) and traffic (network throughput) plus ofc provider markup.

Re: Migrating to Postgres

#139

Post by redcobra762 »

ScalaHanSolo wrote: post #58
esafak wrote: post #17

I read it as: Why You Shouldn't Use Prisma and How Cockroach Hung Us Out To Dry I already knew about prisma from the infamous https://github.com/prisma/prisma/discussions/19748

Author here. Yeah, that's not a bad take away either. I've also been really vocal in Primsa issues for all sorts of things. We are about to embark on a big migration away from Prisma and onto Drizzle once the Drizzle team lands 1.0 We will absolutely share our findings when that migration happens!

Yes, moving to a freshly 1.0 tool/library is often the best way to gain stability...

Re: Migrating to Postgres

#140

Post by monero-xmr »

ringeryless wrote: post #133
luhn wrote: post #55

> By Jan 2024, our largest table had roughly 100 million rows. I did a double take at this. At the onset of the article, the fact they're using a distributed database and the mention of a "mid 6 figure" DB bill made me assume they have some obscenely large database that's far beyond what a single node could do. They don't detail the Postgres setup that replaced it, so I assume it's a pretty standard single primary an…

OTOH they are admittedly using an ORM (Prisma, known for its weight)

It is truly amazing how mature developers always wind up at the same result - old tech that has stood the test of time. Betting the company on alpha solutions of dubious quality keeps the devs employed at least.
Post reply on HN