Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating to Postgres

engineering.usemotion.com

Post reply on HN
Print view View original

271–278 of 278 posts

Re: Migrating to Postgres

#271

Post by TheNewsIsHere »

bigfatkitten wrote: post #256
cies wrote: post #200

Earlier quoted context omitted.

I think these days it's more: were do we find the grey old unix guy who confidently will host your db for you on bare metal.

Which these days seems to mean a 35+ year old who has ever worked on anything other than a big-three cloud.

I’m a mid-30-something who has worked in all three as well as several of the “alt cloud” providers. Typically identified as Linode, DigitalOcean, Hetzner, sometimes Rackspace. I’ve used all of those and more.

Bare metal is absolutely where it’s at for base infrastructure and bang-for-you-buck.

People call me a graybeard for such things, but it’s never been said derisively or sarcastically to my face. Usually it’s asking for advice.

I took a “traditional” path here. I started when “the cloud” was just the fluffy white cloud that said “Internet” or “Other Networks” at the top of the diagram.

Re: Migrating to Postgres

#272

Post by evantbyrne »

marcus_holmes wrote: post #262
evantbyrne wrote: post #206

Earlier quoted context omitted.

Could this be selection bias? I've never worked with a backend engineer that couldn't write SQL. I've worked on plenty of projects were there were bugs in hand written SQL though.

Basic SQL yeah, probably. Actually knowing SQL and how to write good SQL and understanding the tradeoffs between different query plans (and the tradeoffs between different schemas). That's rare. And yeah, obviously, there will be bugs in your SQL. And writing good tests for the database layer is always tricky because there's usually some pushback to setting up a database on the CI instance, and so on. It's not simple…

It makes sense that some kinds of schema optimizations are fairly esoteric, because the services most devs work on don't need them. An ORM is something you import on day one to speed up development; while rearranging tables to remove joins may not be needed (or even justifiable based on metrics) for another five years. I would hope that the ORM is flexible enough to be irrelevant to the trade-off between schemas though. Your comment about not having a database in CI brings back nightmares! Reminds me of how at my first job they had us running SQLite in CI and then shipping to Postgres, which worked about as well as you can imagine haha

Re: Migrating to Postgres

#273

Post by RadiozRadioz »

ScalaHanSolo wrote: post #58
esafak wrote: post #17

I read it as: Why You Shouldn't Use Prisma and How Cockroach Hung Us Out To Dry I already knew about prisma from the infamous https://github.com/prisma/prisma/discussions/19748

Author here. Yeah, that's not a bad take away either. I've also been really vocal in Primsa issues for all sorts of things. We are about to embark on a big migration away from Prisma and onto Drizzle once the Drizzle team lands 1.0 We will absolutely share our findings when that migration happens!

That just sounds irresponsible. The correct choice for prod isn't "the cool new trendy thing that will solve all our problems once it hits 1.0", the correct choice is "the boring stable thing that has existed long enough for everyone to understand its shortcomings".

Re: Migrating to Postgres

#274

Post by reinhardt »

wbercx wrote: post #111
ies7 wrote: post #91

Earlier quoted context omitted.

$500k for only 100 millions rows db also sounds crazy

The largest table was 100 million rows. They could have had hundreds more tables.

Also curious why every comment mentions just the number of rows as the only factor that matters. A 100M rows table of 3 integer columns is quite different from 50+ columns, 5 of which are text up to a few MB long.

Re: Migrating to Postgres

#275

Post by maz1b »

monkeyelite wrote: post #217
maz1b wrote: post #146

Earlier quoted context omitted.

What makes you say that? AFAIK, the largest single dedicated servers you can buy on the market go up to around hundreds of cores and terabytes of ram, and NVME up to a PB~ish if you stack NVME/SSD/HDD as well. this is when i last checked.

Yes. What do you have in mind?

You mentioned how there's ~100 organizations that need more than one instance of postgres, wanted to know how you arrived at that opinion.

Re: Migrating to Postgres

#276

Post by jjani »

cess11 wrote: post #167
esafak wrote: post #17

I read it as: Why You Shouldn't Use Prisma and How Cockroach Hung Us Out To Dry I already knew about prisma from the infamous https://github.com/prisma/prisma/discussions/19748

"Instead, it sends individual queries and joins the data on the application level. However, this doesn't mean that Prisma's relational queries are per se slower" Wow, what the fuck. "Also, this chapter about Query Performance Optimization from the High Performance MySQL book has some great insights. One of the techniques it mentions is JOIN decomposition: Many high-performance web sites use join decomposition. You ca…

> This belief that they can write JavaScript that outperforms decades of bare-metal executed optimisations in mainstream database engines is just astonishing.

In my first job fresh out of uni, I worked with a "senior" backend developer who believed this. He advocated for crusty, bug-ridden ORMs like Sequelize and Prisma (still very early stage back then, so lots more issues than now though I'd still steer well clear of it). Claiming they did "query optimizations". I knew it made zero sense, but also that I wasn't going to be able to convince him.

Re: Migrating to Postgres

#277

Post by hobs »

NegativeLatency wrote: post #258
hobs wrote: post #6

Earlier quoted context omitted.

How does columnar = json? json isn't colunar at all... If you just want to have a schema in json instead of sql, use a no-sql db, postgres nosql features are strong, but the db features are actually much stronger.

json isn't necessarily columnar, but it is a natrual fit for stuff that is columnar that's otherwise harder to model in a traditional relational db here's my usecase: - we have a bunch of attributes (all different names by customer, and many different values for each record that a customer stores) - it's a fairly natural fit for a json value with only one level of key: value mapping - we use mysql on GCP (no columnar…

Too bad mysql doesn't even have materialized views - its not a good combination with a json only situation, but you clearly are suffering through that, good luck :) probably etl it to clickhouse and move on.

Could always try and dump to GCP/Parquet/json and use duckdb/bq to just query it all.

Re: Migrating to Postgres

#278

Post by maz1b »

rastignack wrote: post #208
thehappyfellow wrote: post #162

Earlier quoted context omitted.

It’s incredible how much Postgres can handle. At $WORK, we write ~100M rows per day and keep years of history, all in a single database. Sure, the box is big, but I have beautiful transactional workloads and no distributed systems to worry about!

At $WORK, we are within the range of 2 billion rows per day on one of our apps. We do have beefy hardware and ultra fast SSD storage though.

A single PG database on one server? What are the specs?
Post reply on HN