Squeeze the hell out of the system you have

Post by aidos » Sat, Aug 12, 2023, 11:09 AM UTC

zepolen wrote: post #291

yashap wrote: post #249

Earlier quoted context omitted.

Joins are not inherently expensive, but they can lead to expensive queries. For example, say I want to find the 10 most recent users with a phone number as their primary contact method: SELECT … FROM User JOIN ContactMethod on ContactMethod.userId = User.id WHERE ContactMethod.priority = ‘primary’ AND ContactMethod.type = ‘phoneNumber’ ORDER BY User.createdAt DESC LIMIT 10 If there are a very large number of users, a…

For 10 million users + telephones, this takes 1ms. create table users ( id serial primary key not null, created_at timestamp not null default now() ); create table users_telephones ( user_id int references users(id) not null, is_primary boolean not null default true, telephone varchar not null ); insert into users select i, NOW() + (random() * (interval '90 days')) + '30 days' from generate_series(1, 10000000) i; ins…

Post by blitzar » Sat, Aug 12, 2023, 11:34 AM UTC

crazygringo wrote: post #159

endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

That is a hot take... ;) But joins should never impact performance in a large way if they're on the same server and properly indexed. "It's truly amazing how much faster everything is when you eliminate joins" is just not true if you're using joins correctly. Sadly, many developers simply never bother to learn. On the other hand, having to write a piece of data to 20 different spots instead of 1 is going to be dramat…

Post by sgarland » Sat, Aug 12, 2023, 11:39 AM UTC

hliyan wrote: post #240

sgarland wrote: post #165

Earlier quoted context omitted.

> Google's hallowed SRE system was developed by an engineer who had come up through the ranks of Navy nuclear power Wait, really? That makes _so much sense._ It also makes me upset that all of my attempts to sway other SRE orgs over to Nuclear Navy practices have been met with doubt. - ex nuke submariner

Amazing. Just looked it up. This document references the nuclear navy and civil nuclear power multiple times: https://sre.google/sre-book/lessons-learned/

Post by sgarland » Sat, Aug 12, 2023, 11:43 AM UTC

kreetx wrote: post #267

re-thc wrote: post #257

Earlier quoted context omitted.

> I built a startup and paid little attention to perf for years 1-5, and finally in year 6 we started to get bitten by some perf issues in specific tables, and spent a few engineer-months optimizing. This screams of if I don’t see it it the problem doesn’t exist view of the world. How do you know it’s not a problem? Perhaps customers would have signed up if it as faster? The problem is also treating it in terms of bu…

A denormalized database model is considered bad desig to begin with, and has performance costs on its own. This is why the OP says this is a "hot take". :) Maybe there are situations where this actually helps, although the resulting datastructure to me looks more like a multi-key cache.

Post by dist1ll » Sat, Aug 12, 2023, 12:10 PM UTC

justinlloyd wrote: post #147

Squeeze what you've got, as hard as you can, then realize after squeezing for a while that if you squeezed here, here, and also... here, by changing how you think about a problem, suddenly you've got a lot left to get. I spent two or so months optimizing the crap out of a majestic monolith and went from under 2K RPS when the PM thought, and the team repeatedly reported, that everything had been squeezed as much as it…