Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

331–340 of 383 posts

Re: Squeeze the hell out of the system you have

#331
post #249

Earlier quoted context omitted.

Joins are not inherently expensive, but they can lead to expensive queries. For example, say I want to find the 10 most recent users with a phone number as their primary contact method: SELECT … FROM User JOIN ContactMethod on ContactMethod.userId = User.id WHERE ContactMethod.priority = ‘primary’ AND ContactMethod.type = ‘phoneNumber’ ORDER BY User.createdAt DESC LIMIT 10 If there are a very large number of users, a…

For 10 million users + telephones, this takes 1ms. create table users ( id serial primary key not null, created_at timestamp not null default now() ); create table users_telephones ( user_id int references users(id) not null, is_primary boolean not null default true, telephone varchar not null ); insert into users select i, NOW() + (random() * (interval '90 days')) + '30 days' from generate_series(1, 10000000) i; ins…

Ty for benchmarking, but this isn’t a good benchmark for the issue I’m talking about.

This is only fast because 100% of users have a phone number as a primary contact, so the join filter is essentially meaningless. If in the contact table, the filtered number is a small percentage of the total (e.g. most users have an email as their primary contact, not a phone number), but still a good size (e.g. there’s still hundreds of thousands to millions of phone primary contacts), it’s a much harder query.

It’s probably also fast because you have a warm cache - e.g. there’s enough memory for the DB to have the indexes 100% in memory, which is just not feasible with large DBs in the real world, where you can easily have >100GB of indexes + hot data, and the DB can’t keep it all in memory. In most real world scenarios, having to somewhat frequently read pages of indexes off disk, into memory, to satisfy queries, is common.

Try it again, with the exact same data, but:

- Search for users with a non-primary phone contact (you have 200,000 of these, and 10,000,000 users)

- Give the DB say 1/3 the memory of your total index size, so the complete indexes can’t be in memory

- Run the query right after starting PG up, to ensure the cache is cold (with a hot cache, almost everything is fast, but in real world situations with lots of users the cache isn’t consistently hot)

Re: Squeeze the hell out of the system you have

#332
post #172

Earlier quoted context omitted.

this has opinionated answers. if you ask Amazon, they might suggest that you design around a single table ( https://aws.amazon.com/blogs/compute/creating-a-single-table... ). in my opinion it's easier to use join tables. which are what are sometimes temporarily created when you do a join anyways. in this case, you permanently create table1, table2, and table1_join_table2, and keep all three in sync transactionally. w…

Would it be possible to simply use a materialized view for table1_join_table2?

The problem there would be how they fall out of date, and updating them is a heavy operation.

An alternative I've heard before is using triggers with regular tables, so updating one automatically updates the relevant other ones.

Re: Squeeze the hell out of the system you have

#333

Earlier quoted context omitted.

I’m wondering if indexes and materialized views can be used to do basically the same thing? That is, assuming they contain all the columns you want.

There's always money in the banana sta...materialized views. Materialized views will get you quite a ways on read heavy workloads.

As long as you're okay with the reads being a little out of date after writes occur.

Re: Squeeze the hell out of the system you have

#334

Earlier quoted context omitted.

Some of that, some other bad practices. Lots of low-hanging fruit, then more esoteric changes. https://justinlloyd.li/blog/how-much-cache-you-got-on-you/

He cached everything and delayed writes too. It's easy to make a system fast when it's not realtime.

Incredibly dismissive of somebody's work, aren't ya? I regret breaking my own self-imposed rule of never answering follow-up questions on HN because there's always somebody willing to hand-wave away six months of my life and 40 years of real-world experience with a flippant comment of "oh, but that's easy if you don't have too..."

Re: Squeeze the hell out of the system you have

#335
post #286

Earlier quoted context omitted.

Honestly I couldn’t disagree more. I built a startup and paid little attention to perf for years 1-5, and finally in year 6 we started to get bitten by some perf issues in specific tables, and spent a few engineer-months optimizing. In terms of tech debt it would have been way more expensive to make everything perform well from the start, we would have moved much slower and probably failed during a few crunch points.…

> a few $k/mo Isn’t that the cost of one engineer already?

In the Bay Area, no, an engineer costs an order of magnitude more. (For a round number, think $15-20k/mo including office space, benefits, etc. for a senior engineer; that's perhaps a bit high for the period I'm discussing but it also isn't attempting to price the cost of equity grants. At that time Google was probably spending something like $35-40k/mo (maybe higher, I don't know their office/perk costs) on equivalent talent at SWE5 including the liquid RSU grants.) But of course run the cost/benefit calc for your own cost of labor.

More importantly, it's critical to think in terms of opportunity cost. Like I said, we couldn't hire engineers fast enough at that time, so if I put someone on this work it would be taking them off some other important project. Plausibly for a fast-growing startup that means eschewing work that's worth $1-2m/eng-yr or more (just looking at concrete increases in company valuation, not present value of future gains). So we're talking on the order of $100k/eng-mo opportunity cost.

Re: Squeeze the hell out of the system you have

#336
post #296
post #277

Loads of over-engineering decisions would be avoided if devs understood how to read EXPLAIN/ANALYZE results and do the proper indexing/query optimization. Log queries, filter our the ones that are very frequent or take loads of time to execute, cache the frequent ones, optimize the fat ones, do this systematically and your system will be healthier. Things that help massively from my experience: - APM - slow query log…

Do you know of any good resources to understand sql explain plan. In my current project, we are facing a lot of issues related to query performance on MS SQL server. Do we need to always specify index hint with queries. Sometimes index exists but query does not seem to be using the index. I am thinking using sql execution plan could help us understand this issue better. tia.

I was recommended this video once, but I haven't watched it: https://youtu.be/sGkSOvuaPs4

Use the Index Luke (also recommended by cocoflunchy) was one of my go to resources once.

Also, Tobias Petry does a really good job by covering many advanced topics on a Twitter and his books: https://twitter.com/tobias_petry

Re: Squeeze the hell out of the system you have

#337

Earlier quoted context omitted.

https://en.m.wikipedia.org/wiki/Helmuth_von_Moltke_the_Elder Moltke's thesis was that military strategy had to be understood as a system of options, since it was possible to plan only the beginning of a military operation. As a result, he considered the main task of military leaders to consist in the extensive preparation of all possible outcomes.[3] His thesis can be summed up by two statements, one famous and one l…

I’m reminded of the Eisenhower line: “plans are worthless, but planning is everything.”

Good one indeed.

Apropos of Eisenhower, there is an incredible (fiction) book by Larry Collins, called Fall from Grace. It is about a brilliant long term plan and actions by British and French secret services to deceive the Germans about where the final Allied invasion would happen on the shores of France near the end of WWII. According to the novel, the ruse helped win the war.

Interwoven with a doomed romance.

https://www.google.com/search?q=fall+from+grace+larry+collin...

I read the full book some years ago, and it was gripping, though slow in parts.

Re: Squeeze the hell out of the system you have

#338
post #252

Earlier quoted context omitted.

Nope. Let's juxtapose them and see: Von Moltke: "No battle plan survives contact with the enemy." Tyson: "Everybody has a plan until you get hit in the face." Pretty much the same meaning, and Von Moltke's quote is three words shorter, so no, Tyson's quote is not simpler. Also, Tyson was ungrammatical, IMO: "Everybody" vs. "you" in the same sentence, referring to the same entity. Grammar experts, correct me if I am w…

It was midnight and a few beers after celebrating a birthday. I'm sorry I offended your grammatical sensibilities. But you really did go full orange site there, didn't you! I will admit to misquoting Mike Tyson; "Everyone has a plan until they get punched in the mouth.", which I hope goes someway to restoring peace and order over a tiny, drunken grammatical slip-up.

BTW, what is "orange site"?

I googled it and at least the top few links don't seem relevant.

Re: Squeeze the hell out of the system you have

#339
post #281
post #233

Earlier quoted context omitted.

Most places have "they" instead of "you" in the quote.

That works, too.

It's more like that's the right form, not just "works too", for the reason I said about third and second person forms.

Re: Squeeze the hell out of the system you have

#340

I'll probably get down-voted for saying this (again), but a key way to squeeze unimaginable amounts of performance is to lean into stored procedures . Look, I get it, the devx sucks. And it feels proprietary, icky, COBOL-like experience. It means you have to dwell in the database. What are you, a db admin?! But I'm telling you, the payoff is worth it. (and also, if you ship it you own it so yes you're a db admin). My…

I can't disagree with the results, SPs can change your life. HOWEVER, they require significant discipline and regular audits. All the code for them needs to be in source control with a Process for deployment to the DB. You also need a test suite as part of the Process which runs against a staging server with a comparable configuration to prod. The SPs need to be regularly dumped and compared against what's in source…

True enough; the binding tool handled this. The system of record was in the app server. The db was considered slaved to the app, and more-or-less tightly coupled to it (within reason). This provides a really solid mental model.

Side-note: the GP comment was actually doing pretty well (like +7) now it's (+2) so a cadre of anti-stored proc folks did a drive by down-voting. A bit sad, IMO.

Post reply on HN