Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

61–70 of 383 posts

Re: Squeeze the hell out of the system you have

#61

Post by williamdclt »

endisneigh wrote: post #18
Scarbutt wrote: post #13

Earlier quoted context omitted.

Normalization is not only about data storage but most importantly, data integrity.

Yes, but I assert that it's possible to use transactions to update everything consistently. Serializable transactions weren't really common when MySQL/Postgres first came out, but now that they're common in new DBs + ACID, I think it's not possible to do with reasonable difficulty. If you agree with this, than its easy to prove that denormalized tables performance increase is well worth the annoyance of updating ever…

Transactions are not only (actually mainly not) about atomicity. Of course it’s possible to keep data integrity without normalisation, but that means you need to maintain the invariants yourself at application level and a big could result in data inconsistency. Normalisation isn’t there to make integrity possible, it’s there to make (some) non-integrity impossible.

Nobody says you have to have only one view of your data though. You can have a normalised view of your data to write, and another denormalised for fast reads (you usually have to, at scale). Something like event sourcing is another way (which is actually pushing invariants to application level, in a structured way)

Re: Squeeze the hell out of the system you have

#62

Post by phirschybar »

I agree with this approach. the other added benefit is that when they decided to optimize the app by eliminating or tuning queries and utilizing replicas for reads, they ultimately made the app much more performant while possibly reducing complexity. the "squeeze" mindset pays off in the long-run here. the continued optimization over time is infinitely better than adding the complexity of microservices or expanded infrastructure because the latter will simply bury and compound the potential optimizations which could AND SHOULD have been made. squeeze squeeze squeeze until you just can't squeeze any more!

Re: Squeeze the hell out of the system you have

#65

Post by PeledYuval »

sheepz wrote: post #16

Agree wholeheartedly with the conclusion of the article. But the post makes it seem that there was no real query-level monitoring for the Postgres instance in place, other than perhaps the basic CPU/memory ones provided by the cloud provider. Using an ORM without this kind of monitoring is sure way to shoot yourself in the foot with n+1 queries, queries not using indexes/missing indexes etc The other thing that is am…

What's your recommended way of implementing this in a simple App Server Postgres architecture? Is there a good Postgres plugin or do you utilize something on the App side?

Re: Squeeze the hell out of the system you have

#66

Post by taylodl »

endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

My hot take: always use a materialized view or a stored procedure. Hide the actual, physical tables from the Application's account!

The application doesn't need to know how the data is physically stored in the database. They specify the logical view they need of the data. The DBAs create the materialized view/stored procedure that's needed to implement that logical view.

Since the application is never directly accessing the underlying physical data, it can be changed to make the retrieval more efficient without affecting any of the database's users. You're also getting the experts to create the required data access for you in the fastest, most efficient way possible.

We've been doing this for years now and it works great. It's alleviated so many headaches we used to have.

Re: Squeeze the hell out of the system you have

#67

Post by sakopov »

I thought I was going to read something insightful. Instead it was a post about how to completely ignore your database performance and then consider overcomplicating everything with sharding and microservices because you didn't care to do basic profiling on your queries. I'm glad common sense prevailed, but this is really some junior-level stuff and it's being celebrated as some kind of novelty.

Re: Squeeze the hell out of the system you have

#68

Post by sainez »

endisneigh wrote: post #28
iamwil wrote: post #22

Earlier quoted context omitted.

If you don't use joins, how do you associate records from two different tables when displaying the UI? Do you just join in the application? Or something else?

this has opinionated answers. if you ask Amazon, they might suggest that you design around a single table ( https://aws.amazon.com/blogs/compute/creating-a-single-table... ). in my opinion it's easier to use join tables. which are what are sometimes temporarily created when you do a join anyways. in this case, you permanently create table1, table2, and table1_join_table2, and keep all three in sync transactionally. w…

> what this might mean in practice is that you do mockups of all of the expected pages and what data is necessary on each one. then you design a schema that results in you never having to do joins on the majority, if not all, of them.

Great suggestion! I had a role where I helped a small team develop a full stack, data-heavy application. I felt pretty good about the individual layers but I felt we could have done a better job at achieving cohesion in the big picture. Do you have any resources where people think about these sorts of things deeply?

Re: Squeeze the hell out of the system you have

#69

Post by bayindirh »

iblaine wrote: post #64

TL;DR; do the easy things fist, in this case it was to fix bad SQL Given the options to optimize SQL, move read operations to replicas, shard data or go towards micro services, optimizing SQL is the easy choice.

Actually, I disagree. The "TL;DR:" in the article is "first outgrow, then upgrade". In today's software development practice, efficiency is second class citizen, because moving fast and breaking things is the way to keep the momentum and be hip.

However, sometimes everyone needs to chill and sharpen the tool they have at hand. It might prove much more capable than first anticipated. Or you may be holding the tool wrong to a degree.

Post reply on HN