Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

71–80 of 383 posts

Re: Squeeze the hell out of the system you have

#71

Post by wizofaus »

endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

Can't say I've ever come across a scenario where a join itself was the performance bottleneck. If there's any single principle I have observed is "don't let a table get too big". More often than not it's historical-record type tables that are the issue - but the amount of data you need for day-to-day operations is usually a tiny fraction of what's actually in the table, and you're bound to start finding operations on massive tables get slow no matter what indexes you have (and even the act of adding more indexes becomes problematic. And just indexing all columns isn't enough for traditional RMDBSes at least - you have to index the right combinations of columns for them to be used. Might be different for DynamoDb).

Re: Squeeze the hell out of the system you have

#72

Post by roughly »

i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Rumsfeld's got some great quotes, most of which were delivered in the context of explaining how the Iraq war turned into such a clusterfuck, and boy could that whole situation have used the kind of leadership Donald Rumsfeld's quotes would lead you to believe the man could've provided.

Re: Squeeze the hell out of the system you have

#73

Post by midasuni »

KnobbleMcKnees wrote: post #56
i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

That was Donald Rumsfeld!? I always assumed this came from some techie or agile guru given how much it's used as a concept in project planning.

And unknown unknowns is a great way to communicate with stakeholders too

Re: Squeeze the hell out of the system you have

#74

Post by sheepz »

PeledYuval wrote: post #65
sheepz wrote: post #16

Agree wholeheartedly with the conclusion of the article. But the post makes it seem that there was no real query-level monitoring for the Postgres instance in place, other than perhaps the basic CPU/memory ones provided by the cloud provider. Using an ORM without this kind of monitoring is sure way to shoot yourself in the foot with n+1 queries, queries not using indexes/missing indexes etc The other thing that is am…

What's your recommended way of implementing this in a simple App Server Postgres architecture? Is there a good Postgres plugin or do you utilize something on the App side?

I've used pganalyze which is a non-free SaaS tool. Gives you a very good overview of where the DB time is spent with index suggestions etc. There are free alternatives, but require more work from you.

Re: Squeeze the hell out of the system you have

#75

Post by moffkalast »

roughly wrote: post #72
i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Rumsfeld's got some great quotes, most of which were delivered in the context of explaining how the Iraq war turned into such a clusterfuck, and boy could that whole situation have used the kind of leadership Donald Rumsfeld's quotes would lead you to believe the man could've provided.

Could've at least given them some motivational quotes.

Re: Squeeze the hell out of the system you have

#76

Post by roughly »

midasuni wrote: post #73
KnobbleMcKnees wrote: post #56

Earlier quoted context omitted.

That was Donald Rumsfeld!? I always assumed this came from some techie or agile guru given how much it's used as a concept in project planning.

And unknown unknowns is a great way to communicate with stakeholders too

Žižek has a followup to that quote:

"What he forgot to add was the crucial fourth term: the "unknown knowns," the things we don't know that we know."

I've found it's really critical during the project planning phase to get to not just where the boundaries of our knowledge are, but also where are the things we're either tacitly assuming or not even aware that we've assumed. An awful lot of postmortems I've been a part of have come down to "It didn't occur to us that could happen."

Re: Squeeze the hell out of the system you have

#77

Post by agentultra »

kreetx wrote: post #46
deathanatos wrote: post #23

> Split up the monolith into multiple interconnected services, each with its own data store that could be scaled on its own terms. Just to note: you don't have to split out all the possible microservices at this junction. You can ask, "what split would have the most impact?" In my case, we split out some timeseries data from Mongo into Cassandra. Cass's table structure was a much better fit — that dataset had a well…

In a way, in the article they also did a split: specific heavy select queries were offloaded to a replica.

They could probably squeeze more depending on their workload patterns. RDBMS' typically optimize for fast/convenient writes. If your write load would be fine with a small increase in latency then you can do a lot of de-normalization so that your reads can avoid using tonnes of joins, aggregates, windows, etc at read-time. Update write path so that you update all of the de-normalized views at write time.

Depending on your read load and application structure you can get a lot more scale with caching.

Decent article.

Re: Squeeze the hell out of the system you have

#78

Post by xapata »

roughly wrote: post #72
i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Rumsfeld's got some great quotes, most of which were delivered in the context of explaining how the Iraq war turned into such a clusterfuck, and boy could that whole situation have used the kind of leadership Donald Rumsfeld's quotes would lead you to believe the man could've provided.

> could've

If someone is 83.7% likely to provide good leadership, how would you evaluate the choice to hire that person as a leader in the hindsight that the person failed to provide good leadership -- was it a bad choice, or was it a good choice that was unlucky?

(Likelihood was selected arbitrarily.)

Re: Squeeze the hell out of the system you have

#79

Post by munificent »

roughly wrote: post #76
midasuni wrote: post #73

Earlier quoted context omitted.

And unknown unknowns is a great way to communicate with stakeholders too

Žižek has a followup to that quote: "What he forgot to add was the crucial fourth term: the "unknown knowns," the things we don't know that we know." I've found it's really critical during the project planning phase to get to not just where the boundaries of our knowledge are, but also where are the things we're either tacitly assuming or not even aware that we've assumed. An awful lot of postmortems I've been a part…

> An awful lot of postmortems I've been a part of have come down to "It didn't occur to us that could happen."

Would that not be an unknown unknown?

Re: Squeeze the hell out of the system you have

#80

Post by tedunangst »

i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Mattis "the enemy gets a vote" is another good reminder of reality, although people get very angry about it. Useful in terms of security, privacy, DRM, etc.
Post reply on HN