Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

231–240 of 383 posts

Re: Squeeze the hell out of the system you have

#231

Post by fastball »

endisneigh wrote: post #196
crazygringo wrote: post #159

Earlier quoted context omitted.

That is a hot take... ;) But joins should never impact performance in a large way if they're on the same server and properly indexed. "It's truly amazing how much faster everything is when you eliminate joins" is just not true if you're using joins correctly. Sadly, many developers simply never bother to learn. On the other hand, having to write a piece of data to 20 different spots instead of 1 is going to be dramat…

I agree but I’m talking in the context where you can’t vertically scale anymore. I also don’t think it’s worth the trouble “never using joins” for an existing project. Denormalize as necessary. But for a green one I honestly think since our access patterns can be understood as you continue you can completely get rid of joins. Again, assuming your new project can’t fit on a single machine. If it can you’re best just f…

How is it that your new project can't fit on a single machine?

Re: Squeeze the hell out of the system you have

#232

Post by backendanon »

endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

Performance isn't king. The business is king.

Re: Squeeze the hell out of the system you have

#233

Post by dizhn »

eru wrote: post #218
fuzztester wrote: post #182

Earlier quoted context omitted.

Nope. Let's juxtapose them and see: Von Moltke: "No battle plan survives contact with the enemy." Tyson: "Everybody has a plan until you get hit in the face." Pretty much the same meaning, and Von Moltke's quote is three words shorter, so no, Tyson's quote is not simpler. Also, Tyson was ungrammatical, IMO: "Everybody" vs. "you" in the same sentence, referring to the same entity. Grammar experts, correct me if I am w…

Simplicity isn't a function of number of words alone. > Also, Tyson was ungrammatical, IMO: > "Everybody" vs. "you" in the same sentence, referring to the same entity. Seems perfectly understandable to English speakers. (And that's pretty close to how English grammar is defined by descriptive linguists.) Have a look at http://fine.me.uk/Emonds/ for an exploration of these kinds of concepts.

Most places have "they" instead of "you" in the quote.

Re: Squeeze the hell out of the system you have

#234

Post by porridgeandrice »

taylodl wrote: post #66
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

My hot take: always use a materialized view or a stored procedure. Hide the actual, physical tables from the Application's account! The application doesn't need to know how the data is physically stored in the database. They specify the logical view they need of the data. The DBAs create the materialized view/stored procedure that's needed to implement that logical view. Since the application is never directly access…

Yeah, I like doing this too. Not _always_, but for a few things. I use it to emulate partial functions.

In a show hosting/ticket booking app for example, I never want in any case user facing search/by-id endpoints to serve a show from 2 months ago. So I create a view `select * from shows where time > now`. I can now use this as a 'table' and apply more filters and joins to this if I wish.

Re: Squeeze the hell out of the system you have

#235

Post by dizhn »

batch12 wrote: post #202
i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Sounds like he may have been a Shania Twain fan - "Dance with the One That Brought You" is the similar phrase I've heard

Let's go back to the 70s.

https://en.m.wikipedia.org/wiki/Love_the_One_You%27re_With

Re: Squeeze the hell out of the system you have

#236

Post by chaxor »

endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

Thanks, I hate it.

This seems close to the territory of "why do I need a database? I just keep a bunch of text files with really long names that described exactly what I did to compute the file. They're all in various directories, so if you need to find one just do some greps and finds on the whole system"

I recognize there's a big gap, but boy howdy does what you're suggesting sound messy.

Re: Squeeze the hell out of the system you have

#237

Post by porridgeandrice »

smarkov wrote: post #177
justinlloyd wrote: post #147

Squeeze what you've got, as hard as you can, then realize after squeezing for a while that if you squeezed here, here, and also... here, by changing how you think about a problem, suddenly you've got a lot left to get. I spent two or so months optimizing the crap out of a majestic monolith and went from under 2K RPS when the PM thought, and the team repeatedly reported, that everything had been squeezed as much as it…

Very curious to learn more about what the monolith was doing so incredibly poorly that you managed to squeeze that much performance out of it. Poorly written queries? Too many queries? Lack of any caching? Doing things synchronously when they could've been done concurrently?

+1 I'd like to know as well

Re: Squeeze the hell out of the system you have

#238

Post by xapata »

eru wrote: post #217
xapata wrote: post #78

Earlier quoted context omitted.

> could've If someone is 83.7% likely to provide good leadership, how would you evaluate the choice to hire that person as a leader in the hindsight that the person failed to provide good leadership -- was it a bad choice, or was it a good choice that was unlucky? (Likelihood was selected arbitrarily.)

Is that number (publicly) known when you hire the person? If yes, you just evaluate the choice based on that probability (and other things you knew at the time), not on the actual outcome. Prediction markets are one way to make these kinds numbers known.

No, the likelihood is unknown, but the hiring process includes a model to estimate it. Of a sort.

Re: Squeeze the hell out of the system you have

#239

Post by hliyan »

endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

Ten years ago when DB engines were not as good and servers were not as large, I did something similar -- set up a trigger on insert/updates to certain relations that auto updates the main record with a cache column. Back then it was comma separated, but today I would obviously use JSONB. Back then it reduced latency significantly. Today, I would probably not attempt it.

Re: Squeeze the hell out of the system you have

#240

Post by hliyan »

sgarland wrote: post #165
killjoywashere wrote: post #108

Earlier quoted context omitted.

As a military officer who was watching CNN live from inside an aircraft carrier (moored) when he said that, being in charge of anti-terrorism on the ship at the time, it was absolutely foundational to my approach to so many things after that. Here's the actual footage: https://www.youtube.com/watch?v=REWeBzGuzCc Rumsfeld was complicated, but there's no doubt he was very effective at leading the Department. I think mo…

> Google's hallowed SRE system was developed by an engineer who had come up through the ranks of Navy nuclear power Wait, really? That makes _so much sense._ It also makes me upset that all of my attempts to sway other SRE orgs over to Nuclear Navy practices have been met with doubt. - ex nuke submariner

Amazing. Just looked it up. This document references the nuclear navy and civil nuclear power multiple times: https://sre.google/sre-book/lessons-learned/
Post reply on HN