Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

51–60 of 383 posts

Re: Squeeze the hell out of the system you have

#51

Post by sainez »

huijzer wrote: post #2

> We should always put off significant complexity increases as long as possible. Reminds me of the mantra that I’ve read here to easily go for reversible things and very careful when going for irreversible things.

It is mentioned in this article about the inception of AWS's custom silicon: https://semiconductor.substack.com/p/on-the-origins-of-aws-c...

> “We use the terms one-way door and two-way door to describe the risk of a decision at Amazon. Two-way door decisions have a low cost of failure and are easy to undo, while one-way door decisions have a high cost of failure and are hard to undo. We make two-way door decisions quickly, knowing that speed of execution is key, but we make one-way door decisions slowly and far more deliberately.”

Re: Squeeze the hell out of the system you have

#52

Post by mandevil »

He says to avoid complexity, and the team he was on (cleaning up some bad queries) was probably improving along that axis (or at worst orthogonal to complexity) but, from having done exactly this, adding an optional 'query the read-replica' option for queries- and determining whether this query can safely go there- is definitely extra complexity which will now need to be managed into the future. Definitely less overall than a complete re-arching of the system, but this is where engineering judgement and experience come into play: would you be better off getting those select queries resolved with some other data store or with a pg read-replica? If your query can survive against the read-replica (so stale data is at least sometimes acceptable) would you be better off caching results in redis?

Re: Squeeze the hell out of the system you have

#53

Post by dbg31415 »

I feel like a lot of people do this instead of upgrading to newer versions, even maintenance patches.

And I get that upgrades can be scary, but often they are relatively low cost.

Leaving everyone on the old system unhappy… means they will eventually push to re-platform, or rebuild, instead of just doing suggested maintenance along the way to keep they system they have in good shape.

My advice… do the maintenance. Do all the maintenance! Don’t just drive it into the ground and get mad when it breaks; change the oil and tires and spring for a car wash and some new wiper blades every now and then and you’ll be happier in the long run.

Re: Squeeze the hell out of the system you have

#54

Post by joelshep »

39 wrote: post #11

Strangely obvious advice?

It might be obvious as far as it goes, but it's also incomplete in at least two ways. One is that as tweaks and optimizations and "supplementing the system in some way" often involves increasing its complexity, even if just a little bit at a time. It adds up with time. The more important thing is this: if you're already constrained on vertical scaling, and you don't have a firm grip on how fast your system is scaling, then you can't just stop with making the db more efficient. That's just postponing the inevitable, and possibly not for more than a couple of years. If you're in the position the author portrays, get the database under control first -- for sure -- but then get started on figuring out how you're going to stay in front of your scaling problem, whether that's rearchitecture, off-loading work to systems better suited for it, or whatever. Speaking as a former owner of a very large Amazon database that fought this battle many times, trying to buy enough dev time to build away from it before it completely collapsed. We were too content with performance improvements just like the ones described in this article, before finally recognizing we were just racing the clock.

Re: Squeeze the hell out of the system you have

#55

Post by aidos »

iamwil wrote: post #9

Ugh. I had a colleague that addressed any scaling problem by putting a cache in front of the DB. Praised for solving the immediate problem, but shouldered none of the costs. I admit in the face of finding Prod/market fit, you do the expedient thing, but damned if I'm not often at the receiving end of these sorts of decisions.

Interestingly, I often ask candidates about optimising a slow running db query and the majority of people jump to adding caching and very few ask if they can run an explain or see the indexes.

Re: Squeeze the hell out of the system you have

#56

Post by KnobbleMcKnees »

i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

That was Donald Rumsfeld!? I always assumed this came from some techie or agile guru given how much it's used as a concept in project planning.

Re: Squeeze the hell out of the system you have

#57

Post by giantrobot »

javajosh wrote: post #41

I'll probably get down-voted for saying this (again), but a key way to squeeze unimaginable amounts of performance is to lean into stored procedures . Look, I get it, the devx sucks. And it feels proprietary, icky, COBOL-like experience. It means you have to dwell in the database. What are you, a db admin?! But I'm telling you, the payoff is worth it. (and also, if you ship it you own it so yes you're a db admin). My…

I can't disagree with the results, SPs can change your life. HOWEVER, they require significant discipline and regular audits. All the code for them needs to be in source control with a Process for deployment to the DB. You also need a test suite as part of the Process which runs against a staging server with a comparable configuration to prod. The SPs need to be regularly dumped and compared against what's in source control and marked as what's supposed to be released in prod.

Re: Squeeze the hell out of the system you have

#59

Post by sainez »

i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Great point about working on teams. For the vast majority of tasks, people are only marginally better or worse than each other. A few people with decent communication will outpace a "star" any day of the week.

I try to remind myself of this fact when I'm frustrated with other people. A bit of humility and gratitude go a long way.

Re: Squeeze the hell out of the system you have

#60

Post by romafirst3 »

iamwil wrote: post #25
andrewstuart wrote: post #10

Isn’t Rails wasteful in its database access patterns?

Generally No. But it can be easy to write bad queries using ActiveRecord ORM if you're not aware of N + 1 problems.

100%, it makes it easy for bad programmers to write bad performing queries, but you can easily write performant code. Btw that’s a feature - letting people ramp up to full db knowledge is beneficial, you don’t want to be spending your time writing performant queries before you need to.
Post reply on HN