Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

171–180 of 383 posts

Re: Squeeze the hell out of the system you have

#171

Post by DylanDmitri »

r00fus wrote: post #167
DylanDmitri wrote: post #152

Earlier quoted context omitted.

For Amazon, if we consider everything for website retail purchases, I would estimate tens of thousands of table schemas. This is counting: - Sellers (Amazon, third party, retail store) - Inventory (forecasting, recommendation) - Customers (comments, ratings, returns, preferences) - Warehouses (5+ distinct types, filled with custom machines) - Transit Options (long haul, air, vans, cars, bikes, walking, boats) - Deliv…

But these aren't the website, right? Amazon runs "wide" enterprise systems in the back end for sure.

Oh yeah these are "wide" enterprise systems, and not for the website. I think we read the initial comment differently.

Re: Squeeze the hell out of the system you have

#172

Post by r00fus »

endisneigh wrote: post #28
iamwil wrote: post #22

Earlier quoted context omitted.

If you don't use joins, how do you associate records from two different tables when displaying the UI? Do you just join in the application? Or something else?

this has opinionated answers. if you ask Amazon, they might suggest that you design around a single table ( https://aws.amazon.com/blogs/compute/creating-a-single-table... ). in my opinion it's easier to use join tables. which are what are sometimes temporarily created when you do a join anyways. in this case, you permanently create table1, table2, and table1_join_table2, and keep all three in sync transactionally. w…

Would it be possible to simply use a materialized view for table1_join_table2?

Re: Squeeze the hell out of the system you have

#173

Post by paulddraper »

endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

> Design your application's hot path to never use joins

I had a Chief Architect who decreed this.

So engineers wound up doing joins in application code, with far worse performance, filtering, memory caching, etc.

Re: Squeeze the hell out of the system you have

#174

Post by sgarland »

canucker2016 wrote: post #129
zengid wrote: post #109

The solution they went with, squeezing juice out of the system by finding performance optimizations, brings me so much joy. It reminds me of Richard L. Sites's book _Understanding_Software_Dynamics_ where he basically teaches how to measure and fix latency issues, and how at large scales, reducing latency can have tremendous savings. Measuring and reasoning about those issues are hard, but the solutions are often sim…

The problem I have with their eventual solution is that they only optimized their queries AFTER they had upgraded their instance to the largest config available. They couldn't upgrade their config with a few clicks in the admin console anymore (I'm guessing what's involved here) so now they had to use actual grey matter to fix their capacity problem. Maybe if they had spent more time optimizing specific parts of thei…

Despite the fact that SQL is not a complex language, and relational algebra isn't that hard, people regard it as dark magic.

Administering and tuning RDBMS is dark magic. Doing basic query optimization should be viewed the same as "maybe don't write an O(n^3) algorithm."

Re: Squeeze the hell out of the system you have

#176

Post by jmmv »

Reminds me of something I wrote a while back: https://jmmv.dev/2020/01/system-rewrites-and-tuning.html

TLDR: when facing problems like these, it’s too easy to look at grandiose solutions and, because they look like cool engineering problems, we end up justifying that it’s worth and reasonable to take on such year-long projects. But most often, the boring incremental solutions are easier and cheaper to achieve, while delivering benefit along the way.

This article shows examples of both, and I’m happy to see that the “boring” solution won.

Re: Squeeze the hell out of the system you have

#177

Post by smarkov »

justinlloyd wrote: post #147

Squeeze what you've got, as hard as you can, then realize after squeezing for a while that if you squeezed here, here, and also... here, by changing how you think about a problem, suddenly you've got a lot left to get. I spent two or so months optimizing the crap out of a majestic monolith and went from under 2K RPS when the PM thought, and the team repeatedly reported, that everything had been squeezed as much as it…

Very curious to learn more about what the monolith was doing so incredibly poorly that you managed to squeeze that much performance out of it. Poorly written queries? Too many queries? Lack of any caching? Doing things synchronously when they could've been done concurrently?

Re: Squeeze the hell out of the system you have

#178

Post by icedchai »

downWidOutaFite wrote: post #131
taylodl wrote: post #66

Earlier quoted context omitted.

My hot take: always use a materialized view or a stored procedure. Hide the actual, physical tables from the Application's account! The application doesn't need to know how the data is physically stored in the database. They specify the logical view they need of the data. The DBAs create the materialized view/stored procedure that's needed to implement that logical view. Since the application is never directly access…

This doesn't work because DBAs are rarely on the dev team's sprint schedule. If the DBAs are blocking them devs can and will figure out how to route around the gatekeepers. In general, keep the logic in the app not the db.

Also, many companies don't even have DBAs these days. DBA is, at best, a part time job for a senior+ engineer.

Re: Squeeze the hell out of the system you have

#179

Post by bachmeier »

i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

That's the right attitude for an employee. If management says something like that, look for a new job. It's not sustainable to compete with fewer resources than your opposition. There's a reason college sports is going through a passionate realignment right now.

Re: Squeeze the hell out of the system you have

#180

Post by jodrellblank »

latchkey wrote: post #38
endisneigh wrote: post #33

Earlier quoted context omitted.

yes, this is a fair point. there's no free lunch after all. without knowing more about what happened with Grab I'd say you could mitigate some of that with good management and access patterns, though.

All in all though, I don't think that 'never use joins' is a good solution either since it does create more developer work almost every way you slice it. I think the op's solution of looking more closely at the hot paths and solving for those is a far better solution than re-architecting the application in ways that could, or can, create unintended consequences. People don't consider that enough, at all. Don't forget…

Grab was the company Steve Yegge left Google to go to. He quit during COVID when he could no longer travel to Asia, and his retrospective is glowing, including of the CTO Mark Porter(I think?): "frankly, the “Grab Way” of collaboration teaches you life skills, such as psychological safety and inclusiveness, which work everywhere else as well. [...] We US Grabbers made many mistakes on the journey towards becoming better Grabbers. Mark Porter led the charge on figuring it out, and we’ve learned a ton from him as he has evolved as a person before our eyes [...] I want to thank Mark especially for helping me grow as a leader and as a person. It takes humility to become a better person, and Grab is a humbling place. The core company value of humility comes straight from the CEO and cofounder Anthony Tan, who is one of the most humble, down-to-earth people you’ll ever meet.".

Is that, uhh, rose-tinted glasses?

2020: https://steve-yegge.medium.com/saying-goodbye-to-the-best-gi...

Previously, 2018: https://steve-yegge.medium.com/why-i-left-google-to-join-gra...

[I don't have any knowledge of Grab or the people involved other than reading these two blog posts, or of the parent commenter].

Post reply on HN