Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

131–140 of 383 posts

Re: Squeeze the hell out of the system you have

#131

Post by downWidOutaFite »

taylodl wrote: post #66
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

My hot take: always use a materialized view or a stored procedure. Hide the actual, physical tables from the Application's account! The application doesn't need to know how the data is physically stored in the database. They specify the logical view they need of the data. The DBAs create the materialized view/stored procedure that's needed to implement that logical view. Since the application is never directly access…

This doesn't work because DBAs are rarely on the dev team's sprint schedule. If the DBAs are blocking them devs can and will figure out how to route around the gatekeepers. In general, keep the logic in the app not the db.

Re: Squeeze the hell out of the system you have

#133

Post by marcosdumay »

roughly wrote: post #72
i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Rumsfeld's got some great quotes, most of which were delivered in the context of explaining how the Iraq war turned into such a clusterfuck, and boy could that whole situation have used the kind of leadership Donald Rumsfeld's quotes would lead you to believe the man could've provided.

If I remember it correctly (it was a long time ago), he never fully supported the war. It didn't take a genius to notice that the goals set by the presidency were (literally) impossible and not the kind of thing you do achieve a war.

But whatever position he had, Iraq turning into a clusterfuck wasn't a sign of bad leadership by his part. It was a sign of bad ethics, but not leadership. His options were all of getting out of his position, disobeying the people above him, or leading the US into a clusterfuck.

Re: Squeeze the hell out of the system you have

#134

Post by topspin »

romafirst3 wrote: post #60
iamwil wrote: post #25

Earlier quoted context omitted.

Generally No. But it can be easy to write bad queries using ActiveRecord ORM if you're not aware of N + 1 problems.

100%, it makes it easy for bad programmers to write bad performing queries, but you can easily write performant code. Btw that’s a feature - letting people ramp up to full db knowledge is beneficial, you don’t want to be spending your time writing performant queries before you need to.

> it makes it easy for bad programmers to write bad performing queries

That is true of every ORM in existence. The easiest thing to do is naively follow the object graph in code, because that's what the ORM gives you. If the ORM was to somehow add friction here to encourage some other approach it would be panned as "too hard!!1" and fade away into obscurity.

Re: Squeeze the hell out of the system you have

#135

Post by pipe_connector »

feoren wrote: post #115
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

There are "tall" applications and "wide" applications. Almost all advice you ever read about database design and optimization is for "tall" applications. Basically, it means that your application is only doing one single thing, and everything else is in service of that. Most of the big tech companies you can think of are tall. They have only a handful of really critical, driving concepts in their data model. Facebook…

I agree with the characterization of applications you've laid out and think everyone should consider whether they're working on a "tall" (most users use a narrow band of functionality) or a "wide" (most users use a mostly non-overlapping band of functionality) application.

I also agree with your take that tall applications are generally easier to build engineering-wise.

Where I disagree is that I think in general wide applications are failures in product design, even if profitable for a period of time. I've worked on a ton of wide applications, and each of them eventually became loathed by users and really hard to design features for. I think my advice would be to strive to build a tall application for as long as you can muster, because it means you understand your customers' problems better than anyone else.

Re: Squeeze the hell out of the system you have

#136

Post by oDot »

i_like_apis wrote: post #34

I’m reminded of one of my favorite sayings: You go to war with the army you have, not the army you might want or wish to have at a later time. You may want to ignore that this this comes from Donald Rumsfeld (he has some great ones though: “unknown unknowns …”, etc.) I think about this a lot when working on teams. Everyone is not perfectly agreeable or has the same understanding or collective goals. Some may be subop…

Every time I hear the name Rumsfeld, I am reminded of the time when, for over 10 minutes, he refused to deny being a lizard:

https://www.youtube.com/watch?v=XH_34tqxAjA

Re: Squeeze the hell out of the system you have

#137

Post by bootsmann »

nerdponx wrote: post #127
macNchz wrote: post #121

In my experience, in web apps built on top of ORMs there is often a TON of low hanging fruit for query optimization when database load becomes an issue. Beyond the basics of "do we have N+1 issues", ORMs sometimes just don't generate optimal queries. I wouldn't want to built a complex production web app without an ORM, but being able to eject from it sometimes is key. Profile real world queries being run in productio…

I'm working on something right now with the Python ORM SQLAlchemy. It turns out that getting it to use RETURNING with INSERT is not trivial and requires you to set the non-obvious option `expire_on_commit=False`, which doesn't guarantee use of RETURNING, but is supposed to use it if your db driver and database happen to support it and the ORM happens to support it for that particular combination of driver and databas…

There is certainly an API to inspect your query, you can just call print() on the object iirc.

Re: Squeeze the hell out of the system you have

#138

Post by 8note »

wizofaus wrote: post #71
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

Can't say I've ever come across a scenario where a join itself was the performance bottleneck. If there's any single principle I have observed is "don't let a table get too big". More often than not it's historical-record type tables that are the issue - but the amount of data you need for day-to-day operations is usually a tiny fraction of what's actually in the table, and you're bound to start finding operations on…

Dynamo is quick for that, so long as you are picking good partition keys.

Instead, it'll throw you hot key throttling if you start querying one partition too much

Re: Squeeze the hell out of the system you have

#139

Post by klodolph »

I have personally witnessed the “let’s build microservices to get better performance” argument. I definitely want to nip that in the bud.

It’s easy to fall in love with complexity, especially since you see a lot of complexity in existing systems. But those systems became complex as they evolved to meet user needs, or for other reasons, over time. Complex systems are impressive, but you need to make sure that your team has people who recognize the heavy costs of complexity, and who can throw their engineering efforts directly against the most important problems your team faces.

Re: Squeeze the hell out of the system you have

#140

Post by mickdeek86 »

marcosdumay wrote: post #133
roughly wrote: post #72

Earlier quoted context omitted.

Rumsfeld's got some great quotes, most of which were delivered in the context of explaining how the Iraq war turned into such a clusterfuck, and boy could that whole situation have used the kind of leadership Donald Rumsfeld's quotes would lead you to believe the man could've provided.

If I remember it correctly (it was a long time ago), he never fully supported the war. It didn't take a genius to notice that the goals set by the presidency were (literally) impossible and not the kind of thing you do achieve a war. But whatever position he had, Iraq turning into a clusterfuck wasn't a sign of bad leadership by his part. It was a sign of bad ethics, but not leadership. His options were all of gettin…

Rumsfeld personally advanced the de-baathification directive - the lynchpin of the clusterfuckery - all on his own, and he certainly would have known to expect the 'unexpected' results to be similar to de-nazification. This was absolutely his choice. Another point you have (unintentionally?) brought up is the dignified resignation option. While it is often a naive, self-serving gesture, we can reasonably imagine that the Defense Secretary publicly resigning over opposition to a war during the public consideration of that war, might have had some effect on whether that war was started. I want to like him too, with his grandfatherly demeanor and genuine funnyness ("My god, were there so many vases?!") but, come on.
Post reply on HN