Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

Post reply on HN
Print view View original

311–320 of 383 posts

Re: Squeeze the hell out of the system you have

#311

Post by high_priest »

smarkov wrote: post #177
justinlloyd wrote: post #147

Squeeze what you've got, as hard as you can, then realize after squeezing for a while that if you squeezed here, here, and also... here, by changing how you think about a problem, suddenly you've got a lot left to get. I spent two or so months optimizing the crap out of a majestic monolith and went from under 2K RPS when the PM thought, and the team repeatedly reported, that everything had been squeezed as much as it…

Very curious to learn more about what the monolith was doing so incredibly poorly that you managed to squeeze that much performance out of it. Poorly written queries? Too many queries? Lack of any caching? Doing things synchronously when they could've been done concurrently?

Quick google yields very good examples in huge improvements on a single algorithm level: https://youtu.be/c33AZBnRHks Easy to imagine how even smaller improvements across 10-100 steps of data processing, can become even better.

Re: Squeeze the hell out of the system you have

#312

Post by kaba0 »

crazygringo wrote: post #159
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

That is a hot take... ;) But joins should never impact performance in a large way if they're on the same server and properly indexed. "It's truly amazing how much faster everything is when you eliminate joins" is just not true if you're using joins correctly. Sadly, many developers simply never bother to learn. On the other hand, having to write a piece of data to 20 different spots instead of 1 is going to be dramat…

Also, one could just create a materialized view for the join and solve the problem at the proper layer.

Re: Squeeze the hell out of the system you have

#313

Post by camgunz »

andresp wrote: post #282
atmosx wrote: post #280

> Just think about how massive these costs are. How much feature delivery will have to be delayed or foregone to support the additional architectural complexity? I don’t know if the author has worked with micro services. MS solve a communication issue. If implemented semi-properly teams stop blocking each other and the overall result is _faster_ and _safer_ feature delivery to production because the scope a team (or…

I got to understand from personal experience that the anti-SOA people are usually the ones who stayed at the same company their entire carers, never saw any model other than the monolithic one, see SOA as a threat to their domain knowledge within the company and simply are not able to see its downsides (because they have adapted their ways of work around it and never experienced anything better).

Similar experience to my sibling here: I've had a couple microservice shops and they really soured me on them and SOA in general.

- You have so many deploy stacks that it's literally boggling. I had to deploy code that used everything from make to GitHub CI to Jenkins to serverless just to push a single change.

- You have infinite implementations of your business logic. This is for two major reasons. First it's just hard to keep everything sync'd up; even if you've got "libbiz" you're gonna have services on various versions. But second, a core tenet of microservices is to just fork a new service for this and let other services migrate, but now you've evolved from an ecosystem of library differences to an ecosystem of service differences, which is way more complicated and expensive to maintain. It would be infinitely better if all parts of your app used the same versions of your business logic, but it will never ever happen.

- Your stacks are probably really heterogeneous ("right tool for the job"), but what that means is your devs now probably all have to know some mix of Java, Ruby, JavaScript, Python, Go, and maybe something more esoteric like Clojure or Elixir.

- Maintaining a microservice infra is way more complicated. Good luck monitoring multiple app and deploy stacks. Good luck keeping all of them up to date with security fixes. Good luck with Kubernetes and helm (or whatever). Good luck with multiple persistence systems (Postgres, Maria, Mongo, Redis, Cockroach, BigQuery, etc)

- You have to have an event bus. Boo.

- Your logging is hyper complicated now

- Because of the ecosystem around microservices, you're probably doing a lot of weird enterprisy things in your code (DDD, CQRS) that mostly only add layers of indirection or pull in more complicated dependencies, or inspire you to (against all good advice) build your own framework.

I'm not saying a monolith doesn't have problems, but I think the cons of microservices get very little play.

Re: Squeeze the hell out of the system you have

#314

Post by kaba0 »

rocqua wrote: post #284
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

I wish for a DB that lets me write a completely normalized scheme, and then lets me specify how it should denormalize the scheme for actual storage. There is no reason manual updates to denormalized DBs need to be hand-rolled every time. They are easy to automatically deduce.

These are called material views.

Re: Squeeze the hell out of the system you have

#315

Post by kaba0 »

roughly wrote: post #184
fuzztester wrote: post #126

Earlier quoted context omitted.

https://en.m.wikipedia.org/wiki/Helmuth_von_Moltke_the_Elder Moltke's thesis was that military strategy had to be understood as a system of options, since it was possible to plan only the beginning of a military operation. As a result, he considered the main task of military leaders to consist in the extensive preparation of all possible outcomes.[3] His thesis can be summed up by two statements, one famous and one l…

I’m reminded of the Eisenhower line: “plans are worthless, but planning is everything.”

“No battle was ever won according to plan, but no battle was ever won without one”

I heard it in this form.

Re: Squeeze the hell out of the system you have

#316

Post by tremon »

wizofaus wrote: post #71
endisneigh wrote: post #6

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

Can't say I've ever come across a scenario where a join itself was the performance bottleneck. If there's any single principle I have observed is "don't let a table get too big". More often than not it's historical-record type tables that are the issue - but the amount of data you need for day-to-day operations is usually a tiny fraction of what's actually in the table, and you're bound to start finding operations on…

don't let a table get too big

I'd amend that to "don't let your scan coverage get too big". Understanding how much data must be loaded in memory and compared is essential to writing performant database applications. And yes, those characteristics change over time as the data grows, so there may be a one-size-fits-all solution. But "table too large" can pretty much always be solved by adding better indexes or by partitioning the table, and making sure common queries (query's?) hit only one partition.

As a simple example: a lot of queries can be optimized to include "WHERE fiscal_year = $current". But you need to design your database and application up front to make use of such filtered indexes.

Re: Squeeze the hell out of the system you have

#317

Post by tremon »

kaba0 wrote: post #314
rocqua wrote: post #284

Earlier quoted context omitted.

I wish for a DB that lets me write a completely normalized scheme, and then lets me specify how it should denormalize the scheme for actual storage. There is no reason manual updates to denormalized DBs need to be hand-rolled every time. They are easy to automatically deduce.

These are called material views.

*updatable views.

Not every material view is updatable, and a view doesn't need to be materialized to be updatable.

Re: Squeeze the hell out of the system you have

#318

Post by atmosx »

camgunz wrote: post #313
andresp wrote: post #282

Earlier quoted context omitted.

I got to understand from personal experience that the anti-SOA people are usually the ones who stayed at the same company their entire carers, never saw any model other than the monolithic one, see SOA as a threat to their domain knowledge within the company and simply are not able to see its downsides (because they have adapted their ways of work around it and never experienced anything better).

Similar experience to my sibling here: I've had a couple microservice shops and they really soured me on them and SOA in general. - You have so many deploy stacks that it's literally boggling. I had to deploy code that used everything from make to GitHub CI to Jenkins to serverless just to push a single change. - You have infinite implementations of your business logic. This is for two major reasons. First it's just…

> I'm not saying a monolith doesn't have problems, but I think the cons of microservices get very little play.

I was about to make the same point. I hear you though. In my case CI/CD, application framework, programming language, etc. was common. We manage to shoot ourselves in the foot multiple times because (a) our CI/CD was _way_ too smart, allowing for _way_ to many things to happen for us (SRE team) in non-standard ways and (b) Helm (Duh!) ... allows you to put deployment logic in there which leads to a mess.

Re: Squeeze the hell out of the system you have

#319

Post by Too »

hennell wrote: post #288
otar wrote: post #277

Loads of over-engineering decisions would be avoided if devs understood how to read EXPLAIN/ANALYZE results and do the proper indexing/query optimization. Log queries, filter our the ones that are very frequent or take loads of time to execute, cache the frequent ones, optimize the fat ones, do this systematically and your system will be healthier. Things that help massively from my experience: - APM - slow query log…

Simply understanding how to read explain output can be quite a task in itself though, databases are a whole other thing, especially if you barely do any SQL yourself. Tools like https://explainmysql.com that make it clearer what you actually need to optimise are an easier system for Devs with enough database knowledge to set stuff up, but not enough to understand how it's used. I assume someone's already working on a…

Understanding explain output is usually very simple. 1 Look for any occurrence of “table scan”. 2. Add index on those queried fields or limit the query by filtering on another already indexed field.

This should unclog the most low hanging fruit. Then there is of course more advanced scenarios, especially with joins.

That’s not to say that the UX for explaining (hah) this doesn’t have a lot of room for improvement.

Re: Squeeze the hell out of the system you have

#320

Post by catalypso »

zepolen wrote: post #291
yashap wrote: post #249

Earlier quoted context omitted.

Joins are not inherently expensive, but they can lead to expensive queries. For example, say I want to find the 10 most recent users with a phone number as their primary contact method: SELECT … FROM User JOIN ContactMethod on ContactMethod.userId = User.id WHERE ContactMethod.priority = ‘primary’ AND ContactMethod.type = ‘phoneNumber’ ORDER BY User.createdAt DESC LIMIT 10 If there are a very large number of users, a…

For 10 million users + telephones, this takes 1ms. create table users ( id serial primary key not null, created_at timestamp not null default now() ); create table users_telephones ( user_id int references users(id) not null, is_primary boolean not null default true, telephone varchar not null ); insert into users select i, NOW() + (random() * (interval '90 days')) + '30 days' from generate_series(1, 10000000) i; ins…

Thanks for the effort.

Probably nitpicking but these types of measures are usually tricky to interpret because there is a high chance your indexes (maybe even rows) are still on PostgreSQL shared buffers and OS cache and might not reflect real usage performance.

To get a more "worst-case" measure, after your inserts and indexes creation, you can restart your database server + flush OS pages cache (e.g. drop_caches for Linux), then do the measure.

Sometimes the difference is huge, although I don't suspect it will be in this case.

Post reply on HN