41–47 of 47 posts
Think deeply about what you're making the computer do, and ask it to do less things by being smarter about what you ask it to do.
I'd say 95% of the time most of your OOM gains will come from the above.
* Look for O(N^X), that is nested loops even when they are not necessarily expressed as loops on the language level.
* If possible, get rid of ORMs in favor of raw SQL. Not because ORMs are very bad but because almost nobody bothers to learn them; they often start causing issues with any non-trivial amount of load.
* Study data access patterns and figure out where and what composite indexes might help. I am saying composite indexes because I assume regular indexes are more or less always there, often even too many of them.
Especially with the last one I have achieved impressive results without any kind of impressive effort, just setting aside some time to understand the code.
Easy trick to making joins 50x faster: don't use Postgres and give your tables a primary key which groups related items together. A lot of people don’t know that a database index doesn’t order the actual rows on disk. It’s just a Btree of pointers. If you use clustered index for a table query pattern, the rows are actually ordered on disk. Most DBs load data in 8KiB chunks. So if you query 100 rows that are 100bytes,…
Although it isn't automatic, doesn't the Postgres CLUSTER command reorder the rows on disk? Or am I misunderstanding something?
Oh and also does compression which helps quite a bit with network storages (like cloud disks)
Easy trick to making joins 50x faster: don't use Postgres and give your tables a primary key which groups related items together. A lot of people don’t know that a database index doesn’t order the actual rows on disk. It’s just a Btree of pointers. If you use clustered index for a table query pattern, the rows are actually ordered on disk. Most DBs load data in 8KiB chunks. So if you query 100 rows that are 100bytes,…
Earlier quoted context omitted.
How it usually goes for me. I ask business how often should something be updated - they say “real time” (not going into details what real time means really) - it is hard to explain processing all data all the time so everything is fresh takes forever…. After couple of months it turns out they never ever open their “super important dashboard” or do it once in 6 months. Great after a year of bogging down everything I c…
Why take the unspecific "real time" answers as gospel? Just propose daily initially (or weekly or monthly), listen to their protests, and see if there are good reasons why it needs to be more often. Then pick a suitable interval that fulfills their needs and that you can guarantee. If you offer pink fluffy unicorns for free, people will always pick them, without thinking.
Bonus points for trying to explain you want to implement "eventual consistency" - keep in mind I have added {“real time” (not going into details what real time means really)} in text to indicate that I am not some junior whining around but someone with deep understanding of computing...
1. Establish what is good enough
2. Measure, don't guess
3. Fix the biggest bottleneck first
4. Measure after fixing
And some general things:
5. Avoid micro-benchmarks (i.e. things not at the entire system level)
6. Be careful with synthetic data
7. Know your general estimates (e.g. cache, memory, disk, network speeds)
Push less data through wires. The memory hierarchy is so stark on modern hardware that the 30-year-old adage that "the fastest code is the code you don't run" is maybe less important than, "the fastest code is the code that doesn't spend much time talking to the memory controller." And it's even worse once we start talking about accessing memory that's on an entirely different computer. Serialization/deserialization,…