Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your go to performance optimization?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Ask HN: What is your go to performance optimization?

#31

Post by riskable »

I've seen memoization improve performance by enormous amounts. Even for simple functions that do a few simple calculations before returning a result.

Another go-to of mine is to take conditionals out of loops that really only need to be checked once. For example:

    for foo in whatever:
        for bar in foo:
            if len(foo) > some_value:
                do_something(bar)
Can become:

    for foo in whatever:
        if len(foo) > some_value:
            for bar in foo:
                do_something(bar)
This example is trivial and wouldn't gain much but imagine if `len(foo)` was a more computationally expensive function. You'd only need to call it on each iteration of foo instead of every iteration of foo * bar.

Re: Ask HN: What is your go to performance optimization?

#32

Post by cirrus3 »

riskable wrote: post #31

I've seen memoization improve performance by enormous amounts. Even for simple functions that do a few simple calculations before returning a result. Another go-to of mine is to take conditionals out of loops that really only need to be checked once. For example: for foo in whatever: for bar in foo: if len(foo) > some_value: do_something(bar) Can become: for foo in whatever: if len(foo) > some_value: for bar in foo:…

I wouldn't even really call that loop example an "optimization", it is just the obviously more efficient implementation.

Re: Ask HN: What is your go to performance optimization?

#34

Post by riskable »

cirrus3 wrote: post #32
riskable wrote: post #31

I've seen memoization improve performance by enormous amounts. Even for simple functions that do a few simple calculations before returning a result. Another go-to of mine is to take conditionals out of loops that really only need to be checked once. For example: for foo in whatever: for bar in foo: if len(foo) > some_value: do_something(bar) Can become: for foo in whatever: if len(foo) > some_value: for bar in foo:…

I wouldn't even really call that loop example an "optimization", it is just the obviously more efficient implementation.

Yet I see it constantly. That pattern of `for foo in whatever: for bar in foo` is everywhere. People don't even think about it... They just write the looping part then start thinking about the logic below. It's such a common thing I'm surprised compilers and interpreters don't just optimize it away :shrug:

Re: Ask HN: What is your go to performance optimization?

#35

Post by NicoJuicy »

koliber wrote: post #9

Reduce how often something runs and the amount of work it does when it does run. Sometimes, things are done too often, or process unnecessary data. I once had a project where a SAP system was crawling and literally causing company-wide stoppages. We found a job that literally ran every minute of every day and it processed a table that contained a few thousand tasks. This was something that could be done once per hour…

Just an alternative related to pulling.

We had a task that checked every minute what could be executed.

Instead of doing that, we scheduled the executing code to execute on that time ( if it didn't exist yet) through service bus. Since events can be planned...

Easy peasy and avoided a micro service named "scheduler", a db and an Serverless function... ( Hoboy)

Re: Ask HN: What is your go to performance optimization?

#36

Post by PigeonHolePncpl »

In my case, it was database optimizations that reduced the overall costs significantly

* Instead of writing big complex queries that had nested SELECT's, I split them into smaller bite-sized chunks that could be cached * Better caching strategies - reducing how many caches were flushed when a change was made * Tweaking the index's on database tables to improve WHERE clauses * Storing intermediate calculations into the database (for example, the number of posts a user has could be stored on the user table instead of counting them each time)

When I optimized the database, I could then reduce the size of the DB and the server as they no longer needed to work / wait as much

Re: Ask HN: What is your go to performance optimization?

#37

Post by srcreigh »

Easy trick to making joins 50x faster: don't use Postgres and give your tables a primary key which groups related items together.

A lot of people don’t know that a database index doesn’t order the actual rows on disk. It’s just a Btree of pointers.

If you use clustered index for a table query pattern, the rows are actually ordered on disk.

Most DBs load data in 8KiB chunks. So if you query 100 rows that are 100bytes, if they’re not sorted, you actually need to load nearly 1MiB of data even tho the query result is 10KiB.

Speeds up joins and range queries 50x or more, less cache evictions, etc.

You can do this in any database except for Postgres. Postgres doesn’t have the ability to keep rows sorted on disk.

Re: Ask HN: What is your go to performance optimization?

#39

Post by mthoms »

srcreigh wrote: post #37

Easy trick to making joins 50x faster: don't use Postgres and give your tables a primary key which groups related items together. A lot of people don’t know that a database index doesn’t order the actual rows on disk. It’s just a Btree of pointers. If you use clustered index for a table query pattern, the rows are actually ordered on disk. Most DBs load data in 8KiB chunks. So if you query 100 rows that are 100bytes,…

Although it isn't automatic, doesn't the Postgres CLUSTER command reorder the rows on disk? Or am I misunderstanding something?

Re: Ask HN: What is your go to performance optimization?

#40

Post by jononor »

ozim wrote: post #15
koliber wrote: post #9

Reduce how often something runs and the amount of work it does when it does run. Sometimes, things are done too often, or process unnecessary data. I once had a project where a SAP system was crawling and literally causing company-wide stoppages. We found a job that literally ran every minute of every day and it processed a table that contained a few thousand tasks. This was something that could be done once per hour…

How it usually goes for me. I ask business how often should something be updated - they say “real time” (not going into details what real time means really) - it is hard to explain processing all data all the time so everything is fresh takes forever…. After couple of months it turns out they never ever open their “super important dashboard” or do it once in 6 months. Great after a year of bogging down everything I c…

Why take the unspecific "real time" answers as gospel? Just propose daily initially (or weekly or monthly), listen to their protests, and see if there are good reasons why it needs to be more often. Then pick a suitable interval that fulfills their needs and that you can guarantee. If you offer pink fluffy unicorns for free, people will always pick them, without thinking.
Post reply on HN