Live data from Hacker News

Squeeze the hell out of the system you have

blog.danslimmon.com

291–300 of 383 posts

Re: Squeeze the hell out of the system you have

#291
post #249

Earlier quoted context omitted.

That is a hot take... ;) But joins should never impact performance in a large way if they're on the same server and properly indexed. "It's truly amazing how much faster everything is when you eliminate joins" is just not true if you're using joins correctly. Sadly, many developers simply never bother to learn. On the other hand, having to write a piece of data to 20 different spots instead of 1 is going to be dramat…

Joins are not inherently expensive, but they can lead to expensive queries. For example, say I want to find the 10 most recent users with a phone number as their primary contact method: SELECT … FROM User JOIN ContactMethod on ContactMethod.userId = User.id WHERE ContactMethod.priority = ‘primary’ AND ContactMethod.type = ‘phoneNumber’ ORDER BY User.createdAt DESC LIMIT 10 If there are a very large number of users, a…

For 10 million users + telephones, this takes 1ms.

    create table users (
        id serial primary key not null,
        created_at timestamp not null default now()
    );

    create table users_telephones (
        user_id int references users(id) not null,
        is_primary boolean not null default true,
        telephone varchar not null
    );

    insert into users
    select i, NOW() + (random() * (interval '90 days')) + '30 days' from generate_series(1, 10000000) i;
    insert into users_telephones select id, true, random() :: text from users limit 10000000; -- all users have a primary telephone
    insert into users_telephones select id, false, random() :: text from users limit 200000; -- some users have a non primary telephone
    create index on users(created_at);
    create index on users_telephones(user_id);
    create index on users_telephones(user_id, is_primary) where is_primary;

    select count(*) from users;
    count   
    ----------
    10000000
    (1 row)

    Time: 160.911 ms


    select count(*) from users_telephones;
    count   
    ----------
    10200000
    (1 row)

    Time: 176.361 ms


    select
        *
    from
        users u
        join users_telephones ut on u.id = ut.user_id
    where
        ut.is_primary
    order by
        created_at
    limit
        10;

    id    |         created_at         | user_id | is_primary |     telephone      
    ---------+----------------------------+---------+------------+--------------------
    9017755 | 2023-09-11 11:45:37.65744  | 9017755 | t          | 0.7182410419408853
    6061687 | 2023-09-11 11:45:39.271054 | 6061687 | t          | 0.3608686654204689
    9823470 | 2023-09-11 11:45:39.284201 | 9823470 | t          | 0.3026398665522869
    2622527 | 2023-09-11 11:45:39.919549 | 2622527 | t          | 0.1929579716250771
    7585920 | 2023-09-11 11:45:40.256742 | 7585920 | t          | 0.3830236472843005
    5077138 | 2023-09-11 11:45:41.076164 | 5077138 | t          | 0.9058939392225689
    1496883 | 2023-09-11 11:45:42.459194 | 1496883 | t          | 0.1519510558344308
    9234364 | 2023-09-11 11:45:42.965896 | 9234364 | t          | 0.8254433522266105
    6988331 | 2023-09-11 11:45:43.130548 | 6988331 | t          | 0.9577098184736457
    7916398 | 2023-09-11 11:45:43.559425 | 7916398 | t          | 0.9681218675498862
    (10 rows)

    Time: 0.973 ms

Re: Squeeze the hell out of the system you have

#292

Earlier quoted context omitted.

Very curious to learn more about what the monolith was doing so incredibly poorly that you managed to squeeze that much performance out of it. Poorly written queries? Too many queries? Lack of any caching? Doing things synchronously when they could've been done concurrently?

Some of that, some other bad practices. Lots of low-hanging fruit, then more esoteric changes. https://justinlloyd.li/blog/how-much-cache-you-got-on-you/

He cached everything and delayed writes too. It's easy to make a system fast when it's not realtime.

Re: Squeeze the hell out of the system you have

#293

The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…

I would rather just use a cache.

Re: Squeeze the hell out of the system you have

#294
post #115

Earlier quoted context omitted.

There are "tall" applications and "wide" applications. Almost all advice you ever read about database design and optimization is for "tall" applications. Basically, it means that your application is only doing one single thing, and everything else is in service of that. Most of the big tech companies you can think of are tall. They have only a handful of really critical, driving concepts in their data model. Facebook…

Thanks I think this is a really interesting way to look at things. What is the market for "wide" applications though? It seems like any particular business can only really support one or two of them, for some that will be SAP and for others it might be Salesforce (if they don't need much ERP), or (as you mentioned) some giant semi homebrewed Oracle thing. Usually there is a legacy system which is failing but still ru…

> What is the market for "wide" applications though?

Just my experience, but essentially these target industries, not necessarily consumers or singular entities. Hence the term "enterprise". As someone who worked on a fairly reasonable ERP for academic purposes, even just calculating a GPA is extremely complicated in the backend:

    * There are multiple schemes for calculating GPAs
    * Each scheme needs to support multiple grading types (A-F, pass/fail, etc)
    * Each scheme needs to support multiple rounding rules
    * Displays of GPAs will need to be scaled properly based on the output context
    * GPA values will need to be normalized for use in calculations in other parts of the system
    * State legislatures mandate state-specific usages of GPAs which must be honored for legal compliance
    * All GPA calculations must have historical context in case the rules changes so that old transcripts can be revived correctly
    * Institutions themselves will have custom rules (maybe across schools or departments) for calculations which must be incorporated into everything else
    * This pretty much has to work every time
I don't know exactly how many tables GPAs themselves took, but overall the system was over 4,000 tables and 10,000+ stored procedures/functions. Also, I worked in the State of Texas which has its own institution-supported entity performing customizations to this ERP for multiple universities that are installed separately but required for full compliant operation.

I would compare this to most modern "tall" applications which would more-than-likely offer you maybe up to 3 different GPA options with some basic data syncing or something. They might offer multiple rounding types if they thought that far. These apps are generally extremely niche and typically work for very basic workloads. They can capture a lot of easy value for entry-level stuff but immediately fail at everything else.

Re: Squeeze the hell out of the system you have

#295

Earlier quoted context omitted.

There is certainly an API to inspect your query, you can just call print() on the object iirc.

The problem is with using `session.add(obj)` instead of `session.scalars(insert(TheClass).returning(TheClass), data)`. If there's a way to get generated SQL from an AsyncSession, please do let me know.

I think you have a couple options [0], I currently have a project that uses 'echo' in debug mode.

[0]: https://stackoverflow.com/questions/27748053/how-to-log-sql-...

Re: Squeeze the hell out of the system you have

#296
post #277

Loads of over-engineering decisions would be avoided if devs understood how to read EXPLAIN/ANALYZE results and do the proper indexing/query optimization. Log queries, filter our the ones that are very frequent or take loads of time to execute, cache the frequent ones, optimize the fat ones, do this systematically and your system will be healthier. Things that help massively from my experience: - APM - slow query log…

Do you know of any good resources to understand sql explain plan. In my current project, we are facing a lot of issues related to query performance on MS SQL server. Do we need to always specify index hint with queries. Sometimes index exists but query does not seem to be using the index. I am thinking using sql execution plan could help us understand this issue better. tia.

Re: Squeeze the hell out of the system you have

#297
post #277

Loads of over-engineering decisions would be avoided if devs understood how to read EXPLAIN/ANALYZE results and do the proper indexing/query optimization. Log queries, filter our the ones that are very frequent or take loads of time to execute, cache the frequent ones, optimize the fat ones, do this systematically and your system will be healthier. Things that help massively from my experience: - APM - slow query log…

Can't even count how many "next Gen architecture" sessions I've been at which certainly could've been replaced with due diligence on the current implementation.

You don't fix bad coding with a new architecture. That just puts the problem off by some time.

Re: Squeeze the hell out of the system you have

#298

Earlier quoted context omitted.

There is certainly an API to inspect your query, you can just call print() on the object iirc.

The problem is with using `session.add(obj)` instead of `session.scalars(insert(TheClass).returning(TheClass), data)`. If there's a way to get generated SQL from an AsyncSession, please do let me know.

This is a very blunt tool, but `engine.echo = True` prints all SQL going to the DB.

(I don't have any experience with AsyncSession, so cannot contribute something more specific)

Re: Squeeze the hell out of the system you have

#299
post #272

Earlier quoted context omitted.

If the tables involved in the join are of 100M+ records what I do when the joins use varchar columns to improve the performance is to use an additional integer column of the varchar one that is a CRC of it (or hash if you prefer that) and use the integer one instead in the join.

That seems weirdly convoluted. So you store two columns to represent the foreign key?

If you use a varchar as FK then you're definitely doing something wrong from beginning. OP was talking about getting the phone number under certain conditions, and a phone number column is a varchar.

Re: Squeeze the hell out of the system you have

#300
post #55

Earlier quoted context omitted.

Interestingly, I often ask candidates about optimising a slow running db query and the majority of people jump to adding caching and very few ask if they can run an explain or see the indexes.

"I would make the slow query faster" seems too obvious an answer for an interview question.

Haha, sure, but the very first thing you should ask when faced with a slow query is to see the “explain analyse” output.

Caching, in any form, is the last thing you want to reach for because it’s always more nuanced than you anticipate.

To clarify, when asking the question it’s after drilling through the layers from the frontend -> backend -> query and the actual query is on the screen along with some table metrics as a guide.

Post reply on HN