Let's say there's "slowness" on a route in a Rails app. A
really junior engineer might start blindly optimizing.
A mid-skill engineer might start by trying to diagnose the problem, and usually will get somewhere, but occasionally they'll run out of ideas. They might assume it has to be one of the things they've seen before, but their list is not exhaustive.
A top engineer will start by looking at the big picture and generating hypothesis. They will usually be able to generate more hypothesis and have more ways to test them than less seasoned folks. Often this is because they've seen more things.
For example, off the top of my head, the slowness could be:
- n+1 query problems
- slow remote service calls
- streaming to slow clients
- object allocations
- loops in loops--bad big O
- resource limitation (memory, CPU, IOPS)
- database connection limits
- worker process/thread limit
- frequent cache misses
- noisy neighbor problem
- lack of indexing and data size growth
A top engineer will run through their own list, and for each, they have some ideas on how to check on that item or to rule out what is unlikely. Some of them are trickier than others. The key is they have more angles that they'll know off-hand to follow up on.