Qualities that I believe make the most difference in programmers’ productivity

It was ridiculous, and I was annoyed at the time too. Especially because I had basically laid out how to solve it just before I left that company. Somehow that didn't get communicated to him. When I went back for one day 5 months later, I learned that he was basically still nowhere (he had been doing other things too). I spent an hour with him laying out the solution in detail. From what I last heard, he listened, and that is the solution they are currently using.

> At a certain point, your colleague must have realized this task was simply beyond his "off the cuff" abilities and gone looking for a more efficient way. Surely he had the presence of mind to realize he was beginning to be unproductive? How did he spend weeks on a problem that can be summarized as "retrieve a lot of records from a database"?

If you only have Python experience and no intuition about how much time/memory certain problems should take, you don't necessarily know when you're out of your depth. It was a Django shop. The only exposure most of the people in that company had to the RDBM was through the Django ORM. Multiple times I have seen them do things through the ORM which would be trivial and multiple orders of magnitude faster with a couple of lines of SQL.

That data also didn't come in a database, but in a series of CSV like files. If you don't have RDBM experience, you don't necessarily see how you would use a database for this problem. And Python conveniently has a csv module that will happily parse your files into objects... If you buy into the marketing that Python is 'fast enough', and 'you can always use Pypy if it needs to be faster', you might not realise when you get stuck.

> Furthermore, what was your organization like that someone that grossly incompetent wasn't checked in on about progress? There was no "mercy rule"? No referee to call it when it was clear it was taking weeks instead of hours or days?

Management didn't know how long that task was supposed to take. The technical competence in that company was very limited, and they were used to deadlines slipping all the time.

> I'm self-aware enough to search for a solution and RTFM for the tools at hand.

That puts you ahead of the pack, in my experience. I had almost no experience with RDBMs before I joined that company, but reading the first 100 pages of the PostgreSQL manual the first week put me ahead of people who had been using PostgreSQL as their principal tool for more than a year (a different team, the metrics team which used PostgreSQL directly, not through the Django ORM). They had never heard of simple things like VIEWs and WITH queries.

> I've never encountered a team where it seemed like one person was that far off from the median

I don't think he was that far off from the median in that company, maybe just a bit below. To be fair, it was technically the worst company I've ever worked at. I was only with them for 6 months (and I never would have started there except for personal circumstances).

Consider yourself lucky never having worked at such a company. They are more common than you think.

Probably the biggest aspect not dealt with in these writings along with the discussion around it (is there or is there not such a beast) is bias.

For example, selection bias: e.g. my view over what was the best / worst programmer was much different when working in different teams. I found out there could be much worse than what I thought is the worse. Then I learned that there could be much worse... It's like a fractal :) As you notice these differences a 10x difference doesn't seem that crazy. It's really things that should take a few weeks, which in turn take months or years or never get done.

Also like any other optimization problem, optimizing software development is about hitting a moving target. When the team is balanced, it may be the process that will become a bottleneck, etc.