The Death of Software Engineering by a Thousand Prompts

Once they hit some threshold of project size, they overcommit, bite off too much or don't recognize that they're missing some context. Agents help this by allowing them to see their mistakes and try again, but eventually you hit some death loop.

I think someone around now-ish will realize that there should be two separate RLHF tunes--one for the initial prototype, and another for the hard engineering that follows. I doubt its that hard to make an methodical, engineering-minded tune, but the emphasis has been on the flashy demos and the quick wins. Cursor and folks should be collecting this data as we speak, and I expect curmudgeony agents to start appearing within a year.

Combine this with better feedback loops (e.g. mcp-accessible debuggers), the agent doing its own stackoverflow/github searches, and continued efficiency work driving token costs down by an order-of-magnitude every year or so, and agents will get very very good very fast.

In this atmosphere, humans will shortly exist to get context for the agent that it can't fetch itself, either for security reasons, or because no one's built the integration yet. And that will be short-lived, because integrations will always be built.

So I guess there's a window for the "copilot" reality, but it feels very very brief. I don't think agents will need humans for very long.

This sounds like outsourcing on steroids. Joke aside, what the software engineering will become really depends on the growth of the industry. Many people thought that most of the software engineering jobs would be outsourced to India and software engineer as a profession would soon die in the US. It turned out that the investment to software engineering far outpaced outsourcing, and as software engineers we were incredibly lucky to work in this field. The trend will not last forever, though. If it turns out that the growth areas in the world do not require much of novel software engineering, then the demand of this profession will dwindle, and the investment will diminish. As a result, our jobs will be outsourced or replaced by AI to a large degree, as AI is really good at slicing and dicing mature code for mature use cases.

I'm not so optimistic on this if AI prevails. Think about the chip industry. It's an incredibly challenging field for only the top few to truly understand the art of chip design, yet even the top engineers may not necessarily have the same "lucrative" packages compared to the software engineers in the same percentile, let alone the pay of industry average.

In the end, it is the supply and demand that determines our packages. AI can suppress demand to the point that the entire industry needs fewer senior engineers than now, and we will then be paid less accordingly.

There's a couple possibilities with this:

1. Agents become so powerful that a human can't conceivably keep up with them. And, it becomes a drain on efficiency for any human to try. The only important things are wether or not the prompt fits the desired outcome, and is the creation 'safe'. Safe can mean many things. Will not crash, will not leak data, will not take over the world... Atlas Computing is one startup that's taking this view. By ensuring an AI can only do 'safe' things as defined by some formal ontology/methods.

2. A human stays in the loop, and tries to stay at least reasonably up to date on the code architecture. For this to work long term, the weak link is the human understanding. In which case there's interesting opportunities for AI-generated lessons, animations, and examples that are used to get the human up to speed as fast as possible. If I see a very nice 3Blue1Brown style animation generated by AI about how a piece of software functions, than I can probably start working with it more quickly than if I only had the code. At least if the animation links very closely with the code itself.

A big difference here is that typesetting can be done by individuals as an ongoing task accompanying writing. So, software indeed replaces this profession. On the other hand, we do software engineering all day long. It's a stand-alone profession. A more relatable historical example should be automation in chip industry. With CAD and automation, the chip industry requires a lot fewer engineers, even though chip design and manufacturing are still very challenging. But then, this probably has more to do with limited investment (therefore limited demand for talent) than how much automation in the field.