I have a large monorepo that includes about 15 TypeScript services and many Rust services. Everything is well-documented and organized, with standardized and structured custom code.
When an issue arises, I often test the systems by providing a minimal prompt, like: "this user, this is their email, this isn't working, figure it out in production." I send this to both Opus and ChatGPT, but it doesn't help. I've set up Agents.md and Quote.md identically, with the same access and linkers, so the Harness is consistent.
ChatGPT rarely succeeds. If the task is complex and requires a multi-step process to identify the true cause, ChatGPT usually stops after a few initial ideas and wrongly claims it has found the solution.
- For simple tasks, like identifying a missing item in a to-do list, ChatGPT performs well.
- However, for issues like memory leaks or file system corruption, it struggles.
On the other hand, Opus 4.8 always finds the solution, albeit slowly. I can rely on it without worrying about whether it will succeed. It just gets the job done.
Recently, Fable 5 has emerged, which resolves issues without needing any prompts. It operates even faster than Opus.
When I ask ChatGPT or Opus to create a new feature:
- ChatGPT often produces superficial results, ignoring existing code and building unnecessary independent code.
- Interestingly, the outcome from ChatGPT appears functional, but it's usually incorrect, focusing on a superficial "aha!" moment.
Opus, however, plans thoroughly, executes, and cleans up, ensuring everything works correctly. If needed, I can provide more realistic examples, though it's challenging due to the monorepo's size and complexity, with hundreds of thousands of lines of code.