Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post: > Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D. He makes it sounds like he got a 5-14% boos…
> “replace line 2:f1, replace range 1:a3 through 3:0e, insert after 3:0e.” Not sure what they're calculating, but this seems to me like it could be many times more efficient than 20%.
Yes, this looks like O(1) actions, where before, its likely that harnesses are ingesting and outputting huge portions of the source files for each step, and the local uses of str_replace() are themselves O(N) on the users computer. The excess reads and writes from the LLM are O(N^2).