Earlier quoted context omitted.
>I'm interested to hear about any other data representations you'd like to see, too It would be nice if you can show how much the models drift from the instructions over time
Not sure what you mean. Time series chart of model performance over time to see if proprietary models get degraded? That's in the works, but we will need a couple months more data collection before launch.
Yes, probably performance helps.
The idea is that the larger a coding task is and the longer the coding agent is, the higher the chance is for the agent to not follow the rules and guidelines.