> I gave it the hardest real task that fits on one machine: reverse-engineering a commercial app's license check... Respectfully, tasks that allow for explicit straightforward true/false or done/not-done tests are not the "hardest real task[s]." In fact, those are the ones that see the most gains from AI-assisted coding. Testable tasks are where the largest opportunity is.
It had a very non-standard RSA key implementation that was obfuscated heavily. As well, it has an online license check at first run, and that part typically trips up most of the local models I've tried. I've been running this test for about a year now with different models, and it was the first I've seen not only figure out the RSA key implementation, but the first that didn't just give up once it saw the online license check. Even though it's only a first-time launch check.
That's why I call it one of the hardest, because in my experience, it has been. It's the first local model I've seen pull it off end-to-end. For some of the reverse engineering work that I've done with LLMs, none have been as consistent as this particular test at highlighting a model's failure in this domain.
I have access to Daybreak Blue and I'm approved for Anthropic's Cybersecurity program, so I might run the same test with both of those just to see, because it's been a while since I used a frontier model on this test. I imagine they'll make relatively light work of it, though, assuming it doesn't trip the relaxed guardrails.