11–20 of 26 posts
Interesting. Did you give the agents any skills for playing civ? If not, are you planning to?
Have you tried playing the agents yourself? Do they crush human competition?
This is great. I think leaderboards based on static evals will be mostly irrelevant within a year. Continuous benchmarks like this are the only way to get signal on frontier models You mention Opus 4.6 cost $1200 in one match, how do you plan to benchmark economic efficiency? Looking at a performance vs. cost trade-off you might say a model that plays 80% as well at 1% of the cost is more impressive than the 'top' mo…
In the leaderboards part of the page I'll be autopopulating the token cost of the model as a metric to evaluate on
Incredible and important product. Necessary for developers, users, and industries that want to use agents. Can’t wait to see how it’ll grow
This is an amazing eval metric that no one thought about! such a creative idea. Have you thought of other games? how different it is from chess?