11–20 of 93 posts
Are you freaking kidding me with YC throwing money at something like this? I guess I can fund raise just by having built https://maki.sh , and months ahead of other founders too...
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.
Earlier quoted context omitted.
Thanks for bringing this up and trying it out! We've disabled Cmd+Option+I in the new version and also have a message during signup about sharing chats with Bullet, let us know if you run into any other issues!
> We've disabled Cmd+Option+I in the new version Bulletproof.
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
Are you freaking kidding me with YC throwing money at something like this? I guess I can fund raise just by having built https://maki.sh , and months ahead of other founders too...
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
I guess however this is a harness and it needs to connect to the API?
This is a promising direction! Unfortunately, I think the benchmark result here is essentially meaningless. I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai ). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mi…
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
This is a promising direction! Unfortunately, I think the benchmark result here is essentially meaningless. I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai ). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mi…
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
Congrats on the launch. I can't find which model providers are supported? Are you calling the claude-code CLI directly and make bullet usable with anthropic subscriptions like orca or herdr? I guess however this is a harness and it needs to connect to the API?
This is a promising direction! Unfortunately, I think the benchmark result here is essentially meaningless. I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai ). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mi…
The breakdown with which model, per-task cost, and methodology is in the "Full results and methodology" link in the post, not omitted. Definitely check it out if you haven't. On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publis…