81–90 of 254 posts
So this is the dogpile.com of the askjeeves, alta vista, and lycos approach? Time is a flat circle?
Good ideas are usually still good across time and tools
You needed to search all of them to find something decent.
That's roughly analogous to today. Ignoring cost, you'd be way better off asking all the LLMs to solve a problem (like coding) where you can verify the answer.
So the question is, for things like that -> can a group of models perform better than frontier models, especially at a reasonable cost?
Fable is not a great value, so unless you're trying to find answers to Erdos questions, you can probably do better on cost.
You can probably typically ask 3 or 4 of the top Chinese models for an answer and get a response for the same Fable question... Given that Fable isn't that much better, it's not surprising you can do better for a large subset of problems.
Else, you break the cache by doing a round robin of the same conversation across different models. Likely you'll end up paying more than what it would've cost with a cache aware system
This reminds me on OpenRouters report that combining multiple different models gave comparable performance to Fable 5. I think this approach has lots of potential. Maybe OpenAi was ahead of its time with GPT-5 (it being a router to different models rather than just being one new model)
Earlier quoted context omitted.
> No benchmarks, no info on which models are used, […] The benchmarks are here → https://echo.tracerml.ai/eval/ They are not good benchmarks but at least they exist.
I've been working on a similar project and I found that it's easy to replicate Fable results if you use saturated benchmarks. In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.
Earlier quoted context omitted.
I've been working on a similar project and I found that it's easy to replicate Fable results if you use saturated benchmarks. In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.
I wouldn't be surprised if ≈7% of GPQA Diamond questions simply have the wrong answer in the ground truth data, so that getting such a question correct is graded as an error. Most machine-learning benchmarks are rather badly validated.
Earlier quoted context omitted.
Good ideas are usually still good across time and tools
Dogpile was only a good idea while Search Engines were mostly trash. You needed to search all of them to find something decent. That's roughly analogous to today. Ignoring cost, you'd be way better off asking all the LLMs to solve a problem (like coding) where you can verify the answer. So the question is, for things like that -> can a group of models perform better than frontier models, especially at a reasonable co…
Precisely.
Earlier quoted context omitted.
Dogpile was only a good idea while Search Engines were mostly trash. You needed to search all of them to find something decent. That's roughly analogous to today. Ignoring cost, you'd be way better off asking all the LLMs to solve a problem (like coding) where you can verify the answer. So the question is, for things like that -> can a group of models perform better than frontier models, especially at a reasonable co…
> Dogpile was only a good idea while Search Engines were mostly trash. Precisely.
Earlier quoted context omitted.
Sometimes expensive models are cheaper on easier tasks because they use fewer tokens, too.
Yeah apparently Opus and Sonnet are like that.
> Fable-level results at 1/3 the cost I am guessing this is not targeting those of us on the heavily subsidized $200/mo plans. Sure, these plans may be temporary, but none of us really know how temporary they are. Until then, 1/3rd of the published API pricing is not very appealing.