I’m also collecting the data my side with the hopes of later using it to fine tuning a tiny model later. Unsure whether it’ll work but if I’m using APIs anyway may as well gather it and try to bottle some of that magic of using bigger models
Without benchmarking LLMs, you're likely overpaying
61–70 of 100 posts
Re: Without benchmarking LLMs, you're likely overpaying
#62I paid a total of 13 US Dollars for all my llm usage in about 3 years. Should I analyze my providers and see if there's room for improvement?
Depends on your remaining budget ;)
Re: Without benchmarking LLMs, you're likely overpaying
#63I love the user experience for your product. You're giving a free demo with results within 5 minutes and then encourage the customer to "sign in" for more than 10 prompts. Presumably that'll be some sort of funnel for a paid upload of prompts.
https://evalry.com/question-benchmarks/game-engine-assistant... Here's a bug report, by switching the model group the api hangs in private mode.
Thanks. Will take a look.
Re: Without benchmarking LLMs, you're likely overpaying
#64I love the user experience for your product. You're giving a free demo with results within 5 minutes and then encourage the customer to "sign in" for more than 10 prompts. Presumably that'll be some sort of funnel for a paid upload of prompts.
https://evalry.com/question-benchmarks/game-engine-assistant... Here's a bug report, by switching the model group the api hangs in private mode.
Headsup I think I broke the site.
Re: Without benchmarking LLMs, you're likely overpaying
#65Earlier quoted context omitted.
https://evalry.com/question-benchmarks/game-engine-assistant... Here's a bug report, by switching the model group the api hangs in private mode.
Headsup I think I broke the site.
It's not you, it's the HN hug of death. There's so much load on the server, I'm barely able to download the redis image I need for caching...
Re: Without benchmarking LLMs, you're likely overpaying
#66Anecdotal tip on LLM-as-judge scoring - Skip the 1-10 scale, use boolean criteria instead, then weight manually e.g. - Did it cite the 30-day return policy? Y/N - Tone professional and empathetic? Y/N - Offered clear next steps? Y/N Then: 0.5 * accuracy + 0.3 * tone + 0.2 * next_steps Why: Reduces volatility of responses while still maintaining creativeness (temperature) needed for good intuition
Funny, this move is exactly what YouTube did to their system of human-as-judge video scoring, which was a 1-5 scale before they made it thumbs up/thumbs down in 2010.
I hate thumbs up/down. 2 values is too little. I understand that 5 was maybe too much, but thumbs up/down systems need an explicit third "eh, it's okay" value for things I don't hate, don't want to save to my library, but I would like the system to know I have an opinion on.
I know that consuming something and not thumbing it up/down sort-of does that, but it's a vague enough signal (that could also mean "not close enough to keyboard / remote to thumbs up/down) that recommendation systems can't count it as an explicit choice.
Re: Without benchmarking LLMs, you're likely overpaying
#67Earlier quoted context omitted.
Funny, this move is exactly what YouTube did to their system of human-as-judge video scoring, which was a 1-5 scale before they made it thumbs up/thumbs down in 2010.
I hate thumbs up/down. 2 values is too little. I understand that 5 was maybe too much, but thumbs up/down systems need an explicit third "eh, it's okay" value for things I don't hate, don't want to save to my library, but I would like the system to know I have an opinion on. I know that consuming something and not thumbing it up/down sort-of does that, but it's a vague enough signal (that could also mean "not close e…
Here's the discussion from back in the day when this changed: https://news.ycombinator.com/item?id=837698
In practice, people generally didn't even vote with two options, they voted with one!
IIRC youtube did even get rid of downvotes for a while, as they were mostly used for brigading.
Re: Without benchmarking LLMs, you're likely overpaying
#68Depends on what you’re doing. Using the smaller / cheaper LLMs will generally make it way more fragile. The article appears to focus on creating a benchmark dataset with real examples. For lots of applications, especially if you’re worried about people messing with it, about weird behavior on edge cases, about stability, you’d have to do a bunch of robustness testing as well, and bigger models will be better. Another…
One point in favor of smaller/self-hosted LLMs: more consistent performance, and you control your upgrade cadence, not the model providers. I'd push everyone to self-host models (even if it's on a shared compute arrangement), as no enterprise I've worked with is prepared for the churn of keeping up with the hosted model release/deprecation cadence.
Where can I find information on self-hosting models success stories? All of it seems like throwing tens of thousands away on compute for it to work worse than the standard providers.
The self-hosted models seem to get out of date, too. Or there ends up being good reasons (improved performance) to replace them
Re: Without benchmarking LLMs, you're likely overpaying
#69Earlier quoted context omitted.
You may also be getting a worse result for higher cost. For a medical use case, we tested multiple Anthropic and OpenAI models as well as MedGemma. Pleasantly surprised when the LLM as Judge scored gpt5-mini as the clear winner. I don't think I would have considered using it for the specific use cases - assuming higher reasoning was necessary. Still waiting on human evaluation to confirm the LLM Judge was correct.
You obviously know what you’re looking for better than me, but personally I’d want to see a narrative that made sense before accepting that a smaller model somehow just performs better, even if the benchmarks say so. There may be such an explanation, it feels very dicey without one.
Volume and statistical significance? I'm not sure what kind of narrative I would trust beyond the actual data.
It's the hard part of using LLMs and a mistake I think many people make. The only way to really understand or know is to have repeatable and consistent frameworks to validate your hypothesis (or in my case, have my hypothesis be proved wrong).
You can't get to 100% confidence with LLMs.
Re: Without benchmarking LLMs, you're likely overpaying
#70I paid a total of 13 US Dollars for all my llm usage in about 3 years. Should I analyze my providers and see if there's room for improvement?
How? All LLM-as-a-Servive's are prohibitively expensive for me. $13 over 3 years sounds too-good-to-be-true.