Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Without benchmarking LLMs, you're likely overpaying

karllorey.com

Post reply on HN
Print view View original

41–50 of 100 posts

Re: Without benchmarking LLMs, you're likely overpaying

#41

Post by verdverm »

sixtyj wrote: post #18
dpoloncsak wrote: post #9

Earlier quoted context omitted.

Yeah, one of my first projects one of my buddies asked "Why aren't you using [ChatGPT 4.0] nano? It's 99% the effectiveness with 10% the price." I've been using the smaller models ever since. Nano/mini, flash, etc.

Yup. I have found out recently that Grok-4.1-fast has similar pricing (in cents) but 10x larger context window (2M tokens instead of ~128-200k of gpt-4-1-nano). And ~4% hallucination, lowest in blind tests in LLM arena.

You use stuff from xAi and Elmo?

I'm unwilling to look past Musk's politics, immorality, and manipulation on a global scale

Re: Without benchmarking LLMs, you're likely overpaying

#42

Post by lorey »

epolanski wrote: post #15

The author of this post should benchmark his own blog for accessibility metrics, text contrast is dreadful.. On the other hand, this would be interesting for measuring agents in coding tasks, but there's quite a lot of context to provide here, both input and output would be massive.

Pushed a fix. Could you check, please?

Any resources you can recommend to properly tackle this going forward?

Re: Without benchmarking LLMs, you're likely overpaying

#43

Post by verdverm »

r_lee wrote: post #5
verdverm wrote: post #3

I'd second this wholeheartedly Since building a custom agent setup to replace copilot, adopting/adjusting Claude Code prompts, and giving it basic tools, gemini-3-flash is my go-to model unless I know it's a big and involved task. The model is really good at 1/10 the cost of pro, super fast by comparison, and some basic a/b testing shows little to no difference in output on the majority of tasks I used Cut all my sub…

Plus I've found that overall with "thinking" models, it's more like for memory, not even actual perf boost, it might even be worse because if it goes even slightly wrong on the "thinking" part, it'll then commit to that for the actual response

for sure, the difference in the most recent model generations makes them far more useful for many daily tasks. This is the first gen with thinking as a significant mid-training focus and it shows

gemini-3-flash stands well above gemini-2.5-pro

Re: Without benchmarking LLMs, you're likely overpaying

#44

Post by lorey »

jmathai wrote: post #28
andy99 wrote: post #13

Depends on what you’re doing. Using the smaller / cheaper LLMs will generally make it way more fragile. The article appears to focus on creating a benchmark dataset with real examples. For lots of applications, especially if you’re worried about people messing with it, about weird behavior on edge cases, about stability, you’d have to do a bunch of robustness testing as well, and bigger models will be better. Another…

You may also be getting a worse result for higher cost. For a medical use case, we tested multiple Anthropic and OpenAI models as well as MedGemma. Pleasantly surprised when the LLM as Judge scored gpt5-mini as the clear winner. I don't think I would have considered using it for the specific use cases - assuming higher reasoning was necessary. Still waiting on human evaluation to confirm the LLM Judge was correct.

That's interesting. Similarly, we found out that for very simple tasks the older Haiku models are interesting as they're cheaper than the latest Haiku models and often perform equally well.

Re: Without benchmarking LLMs, you're likely overpaying

#45

Post by rudhdb773b »

verdverm wrote: post #41
sixtyj wrote: post #18

Earlier quoted context omitted.

Yup. I have found out recently that Grok-4.1-fast has similar pricing (in cents) but 10x larger context window (2M tokens instead of ~128-200k of gpt-4-1-nano). And ~4% hallucination, lowest in blind tests in LLM arena.

You use stuff from xAi and Elmo? I'm unwilling to look past Musk's politics, immorality, and manipulation on a global scale

Grok is the best general purpose LLM in my experience. Only Gemini is comparable. It would be silly to ignore it, and xAI is less evil than Google these days.

Re: Without benchmarking LLMs, you're likely overpaying

#46

Post by verdverm »

rudhdb773b wrote: post #45
verdverm wrote: post #41

Earlier quoted context omitted.

You use stuff from xAi and Elmo? I'm unwilling to look past Musk's politics, immorality, and manipulation on a global scale

Grok is the best general purpose LLM in my experience. Only Gemini is comparable. It would be silly to ignore it, and xAI is less evil than Google these days.

[flagged]

Re: Without benchmarking LLMs, you're likely overpaying

#47

Post by verdverm »

phainopepla2 wrote: post #11
dpoloncsak wrote: post #9

Earlier quoted context omitted.

Yeah, one of my first projects one of my buddies asked "Why aren't you using [ChatGPT 4.0] nano? It's 99% the effectiveness with 10% the price." I've been using the smaller models ever since. Nano/mini, flash, etc.

I have been benchmarking many of my use cases, and the GPT Nano models have fallen completely flat one every single except for very short summaries. I would call them 25% effectiveness at best.

Flash is not a small model, it's still over 1T parameters. It's a hyper MoE aiui

I have yet to go back to small models, waiting for the upstream feature / GPU provider has been seeing capacity issues, so I am sticking with the gemini family for now

Re: Without benchmarking LLMs, you're likely overpaying

#48

Post by deepsquirrelnet »

lorey wrote: post #32
deepsquirrelnet wrote: post #20

This is just evaluation, not “benchmarking”. If you haven’t setup evaluation on something you’re putting into production then what are you even doing. Stop prompt engineering, put down the crayons. Statistical model outputs need to be evaluated.

This went straight to prod, even earlier than I'd opted for. What do you mean?

I’m totally in alignment with your blog post (other than terminology). I meant it more as a plea to all these projects that are trying to go into production without any measures of performance behind them.

It’s shocking to me how often it happens. Aside from just the necessity to be able to prove something works, there are so many other benefits.

Cost and model commoditization are part of it like you point out. There’s also the potential for degraded performance because of the shelf benchmarks aren’t generalizing how you expect. Add to that an inability to migrate to newer models as they come out, potentially leaving performance on the table. There’s like 95 serverless models in bedrock now, and as soon as you can evaluate them on your task they immediately become a commodity.

But fundamentally you can’t even justify any time spent on prompt engineering if you don’t have a framework to evaluate changes.

Evaluation has been a critical practice in machine learning for years. IMO is no less imperative when building with llms.

Re: Without benchmarking LLMs, you're likely overpaying

#50

Post by piskov »

hamiltont wrote: post #19

Anecdotal tip on LLM-as-judge scoring - Skip the 1-10 scale, use boolean criteria instead, then weight manually e.g. - Did it cite the 30-day return policy? Y/N - Tone professional and empathetic? Y/N - Offered clear next steps? Y/N Then: 0.5 * accuracy + 0.3 * tone + 0.2 * next_steps Why: Reduces volatility of responses while still maintaining creativeness (temperature) needed for good intuition

How come accuracy has only 50% weight?

“You’re absolutely right! Nice catch how I absolutely fooled you”

Post reply on HN