Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Without benchmarking LLMs, you're likely overpaying

karllorey.com

Post reply on HN
Print view View original

11–20 of 100 posts

Re: Without benchmarking LLMs, you're likely overpaying

#11

Post by phainopepla2 »

dpoloncsak wrote: post #9
verdverm wrote: post #3

I'd second this wholeheartedly Since building a custom agent setup to replace copilot, adopting/adjusting Claude Code prompts, and giving it basic tools, gemini-3-flash is my go-to model unless I know it's a big and involved task. The model is really good at 1/10 the cost of pro, super fast by comparison, and some basic a/b testing shows little to no difference in output on the majority of tasks I used Cut all my sub…

Yeah, one of my first projects one of my buddies asked "Why aren't you using [ChatGPT 4.0] nano? It's 99% the effectiveness with 10% the price." I've been using the smaller models ever since. Nano/mini, flash, etc.

I have been benchmarking many of my use cases, and the GPT Nano models have fallen completely flat one every single except for very short summaries. I would call them 25% effectiveness at best.

Re: Without benchmarking LLMs, you're likely overpaying

#12

Post by walthamstow »

dpoloncsak wrote: post #9
verdverm wrote: post #3

I'd second this wholeheartedly Since building a custom agent setup to replace copilot, adopting/adjusting Claude Code prompts, and giving it basic tools, gemini-3-flash is my go-to model unless I know it's a big and involved task. The model is really good at 1/10 the cost of pro, super fast by comparison, and some basic a/b testing shows little to no difference in output on the majority of tasks I used Cut all my sub…

Yeah, one of my first projects one of my buddies asked "Why aren't you using [ChatGPT 4.0] nano? It's 99% the effectiveness with 10% the price." I've been using the smaller models ever since. Nano/mini, flash, etc.

Flash Lite 2.5 is an unbelievably good model for the price

Re: Without benchmarking LLMs, you're likely overpaying

#13

Post by andy99 »

Depends on what you’re doing. Using the smaller / cheaper LLMs will generally make it way more fragile. The article appears to focus on creating a benchmark dataset with real examples. For lots of applications, especially if you’re worried about people messing with it, about weird behavior on edge cases, about stability, you’d have to do a bunch of robustness testing as well, and bigger models will be better.

Another big problem is it’s hard to set objectives is many cases, and for example maybe your customer service chat still passes but comes across worse for a smaller model.

Id be careful is all.

Re: Without benchmarking LLMs, you're likely overpaying

#14

Post by montroser »

petcat wrote: post #2

> He's a non-technical founder building an AI-powered business. It sounds like he's building some kind of ai support chat bot. I despise these things.

The whole post is just an advert for this person's startup. Their "friend" doesn't exist...

Re: Without benchmarking LLMs, you're likely overpaying

#15

Post by epolanski »

The author of this post should benchmark his own blog for accessibility metrics, text contrast is dreadful..

On the other hand, this would be interesting for measuring agents in coding tasks, but there's quite a lot of context to provide here, both input and output would be massive.

Re: Without benchmarking LLMs, you're likely overpaying

#16

Post by candiddevmike »

andy99 wrote: post #13

Depends on what you’re doing. Using the smaller / cheaper LLMs will generally make it way more fragile. The article appears to focus on creating a benchmark dataset with real examples. For lots of applications, especially if you’re worried about people messing with it, about weird behavior on edge cases, about stability, you’d have to do a bunch of robustness testing as well, and bigger models will be better. Another…

One point in favor of smaller/self-hosted LLMs: more consistent performance, and you control your upgrade cadence, not the model providers.

I'd push everyone to self-host models (even if it's on a shared compute arrangement), as no enterprise I've worked with is prepared for the churn of keeping up with the hosted model release/deprecation cadence.

Re: Without benchmarking LLMs, you're likely overpaying

#18

Post by sixtyj »

dpoloncsak wrote: post #9
verdverm wrote: post #3

I'd second this wholeheartedly Since building a custom agent setup to replace copilot, adopting/adjusting Claude Code prompts, and giving it basic tools, gemini-3-flash is my go-to model unless I know it's a big and involved task. The model is really good at 1/10 the cost of pro, super fast by comparison, and some basic a/b testing shows little to no difference in output on the majority of tasks I used Cut all my sub…

Yeah, one of my first projects one of my buddies asked "Why aren't you using [ChatGPT 4.0] nano? It's 99% the effectiveness with 10% the price." I've been using the smaller models ever since. Nano/mini, flash, etc.

Yup.

I have found out recently that Grok-4.1-fast has similar pricing (in cents) but 10x larger context window (2M tokens instead of ~128-200k of gpt-4-1-nano). And ~4% hallucination, lowest in blind tests in LLM arena.

Re: Without benchmarking LLMs, you're likely overpaying

#19

Post by hamiltont »

Anecdotal tip on LLM-as-judge scoring - Skip the 1-10 scale, use boolean criteria instead, then weight manually e.g.

- Did it cite the 30-day return policy? Y/N - Tone professional and empathetic? Y/N - Offered clear next steps? Y/N

Then: 0.5 * accuracy + 0.3 * tone + 0.2 * next_steps

Why: Reduces volatility of responses while still maintaining creativeness (temperature) needed for good intuition

Re: Without benchmarking LLMs, you're likely overpaying

#20

Post by deepsquirrelnet »

This is just evaluation, not “benchmarking”. If you haven’t setup evaluation on something you’re putting into production then what are you even doing.

Stop prompt engineering, put down the crayons. Statistical model outputs need to be evaluated.

Post reply on HN