Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Without benchmarking LLMs, you're likely overpaying

karllorey.com

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Without benchmarking LLMs, you're likely overpaying

#81

Post by verdverm »

PunchyHamster wrote: post #73
verdverm wrote: post #3

I'd second this wholeheartedly Since building a custom agent setup to replace copilot, adopting/adjusting Claude Code prompts, and giving it basic tools, gemini-3-flash is my go-to model unless I know it's a big and involved task. The model is really good at 1/10 the cost of pro, super fast by comparison, and some basic a/b testing shows little to no difference in output on the majority of tasks I used Cut all my sub…

LLM bubble will burst the second investors figure out how much well managed local model can do

Except that

1. There is still night and day difference

2. Local is slow af

3. The vast majority of people will not run their own models

4. I would have to spend more than $200+ a month on frontier AI to come close the same price it would cost for any decent AI at home rig. Why would I not use frontier models at this point?

Re: Without benchmarking LLMs, you're likely overpaying

#83

Post by xmcqdpt2 »

This is useful when selecting a model for an initial application. The main issue I'm concerned about though is ongoing testing. At work we have devs slinging prompt changes left and right into prod, after "it works on my machine" local testing. It's like saying the words "AI" is sufficient to get rid of all engineering knowledge.

Where is TDD for prompt engineering? Does it exist already?

Re: Without benchmarking LLMs, you're likely overpaying

#84

Post by cap11235 »

xmcqdpt2 wrote: post #83

This is useful when selecting a model for an initial application. The main issue I'm concerned about though is ongoing testing. At work we have devs slinging prompt changes left and right into prod, after "it works on my machine" local testing. It's like saying the words "AI" is sufficient to get rid of all engineering knowledge. Where is TDD for prompt engineering? Does it exist already?

Evals have always existed, and not using them when building systems is relying on superstition.

Re: Without benchmarking LLMs, you're likely overpaying

#85

Post by machomaster »

rednafi wrote: post #78
PunchyHamster wrote: post #72

Earlier quoted context omitted.

> IIRC youtube did even get rid of downvotes for a while, as they were mostly used for brigading. No, they got rid of them most likely because advertisers complained that when they dropped some flop they got negative press from media going "lmao 90% dislike rate on new trailer of ". Stuff disliked to oblivion was either just straight out bad, wrong (in case of just bad tutorials/info) and brigading was very tiny perc…

Oh, didn't they remove the dislike count after people absolutely annihilated one of their yearly rewind with dislikes?

The original sin is argued to be the Youtube Rewind 2018. But it took them until 2021 to roll it out.

Re: Without benchmarking LLMs, you're likely overpaying

#86

Post by machomaster »

giobox wrote: post #79
steveklabnik wrote: post #67

Earlier quoted context omitted.

Here's the discussion from back in the day when this changed: https://news.ycombinator.com/item?id=837698 In practice, people generally didn't even vote with two options, they voted with one! IIRC youtube did even get rid of downvotes for a while, as they were mostly used for brigading.

Youtube always kept downvotes and the 'dislike' button, the change (which still applies today) was that they stopped displaying the downvote count to users - the button never went away though. Visit a youtube video today, you can still upvote and downvote with the exact same thumbs up or down, the site however only displays to you the count of upvotes. The channel owners/admins can still see the downvote count and th…

There is also an independent "Return Youtube Dislike" browser extension that shows the dislike numbers. It's very convenient.

Re: Without benchmarking LLMs, you're likely overpaying

#87

Post by lorey »

xmcqdpt2 wrote: post #83

This is useful when selecting a model for an initial application. The main issue I'm concerned about though is ongoing testing. At work we have devs slinging prompt changes left and right into prod, after "it works on my machine" local testing. It's like saying the words "AI" is sufficient to get rid of all engineering knowledge. Where is TDD for prompt engineering? Does it exist already?

This is a very good point. When I came in, the founder did a lot of evaluation based on a few prompts and with manual evaluation, exactly as described. Showing the results helped me underline the fact that "works for me" (tm) does not match the actual data in many cases.

Re: Without benchmarking LLMs, you're likely overpaying

#88

Post by lorey »

cap11235 wrote: post #84
xmcqdpt2 wrote: post #83

This is useful when selecting a model for an initial application. The main issue I'm concerned about though is ongoing testing. At work we have devs slinging prompt changes left and right into prod, after "it works on my machine" local testing. It's like saying the words "AI" is sufficient to get rid of all engineering knowledge. Where is TDD for prompt engineering? Does it exist already?

Evals have always existed, and not using them when building systems is relying on superstition.

This is true with one caveat.

In most cases, e.g. with regular ML, evals are easy and not doing them results in inferior performance. With LLMs, especially frontier LLMs, this has flipped. Not doing them will likely give you alight performance and at the same time proper benchmarks are tricky to implement.

Re: Without benchmarking LLMs, you're likely overpaying

#89

Post by lorey »

PeterStuer wrote: post #82

All true, but from what I see in the field it is most often an "ain't nobody got time for that" as teams rush into adoption the costs be dammed for now. We'll deal with it only if cost becomes a major issue.

Haha, very true. Exactly as described in the article.
Post reply on HN