Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Without benchmarking LLMs, you're likely overpaying

karllorey.com

Post reply on HN
Print view View original

91–100 of 100 posts

Re: Without benchmarking LLMs, you're likely overpaying

#91

Post by steveklabnik »

machomaster wrote: post #86
giobox wrote: post #79

Earlier quoted context omitted.

Youtube always kept downvotes and the 'dislike' button, the change (which still applies today) was that they stopped displaying the downvote count to users - the button never went away though. Visit a youtube video today, you can still upvote and downvote with the exact same thumbs up or down, the site however only displays to you the count of upvotes. The channel owners/admins can still see the downvote count and th…

There is also an independent "Return Youtube Dislike" browser extension that shows the dislike numbers. It's very convenient.

That doesn't show the real number, only "a combination of scraped dislike stats and estimates extrapolated from extension user data."

Re: Without benchmarking LLMs, you're likely overpaying

#92

Post by [deleted] »

piskov wrote: post #50
hamiltont wrote: post #19

Anecdotal tip on LLM-as-judge scoring - Skip the 1-10 scale, use boolean criteria instead, then weight manually e.g. - Did it cite the 30-day return policy? Y/N - Tone professional and empathetic? Y/N - Offered clear next steps? Y/N Then: 0.5 * accuracy + 0.3 * tone + 0.2 * next_steps Why: Reduces volatility of responses while still maintaining creativeness (temperature) needed for good intuition

How come accuracy has only 50% weight? “You’re absolutely right! Nice catch how I absolutely fooled you”

[deleted]

Re: Without benchmarking LLMs, you're likely overpaying

#93

Post by rudhdb773b »

naught0 wrote: post #80
rudhdb773b wrote: post #45

Earlier quoted context omitted.

Grok is the best general purpose LLM in my experience. Only Gemini is comparable. It would be silly to ignore it, and xAI is less evil than Google these days.

When's the last time Sundar Pichai did a Hitler salute or had his creation calling itself "Mecha Hitler"?

In the big picture, those events are insignificant compared to the negative impacts on society from Google's trillion dollar advertising business and the associated destruction of privacy.

Re: Without benchmarking LLMs, you're likely overpaying

#94

Post by PunchyHamster »

rednafi wrote: post #78
PunchyHamster wrote: post #72

Earlier quoted context omitted.

> IIRC youtube did even get rid of downvotes for a while, as they were mostly used for brigading. No, they got rid of them most likely because advertisers complained that when they dropped some flop they got negative press from media going "lmao 90% dislike rate on new trailer of ". Stuff disliked to oblivion was either just straight out bad, wrong (in case of just bad tutorials/info) and brigading was very tiny perc…

Oh, didn't they remove the dislike count after people absolutely annihilated one of their yearly rewind with dislikes?

well, people annihilated every of their rewinds with dislikes. But yeah, that might've contributed.

Re: Without benchmarking LLMs, you're likely overpaying

#95

Post by iugtmkbdfil834 »

steveklabnik wrote: post #91
machomaster wrote: post #86

Earlier quoted context omitted.

There is also an independent "Return Youtube Dislike" browser extension that shows the dislike numbers. It's very convenient.

That doesn't show the real number, only "a combination of scraped dislike stats and estimates extrapolated from extension user data."

I think that just the absence in official app and the existence of this tool makes this point largely irrelevant. Company in question could easily reverse this decision overnight as the data exist, but absent that people adjust to an available proxy estimate. It is interesting though, because it shows clear intent of "we don't want to show actual sentiment".

Re: Without benchmarking LLMs, you're likely overpaying

#96

Post by direwolf20 »

rednafi wrote: post #78
PunchyHamster wrote: post #72

Earlier quoted context omitted.

> IIRC youtube did even get rid of downvotes for a while, as they were mostly used for brigading. No, they got rid of them most likely because advertisers complained that when they dropped some flop they got negative press from media going "lmao 90% dislike rate on new trailer of ". Stuff disliked to oblivion was either just straight out bad, wrong (in case of just bad tutorials/info) and brigading was very tiny perc…

Oh, didn't they remove the dislike count after people absolutely annihilated one of their yearly rewind with dislikes?

It was removed after some presidential speeches attracted heavy dislikes.

Re: Without benchmarking LLMs, you're likely overpaying

#97

Post by machomaster »

steveklabnik wrote: post #91
machomaster wrote: post #86

Earlier quoted context omitted.

There is also an independent "Return Youtube Dislike" browser extension that shows the dislike numbers. It's very convenient.

That doesn't show the real number, only "a combination of scraped dislike stats and estimates extrapolated from extension user data."

The official youtube stats (views, comments, upvotes) are not real/real-time either. But that's the best we have. And dislike numbers are in the same universe of credibility and closeness to reality. It's definitely good enough.

If you want downvote data be more precise, do your part and install the extension! :-)

Re: Without benchmarking LLMs, you're likely overpaying

#98

Post by naught0 »

rudhdb773b wrote: post #93
naught0 wrote: post #80

Earlier quoted context omitted.

When's the last time Sundar Pichai did a Hitler salute or had his creation calling itself "Mecha Hitler"?

In the big picture, those events are insignificant compared to the negative impacts on society from Google's trillion dollar advertising business and the associated destruction of privacy.

fair points, but we'll have to see now that grok is in the pentagon. sky's the limit

Re: Without benchmarking LLMs, you're likely overpaying

#99

Post by gforce_de »

iFire wrote: post #56

I love the user experience for your product. You're giving a free demo with results within 5 minutes and then encourage the customer to "sign in" for more than 10 prompts. Presumably that'll be some sort of funnel for a paid upload of prompts.

Wow - interesting how strong the differences are!

What seems missing: I can not see the answer from the different models. One have to rely on the "correctness" score.

Another minor thing: the scoring seems hardcoded to: 50% correctness, 30% cost, 20% latency - which is OK, but in my case i care more about correctness and latency I don't care.

Wow! This was my testprompt:

  You are an expert linguist and translator engine.  
  Task: Translate the input text from English into the languages listed below.  
  Output Format: Return ONLY a valid, raw JSON object.  
  Do not use Markdown formatting (no ```json code blocks).  
  Do not add any conversational text.
  
  Keys: Use the specified ISO 639-1 codes as keys.
  
  Target Languages and Codes:  
  - English: "en" (Keep original or refine slightly)  
  - Mandarin Chinese (Simplified): "zh"  
  - Hindi: "hi"  
  - Spanish: "es"  
  - French: "fr"  
  - Arabic: "ar"  
  - Bengali: "bn"  
  - Portuguese: "pt"  
  - Russian: "ru"  
  - German: "de"  
  - Urdu: "ur"
  
  Input text to translate:  
  "A smiling boy holds a cup as three colorful lorikeets perch on his arms and shoulder in an outdoor aviary."

Re: Without benchmarking LLMs, you're likely overpaying

#100

Post by vercaemert »

andy99 wrote: post #39
jmathai wrote: post #28

Earlier quoted context omitted.

You may also be getting a worse result for higher cost. For a medical use case, we tested multiple Anthropic and OpenAI models as well as MedGemma. Pleasantly surprised when the LLM as Judge scored gpt5-mini as the clear winner. I don't think I would have considered using it for the specific use cases - assuming higher reasoning was necessary. Still waiting on human evaluation to confirm the LLM Judge was correct.

You obviously know what you’re looking for better than me, but personally I’d want to see a narrative that made sense before accepting that a smaller model somehow just performs better, even if the benchmarks say so. There may be such an explanation, it feels very dicey without one.

You just need a robust benchmark. As long as you understand your benchmark, you can trust the results.

We have a hard OCR problem.

It's very easy to make high-confidence benchmarks for OCR problems (just type out the ground truth by hand), so it's easy to trust the benchmark. Think accuracy and token F1. I'm talking about highly complex OCR that requires a heavyweight model.

Scout (Meta), a very small/weak model, is outperforming Gemini Flash. This is highly unexpected and a huge cost savings.

Some problems aren't so easily benchmarked.

Post reply on HN