Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

seqpu.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#31

Post by stavros »

drivebyhooting wrote: post #12

That was prolix and repetitive. I wish the purported simple fixes were shown on the page.

I wish the page were just the prompt they used to generate the article. I like LLMs as much as the next person, but we don't really need two intermediate LLM layers (expand and summarise) between your brain and mine.

Edit: the author's comment below is dead, so I'll reply here: The tape and general effort is great, it's the overused LLM-style intro above that that grates. LLM writing is now like the Bootstrap of old, it's so overused that it's tedious to read.

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#32

Post by j-bos »

declan_roberts wrote: post #25

I'm very surprised at the quality of the new Gemma 4 models. On my 32 gig Mac mini I can be very productive with it. Not close to replacing paid AI by a long shot, but if I had to tighten the belt I could do it as someone who already knows how to program.

What's your setup/usecase? Enhanced intellisense?

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#33

Post by smallerize »

operatingthetan wrote: post #19
svnt wrote: post #7

> The model does not need to be retrained. It needs surgical guardrails at the exact moments where its output layer flinches. > With those guardrails — a calculator for arithmetic, a logic solver for formal puzzles, a per-requirement verifier for structural constraints, and a handful of regex post-passes — the projected score climbs to ~8.2. Surgical guardrails? Tools, those are just tools.

>It needs surgical guardrails at the exact moments where its output layer flinches. This article is very clearly shitty LLM output. Abstract noun and verb combos are the tipoff. It's actually quite horrible, it repeats lines from paragraph to paragraph.

I know that's one of the tells of AI-generated text, but if anything there's too much of it on this page. The article barely has any complete sentences. I think a human learned "sentence fragments == punchy" and then had too much fun writing at least some of this article.

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#34

Post by operatingthetan »

smallerize wrote: post #33
operatingthetan wrote: post #19

Earlier quoted context omitted.

>It needs surgical guardrails at the exact moments where its output layer flinches. This article is very clearly shitty LLM output. Abstract noun and verb combos are the tipoff. It's actually quite horrible, it repeats lines from paragraph to paragraph.

I know that's one of the tells of AI-generated text, but if anything there's too much of it on this page. The article barely has any complete sentences. I think a human learned "sentence fragments == punchy" and then had too much fun writing at least some of this article.

My guess is they used the 2b model to write the article as a proof of concept. Which did not prove the concept.

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#36

Post by fredmendoza »

ComputerGuru wrote: post #27

Seems to be llm written article and the tooling around the model is undeniably influenced by knowledge of the tests. In all cases, GPT 3.5 isn’t a good benchmark for most serious uses and was considered to be pretty stupid, though I understand that isn’t the point of the article.

really appreciate you reading the article. the benchmark data, grading, and error classes were all done by hand though. the ~8.0 is the raw model with zero tooling, and the guardrail projections are documented separately. and yeah gpt-3.5 isn't the gold standard anymore, we're on the same page there. we just wanted to show that the quality people are still paying for can be free, private, and customized to whatever you need. thanks again for taking the time to check it out.

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#38

Post by throwanem »

mrtesthah wrote: post #18
polotics wrote: post #9

Earlier quoted context omitted.

"Surgical "is the kind of wordage that LLMs seem to love to output. I have had to put in my .md file the explicit statement that the word "surgical" should only be used when referring to an actual operation at the block...

The core issue is that the LLM is using rhetoric to try to convince or persuade you. That's what you need to tell it not to do.

Which will not work. Don't think of a pink genitalia, I mean elephant...

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#39

Post by fredmendoza »

MarsIronPI wrote: post #21

> A weekend of focused work, Claude as pair programmer, no ML degree required It's not caught up if you're using Claude as your pair programmer instead of the model you're touting. Gemma 4 may be equivalent to GPT-3.5 Turbo, but GPT-3.5 isn't SOTA anymore. Opus 4.5 and 4.6 are in a different league.

good callout, want to clarify. claude helped us set up the test harness. gemma took every question alone with zero help. the ~8.0 is all gemma. and you're right, opus is in a completely different league. we're not arguing otherwise. we just found it interesting that a free 2B on a cpu matches what a lot of people are still paying for daily. every tool has a cost. some are free, some are expensive, some have rate limits. the right move is matching the tool to the job. thought it was worth showing where that floor actually is now.

Re: CPUs Aren't Dead. Gemma2B Out Scored GPT-3.5 Turbo on Test That Made It Famous

#40

Post by fredmendoza »

stavros wrote: post #31
drivebyhooting wrote: post #12

That was prolix and repetitive. I wish the purported simple fixes were shown on the page.

I wish the page were just the prompt they used to generate the article. I like LLMs as much as the next person, but we don't really need two intermediate LLM layers (expand and summarise) between your brain and mine. Edit: the author's comment below is dead, so I'll reply here: The tape and general effort is great, it's the overused LLM-style intro above that that grates. LLM writing is now like the Bootstrap of old,…

[dead]
Post reply on HN