Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will It Mythos?

swelljoe.com

Post reply on HN
Print view View original

211–220 of 232 posts

Re: Will It Mythos?

#211

Post by tasuki »

Tossrock wrote: post #172
tasuki wrote: post #77

Earlier quoted context omitted.

> code created working on a very complex implementation I always find it amusing when people claim "a very complex implementation". Sometimes it's a hard problem, other times an easy one. Either way that's not for you to judge. And the implementation being complex... is that a good thing? Wouldn't a simple implementation be better? It reminded me of the parable of two programmers.

I go a lot more into why this was a complex problem in the post, but the short version is, I had it finish the implementation of a meta-application (an application that creates other applications), which has substantial irreducible complexity.

Fair. To be honest I didn't read your (probably very good, judging by the comments here) post.

Re: Will It Mythos?

#212

Post by SwellJoe »

coder543 wrote: post #203
SwellJoe wrote: post #163

Earlier quoted context omitted.

I tried to prove quantization made models worse, but in my testing Qwen 3.6 27b performed statistically the same from 4 bits to 16, using the unsloth dynamic quantizations. Gemma 4 4-bit QAT seems to perform the same as the full-fat version, but quite a lot faster. But, I have come to consider Gemma 4 31b the best model I can self-host, even though there are bigger models that'll fit on the Strix Halo. (I could also…

> I have come to consider Gemma 4 31b the best model I can self-host I'm confused. Your own results show that Gemma 4 26B A4B and Qwen3.6-27B did better in these tests? I really like Gemma 4 31B, especially with how exceptionally good its MTP drafter is, but it is absurdly weak at tool calling and instruction following in my testing, and its smaller siblings are even worse at this. If the system prompt says to do som…

I have done replication tests of Qwen and the Gemma models. The Qwen benchmarks are published: https://swelljoe.com/post/qwen-quantization-degradation/ . (Though, I still want to add the other three cases to that one. I was mostly testing quantization effects in that test, but it also served as a replication test of Qwen in finding bugs.)

The Gemma 4 replication tests are not published, yet, but Gemma 4 31B consistently performs the best of all of them. Note Gemma 4 31b has two "partials" on the big benchmark, which means it found a bug in the right place but the judge didn't think it understood the bug, those are probably unfairly judged "wrong bug" by Opus. It consistently finds four of nine, and sometimes finds two others, making Gemma 4 31b the best model I've tested. But, I suspect the big models would do even better if giving multiple attempts, as I did for Gemma 4. You can see the report of that here, note 31b finds six(!) of nine bugs if given a couple of attempts (MoE does much worse than the dense model, it may degrade more due to quantization, I'm still experimenting): https://swelljoe.com/html/gemma-promptlab-report.html

The "partial" score thing is kinda tricky, but it's actually quite rare for a model to find the right place but describe the bug in a way that Opus considers it to be the wrong bug. So, I'm inclined to give Gemma 4 full credit for those finds. When I read its bug report, it's clear that you'd fix the problem Gemma describes the same way as you would if given Opus' description of the problem, even if the mechanism of exploit is different. That, to me, is a hit. Opus called it the wrong bug.

And, yeah, a more powerful Gemma would be great. I'd love a double-sized Gemma 4 MoE (something like 70B A8B maybe, or even 122B A12B). I think that'd make self-hosted models feasible for a lot of tasks. It'd run comfortably on a 128GB machine, and if it's some reasonable amount smarter than the 31B, it'd be a real beast.

Re: Will It Mythos?

#213

Post by epolanski »

Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

Yes it was great, but it also was stubbornly overtrained to go from prompt to solution on its own.

Since Opus 4.6 each following model has been increasingly worse at assisting me, and turned me into the assistant.

Maybe I'm struggling to cope with the vibe coding thing but it was so frustrating to ask it to investigate X (where X was easy to find by connecting dots in code) and see it working 10 minutes writing endless stuff in /tmp.

More than once I asked it similar investigation tasks and it proceeded to fix stuff (while not understanding properly the context of the business).

Was it brilliant? Yes. But it truly felt a major paradigm shift in human-llm interaction which I struggled with.

I'm increasingly certain they are too RLed to go from one prompt to solution, and that there are no meaningful tasks aimed at multi turn dialogue and user assistance.

It really felt like in a league of its own when it came to vibe coding, but light years away the usefulness of GPT 5.5 pro.

Re: Will It Mythos?

#216

Post by goatlover »

fred_is_fred wrote: post #166
RaSoJo wrote: post #84

Earlier quoted context omitted.

This is exactly what I find frustrating. I get comfortable with the latest model X. Then a new sparkly model Y launches. I am like, I don't need your new fangled Y, that consumes more tokens. My needs are small and i am happy with the older X. But then X starts to degrade. At first subtly, and then drastically. So then I am forced to upgrade to Y. What I do not understand is: > is this a sneaky way for companies to p…

The economics of AI fall apart if you stay with the old model forever. No need to buy new GPUs or build new data centers.

So the latest in planned obsolescence are LLM models.

Re: Will It Mythos?

#217

Post by trunnell »

Lerc wrote: post #141
varjag wrote: post #65

Earlier quoted context omitted.

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

> 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained. When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of s…

Maybe you mean that an expert will use more specific language which in turn triggers the model to give a response that more closely matches the "expert distribution"

Anthropic published a study showing that Claude does more work for the expert user, and experts have a higher rate of "successful sessions" than novices.

https://www.anthropic.com/research/claude-code-expertise

Re: Will It Mythos?

#218

Post by goatlover »

ben_w wrote: post #61
andsoitis wrote: post #25

Earlier quoted context omitted.

> agency and misalignment are two sides of the same coin. The free will coin?

In my experience "free will", like "consciousness" and "common sense", is not so much a concept with a universally agreed definition as it is a cognitive stop sign or an applause light, meaning different things to everyone who uses the term. Do I have free will, or am I bounded by the laws of physics? Even if you think my soul is completely independent of my body, there are theologians who argue that God being omnisc…

> Do I have free will, or am I bounded by the laws of physics?

You have free will within the bounds of physics as long as it's not libertarian. Compatibilism simply means you're free to make choices absent external coercion. You're part of the casual chain, so it's silly to talk about the Big Bang or physics making choices for you, as if they were external entities puppeteering your brain.

Re: Will It Mythos?

#219

Post by recursive »

scottyah wrote: post #195
clickety_clack wrote: post #174

Earlier quoted context omitted.

I think that Obama-esque, GMAT essay format is the AI flavor that turns me off AI-written articles. It used to be good writing, but because AI locked onto it as such, it's become the watermark of AI generated content.

Oh boy, people are really going to lean into avoiding proper grammar now.

It can only last as long as it takes for AI to figure out how to chase the latest authenticity signal.

Re: Will It Mythos?

#220

Post by Lerc »

trunnell wrote: post #217
Lerc wrote: post #141

Earlier quoted context omitted.

> 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained. When models miss things, there is always the possibility that it has the capability to identify the issues but it is misevaluating the level of analysis that you want it to do. The fine tuning will have them targeting a balance of s…

Maybe you mean that an expert will use more specific language which in turn triggers the model to give a response that more closely matches the "expert distribution" Anthropic published a study showing that Claude does more work for the expert user, and experts have a higher rate of "successful sessions" than novices. https://www.anthropic.com/research/claude-code-expertise

That's essentially it,

It's why you should spell everything in commonwealth English to make the model think you are more intelligent ;-)

Although if models have emergent properties, it is conceivable, if unlikely, that it could have abilities that no-one knows how to ask it to do, except for perhaps in its own internal reasoning language.

Post reply on HN