Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will It Mythos?

swelljoe.com

Post reply on HN
Print view View original

81–90 of 232 posts

Re: Will It Mythos?

#81

Post by SwellJoe »

coldtea wrote: post #76
SwellJoe wrote: post #53

Earlier quoted context omitted.

Early on, I had a vague suspicion that the reason some of the Chinese models, including quite small ones, perform so well on this task, especially relative to their size and cost, is because they don't have the same safety guardrails baked in regarding software security that US models seem to have. Gemini 3.1 Pro doing so poorly sort of reinforced that gut feeling. But, then Gemma 4 proved to be extraordinarily good…

> But, then Gemma 4 proved to be extraordinarily good for its size (better than Qwen), and kinda disproved that US models are any weaker at small sizes. Did it "disprove" it retroactively or just changed what the situation is, given that until then they were indeed weaker at small sizes?

I don't know. I think it proves that if Google is baking guardrails into their models that prevent them from finding security bugs, they didn't bake those guardrails into Gemma 4, because it is very good at it. Maybe that means Google devs had a change of heart. Maybe it means something about Gemma 4 architecture is better for this task than Gemini 3.1 Pro. Gemini Flash 3.5 did OK though.

Anyway, I kinda think among US models only Fable really tries to block security work like this, based on my experience so far.

Re: Will It Mythos?

#82

Post by raphman »

mejutoco wrote: post #72
raphman wrote: post #16

Earlier quoted context omitted.

> It's been really frustrating that neither Codex nor Opus can make targetted edits to Fable's code without screwing something subtle up. Reminds me of the old adage: don't try to be too smart when writing code. Otherwise, dumber people - including your future self - will have trouble working with it.

For reference: if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it

Ah thanks - I couldn't remember the original version.

For reference: it's called Kernighan's Law, and can be found in the Second Edition of "The Elements of Programming Style", page 10 [1].

The original phrasing is:

> Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?

[1] https://archive.org/details/the-elements-of-programming-styl...

Re: Will It Mythos?

#83

Post by mirsadm »

varjag wrote: post #65
Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

To me it feels like they're basically tweaking these things around the edges. I'm not seeing any difference in capability just preference. This has been the case for a while.

Re: Will It Mythos?

#84

Post by RaSoJo »

airstrike wrote: post #54

Around February, Opus 4.6 was excellent. Smart, fast, proactive. Then it got lobotomized and it's never been the same after that nerf. 4.7 came along and it too was disappointing—not unlike 4.8, which despite feeling a smidge smarter, tends to write word salad and is basically unusable for some workflows. Fable felt like having access to that "old Opus" again, but a little smarter. Sort of like I'd expect an Opus 5 t…

This is exactly what I find frustrating. I get comfortable with the latest model X. Then a new sparkly model Y launches. I am like, I don't need your new fangled Y, that consumes more tokens. My needs are small and i am happy with the older X.

But then X starts to degrade. At first subtly, and then drastically. So then I am forced to upgrade to Y.

What I do not understand is:

> is this a sneaky way for companies to push users up the chain?

> Or is this a genuine fault in model design/resource allocation?

Re: Will It Mythos?

#85

Post by TeMPOraL »

hypfer wrote: post #27
sigbottle wrote: post #26

Earlier quoted context omitted.

To be fair, labs silently nerf models all the time. Fable's probably objectively better at full power. I mean, I definitely felt the same difference in competency between Fable and current Opus. But Opus itself has definitely been nerfed, and Fable, even if it comes back the public forever (probably won't), will get nerfed.

I remember a time where a product didn't suddenly get worse while you were blinking. That was a nice time. Let us get back to that time. Use open weights models. Own stuff.

That was before SaaS became a thing. Products didn't degrade over time because they couldn't easily reach out to your machine and remotely overwrite bytes on the CD-ROM the product came on.

Re: Will It Mythos?

#86

Post by jonplackett »

I thought the whole point was that it doesn’t need to be pointed at the problem. That’s a much easier problem to solve. Also you eliminate 10000 false positives.

Re: Will It Mythos?

#87

Post by SwellJoe »

jonplackett wrote: post #86

I thought the whole point was that it doesn’t need to be pointed at the problem. That’s a much easier problem to solve. Also you eliminate 10000 false positives.

They were not pointed at the problem. You're reading the section about corpus selection and mixing it up with the benchmark rules.

And, false positives are reported in the results.

Re: Will It Mythos?

#88

Post by kingkongjaffa »

mirsadm wrote: post #83
varjag wrote: post #65

Earlier quoted context omitted.

Interesting. I tried Fable vs Codex 5.5 xhigh on three different cases. 1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly. 2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where t…

To me it feels like they're basically tweaking these things around the edges. I'm not seeing any difference in capability just preference. This has been the case for a while.

Most people thought Fable had more 'taste' than Opus, there was certainly a better quality of writing that felt more 'smart human' and not 'stochastic parrot stringing sentences together'.

Re: Will It Mythos?

#89

Post by sigmoid10 »

RaSoJo wrote: post #84
airstrike wrote: post #54

Around February, Opus 4.6 was excellent. Smart, fast, proactive. Then it got lobotomized and it's never been the same after that nerf. 4.7 came along and it too was disappointing—not unlike 4.8, which despite feeling a smidge smarter, tends to write word salad and is basically unusable for some workflows. Fable felt like having access to that "old Opus" again, but a little smarter. Sort of like I'd expect an Opus 5 t…

This is exactly what I find frustrating. I get comfortable with the latest model X. Then a new sparkly model Y launches. I am like, I don't need your new fangled Y, that consumes more tokens. My needs are small and i am happy with the older X. But then X starts to degrade. At first subtly, and then drastically. So then I am forced to upgrade to Y. What I do not understand is: > is this a sneaky way for companies to p…

I suppose it is both. Basically all frontier models are inference-time compute bound thanks to reasoning. And actual reasoning traces are locked behind closed doors at all American labs. So whenever they want to push a new model and need to give it hardware, it would make sense to cut into the reasoning budgets of older models. Users will not be able to see that directly, it will only become apparent on high-end, difficult tasks - exactly the kind of tasks where the provider wants you to use the new model anyway, so they can further improve it.
Post reply on HN