Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will It Mythos?

swelljoe.com

Post reply on HN
Print view View original

61–70 of 232 posts

Re: Will It Mythos?

#61

Post by ben_w »

andsoitis wrote: post #25
somesortofthing wrote: post #12

Earlier quoted context omitted.

In LLMs, much like in humans, agency and misalignment are two sides of the same coin.

> agency and misalignment are two sides of the same coin. The free will coin?

In my experience "free will", like "consciousness" and "common sense", is not so much a concept with a universally agreed definition as it is a cognitive stop sign or an applause light, meaning different things to everyone who uses the term.

Do I have free will, or am I bounded by the laws of physics?

Even if you think my soul is completely independent of my body, there are theologians who argue that God being omniscient means that who goes to heaven and hell is predetermined before birth and therefore no action you take will ever change the afterlife you go to, and that to think God isn't omniscient would be blasphemy; do they think I have free will?

And then there's Thelma with "Do what thou wilt shall be the whole of the Law", which can be understood in terms of (amongst other things) "Don't let peer pressure manipulate you into thinking you want other things than you really want", though this is of course a simplification much as the omniscient example above: https://en.wikipedia.org/wiki/True_Will

Re: Will It Mythos?

#62

Post by shshnsnnsma »

Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

This is very cool, thank you for the write-up.

What caught my eye is the complexity you assign to a project like this. It’s hairy but I wouldn’t call it super complicated. I find that super interesting to be honest because it probably means that it is really hard and I am just used to this shit now and it all looks doable to me now.

I never think of anything as “complex”, certainly not my own work and I always think what other people do is so much more impressive but I’m starting to realize it might be a me-issue.

I worked on some pretty hairy nonsense like say a DB replication solution but I still think it was just tangly, not complex like say a particle collider. Maybe I also need to call my work super complex and highly abstract. Now that I think of it I have a history of not being taken seriously while others with easy shit get credits.

Re: Will It Mythos?

#63

Post by davedx »

I don't understand the article.

"I’d say this benchmark answers with a resounding, “Maybe.”

Mythos maybe really is better than the other current models at finding security bugs"

Yet in the results, I don't see Mythos?

It seems like a really well researched article with lots of results for other models, yet the title seems to be clickbait because the results don't contain Mythos, do they?

Re: Will It Mythos?

#64

Post by scotty79 »

davedx wrote: post #63

I don't understand the article. "I’d say this benchmark answers with a resounding, “Maybe.” Mythos maybe really is better than the other current models at finding security bugs" Yet in the results, I don't see Mythos? It seems like a really well researched article with lots of results for other models, yet the title seems to be clickbait because the results don't contain Mythos, do they?

Bugs the other models were benchmarked on are from the corpus that Mythos found. So Mythos might have 100% in this benchmark.

Although the benchmark had 100$ budget cap and rudimentary tooling so probably a bit less than 100%.

GPT-5.5-pro attemted only 4 problems out of 9 before the budget ran out and got 2 of them right.

It's a shame that the author didn't try GPT-5.5-pro on all 9 just for completeness, pehaps on subscription to save money.

Re: Will It Mythos?

#65

Post by varjag »

Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

Interesting.

I tried Fable vs Codex 5.5 xhigh on three different cases.

1. A resource leak with unknown cause. Both of them zoomed onto the same potential issue and proposed almost identical patches. Fable missed an edge case that Codex handled correctly.

2. Review of a SPICE model. Models had different comments, none substantial. Both missed important issues that were simulated inadequately. Clearly a valley where they are undertrained.

3. An open research problem in CS, presented as a codebase with documentation and performance metrics over datasets. Both were spinning wheels. Which can certainly mean the whole approach had run its course but older models were not able to identify the previous round of improvement either.

I liked the prose coming out of Fable more: it was almost like if Obama was giving tech speeches. By actual solution metrics however they both appear in the same place, naturally with the caveat that we didn't really have more time with Fable to compare further.

Re: Will It Mythos?

#66

Post by NetOpWibby »

Tossrock wrote: post #39

As I posted in another comment, I found Fable to be substantially more powerful than any previous model. However, this isn't just an ungrounded opinion - I uploaded my full session transcript and code created working on a very complex implementation, so people can judge for themselves, if they're interested: https://tossrock.substack.com/p/36-hours-with-fable

Great post. I miss Fable.

Re: Will It Mythos?

#67

Post by olmo23 »

davedx wrote: post #63

I don't understand the article. "I’d say this benchmark answers with a resounding, “Maybe.” Mythos maybe really is better than the other current models at finding security bugs" Yet in the results, I don't see Mythos? It seems like a really well researched article with lots of results for other models, yet the title seems to be clickbait because the results don't contain Mythos, do they?

> Yet in the results, I don't see Mythos?

Mythos is the 100% against which the other models are compared.

Re: Will It Mythos?

#68

Post by himata4113 »

What makes mythos special is the fact that someone with zero expertise in the field could find and weaponize a zero-day. Real threat actors already use llms em masse and the recent advancements with glm-5.2 will probably enable way more cyber attacks than fable ever could.

Re: Will It Mythos?

#70

Post by hodgehog11 »

dimgl wrote: post #21
hodgehog11 wrote: post #11

Earlier quoted context omitted.

Hard disagree. Opus reports to me like a student. Fable reported to me like a colleague (researcher). It genuinely seemed to pick up on nuance that the other models just don't, even when I tell them explicitly. It's been really frustrating that neither Codex nor Opus can make targetted edits to Fable's code without screwing something subtle up. For context, this is for computational geometry work, so your mileage may…

Maybe I was getting downgraded to Opus 4.8 but I saw nothing even close to resembling this behavior when using Fable.

It very much depends on the task. What were you trying it on?
Post reply on HN