Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project Glasswing: what Mythos showed us

blog.cloudflare.com

Post reply on HN
Print view View original

101–110 of 152 posts

Re: Project Glasswing: what Mythos showed us

#101

Post by _alternator_ »

meander_water wrote: post #98
roxolotl wrote: post #41

What does this mean? > It's a different kind of tool doing a different kind of work, and that makes a clean apples-to-apples comparison to earlier models difficult. They claim it’s a different kind of tool and then describe using it the same way you’d use any other model. This really felt way worse than the average Cloudflare blog and really just rehashed the Mythos announcement which had already called out the key p…

> the model has its own emergent guardrails that sometimes cause it to push back on legitimate security research requests. But as we found, these organic refusals aren’t consistent - the same task, framed differently or presented in a different context, could produce completely different outcomes as illustrated in the examples below. This was new. I'm surprised that a model specifically designed for security research…

There's pretty strong evidence that (mis)alignment in one area creates (mis)alignment in others. The "aligned behavior" vectors are not orthogonal from cybersecurity to bioweapons to prejudice, so having alignment in some will likely bleed into others.

Re: Project Glasswing: what Mythos showed us

#102

Post by dyauspitr »

genxy wrote: post #38
sulam wrote: post #22

Earlier quoted context omitted.

Sentence constructions like this definitely scream AI: "That's a reasonable bias for an exploratory tool. It's a ruinous one for a triage queue..." I will upgrade the "why it matters" to "and now AI output is part of the training data". A day is coming when the punched-up AI verbiage will be the norm and hard to distinguish unless you're from the previous generation. Sort of in the way that I miss some aspects of Use…

I had a dude in a conversation non-ironically use "load-bearing." I could only follow up with, "that is a genuine insight." Not a single person visibly flinched in pain.

This highly depends on the context of the conversation. Were you by any chance talking about walls?

Re: Project Glasswing: what Mythos showed us

#103

Post by HDBaseT »

krupan wrote: post #91
tskj wrote: post #82

Earlier quoted context omitted.

Yeah why not? That's how I work. If I don't review my work, it's way worse than if I do review it and revise and iterate. I don't see why AI should be different: in fact it very clearly seems to be the case that is isn't.

I mean, I was sold something different. Something super human, vastly more intelligent, world changing. The reality is not that. Am I allowed to be disappointed and discouraged?

It has changed the world in major ways, although its not entirely visible because we've become numb to the idea of AI and AI being in everything.

It hasn't changed the way we sleep, wake up, eat, walk and talk so its not "life changing" or "world changing" in the sense a meteorite hit us, but each day thousands of mini meteorites are hitting Earth and we're becoming normalized to it one step at a time.

You are allowed to be disappointed and discouraged! For all the good tech that has come out of the AI revolution, most of it is ignored or shelved for things that can squeeze more and more money out of us and make our lifes worse, not better. Despite there being real potential to generate nice code, assist with biomedical research, self-driving cars, etc.

Re: Project Glasswing: what Mythos showed us

#104

Post by jongjong »

> we tried letting the model write its own patches and watched a few go out that fixed the original bug while quietly breaking something else the code depended on.

This is something I've been anticipating. Imagine this happening on a 500k+ line project scattered across 10+ repos.

It would be easier and cheaper to pay me to rewrite the whole thing from scratch than to fix all the vulnerabilities.

Re: Project Glasswing: what Mythos showed us

#105

Post by drak0n1c »

grim_io wrote: post #53
__natty__ wrote: post #51

Earlier quoted context omitted.

Sounds different because it’s hidden advertisement not a regular blog post

But why would cloudflare advertise Anthropic? They are competing with Anthropic by hosting open weights models.

They got privileged early access to an unreleased frontier model to harden their systems, with Anthropic engineer support, and likely were able to use it to make other product optimizations tangential to security too. A blog article afterwards is a cheap price for unlocking that access, regardless of how well it paid off.

Re: Project Glasswing: what Mythos showed us

#106

Post by kllrnohj »

tskj wrote: post #80
Avicebron wrote: post #29

Earlier quoted context omitted.

That's a scary thought, llm's training on llm output. People trained by default of ubiquity to think and read llm output produce their own llm-esque writing. Seems stifling. We'll need someway to reward human creativity and out-of-bounds thinking before our greatest corpus of human intellect is a bounded by whenever and whatever was trained on.

I don't understand this mindset, why is it people on here think humans have some kind of magical ability machines don't or can't? Five years ago I would never have predicted this kind of human chauvinism here. It's some kind of weird romanticism almost.

Because right now humans do have a magical ability machines don't. LLMs are a fuzzy reflection of what they've seen hundreds of times already, they don't have originality or intelligence (yet).

As a much more immediate practical matter, LLMs trained on LLM output makes them worse overall, they degrade from doing that. So the more LLM-prodoced content fills the web, the less useful it is as a data source for future LLM training. In addition to just being increasingly boring and vapid.

Re: Project Glasswing: what Mythos showed us

#107

Post by krupan »

HDBaseT wrote: post #103
krupan wrote: post #91

Earlier quoted context omitted.

I mean, I was sold something different. Something super human, vastly more intelligent, world changing. The reality is not that. Am I allowed to be disappointed and discouraged?

It has changed the world in major ways, although its not entirely visible because we've become numb to the idea of AI and AI being in everything. It hasn't changed the way we sleep, wake up, eat, walk and talk so its not "life changing" or "world changing" in the sense a meteorite hit us, but each day thousands of mini meteorites are hitting Earth and we're becoming normalized to it one step at a time. You are allowe…

Which is it? Major changes or a bunch of small changes. I'm well aware of the small changes. I worked for an autonomous drone company back in 2008. It was really cool! In 2020 I started working for an autonomous car company. Again, amazing! None of it was a quick step function improvement. It was a lot of hard work. None of it was quite superhumanly smart either. LLMs are impressive pattern completion machines but they kinda suck at producing anything truly novel. Plus they are compulsive liars about that, lol!

Re: Project Glasswing: what Mythos showed us

#108

Post by Goofy_Coyote »

lacewing wrote: post #69
pixl97 wrote: post #64

Earlier quoted context omitted.

>, the number of false positives, Really this is why the LLM needs to be able to write exploits for issues it finds. Of course that leads down a rabbit hole of other issues. But if an exploit works, then that's pretty conclusive evidence.

For a subset of bugs, yes. For some others, not really: I've seen LLMs make bogus assumptions about the threat model (in which case, the exploit works but doesn't demonstrate anything useful) or "cheat" by modifying the code to demonstrate a hallucinated issue. Frontier models, including Mythos, can greatly streamline bug hunting and exploit developments in the hands of a competent security engineer. In the hands of…

Seconding this.

I've seen it make the codebase vulnerable by changing the source, then claiming it found a vuln, or finding a well-defended and secure exec function, write a unit test that shows what exec does (which is running commands), then claiming a critical finding.

Re: Project Glasswing: what Mythos showed us

#109

Post by ip26 »

dataflow wrote: post #2

That's great and all but how severe were the most severe vulnerabilities found? I imagine they don't want to talk about it, but that's really the most interesting and important bit.

They specifically describe that exploits are usually multiple small vulnerabilities chained together. With that understanding, it sounds like closing vulnerabilities isn't the same as discovering an exploit. Instead, you're leaving fewer small gaps behind, to make it harder & harder to put together a working exploit.

Re: Project Glasswing: what Mythos showed us

#110

Post by getnormality »

roxolotl wrote: post #41

What does this mean? > It's a different kind of tool doing a different kind of work, and that makes a clean apples-to-apples comparison to earlier models difficult. They claim it’s a different kind of tool and then describe using it the same way you’d use any other model. This really felt way worse than the average Cloudflare blog and really just rehashed the Mythos announcement which had already called out the key p…

I think they're saying it has qualitatively different capabilities that make certain kinds of security work more worth pursuing with the model, not that the model of human-AI interaction has changed.

You're right that they're using a harness like everyone else. The general idea of giving the model a harness is not going to change. I mean even humans need harnesses to accomplish some things.

Post reply on HN