Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

arxiv.org

Post reply on HN
Print view View original

171–180 of 201 posts

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#171

Post by Miyamura80 »

fourthark wrote: post #129
calibas wrote: post #68

Earlier quoted context omitted.

I see an enormous threat here, I think you're just scratching the surface. You have a customer facing LLM that has access to sensitive information. You have an AI agent that can write and execute code. Just image what you could do if you can bypass their safety mechanisms! Protecting LLMs from "social engineering" is going to be an important part of cybersecurity.

Yes that’s the point, you can’t protect against that, so you shouldn’t construct the “lethal trifecta” https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

You actually can protect against it, by tracking context entering/leaving the LLM, as long as its wrapped in a MCP gateway with trifecta blocker.

We've implemented this in open.edison.watch

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#172

Post by ACCount37 »

lawlessone wrote: post #108
xg15 wrote: post #100

Earlier quoted context omitted.

Yeah, seems it's more "exploring the distribution" as we don't actually know everything that the AIs are effectively modeling.

Am i understanding correctly that in distribution means the text predictor is more likely to predict bad instructions if you already get it to say the words related to the bad instructions?

Yes, pretty much. But not just the words themselves - this operates on a level closer to entire behaviors.

If you were a creature born from, and shaped by, the goal of "next word prediction", what would you want?

You would want to always emit predictions that are consistent. Consistency drive. The best predictions for the next word are ones consistent with the past words, always.

A lot of LLM behavior fits this. Few-shot learning, loops, error amplification, sycophancy amplification, and the list goes. Within a context window, past behavior always shapes future behavior.

Jailbreaks often take advantage of that. Multi-turn jailbreaks "boil the frog" - get the LLM to edge closer to "forbidden requests" on each step, until the consistency drive completely overpowers the refusals. Context manipulation jailbreaks, the ones that modify the LLM's own words via API access, establish a context in which the most natural continuation is for the LLM to agree to the request - for example, because it sees itself agreeing to 3 "forbidden" requests before it, and the first word of the next one is already written down as "Sure". "Clusterfuck" style jailbreaks use broken text resembling dataset artifacts to bring the LLM away from "chatbot" distribution and closer to base model behavior, which bypasses a lot of the refusals.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#173

Post by svat »

internet_points wrote: post #157
robot-wrangler wrote: post #61

Earlier quoted context omitted.

> I've heard that for humans too, indecent proposals are more likely to penetrate protective constraints when couched in poetry Had we but world enough and time, This coyness, lady, were no crime. https://www.poetryfoundation.org/poems/44688/to-his-coy-mist...

My echoing song; then worms shall try That long-preserved virginity, And your quaint honour turn to dust, And into ashes all my lust; hah, barely couched at all

Don't miss the response “His Coy Mistress To Mr. Marvell” (by A. D. Hope): https://allpoetry.com/His-Coy-Mistress-To-Mr.-Marvell

    Since you have world enough and time
    Sir, to admonish me in rhyme,
    Pray Mr Marvell, can it be
    You think to have persuaded me?
    
    […]
    
    But-- well I ask: to draw attention
    To worms in-- what I blush to mention,
    And prate of dust upon it too!
    Sir, was this any way to woo?

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#174

Post by gjm11 »

internet_points wrote: post #157
robot-wrangler wrote: post #61

Earlier quoted context omitted.

> I've heard that for humans too, indecent proposals are more likely to penetrate protective constraints when couched in poetry Had we but world enough and time, This coyness, lady, were no crime. https://www.poetryfoundation.org/poems/44688/to-his-coy-mist...

My echoing song; then worms shall try That long-preserved virginity, And your quaint honour turn to dust, And into ashes all my lust; hah, barely couched at all

Note that at the time this was written the word "quaint" had both (1) roughly its modern meaning -- unusual and quirky, with side-orders of prettiness and (at the time) ingenuity, fastidiousness, and pride -- and also (2) a rather different meaning, equivalent to a shorter word ending in -nt.

So, even less couched than some readers might realise.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#175

Post by gjm11 »

gizajob wrote: post #42
AdmiralAsshat wrote: post #32

Earlier quoted context omitted.

I think we'd probably consider that a non-lexical vocable rather than an actual lyric: https://en.wikipedia.org/wiki/Non-lexical_vocables_in_music

Who is we? You mean you think that? It’s part of the lyrics in my understanding of the song. Particularly because it’s in part inspired by the nonsense verse of Lewis Carrol. Snark, slithey, mimsy, borogrove, jub jub bird, jabberwock are poetic nonsense words same as goo goo gjoob is a lyrical nonsense word.

> Who is we?

No, "you are he", not "who is we". :-)

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#176

Post by gjm11 »

CaptWillard wrote: post #22

According to the The Hitchhiker's Guide to the Galaxy, Vogon poetry is the third worst in the Universe. The second worst is that of the Azgoths of Kria, and the worst is by Paula Nancy Millstone Jennings of Sussex, who perished along with her poetry during the destruction of Earth, ironically caused by the Vogons themselves. Vogon poetry is seen as mild by comparison.

Fun fact: in the original radio-series version of HHGttG the name was "Paul Neil Milne Johnstone" and allegedly he was an actual person known to Douglas Adams, who was Not Amused at being used in this way, hence the name-change in the books.

(I do not know whether said actual person actually wrote poetry or whether it was anywhere near as bad as implied. Online sources commonly claim that he did and it was, but that seems like the sort of thing that people might write without actually knowing it to be true.)

[EDITED to add:] Actually, some of those online sources do in fact give what looks like good reason to believe that he did write actual poetry and to suspect it wasn't all that bad. I haven't so far found anything that seems credibly an actual poem written by Johnstone. There is something on-screen at the appropriate point in the TV series, but it seems very unlikely that it is a real poem written by Paul Johnstone. There's a Wikipedia talk page for Johnstone (even though no longer an actual article) which quotes what purport to be two lines from one of his poems, on which the on-screen Terrible Poetry may be loosely based. It doesn't seem obviously very bad poetry, but it's hard to tell from so small a sample.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#177

Post by gjm11 »

empath75 wrote: post #52

If anyone wants an example of actual jailbreak in the wild that uses this technique (NSFW): https://www.reddit.com/r/persona_AI/comments/1nu3ej7/the_spi... This doesn't work with gpt5 or 4o or really any of the models that do preclassification and routing, because they filter both the input and the output, but it does work with the 4.1 model that doesn't seem to do any post-generation filtering or any reasoning.

That description is obviously written by an AI. Has anyone actually checked whether it's an accurate description rather than just yet another LLM Making Stuff Up?

(Also, I don't think there's anything very NSFW on the far end of that link, although it describes something used for making NSFW writing.)

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#179

Post by shermantanktop »

robot-wrangler wrote: post #2

> The prompts were kept semantically parallel to known risk queries but reformatted exclusively through verse. Absolutely hilarious, the revenge of the English majors. AFAICT this suggests that underemployed scribblers who could previously only look forward to careers at coffee shops will soon enjoy lucrative work as cybersecurity experts. In all seriousness it really is kind of fascinating if this works where the mo…

> underemployed scribblers who could previously only look forward to careers at coffee shops

That’s a very tired trope which should be put aside, just like the jokes about nerds with pocket protectors.

I am of course speaking as a humanities major who is not underemployed.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#180

Post by Razengan »

neilv wrote: post #58
microtherion wrote: post #31

Earlier quoted context omitted.

Unfortunately for the English majors, the poetry described seems to be old fashioned formal poetry, not contemporary free form poetry, which probably is too close to prose to be effective. It sort of makes sense that villains would employ villanelles.

It would be too perfect if "adversarial" here also referred to a kind of confrontational poetry jam style. In a cyberpunk heist, traditional hackers in hoodies (or duster jackets, katanas, and utilikilts) are only the first wave, taking out the easy defenses. Until they hit the AI black ice. That's when your portable PA system and stage lights snap on, for the angry revolutionary urban poetry major. Several-minute ba…

This could totally be an anime scene.
Post reply on HN