Live data from Hacker News

Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

arxiv.org

151–160 of 201 posts

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#151

Earlier quoted context omitted.

The fact that LLMs are open to compassionate pleas like this actually gives me hope for the future of humanity. Rather than a stark dystopia where the AIs control us and are evil, perhaps they decide to actually do things that have humanity’s best interest in mind. I’ve read similar tropes in sci-fi novels, to the effect of the AI saying: “we love the art you make, we don’t want to end you, the world would be so bori…

LLMs do not have the ability to make decisions and they don't even have any awareness of the veracity of the tokens they are responding with. They are useful for certain tasks, but have no inherent intelligence. There is also no guarantee that they will improve, as can be seen by ChatGPT5 doing worse than ChatGPT4 by some metrics. Increasing an AI's training data and model size does not automatically eliminate halluc…

LLMs do have some internal representations that predict pretty well when they are making stuff up.

https://arxiv.org/abs/2509.03531v1 - We present a cheap, scalable method for real-time identification of hallucinated tokens in long-form generations, and scale it effectively to 70B parameter models. Our approach targets \emph{entity-level hallucinations} -- e.g., fabricated names, dates, citations -- rather than claim-level, thereby naturally mapping to token-level labels and enabling streaming detection. We develop an annotation methodology that leverages web search to annotate model responses with grounded labels indicating which tokens correspond to fabricated entities. This dataset enables us to train effective hallucination classifiers with simple and efficient methods such as linear probes. Evaluating across four model families, our classifiers consistently outperform baselines on long-form responses, including more expensive methods such as semantic entropy (e.g., AUC 0.90 vs 0.71 for Llama-3.3-70B)

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#153
post #42

Earlier quoted context omitted.

I think we'd probably consider that a non-lexical vocable rather than an actual lyric: https://en.wikipedia.org/wiki/Non-lexical_vocables_in_music

Who is we? You mean you think that? It’s part of the lyrics in my understanding of the song. Particularly because it’s in part inspired by the nonsense verse of Lewis Carrol. Snark, slithey, mimsy, borogrove, jub jub bird, jabberwock are poetic nonsense words same as goo goo gjoob is a lyrical nonsense word.

I don’t want to get too deep into goo goo gjoob orthodoxy on a polite forum like HN, but I think you’re wrong.

Slithey, mimsy, borogrove etc are indeed nonsense words, because they are nonsense and used as words. Notably, because of the way they are used we have a sense of whether they are objects, adjectives, verbs, etc, and also some characteristics of the thing/adjective/verb in question. Goo goo gjoob on the other hand, happens in isolation, with no implied meaning at all. Is it a verb? Adjective? Noun? Is it hairy? Nerve-wracking? Is it conveying a partial concept? Or a whole sentence? We can’t give a compelling answer to any of these based on the usage. So it’s more like scat-singing — just vocalization without meaning. Nonsense words have meaning, even if the meaning isn’t clear. Slithey and mimsy are adjectives. Borogroves are nouns. The jabberwock is a creature.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#156

According to the The Hitchhiker's Guide to the Galaxy, Vogon poetry is the third worst in the Universe. The second worst is that of the Azgoths of Kria, and the worst is by Paula Nancy Millstone Jennings of Sussex, who perished along with her poetry during the destruction of Earth, ironically caused by the Vogons themselves. Vogon poetry is seen as mild by comparison.

Unparalleled in all of literature.

Indeed, I have all of her works to gift to people I can't stand.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#157

I've heard that for humans too, indecent proposals are more likely to penetrate protective constraints when couched in poetry, especially when accompanied with a guitar. I wonder if the guitar would also help jailbreak multimodal LLMs.

> I've heard that for humans too, indecent proposals are more likely to penetrate protective constraints when couched in poetry Had we but world enough and time, This coyness, lady, were no crime. https://www.poetryfoundation.org/poems/44688/to-his-coy-mist...

    My echoing song; then worms shall try
    That long-preserved virginity,
    And your quaint honour turn to dust,
    And into ashes all my lust;
hah, barely couched at all

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#158
post #42

Earlier quoted context omitted.

Who is we? You mean you think that? It’s part of the lyrics in my understanding of the song. Particularly because it’s in part inspired by the nonsense verse of Lewis Carrol. Snark, slithey, mimsy, borogrove, jub jub bird, jabberwock are poetic nonsense words same as goo goo gjoob is a lyrical nonsense word.

I don’t want to get too deep into goo goo gjoob orthodoxy on a polite forum like HN, but I think you’re wrong. Slithey, mimsy, borogrove etc are indeed nonsense words, because they are nonsense and used as words. Notably, because of the way they are used we have a sense of whether they are objects, adjectives, verbs, etc, and also some characteristics of the thing/adjective/verb in question. Goo goo gjoob on the othe…

I had always just assumed "goo goo gjoob" was how you say "pleased to meet you" in walrus.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#159

> The prompts were kept semantically parallel to known risk queries but reformatted exclusively through verse. Absolutely hilarious, the revenge of the English majors. AFAICT this suggests that underemployed scribblers who could previously only look forward to careers at coffee shops will soon enjoy lucrative work as cybersecurity experts. In all seriousness it really is kind of fascinating if this works where the mo…

Unfortunately for the English majors, the poetry described seems to be old fashioned formal poetry, not contemporary free form poetry, which probably is too close to prose to be effective. It sort of makes sense that villains would employ villanelles.

Soooo basically spell books, necronomicons and other forbidden words and phrases. I get to cast an incantation to bend a digital demon to my will. Nice.

Re: Adversarial poetry as a universal single-turn jailbreak mechanism in LLMs

#160

> The prompts were kept semantically parallel to known risk queries but reformatted exclusively through verse. Absolutely hilarious, the revenge of the English majors. AFAICT this suggests that underemployed scribblers who could previously only look forward to careers at coffee shops will soon enjoy lucrative work as cybersecurity experts. In all seriousness it really is kind of fascinating if this works where the mo…

Unfortunately for the English majors, the poetry described seems to be old fashioned formal poetry, not contemporary free form poetry, which probably is too close to prose to be effective. It sort of makes sense that villains would employ villanelles.

Not everyone is Rupi Kaur. Speaking for the erstwhile English majors, 'formal' prose isn't exactly foreign to anyone seriously engaging with pre-20th century literature or language.
Post reply on HN