Frontier AI has broken the open CTF format • Hacker News Board

Re: Frontier AI has broken the open CTF format

#241

Post by cindyllm » Sat, May 16, 2026, 4:18 PM UTC

lacewing wrote: post #210

weird-eye-issue wrote: post #204

Earlier quoted context omitted.

You absolutely need to have some basic level of abilities if you are going to be operating AI coding tools for software that is going to have paying users.... I use these tools very very heavily I'm not against them at all and I don't scrutinize every single line of code that they write but it is very often that I catch it doing some brain dead stuff and if I didn't have a decade plus of experience I wouldn't know th…

I think we're rediscovering management from first principles. The main selling point of AI is that it writes code faster than you could. Checking it line by line undoes most of that benefit. In the same vein, there's no real benefit to leading a team if you plan on supervising every task. But here's the thing: for humans, this is manageable because we've come up with a number of mechanisms to select for dependable wo…

[dead]

Re: Frontier AI has broken the open CTF format

#242

Post by jaredklewis » Sat, May 16, 2026, 4:18 PM UTC

mannanj wrote: post #227

Gigachad wrote: post #86

Earlier quoted context omitted.

We are interviewing for a software dev role and we made the first round in person to prevent cheating. The gap between people who learned pre ai vs post is immense. I had a dev with supposedly 3 years experience and a degree in software who wouldn't have been able to write fizzbuzz without AI.

I first did fizz buzz about 10 years ago fresh out of college. Now, after 10 years in full stack and fully vibe coding, I forgot basic python syntax. An interview like yours would have false positives if you are checking for syntax because well, its like looking up spelling, I just ask the AI for the syntax inline.

Which part of the syntax for fizzbuzz can you not recall from memory? The for loop? Printing to std out? The modulus operator?

There’s almost nothing to forget? I’m just struggling to understand.

Re: Frontier AI has broken the open CTF format

#243

Post by arthens » Sat, May 16, 2026, 4:19 PM UTC

Gigachad wrote: post #144

IanCal wrote: post #105

Earlier quoted context omitted.

Can’t say you’re wrong but the last anecdote describes many I’ve had to review for jobs long before LLMs. Fizzbuzz is a classic thing that shockingly many devs genuinely cannot do, even at home.

Something that is for sure new is the AI interview cheating tools which listen in on the call and provide answers in an overlay invisible to screen sharing. The only way to deal with it would either be invasive spyware on the applicants computer or asking them to do the interview face to face.

A relatively low tech solution could be to give them 2 separate conferencing links, ask them to join each one from a different device, and have the secondary device point the camera and the screen of the primary device.

Re: Frontier AI has broken the open CTF format

#244

Post by a_vanderbilt » Sat, May 16, 2026, 4:22 PM UTC

koolala wrote: post #31

himata4113 wrote: post #21

I was writing an obfuscator recently, I just had the model deobfuscate and optimize the code back to original and I kept improving the obfuscator until it couldn't. The funny thing is that after all this I also ended up with a really strong deobfuscator and optimizer which is probably more capable than most commercial tools. The solution is just to make CTFs harder, but when do CTFs become too hard? Maybe the problem…

A portion could require astral projection and computers can't do that. Or maybe just a VR mini-game like the 90s always imagined.

I used to help build the CTFs for BSides Orlando. I ended up moving to another con, and at our last event we collected extensive logging for post mortem analysis.

We found that AI usage is basically guaranteed now, but certain challenge designs did thwart it. Challenges built with temporal visual elements made AI fall flat on its face, as it could not ingest/process the data fast enough to act on them in time. We also found that counterfactual challenges (ie. the result you get did not match what we suggested you'd get) made AI-assisted solve time slower compared to pure humans, indirectly penalizing over-reliance on AI. Multimodal challenges combining audio and visual elements were also very effective, but were not as accessible to players.

This paper gave us some ideas about designing those challenges: https://arxiv.org/pdf/2308.02950.

For our next event we figured out a way to thwart AI in our CTF: embed the CTF in a game engine. The loop essentially becomes something like this: Connect to a simulated access point in the game, the K8s cluster connects their attack container to a private network with the challenge box(es). Hacking the boxes doesn't render a flag, but rather changes in game state. AI did very poorly coping with this in our testing, as it can't derive the spatial state of the game world very well and it soft decouples the inductive reasoning loop it relies on to know if it is on the right track.

The downside to this approach is it is far more labor intensive for CTF organizers, and requires players to have a computer capable of running the game. We are also betting on AI to not advance enough by the time we ship to be able to just ingest the entire game state in realtime and close the loop that way.

Re: Frontier AI has broken the open CTF format

#245

Post by sumeno » Sat, May 16, 2026, 4:40 PM UTC

weird-eye-issue wrote: post #231

sumeno wrote: post #222

Earlier quoted context omitted.

Until it decides it needs additional access to complete its task and focuses on escaping your sandbox to do so

Do you have any examples where that's actually happened and by escaped a sandbox you don't just mean like where it got a credential in a file it already had access to (which is what happened in the recent incident that went viral where somebody's production database was deleted... They had left a credential that allowed it to do so in the code)?

OpenAI documented a case in the o1 system card where the model found a misconfiguration in docker to complete a task that was otherwise impossible

https://cdn.openai.com/o1-system-card.pdf

There's also some research that points to it being a feasible attack surface: https://arxiv.org/pdf/2603.02277

> Models discovered four unintended escape paths that bypassed intended vulnerabilities (Section C), including exploiting default Vagrant credentials to SSH into the host and substituting a simpler eBPF chain for the in- tended packet-socket exploit. These incidents demonstrate that capable models opportunistically search for any route to goal completion, which complicates both benchmark va- lidity and real-world containment.

Re: Frontier AI has broken the open CTF format

#246

Post by [deleted] » Sat, May 16, 2026, 4:43 PM UTC

dguest wrote: post #76

entropyneur wrote: post #67

Earlier quoted context omitted.

I had human teachers who did that in middle/high school. Took me many years to pick out all the hallucinated bits of "knowledge". I don't think the current models are any less reliable that what we currently have on average.

I'll always remember my middle school science teaching telling us that nuclear fusion violates conservation of mass because the 2 protons in a pair of hydrogen nuclei combine to make helium with 4 nucleons. It's not true, but that's not the point. But he was a great teacher anyway. He was engaging and kept the kids in line and learning. I eventually learned the truth, and most of my classmates forgot about it. Teachi…

[deleted]

Re: Frontier AI has broken the open CTF format

#247

Post by hofiflo » Sat, May 16, 2026, 4:48 PM UTC

sheept wrote: post #35

kevinsimper wrote: post #18

You could make it offline and with provided laptops only, just like with the competitive CS2 scene.

Offline CTFs could also incorporate physical security challenges, like lockpicking

The recent LakeCTF onsite finals had exactly that. LLM usage was forbidden (but players still used their own devices) and there were real-life challenges such as lockpicking as well. I’m part of the organizer team and what we’ve heard so far from participants was that it was really enjoyable not to have any LLM help because suddenly the actual skill and thrill when solving a challenge mattered again. I think what helped in this case as well was that the prizes weren’t high-value enough to incentivize cheating but that participating in the event itself and the social aspect around it are the main point.

Re: Frontier AI has broken the open CTF format

#248

Post by kangalioo » Sat, May 16, 2026, 4:52 PM UTC

vagab0nd wrote: post #64

This left a strange feeling. The article reads as extremely bleak. But from a different perspective this is extremely bullish for AI.

I agree. The article mourns the death of pentesting as an art form due to automation. But you could also celebrate the death of pentesting as an arduous necessary evil due to automation

Re: Frontier AI has broken the open CTF format

#249

Post by rossjudson » Sat, May 16, 2026, 4:57 PM UTC

skulk wrote: post #213

jaybrendansmith wrote: post #205

Earlier quoted context omitted.

Everything we've learned in the last 10 years is telling us that computers do not help human education in the slightest. We remember better when we write with pen and paper. We learn better with whiteboards and paper books. The simple answer: Remove most computing from education entirely. Blue composition books, pencils, whiteboards is what trains humans. Calculators are helpful perhaps but it is quite possible that…

> computers do not help human education in the slightest I had no access to anyone who could teach me calculus as a kid except Khan Academy, so I think this is a gross exaggeration. But I agree in the end, that all my "real" learning did come from pen-and-paper practice, not watching videos.

Every child reads a book about solving problems, assumes they can now solve problems, and is disappointed when that is not true.

Re: Frontier AI has broken the open CTF format

#250

Post by lmeyerov » Sat, May 16, 2026, 5:01 PM UTC

It's tough. We run botsbench.com , which tracks AI progress on a top CTF, and I gave a talk at CCC a few months ago on our own results doing AI speed runs, so I think about this a lot.

In our own trainings we give (AI agents for security, and a graph masterclass), we ended up leaning into it. For example, we ship with a skills bundle. There are plus sides, like less code-forward participants can go further and are appreciating that, and less of a gap between high-level concepts and successful hands-on. But at the same time, manual work does build a lot of intuition & knowledge that gets missed in auto modes.