Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Frontier AI has broken the open CTF format

kabir.au

Post reply on HN
Print view View original

251–260 of 502 posts

Re: Frontier AI has broken the open CTF format

#251

Post by nine_k »

Jenk wrote: post #98
IanCal wrote: post #65

Earlier quoted context omitted.

If it helps I understand the second much better and feels less clickbaity and includes more info. I do agree with the points you made about the confusion although I find frontier a term used in this area a lot, “frontier AI models have” would probably resolve that.

If the title simply said "AI is out-performing humans at CTF" then none of this confusion exists. Nothing is "broken," we don't need to be superfluous with "frontier," and the point is still there.

It's like "Forklifts outperform humans in weightlifting". The problem, of course, is that a forklift is much easier to spot among athletes than an AI among CTF players.

Re: Frontier AI has broken the open CTF format

#252

Post by nine_k »

lmeyerov wrote: post #250

It's tough. We run botsbench.com , which tracks AI progress on a top CTF, and I gave a talk at CCC a few months ago on our own results doing AI speed runs, so I think about this a lot. In our own trainings we give (AI agents for security, and a graph masterclass), we ended up leaning into it. For example, we ship with a skills bundle. There are plus sides, like less code-forward participants can go further and are ap…

Will this bring back the age of LAN parties, where the LAN is disconnected from the internet, and mobile connectivity is blocked?

Re: Frontier AI has broken the open CTF format

#253

Post by whichdan »

skulk wrote: post #213
jaybrendansmith wrote: post #205

Earlier quoted context omitted.

Everything we've learned in the last 10 years is telling us that computers do not help human education in the slightest. We remember better when we write with pen and paper. We learn better with whiteboards and paper books. The simple answer: Remove most computing from education entirely. Blue composition books, pencils, whiteboards is what trains humans. Calculators are helpful perhaps but it is quite possible that…

> computers do not help human education in the slightest I had no access to anyone who could teach me calculus as a kid except Khan Academy, so I think this is a gross exaggeration. But I agree in the end, that all my "real" learning did come from pen-and-paper practice, not watching videos.

Yeah I agree. I grew up in a very blue-collar town, and anything I wanted to learn (outside of public schooling) either came from emaciated websites or whatever books I could find at the library. Having YouTube and Khan Academy and everything else would have made such a huge difference for me.

Re: Frontier AI has broken the open CTF format

#254

Post by unethical_ban »

josh2600 wrote: post #165
IanCal wrote: post #105

Earlier quoted context omitted.

Can’t say you’re wrong but the last anecdote describes many I’ve had to review for jobs long before LLMs. Fizzbuzz is a classic thing that shockingly many devs genuinely cannot do, even at home.

Why is it important that a dev can’t do fizzbuzz without ai? If they can ship code that matches a spec, why does it matter if they’re using ai or not? Genuinely curious.

First: FizzBuzz is a test to know if you understand the most basic constructs of programming. The kind of thing you learn in the first week of CS101. I forgot what it was, and when I looked at the problem I knew the answer.

More broadly: In the short/medium term, we still need humans who have the skills to understand software largely on their own. We will always need those who understand software engineering and architecture. Perhaps in 25 years LLMs will be so good that learning Python by hand will be like learning assembly today. But not yet.

The field is not ready for new practitioners to be know-nothing Prompt engineers. If we do that, we cut the legs out from under the education pipeline for programming.

Re: Frontier AI has broken the open CTF format

#255

Post by unethical_ban »

brookst wrote: post #206
raincole wrote: post #193

Earlier quoted context omitted.

Because "the next generation is ruined" is always a popular sentiment. It has been with us for at least two thousand years, and it surely won't go away in our lifetime. When this AI era's devs grow older they'll complain the newer generation can't even vide code too.

I remember when everyone bemoaned the kids not knowing assembly language. How can anyone understand software if you don’t know assembly? “Kids these days don’t work as hard / know as much / value the important things” is as tired as it is universal.

The time may come when we can treat regular programming as a lower layer niche field the way we treat assembly today.

I don't think we're close to that time yet. Just like as a kid I was told to prove my work by hand even if I could do it in my head, and just like we learned how to do calculus without a calculator and then learned how to use the calculator to get the same result, I think we still need the software field to learn programming concepts independent of the use of AI to create code.

I don't think you can be a good "prompt engineer" for solid software in 2026 if you don't understand programming concepts and software architecture and flow.

Re: Frontier AI has broken the open CTF format

#256

Post by lg5689 »

This is happening to other forms of competitive programming too. The most recent AIs have problem solving skills rivaling top humans, and so if AI can't be easily banned, the competition is dominated by AI agents.

I thought code golf would take longer for AIs because there's so little training data (it's more niche), but we're seeing AIs starting to match expert humans there too. Sucks because golf has been my favorite type of programming puzzle.

It's crazy how far AIs have come in problem solving ability.

Re: Frontier AI has broken the open CTF format

#257

Post by brookst »

jkubicek wrote: post #236
brookst wrote: post #206

Earlier quoted context omitted.

I remember when everyone bemoaned the kids not knowing assembly language. How can anyone understand software if you don’t know assembly? “Kids these days don’t work as hard / know as much / value the important things” is as tired as it is universal.

OK sure, but back when old heads were complaining about the kids not knowing assembly, those same kids knew C or Fortran or something. In 2026, if you call yourself a developer and can't solve FizzBuzz without help, it's hard to argue that you know anything useful at all.

Do modern languages and compilers count as “help”? Because I could probably do fizzbuzz in x86 assembly, but it would take a while to page that back in, and I suspect most people who call themselves developers today simply could not do it without help.

Re: Frontier AI has broken the open CTF format

#258

Post by brookst »

unethical_ban wrote: post #255
brookst wrote: post #206

Earlier quoted context omitted.

I remember when everyone bemoaned the kids not knowing assembly language. How can anyone understand software if you don’t know assembly? “Kids these days don’t work as hard / know as much / value the important things” is as tired as it is universal.

The time may come when we can treat regular programming as a lower layer niche field the way we treat assembly today. I don't think we're close to that time yet. Just like as a kid I was told to prove my work by hand even if I could do it in my head, and just like we learned how to do calculus without a calculator and then learned how to use the calculator to get the same result, I think we still need the software fi…

I generally agree, but it’s just a matter of time, and even today people with domain expertise in other areas (accounting, weather, etc) are producing adequate tools using nothing but prompt engineering. Many caveats of course, but I still think 90% of the distaste for mere prompt engineers comes from “kids these days; my unique knowledge is irreplaceable and they don’t even value it” thing.

Re: Frontier AI has broken the open CTF format

#259

Post by jaybrendansmith »

jaybrendansmith wrote: post #205
baq wrote: post #44

Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence. We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part. LLMs can be wonderful teachers, but the temptation to just tell it ‘do it for me’ is almost impossible to resist.

Everything we've learned in the last 10 years is telling us that computers do not help human education in the slightest. We remember better when we write with pen and paper. We learn better with whiteboards and paper books. The simple answer: Remove most computing from education entirely. Blue composition books, pencils, whiteboards is what trains humans. Calculators are helpful perhaps but it is quite possible that…

I'm not going to disagree with step by step videos ... those are a HUGE help. I'm really talking about solving problems using pen and paper, whether math or writing, is how my problem-solving patterns actually changed.

Re: Frontier AI has broken the open CTF format

#260

Post by mpeg »

tardedmeme wrote: post #174

When I did my first CTF, it was close to the deadline and I thought I had the extracted the flag from the program and the rest of the program was just filler, so I entered the flag, and it told me it was not the flag. It turns out the program multiplies the input by a pseudorandom matrix before comparing it against the flag, so I had to implement a matrix inversion and then get the flag. That's not the story though.…

Yes but you can't compare some ollydbg script that would maybe be useful in a super specific challenge to LLMs which trivialise absolutely every challenge in a ctf and are de facto necessary to compete now
Post reply on HN