Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Frontier AI has broken the open CTF format

kabir.au

Post reply on HN
Print view View original

281–290 of 502 posts

Re: Frontier AI has broken the open CTF format

#281

Post by jhbadger »

sigmoid10 wrote: post #126
IanCal wrote: post #105

Earlier quoted context omitted.

Can’t say you’re wrong but the last anecdote describes many I’ve had to review for jobs long before LLMs. Fizzbuzz is a classic thing that shockingly many devs genuinely cannot do, even at home.

Yeah, I've interviewed people like this 15 years ago. Degrees and experience mean nothing in this field. The best predictor I found was personal passion projects. Let them get as nerdy as possible, then you will see pretty quickly where their skills are at and what their limits are. And you will immediately filter out people who just studied CS because they heard you can make good money.

Maybe. There are certainly people in all fields who are book smart and did well in classes but are useless at actually practicing their field (not to mention people who cheated in school and got away with it and aren't even that), and it is worth filtering them out. But I think it is weird that CS expects good workers to have these passion projects. Do we expect civil engineers to build bridges in their back yard on the weekends? Can't someone just be good at their job and have other interests outside it?

Re: Frontier AI has broken the open CTF format

#282

Post by mr_mitm »

tardedmeme wrote: post #279
mr_mitm wrote: post #188

Earlier quoted context omitted.

Mass is not the same as energy. Mass can be converted to energy or has energy, but a photon, for example, is massless while carrying energy.

That is incorrect. Photons have mass. They have no rest mass. They also cannot rest, so you might wonder how relevant that is.

The concepts of rest mass and relativistic mass are considered outdated. In modern physics, "mass" means what they meant by "rest mass".

Here some indication I'm not making this up: https://hsm.stackexchange.com/questions/2465/when-and-why-di...

In any case, I never use those concepts, and I know no professional particle physicist that does. By "mass", I mean rest mass.

Re: Frontier AI has broken the open CTF format

#283

Post by tannertech »

voxl wrote: post #272
skulk wrote: post #213

Earlier quoted context omitted.

> computers do not help human education in the slightest I had no access to anyone who could teach me calculus as a kid except Khan Academy, so I think this is a gross exaggeration. But I agree in the end, that all my "real" learning did come from pen-and-paper practice, not watching videos.

The reality is that a human will learn, given any materials including LLMs, but only if they truly desire to learn. We've had MOOCs, gigantic libraries, all full of free information. You can obtain a PhD level understanding in any technical field of your choice today just by consistently going to the library and consistently applying yourself. It's not unlike going to the gym, and we see how many people do that regul…

The annoying thing is a PhD level understanding does not get you jobs.

Re: Frontier AI has broken the open CTF format

#284

Post by funimpoded »

SquareWheel wrote: post #276
12_throw_away wrote: post #270

Earlier quoted context omitted.

> I forgot basic python syntax If you cannot write "basic syntax" for any language then you are not a programmer, and certainly not a software engineer? This is not a value judgement, it's ok (probably good tbh) to not be a programmer. But you are wasting everyone's time by interviewing for a programming position in this case.

Personally, I forget syntax all the time. There's always a warm up period after I switch languages, and it takes me longer to be start writing good, idiomatic code. Like sure, I can probably write some python, but will it be pythonic? I might still be Java-minded for a while, trying to OOP my way into solutions. Earlier today I needed to write some PHP and couldn't remember if it used length, count, or size. I had to…

Same, I can't pass any test that relies on getting syntax correct. If you want me to fizzbuzz on a whiteboard in a language I've been writing dozens or more of lines of per day for a year up to and including the day before, and require that I don't mess up the syntax, I reckon I've got a coin-flip chance of passing at best (meanwhile, sure, of course the actual logic of fizzbuzz isn't tricky for me)

I once got the method invocation syntax wrong for PHP in an interview. I'd written thousands of lines of PHP and had most-recently written some the week before.

This, despite starting off my programming journey in editors with no hinting or automatic correction. If anything, I've gotten even worse about remembering syntax as I've gotten better at the rest of the job, but I was never great at it.

I rely on surrounding code to remind me of syntax and the exact names of basic things constantly. On a blank screen without syntax hints and autocompletion, or a blank whiteboard, I'm guaranteed to look like a moron if you don't let me just write pseudocode.

Been paid to write code for about 25 years. This has never been any amount of a problem on the job but is sometimes a source of stress in interviews and has likely lost me an offer or two (most of the sources of stress in an interview have little to do with the job, really)

Re: Frontier AI has broken the open CTF format

#285

Post by hemlock4593 »

I feel the post. For me AI has ruined both, playing CTFs and also building CTFs challenges. The most annoying thing to me is the "yeah idk but here is the flag" mentality.

Before when playing CTFs with my mates was usually sitting there for hours tackling a challenge until some other mate joined, had some look together and solved it with you together in 30 minutes which is the most rewarding learning experience. Nowadays mate joins in throws the clanker on it and solved it in 5 minntes. Asking on how it worked you always get the "yeah idk what it did, but who cares, here is the flag" response.

Same for creating challenges. Whenever I ask for writeups or if some people solved it differently I usually get the "yeah idk, clanker solved that one" response taking the fun out of it.

So yep, this CTF format is definitely dead. Mainly because the strong competitiveness and prices. This encourages people to cheese challenges and sometimes solving them differently was fine as you still had a creative out-of-the-box thinking moment, but nowadays with AI there is no brainpower needed, no cheesing needed, no human needed. As you mentioned, it's pay to win.

My two cents is that the 24/7 CTFs will get more attraction as the scoreboard doesn't matter there and simply doesn't give you any price.

Re: Frontier AI has broken the open CTF format

#286

Post by gbnwl »

chrismorgan wrote: post #53

Meta: this was submitted with the article’s title “The CTF scene is dead” which I found very easy to understand. It has just been updated to use the subtitle’s first sentence, “Frontier AI has broken the open CTF format”. I find that much harder to grasp, rather like a garden-path sentence. My immediate thoughts were that “Frontier” was a company name, and that there was some file format named CTF. If you don’t know…

Frontier as in "Frontier Model" is a legitimate vocabulary term you should probably be aware of in 2026. It's not something the author made up or chose randomly, it's common parlance in the space.

Re: Frontier AI has broken the open CTF format

#287

Post by charcircuit »

>Imagine giving every competitive chess player the best chess engine and letting them use it freely during matches. Would that be considered fair?

Imagine every competitive chess player being allowed to video call with a hundred other people to help them make a move. CTF have never been fair, nor has it ever been effectively structured for learning.

Re: Frontier AI has broken the open CTF format

#288

Post by Wolfbeta »

selimthegrim wrote: post #280
whichdan wrote: post #253

Earlier quoted context omitted.

Yeah I agree. I grew up in a very blue-collar town, and anything I wanted to learn (outside of public schooling) either came from emaciated websites or whatever books I could find at the library. Having YouTube and Khan Academy and everything else would have made such a huge difference for me.

Now I’m wondering how a website is emaciated

One simply forgets to hydrate.

Re: Frontier AI has broken the open CTF format

#289

Post by lmeyerov »

nine_k wrote: post #252
lmeyerov wrote: post #250

It's tough. We run botsbench.com , which tracks AI progress on a top CTF, and I gave a talk at CCC a few months ago on our own results doing AI speed runs, so I think about this a lot. In our own trainings we give (AI agents for security, and a graph masterclass), we ended up leaning into it. For example, we ship with a skills bundle. There are plus sides, like less code-forward participants can go further and are ap…

Will this bring back the age of LAN parties, where the LAN is disconnected from the internet, and mobile connectivity is blocked?

I think that ship has sailed as well --

botsbench.com shows Sonnet 4.5+ with Claude Code harness does pretty well, and Sonnet roughly tracks the edge of what self-hosted models do on the upper tier of affordable GPUs, like running 1-2 DGX Sparks and waiting 6mo for oss to catch up a bit

Re: Frontier AI has broken the open CTF format

#290

Post by dabbledash »

josh2600 wrote: post #165
IanCal wrote: post #105

Earlier quoted context omitted.

Can’t say you’re wrong but the last anecdote describes many I’ve had to review for jobs long before LLMs. Fizzbuzz is a classic thing that shockingly many devs genuinely cannot do, even at home.

Why is it important that a dev can’t do fizzbuzz without ai? If they can ship code that matches a spec, why does it matter if they’re using ai or not? Genuinely curious.

If the job does not require a person to be able to fizzbuzz, it probably doesn't require a person at all.
Post reply on HN