Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Build your own vulnerability harness

blog.cloudflare.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Build your own vulnerability harness

#11

Post by joshka »

TacticalCoder wrote: post #7
joshka wrote: post #5

If I was doing this at scale, I'd be looking at two features in particular. Cached input tokens - leveraging the LLM's token cache to get a 10:1 cost advantage, and batch tokens, 2:1 cost saving. The benefits of doing your own harness here is that you get to explicitly program around those specific things to optimize cost. And they both heavily benefit you in the way that these sorts of jobs work - at least for the h…

> The benefits of doing your own harness here is that you get to explicitly program around those specific things to optimize cost. I've toying with pi.dev to do something like that. It's nearly as if, in the end, we'll still be coding after all.

or just let pi create that approach - i.e. slap the above input into a clanker's input field and see what comes out.

Re: Build your own vulnerability harness

#13

Post by throw1234567891 »

jerf wrote: post #3

This whole vulnerability thing is the first time I'm really feeling like AI tech is a shakedown more than a value-add to my job. So I have to take multiple different frontier models, burn tokens constantly scanning all my code bases with complicated harnesses that eat tokens by the hundreds of millions, burn tokens cross-checking the cross-checks of the cross-checks in some eight-phase process made out of non-determi…

> burn tokens constantly

Only if you choose to use the most expensive option.

Re: Build your own vulnerability harness

#14

Post by dragonwriter »

alberth wrote: post #12

Dumb question: Can someone help me understand whag it means to have a “harness” for agents (and why you need one)? I’ve done some reading on the topic and am just not getting it.

The harness is the software that runs the model, and, in the case of agents, implements the agentic loop, and provides capabilities external to the model (either directly or by interfacing with other systems.)

You need a harness for ANY use of a model because otherwise you just have a bunch of numbers that are the weights and no actual software that does anything with them, and you can’t have agents without a harness that provides the capabilities that define agents.

Re: Build your own vulnerability harness

#15

Post by lanyard-textile »

alberth wrote: post #12

Dumb question: Can someone help me understand whag it means to have a “harness” for agents (and why you need one)? I’ve done some reading on the topic and am just not getting it.

So, you have these models that you put things into (like text), and get things out of.

For many models, you can include a "tool definition" in that text input. Remember this definition is just JSON :)

Let's say you send a tool definition to the model, plus a text question.

The model responds with a request for you to call the tool. It's also JSON.

What shall you, the human, do with it? :) That's what the harness is for: In this case, it interprets that JSON request and returns a JSON response that the model can understand.

In general, a harness is anything that manipulates the model input and output for your benefit: It may recall memories and place them into your context automatically, handle tool requests, prune long conversations, injecting parts of old conversations, and so on.

Re: Build your own vulnerability harness

#16

Post by neoncontrails »

TZubiri wrote: post #8
jerf wrote: post #3

This whole vulnerability thing is the first time I'm really feeling like AI tech is a shakedown more than a value-add to my job. So I have to take multiple different frontier models, burn tokens constantly scanning all my code bases with complicated harnesses that eat tokens by the hundreds of millions, burn tokens cross-checking the cross-checks of the cross-checks in some eight-phase process made out of non-determi…

Did you see the article about how security is becoming PoW? Whoever spends more tokens wins

Ah yes. It is clear that the root cause of this CVE last month is a lack of spend. Lessons learned: 2.1 trillion tokens is insufficient to detect obvious malware; increase token spend by 10x.

https://nesbitt.io/2026/06/26/incident-report-cve-2026-lgtm....

Re: Build your own vulnerability harness

#17

Post by ozim »

dragonwriter wrote: post #14
alberth wrote: post #12

Dumb question: Can someone help me understand whag it means to have a “harness” for agents (and why you need one)? I’ve done some reading on the topic and am just not getting it.

The harness is the software that runs the model, and, in the case of agents, implements the agentic loop, and provides capabilities external to the model (either directly or by interfacing with other systems.) You need a harness for ANY use of a model because otherwise you just have a bunch of numbers that are the weights and no actual software that does anything with them, and you can’t have agents without a harness…

I think that is what OP was fishing for. Wrong answers ;).

Llama.cpp is not harness, setting up prompt template to have instruct model is also not what people call harness.

What people call harness nowadays is software that you interact with and it proxies model interactions. It can do agentic loop but the loop is not really much use for models that do “thinking” unless you want to layer your own loop on top for dealing with results and good harness can help - but usually if something runs loop it is called an agent not a harness.

So if harness proxies all interaction with model it also provides tools for model. It can provide context management and security features, where as security features would be preventing injections and making sure model doesn’t access stuff it shouldn’t.

Re: Build your own vulnerability harness

#18

Post by yorwba »

neoncontrails wrote: post #16
TZubiri wrote: post #8

Earlier quoted context omitted.

Did you see the article about how security is becoming PoW? Whoever spends more tokens wins

Ah yes. It is clear that the root cause of this CVE last month is a lack of spend. Lessons learned: 2.1 trillion tokens is insufficient to detect obvious malware; increase token spend by 10x. https://nesbitt.io/2026/06/26/incident-report-cve-2026-lgtm....

Are you aware that is a satirical story, not a real CVE? (Security was proof-of-work before LLMs, with humans as workers. A CVE getting issued before it's exploited in the wild means the defenders won this round.)

Re: Build your own vulnerability harness

#19

Post by xeyownt »

alberth wrote: post #12

Dumb question: Can someone help me understand whag it means to have a “harness” for agents (and why you need one)? I’ve done some reading on the topic and am just not getting it.

The harness is the tool you use to communicate with the model.

It can be as basic as a simple conversation tool (like ChatGPT back in the day), or more advanced like GitHub Copilot, Claude Code, etc.

Advanced harnesses will give precise instructions to the model to tell him what tooling / skills / MCPs / etc are available, and will orchestrate all the actions around the model (like starting subagents, calling external tools, etc).

Re: Build your own vulnerability harness

#20

Post by nullbio »

jerf wrote: post #3

This whole vulnerability thing is the first time I'm really feeling like AI tech is a shakedown more than a value-add to my job. So I have to take multiple different frontier models, burn tokens constantly scanning all my code bases with complicated harnesses that eat tokens by the hundreds of millions, burn tokens cross-checking the cross-checks of the cross-checks in some eight-phase process made out of non-determi…

Agreed that they shouldn't be writing vulnerable code to begin with. You'd think the models would be trained to know when they are working on something that has security implications, and to validate the security of what they're building, as they're building it.
Post reply on HN