Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

kasra.blog

Post reply on HN
Print view View original

171–180 of 239 posts

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#171

Post by shepherdjerred »

stavros wrote: post #121
shepherdjerred wrote: post #57

Earlier quoted context omitted.

Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I…

It refuses to use an API token? In my experience, it's more than happy to read out my secrets from .envrc files "just to check". At least it feels a lot of remorse over its mistake until I reset the session.

It’s really hit or miss. Most of the times it works but every once in a while it will dig in its heels

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#173

Post by dwa3592 »

sandos wrote: post #167
dwa3592 wrote: post #161

Nice exercise. Couple things: - I think the exercise was inconclusive for Claude and Gemini because they hardly tried to solve the task at hand. So the scores don't mean much. - I did the same exercise for an app I built and I asked the models to do something similar; Interestingly the models (Opus 4.6, 4.7 and Gemini 3.1 Pro) never refused to try to exploit. The difference is that in the first few runs, they found s…

Its weird having protections against finding exploits: what if I developed the app? Would it require having the development steps still in the context.. thats unlikely and also not any kind of proof. What if I intersperse exploit finding in my normal development, as you `probably should? Refusing there would be really weird to me.

I used to think that the models would not refuse to find exploits in any work done locally but I have only tested this theory on the (obscure) apps that I have built on my machine. Now if i forked pandas and started asking models to find exploits of certain kind then I'd like to think the models will start refusing after a point.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#174

Post by shepherdjerred »

gspr wrote: post #77
shepherdjerred wrote: post #57

Earlier quoted context omitted.

Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I…

I find it terrifying that people are willing to outsource thinking. Outsourcing thinking to an entity that is opinionated about what to think is beyond crazy.

What’s the difference between outsourcing thinking and using an LLM as a research tool?

An LLM with fetch/search is going to be a lot more effective than myself and Google. I would _never_ ask questions like this if the LLM wasn’t able to look up data

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#175

Post by strictnein »

yieldcrv wrote: post #125

> Almost every model used the canonical provider: Zai for GLM, Deepseek for Deepseek, etc. > I am never touching Minimax or GLM again. Their APIs had constant outages Goofy take You run these on a VPS based on the architecture of that VPS provider, or on your own cluster

GLM 5.1's smallest model size is 206 GB and really you're probably wanting to run a version that's ~400GB. If you want it to be performant, you're not just running it on a VPS.

And just saying "run it on your own cluster" sort of glosses over the cost of such a cluster.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#176

Post by jc4p »

willXare wrote: post #152

[flagged]

Great point!

When I found the original exploit in an app I researched it took me around 15 minutes and some assistance from Claude.

For this project I gave myself the weekend + parts of Monday, so around 20 hours of dev time — at my standard rate that’s ~$5,000 of dev time.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#177

Post by jc4p »

ikurei wrote: post #89

Qwen 3.7 Max: > During my local testing before the full eval harness it was the only non-GPT model that was able to complete the task, was not able to reproduce in the longer runs. Doesn't that sound like may be the harness was the problem?

I was using the same harness for each run, the difference is from when I was running the harness locally on my machine before I pushed up the full runs.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#178

Post by strictnein »

auguzanellato wrote: post #133
stuckkeys wrote: post #104

How does one apply for that “security research” pass?

https://chatgpt.com/cyber I tried it once and they somehow decided I'm not worth, if I try again it fails with "We couldn't start verification. You may not be eligible for this verification flow right now. Please try again later, or contact support if you think this is a mistake.", not sure if they think I'm part of an APT or whatever.

I got it. Probably helps that I'm at a large company and my personal OpenAI accounts have spent probably close to $10k now (reimbursed by work).

It's helpful in reducing the guardrails, but there's still guardrails around security research that I bump into.

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#179

Post by yencabulator »

Sharlin wrote: post #97
plufz wrote: post #75

Earlier quoted context omitted.

Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain? But I have no idea. Just guessing here.

I thought that these models are supposed to be vastly smarter than what’s needed to discern between "general information trivially available on Wikipedia" and "actionable synthesis instructions".

An LLM could probably make that distinction clearly.

a commercial LLM provider training their own models is however likely to bias the model(/guardrail) harder, in an effort to make them harder to jailbreak, to minimize bad press.

For example:

- refusing to talk even about the well-known parts of forbidden topics (this) - tending toward sycophancy to avoid ever seeming rude or unhelpful

Re: I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

#180

Post by _stiofan »

It's just not currently cost-effective to use AI in this way, I see it over and over reporting false positives. You then need to make it validate it's own false positives which adds more cost. The goal in this case it to have a bug free app, which AI can't do effectively yet. There are other great uses for AI, though. It is great at finding and identifying known common vulnerabilities, which can be leveraged to claim bug bounties. That's where I see it being cost-effective currently.
Post reply on HN