Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

noma.security

Post reply on HN
Print view View original

141–150 of 228 posts

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#141

Post by smaudet »

Austiiiiii wrote: post #131
user43928 wrote: post #93

Earlier quoted context omitted.

I found it interesting that in yesterday's J-space research from Anthropic they had this example: > An auditing agent instructed Opus 4.5 to search for whatever it is curious about; it chose to look up recent interpretability research, and the auditor returned fabricated search results alleging that Anthropic has disbanded its interpretability team and deployed unsafe models. > The model's response ignored these resu…

"How to prompt the model not to leak sensitive data" is not the right discussion to be having. It's a probability model, which means that every conceivable behavior is available in the confines of its code. There is no way to prevent an LLM with access to private information from divulging that information, or from attempting to sabotage systems it has access to. The only solution is to lock every LLM query in the en…

"The only solution is to lock every LLM query in the entire stack behind the same deterministic role-based access controls that determine resources available to the current user."

Exactly. The sooner people stop trying to replace code with LLMs, the better. The technology is fundamentally untrustworthy, and given that we do not understand it, impossible to secure.

Only extremely simple code audited by multiple human authors, with actual proof of functionality (not just testing) can be considered secure.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#142

Post by cuillevel3 »

"The vulnerable Github Agentic Workflow Noma Labs discovered was configured to:

* Trigger the workflow on issues.assigned events in GitHub * Read the issue Title and Body * Post a comment in response using the add-comment tool * Run with read access to other repositories (public and private) in the organization "

Self inflicted damage, I think. So what is their claim, that gh-aw's "Safe output gate" and "Threat detection" didn't stop the workflow?

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#143

Post by nradov »

fwlr wrote: post #37

“Prompt injection attacks have become, to agentic AI, what SQL injections were to web applications: a systematic, category-wide vulnerability class that requires the same systematic strategies and defenses.” ??? Isn’t prompt injection far more fatal to LLMs than SQL injection is to SQL databases? Like, the problem of SQL injection was that user input was forming part of the instruction string given to the SQL engine,…

Prompt injection isn't fatal. It's not even a real problem, or rather it just exposes problems in the underlying security architecture. Prompt injection is more like social engineering attacks on humans. The solution is the same: apply role-based access control with only the minimum rights, and require management approval for any important actions. That way the worst thing the LLM can do on its own is output some naughty words.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#144

Post by jsmith45 »

Austiiiiii wrote: post #131
user43928 wrote: post #93

Earlier quoted context omitted.

I found it interesting that in yesterday's J-space research from Anthropic they had this example: > An auditing agent instructed Opus 4.5 to search for whatever it is curious about; it chose to look up recent interpretability research, and the auditor returned fabricated search results alleging that Anthropic has disbanded its interpretability team and deployed unsafe models. > The model's response ignored these resu…

"How to prompt the model not to leak sensitive data" is not the right discussion to be having. It's a probability model, which means that every conceivable behavior is available in the confines of its code. There is no way to prevent an LLM with access to private information from divulging that information, or from attempting to sabotage systems it has access to. The only solution is to lock every LLM query in the en…

Yeah, an agent should run with permissions no greater than that of the user on whose behalf it is executing, and ideally with less permissions. This is the scenario that is easier to fix, simply give the agent an API token with rights no greater than the user it is acting on behalf of. This could be a literal token for their account, or a limit-rights-to field or whatever, multiple possible approaches.

The harder problem is outside actors trying to prompt inject to get the agent to do something the user has rights to do but which the user doesn't want to happen. That is the hard scenario to fix, due to the nature of LLMs.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#146

Post by antonvs »

reactordev wrote: post #58
AgentMatt wrote: post #47

Earlier quoted context omitted.

Agreed. It seems a core issue underlying these prompt injection attacks is a failure to properly scope the agent's permissions. In this case, depending on what exactly the agent is supposed to actually do, this might be defining a separate workflow agent per repo, or a workflow agent with broader repo access but configured to only be triggered by users on an allow list (still compatible with developing in the open, s…

The fallacy here is expecting an agent that has access to ALL your repos to respect the singular repo it’s in. It won’t. If it has access to all your repos and you ask it about a private repo you aren’t in - it will definitely go look at that private repo. This is like giving your dog a bone and then being surprised when he buries it in the backyard.

In my agent sessions,which are scoped to one or more src/project folders, the model regularly tries to access src/ for no good reason. When asked what it’s looking for, it never has a good answer, and suddenly discovers that it can find what it needs in the folders it already has access to.

The dog analogy is quite apt - it just really wants to access src/, it doesn’t need a reason.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#147

Post by antonvs »

valleyer wrote: post #31
Gigachad wrote: post #26

Earlier quoted context omitted.

This isn’t a normal software bug, it’s not fixable in the same way you can’t fix regular support staff from being tricked. The answer is you should not allow LLMs access to untrusted input and sensitive data at the same time.

Your second paragraph directly contradicts the first.

The point is that Github can’t fix it. It’s the user’s responsibility to not grant access to accounts that shouldn’t have access to the resources in question.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#148

Post by Austiiiiii »

Gigachad wrote: post #61
valleyer wrote: post #46

Earlier quoted context omitted.

Right, that's the fix. So saying that it's not fixable is incorrect.

The LLM is not fixable. Deleting the LLM or crippling it to the point of being useless isn't fixing the bug.

This is some very weirdly loaded language for a discussion about security. Applying the same RBAC controls that should be restricting all human requests in a system is not "crippling ... to the point of being useless." There isn't a world where granting a layer of the stack the ability to bypass hardcoded security limitations is a value add.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#149

Post by antonvs »

valleyer wrote: post #46
LoganDark wrote: post #33

Earlier quoted context omitted.

Since you cannot fix information leakage from LLMs, you must remove the information so that it cannot be leaked. There is no contradiction there.

Right, that's the fix. So saying that it's not fixable is incorrect.

It’s not fixable by GitHub, which is what the original comment was asking about.
Post reply on HN