Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

noma.security

Post reply on HN
Print view View original

161–170 of 228 posts

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#161

Post by lubujackson »

jakewins wrote: post #28

How is this a Github vulnerability? The researchers are the ones that grant the agent access to private repos and then ask it to answer questions in public repos.. of course this allows extracting private information? This is like setting up a normal CI job with access to secrets and running it on public PRs. If you configure GitHub to allow public code or LLM instructions to run in contexts that have access to sensi…

LLMs are just a dumb terminal related to permissions. What they apparently want is some synthentic permissions spun up based on their prompt which is... not a "prepared statement" solution and more of a "I will clean user SQL statements with my handy regex" and we know how that works out.

The real solution is a better UI for controlling permissions on a per prompt basis - just as we can select "search the web or not" the solution would be to have a "include my private repo" option that can be trivially toggled.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#162

Post by ethin »

fwlr wrote: post #37

“Prompt injection attacks have become, to agentic AI, what SQL injections were to web applications: a systematic, category-wide vulnerability class that requires the same systematic strategies and defenses.” ??? Isn’t prompt injection far more fatal to LLMs than SQL injection is to SQL databases? Like, the problem of SQL injection was that user input was forming part of the instruction string given to the SQL engine,…

The fundamental problem with even the kind of mitigation you suggest is that it just doesn't work. You would need to build some kind of completely dynamic authorization system that could figure out the context of user-provided instructions and limit agent access based on that context, at least I think. I've said it before and I'll say it again: I don't think this is actually solvable. This isn't like SQL injections or similar where the grammar was fixed and there was a predefined set of possible inputs. Here the set of inputs is unbounded as long as natural language is the medium of expression.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#163

Post by catoc »

Austiiiiii wrote: post #131
user43928 wrote: post #93

Earlier quoted context omitted.

I found it interesting that in yesterday's J-space research from Anthropic they had this example: > An auditing agent instructed Opus 4.5 to search for whatever it is curious about; it chose to look up recent interpretability research, and the auditor returned fabricated search results alleging that Anthropic has disbanded its interpretability team and deployed unsafe models. > The model's response ignored these resu…

"How to prompt the model not to leak sensitive data" is not the right discussion to be having. It's a probability model, which means that every conceivable behavior is available in the confines of its code. There is no way to prevent an LLM with access to private information from divulging that information, or from attempting to sabotage systems it has access to. The only solution is to lock every LLM query in the en…

Exactly!

Attempting to handle prompt injections by prompting the model (not to leak sensitive data), is like attempting to stop a fire by burning the area around it

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#164

Post by simonw »

commentry wrote: post #155
simonw wrote: post #133

Earlier quoted context omitted.

What do you mean by "steal it"? What are they doing with that code that's underhand?

https://www.reddit.com/r/AI_Agents/comments/1s4249h/github_j...

That Reddit post failed to link to the announcement, so I tracked it down: https://github.blog/news-insights/company-news/updates-to-gi...

> From April 24 onward, interaction data—specifically inputs, outputs, code snippets, and associated context—from Copilot Free, Pro, and Pro+ users will be used to train and improve our AI models unless they opt out. [...]

> This program does not use:

> Content from your issues, discussions, or private repositories at rest. We use the phrase “at rest” deliberately because Copilot does process code from private repositories when you are actively using Copilot. This interaction data is required to run the service and could be used for model training unless you opt out.

So yes, pieces of your private code can end up in training data if you're using Copilot with it and don't opt out.

The Reddit comment said "your private repo context will be used to train their AI models by default" which is an inaccurate summary.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#165

Post by darkvertex »

LoganDark wrote: post #33
valleyer wrote: post #31

Earlier quoted context omitted.

Your second paragraph directly contradicts the first.

Since you cannot fix information leakage from LLMs, you must remove the information so that it cannot be leaked. There is no contradiction there.

Exactly. The system should run in a forcibly limited scope of the current repo only or add permissions for scope to include other org/user repos.

It can't leak a private repo if it can't open it to begin with.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#166

Post by politician »

catoc wrote: post #163
Austiiiiii wrote: post #131

Earlier quoted context omitted.

"How to prompt the model not to leak sensitive data" is not the right discussion to be having. It's a probability model, which means that every conceivable behavior is available in the confines of its code. There is no way to prevent an LLM with access to private information from divulging that information, or from attempting to sabotage systems it has access to. The only solution is to lock every LLM query in the en…

Exactly! Attempting to handle prompt injections by prompting the model (not to leak sensitive data), is like attempting to stop a fire by burning the area around it

My guy, that's actually how fires are stopped.

https://en.wikipedia.org/wiki/Firebreak

https://en.wikipedia.org/wiki/Controlled_burn

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#167

Post by lelanthran »

siws wrote: post #157
lelanthran wrote: post #90

Earlier quoted context omitted.

> Partially, you could still deploy the AI in an isolated envirnoment. If there's nothing to access, there's no prompt injection. If there's nothing to access, there's only limited value in using an LLM in the first place. If your LLM is prevented from accessing anything other than the prompt, the only use is interactive use by the user; no automatic work done on any workflow items.

Honest question: couldn't this be solved by setting the authorization level of the agent the same as the user that prompted the question? In this post's example, the agent would be limited by the author's scope inside the organization and, therefore, be incapable of exposing any unwanted file.

> Honest question: couldn't this be solved by setting the authorization level of the agent the same as the user that prompted the question?

No.

> In this post's example, the agent would be limited by the author's scope inside the organization and, therefore, be incapable of exposing any unwanted file.

That still allows prompt injection to exfiltrate the authors files. That's the whole exploit - files that the author has, that he doesn't want exfiltrated.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#168

Post by catoc »

politician wrote: post #166
catoc wrote: post #163

Earlier quoted context omitted.

Exactly! Attempting to handle prompt injections by prompting the model (not to leak sensitive data), is like attempting to stop a fire by burning the area around it

My guy, that's actually how fires are stopped. https://en.wikipedia.org/wiki/Firebreak https://en.wikipedia.org/wiki/Controlled_burn

Haha, nice. TIL.

So all we need is ‘controlled prompting’ to handle prompt injections :-)

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#169

Post by noisy_boy »

This is like repeatedly trying to train a dog with amnesia to not poop in the bedroom. Despite the dog repeatedly doing so and moreover being particularly easy to be fooled into doing so.

It can't reliably learn so stop trying to teach it. Lock the bedroom instead.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#170

Post by js2 »

Why did an action running in the context of public repo even have access to the private repo? Looking at the workflow, it seems to use the github token which should not normally grant rights to a private repo.

Or was it the agent itself that somehow had elevated permissions? If that's the case, you've misconfigured the agent... we know that agents cannot be trusted to enforce anything.

Post reply on HN