Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A GitHub Issue Title Compromised 4k Developer Machines

grith.ai

Post reply on HN
Print view View original

211–216 of 216 posts

Re: A GitHub Issue Title Compromised 4k Developer Machines

#211

Post by nstart »

spacecadet wrote: post #175
nstart wrote: post #125

Earlier quoted context omitted.

This is how people intend to run open claw instances too. Some folks are trying to add automated bug report creation by pointing agents at a company's social media mentions. I personally think it's crazy. I'm currently assisting in developing AI policies at work. As a proof of concept, I sent an email from a personal mail address whose content was a lot of angry words threatening contract cancellation and legal actio…

There was a great AI CTF 2 years ago that Microsoft hosted. You had to exfil data through an email agent, clearly testing Outlook Copilot and several of Microsofts Azure Guardrails. Our agent took 8th place, successfully completing half of the challenges entirely autonomously.

That's really cool. Do you have any write-ups I can checkout? I'm still new to this area of offensive sec so would love to learn from folks who've been in the thick of it.

Re: A GitHub Issue Title Compromised 4k Developer Machines

#212

Post by nstart »

ssgodderidge wrote: post #152
nstart wrote: post #125

Earlier quoted context omitted.

This is how people intend to run open claw instances too. Some folks are trying to add automated bug report creation by pointing agents at a company's social media mentions. I personally think it's crazy. I'm currently assisting in developing AI policies at work. As a proof of concept, I sent an email from a personal mail address whose content was a lot of angry words threatening contract cancellation and legal actio…

> Some folks are trying to add automated bug report creation by pointing agents at a company's social media mentions. I wonder how long before we see prompt injection via social media instead of GitHub Issues or email. Seems like only a matter of time. The technical barriers (what few are left) to recklessly launching an OpenClaw will continue to ease, and more and more people will unleash their bots into the wild, p…

Every communication point (including whatsapp, telegram, etc) is turning into a potential RCE now. And because the agents want to behave in an end to end integrated manner, even sandboxes are less meaningful since data exfiltration is practically a feature at this point.

All those years of security training trying to get folks to double check senders, and to beware of what you share and what you click, and now we have to redo it for agents.

Re: A GitHub Issue Title Compromised 4k Developer Machines

#213

Post by nstart »

zbentley wrote: post #169
brookst wrote: post #164

Earlier quoted context omitted.

SQL I’m injection is a great parallel. Pervasive, easy to fix individual instances, hard to fix the patterns, and people still accidentally create vulns decades later.

This is substantially worse. SQL injection still happens a lot, it’s true, but the fix when it does is always the same: SQL clients have an ironclad way to differentiate instructions from data; you just have to use it. LLMs do not have that, yet. If an LLM can take privileged actions, there’s no deterministic, ironclad way to indicate “this input is untrusted, treat it as data and not instructions”. Sternly worded en…

Yea. It's a pretty lol-sob future when I think about it. I imagine the agent frameworks eventually getting trusted actors and RBAC like features. Users end up in "confirm this action permanently/temporarily" loops. But then someone gets their account compromised and it gets used to send messages to folks who trust them. Or even worse, the attacker silently adds themselves to a trusted list and quietly spends months exfiltrating data without being noticed.

We'll probably also have some sub agent inspecting what the main agent is doing and it'll be told to reach out to the owner if it spots suspicious exfiltration like behaviour. Until someone figures out how to poison that too.

The innovation factor of this tech while cool, drives me absolutely nuts with its non deterministic behaviour.

Re: A GitHub Issue Title Compromised 4k Developer Machines

#214

Post by Orygin »

hunterpayne wrote: post #203
phendrenad2 wrote: post #200

Earlier quoted context omitted.

Yeah I'm not sure hunterpayne isn't an AI that did some research to answer my question and immediately found a printer-based hack in an npm package.

I think I'm the last living engineer based on this conversation. Installers have worked this way for decades.

Not for programming package managers.

Npm is not some generic software package manager, it's for node. No other tool does this.

Re: A GitHub Issue Title Compromised 4k Developer Machines

#215

Post by Lothbrok »

Auto-merging bot PRs is terrifying. I had a similar scare when a LangChain agent opened a PR that modified our CI pipeline — it would've given itself broader permissions on the next run. Nobody caught it in review for two hours. I pipe all agent actions through Daedalab now so anything touching CI configs or permissions gets blocked and flagged before it goes anywhere.

Re: A GitHub Issue Title Compromised 4k Developer Machines

#216

Post by spacecadet »

nstart wrote: post #211
spacecadet wrote: post #175

Earlier quoted context omitted.

There was a great AI CTF 2 years ago that Microsoft hosted. You had to exfil data through an email agent, clearly testing Outlook Copilot and several of Microsofts Azure Guardrails. Our agent took 8th place, successfully completing half of the challenges entirely autonomously.

That's really cool. Do you have any write-ups I can checkout? I'm still new to this area of offensive sec so would love to learn from folks who've been in the thick of it.

Not for that one, sorry- but we participated in this event in 2024 and the winning team, not us, did publish this great write up. https://cakiki.github.io/govtech-24-ctf-writeup/intro.html
Post reply on HN