Easy solution: block any “agentic AI” from interacting with your systems at all.
Disrupting the first reported AI-orchestrated cyber espionage campaign
51–60 of 298 posts
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#52I have the feeling that we are still in the early stages of AI adoption, where regulation hasnt fully caught up yet. I can imagine a future where LLMs sit behind KYC identification and automatically report any suspicious user activity to the authorities... I just hope we won’t someday look back on this period with nostalgia :)
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#53I have the feeling that we are still in the early stages of AI adoption, where regulation hasnt fully caught up yet. I can imagine a future where LLMs sit behind KYC identification and automatically report any suspicious user activity to the authorities... I just hope we won’t someday look back on this period with nostalgia :)
Being colored and/or poor is about to get (even) worse
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#54Easy solution: block any “agentic AI” from interacting with your systems at all.
How would this be implemented?
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#55I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain.
My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and government agencies"). I'm sorry but I don't see this as some fancy AI cyberattack, this is a security failure on Anthropic's part and that too at a very basic level that should never have happened at a company of their caliber.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#56I think as AI gets smarter, defenders should start assembling systems how NixOS does it. Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed. Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running sy…
Nix makes everything else so hard that I've seen problems with production configuration persist well beyond when they should because the cycle time on figuring out the fix due to evaluations was just too long. In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.
Sounds like it’s a gap that AI could fill to make Nix more usable.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#57I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain. My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and…
there's no mention of any victims having Anthropic accounts, presumably the attackers used Claude to run exploits against public-facing systems
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#58Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#59Earlier quoted context omitted.
Nix makes everything else so hard that I've seen problems with production configuration persist well beyond when they should because the cycle time on figuring out the fix due to evaluations was just too long. In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.
Sounds like it’s a gap that AI could fill to make Nix more usable.
If you make a conventional AI agent do packaging and configuration tasks, it has to do one imperative step after the other. While it can forget, it can't really undo the effects of what it already did.
If you purpose-build these tools to work with Nix, in the big picture view how these functional units of composition can affect each other is much more constrained. At the same time within one unit of composition, you can iterate over a whole imperative multi-step process in one go, because you're always rerunning the whole step in a fresh sandbox.
LLMs and Nix work together really well in that way.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#60Earlier quoted context omitted.
> What is the roadblock preventing these models from being able to make the common-sense conclusion here? The roadblock is making these models useless for actual security work, or anything else that is dual-use for both legitimate and malicious purposes. The model becomes useless to security professionals if we just tell it it can't discuss or act on any cybersecurity related requests, and I'd really hate to see the…
I think one could certainly make the case that model capabilities should be open. My observation is just about how little it took to flip the model from refusal to cooperation. Like at least a human in this situation who is actually fooled into believing they're doing legitimate security work has a lot of concrete evidence that they're working for a real company (or a lot of moral persuasion that their work is actual…
Stop thinking of models as a 'normal' human with a single identity. Think of it instead as thousands, maybe tens of thousands of human identities mashed up in a machine monster. Depending on how you talk to it you generally get the good models as they try to train the bad modes out, problem is there are a nearly uncountable means to talking to the model to find modes we consider negative. It's one of the biggest problems in AI safety.