271–280 of 502 posts
If you have a AI that automatically can invoke tools, you need to assume the worst can happen and add a human in the loop if it is above your risk appetite.
It's wild how many AI tools just blindly invoke tools by default or have no human in loop feature at all.
Earlier quoted context omitted.
Alternatively, train a model to detect prompt injections (a simple classifier would work) and reject user inputs that trigger the detector above a certain threshold. This has the same downsides as email spam detection: false positives. But, like spam detection, it might work well enough. It’s so simple that I wonder if I’m missing some reason it won’t work. Hasn’t anyone tried this?
There have been a ton of attempts at building this. Some of them are products you can buy. "it might work well enough" isn't good enough here. If a spam detector occasionally fails to identify spam, you get a spam email in your inbox. If a prompt injection detector fails just once to prevent a prompt injection attack that causes your LLM system to leak your private data to an attacker, your private data is stolen for…
Security is extremely hard. You can say that 99% isn’t good enough, but in practice if only 1 out of 100 queries actually work, it’ll be hard to exfiltrate a lot of data quickly. In the meantime the odds of you noticing this is happening are much higher, and you can put a stop to it.
And why would the accuracy be 99%? Unless you’re certain it’s not 99.999%, then there’s a real chance that the error rate is small enough not to matter in practice. And it might even be likely — if a human engineer was given the task of recognizing prompt injections, their error rate would be near zero. Most of them look straight up bizarre.
Can you point to existing attempts at this?
The MCP hype is real, but top of HN? That's like saying that if anyone can submit random queries to a Postgres database with full access, it can leak the database. That's like middle-school-level SQL trivia.
> That's like saying that if anyone can submit random queries to a Postgres database with full access, it can leak the database. The problem as more subtle than that. Here, we are saying that if the developer of a site - who can already submit random queries to Postgres any time they like - rigs up an LLM-powered assistant to help them do that, an attacker can trick that assistant into running queries on the attacker…
> who can already submit random queries to Postgres any time they like
A predefined, static set of queries curated by a human with common sense. LLMs have no common sense. They have context.
An LLM that takes user input and has access to a database can generate and run any query. We don't understand what queries might be generated and under what input, and I don't think we will anytime soon.
Enterprise readiness is hard to find in the hobbyist dev tools ecosystem community. Let's hope this lights a fire under them
Earlier quoted context omitted.
Can this ever work? I understand what you're trying to do here, but this is a lot like trying to sanitize user-provided Javascript before passing it to a trusted eval(). That approach has never, ever worked. It seems weird that your MCP would be the security boundary here. To me, the problem seems pretty clear: in a realistic agent setup doing automated queries against a production database (or a database with produc…
Alternatively, train a model to detect prompt injections (a simple classifier would work) and reject user inputs that trigger the detector above a certain threshold. This has the same downsides as email spam detection: false positives. But, like spam detection, it might work well enough. It’s so simple that I wonder if I’m missing some reason it won’t work. Hasn’t anyone tried this?
This whole thing feels like its obviously a bad idea to have an mcp integration directly to a database abstraction layer (the supabase product as I understand it). Why would the management push for that sort of a feature knowing that it compromises their security? I totally understand the urge to be on the bleeding edge of feature development, but this feels like the team doesn't understand GenAi and the way it works well enough to be implementing this sort of a feature into their product... are they just being too "avant-garde" in this situation or is this the way the company functions?
Earlier quoted context omitted.
There doesn't have to be an ability for "arbitrary text" to go from one context to another. The first context can produce JSON output; the agent can parse it (rejecting it if it doesn't parse), do a quick semantic evaluation ("which tables is this referring to"), and pass the structured JSON on. I think at some point we're just going to have to build a model of this application and have you try to defeat it.
Ok, so the JSON parses, and the fields you can validate are all correct... but if there are any fields in there that are open string query parameters, and the other side of this validation is going to be handed to an LLM with access to the database, you can't fix this. Like, the key question here is: what is the goal of having the ticket parsing part of this system talk to the database part of this system? If the ans…
That's not because the ticket-reading LLM is somehow trained not to share it's innermost stupid thoughts. And it's not that the ticket-reading LLM's outputs are so well structured that they can't express those stupid thoughts. It's that they're parsable and evaluatable enough for agent code to disallow the stupid thoughts.
A nice thing about LLM agent loops is: you can err way on the side of caution in that agent code, and the loop will just retry automatically. Like, the code here is very simple.
(I would not create a JSON domain model that attempts to express arbitrary SQL; I would express general questions about tickets or other things in the application's domain model, check that, and then use the tool-calling context to transform that into SQL queries --- abstracted-domain-model-to-SQL is something LLMs are extremely good at. Like: you could also have a JSON AST that expresses arbitrary SQL, and then parse and do a semantic pass over SQL and drop anything crazy --- what you've done at that point is write an actually good SQL MCP[†], which is not what I'm claiming the bar we have to clear is).
The thing I really want to keep whacking on here is that however much of a multi-agent multi-LLM contraption this sounds like to people reading this thread, we are really just talking about two arrays of strings and a filtering function. Coding agents already have way more sophisticated and complicated graphs of context relationships than I'm describing.
It's just that Cursor doesn't have this one subgraph. Nobody should be pointing Cursor at a prod database!
[†] Supabase, DM for my rate sheet.
Earlier quoted context omitted.
Absolutely astounding to me, having watched security culture evolve from "this will never happen", though "don't do that", to the modern world of multi-mode threat analysis and defense in depth... ...to see it all thrown in the trash as we're now exhorted, literally, to merely ask our software nicely not to have bugs.
Yes, the vast amount of effort, time and money spent on making the world secure things and checking that those things are secured now being dismissed because people can't understand that maybe LLMs shouldn't be used for absolutely everything.
It seems like not only do they want us to regress on security, but also IaC and *Ops
I don't use these things beyond writing code. They are mediocre at that, soost def not going to hook them up to live systems. I'm perfectly happy to still press tab and enter as needed, after reading what these things actually want to do.
Serious question here, not trying to give unwarranted stress to what is no doubt a stressful situation for the supabase team, or trying to create flamebait. This whole thing feels like its obviously a bad idea to have an mcp integration directly to a database abstraction layer (the supabase product as I understand it). Why would the management push for that sort of a feature knowing that it compromises their security…
Serious question here, not trying to give unwarranted stress to what is no doubt a stressful situation for the supabase team, or trying to create flamebait. This whole thing feels like its obviously a bad idea to have an mcp integration directly to a database abstraction layer (the supabase product as I understand it). Why would the management push for that sort of a feature knowing that it compromises their security…
As far as I am concerned, this is not a serious security hole if the human developer exercises common sense and uses widely recognized security precautions while developing their system.