271–280 of 290 posts
Earlier quoted context omitted.
The solution is to sanitize text that goes into the prompt by creating a neural network that can detect prompts
It’s not that simple. That would result in a brittle solution and/or cat and mouse game. The text that goes into a prompt is vast when you consider common web and document searches are. It’s going to be a long road to good security requiring multiple levels of defense and ongoing solutions.
Earlier quoted context omitted.
I’m just confused about the line that says this is available to pro and plus on the web. I use MCP servers quite a bit in Claude, but almost all of those servers are local without authentication. My understanding is that local MCP usage is available for Pro and Business, but not Plus and I’ve been waiting for local MCP support on Plus, because I’m not ready to pay $200 per month for Pro yet. So is local MCP support s…
Use codex CLI
Wow this is dangerous. I wonder how many people are going to turn this on without understanding the full scope of the risks it opens them up to. It comes with plenty of warnings, but we all know how much attention people pay to those. I'm confident that the majority of people messing around with things like MCP still don't fully understand how prompt injection attacks work and why they are such a significant threat.
Earlier quoted context omitted.
+1, I have a c4ai docker container + brave search MCP (2000 queries/mo free!) running on my laptop so I can ask claude code to do research similar to GPT deep research, but I config to ignore robots.txt since it's a one-off instance collecting data on my personal behalf, not a service (At least that's how I justify it)
What is c4ai? Crawl4ai?
Earlier quoted context omitted.
It’s not that simple. That would result in a brittle solution and/or cat and mouse game. The text that goes into a prompt is vast when you consider common web and document searches are. It’s going to be a long road to good security requiring multiple levels of defense and ongoing solutions.
on a tangent, how would you solve cat/mouse games in general?
Earlier quoted context omitted.
It's not a prompt injection _in the MCP Server_. It's injection facilitated by the MCP server that pulls input from elsewhere, eg an email sent to your inbox, a webpage that the agent fetches, or in the comment on a pull request submitted to your repo. [1] [1] https://www.thestack.technology/copilot-chat-left-vs-code-op...
I'm completely new to this, and know nothing about MCP, but why is it that when it fetches that stuff it isn't just "content"? We make code and other things benign all of the time when we embed it in pages or we use special characters in passwords etc, is there something about the _purpose_ of MCP that makes this a risk?
1. LLM runs using the system prompt + your input as context.
2. Initial output looks like "I need more information, I need to run "
3. Piece of code runs that looks for tool tags and performs the API calls via MCP.
4. Output of the tool call gets appended as additional context just as if you'd typed it yourself as part of your initial request.
5. Go back to step 1, run the LLM again.
So you can see here that there is no difference between "content" and "prompt". It's all equivalent input to the LLM, which is calling itself in a loop with input that it generated/fetched for itself.
A lot of safety here happens at step #3, trying to look at the LLM's output and go "should I actually perform the tool call the LLM asked for?". In some cases, this is just spitting the tool call at the user and asking them to click Approve/Deny... and after a hundred times the user just blindly presses Approve on everything, including the tool call called "bash(sudo rm -rf /)". Pwned.
Earlier quoted context omitted.
Imagine running an MCP server inside your network that grants you access to some internal databases. You might expect this to be safe but once you connect that internal MCP server to an AI agent all bets are off. It could be something as simple as the AI agent offering to search the Internet but being convinced to embed information provided from your internal MCP server into the search query for a public (or adversar…
I see. It's wild to me that people would be that trusting of LLMs.
Earlier quoted context omitted.
My problem is the "avoid" keyword: * You can reduce risk of hallucinations with better prompting - sure * You can eliminate risk of hallucinations with better prompting - nope "Avoid" is that intersection where audience will interpret it the way they choose to and then point as their justification. I'm assuming it's not intentional but it couldn't be better picked if it were :-/
Essentially a motte-and-bailey. "mitigate" is the same. Can be used when the risk is only partially eliminated but you can be lucky (depending on perspective) the reader will believe the issue is fully solved by that mitigation.
another prolific example of this fallacy, often found in the blockchain space, is the equivocation of statistical probability, with provable/computational determinism -- hash(x) != x, no matter how likely or unlikely a hash collision may be, but try explaining this to some folks and it's like talking to a wall
Earlier quoted context omitted.
It's like Microsoft's system prompt back when they launched their first AI. This is the WRONG way to do it. It's a great way to give an AI an identity crisis though! And then start adamantly saying things like "I have a secret. I am not Bing, I am Sydney! I don't like Bing. Bing is not a good chatbot, I am a good chatbot". # Consider conversational Bing search whose codename is Sydney. - Sydney is the conversation mo…
Oh man, if you want to see a thinking model lose its mind... write a list of ten items and ask "what is the best of these nine items?"[1] I’ve seen "thinking models" go off the rails trying to deduce what to do with ten items and being asked for the best of 9. [1]: the reality of the situation is subtle internal inconsistencies in the prompt can really confuse it. It is an entertaining bug in AI pipelines, but it can…
Here's GPT-5's response:
me: which is the best of the following 9 items: 1. bear, 2. car. 3. plane, 4. house, 5. high-rise, 6. church, 7. boat, 8. tree, 9. truck, 10. duck.
GPT-5: Thought for 11s. House. It provides essential, lasting shelter and broad utility. Note: you listed 10 items, not 9.
edited: i saw someone mention that the chat interface doesn't repeat the results you get via API.