Installing a self signed cert system wide to do MITM? Sign me up!
CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
61–70 of 70 posts
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#62Comments like this don't fill me with confidence: https://github.com/brexhq/CrabTrap/blob/4fbbda9ca00055c1554a... // The policy is embedded as a JSON-escaped value inside a structured JSON object. // This prevents prompt injection via policy content — any special characters, // delimiters, or instruction-like text in the policy are safely escaped by // json.Marshal rather than concatenated as raw text.
Why do you say that? I thought this pattern was well established, or are you aware of known issues with it?
Robots struggle with syntax-in-syntax. Really easy to confuse them when asking it to write a SQL query that targets a JSON column but it must respond with a JSON envelope so the harness can parse the result. Lots of escaping that needs to happen. Deeply nested structures in JSON also end up with foibles like missing a ] or } in a string of }}]}]}. Aside from the prompt injection possibility, just the result being straight up broken and requiring another LLM call is tokens flushed.
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#63How is the judge protected from injection?
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#64Earlier quoted context omitted.
I'm willing to wager that your comment was generated from the body of the article plus a prompt to work in an advertisement for your product, which gets a mention in nearly every comment you make (and every submission you make, sometimes on a daily basis).
Hand written I’m afraid… regular comments on this topic is true - it’s an area I’m very interested in.
Curiously, the accounts whose comments/articles I'm most confident about being AI-written tend to focus heavily on AI, yet deny using it themselves.
I prompted Claude 5 times with a simple "What do you think about ?" and the text it generated was remarkably similar. In fact in every response it used the adjective "genuinely", as in your "genuinely novel", which is the LLM glazing that initially struck me.
Claude goes on to hit several of the same notes, in the same tone, basically summarizing bits of the article (using `HTTP_PROXY/HTTPS_PROXY` as a compound; referencing iptables for containers, structured JSON escaping, request size caps). It used the phrase "blast radius" and a similar three-point attack sequence in one response.
I am confident you are using LLMs to write—the Grith.ai blog is basically entirely LLM slop. Please stop posting it here.
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#65Comments like this don't fill me with confidence: https://github.com/brexhq/CrabTrap/blob/4fbbda9ca00055c1554a... // The policy is embedded as a JSON-escaped value inside a structured JSON object. // This prevents prompt injection via policy content — any special characters, // delimiters, or instruction-like text in the policy are safely escaped by // json.Marshal rather than concatenated as raw text.
Why do you say that? I thought this pattern was well established, or are you aware of known issues with it?
Well-established where and amongst who, exactly? Is it seriously a common belief that this prevents prompt injection?
That would be more than a little alarming.
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#66Earlier quoted context omitted.
If you're working in a mission-critical field like healthcare, defense, etc. you need a way to make static and verifiable guarantees that you can't leak patient data, fighter jet details etc. through your software. This is either mandated by law or in your contract details. The entire purpose of LLMs is to be non-static: they have no deterministic output and can't be validated the same way a non-LLM function can be.…
> The entire purpose of LLMs is to be non-static: they have no deterministic output and can't be validated the same way a non-LLM function can be. Adding another LLM layer is just adding another layer of swiss cheese and praying the holes don't line up. You have no way of predicting ahead of time whether or not they will. This is exactly the point though. A LLM is great at finding work-around for static defenses. We…
Where are the examples of a system that had insufficient static guarantees that was made any better by an LLM judge?
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#67[dead]
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#68Interesting approach! I’ve been building something complementary on the deterministic side. LLM-as-judge guardrails are fundamentally probabilistic and can be gamed or hallucinate themselves (as several comments pointed out). That’s why I built EvalView — it does full trajectory snapshots + diffs so you can see exactly what changed, plus a lightweight zero-judge model-check that directly pings the model and reports d…
Securing agents in real time and testing them for drift in CI are pretty different use-cases… This post is an AI-generated ad, isn’t it? It’s getting too hard to tell!
You’re right that I mixed runtime enforcement with CI drift/regression testing. Different layer, different job.
I meant it as complementary, not equivalent. CrabTrap for runtime control, EvalView for deterministic testing/diffing. My bad on making it sound like a drive-by promo.
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#69[dead]
Re: CrabTrap: An LLM-as-a-judge HTTP proxy to secure agents in production
#70Interesting approach. One thing worth considering alongside this is that a lot of agent risk sits at the network layer before the HTTP payload. An agent that can reach any endpoint at all is already a problem regardless of whether the content looks malicious. Pilot Protocol handles this differently: agents are invisible by default and can only reach peers they've mutually handshaked with. Complementary to what you're doing here, not a replacement.