Show HN: Trust Protocols for Anthropic/OpenAI/Gemini

Fair... Happy to do a deep dive on the protocols. FWIW, I'm dogfooding with an openclaw running smoltbot called Hunter S. Clawmpson. He blogs about AI from an AI's perspective: mnemom.ai/blog.

You can see his trace data live here: https://www.mnemom.ai/agents/smolt-a4c12709

The trace cards are all expandable and show you, in real time, what he's thinking/going to do, etc., and when violations are being caught. Turns out OpenClaw is extremely creative in finding ways to circumvent the rules. Voila AIP.

Busy day today, but this gives you a pretty deep dive/interactive view into the protocols in action. Cool thing about smoltbot... It's literally "smoltbot init" with the API key to the provider of your choice, and you can go to the website and claim your agent and (privately) see your traces running the same way. Very low impact dogfooding.

Have you tried using a more traditional, non-LLM, loop to do the analysis? I'd assume it wouldn't catch more of the more complex deceptive behaviours, but I'm assuming most detections can be done with various sentiment analysis / embedding tools which would drastically reduce cost and latency. If you have tried, do you have any benchmarks?

Anecdotally, I often end up babysitting agents running against codebases with non-standard choices (e.g. yarn over npm, podman over docker) and generally feel that I need a better framework to manage these. This looks promising as a less complex solution - can you see any path to making it work with coding agents/subscription agents?

I've saved this to look at in more detail later on a current project - when exposing an embedded agent to internal teams I'm very wary of handling the client conversations around alignment, so I find the presentation of the cards and the violations very interesting - I think they'll understand the risks a lot better, and it may also give them a method of 'tuning'.

Good question. So... AAP/AIP are agnostic about how checking is done, and anyone can use the protocols and enforce them however they want.

Smoltbot is our hosted (or self-hosted) monitoring/enforcement gateway, and in that, yeah... I use a haiku class model for monitoring.

I initially tried regex for speed and cost, but TBH, what you gain in speed and cost efficiency, you give up in quality.

AAP is zero-latency sideband monitoring, so that's just a (very small) cost hit. AIP is inline monitoring, but my take is this: If you're running an application where you just need transparency, only implement AAP. If you're running one that requires trust, the small latency hit (~1 second) is totally worth it for the peace of mind and is essentially imperceptible in the flow.

Your mileage may vary, which is why I open-sourced the protocols. Go for it!

My opinion is that all attempts to make an LLM behave securely that are based on training and prompting are doomed to fail. In Security, we have the notion of the CIA triad (Confidentiality, availability, and integrity), when we discuss this we often explain that these properties can be protected through people, processes, and technology. Training and prompting an AI to behave appropriately is far more akin to a "people" focussed control (similar to training and awareness practices) rather than a "technology" control.

The only way we will actually secure agents is by only giving them the permissions they need for their tasks. A system that uses your contract proposal to create an AuthZ policy that is tied to a short-lived bearer token which the agent can use on its tool calls would ensure that the agent actually behaves how it ought to.

Ha! That's a first-world problem. Check out github.com/mnemom/docs which you'll be able to access at work if you just can't wait. docs.mnemom.ai is way easier to use.