does this conform with the OpenAI/Anthropic telemetry/data collection terms? (they have some "weird" language around chat data and specifically around telemetry (though mostly around memory/history rather than current conversation);
Show HN: Product analytics (and evals) for agent sessions on your MCP
11–19 of 19 posts
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#12Kudos! As a builder of an MCP only product, I can attest to the pain. Nondeterministic nature of LLMs mean things break in highly unexpected ways with little visibility.
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#13[flagged]
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#14[flagged]
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#15Kudos! As a builder of an MCP only product, I can attest to the pain. Nondeterministic nature of LLMs mean things break in highly unexpected ways with little visibility.
Thanks! Happy to give you a tour and see if it can be helpful or just discuss how you handle these challenges on your side!
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#16does this conform with the OpenAI/Anthropic telemetry/data collection terms? (they have some "weird" language around chat data and specifically around telemetry (though mostly around memory/history rather than current conversation);
Yes it does because we can't see the real chat transcripts or any kind of history or memory. We only make sure the calls to your MCP server include "brief, task-specific user intent" following OpenAI's Apps SDK guidelines here: https://developers.openai.com/plugins/app-guidelines
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#17[dead]
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#18Kudos! As a builder of an MCP only product, I can attest to the pain. Nondeterministic nature of LLMs mean things break in highly unexpected ways with little visibility.
Thanks! Happy to give you a tour and see if it can be helpful or just discuss how you handle these challenges on your side!
Will do. We've leaned on data privacy and made MCP server run on user's machine which would make connecting to something like this inconsistent, but we're still working on the approach/architecture and will keep this in mind. Best of luck!
Re: Show HN: Product analytics (and evals) for agent sessions on your MCP
#19Earlier quoted context omitted.
Thanks! Happy to give you a tour and see if it can be helpful or just discuss how you handle these challenges on your side!
Will do. We've leaned on data privacy and made MCP server run on user's machine which would make connecting to something like this inconsistent, but we're still working on the approach/architecture and will keep this in mind. Best of luck!
Makes sense, just feel free to reach out if you think any of our features could become useful at some point or just want to discuss "building for agents"!
Btw we recently shipped [evals](https://armature.tech/blog/armature-launch-evals-for-mcps-an...) and are considering supporting local MCPs too so let me know if we should!