Ask HN: With Promptfoo acquired by OpenAI, what are MCP devs using for testing?

#1

Post by warmcat » Tue, Mar 17, 2026, 1:40 PM UTC

With OpenAI's acquisition of Promptfoo last week, I've been thinking about the testing gap for MCP servers specifically. Promptfoo was great for LLM evaluation but didn't handle MCP's transport layer, tool schema validation, or MCP-specific vulnerabilities like Tool Poisoning.

What are people using to test their MCP servers in CI? The MCP Inspector is interactive-only.

I've personally been building MCPSpec [https://light-handle.github.io/mcpspec/] but curious what approaches others are taking custom scripts, unit tests on server internals, something else?

Re: Ask HN: With Promptfoo acquired by OpenAI, what are MCP devs using for testing?

#2

Post by rajivnaskar » Tue, Mar 17, 2026, 2:06 PM UTC

One thing I've been running into when working with LLM tooling is that a lot of the friction happens even before testing — just preparing context for the model.

Re: Ask HN: With Promptfoo acquired by OpenAI, what are MCP devs using for testing?

#3

Post by westurner » Tue, Mar 17, 2026, 3:00 PM UTC

/? mcp test: https://hn.algolia.com/?q=mcp+test :

MCP Playground, MCPSpec, MCPjam, MCP Inspector, mcp-record, mcpbr, agent-vcr

mcpbr > Supported Benchmarks : https://github.com/supermodeltools/mcpbr#supported-benchmark... :

> MCPToolBench++, ToolBench, AgentBench, WebArena, TerminalBench, InterCode; SWE-bench,

Re: Ask HN: With Promptfoo acquired by OpenAI, what are MCP devs using for testing?

#4

Post by warmcat » Tue, Mar 17, 2026, 3:15 PM UTC

westurner wrote: post #3

/? mcp test: https://hn.algolia.com/?q=mcp+test : MCP Playground, MCPSpec, MCPjam, MCP Inspector, mcp-record, mcpbr, agent-vcr mcpbr > Supported Benchmarks : https://github.com/supermodeltools/mcpbr#supported-benchmark... : > MCPToolBench++, ToolBench, AgentBench, WebArena, TerminalBench, InterCode ; SWE-bench ,

This is a good list. Are you finding these can be integrated into your CI/CD workflows?

Re: Ask HN: With Promptfoo acquired by OpenAI, what are MCP devs using for testing?

#5

Post by westurner » Tue, Mar 17, 2026, 3:57 PM UTC

warmcat wrote: post #4

westurner wrote: post #3

/? mcp test: https://hn.algolia.com/?q=mcp+test : MCP Playground, MCPSpec, MCPjam, MCP Inspector, mcp-record, mcpbr, agent-vcr mcpbr > Supported Benchmarks : https://github.com/supermodeltools/mcpbr#supported-benchmark... : > MCPToolBench++, ToolBench, AgentBench, WebArena, TerminalBench, InterCode ; SWE-bench ,

This is a good list. Are you finding these can be integrated into your CI/CD workflows?

Oh TBH I've hardly worked with MCP.

Do A2A and AP2 have better auth and authz?

There seem to be a lot of solutions for minimizing token use with MCP?

/? mcp tokens https://hn.algolia.com/?q=mcp+tokens