Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Cekura (YC F24) – Testing and monitoring for voice and chat AI agents

news.ycombinator.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Launch HN: Cekura (YC F24) – Testing and monitoring for voice and chat AI agents

#21

Post by atarus »

laalshaitaan wrote: post #20

The full-session evaluation framing is the right call - most teams don't realize the failure happened in turn 2 until they've spent 3 hours blaming the model. One thing worth thinking about as you grow: connecting caught regressions to production conversation data. When your simulation flags a new failure mode, being able to say "this pattern has already surfaced X times in prod this week" cuts the prioritization deb…

We track the failure modes in production directly instead of relying on simulation. So if suddenly we are seeing a failure mode pop up too often, we can alert timely. In the approach of going from simulation to monitoring, I am worried the feedback might be delayed.

Doing it in production also helps to go run simulations by replaying those production conversations ensuring you are handling regression.

Re: Launch HN: Cekura (YC F24) – Testing and monitoring for voice and chat AI agents

#22

Post by atarus »

rush86999 wrote: post #19
atarus wrote: post #12

Earlier quoted context omitted.

Training is an overkill at this point imo. I have seen agents work quite well with a feedback loop, some tools and prompt optimisation. Are you doing fine-tuning on the models when you say training?

Nope - just use memory layer with model routing system. https://github.com/rush86999/atom/blob/main/docs/EPISODIC_ME...

Memory is usually slow and haven't seen many voice agents atleast leverage it. Are you building in text modality or audio as well?

Re: Launch HN: Cekura (YC F24) – Testing and monitoring for voice and chat AI agents

#24

Post by michaellee8 »

atarus wrote: post #9
michaellee8 wrote: post #8

Interesting, I have built https://github.com/michaellee8/voice-agent-devkit-mcp exactly for this, launch a chromium instance with virtual devices powered by Pulsewire and then hook it up with tts and stt so that playwright can finally have mouth and ears. Any chance we can talk?

That's actually interesting. Is it a dependancy on user to create the HTTP endpoints for the /speak and /transcript? One of our learnings has been to allow plugging into existing frameworks easily. Example - livekit, pipecat etc. Happy to talk if you can reach out to me on linkedin - https://www.linkedin.com/in/tarush-agarwal/

Just sent an connection invitation on Linkedin. This is actually designed for allow e2e automation using playwright-mcp for a previous startup i worked in that does voice-based job interview agents. The http endpoints is provided by a daemom sitting on the background, listening all input to the virtual mic and transcribing and storing it. The agent can hit /speak and /transcript through an mcp. We have built Livekit Agents specific solutions by injecting text responses but felt that is not enough since we want to be able to test the whole thing end to end so I hacked a way to do virtual mic/speaker. It was designed for closing the dev-test-debug loop so that Claude Code can develop on its own rather than relying on human to test it.

Re: Launch HN: Cekura (YC F24) – Testing and monitoring for voice and chat AI agents

#28

Post by MickeyShmueli »

the mock tool platform thing is smart. testing agents against real APIs is a nightmare, you get flakiness, you burn through rate limits, and you can't reproduce failures

one thing i'm curious about: how do you handle testing the tool selection logic itself? like the agent choosing WHICH tool to call is often where things break, not the tool execution

we had a support agent that would sometimes call the "refund order" tool when the user just wanted to check order status. the tool worked perfectly, the LLM just kept picking the wrong one. your mock platform lets you verify the tool returns the right data, but does it catch when the agent calls the wrong tool entirely?

also the full-session evaluation vs turn-by-turn is spot on. had a similar issue with a verification flow where each individual turn looked fine in langsmith but the overall flow was completely broken. you'd see "assistant asked for name" (good), "assistant asked for phone" (good), "assistant processed request" (good), but it never actually verified the phone number matched the account

tbh this feels like one of those problems that's obvious in hindsight but nobody builds the tooling for until they get burned in production

Re: Launch HN: Cekura (YC F24) – Testing and monitoring for voice and chat AI agents

#29

Post by michaellee8 »

MickeyShmueli wrote: post #28

the mock tool platform thing is smart. testing agents against real APIs is a nightmare, you get flakiness, you burn through rate limits, and you can't reproduce failures one thing i'm curious about: how do you handle testing the tool selection logic itself? like the agent choosing WHICH tool to call is often where things break, not the tool execution we had a support agent that would sometimes call the "refund order"…

In that case I think you can have a refund subagent that is responsible for checking if the user really asked for refund before doing these dangerous things. But it only minimize errors, LLMs are non-determinitic by nature.
Post reply on HN