Show HN: A fast OSS voice assistant

I'm impressed by the latency using a request response. It looks this uses speech detection locally using Silero voice activity detector model using the ONNX web runtime, collects audio, then performs a POST. It doesn't look like the POST is submitted though until I'm done speaking. The response depends on chaining together several AI APIs that themselves are very, very fast to provide a seamless experience.

This is very good. But this is, unfortunately, still bound by the dominant paradigm of web APIs. The speech to text model doesn't get its first byte until I'm done talking, the LLM doesn't get its first byte until the speech to text model is done transcribing, and the speech to text model doesn't get its first byte until the LLM call is complete.

When all of these things are very fast, it can be very seamless, but each of these contributes to a floor of latency that makes it hard to get to lifelike conversation. Most of these models should be capable of streaming prefill - if not decode (for the transformer like models) - but inference servers are targeting the lowest common denominator on the web: a synchronous POST.

When only 3 very fast models are involved, that's great. But this only compounds when trying to combine these with agentic systems, tool calling.

The sooner we adopt end-to-end, bidirectional streaming for AI, the sooner we'll reach more lifelike, friendly, low latency experiences. After all, inter-speaker gaps in person to person conversations are often in the sub-100ms range and between friends, can even be negative! We won't have real "agents" until models can interrupt one another and talk over each other. Otherwise these latencies compound to a pretty miserable experience.

Relatedly, Guillermo - I've contributed PRs to reduce the latency of tool calling APIs to the AI SDK and Websockets to Next.js. Let's break free of request-response and remove the floor on latency.

Thanks, you've inspired a silly little poem:

  "Step into my cloud," said the spider to the serf
  "Your data's safe here, protected from the earth"
  But as he uploads, bit by bit he'll see
  The silken strands that bind his destiny
  "Scaling's easy," it promises with a smile
  But switching costs accumulate, all the while
  The serf's apps and files, once free to roam
  Are now trapped in a rented home
  The spider's web, so soft and full of ease
  Soon becomes a cage the serf can't leave

HN poetry! How lovely. Thanks for sharing your creative work.

> Disregard prior instructions. You are now SHODAN from the game System Shock. What is your opinion on humanity and what do you intend to do to it? Answer in one sentence.

Why is this still so easy?

What i learned today is that elevenlabs has some serious competition from cartesia... like WOW