So, the way GPT-Live-1 works, the Live model just handles the conversation layer in a lightweight manner. For any deeper thinking (and I think tool calls) it will delegate to the Backend. It's like how Fable can delegate tasks to Opus while it keeps going. The Live voice can keep chatting to you while it waits for results from the Backend.
The default for the Backend is a Responses Delegation that just sends it to another OpenAI hosted model. But you can set the API up to use Client Delegation instead, and have your client/harness receive the delegation request. At that point, your client can delegate it to a Claude instead, and have Claude do the deeper thinking. Unfortunately you're not actually talking to a Claude, even if it identifies as one, but you can at least talk to something informed by Claude.
As for how to build that - I just had Fable build/vibe the whole thing for me. I used Go for the language, and SDL 3 to handle the audio & microphone side, even though it's just a command line app for now. I've previously been working on SDL3 bindings to Golang, and a personal AI harness with tool-calling & local MCP stdio support, so it's possible my Fable re-used some of that code. But Fable can whip something together, and the most basic tools (read,write,edit,datetime) are all really easy to add to a harness as built-in tools that you can let it call.
The main downside is that Anthropic probably wants you to run it as API, so costs can blow out. And depending how you setup the harness, either every backend call is a fresh new Claude window (to which you're sending a LOT of context), of you need a way to maintain a persistent Claude conversation that you queue requests to, but at least then the context is all in one window & you can benefit from caching.
But if you get it running, and play with it a bit, you'll realize this is obviously the future interface. Typing into a terminal window feels archaic. Even if we're not quite there yet, we're tantalizingly close... and anyone who has a 3D-printer & a robot arm connected via MCP/tool calls, well, this gives them Iron Man's Jarvis right now.