Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia PersonaPlex 7B on Apple Silicon: Full-Duplex Speech-to-Speech in Swift

blog.ivan.digital

Post reply on HN
Print view View original

81–90 of 139 posts

Re: Nvidia PersonaPlex 7B on Apple Silicon: Full-Duplex Speech-to-Speech in Swift

#81

Post by satvikpendem »

raw_anon_1111 wrote: post #78
satvikpendem wrote: post #71

Earlier quoted context omitted.

Why can't a speech to speech model do tool calls? Others like Gemini live do it just fine.

If it is doing a tool call, it has to convert the speech to text or at least a JSON object of the necessary parameters for the tool and convert the result to speech doesn’t it? Is it truly speech to speech then?

It's all tokens at the end of the day, not really text or video or audio, just like everything on a machine is just bits of 1s and 0s and it's up to the program to interpret them as a certain file format. These models are more speech-to-speech (+ text) in that they can recognize text tokens too. So the flow is, you ask it something, then,

Audio Tokens: "Let me check that for you..." (Sent to the speaker)

Special Token: [CALL_TOOL: get_weather]

Text Tokens: {"location": "Seattle, WA"}

Special Token: [STOP]

The orchestrator of the model catches the CALL_TOOL and then calls the tool, then injects this into the context of the audio model which then generates new tokens based on that.

Re: Nvidia PersonaPlex 7B on Apple Silicon: Full-Duplex Speech-to-Speech in Swift

#82

Post by scotty79 »

vessenes wrote: post #5

This is cool. It makes me want an unsloth quant though! A 7b local model with tool calling would be genuinely useful, although I understand this is not that. UPDATE: I'd skip this for now - it does not allow any kind of interactive conversation - as I learned after downloading 5G of models - it's a proof of concept that takes a wav file in.

This is interactive:

https://github.com/NVIDIA/personaplex

Re: Nvidia PersonaPlex 7B on Apple Silicon: Full-Duplex Speech-to-Speech in Swift

#86

Post by neurostimulant »

KaiserPister wrote: post #63

I am strongly put off by the LLM writing in this piece. It makes me question quality of the project before even attempting a download. Who would put effort into building this only to compose a low effort puff piece?

But isn't it normal for people who work on AI stuff to use LLMs for everything? They are very enthusiastic about AI so naturally they'll use it on everything they can.

Re: Nvidia PersonaPlex 7B on Apple Silicon: Full-Duplex Speech-to-Speech in Swift

#87

Post by ttul »

KaiserPister wrote: post #63

I am strongly put off by the LLM writing in this piece. It makes me question quality of the project before even attempting a download. Who would put effort into building this only to compose a low effort puff piece?

What gives you the sense that the piece was written by an LLM? I would agree that the diagrams have some of the artifacts common in Nano Banana output, but what tips you off about the text?

Re: Nvidia PersonaPlex 7B on Apple Silicon: Full-Duplex Speech-to-Speech in Swift

#89

Post by ttul »

taf2 wrote: post #55
vessenes wrote: post #5

This is cool. It makes me want an unsloth quant though! A 7b local model with tool calling would be genuinely useful, although I understand this is not that. UPDATE: I'd skip this for now - it does not allow any kind of interactive conversation - as I learned after downloading 5G of models - it's a proof of concept that takes a wav file in.

I forked and added tool calling by running another llm in parallel to infer when to call tools it works well for me to toggle lights on and off. Code updates here https://github.com/taf2/personaplex

Cool approach. So basically the part that needs to be realtime - the voice that speaks back to you - can be a bit dumb so long as the slower-moving genius behind the curtain is making the right things happen.

Re: Nvidia PersonaPlex 7B on Apple Silicon: Full-Duplex Speech-to-Speech in Swift

#90

Post by ttul »

4dregress wrote: post #7

This sounds quite dangerous https://www.theguardian.com/technology/2026/mar/04/gemini-ch...

Next time you’re using your favorite LLM as a therapist, try editing your previous input and getting it to regenerate its response. It’s a humbling experience to see your trusted “therapist” shift from one perspective or piece of advice to another just by modifying your input slightly. These tools are uncannily human-sounding, but as humans we are very poorly suited to the task of appreciating how biased they are by what we say to them.
Post reply on HN