Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

github.com

Post reply on HN
Print view View original

101–110 of 167 posts

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#101

Post by sanchitmonga22 »

mirekrusin wrote: post #84
shubham2802 wrote: post #76

Earlier quoted context omitted.

RunAnywhere builds software that makes AI models run fast locally on devices instead of sending requests to the cloud. Right now, our focus is Apple Silicon. Today there are two parts: MetalRT - our proprietary inference engine for Apple Silicon. It speeds up local LLM, speech-to-text, and text-to-speech workloads. We’re expanding model coverage over time, with more modalities and broader support coming next. RCLI -…

From LLM benchmarks it looks like it's better to use open source uzu than RunAnywhere's proprietary inference engine. [0] https://github.com/trymirai/uzu

uzu is a strong engine, it beat us on Llama-3.2-3B (222 vs 184 tok/s) and we reported that honestly in our benchmarks.

But looking at the full picture across all four models tested:

Qwen3-0.6B: MetalRT 658, uzu 627

Qwen3-4B: MetalRT 186, uzu 165

Llama-3.2-3B: uzu 222, MetalRT 184

LFM2.5-1.2B: MetalRT 570, uzu 550

MetalRT wins 3 of 4. The bigger difference is that MetalRT also handles STT and TTS natively, uzu is LLM-only. For a voice pipeline where you need all three modalities running on one engine with shared memory management, that matters.

That said, uzu is great open-source software and worth checking out if your looking for an OSS LLM-only engine on Apple Silicon.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#102

Post by sanchitmonga22 »

pzo wrote: post #51

FWIW this RCLI is only MIT license but their engine MetalRT is commercial. Not sure the license of their models I guess also not MIT. So IMHO this repo is misleading. Not sure why they decided to reinvent the wheel and write yet another ML engine (MetalRT) which is proprietary. I would most likely bet on CoreML since it have support for ANE (apple NPU) or MLX. Other popular repos for such tasks I would recommend: htt…

Fair feedback on the README clarity, we've updated it to make the licensing distinction between RCLI (MIT) and MetalRT (proprietary) more prominent. That should have been clearer from day one.

On why we built MetalRT instead of using CoreML or MLX:

CoreML is optimized for classification and vision models, not autoregressive text generation. ANE is powerful for fixed-shape workloads but doesn't handle the dynamic shapes in LLM decode well.

MLX is much closer to what we need, and we respect what Apple has built. But MLX is a general-purpose array framework, it carries abstractions for developer ergonomics and portability that add overhead. MetalRT is purpose-built for inference only, and the numbers reflect that: 1.1-1.2x faster on LLMs (same model files) and 4.6x faster on STT.

We also needed one unified engine for LLM + STT + TTS rather than stitching three separate runtimes together. That doesn't exist in any of the alternatives listed.

The libraries you mentioned (FluidAudio, mlx-swift-audio, sherpa-onnx) are good projects. RCLI actually uses sherpa-onnx as it's fallback engine when MetalRT isn't installed. They solve different problems at different layers of the stack.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#103

Post by sanchitmonga22 »

mips_avatar wrote: post #73

Have you tried any really big models on a mac studio? I'm wondering what latency is like for big qwens if there's enough memory.

Not yet with MetalRT, right now we support models up to ~4B parameters (Qwen3 4B, Llama 3.2 3B, LFM2.5 1.2B). These are optimized for the voice pipeline use case where decode speed and latency matter more then model size.

Expanding to larger models (7B, 14B, 32B) on machines with more unified memory is on the roadmap. The Mac Studio with 192GB would be an interesting target, a 32B model at 4-bit would fit comfortably and MetalRT's architectural advantages (fused kernels, minimal dispatch overhead) should scale well.

What model / use case are you thinking about? That helps us prioritize.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#104

Post by nostrebored »

fragmede wrote: post #56
alfanick wrote: post #9

I'm not looking for STT->AI ->TTS, I'm looking for truly good voice-to-text experience* on Linux (and others). Siri/iOS-Dictation is truly good when it comes to understanding the speech. Something this level on Linux (and others) would be great, yeah always listening, maybe sending the data somewhere, but give me UX - hidden latency, optimizing for first chars recognized - a good (virtual) input device.

> I'm not looking for STT->AI->TTS, I'm looking for truly good voice-to-text experience Umm, ah, wait no, uhh yes you are. Unless, hang on, you are possessed with greater umm speech capabilities than most, wait nevermind start over. Unless you never make a mistake while talking, you want AI to take out the "three, wait no four" and just leave the output with "four" from what you actually spoke. Depending on your use…

It’s the TTS layer that is weird. I’m in the same boat — speech out is just a much worse modality than text when possible.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#105

Post by sanchitmonga22 »

woadwarrior01 wrote: post #93

> Apple M3 or later required. MetalRT uses Metal 3.1 GPU features available on M3, M3 Pro, M3 Max, M4, and later chips. M1/M2 support is coming soon. On M1/M2, RCLI automatically falls back to the open-source llama.cpp engine. So, no support for M5 Neural Accelerators, eh? (Requires Metal 4) ¯\_(ツ)_/¯

Ha, not yet. Metal 4 is interesting and we're keeping an eye on it.

MetalRT currently targets Metal 3.1 GPU compute because that's where we get the most control over the decode pipeline. Neural Engine / ANE is powerful for fixed-shape inference (vision, classification) but autoregressive LLM decode, where you're generating one token at a time with dynamic KV cache, doesn't map as cleanly to ANE today.

That said, if Metal 4 opens up new capabilities that help with sequential token generation or gives better programmable access to the neural accelerator, we'll absolutely look at it. The M5 will be a fun chip to benchmark on.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#106

Post by sanchitmonga22 »

vessenes wrote: post #7

Just tried it. really cool, and a fun tech demo with rcli. I filed a bug report; not everything is loading properly when installed via homebrew. Quick request: unsloth quants; bit per bit usually better. Or more generally UI for huggingface model selections. I understand you won't be able to serve everything, but I want to mix and match! Also - grounding: "open safari" (safari opens, voice says: "I opened safari") "n…

Thanks for trying it and for filing the bug, we're looking into the homebrew install issue.

On unsloth quants: agreed, they're consistently better bit-for-bit. Adding broader quantization format support (including unsloth's approach) is on the roadmap. Right now MetalRT works with MLX 4-bit files and GGUF Q4_K_M, we want to expand that.

On the grounding issue ("navigate to google.com" not actually navigating): you're right, that's a gap. The "open_url" action exists but the LLM doesn't always route to it correctly, especially with compound commands. Small models (0.6B-1.2B) have limited tool-calling accuracy, upgrading to Qwen3.5 4B via rcli upgrade-llm helps significantly. We're also improving the action routing prompts.

Appreciate the detailed feedback, this is exactly what we need.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#107

Post by sanchitmonga22 »

blks wrote: post #75
vessenes wrote: post #7

Just tried it. really cool, and a fun tech demo with rcli. I filed a bug report; not everything is loading properly when installed via homebrew. Quick request: unsloth quants; bit per bit usually better. Or more generally UI for huggingface model selections. I understand you won't be able to serve everything, but I want to mix and match! Also - grounding: "open safari" (safari opens, voice says: "I opened safari") "n…

> "open safari" (safari opens, voice says: "I opened safari") "navigate to google.com in safari" (nothing happens, voice says: "I navigated to google.com") So you’re describing a core broken feature. Application breaking at easiest test.

Fair criticism. The action executed on the LLM side but didn't translate to the correct macOS action, the model hallucinated success instead of routing to the open_url tool.

This is a known limitation with small LLMs (0.6B-1.2B) doing tool calling. They sometimes confuse "I know what you want" with "I did it." Upgrading to a larger model improves tool-calling accuracy significantly.

We're also working on verification, having the pipeline confirm the action actually succeeded before reporting back. Thats a fair expectation and we should meet it.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#108

Post by sanchitmonga22 »

jawns wrote: post #50

Based on the demo video, the TTS sounds like it's 10 years out of date. I would not enjoy interacting with it.

The default TTS voice (Piper) is a lightweight model optimized for speed over quality. It's fast but yeah, it doesn't sound great.

If you install Kokoro TTS (rcli models > TTS section), the voice quality is dramatically better, it's a neural TTS model with 28 different voices. MetalRT synthesizes Kokoro at 178ms for short responses, so you don't pay a speed penalty for the upgrade.

We should probably make Kokoro the default or atleast make the upgrade path more obvious in the first-run experience. Fair feedback.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#109

Post by sanchitmonga22 »

tiku wrote: post #22

Personally I'm so disappointed about the state of local AI. Only old models run "decent" but decent is way to slow to be usable.

This is exactly the problem we're trying to solve. The models themselves have gotten surprisingly capable at small sizes, Qwen3.5 4B with 262K context, LFM2 1.2B for fast tool calling, but the inference infrastructure hasn't kept up.

When people say "local AI is too slow," they usually mean the engine is too slow, not the model. A 4B model at 186 tok/s (MetalRT on M4 Max) feels genuinely responsive for interactive chat. The same model at 87 tok/s (llama.cpp) feels sluggish. Same weights, same quality, 2x the speed, that's a usability cliff.

We think the gap between cloud and on-device inference is a infrastructure problem, not a model problem. That's what we're working on.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#110

Post by sanchitmonga22 »

tristor wrote: post #8

> What would you build if on-device AI were genuinely as fast as cloud? I think this has to be the future for AI tools to really be truly useful. The things that are truly powerful are not general purpose models that have to run in the cloud, but specialized models that can run locally and on constrained hardware, so they can be embedded. I'd love to see this able to be added in-path as an audio passthrough device so…

This is a great idea. A virtual audio device that sits in the path of any audio stream and provides live transcription, that would be huge for video conferencing, lectures, podcasts.

MetalRT's STT numbers make this feasible: 70 seconds of audio transcribed in 101ms means you could process audio chunks in real-time with massive headroom. The latency would be imperceptible.

We haven't built this yet but it's a compelling use case. CoreAudio supports virtual audio devices (aggregate devices) that could pipe audio through the pipeline. If anyone in this thread has experience building macOS audio HAL plugins and wants to collaborate, we're very open to contributions, RCLI is MIT.

Post reply on HN