Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

github.com

Post reply on HN
Print view View original

151–160 of 167 posts

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#151

Post by sanchitmonga22 »

elpakal wrote: post #147
sanchitmonga22 wrote: post #107

Earlier quoted context omitted.

Fair criticism. The action executed on the LLM side but didn't translate to the correct macOS action, the model hallucinated success instead of routing to the open_url tool. This is a known limitation with small LLMs (0.6B-1.2B) doing tool calling. They sometimes confuse "I know what you want" with "I did it." Upgrading to a larger model improves tool-calling accuracy significantly. We're also working on verification…

> This is a known limitation with small LLMs (0.6B-1.2B) doing tool calling. To me this is this nut to crack, wrt tool calling and locally running inference. This seems like a really cool project and I'm going to dive around a little later but if it's hallucinating for something as basic as this makes me think it's more of POC stage right now (to echo other sentiment here).

That's a fair read. Tool calling reliability with sub-4B models is genuinely the hardest unsolved problem in on-device AI right now.

The inference engine (MetalRT) is production-grade, the pipeline architecture is solid, but the models at this size are still the weak link for complex tool routing. Larger model support (where tool calling is much more reliable) is next on the roadmap. Please stay tuned!

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#152

Post by sanchitmonga22 »

brainless wrote: post #141

I am interested in MetalRT. I am an indie builder, focused mostly on building products with LLM assistance that run locally. Like: https://github.com/brainless/dwata I would be interested if MetalRT can be used by other products, if you have some plans for open source products?

Yes, that's the plan. MetalRT will ship as part of the RunAnywhere SDK so other developers can integrate it into their own apps. We're working on making that available. If you want to be in the early access group, drop me a line at founder@runanywhere.ai or open an issue on the RCLI repo. Happy to look at your project.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#153

Post by sanchitmonga22 »

ifh-hn wrote: post #146

Faster AI inference of Apple silicon... So not run anywhere then...

please check our main repo: https://github.com/RunanywhereAI/runanywhere-sdks/

We are running anywhere, hence RunAnywhere, MetalRT is the fastest inference engine we made for Apple silicon, and we'll be covering other edge devices as well, All edge about to hit Warp speed!

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#154

Post by tristor »

sanchitmonga22 wrote: post #110
tristor wrote: post #8

> What would you build if on-device AI were genuinely as fast as cloud? I think this has to be the future for AI tools to really be truly useful. The things that are truly powerful are not general purpose models that have to run in the cloud, but specialized models that can run locally and on constrained hardware, so they can be embedded. I'd love to see this able to be added in-path as an audio passthrough device so…

This is a great idea. A virtual audio device that sits in the path of any audio stream and provides live transcription, that would be huge for video conferencing, lectures, podcasts. MetalRT's STT numbers make this feasible: 70 seconds of audio transcribed in 101ms means you could process audio chunks in real-time with massive headroom. The latency would be imperceptible. We haven't built this yet but it's a compelli…

Something that could be possible is serving the model as a virtual audio device and then you can use existing tools on macOS like Rogue Amoeba's Loopback to direct audio to split to that virtual device and your other output (you'd configure your Loopback device as the output in your system audio settings).

I have never written audio drivers on macOS, but maybe something worth exploring to see if I can make this happen. I really appreciate high quality AI transcripts in my meetings, but right now only Webex has good transcriptioning, and a lot of meetings use other services like MS Teams, Zoom, Meet, et al.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#155

Post by elpakal »

sanchitmonga22 wrote: post #151
elpakal wrote: post #147

Earlier quoted context omitted.

> This is a known limitation with small LLMs (0.6B-1.2B) doing tool calling. To me this is this nut to crack, wrt tool calling and locally running inference. This seems like a really cool project and I'm going to dive around a little later but if it's hallucinating for something as basic as this makes me think it's more of POC stage right now (to echo other sentiment here).

That's a fair read. Tool calling reliability with sub-4B models is genuinely the hardest unsolved problem in on-device AI right now. The inference engine (MetalRT) is production-grade, the pipeline architecture is solid, but the models at this size are still the weak link for complex tool routing. Larger model support (where tool calling is much more reliable) is next on the roadmap. Please stay tuned!

Sorry, I scrolled through some of the rest of the comments on this thread and can’t stay tuned.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#156

Post by j45 »

Tacite wrote: post #44
j45 wrote: post #38

"Apple M3 or later required. MetalRT uses Metal 3.1 GPU features available on M3, M3 Pro, M3 Max, M4, and later chips. M1/M2 support is coming soon. On M1/M2, RCLI automatically falls back to the open-source llama.cpp engine."

Funny you mention that because on their github they just pushed an update to say that it didn't work M3 and M4.

The quote was from the Github page.

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#158

Post by druide67 »

concats wrote: post #138
shubham2802 wrote: post #76

Earlier quoted context omitted.

RunAnywhere builds software that makes AI models run fast locally on devices instead of sending requests to the cloud. Right now, our focus is Apple Silicon. Today there are two parts: MetalRT - our proprietary inference engine for Apple Silicon. It speeds up local LLM, speech-to-text, and text-to-speech workloads. We’re expanding model coverage over time, with more modalities and broader support coming next. RCLI -…

How does it compare for models of any meaningful size? These 0.6B-4B models are, frankly, just amusing curiosities. But commonly regarded as too error prone for any non-demo work. The reason why people are buying Apple Silicon today is because the unified memory allows them to run larger models that are cost prohibitive to run otherwise (usually requiring Nvidia server GPUs). It would be much more interesting to see…

[dead]

Re: Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

#160

Post by solarkraft »

coder543 wrote: post #25
alfanick wrote: post #9

I'm not looking for STT->AI ->TTS, I'm looking for truly good voice-to-text experience* on Linux (and others). Siri/iOS-Dictation is truly good when it comes to understanding the speech. Something this level on Linux (and others) would be great, yeah always listening, maybe sending the data somewhere, but give me UX - hidden latency, optimizing for first chars recognized - a good (virtual) input device.

> Siri/iOS-Dictation is truly good when it comes to understanding the speech. What...? It is terrible, even compared to Whisper Tiny , which was released years ago under an Apache 2.0 license so Apple could have adopted it instantly and integrated it into their devices. The bigger Whisper models are far better, and Parakeet TDT V2 (English) / V3 (Multilingual) are quite impressive and very fast. I have no idea what w…

Its quality isn’t great, but it is damn fast and that matters a lot! Whisper doesn’t even work live without hacks.
Post reply on HN