Viewing profile — sanchitmonga22

[flagged]

please check our main repo: https://github.com/RunanywhereAI/runanywhere-sdks/ We are running anywhere, hence RunAnywhere, MetalRT is the fastest inference engine we made for Apple…

Yes, that's the plan. MetalRT will ship as part of the RunAnywhere SDK so other developers can integrate it into their own apps. We're working on making that available. If you want…

That's a fair read. Tool calling reliability with sub-4B models is genuinely the hardest unsolved problem in on-device AI right now. The inference engine (MetalRT) is production-gr…

That tracks with what we've seen too. For agent workflows with reliable tool calling, you really do need the larger models. Larger model support is a priority for us. Thanks for th…

Fair criticism. Our benchmarks are on small models because MetalRT was built for the voice pipeline use case, where decode latency on 0.6B-4B models is the bottleneck. You're right…

Good correction, thanks. You're right that NAX and ANE are distinct, I shouldn't have conflated them. NAX's ability to accelerate LLM prefill is exactly the kind of capability that…

Yes, mobile is our primary offering and it is on the roadmap. The same Metal GPU pipeline that powers MetalRT on macOS maps directly to iOS (same Apple Silicon, same Metal API)

Agreed for a lot of use cases. RCLI supports text-only mode (--no-speak flag or just type in the TUI instead of using push-to-talk). TTS makes sense for hands-free / eyes-free scen…

We use AI tools in our workflow, same as a lot of teams at this point. The pipeline architecture, Metal integration, and engine design are ours. The code is MIT and open for anyone…

RCLI includes local RAG out of the box. You can ingest PDFs, DOCX, and plain text, then query by voice or text: rcli rag ingest ~/Documents/notes rcli ask --rag ~/Library/RCLI/inde…

Fair point. The install script shouldn't silently install Homebrew without explicit consent. We'll update it to detect when Homebrew is missing and prompt the user before installin…

Cool, just checked out dlgo. Looks like you're targeting Go bindings for on-device inference? Different approach but same conviction that this should run locally. Happy to compare …

Apple has the silicon, the frameworks (MLX, CoreML), and the models. The gap is putting it all together into a fast, unified on-device pipeline. That's what we're focused on, and h…

Absolutely, we'd welcome a Portfile contribution. Happy to review and merge. If halostatue wants to co-maintain, even better. Feel free to open a PR or issue on the RCLI repo and w…

Understood, you want dictation, not a chatbot. That's a valid and different use case. RCLI is Apple Silicon only today because MetalRT is built on Metal. For Linux, the closest thi…

This is a great idea. A virtual audio device that sits in the path of any audio stream and provides live transcription, that would be huge for video conferencing, lectures, podcast…

This is exactly the problem we're trying to solve. The models themselves have gotten surprisingly capable at small sizes, Qwen3.5 4B with 262K context, LFM2 1.2B for fast tool call…

The default TTS voice (Piper) is a lightweight model optimized for speed over quality. It's fast but yeah, it doesn't sound great. If you install Kokoro TTS (rcli models > TTS sect…

Fair criticism. The action executed on the LLM side but didn't translate to the correct macOS action, the model hallucinated success instead of routing to the open_url tool. This i…

Thanks for trying it and for filing the bug, we're looking into the homebrew install issue. On unsloth quants: agreed, they're consistently better bit-for-bit. Adding broader quant…

Ha, not yet. Metal 4 is interesting and we're keeping an eye on it. MetalRT currently targets Metal 3.1 GPU compute because that's where we get the most control over the decode pip…

Not yet with MetalRT, right now we support models up to ~4B parameters (Qwen3 4B, Llama 3.2 3B, LFM2.5 1.2B). These are optimized for the voice pipeline use case where decode speed…