What is the “middlest” Mac one could get for this? I’m in the market but keep going back and forth between a 64gb m5 pro or “lower end m5 air and screw it I’ll just pay for cloud tokens”. At current prices the 2-3k diff to try to run something local that isn’t as powerful could buy a lot of tokens.
Nativ: Run frontier open models locally on your Mac
111–120 of 144 posts
Re: Nativ: Run frontier open models locally on your Mac
#112Earlier quoted context omitted.
That's not what people are normally referring to when they say "frontier models". It means the most capable models full stop. Not the most capable that you can run locally.
I’ve always seen the frontier shown as a curve on a plot. Frontier ≠ SOTA
Frontier = SOTA.
Pareto frontier ≠ frontier.
Re: Nativ: Run frontier open models locally on your Mac
#113How does this compare to LM Studio Bionic?
Why do ppl downvote this comment? HN makes no sense sometimes...
Re: Nativ: Run frontier open models locally on your Mac
#114What is the “middlest” Mac one could get for this? I’m in the market but keep going back and forth between a 64gb m5 pro or “lower end m5 air and screw it I’ll just pay for cloud tokens”. At current prices the 2-3k diff to try to run something local that isn’t as powerful could buy a lot of tokens.
You can run a lot of these on e.g. M1 max 64gb
Re: Nativ: Run frontier open models locally on your Mac
#115I really don't like the marketing texts.
"Why we’re open source when nobody else is."
I'm using oMLX which is open source and seems to be doing everything Nativ offers. I'd rather see the comparison with existing "non-existing" open source competitors.
Re: Nativ: Run frontier open models locally on your Mac
#116I really don't like the marketing texts. "Why we’re open source when nobody else is." I'm using oMLX which is open source and seems to be doing everything Nativ offers. I'd rather see the comparison with existing "non-existing" open source competitors.
What they likely mean is, why options like LM Studio are not open source.
Re: Nativ: Run frontier open models locally on your Mac
#117I am still rocking Sequoia and this targets Tahoe purely from UI constraints, no like.
Re: Nativ: Run frontier open models locally on your Mac
#118So should I ditch ollama for this?
Re: Nativ: Run frontier open models locally on your Mac
#119To clarify, this MIT-licensed app is from the very same dev, 'Prince Canuma', who maintains the popular MLX-VLM library ( https://github.com/Blaizzy/mlx-vlm ). MLX-VLM is a long-time dependency of the excellent LM Studio and others because it can provide faster inference on Apple devices than llama.cpp. Historically, MLX is a smaller community than CUDA, but has some of the fastest updates upon the release of new mod…
Switching to mlx-vlm is basically harmful since it (like vllm and sglang) have such garbage support for modern samplers. To be clear, I am one of the authors on the min_p paper, and if min_p is the best you have (when llamacpp supports the far superior top-n-sigma), than I have no reason to switch even if you are somehow faster. ( https://arxiv.org/abs/2411.07641 ) And if you do care to support modern samplers, you c…
Thanks for bringing this up. You're 100% right. But most people, even technical ones are oblivious to how much of a difference modern samplers and higher quality quantization algorithms make for on-device LLM inference and are stuck with good old top-p, top-k samplers and RTN quantization.
TBF mlx-vlm does support min-p sampling, but none of the other modern samplers that you list. Ollama and LM Studio are even worse with only top-p and top-k samplers.
Re: Nativ: Run frontier open models locally on your Mac
#120Earlier quoted context omitted.
> tfa's title says "frontier open models" Does it? I read: "Run AI models locally on your Mac."
The submission is "Nativ: Run frontier open models locally on your Mac" and on the page it says "Nativ puts frontier intelligence on your desk."
> puts frontier intelligence on your desk
Which of the models it puts on your desk do you consider frontier intelligence?