321–330 of 382 posts
Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea. Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s. Page cache seems to be why it beats the M5 Pro. W…
Thank you very much for sharing! Great results and useful info!
Earlier quoted context omitted.
Any idea if madvise helps? Admittedly I have very limited experience and only on Linux
MADV_SEQUENTIAL might help a bit, but not that much. Biggest problem here is throughput-vs-latency. With mmap()-ed file, for each pagefault, kernel will conservatively estimate block size to page in, so you'll have a ton of relatively small requests going to SSD. This would be IOPS-bound, and likely under-perform relative to maximum possible bytes/second throughput. With explicit read()/pread(), kernel & SSD can work…
Yeah, I was thinking MADV_WILLNEED might work there but not sure
Earlier quoted context omitted.
It depends on the use case. I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s. The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly co…
can I vary the context length depending on RAM available?
For CLI and Server, use --max-context
Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…
Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…
> Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about. The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.
And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol).
It's not about the people there being smart or not, it's about their and the company's priorities, resources and what they choose to focus on.
Earlier quoted context omitted.
> Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about. The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.
Anthropic's API has two nines availability and Claude Code is a TUI made with React that can regularly consume more than 1GB of RAM, and the codebase is utter slop. They couldn't fix the flickering bug for over a year! And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol). It's not about the people there being smart or not, it's about their and the company's priorities, resou…
The second would be hardware savings on the order of tens of billions of dollars if they were stupid not do try all possible optimizations.
Spot the difference.
Earlier quoted context omitted.
The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood
Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache. Try to leave only a gigabyte or two free, speed likely would drop dramatically. Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.
Earlier quoted context omitted.
The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood
Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache. Try to leave only a gigabyte or two free, speed likely would drop dramatically. Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.
Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea. Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s. Page cache seems to be why it beats the M5 Pro. W…