Viewing profile — gitpusher42

Yeah, original TurboFieldfare supports Gemma https://github.com/drumih/turbo-fieldfare

Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README. I am glad it inspired more people to explore area of on-devic…

It is only a wild guess, but if you have 256gb version and a lot of apps running it can be pretty slow.

Thank you for testing and sharing results! I think I might understand your use case. You ssh the Mac and want something like `ollama run` with an interactive chat in terminal. Am I…

I think there is a limit based on MoE number of active parameters and quantisation, bytes count for active experts. But I believe we will see more project like this for different m…

Uh, maybe, who knows. I am pretty bad solving leetcode, btw

Yeah, gpt-oss-120b is also MoE, so the same ssd-streaming and caching ideas should work. Feel free to fork and try implementing it!

Yeah, it must be exactly the same. The same weights are used, nothing skipped or pruned. But it might have differ to MLX for greedy decode because of small floating-point nums diff…

yeah, looks like a page cache matters a lot I tested on mine m5 pro with 8gb memory pressure, got 27t/s instead of 35t/s Someone tested on m4 max. In regular state it was 48tok/s, …

Apple does something similar with their latest foundation model. They process input prompt and based on results they preload required experts. Quite neat solution for the edge devi…

Yeah, sure! You can select different options in the app settings at the right panel, it shows how much memory it will use For CLI and Server, use --max-context

uh, tried most of this mmap benchmark did basically page touch experiment and cold reads were much slower, unfortunately (10ms vs 3ms) I tried MADV_WILLNEED, F_RDADVISE and preadv.…

afaik there is some research at this area. Also the new apple foundation model uses related idea. they process the whole prompt and based on prompt load required experts and use on…

Yeah, must be possible. Not fast, but possible if you have enough ram. I think you can search online for projects, I think I saw something related

uh, don't worry. Just install the latest Xcode from the App Store. It includes everything you need to run this project

It heavily relies on M-series Mac unified memory architecture. And shaders are written using Metal, Apple's own gpu programming technology. It cannot be ported directly to classic …

Not sure it will be really usable. Check for Flash-Moe and Colibri repos A lot of request for qwen3.6 moe, it might worth exploring

Yeah, Gemma is not the best for coding I guess. qwen must be better

Yeah, the same ideas should work for qwen. You can try porting this engine to use Owen. Owen 3.6-35b-a3b was my initial idea, but I switched to Gemma because of its simpler archite…

Thank you for testing and sharing, it is useful info!

Thank you very much for sharing! Great results and useful info!

Thanks! Will try!

uh, it's a bit difficult to discuss the classical approach with vRAM and regular RAM. Not really familiar with optimisations and hacks, I always worked with apple platforms and sha…