141–150 of 151 posts
-31b It's a dense model
-how many tokens/s is it running at
-What temps are the M1 max GPU/CPU running at
-Is it mlx or gguf
-Why 31b and not 26b which is moe and much more efficient on the m1 max at 50tokens/s & low temps.
I personally use (MLX) qwen3.6-35b-8bit mostly, but use Gemma-4-26b-4bit for image analysis, its mind blowing how fast it is at identifying the scene in a photograph.
Earlier quoted context omitted.
You need to ask macOS people for their prefill speed as well, there are two numbers you care about here, and current MacBooks have generally terrible numbers when it comes to prefill performance. Surely it'll get better with time, but if you already have a desktop, I'd go the "beefy GPU" route first.
> current MacBooks have generally terrible numbers when it comes to prefill performance Previous MacBooks. Prefill speed on M4 Pro and M5 Pro are hugely different.
Earlier quoted context omitted.
> current MacBooks have generally terrible numbers when it comes to prefill performance Previous MacBooks. Prefill speed on M4 Pro and M5 Pro are hugely different.
Alright, show me the numbers then, whenever I ask any macOS people about their prefill speed instead of generation speed, they all seem to disappear :P
did you know that this existed and is pretty good and doesn't hog 50GB of swap? https://github.com/iliashad/edit-mind
did you know that this existed and is pretty good and doesn't hog 50GB of swap? https://github.com/iliashad/edit-mind
Awesome, thank you for mentioning my project. Much appreciated
Earlier quoted context omitted.
Awesome, thank you for mentioning my project. Much appreciated
When I saw the original article - i thought this should totally be possible without needing that much ram, so let me just build it. So I started looking at the models I needed for each (whisper, florence etc). Before building, i decided to do one more search on the internet, and I found your project. That's exactly what I was going to build. Good work!!
Earlier quoted context omitted.
> the fans spinning at max speed This always confuses me - don't people want their computations to run as fast as possible and thus inevitably produce more heat that needs to be vented? I suppose sometimes it is just an analogy for "its utilizing 100% of my resources" (which I'm guessing it is here), but I've definitely had people say it as an actual complaint in different contexts
> I've definitely had people say it as an actual complaint in different contexts I think fan loudness is an outgrowth of conspicuous consumption because a certain OEM decided to make it a marketing bullet-point. I was equally disappointed by by people - especially device reviewers - banging on the drum that phones made of plastic "didn't feel premium", and we got phones with glass backs that have to be shoved into pl…
Earlier quoted context omitted.
Use function calling/tool use, not XML output. The models are all trained for that now. Ie, instead of telling it to generate Name 19 whatever give it a function details(name: string, age: int, address: string) That is actually a JSON schema, and the models do great at it. Here's the claude docs, but they are all similar: https://platform.claude.com/docs/en/agents-and-tools/tool-us...
Hey, just want to thank you for this suggestion. Spent this morning swapping to open router and changing all my prompts to use tools instead of XML. Not only is Gemma and Gemini much cheaper, the output tokens from the tool call are much less too. Cost to analyse one 20 minute video with 10 snapshots went from $0.21 to $0.009, and I'm even sending full HD snapshots instead of the 960x540 ones I was sending before (to…
Very interested in the full run details.
Earlier quoted context omitted.
Hey, just want to thank you for this suggestion. Spent this morning swapping to open router and changing all my prompts to use tools instead of XML. Not only is Gemma and Gemini much cheaper, the output tokens from the tool call are much less too. Cost to analyse one 20 minute video with 10 snapshots went from $0.21 to $0.009, and I'm even sending full HD snapshots instead of the 960x540 ones I was sending before (to…
Ha! So glad it helped you! Very interested in the full run details.