Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

blog.simbastack.com

Post reply on HN
Print view View original

71–80 of 151 posts

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#71

Post by cortesoft »

bonoboTP wrote: post #44
jaggederest wrote: post #28

Earlier quoted context omitted.

Hey friend, try something in this ballpark, your post has a bunch of painful AI tropes: https://github.com/blader/humanizer You get a pass here because you're doing really cool stuff but it's kinda tough to read past the AI nonsense, and it's relatively easy to screen out "it's not x it's y" kind of things and the bolded bullet points.

I don't dislike those tropes because they are frequent or because they are not pleasing to read intrinsically. I dislike them because it tells me it was made by AI and AI output varies strongly in quality and most of it is low on insight but rings the right bells to make it seem insightful. It indicates a lack of human care. Hiding these clues by another AI pass doesn't solve the core problem. Now you just end up wit…

I feel like human copywriters have been using those same tricks for clickbait articles for years…

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#72

Post by Confiks »

I'm not quite sure why all that swapping is necessary. I really does age your SSD quite fast considering the enormous memory bandwidth required. Gemma 4 31B at 4-bit quantization should only be around 19 GiB [1], not 28.4 GiB. I'm not feeding it images regularly, so I'm not sure how much memory it needs to get those into context, but I can't imagine it is more than 10 GiB.

The activity monitor does show all kinds of Electron apps active, on top of a presumably model-loaded Handy and a virtual machine for Claude Code, so I guess that's the real root cause for all the swapping. If your laptop starts trashing I can't imagine you have any use for those apps, which will grind to a halt.

[1] https://huggingface.co/mlx-community/gemma-4-31b-it-4bit

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#74

Post by egorfine »

busfahrer wrote: post #65
egorfine wrote: post #17

Earlier quoted context omitted.

Qwen 3.6 35B running on oMLX 0.3.9rc1: on oMLX I get 86 t/s on Q4 and 74 t/s on Q6. Bear in mind that ttft on MLX is much much faster on M5 Pro as compared to M4 Pro. Also bear in mind that those figures are with NO optimizations whatsoever: no MCP, no DFlash. I am waiting for both to be released for the Qwen models.

Great, thanks! :-) and to mirror another poster: what kind of prompt parsing (prefill) speed do you get for that model? Also how is the speed for the 27B model?

35B: 1300-1800 t/s on both Q4 and Q6.

27B: give me 20 minutes

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#75

Post by asenna »

enos_feedler wrote: post #57
asenna wrote: post #51

Earlier quoted context omitted.

A huge thing here was the massive amount of data that was just processed - I went through about 1TB of files over 24 hours. Using API to analyze even a subset of this would've been painful imo.

I thought about that in this video case and it's true. I thought the parent comment was making a broader statement about local models in general. But even with video, if it was stored in private cloud storage near the LLM could this still have worked efficiently? What are the most painful elements of this whole setup / work environment if everything was cloud?

Oh yes, if everything is cloud, then this is a non-issue.

The few other points of consideration would be:

1) Cost - I was considering using Sonnet for this but there's always the concern of reaching limits OR the API cost if you're using the API.

The feeling of knowing you have a capable model in your hands without any limits is actually pretty awesome. Your mind starts running at what else can I throw at it to do grunt work.

2) Privacy issues - same as with moving to cloud.

3) Reliability issues - I know from experience Claude uptime has been pretty bad the past few months

4) Restrictions - Claude has been pretty heavy handed with their restrictions lately, anything which remotely triggers there flags gets an instant denial (or worse, an account ban). Often these are false-positives.

I love the value I get from Claude but there's a different kind of freedom you get with local, capable models.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#76

Post by egorfine »

busfahrer wrote: post #10
egorfine wrote: post #9

Thanks for the article! I have a beefy M5 Pro and I'm eagerly looking around for ways to use local models (specifically Gemma4 & Qwen3.6). This is an excellent thing to do. Especially that LLMs excel at batching thus you can index multiple photos and videos in parallel for no performance penalty.

I have been contemplating a M5 Pro MBP, but for the life for me I wasn't able to find benchmarks for real-world models, do you happen to know how many tokens per second roughly you get with MoE models like Qwen 3.6 35B/A3B or Gemma 4 26B?

Qwen3.6 27B oQ6: 12.5 t/s generation, 340-360 t/s pp.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#77

Post by egorfine »

busfahrer wrote: post #10
egorfine wrote: post #9

Thanks for the article! I have a beefy M5 Pro and I'm eagerly looking around for ways to use local models (specifically Gemma4 & Qwen3.6). This is an excellent thing to do. Especially that LLMs excel at batching thus you can index multiple photos and videos in parallel for no performance penalty.

I have been contemplating a M5 Pro MBP, but for the life for me I wasn't able to find benchmarks for real-world models, do you happen to know how many tokens per second roughly you get with MoE models like Qwen 3.6 35B/A3B or Gemma 4 26B?

Native MCP:

For Qwen 35B enabling native MCP on MLX models slows it down by 10%.

For Qwen 27B enabling native MCP on MLX models speeds token generation up almost exactly 1.5x.

(all tested on M5 pro).

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#79

Post by genxy »

dang wrote: post #69
oceanus wrote: post #67

[flagged]

Could you please not post generated comments to HN? It's not allowed here. See https://news.ycombinator.com/newsguidelines.html#generated and https://news.ycombinator.com/item?id=47340079 . We ban accounts that do this and I don't want to ban you, so please write everything that you post to HN by hand. Of course, it's impossible to know for sure what was LLM processed or not, but we're getting complaints about some o…

The article itself has many AI tells. Can we update the guidelines on AI generated content ?

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#80

Post by asenna »

teach wrote: post #62
asenna wrote: post #49

Earlier quoted context omitted.

Tbh, I did spend a lot of time trying to ground it and de-slopify it - verified nothing was halucinated and went through 10 iterations to get to this. It's almost like wrestling with Claude and I knew it would be tough on HN. But because of the fear of non-perfection, I used to put away things like creating this article or even posting it anywhere. And I do think the article has real value that HN would appreciate (I…

For what it's worth, I really enjoyed this read and almost came here to comment "this is the most enjoyable llm-assisted article I've read in a while" The tells were unmistakable but it still had a human touch, so I for one am glad you published anyway.

I'm definitely learning and hope to do better next time but your comment truly means a lot.

I kid you not, I've taken a screenshot of this to motivate me next time I'm doubting publishing :)

Post reply on HN