Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

blog.simbastack.com

Post reply on HN
Print view View original

11–20 of 151 posts

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#11

Post by ahknight »

busfahrer wrote: post #10
egorfine wrote: post #9

Thanks for the article! I have a beefy M5 Pro and I'm eagerly looking around for ways to use local models (specifically Gemma4 & Qwen3.6). This is an excellent thing to do. Especially that LLMs excel at batching thus you can index multiple photos and videos in parallel for no performance penalty.

I have been contemplating a M5 Pro MBP, but for the life for me I wasn't able to find benchmarks for real-world models, do you happen to know how many tokens per second roughly you get with MoE models like Qwen 3.6 35B/A3B or Gemma 4 26B?

I'm not normally one to share videos as answers, but this particular fellow does a LOT of work with local AIs and Macs and happens to have a nuanced answer. https://youtu.be/XGe7ldwFLSE

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#12

Post by satvikpendem »

egorfine wrote: post #9

Thanks for the article! I have a beefy M5 Pro and I'm eagerly looking around for ways to use local models (specifically Gemma4 & Qwen3.6). This is an excellent thing to do. Especially that LLMs excel at batching thus you can index multiple photos and videos in parallel for no performance penalty.

Unsloth Studio [0] is what I recommend these days, open source alternative to the more widely known LM Studio, and also built by the people who make good quantizations of released models. With MTP support not merged in you should get 2x token generation speed with no accuracy difference. They also have MLX quants if you scroll down a bit, which is a format specifically for macOS' Metal GPU acceleration but that's not integrated into Unsloth Studio just yet.

[0] https://unsloth.ai/docs/models/qwen3.6#mtp-guide

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#13

Post by ahknight »

asenna wrote: post #5
andai wrote: post #2

Awesome. Say, this is very comprehensive. I was vaguely aware of all these pieces existing (except for running a facial recognition database at home o_o), but it's really neat to put them all together like that.

Thanks! I was honestly casually trying it out on the side with Claude's help. And I was actually pleasantly surprised to see how good the result was. Still blows my mind I can do all this from my 2021 MBP. I'll try to do a post once I have the next steps working (helping with planning and editing videos with Davinci Resolve).

I also have a 64GB M1 Max and am similarly impressed with what that workhorse can do. The M5 tempted me -- a lot -- but then I looked at what I was already getting done on that machine and just couldn't justify it ... yet. Someday, surely, but not yet. Gemma4 gave all my local projects new life, just like what you did here.

Great job. Long live the M1 Max!

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#14

Post by asenna »

asenna wrote: post #8
desro wrote: post #7

> The skill is open at ~/.claude/skills/video-index/. If you're working on something similar (indexing personal archives, getting a local model to do real archival work, building agents that drive editing tools), I'd be glad to compare notes. When your Claude wrote this post they might not have selected the right URL to share, unless your home folder is exposed. Care to share the skill files?

Oops! My bad. Fixing it now. And yeah, I can share the Skill file. Give me 5 mins.

Ok I scrambled to finalize a name for it and create a new repo for it - https://github.com/Simbastack-hq/framedex

PS - I just put this together in the last few mins, removed my personal files and references. So it's not tested properly, please let me know if any issues.

It's still an early hack, but I have thousands of still images as well from my camera which I've not processed and I need to do the same analysis for those.

So I'll continue working on it, but happy to receive any PRs if anyone finds any use for it.

I'm tired of having a backlog of thousands of images and videos, leaving it for later.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#15

Post by herf »

Two questions:

1. What is the search index?

2. The "description.md" example has things like "faces -> cluster_id". Is this from Davinci Resolve's face index? Things like faces+names and locations are really important with photo collections, but general LLMs don't handle them so well.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#16

Post by asenna »

UPDATE: Quickly created a repo for this - https://github.com/Simbastack-hq/framedex (MIT License)

It's not tested properly after I genericized it. Will try to go through it properly and add more updates.

Two big things on my TODO: 1) Make use of this indexing and using Claude's help, make video editing faster with Davinci Resolve (now that I have a good index of all the content)

2) I currently did this for videos, but I want to add more things to this for my thousands of still images of my camera - need to make sense of them. So I'll be working on this as well.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#17

Post by egorfine »

busfahrer wrote: post #10
egorfine wrote: post #9

Thanks for the article! I have a beefy M5 Pro and I'm eagerly looking around for ways to use local models (specifically Gemma4 & Qwen3.6). This is an excellent thing to do. Especially that LLMs excel at batching thus you can index multiple photos and videos in parallel for no performance penalty.

I have been contemplating a M5 Pro MBP, but for the life for me I wasn't able to find benchmarks for real-world models, do you happen to know how many tokens per second roughly you get with MoE models like Qwen 3.6 35B/A3B or Gemma 4 26B?

Qwen 3.6 35B running on oMLX 0.3.9rc1: on oMLX I get 86 t/s on Q4 and 74 t/s on Q6.

Bear in mind that ttft on MLX is much much faster on M5 Pro as compared to M4 Pro.

Also bear in mind that those figures are with NO optimizations whatsoever: no MCP, no DFlash. I am waiting for both to be released for the Qwen models.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#18

Post by egorfine »

satvikpendem wrote: post #12
egorfine wrote: post #9

Thanks for the article! I have a beefy M5 Pro and I'm eagerly looking around for ways to use local models (specifically Gemma4 & Qwen3.6). This is an excellent thing to do. Especially that LLMs excel at batching thus you can index multiple photos and videos in parallel for no performance penalty.

Unsloth Studio [0] is what I recommend these days, open source alternative to the more widely known LM Studio, and also built by the people who make good quantizations of released models. With MTP support not merged in you should get 2x token generation speed with no accuracy difference. They also have MLX quants if you scroll down a bit, which is a format specifically for macOS' Metal GPU acceleration but that's not…

I have researched for quite a bit and so far the fastest runtime is the oMLX one. But there's a caveat: ttft on MLX on M4 Pro is enormous. On M5 Pro it has been greatly sped up.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#20

Post by asenna »

herf wrote: post #15

Two questions: 1. What is the search index? 2. The "description.md" example has things like "faces -> cluster_id". Is this from Davinci Resolve's face index? Things like faces+names and locations are really important with photo collections, but general LLMs don't handle them so well.

1) It's just simple plain-text `.description.md` sidecar files, one per clip, sitting next to each video.

Something which I can query later - Like when brainstorming with Claude "I wanna make some videos of the Luxury rooms in the lodge" and it knows what all videos could help here (going through the files).

There's also a folder root level files that aggregates the text descriptions to make it easier to find.

I've just attached an image in the blog showing an example - https://blog.simbastack.com/_media/gvcycx2n.png

2) No - nothing from DaVinci Resolve. Framedex is a standalone pipeline. Resolve isn't involved.

Faces come from insightface (the open-source buffalo_l pack - RetinaFace for detection), running locally on CPU. For each clip it detects faces in the sampled frames, embeds them, and writes rows to ~/.framedex/faces.db.

Tbh, this part I know it's building up in my local DB but I haven't tested how good is it. Will check them out properly soon.

But yeah, on your broader point that's why framedex deliberately does not ask the LLM to handle faces or locations.

----

Faces → insightface / ArcFace embeddings. Deterministic, comparable across clips. The vision model only contributes a rough people_count; it never tries to identify anyone.

Locations → EXIF GPS via exiftool, reverse-geocoded through Nominatim/OpenStreetMap. Hard metadata, not a guess.

The LLM only does what it's good at: scene description, mood, shot type, keywords, keep/review/cull rating (this last part is also debatable though).

Post reply on HN