Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

blog.simbastack.com

Post reply on HN
Print view View original

101–110 of 151 posts

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#101

Post by nl »

carpo wrote: post #100
nl wrote: post #98

Earlier quoted context omitted.

Try some of the models on OpenRouter if you are looking to save money. Gemma 4 31B is $0.12/M input, $0.37/M output vs $1/M input, $5/M output for Haiku. There are other options that are good too. Gemini 3.1 Flash Lite is great for this kind of thing (NOT Gemini 3.5 Flash though - the pricing for that is bad). https://openrouter.ai/google/gemma-4-31b-it

Cheers, I'll give it a try. How are those models at returning structured results? When I was writing the prompts for the analysis step and testing with older Claude models, it would have trouble structuring the XML consistently. Sonnet 4.6 handles it really well.

Use function calling/tool use, not XML output. The models are all trained for that now.

Ie, instead of telling it to generate

  Name
  19
  whatever
give it a function

  details(name: string, age: int, address: string)
That is actually a JSON schema, and the models do great at it. Here's the claude docs, but they are all similar: https://platform.claude.com/docs/en/agents-and-tools/tool-us...

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#102

Post by bahmboo »

throwawaytea wrote: post #56
cold_harbor wrote: post #47

the reason 50GB swap is even viable here is Apple Silicon's memory bandwidth. on x86 that much swap would make inference unusably slow

Memory bandwidth or storage bandwidth?

potato potatoh

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#103

Post by carpo »

nl wrote: post #101
carpo wrote: post #100

Earlier quoted context omitted.

Cheers, I'll give it a try. How are those models at returning structured results? When I was writing the prompts for the analysis step and testing with older Claude models, it would have trouble structuring the XML consistently. Sonnet 4.6 handles it really well.

Use function calling/tool use, not XML output. The models are all trained for that now. Ie, instead of telling it to generate Name 19 whatever give it a function details(name: string, age: int, address: string) That is actually a JSON schema, and the models do great at it. Here's the claude docs, but they are all similar: https://platform.claude.com/docs/en/agents-and-tools/tool-us...

Very interesting. Thank you!

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#104

Post by fenix1851 »

bonoboTP wrote: post #44
jaggederest wrote: post #28

Earlier quoted context omitted.

Hey friend, try something in this ballpark, your post has a bunch of painful AI tropes: https://github.com/blader/humanizer You get a pass here because you're doing really cool stuff but it's kinda tough to read past the AI nonsense, and it's relatively easy to screen out "it's not x it's y" kind of things and the bolded bullet points.

I don't dislike those tropes because they are frequent or because they are not pleasing to read intrinsically. I dislike them because it tells me it was made by AI and AI output varies strongly in quality and most of it is low on insight but rings the right bells to make it seem insightful. It indicates a lack of human care. Hiding these clues by another AI pass doesn't solve the core problem. Now you just end up wit…

it’s a different story.

I was highly interested in reading this article from start to finish.

Ofc there are was a lot of slop moments, but author experience itself is great!

And i genuinely don’t care if he would share it through LLM article.

Just please remove slop markers :)

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#105

Post by 0x38B »

embedding-shape wrote: post #22
desro wrote: post #7

> The skill is open at ~/.claude/skills/video-index/. If you're working on something similar (indexing personal archives, getting a local model to do real archival work, building agents that drive editing tools), I'd be glad to compare notes. When your Claude wrote this post they might not have selected the right URL to share, unless your home folder is exposed. Care to share the skill files?

We just got a modern example of the classic message from a friend who just picked up programming, containing: "I just created my own web app, wanna check it out? It's here: http://localhost:8080"

Different context, but I sent a message like that in Signal the other day to a family member with a link to my IP, pointing to `Python -m http.server` running in a directory with a file for them to try (1). Easier than having them open my Samba share.

1: To get an Android app working that has been delisted and requires a 'key' app that you purchase. We did purchase it, but didn't think to make any backups.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#106

Post by asenna »

Confiks wrote: post #72

I'm not quite sure why all that swapping is necessary. I really does age your SSD quite fast considering the enormous memory bandwidth required. Gemma 4 31B at 4-bit quantization should only be around 19 GiB [1], not 28.4 GiB. I'm not feeding it images regularly, so I'm not sure how much memory it needs to get those into context, but I can't imagine it is more than 10 GiB. The activity monitor does show all kinds of…

Yeah to be fair, I could've cleaned everything up but this was taken when I was doing other work on my laptop while the screenshot was taken.

Although slightly laggy, I was impressed by the fact that I was still able to work on other things and have a bunch of tabs open on my Brave browser.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#107

Post by pavlov »

The content is good, but this LLM writing style gets tiresome. Everything is a revelation:

>“I bought it for Chrome. It's running a model that didn't exist when I bought it.”

Well duh, personal computers run new software. That’s literally the whole point. The Apple II didn’t sell on the strength of the preinstalled apps.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#109

Post by asenna »

cataphract wrote: post #68
gitowiec wrote: post #32

Reading this text feels strange, sentences seems to be detached

I had exactly the same impression, and I recall seeing this style other times recently. First time I thought it was just bad writing skills, now I'm thinking it's AI generated.

I'm the author, yes it is AI-assisted.

You can make AI-generated content without it being slop. Slop, to me at least, is content that's wrong, padded, or generic.

I see the cadence / short-sentence issues but if there's something else beyond those, I'd actually want to know what made it feel bad.

I would've put off documenting what I did over the weekend but instead, I did document everything, spent quite some time (several iterations) and effort to make sure it does not hallucinate and writes in my own tone and voice. I'm sure it could be better but the content is not made-up.

At a time where most of us software engineers have changed our workflows to let AI write 80+% of our code using agents, I feel writing is heading the same way. It then becomes a matter of taste, whether it's done well or not.

If you're looking clues and signs for whether a content has used AI, you're going to be disappointed over the next 12 months.

If it feels jarring right now, I'll work harder on the workflow so it feels more natural next time (someone shared this project with me - https://github.com/blader/humanizer).

But this clearly allows me to make content which I wouldn't have done earlier.

Re: Indexing a year of video locally on a 2021 MacBook with Gemma4-31B (50GB swap)

#110

Post by asenna »

pavlov wrote: post #107

The content is good, but this LLM writing style gets tiresome. Everything is a revelation: >“I bought it for Chrome. It's running a model that didn't exist when I bought it.” Well duh, personal computers run new software. That’s literally the whole point. The Apple II didn’t sell on the strength of the preinstalled apps.

Author here. I totally hear you. I wasn't expecting this to do well on HN for exactly this reason.

But I've mentioned elsewhere - if it wasn't for all the AI-assistance, I would've put-off documenting everything that I did and not even get to the writing part.

But yeah, I'll be working on the workflow to make the next write-up better, more humanized.

Post reply on HN