I’m currently punishing Fable by making it watch the entire series of 7th Heaven.
Inhumane
Was it bad? I was too young too tell and thought it was nice.
Claude-real-video － any LLM can watch a video
31–40 of 86 posts
Re: Claude-real-video － any LLM can watch a video
#32Pretty terribly expensive way to watch a video with Claude. Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens. Check out this library: https://vlm-run.github.io/mm/ You can swap models and try out different encoding methods for videos ( https://vlm-run.github.io/mm/encoders/#video )
Do you mean that Gemini is most token-efficent at watching videos? Is that the case for e.g. just giving it a video in the browser? I admit, I dont give LLMs videos as I just assume it'll burn too many tokens.
Yes, Gemini is very token efficient at video. It also has "lower resolution" options which can make it even cheaper if. With Gemini 3.1 flash lite an hour of video works out to $0.24 at the API rates.
Re: Claude-real-video － any LLM can watch a video
#33Pretty terribly expensive way to watch a video with Claude. Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens. Check out this library: https://vlm-run.github.io/mm/ You can swap models and try out different encoding methods for videos ( https://vlm-run.github.io/mm/encoders/#video )
Exactly this. Gemini is best at this. Just give it video link - YouTube works best - and it will analyse the video.
Re: Claude-real-video － any LLM can watch a video
#34So this work basically by dividing into frames..
Re: Claude-real-video － any LLM can watch a video
#35Pretty terribly expensive way to watch a video with Claude. Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens. Check out this library: https://vlm-run.github.io/mm/ You can swap models and try out different encoding methods for videos ( https://vlm-run.github.io/mm/encoders/#video )
Seems cool from the docs page, I was about to give it a shot but https://github.com/vlm-run/mm goes 404 …
Re: Claude-real-video － any LLM can watch a video
#36I was creating a scene by scene remake of a cutscene from an old DOS game. The sprite sheet had several sprites which were cycled (e.g. a horse with it's head down and up). The engine would cycle through these regularly to create some "liveliness" in the background. It was tedious and I didn't want to figure out which sprites belonged at which pixel location.
I recorded a video of the relevant part of the cutscene using dosbox and then split it into numbered frames using ffmpeg. Then I gave that + the spritesheet to Claude Code and asked it to figure it out and tell me which ones are at what position. I should probably have deduped it but in any case, it churned through the whole thing and got one or two out of 15 or 16 sprites right. The rest, it just dropped into random places. YMMV
Re: Claude-real-video － any LLM can watch a video
#37[flagged]
Re: Claude-real-video － any LLM can watch a video
#38Pretty terribly expensive way to watch a video with Claude. Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens. Check out this library: https://vlm-run.github.io/mm/ You can swap models and try out different encoding methods for videos ( https://vlm-run.github.io/mm/encoders/#video )
Exactly this. Gemini is best at this. Just give it video link - YouTube works best - and it will analyse the video.
Really, does this work now? What about NotebookLM? I was using it a lot until i realised it was only analysing the transcripts and not the video because i was mostly using it for technical ones with important charts.
Re: Claude-real-video － any LLM can watch a video
#39Based on my tests, a frame rate of 2fps is generally sufficient to resolve video content very well.