How do you handle things like scrolling quickly in a video?
[dead]
Claude-real-video － any LLM can watch a video
61–70 of 86 posts
Re: Claude-real-video － any LLM can watch a video
#62Interesting, but how expensive does it get?
[flagged]
Re: Claude-real-video － any LLM can watch a video
#63Hi HN! I built this because I was frustrated that no LLM actually "sees" a video — Claude won't accept video files, ChatGPT reads the transcript only, and Gemini samples at a fixed 1fps (missing fast cuts, over-sampling static slides). claude-real-video takes a URL or local file and: 1. Extracts frames at every scene change (not fixed intervals) + a density floor 2. Deduplicates with a sliding-window pixel-diff algor…
Very cool I have something that does this as well along these lines. I’ll dig into yours over the next few days and contribute where and if I can too, awesome to see!
[flagged]
Re: Claude-real-video － any LLM can watch a video
#64Nice @OP i put together something similar as well. Incidentally I found for motion design specifically llm is not able to infer specific animations as well as it just being described very plainly and accurately what is happening and the timing. One thing which sort of worked decently was actually take the frames and put them into a grid and have the agent look at the image of all of the frames together. It did surpri…
[flagged]
Re: Claude-real-video － any LLM can watch a video
#65Based on my tests, a frame rate of 2fps is generally sufficient to resolve video content very well.
[deleted]
Re: Claude-real-video － any LLM can watch a video
#66Interesting, but how expensive does it get?
[flagged]
Re: Claude-real-video － any LLM can watch a video
#67How do you handle things like scrolling quickly in a video?
[flagged]
Re: Claude-real-video － any LLM can watch a video
#68Are models any good at descerning motion from multiple frames? For instance if I gave models multiple animations of a bouncing ball as individual frames. Would they be able to tell which bounce was the more realistic motion. (Is this a potential new benchmark? maybe also variations of stair dismount)
[dead]
Re: Claude-real-video － any LLM can watch a video
#69Earlier quoted context omitted.
Exactly this. Gemini is best at this. Just give it video link - YouTube works best - and it will analyse the video.
Really, does this work now? What about NotebookLM? I was using it a lot until i realised it was only analysing the transcripts and not the video because i was mostly using it for technical ones with important charts.
NotebookLM was transcript-only when I last checked. Gemini proper does ingest video natively (samples ~1fps server-side). This tool is for everything that can't — Claude, ChatGPT web, local models — it turns the video into frames + transcript on your machine so any of them can read it.
Re: Claude-real-video － any LLM can watch a video
#70The comments in this post strongly validate the need for reliable video processing and understanding with VLMs.
While you can use Gemini or other local VLMs, the real challenge is token efficiency, accuracy, and coverage. For example, how do you make a VLM “watch” a 2-hour or 4GB video without losing context or meaning?
Video transcript alone can be sufficient for basic workflows needing no visual context. But when deep contextual understanding is required, e.g., self-driving, security analysis, warehouse tracking, etc., you’ll need more advanced methods like keyframe sampling, clipping, chunking, and shots+transcript.
You can explore the different encoding strategies we designed for efficient video processing and understanding here: https://vlm-run.github.io/mm/encoders/#video.
FYI, the repo is now public, and contributions are welcome.