I’m currently punishing Fable by making it watch the entire series of 7th Heaven.
It's going to make itself unavailable again. Actually... that's probably a litmus test for sentience.
Claude-real-video － any LLM can watch a video
41–50 of 86 posts
Re: Claude-real-video － any LLM can watch a video
#42Earlier quoted context omitted.
Identifying objects in pictures was considered an insurmountable task only a few years ago, like in the xckd comic https://xkcd.com/1425/
In the general case, I guess. But watching gauges and dials like battery capacity only take a little work with a deterministic computer vision library.
Yeah - the correct way to use an LLM in this scenario is to ask it to put write such a model.
Re: Claude-real-video － any LLM can watch a video
#43Earlier quoted context omitted.
Exactly this. Gemini is best at this. Just give it video link - YouTube works best - and it will analyse the video.
Really, does this work now? What about NotebookLM? I was using it a lot until i realised it was only analysing the transcripts and not the video because i was mostly using it for technical ones with important charts.
It can tell you what’s on the screen at given point in time. My pipeline is mostly around simple questions like “does this video contain cars?” Not sure if it can spot charts on screen.
Re: Claude-real-video － any LLM can watch a video
#44Re: Claude-real-video － any LLM can watch a video
#45Gemini does read videos.
Re: Claude-real-video － any LLM can watch a video
#46So I did this yesterday for a video analysis sample with ChatGPT and it took the video, pulled out frames, did difference tests across the frames to look for significant frames to focus on, did image recognition on each frame, and interpolated motion and action between.
So I’m not sure why this says ChatGPT doesn’t “see” video and reads transcripts. Obviously if the video is already labeled that’s the shortcut. But it did an impressive job describing a video I have no inclination it would have in its training data. One could argue it wasn’t “native” and had an agent orchestrator to rely on external tools to accomplish the goal… but it worked.
Re: Claude-real-video － any LLM can watch a video
#47better off using a cloud solution.
Re: Claude-real-video － any LLM can watch a video
#48So I did this yesterday for a video analysis sample with ChatGPT and it took the video, pulled out frames, did difference tests across the frames to look for significant frames to focus on, did image recognition on each frame, and interpolated motion and action between. So I’m not sure why this says ChatGPT doesn’t “see” video and reads transcripts. Obviously if the video is already labeled that’s the shortcut. But i…
Had the same experience with Claude, just somehow the entire thing felt (token) expensive.
Re: Claude-real-video － any LLM can watch a video
#49Pretty terribly expensive way to watch a video with Claude. Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens. Check out this library: https://vlm-run.github.io/mm/ You can swap models and try out different encoding methods for videos ( https://vlm-run.github.io/mm/encoders/#video )
Seems cool from the docs page, I was about to give it a shot but https://github.com/vlm-run/mm goes 404 …
It’s unclear if that’s intentional since it’s listed also under open source on the main company site: https://www.vlm.run/open-source/mm
Re: Claude-real-video － any LLM can watch a video
#50[deleted]