Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude-real-video － any LLM can watch a video

github.com

Post reply on HN
Print view View original

21–30 of 86 posts

Re: Claude-real-video － any LLM can watch a video

#21

Post by yieldcrv »

idiotsecant wrote: post #16
ElijahLynn wrote: post #5

I was just thinking about this exact use case yesterday: And it's for me measuring different charged speeds at different starting battery capacities and different temperatures and I was like well. What if I just had a video camera pointing at the voltage going in and out and then I could see the battery percentage increase and I can have a temperature gun pointed at the phone as well. And I couldn't know what tempera…

It's kind of wild how much we are abandoning basic problem solving skills in favor of just pointing an enormous stack of GPUs at it

[dead]

Re: Claude-real-video － any LLM can watch a video

#23

Post by Lerc »

Are models any good at descerning motion from multiple frames?

For instance if I gave models multiple animations of a bouncing ball as individual frames. Would they be able to tell which bounce was the more realistic motion.

(Is this a potential new benchmark? maybe also variations of stair dismount)

Re: Claude-real-video － any LLM can watch a video

#25

Post by smallerize »

siriusastrebe wrote: post #17
idiotsecant wrote: post #16

Earlier quoted context omitted.

It's kind of wild how much we are abandoning basic problem solving skills in favor of just pointing an enormous stack of GPUs at it

Identifying objects in pictures was considered an insurmountable task only a few years ago, like in the xckd comic https://xkcd.com/1425/

In the general case, I guess. But watching gauges and dials like battery capacity only take a little work with a deterministic computer vision library.

Re: Claude-real-video － any LLM can watch a video

#26

Post by danbrooks »

Lerc wrote: post #23

Are models any good at descerning motion from multiple frames? For instance if I gave models multiple animations of a bouncing ball as individual frames. Would they be able to tell which bounce was the more realistic motion. (Is this a potential new benchmark? maybe also variations of stair dismount)

I’d imagine they could. I’d try Gemini 3.5 flash with high fps.

Re: Claude-real-video － any LLM can watch a video

#28

Post by mh- »

fzysingularity wrote: post #18

Pretty terribly expensive way to watch a video with Claude. Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens. Check out this library: https://vlm-run.github.io/mm/ You can swap models and try out different encoding methods for videos ( https://vlm-run.github.io/mm/encoders/#video )

Assuming that's your project, the GitHub link from the PyPi page is a 404.

Re: Claude-real-video － any LLM can watch a video

#30

Post by Tenoke »

fzysingularity wrote: post #18

Pretty terribly expensive way to watch a video with Claude. Use Gemini or some local VLM to do this way more efficiently. We spent quite a bit of time on video understanding, and Claude will just burn tokens. Check out this library: https://vlm-run.github.io/mm/ You can swap models and try out different encoding methods for videos ( https://vlm-run.github.io/mm/encoders/#video )

Do you mean that Gemini is most token-efficent at watching videos? Is that the case for e.g. just giving it a video in the browser? I admit, I dont give LLMs videos as I just assume it'll burn too many tokens.
Post reply on HN