Live data from Hacker News

Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

the-decoder.com

41–50 of 86 posts

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#41

Not so relevant to the thread but ive been uploading screenshots from citrix guis and asking qwen3-vl for the appropriate next action eg Mouseclick, and while it knows what to click it struggles to accurately return which pixel coordinates to click. Anyone know a way to get accurate pixel coordinates returned?

Could you combine it with a classic OCR segmentation process, so that along with the image you also provide box coordinates of each string?

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#42

Not so relevant to the thread but ive been uploading screenshots from citrix guis and asking qwen3-vl for the appropriate next action eg Mouseclick, and while it knows what to click it struggles to accurately return which pixel coordinates to click. Anyone know a way to get accurate pixel coordinates returned?

It’s been about a year since I looked into this sort of thing, but molmo will give you x,y coordinates. I hacked together a project about it. I also think Microsoft’s omniparser is good at finding coordinates too.

https://huggingface.co/allenai/Molmo-7B-D-0924

https://github.com/logankeenan/george

https://github.com/microsoft/OmniParser

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#43

Not so relevant to the thread but ive been uploading screenshots from citrix guis and asking qwen3-vl for the appropriate next action eg Mouseclick, and while it knows what to click it struggles to accurately return which pixel coordinates to click. Anyone know a way to get accurate pixel coordinates returned?

you want get the exact coordinated by running a key point network to pinpoint which coordinates does the next click point is you can. here I show a example simple prompt which returns the keypoint location of the next botton to click and visually localize the point with a keypoint in the image

https://chat.vlm.run/c/e12f0153-7121-4599-9eb9-cd8c60bbbd69

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#44
post #27

Hope this on day will be used for auto-tagging all video assets with time codes. The dream of being able to search for running horse and find a clip containing a running horse at 4m42s in one of thousands of clips.

you can do that with Morphik already :)

We use an embedding model that processes videos and allows you to perform RAG on them.

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#45

Not so relevant to the thread but ive been uploading screenshots from citrix guis and asking qwen3-vl for the appropriate next action eg Mouseclick, and while it knows what to click it struggles to accurately return which pixel coordinates to click. Anyone know a way to get accurate pixel coordinates returned?

It's very not accurate, but sometimes instructing to return pyautogui code works.

  prompt: I attach a screenshot (1920x1080). Write code to click the submit button using pyautogui.
  attachment: 
  reply:
    import pyautogui
    pyautogui.click(100, 200)

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#46
post #37

Earlier quoted context omitted.

In surveillance and police states like The Netherlands it has been used since forever: https://www.theguardian.com/cities/2018/mar/01/smart-cities-... Now people will say again that this project has been abandoned, which just isn't true (2024): https://www.dutchnews.nl/2024/06/smart-street-surveillance-o...

I was watching a crime solving show from the UK. A huge percentage of the crimes are solved using camera footage. Also, they use geofencing, looking at which phones went in and out of the crime location at the time of the crime.

I would be surprised if this hasn't existed for a few decades already.

Back in 2009 I was working at a place where O2 was a client, and they gave us an API that could identify the cell tower (inc. lat/lng) any of their customers were connected to. The network needs to track this data internally to function, so the API is basically the equivalent of their DNS.

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#48
post #27

Hope this on day will be used for auto-tagging all video assets with time codes. The dream of being able to search for running horse and find a clip containing a running horse at 4m42s in one of thousands of clips.

It’s not difficult to hack this together with CLIP. I did this with about a tenth of my movie collection last week with a GTX 1080 - though it lacks temporal understanding so you have to do the scene analysis yourself

Would you be willing to share more details of what you did?

Re: Qwen3-VL can scan two-hour videos and pinpoint nearly every detail

#50

For anyone using Qwen3-VL: where are you running it? I had tons of reliability problems with Qwen3-VL inference providers on OpenRouter — based on uptime graphs I wasn’t alone. But when it worked, Qwen3-VL was pack-leading good at AI Vision stuff.

I run it on ollama
Post reply on HN