Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v4-flash-vision-exp

api-docs.deepseek.com

Post reply on HN
Print view View original

91–100 of 169 posts

Re: DeepSeek-v4-flash-vision-exp

#91

Post by leumon »

emosenkis wrote: post #85
leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

This is not a normal looking clock - most clocks have either one color for all hands (second hand is thinnest and maybe also longest) or one color for hour/minute and one for second. I know that the hand lengths and thicknesses on this image are correct but for some reason I, a totally human person who grew up when analog clocks were still common, see this and think the hand on the 5 is the minute hand. How does the…

then deepseek answers: "The clock shows 8:25. The short hour hand is pointing to the 8, and the long minute hand is pointing to the 5, which represents 25 minutes."

and qwen still answers: "The clock shows *8:10* (with the second hand on the 5, i.e., 25 seconds). - *Hour hand* points to the 8 - *Minute hand* points to the 2 (= 10 minutes) - *Second hand* points to the 5 (= 25 seconds) So the time is *8:10:25*, or simply *8:10*."

Re: DeepSeek-v4-flash-vision-exp

#92

Post by zdragnar »

5kyn3t wrote: post #37

For what do you guys use vision in those models? surveillance is the obvious use case... but are there some "nicer" ways to use it?

https://stencil.so/blog/snapcompact - some agents (notably oh my pi, i forget which others) come with snapcompact as a primary means of compaction. Take the entire context, stick it in a small font in a PNG, and vision capable models can summarize and pull out the most useful information in many fewer vision tokens than the original context used.

I've not used it myself, but it's there.

Re: DeepSeek-v4-flash-vision-exp

#93

Post by dghlsakjg »

leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock.

Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I guess maybe now my llm philosophy: I’m not interested in seeing how dumb I can make you look, I’m more interested in how smart you can be.

Re: DeepSeek-v4-flash-vision-exp

#94

Post by dghlsakjg »

leumon wrote: post #91
emosenkis wrote: post #85

Earlier quoted context omitted.

This is not a normal looking clock - most clocks have either one color for all hands (second hand is thinnest and maybe also longest) or one color for hour/minute and one for second. I know that the hand lengths and thicknesses on this image are correct but for some reason I, a totally human person who grew up when analog clocks were still common, see this and think the hand on the 5 is the minute hand. How does the…

then deepseek answers: "The clock shows 8:25. The short hour hand is pointing to the 8, and the long minute hand is pointing to the 5, which represents 25 minutes." and qwen still answers: "The clock shows *8:10* (with the second hand on the 5, i.e., 25 seconds). - *Hour hand* points to the 8 - *Minute hand* points to the 2 (= 10 minutes) - *Second hand* points to the 5 (= 25 seconds) So the time is *8:10:25*, or sim…

Qwen still got the wrong answer, though.

Are we more forgiving because it’s the same type of mistake a human would make?

Re: DeepSeek-v4-flash-vision-exp

#95

Post by segmondy »

leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

most likely a preview. they often release the preview via API, get more training data, post train some more then release the weight. i would expect to see it perform better in a few weeks or a month.

Re: DeepSeek-v4-flash-vision-exp

#96

Post by vasco »

dghlsakjg wrote: post #93
leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

It's still useful to find things it can't do if anything so we can tell when it starts being able to do them.

Re: DeepSeek-v4-flash-vision-exp

#97

Post by johnnyApplePRNG »

leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I was wasting hours yesterday trying to get DeepSeek V4 Flash (with Qwen 3.8 27b as the vision agent, actually) to read sheet music to pass a Terminal Bench 3 benchmark and none of it was working... nothing... I changed models to gemma 31b, I tried OCR models... nothing could get it...

And then I realized, wait a second... you're testing the harness not only against a difficult benchmarking problem, but it's one you're literally never going to use the coding harness for either, lol. I don't write programs that read or interact with sheet music and I never will.

tl;dr Being frustrated that a "state of the art" vision model doesn't have perfect vision is a fools errand.

It can read and extract information from screenshots and PDFs just fine (my setup). No need to worry about edge cases.

Re: DeepSeek-v4-flash-vision-exp

#98

Post by CooCooCaCha »

dghlsakjg wrote: post #93
leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

Is being asked to read a clock really a gotcha?

Re: DeepSeek-v4-flash-vision-exp

#99

Post by johnnyApplePRNG »

mavamaarten wrote: post #71
LorenDB wrote: post #2

I've heard that DeepSeek v4 Flash 0731 has frequently assumed that it has vision capabilities and then resorts to inventing text-based image analysis tools when it finds that it actually can't see. In that case, this is a great upgrade for the model. Anecdotally, I had to tell 0731 to refrain from viewing screenshots since it kept breaking its sessions by trying to read images.

Yeah I've seen it a lot. It goes through the effort, unasked, of pulling screenshots off a connected device and then it's like... Oh shit yeah I can't see.

It's doing it's best to accomplish whatever task you've thrown at it.

It's expecting you to have done at least something besides select DS4 on Ollama, essentially.

Re: DeepSeek-v4-flash-vision-exp

#100

Post by arjie »

CooCooCaCha wrote: post #98
dghlsakjg wrote: post #93

Earlier quoted context omitted.

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

Is being asked to read a clock really a gotcha?

Not if you are aiming at a general intelligence but it’s worth considering that this is a tool that may not be able to count the number of strawberries in the letter R but can still center a div.
Post reply on HN