Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v4-flash-vision-exp

api-docs.deepseek.com

Post reply on HN
Print view View original

101–110 of 169 posts

Re: DeepSeek-v4-flash-vision-exp

#101

Post by jubilanti »

dghlsakjg wrote: post #93
leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

Reading any analog clock at any time level (edit: and a non-noisy vector rendered image at that) is absolutely table stakes for an allegedly frontier flagship vision model. As much as 1:1 OCR. If the model can't do that, there's something wrong. Doesn't matter if it's memorized some random thing you think is esoteric but is in all the training data and benchmarks.

The whole point of LLM/FMs vs good old fashioned ML is generalization to unknown domains, not just unknown tasks. The hunt for "gotchas" is the hunt for "not in your training data".

Re: DeepSeek-v4-flash-vision-exp

#102

Post by v9v »

johndough wrote: post #34
v9v wrote: post #11

Interesting. Wasn't Deepseek's founder saying that they had explicitly decided not to focus on multimodal models at all and were going text-only because they believed it was enough to achieve AGI?

It was explicitly said that they are pursuing multimodal support. A quote from the meeting transcript: https://github.com/demo-zexuan/liang-wenfeng-investor-meetin... Nevertheless, as a component, we will undoubtedly implement multimodal support — and we are already doing so. We plan to develop relevant models, ensuring that versions like V4 and subsequent iterations will natively support multimodal functionality. Ea…

Thanks, I seem to have grossly misremembered what I read.

Re: DeepSeek-v4-flash-vision-exp

#103

Post by mejutoco »

dghlsakjg wrote: post #93
leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

It is like asking a politician how much a coffee costs, to show how disconnected they are from common people. Super intelligence not being able to read a simple analog clock does the same.

> Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot

This is about a _vision_ model.

Re: DeepSeek-v4-flash-vision-exp

#104

Post by dghlsakjg »

CooCooCaCha wrote: post #98
dghlsakjg wrote: post #93

Earlier quoted context omitted.

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

Is being asked to read a clock really a gotcha?

If I had to hire an engineer and there was one that could one shot the wang algorithm, but couldn’t read an analog clock, I would have no problem hiring them.

Also worth noting that both models got it wrong. Qwen made a mistake that humans very good at reading clocks would make. Deepseek made a mistake that a human who had just learned to read clocks would make.

Re: DeepSeek-v4-flash-vision-exp

#105

Post by altruios »

dghlsakjg wrote: post #93
leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

Knowing where it fails is just as important as knowing where is excels.

Re: DeepSeek-v4-flash-vision-exp

#106

Post by atomicnumber3 »

dghlsakjg wrote: post #93
leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

It's because the messaging for what the point of these things is supposed to be is all over the place. Ask 10 different people and you'll get 10 different answers:

- A superintelligence that will usher in an age of human enlightenment

- A superintelligence that will usher in an age of human enslavement

- A really cool way to rake in trillion of rich VC/investor money by promising you're building a superintelligence that will usher in an age of human en[slave/lighten]ment

- A transformer model for predicting output tokens given a series of input tokens, informed primarily by reddit, stack overflow, and 6000 years of classical literature.

- A replacement for white collar labor. Start now or join the permanent underclass.

- A convenient fuzzy-find tool also capable of some probably-correct code generation.

- The ultimate customizable text RPG experience (you can pick if G stand for game or...)

And so on.

So, some people see a new model and check for how close humanity is to enslavement. Some people check to see if it got better at fixing broken unit tests.

Re: DeepSeek-v4-flash-vision-exp

#107

Post by knollimar »

andai wrote: post #78
knollimar wrote: post #72

Earlier quoted context omitted.

Downsizing a higher res image to lower res means the zoom will be blurry.

They process the original image file with Python on the local device. (And I've seen the web chats do this with their "computer use" features too.) The really wild one is even blind models will do this and they'll try to run stats on the pixels to figure out what it looks like... the even wilder thing is that it kind of works!

If the API accepts only 800 by 800, the aegument youre making is "fix it in the harness".

I don't think the n by n subgrid fixes this the way most harnesses do, as it'll fail to count things if you have more overlap and fail relatiomships if you have less

Re: DeepSeek-v4-flash-vision-exp

#108

Post by mkatx »

leumon wrote: post #74

It fails the simple clock test for me which Qwen3.8 27B got (nearly) right. given an image of a clock https://files.catbox.moe/kgwa5e.png I asked it "what time does the clock show?" (both on reasoning: high) DS answered: The clock shows *5:10* (and 45 seconds). Here is the breakdown: * *Hour hand (red, shortest):* Pointing at the *5*. * *Minute hand (green, longest):* Pointing at the *2*, which represents 10 minutes.…

I would say, try without thinking on. I find reasoning on any rag type request seems to increase hallucinations, probably due to the thinking tokens taking attention away from the, in this case, vision tokens.

I'd recommend non-thinking for any non-prompt input, and leave the thinking where it has to actually reason.

Re: DeepSeek-v4-flash-vision-exp

#109

Post by RobertLong »

The benchmark results look promising when compared to Opus 4.8, but for agentic usecases it's lacking images as tool call result types. Giving the model a tool to take screenshots and verify its work is my main usecase for vision models, but this is more oriented towards "build a website that looks like this" type prompts. Hopefully we'll see this by release.

Re: DeepSeek-v4-flash-vision-exp

#110

Post by KerrAvon »

atomicnumber3 wrote: post #106
dghlsakjg wrote: post #93

Earlier quoted context omitted.

I’ll keep that in mind next time I need to tell what time it is by asking an llm to read an analog clock. Snark aside, I’m not sure that these gotcha tests are any more useful than asking politicians gotcha questions. Sure, the model can’t tell me what time it is, but it can code the Wang algorithm for noisy audio matching in one shot. Maybe this is just me being an optimist, but this is my hiring philosophy and I gu…

It's because the messaging for what the point of these things is supposed to be is all over the place. Ask 10 different people and you'll get 10 different answers: - A superintelligence that will usher in an age of human enlightenment - A superintelligence that will usher in an age of human enslavement - A really cool way to rake in trillion of rich VC/investor money by promising you're building a superintelligence t…

What's amazing is that all of these are true at once. If you allow for some significant slack in what "superintelligence" means.
Post reply on HN