Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can see and hear without any training

github.com

Post reply on HN
Print view View original

21–30 of 74 posts

Re: LLMs can see and hear without any training

#21

Post by EncomLab »

JoBrad wrote: post #15
EncomLab wrote: post #10

My photoresistor nightlight can "see" that it is dark and it "knows" to turn on the light - not only does it not have training, it does not have any code! And if you think that is amazing, my bi-metallic strip thermostat "feels" the temperature and then modifies the environment because it "knows" if it's hot to turn on the A/C, and if it's cold to turn on the heat - no training or code! All of this AI stuff is just u…

The nightlight and thermostat's response to stimulus is nowhere near analyzing a picture of a clock tower and responding with "Image of a city's tallest, historic landmark with a sepia filter." To me, recognizing the umbrella in the spoon is one of the most impressive items they list.

It's not the technology that is bad - it's the extreme anthropomorphizing language that's used to describe it.

Re: LLMs can see and hear without any training

#22

Post by lukeinator42 »

oneseven wrote: post #19
viraptor wrote: post #11

That looks like a classic Actor/Critic setup, yet it's not mentioned even once in the paper. Am I missing some large difference here?

Yes, apparently they've developed new names: Generator and Scorer. This feels a bit like "Tai's Model" https://news.ycombinator.com/item?id=17863514

Haha "Tai's Model" is absolutely hilarious, that gave me a good chuckle. I checked and it currently is cited 568 times.

Re: LLMs can see and hear without any training

#24

Post by throwaway4aday »

scribu wrote: post #5

This seems to be a system to generate better prompts to be fed into a base multimodal model. Interesting, but title is definitely clickbait.

They only did that for image generation. The more interesting part is that an LLM can approach or find the correct caption for an image, video or audio during test time with no training using only the score as a guide. It's essentially working blind almost like the game Marco Polo where the scorer is saying "warmer" or "colder" while the LLM is finding its way towards the goal. This is an example of emergent capabilities since there are no examples of this in the training data.

Re: LLMs can see and hear without any training

#25

Post by dragonwriter »

suddenlybananas wrote: post #17
underdeserver wrote: post #2

Paper: https://arxiv.org/pdf/2501.18096

I don't understand how the title relates to the content of this article at all. They're even using CLIP which definitely has been trained.

You don't have to train the LLM soecifically for the tasks and even the auxiliary tools aren't trained on the tasks they are used as scorers for (because they aren't doing the task,just evaluating how well the LlM is), so there is no task-specific training.

Re: LLMs can see and hear without any training

#27

Post by dragonwriter »

jagged-chisel wrote: post #3

Computers can receive input without any programming. Not sure what’s interesting here.

This isn't receiving input, its generating output competitive with models with task-specific training.

I’m guessing the iterative approach burns a lot of tokens though, though that may not matter too much with 8B Llama as the LLM.

Re: LLMs can see and hear without any training

#28

Post by horacemorace »

EncomLab wrote: post #21
JoBrad wrote: post #15

Earlier quoted context omitted.

The nightlight and thermostat's response to stimulus is nowhere near analyzing a picture of a clock tower and responding with "Image of a city's tallest, historic landmark with a sepia filter." To me, recognizing the umbrella in the spoon is one of the most impressive items they list.

It's not the technology that is bad - it's the extreme anthropomorphizing language that's used to describe it.

It might be bad if its behavior wasn’t so anthropomorphic.

Re: LLMs can see and hear without any training

#30

Post by vessenes »

I’ve read the paper and the skeptical comments here, to wit: it’s just an actor/critic pipeline by another name.

I’ll bite and say this is actually interesting — and the paper title is misleading.

What they’ve done here is hooked up a text-only LLM to multimodal critics, given it (mostly) an image diffusion generation task, and asked it to improve its prompting of the multimodal generation by getting a set of scores back.

This definitely works, based on their outputs. Which is to say, LLMs can, zero shot, with outside tool feedback, iteratively improve their prompting using only that tooling feedback.

Why is this interesting? Well, this did not work in the GPT-3 era; it seems to do so now. I see this as an interesting line to be added in the ‘model capabilities’ box as our models get larger and more sophisticated — the LLMs can perform some sort of internally guided search against a black box generator and use a black box scorer to improve at inference time.

That’s pretty cool. It’s also generalizable, and I think is worth keeping in mind on the stack of possible approaches for, say agentic coding, that you can use a critic to not just ‘improve’ generated output, but most likely do some guided search through output space.

Post reply on HN