Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mercury: Commercial-scale diffusion language model

inceptionlabs.ai

Post reply on HN
Print view View original

171–180 of 189 posts

Re: Mercury: Commercial-scale diffusion language model

#171

Post by pama »

GistNoesis wrote: post #144
niemandhier wrote: post #118

Earlier quoted context omitted.

This is tremendously interesting! Could you point me to some literature? Especially regarding mathematical proofs of your intuition? I’d like to recalibrate my priors to align better with current research results.

From the mathematical point of view the literature is about the distinction between a "filtering" distribution and a "smoothing" distribution. The smoothing distribution is strictly more powerful. In theory intuitively the smoothing distribution has access to all the information that the filtering distribution has and some additional information therefore has a minimum lower than the filtering distribution. In practi…

Although what you say about smoothing vs filtering is true in principle, for conditional generation of the eventual joint distribution starting from the same condition and using an autoregresive vs diffusive LLM, it is the smoothing distribution that has less power. In other words, during inference starting from J tokens and writing token number K is of course better with diffusion if you also have some given tokens after token K and up to the maximal token N. However, if your input is fixed (tokens up to J) and you have to predict those additional tokens (from J+1 to N), you are solving a harder problem and have a lower joint probability at the end of the inference for the full generated sequence from J+1 up to N.

Re: Mercury: Commercial-scale diffusion language model

#172

Post by janalsncm »

geysersam wrote: post #163
otabdeveloper4 wrote: post #158

Earlier quoted context omitted.

> it's still useful for debugging How so? What the LLM says is whatever is more likely given the context. It has no relation to the underlying reality whatsoever.

Not sure what you mean by "relation to the underlying reality". The explanation is likely to be correlated with the underlying reason for the answer. For example, here is a simple query: > I put my bike in the bike stand, I removed my bike from the bike stand and biked to the beach, then I biked home, where is my bike. Answer only with the answer and no explanation > Chatgpt: At home > Why did you answer that? > I an…

You need to find an example of the LLM making a mistake. In your example, ChatGPT answered correctly. There are many examples online of LLMs answering basic questions incorrectly, and then the person asking the LLM why it did so. The LLM response is usually nonsense.

Then there is the question of what you would do with its response. It’s not like code where you can go in and update the logic. There are billions of floating point numbers. If you actually wanted to update the weights you’ll quickly find yourself fine-tuning the monstrosity. Orders of magnitude more work than updating an “if” statement.

Re: Mercury: Commercial-scale diffusion language model

#173

Post by [deleted] »

Nevermark wrote: post #75
kazinator wrote: post #68

Earlier quoted context omitted.

That's totally cribbed from some discussion hat occurred in its training.

As apposed to humans who all derive the physics of heat transfer independently when given a question like this? Not picking on you - this brings up something we could all get better at: There should be a "First Rule of Critiquing Models": Define a baseline system to compare performance against. When in doubt, or for general critiques of models, compare to real world random human performance. Without a real practical…

[deleted]

Re: Mercury: Commercial-scale diffusion language model

#174

Post by suddenlybananas »

mhh__ wrote: post #160
suddenlybananas wrote: post #138

Earlier quoted context omitted.

You have not read every accessible piece of text in existence.

There is more to life than just text e.g. this is part of lecun argument against LLMs

Lecun's argument is based off a bad interpretation of how data is processed by the optic nerve, we don't receive that much raw data.

What we do have, is billions of years of evolution that has given a lot of innate knowledge which means we are radically more capable than LLMs despite having little data.

Re: Mercury: Commercial-scale diffusion language model

#175

Post by otabdeveloper4 »

geysersam wrote: post #163
otabdeveloper4 wrote: post #158

Earlier quoted context omitted.

> it's still useful for debugging How so? What the LLM says is whatever is more likely given the context. It has no relation to the underlying reality whatsoever.

Not sure what you mean by "relation to the underlying reality". The explanation is likely to be correlated with the underlying reason for the answer. For example, here is a simple query: > I put my bike in the bike stand, I removed my bike from the bike stand and biked to the beach, then I biked home, where is my bike. Answer only with the answer and no explanation > Chatgpt: At home > Why did you answer that? > I an…

> The explanation is likely to be correlated with the underlying reason for the answer.

No it isn't. You misunderstand how LLMs work. They're giant Mad Libs machines: given these surrounding words, fill in this blank with whatever statistically is most likely. LLMs don't model reality in any way.

Re: Mercury: Commercial-scale diffusion language model

#176

Post by otabdeveloper4 »

IshKebab wrote: post #170
otabdeveloper4 wrote: post #159

Earlier quoted context omitted.

Incorrect. LLM's have no self-reflection capability. That's a key prerequisite for "thinking". ("I think, therefore I am.") They are simple calculators that answer with whatever tokens are most likely given the context. If you want reasonable or correct answers (rather than the most likely) then you're out of luck.

It is not a key prerequisite for "thinking". It's "I think therefore I am" not "I am self-aware therefore I think". In the 90s if your cursor turned into an hourglass and someone said "it's thinking" would you have pedantically said "NO! It is merely calculating!" Maybe you would... but normal people with normal English would not.

Self-reflection is not the same thing as self-awareness.

Computers have self-reflection to a degree - e.g., they react to malfunctions and can evaluate their own behavior. LLMs can't do this, in this respect they are even less of a thinking machine than plain old dumb software.

Re: Mercury: Commercial-scale diffusion language model

#177

Post by geysersam »

otabdeveloper4 wrote: post #175
geysersam wrote: post #163

Earlier quoted context omitted.

Not sure what you mean by "relation to the underlying reality". The explanation is likely to be correlated with the underlying reason for the answer. For example, here is a simple query: > I put my bike in the bike stand, I removed my bike from the bike stand and biked to the beach, then I biked home, where is my bike. Answer only with the answer and no explanation > Chatgpt: At home > Why did you answer that? > I an…

> The explanation is likely to be correlated with the underlying reason for the answer. No it isn't. You misunderstand how LLMs work. They're giant Mad Libs machines: given these surrounding words, fill in this blank with whatever statistically is most likely. LLMs don't model reality in any way.

Did you read the example above? Do you disagree that the LLM provided a correct explanation for the reason it answered as it did?

> They're giant Mad Libs machines: given these surrounding words, fill in this blank with whatever statistically is most likely. LLMs don't model reality in any way.

Not sure why you think this is incompatible with the statement you disagreed with.

Re: Mercury: Commercial-scale diffusion language model

#178

Post by geysersam »

janalsncm wrote: post #172
geysersam wrote: post #163

Earlier quoted context omitted.

Not sure what you mean by "relation to the underlying reality". The explanation is likely to be correlated with the underlying reason for the answer. For example, here is a simple query: > I put my bike in the bike stand, I removed my bike from the bike stand and biked to the beach, then I biked home, where is my bike. Answer only with the answer and no explanation > Chatgpt: At home > Why did you answer that? > I an…

You need to find an example of the LLM making a mistake. In your example, ChatGPT answered correctly. There are many examples online of LLMs answering basic questions incorrectly, and then the person asking the LLM why it did so. The LLM response is usually nonsense. Then there is the question of what you would do with its response. It’s not like code where you can go in and update the logic. There are billions of fl…

I don't think llms always can give correct explanations for their answers. That's a misunderstanding.

> Then there is the question of what you would do with its response. I

Sure but that's a separate question. I'd say the first course of action would be to edit the prompt. If you have to resort to fine tuning I'd say the approach has failed and the tool was insufficient for the task.

Re: Mercury: Commercial-scale diffusion language model

#179

Post by geysersam »

kadushka wrote: post #161
geysersam wrote: post #157

Earlier quoted context omitted.

The premise here is that the model runs fast and cheap. With the current state of the technology running a vending machine using an LLM is of course absurd. The point is that accuracy is not the only dimension that brings qualitative change to the kind of applications that LLMs are useful for.

Running a vending machine using an LLM is absurd not because we can't run LLMs fast or cheap enough - it's because LLMs are not reliable, and we don't know yet how to make them more reliable. Our best LLM - o3 - doubled the previous model (o1) hallucination rate. OpenAI says it hallucinated a wrong answer 33% of the time in benchmarks. Do you want a vending machine that screws up 33% of the time? Today, the accuracy…

Do you seriously think a typical contemporary LLM would screw up 33% of vending machine orders?

I don't know what benchmark you're looking at but I'm sure the questions in it were more complicated than the logic inside a vending machine.

Why don't you just try it out? It's easy to simulate, just tell the bot about the task and explain to it what actions to perform in different situations, then provide some user input and see if it works or not.

Re: Mercury: Commercial-scale diffusion language model

#180

Post by adammarples »

selcuka wrote: post #72
adammarples wrote: post #27

Earlier quoted context omitted.

If you let it sit for 2 minutes your time is up and you don't have time to add the cold milk

By this logic you can't let it sit for 2 mins after you add the cold milk either, so both options are invalid. In math/science questions some things are assumed to be (practically impossibly) instant.

Ah but if you do that you get time up after 1.58s and you still win
Post reply on HN