Earlier quoted context omitted.
AI field desperately needs smarter models - not faster models.
LLM's can't think, so "smarter" is not possible.
Technically correct and completely besides the point.
Mercury: Commercial-scale diffusion language model
111–120 of 189 posts
Re: Mercury: Commercial-scale diffusion language model
#112I'd hope that with diffusion, it would be able to go back and forth between parts of the output to adjust issues with part of the output which it had previously generated. This would not be possible with a purely sequential model. However, > Prompt: Write a sentence with ten words which has exactly as many r’s in the first five words as in the last five > > Response: Rapidly running, rats rush, racing, racing.
Why not possible with autoregressive model?
o4 mini
https://chatgpt.com/share/681315c2-aa90-800d-b02d-c3ba653281...
Re: Mercury: Commercial-scale diffusion language model
#113Earlier quoted context omitted.
I think natural language to code is the right abstraction. Easy enough barrier to entry but still debuggable. Debugging why an LLM randomly gives you Mountain Dew instead of Sprite if you have a southern accent sounds like a nightmare.
I'm not sure it would be that hard to debug. Make sure you can reproduce the llm state (by storing the random seed for the session, or something like that) and then ask it "why did you just now give that customer mountain dew when they ordered sprite?"
Why not just store the state in the code and debug as usual, perhaps with LLM assistance? At least that’s tractable.
Re: Mercury: Commercial-scale diffusion language model
#114Earlier quoted context omitted.
Claude 3.7 gets it exactly right: To determine which option cools coffee the most, I'll analyze the heat transfer physics involved. The key insight is that the rate of heat loss depends on the temperature difference between the coffee and the surrounding air. When the coffee is hotter, it loses heat faster. Option 1 (add milk first, then wait): - Adding cold milk immediately lowers the coffee temperature right away -…
That's totally cribbed from some discussion hat occurred in its training.
If it was just ‘in the training data’ they’d all get it right.
But they don’t.
Re: Mercury: Commercial-scale diffusion language model
#115Interesting approach. However, I never thought of auto regression being _the_ current issue with language modeling. If anything it seems the community was generally surprised just how far next "token" prediction took us. Remember back when we did char generating RNNs and were impressed they could make almost coherent sentences? Diffusion is an alternative but I am having a hard time understanding the whole "built in…
Consider the entropy of the distribution of token X in these examples: "Four X" and "Four X and seven years ago". In the first case X could be pretty much anything, but in the second case we both know the only likely completion. So it seems like there would be a huge advantage in not having to run autogressively. But in practice it's less significant then you might imagine because the AR model can internally model th…
> in the second case we both know the only likely completion.
You two may, but I don't. 'Decades'? 'Months'? 'Wives'? 'Jobs'? 'Conservative PMs'?
Re: Mercury: Commercial-scale diffusion language model
#116Ok. My go to puzzle is this: You have 2 minutes to cool down a cup of coffee to the lowest temp you can You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add the cold milk. Which one cools the coffee to the lowest temperature and why? And Mercury gets this right - while as of right now ChatGPT 4o get it wrong. So that’s pretty impressive.
For me, ChatGPT (the free version, GPT-4o mini I believe?) gets it right, choosing option 2 because the coffee will cool faster due to the larger temperature difference. Unless there's a gotcha somewhere in your prompt that I'm missing, like what if the temperature of the room is hotter than the coffee, or so cold that the coffee becomes colder than the milk, or something? I would be suprised if any models get it wro…
This is what I got from full-fat 4o. Maybe thinking less helps!
ChatGPT:
Option 1 — Add the cold milk immediately — will result in a lower final temperature after 2 minutes.
Why: • Heat loss depends on the temperature difference between the coffee and the environment (usually room temperature). • If you add the milk early, the overall temperature of the coffee-milk mixture is reduced immediately. This lowers the average temperature over the 2 minutes, so less heat is lost to the air. • If you wait 2 minutes to add the milk, the hotter coffee loses more heat to the environment during those 2 minutes, but when you finally add the milk, it doesn’t cool it as much because the coffee’s already cooler and the temp difference between the milk and the coffee is smaller.
Summary: • Adding milk early = cooler overall drink after 2 minutes. • Adding milk late = higher overall temp after 2 minutes, because more heat escapes during the time the coffee is hotter.
Want me to show a simple simulation or visualisation of this?
Re: Mercury: Commercial-scale diffusion language model
#117Ok. My go to puzzle is this: You have 2 minutes to cool down a cup of coffee to the lowest temp you can You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add the cold milk. Which one cools the coffee to the lowest temperature and why? And Mercury gets this right - while as of right now ChatGPT 4o get it wrong. So that’s pretty impressive.
> My go to puzzle is this: > Mercury gets this right - while as of right now ChatGPT 4o get it wrong. This is so common a puzzle it's discussed all over the internet. It's in the data used to build the models. What's so impressive about a machine that can spit out something easily found with a quick web search?
Just that what I thought would be better models don’t do it right.
I was expecting this model to be no-where near chatGPT
Although someone above is saying 4o-mini got it right so maybe it’s meaningless. Or maybe thinking less helps…
Re: Mercury: Commercial-scale diffusion language model
#118Earlier quoted context omitted.
So my personal belief is that diffusion models will enable higher degrees of accuracy. This is because unlike an auto-regressive model it can adjust a whole block of tokens when it encounters some kind of disjunction. Think of the old example where an auto regressive model would output: "There are 2 possibilities.." before it really enumerated them. Often the model has trouble overcoming the bias and will hallucinate…
Unfortunately the intuition and the math proofs so far suggest that autoregressive training is learning the joint distribution of probabilistic streams of tokens much better than diffision models do or will ever do. My intuitive take is that the conditional probability distribtion of decoder-only autoregressive models is at just the right level of complexity for probabilistic models to learn accurately enough. Intuit…
This is tremendously interesting!
Could you point me to some literature? Especially regarding mathematical proofs of your intuition?
I’d like to recalibrate my priors to align better with current research results.
Re: Mercury: Commercial-scale diffusion language model
#119Ok. My go to puzzle is this: You have 2 minutes to cool down a cup of coffee to the lowest temp you can You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add the cold milk. Which one cools the coffee to the lowest temperature and why? And Mercury gets this right - while as of right now ChatGPT 4o get it wrong. So that’s pretty impressive.
Hmm a good nerd-snipe puzzle. I was never very good at physics, so hopefully someone can check my work... assuming upon mixing coffee is at Tc and milk at Tm, and simplifying to assume equivalent mass & specific temp we have (Tf - Tc) = -(Tf - Tm) => Tf = (Tc+Tm)/2 which is intuitive (upon mixing we get the average temperature). On the assumption that the cold milk is always at a fixed temperature until it's mixed in…
There is a more intuitive way: Adding the milk later has a chance to get the cup below room temperature, whereas adding it early never will.
Re: Mercury: Commercial-scale diffusion language model
#120Look into groq.com guys. some good models at similar speed to inception labs