Is this more evidence that the days of insatiable demand for GPU data centers and electricity are behind us?
Gemini Diffusion
151–160 of 252 posts
Re: Gemini Diffusion
#152Earlier quoted context omitted.
Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…
Great explanation. I think I have seen where text diffusion models can “edit” as it’s running inference. Or in other words, a “final” token isn’t necessarily “final” and could change but at some later iteration the model decides it truly is. How does that work?
Correct, diffusion LMs can edit their intermediate predictions, so "final" tokens aren't necessarily final. This is an exciting property because it allows models to correct errors in what's generated so far -- something that GPT-like models can't.
This editing is based on the Transformer's encoder property to predict token probabilities for __every__ token in a sequence, not just for [MASK]s. So when you input a sentence of three tokens `[MASK] cat barks`, Transformer will generate a probability distribution over the vocabulary for each of the three tokens, for free.
Now you can come up with many ways of how to decide whether you want to edit token or keep it as is. In the simplest case, take a new token if its probability higher than the original by some margin. In our example, say model returns the probability of the token "cat" on the second position as p_2("cat") = 0.3, while p_2("dog") = 0.6. We may want to replace "cat" with dog, and use it in the subsequent iterations.
Actual heuristics are slightly more complicated, but the base idea is this.
P.S. In order to teach LM not to just copy input unmasked tokens but to try to find a better replacement, your training objective should include replacing some % of input tokens with some other random token. Now you have part of the input masked, and part of the input corrupted, so the model can't blindly assume that all input tokens are here to stay.
Re: Gemini Diffusion
#153That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…
A LLM could easily use its own knowledge to create a list of things to check inside the code base and generate a fact sheet and use best practices and similar knowledge to extend on it.
Just because one query might not be able to do so doesn't mean there are no ways around it
Re: Gemini Diffusion
#154Earlier quoted context omitted.
I would suggest using an agentic system like Cline, so that the LLM can wander through the codebase by itself and do research and build a "mental model" and then set up an implementation plan. The you iterate in that and hand it off for implementation. This flow works significantly better than what you're describing.
> LLM can wander through the codebase by itself and do research and build a "mental model" It can't really do that due to context length limitations.
1k LOC is perfectly fine, I did not experience issues with Claude with most (not all) projects around ~1k LOC.
Re: Gemini Diffusion
#155Is this more evidence that the days of insatiable demand for GPU data centers and electricity are behind us?
If the diffusion models are an improvement over autoregression models, then the answer is No, due to Jevon's paradox. That is, as these models get cheaper and better, they provide more utility, driving more demand. Even as your datacenters become more productive, the demand for their compute power increases at an even faster pace.
The thing that will limit demand for compute is when the world decides it has sufficient capacity of the form of "intelligence" these models provide. I don't think anyone has any idea what that world will even look like.
Re: Gemini Diffusion
#156Are there any open diffusion ones already? Or too early for that?
Re: Gemini Diffusion
#157Earlier quoted context omitted.
"...what is not in a codebase, and there is meaningful signal in that negative space." Man, I'm writing software for money for decades now, but this fundamental truth never occured to me, at least not consciously and with such clarity. So, thank you!
I understand what negative space is in art. Can you explain how this applies to writing software ?
A quick example is a basic 2d game. If you’re not using an engine (just a graphic library) and you have some animations, experience will tell you to not write most of the code with numbers only. More often than not, you will write a quick vector module. Just how you will use local origin for transformations.
But more often than not, the naive code is the result of not doing the above and just writing the feature. It technically does the job, but it’s verbose and difficult to maintain.
So just like in drawing, you need to think holistically about the program. Every line of code should support an abstraction. And that will dictate which code to write and which to not write.
That’s why you often see the concept of patterns in software. The code is not important. The patterns are. The whole structure more so. Code is just what shape these.
Re: Gemini Diffusion
#158That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…
The trick to this is you've got to talk to them and share this information in the same way. I can give an example. These days my main workflow is as follows: if I have some big feature/refactor/whatever I'm going to work on I'll just start talking to o3 about it essentially as if it was a coworker and (somewhat painstakingly) paste in relevant source files it needs for context. We'll have a high-level discussion abou…
I find myself using a similar workflow with Aider. I'll use chat mode to plan, adjust context, enable edits, and let it go. I'll give it a broad objective and tell it to ask me questions until the requirements are clear, then a planning summary. Flipping the script is especially helpful when I'm unsure what I actually want.
Re: Gemini Diffusion
#159I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear attention), and created such franken-stein just by post-training. The big wow moment about that is that it sort of implies that most of the useful knowledge is in the FFN, and attention it…
Attention is just completely arbitrary way to split the network so the learning can be parallelized.
What contributed more towards success in my opinion are "shortcut connections" through layers which enable more influence on early layers during learning.
Re: Gemini Diffusion
#160Earlier quoted context omitted.
"...what is not in a codebase, and there is meaningful signal in that negative space." Man, I'm writing software for money for decades now, but this fundamental truth never occured to me, at least not consciously and with such clarity. So, thank you!
Then document it. Whenever you choose one algorithm/library/tech stack but not another, write your consideration in the documents.
[deleted]