Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini Diffusion

simonwillison.net

Post reply on HN
Print view View original

191–200 of 252 posts

Re: Gemini Diffusion

#191

Post by lukan »

skydhash wrote: post #157
stef25 wrote: post #145

Earlier quoted context omitted.

I understand what negative space is in art. Can you explain how this applies to writing software ?

A quick example is a basic 2d game. If you’re not using an engine (just a graphic library) and you have some animations, experience will tell you to not write most of the code with numbers only. More often than not, you will write a quick vector module. Just how you will use local origin for transformations. But more often than not, the naive code is the result of not doing the above and just writing the feature. It…

I have written 2D games, but maybe the metapher is just lost on me or I simply disagree to its usefulness here.

Negative space in art achieves a certain effect. Like in the linked sibling comment, the empty space is part of the sculpture.

So the empty space has purpose and meaning.

But if I didn't choose a certain libary .. the empty place of that libary serves no function. It does change my code and might make my dev life easier or harder, but has no meaning in itself for the result.

Re: Gemini Diffusion

#192

Post by simonw »

Jackson__ wrote: post #83

What is this blog spam doing here? This is has literally no new information compared to the official release page. It would make a lot more sense to change the link to https://deepmind.google/models/gemini-diffusion/ to discuss the topic.

This content is from my link blog, and the page you linked to is the primary link in my own post.

I think I added value over the official landing page:

1. I included a video showing how fast it runs. They don't have a video on that page.

2. I compared it to Cerebras (which is even faster). They obviously aren't going to compare themselves with a competitor in their own marketing material!

3. These are updates since first publishing, but my post now highlights a couple of Hacker News comments that help explain how this actually works.

I wrote a bunch about how I try to add value when link blogging here: https://simonwillison.net/2024/Dec/22/link-blog/#trying-to-a...

Re: Gemini Diffusion

#193

Post by Workaccount2 »

nvtop wrote: post #97
theptip wrote: post #64

Earlier quoted context omitted.

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…

Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…

Can you have a hybrid model that can do autoregression and diffusion? It doesn't seem like there is something that would fundamentally prevent this. A model with diffusion CoT for rapid "thought" generation, and then autoregression for the answer on the output.

Re: Gemini Diffusion

#194

Post by Workaccount2 »

seydor wrote: post #93

Just the idea of generating text by removing noise is so profound. Maybe each step is a level of hierarchy. Linguists must be astonished at the things happening these past years. I have to read more about it

You might find this article interesting:

https://www.quantamagazine.org/when-chatgpt-broke-an-entire-...

How transformers pretty much blindsided the whole field of NLP.

Re: Gemini Diffusion

#195

Post by paulsmith »

nvtop wrote: post #152
yahoozoo wrote: post #140

Earlier quoted context omitted.

Great explanation. I think I have seen where text diffusion models can “edit” as it’s running inference. Or in other words, a “final” token isn’t necessarily “final” and could change but at some later iteration the model decides it truly is. How does that work?

Correct, diffusion LMs can edit their intermediate predictions, so "final" tokens aren't necessarily final. This is an exciting property because it allows models to correct errors in what's generated so far -- something that GPT-like models can't. This editing is based on the Transformer's encoder property to predict token probabilities for __every__ token in a sequence, not just for [MASK]s. So when you input a sent…

> say model returns the probability of the token "cat" on the second position as p_2("cat") = 0.3, while p_2("dog") = 0.6. We may want to replace "cat" with dog, and use it in the subsequent iterations.

Might one tradeoff of speed/quality be a tree "search" for better outcomes by branching on logit choices? If a diffusion model is so much faster overall than AR, then I might not mind that I hunt or backtrack for the best probabilities overall.

Re: Gemini Diffusion

#196

Post by airstrike »

lucasacosta_ wrote: post #172
airstrike wrote: post #21

Earlier quoted context omitted.

I thought of that as I wrote my comment, but I think the infrastructure and glue to make that possible in a consistent, fast and scalable way is still a few years out.

Definitely. For now the "frontier-level" papers (working with repository-level coding maintenance) need to necessarily depend on previously (and statically) generated Code Knowledge Graphs or Snippet-Retrieval systems, which makes the scalable and fast aspects complicated, as any change in the code would represent a change in the graph, hence requiring a rebuild. But given the context limit, you need to rely on Graph…

Thanks for these links. I really appreciate it.

Re: Gemini Diffusion

#197

Post by bilsbie »

> Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling.

Has anyone tried making text the way we do image diffusion? What happens?

Re: Gemini Diffusion

#198

Post by moneywoes »

vFunct wrote: post #53
renjimen wrote: post #51

The speed this can build makes me think software is soon to become a lot more fluid than our traditional iterative approach. Apps could ship minimal and build whatever else they need to at the user’s behest.

The challenge for LLMs over the next year is to get them to operate on large data sets/code bases with millions/billions of tokens through some kind of distributed hierarchical framework, with each LLM operating on a local set of 20k or whatever subset of tokens.

any reading?

Re: Gemini Diffusion

#199

Post by albertzeyer »

> Google's first LLM to use diffusion in place of transformers.

But this is a wrong statement? Google never made this statement? You can have a Transformer diffusion models. Actually Transformers are very standard for all of the discrete diffusion language models, so I would expect Gemini Diffusion also uses Transformers.

Edit Ah sorry, I missed, this was already addressed, also linked in the post: https://news.ycombinator.com/item?id=44057939 Maybe my remaining post is still useful to some.

The difference is, it's an encoder-only Transformer, and not a decoder-only Transformer. I.e. it gets fed in a full sequence (but noisy/corrupted), and it predicts the full correct sequence. And then you can iterate on that. All frames in the sequence can be calculated in parallel, and if you need only a few iterations, this is faster than the sequential decoding in decoder-only models (although speculative decoding also gets you some speedup for similar reasons). Those discrete diffusion models / encoder-only Transformers are usually trained with BERT-like masking, but that's actually an active field of research. It's really a pity that they don't provide any details here (on training and modeling).

I wonder how this relates to Gemini. Does it use the same modeling? Was the model checkpoint even imported from Gemini, and then further finetuned for discrete diffusion? Or knowledge distillation? Or is it just branding?

Re: Gemini Diffusion

#200

Post by yencabulator »

internet_points wrote: post #118
wenc wrote: post #74

Earlier quoted context omitted.

Can you give me an example? Spell check only checks if a word is in dictionary. It doesn’t check grammar or context.

Finnish would like a word. Take a random noun like kauppa "shop". It has at least 6000 forms: https://flammie.github.io/omorfi/genkau3.html and that's excluding compounds (written as one word in Finnish) like "bookshop" or "shop-manager" etc. etc. And then you have loan words and slang, derivations into other words classes; all of this is impossible to compactly represent in a full-form word list. Now consider the ma…

Finnish is very different from most other languages, and does not have the user base to be well represented in training data, but that webpage is ridiculous and does not reflect the actual language. No one in the history of Finnish has ever spoken most of those forms. Grammar describes language, it does not define it!
Post reply on HN