Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini Diffusion

simonwillison.net

Post reply on HN
Print view View original

131–140 of 252 posts

Re: Gemini Diffusion

#131

Post by moralestapia »

nvtop wrote: post #97
theptip wrote: post #64

Earlier quoted context omitted.

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…

Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…

Whoa man, thanks.

This is a great explanation.

Re: Gemini Diffusion

#132

Post by NitpickLawyer »

matsemann wrote: post #110
wenc wrote: post #74

Earlier quoted context omitted.

Can you give me an example? Spell check only checks if a word is in dictionary. It doesn’t check grammar or context.

Butt wouldn't you liked if a spell cheque could of fixed these command?

Hah! Apple caught "of" and suggested "consider have instead", but left the rest untouched. Great qed for spell checkers.

Chatgpt fixed it though: "But wouldn't you like it if a spell check could have fixed these commands?"

Re: Gemini Diffusion

#133

Post by lysecret »

So what's its going to be in the end Diffusion or Auto-Regression? After OpenAi (probably) released an Auto-Regressive model for their image generator I thought things might sway the other way.

Re: Gemini Diffusion

#134

Post by moralestapia »

janalsncm wrote: post #94
pertymcpert wrote: post #67

Earlier quoted context omitted.

I have the exact same questions as you. I can barely understand how diffusion works for images, for sequential data like text it makes no sense to me.

Let’s suppose we have 10k possible tokens in the vocabulary. Then text would be an image 10k pixels tall and N pixels wide, where N is the length of the text. For each column, exactly 1 pixel is white (corresponding to the word which is there) and the rest are black. Then the diffusion process is the same. Repeatedly denoising.

No, that intuition is incorrect.

Denoising models work because a lot of regions turn out to be smooth, you cannot do that "in a discrete way" if that makes sense.

Re: Gemini Diffusion

#135

Post by m101 »

It makes one wonder what intelligence really is. The more I think about it the more I feel that speed is a fundamental unit of intelligence, with the other being some simple computation unit. As in, intelligence = speed * simple computation.

If you look around us it is the ability to iterate that drives innovation (and thereby evidence of "intelligence"). LLMs in industry are more useful, and used, the faster they are.

Re: Gemini Diffusion

#136

Post by padolsey »

seydor wrote: post #93

Just the idea of generating text by removing noise is so profound. Maybe each step is a level of hierarchy. Linguists must be astonished at the things happening these past years. I have to read more about it

I feel the same. On reflection, it's how I think I experience thoughts emerging in my head. Language gets derived from initially noisy embeddings. It's quite beautiful that we've ended up closer to a de-noising architecture than auto-complete on steroids.

Re: Gemini Diffusion

#137

Post by manmal »

Cthulhu_ wrote: post #101
manmal wrote: post #54

Earlier quoted context omitted.

They could read the whole git history and have all issue tracker tickets in the context, and maybe even recordings from meetings. It remains to be seen though if such large context will yield usable results.

I find most meetings I'm in nowadays are mostly noise; there's no clear "signal" that "this is the outcome", which I think is what an AI should be able to filter out. Of course, it'd be even better if people communicated more clearly and succinctly.

Maybe time to find an employer with a better culture? I rarely have meetings that I would be comfortable skipping.

Re: Gemini Diffusion

#138

Post by otabdeveloper4 »

danielbln wrote: post #114
Incipient wrote: post #111

Earlier quoted context omitted.

Was more of a general comment - I'm surprised there is significant variation between any of the frontier models? However, vscode with various python frameworks/libraries; dash, fastapi, pandas, etc. Typically passing the 4-5 relevant files in as context. Developing via docker so I haven't found a nice way for agents to work.

I would suggest using an agentic system like Cline, so that the LLM can wander through the codebase by itself and do research and build a "mental model" and then set up an implementation plan. The you iterate in that and hand it off for implementation. This flow works significantly better than what you're describing.

> LLM can wander through the codebase by itself and do research and build a "mental model"

It can't really do that due to context length limitations.

Re: Gemini Diffusion

#139

Post by macleginn »

thinkingemote wrote: post #103
wenc wrote: post #74

Earlier quoted context omitted.

Can you give me an example? Spell check only checks if a word is in dictionary. It doesn’t check grammar or context.

"Bob went to Venice to pick up the doge." Where doge is both the name of a title (like duke) but it is misspelt "dog". The use of "Venice" where doge's are could increase a the likelihood of a smarter spell check keeping doge and not correcting to dog. Looking at a wider context might see that Bob is talking about a pupper. A simpler example would be "spell cheque"

A spelling error, using one dictionary definition, is "an error in the conventionally accepted form of spelling a word" --- mistaking one word for another does not fall under this definition. It is true that we now expect spell checkers to do grammatical checking as well, but a pure spell checker can indeed rely on a wordlist for English (this wouldn't work in languages with more developed morphology and/or frequent compounding).

Re: Gemini Diffusion

#140

Post by yahoozoo »

nvtop wrote: post #97
theptip wrote: post #64

Earlier quoted context omitted.

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…

Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…

Great explanation. I think I have seen where text diffusion models can “edit” as it’s running inference. Or in other words, a “final” token isn’t necessarily “final” and could change but at some later iteration the model decides it truly is. How does that work?
Post reply on HN