Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini Diffusion

simonwillison.net

Post reply on HN
Print view View original

161–170 of 252 posts

Re: Gemini Diffusion

#161

Post by dimitri-vs »

westoncb wrote: post #123
airstrike wrote: post #3

That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…

The trick to this is you've got to talk to them and share this information in the same way. I can give an example. These days my main workflow is as follows: if I have some big feature/refactor/whatever I'm going to work on I'll just start talking to o3 about it essentially as if it was a coworker and (somewhat painstakingly) paste in relevant source files it needs for context. We'll have a high-level discussion abou…

Interesting approach, I'm definitely going to steal your wording for "generate an implementation plan that...".

I do something similar but entirely within Cursor:

1. create a `docs/feature_name_spec.md`, use voice-to-text to brain dump what I am trying to do 2. open up a the AI chat panel in "Ask" mode while referencing that spec file, ask (paste) a boilerplate snippet like: "1) Ask clarifying questions about intent, domain, restrictions, ambiguity or missing details 2) Briefly identify any missing documents, data, or background information that would help you complete the task thoroughly" 3. move that list of questions into the spec doc and answer them there, attach the files it asked for and just rerun the above request (optionally, switching to a different model, like gemini-2.5-pro -> o3, for different perspective) 4. ask it to make an execution plan and at that point i have a fully spec'd out feature and documented business logic, I either use the Edit mode on each step or Agent mode

That's for more complex features touching many files or refactors, but I essentially do a simplified version of that within the same chat by editing my original chat prompt until I'm confident I explained myself well

Re: Gemini Diffusion

#162

Post by scotty79 »

airstrike wrote: post #3

That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…

> they will continue to be handicapped by that lack of institutional knowledge, so to speak

Until we give them access to all Jira tickets instead of just one so they know what's missing.

Re: Gemini Diffusion

#163

Post by skydhash »

nvtop wrote: post #152
yahoozoo wrote: post #140

Earlier quoted context omitted.

Great explanation. I think I have seen where text diffusion models can “edit” as it’s running inference. Or in other words, a “final” token isn’t necessarily “final” and could change but at some later iteration the model decides it truly is. How does that work?

Correct, diffusion LMs can edit their intermediate predictions, so "final" tokens aren't necessarily final. This is an exciting property because it allows models to correct errors in what's generated so far -- something that GPT-like models can't. This editing is based on the Transformer's encoder property to predict token probabilities for __every__ token in a sequence, not just for [MASK]s. So when you input a sent…

But what about the dependency graph between symbols in the program. Because all those symbols have high constraints around them which is the program design.

The issue comes in image diffusion as well. When you ask it for a portrait and some details are wrong. That’s because the face has constraints (which you learn about as an artist). Patterns and probability won’t help you.

Re: Gemini Diffusion

#164

Post by macleginn »

kmacdough wrote: post #142
macleginn wrote: post #139

Earlier quoted context omitted.

A spelling error, using one dictionary definition, is "an error in the conventionally accepted form of spelling a word" --- mistaking one word for another does not fall under this definition. It is true that we now expect spell checkers to do grammatical checking as well, but a pure spell checker can indeed rely on a wordlist for English (this wouldn't work in languages with more developed morphology and/or frequent…

Ok, but this is a technicality. Spell-checkers have slowly evolved into grammar checkers and what people really want is error correction. Whether people call it a spell checker a minor language issue (and the kind of things humans do all the time). When teaching for your dictionary, ask: "is it obvious what they mean if I'm not being pedantic?"

We expect different outputs in these two cases, though. A wrong word choice is usually accompanied by a hint that another word may have been intended, while a wrong spelling can be unambiguously marked as a mistake. These two behaviours can be turned on and off independently, and they need two different labels.

Re: Gemini Diffusion

#165

Post by jacob019 »

Incipient wrote: post #65
8n4vidtmkvmk wrote: post #46

Earlier quoted context omitted.

That's not been my experience so far. LLMs are good at mimicking existing good, it doesn't usually bring in new things when not asked. Sometimes I have to go out of my way to point to other bits of code in the project to copy from because it hasn't ingested enough of the codebase. That said, a negative prompt like we have in stable diffusion would still be very cool.

I'm in the camp of 'no good for existing'. I try to get ~1000 line files refactored to use different libraries, design paradigms, etc and it usually outputs garbage - pulling db logic into the UI, grabbing unrelated api/function calls, to entirely just corrupting the output. I'm sure there is a way to correctly use this tool, so I'm feeling like I'm "just holding it wrong".

I've refactored some files over 6000 loc. It was necessary to do it iteratively with smaller patches. "Do not attempt to modify more than one function per iteration" It would just gloss over stuff. I would tell it repeatedly: I noticed you missed something, can you find it? I kept doing that until it couldn't find anything. Then I had to manually review and ask for more edits. Also lots of style guidelines and scope limit instructions. In the end it worked fine and saved me hours of really boring work.

Re: Gemini Diffusion

#166

Post by oliwary »

nvtop wrote: post #97
theptip wrote: post #64

Earlier quoted context omitted.

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…

Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…

Fascinating, and great explanation.

What about insert and delete operations however? Isn't there a risk of there being too few tokens to properly finish the code in-between the "final" tokens?

Re: Gemini Diffusion

#167

Post by macleginn »

ealexhudson wrote: post #148
kmacdough wrote: post #142

Earlier quoted context omitted.

Ok, but this is a technicality. Spell-checkers have slowly evolved into grammar checkers and what people really want is error correction. Whether people call it a spell checker a minor language issue (and the kind of things humans do all the time). When teaching for your dictionary, ask: "is it obvious what they mean if I'm not being pedantic?"

Agreed. "Dessert" vs "desert" - mistaking these two is often not a grammatical error (they're both nouns), but is a spelling error (they have quite different meanings, and the person who wrote the word simply spelled it wrongly).

I agree, but this is definitely the kind of spelling error (along with complementary/complimentary, discrete/discreet, etc.) that we normally don't expect our spellcheckers to catch.

Re: Gemini Diffusion

#168

Post by skydhash »

GistNoesis wrote: post #99
djmips wrote: post #79

Earlier quoted context omitted.

Is this a Shaggy Dog Story?

If only... When you first encounter diffusion models, you usually see a well formed picture emerge from noise. And then you realize, there is no reason it shouldn't work for anything where you can add noise to. Which means everything. From picture, to audio to text, to anything encoded in data. An infinite world of images and human creations in 10GB of weights. A meaningful universe lost in speckle of dusts. I rememb…

You forgot about constraints, especially cascading ones. Where one detail can shape the whole thing (think shadows and lights location and direction)

Re: Gemini Diffusion

#169

Post by ctxc »

nvtop wrote: post #97
theptip wrote: post #64

Earlier quoted context omitted.

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…

Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…

Thank you for the explanation!

Re: Gemini Diffusion

#170

Post by danielbln »

exe34 wrote: post #150
otabdeveloper4 wrote: post #138

Earlier quoted context omitted.

> LLM can wander through the codebase by itself and do research and build a "mental model" It can't really do that due to context length limitations.

It doesn't need the entire codebase, it just needs the call map, the function signatures, etc. It doesn't have to include everything in a call - but having access to all of it means it can pick what seems relevant.

Yes, that's exactly right. The LLM gets a rough overview over the project (as you said, including function signatures and such) and will then decide what to open and use to complete/implement the objective.
Post reply on HN