Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini Diffusion

simonwillison.net

Post reply on HN
Print view View original

201–210 of 252 posts

Re: Gemini Diffusion

#201

Post by collingreen »

lukan wrote: post #191
skydhash wrote: post #157

Earlier quoted context omitted.

A quick example is a basic 2d game. If you’re not using an engine (just a graphic library) and you have some animations, experience will tell you to not write most of the code with numbers only. More often than not, you will write a quick vector module. Just how you will use local origin for transformations. But more often than not, the naive code is the result of not doing the above and just writing the feature. It…

I have written 2D games, but maybe the metapher is just lost on me or I simply disagree to its usefulness here. Negative space in art achieves a certain effect. Like in the linked sibling comment, the empty space is part of the sculpture. So the empty space has purpose and meaning. But if I didn't choose a certain libary .. the empty place of that libary serves no function. It does change my code and might make my de…

Let me take a crack at it.

I think the negative space metaphor in software can be in the shape of the abstractions and hitting the sweet spot of making the right things easy/railroaded while not over engineering it.

In visual art, negative space is part of the layout and the visual journey. It helps define the relationships between things as much as those things themselves and, used judiciously, is one of the differences between elegance and clutter.

I think "not choosing a library" is important info but isn't the same thing as negative space and is instead more like restrictions, framing, or limitation. You can do a lot with what isn't shown but in this area I think good art and good software diverge in goals - to me good art makes me think or feel or speculate while good software instead makes me understand with as little of those other things as possible.

The caveat here might be not choosing things for very good but not obvious reasons, which should be loudly documented. Things like licensing or other external influences or specific hardware requirements maybe. For example I once banned the creation of a graphQL api in a product that could have benefited from it because we still needed to support the existing api for third parties forever so the suggestion to replace the api was actually secretly the suggestion to maintain two APIs in lockstep.

Re: Gemini Diffusion

#202

Post by machiaweliczny »

I feel like diffusion would be much more useful for code it it could only mark tokens as "valid" if they were passing code checks. So it could be thought as adding more of "semantic chunks" instead of just words. Not sure how to validate it as some additions always will result in invalid code. You could argue that running tests, linters is the same but I think one could make it that generations are validated much more often with diffusion models.

Example: You remove some function, you also remove all uses of it. You can't use not existing variable etc. This could be trained on well commited git repos or stalking/stealing the work of developers via editor

Re: Gemini Diffusion

#203

Post by skydhash »

lukan wrote: post #191
skydhash wrote: post #157

Earlier quoted context omitted.

A quick example is a basic 2d game. If you’re not using an engine (just a graphic library) and you have some animations, experience will tell you to not write most of the code with numbers only. More often than not, you will write a quick vector module. Just how you will use local origin for transformations. But more often than not, the naive code is the result of not doing the above and just writing the feature. It…

I have written 2D games, but maybe the metapher is just lost on me or I simply disagree to its usefulness here. Negative space in art achieves a certain effect. Like in the linked sibling comment, the empty space is part of the sculpture. So the empty space has purpose and meaning. But if I didn't choose a certain libary .. the empty place of that libary serves no function. It does change my code and might make my de…

I’m talking more about architecting code instead of naively writing them. The same point can be made about libraries but the considerations are more subjective.

Most naive approaches to writing software looks like assembly. But instead of opcodes, you have libraries functions. But we move away from assembly and assembly like programming because it’s essentially one shot. Any modification to the program is difficult and/or tedious. So instead of having that one blob of instructions, we introduce gaps so that it becomes more flexible. We have functions, objects, modules… but the actual links between them still needs to be shaped.

A library can have some influence on the shape, but it is minor if you favor the solution over the means. But sometimes you see people really going hard to fill the gaps of the shape, and that’s when you start to shout KISS and YAGNI. Sometimes they want to alter the shape and you bring out SOLID and other principles…

Re: Gemini Diffusion

#204

Post by skydhash »

collingreen wrote: post #201
lukan wrote: post #191

Earlier quoted context omitted.

I have written 2D games, but maybe the metapher is just lost on me or I simply disagree to its usefulness here. Negative space in art achieves a certain effect. Like in the linked sibling comment, the empty space is part of the sculpture. So the empty space has purpose and meaning. But if I didn't choose a certain libary .. the empty place of that libary serves no function. It does change my code and might make my de…

Let me take a crack at it. I think the negative space metaphor in software can be in the shape of the abstractions and hitting the sweet spot of making the right things easy/railroaded while not over engineering it. In visual art, negative space is part of the layout and the visual journey. It helps define the relationships between things as much as those things themselves and, used judiciously, is one of the differe…

Yes the code is not actually important as two different teams will solve the same problem in different manners. Just like a great painting and a bad one can use the same base materials. What’s important is the purpose and the constraints of any solution. Any decision you take propagates down the timeline and outward in the project. And they preclude other decisions from being taken.

So whatever you do will live a mark. But there are some spaces that should not be filled in. While it may look nice in the moment or taken in isolation. When looking at the whole, it makes it a mess.

Re: Gemini Diffusion

#205

Post by CooCooCaCha »

m101 wrote: post #135

It makes one wonder what intelligence really is. The more I think about it the more I feel that speed is a fundamental unit of intelligence, with the other being some simple computation unit. As in, intelligence = speed * simple computation. If you look around us it is the ability to iterate that drives innovation (and thereby evidence of "intelligence"). LLMs in industry are more useful, and used, the faster they ar…

I really like Gwern’s definition of intelligence which is “search over Turing machines”.

In other words, searching for the right program given some goal.

Re: Gemini Diffusion

#206

Post by lostmsu »

moralestapia wrote: post #134
janalsncm wrote: post #94

Earlier quoted context omitted.

Let’s suppose we have 10k possible tokens in the vocabulary. Then text would be an image 10k pixels tall and N pixels wide, where N is the length of the text. For each column, exactly 1 pixel is white (corresponding to the word which is there) and the rest are black. Then the diffusion process is the same. Repeatedly denoising.

No, that intuition is incorrect. Denoising models work because a lot of regions turn out to be smooth, you cannot do that "in a discrete way" if that makes sense.

They may be smooth in embedding space

Re: Gemini Diffusion

#208

Post by skydhash »

angusturner wrote: post #184
skydhash wrote: post #163

Earlier quoted context omitted.

But what about the dependency graph between symbols in the program. Because all those symbols have high constraints around them which is the program design. The issue comes in image diffusion as well. When you ask it for a portrait and some details are wrong. That’s because the face has constraints (which you learn about as an artist). Patterns and probability won’t help you.

You assume that for small steps (I.e taking some noisy code and slightly denoising) you can make an independence assumption. (All tokens conditionally independent, given the current state). Once you chain many steps you get a very flexible distribution that can model all the interdependencies. A stats person could probably provide more nuance, although two interesting connection I’ve seen: There is some sense in whic…

There’s a reason we have formal verification as the highest guarantee for software. To ensure that we have a complete assurance of what the program can and can not do, the semantic of each of its components needs to be known. Recursively.

A minor change in one token can change the meaning of the whole software. Programming is just trying to enforce semantics on instructions (how well is that done is software engineering’s realm)

An algorithm like merge sort is just semantic constraints. Which is why most books go with their own notations as code does not really matter.

At most, LLMs and diffusion can be regarded as fancy searches. But, what you actually want is semantics and that’s why you can design lots of stuff on paper. But we do it with the code editor because feedbacks are nice and libraries’ documentations (if they exist) lie about their semantics. And we read code because there’s nothing more complete about semantics than that.

Re: Gemini Diffusion

#209

Post by shawntan »

nvtop wrote: post #97
theptip wrote: post #64

Earlier quoted context omitted.

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…

Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…

I'm curious how the speed is achieved is this is the technique used. Generally I expected this "masked language model" technique to be far slower since the full vocab projection needs to be computed every iteration.

I always thought the eventual technique would be some form of diffusion in continuous space, then decoding into the discrete tokens.

Also I'm guessing this is a "best guess" of how Gemini Diffusion is done?

Re: Gemini Diffusion

#210

Post by vFunct »

moneywoes wrote: post #198
vFunct wrote: post #53

Earlier quoted context omitted.

The challenge for LLMs over the next year is to get them to operate on large data sets/code bases with millions/billions of tokens through some kind of distributed hierarchical framework, with each LLM operating on a local set of 20k or whatever subset of tokens.

any reading?

I’m just a user, trying out the models first hand on a large project, learning as I go.
Post reply on HN