Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini Diffusion

simonwillison.net

Post reply on HN
Print view View original

211–220 of 252 posts

Re: Gemini Diffusion

#211

Post by lukan »

skydhash wrote: post #203
lukan wrote: post #191

Earlier quoted context omitted.

I have written 2D games, but maybe the metapher is just lost on me or I simply disagree to its usefulness here. Negative space in art achieves a certain effect. Like in the linked sibling comment, the empty space is part of the sculpture. So the empty space has purpose and meaning. But if I didn't choose a certain libary .. the empty place of that libary serves no function. It does change my code and might make my de…

I’m talking more about architecting code instead of naively writing them. The same point can be made about libraries but the considerations are more subjective. Most naive approaches to writing software looks like assembly. But instead of opcodes, you have libraries functions. But we move away from assembly and assembly like programming because it’s essentially one shot. Any modification to the program is difficult a…

"I’m talking more about architecting code instead of naively writing them."

Yeah, we are talking about code designing.

And I got my head filled with all the design patterns back then in university, but my first bigger real world projects were somehow horribly overengineered and still unflexible. And I don't think it was just lack of experience.

Nowdays I prefer a very, very simple and clear approach.

No dark empty space I want to design around.

No clever hidden layers, that prevent the introduction of a pragmatic new API.

I guess I get what you probably mean and it ain't that, but to me it has too much of the vibe of the time when I was amazed at myself for coming up with a seemingly super clever (complex) design, that sounded great in theory.

Re: Gemini Diffusion

#212

Post by astrodude »

Jackson__ wrote: post #83

What is this blog spam doing here? This is has literally no new information compared to the official release page. It would make a lot more sense to change the link to https://deepmind.google/models/gemini-diffusion/ to discuss the topic.

Simon's blog explains things simply, without all that marketing/hype terminology. Compares performance with competitors. Love reading it

Re: Gemini Diffusion

#213

Post by skydhash »

lukan wrote: post #211
skydhash wrote: post #203

Earlier quoted context omitted.

I’m talking more about architecting code instead of naively writing them. The same point can be made about libraries but the considerations are more subjective. Most naive approaches to writing software looks like assembly. But instead of opcodes, you have libraries functions. But we move away from assembly and assembly like programming because it’s essentially one shot. Any modification to the program is difficult a…

"I’m talking more about architecting code instead of naively writing them." Yeah, we are talking about code designing. And I got my head filled with all the design patterns back then in university, but my first bigger real world projects were somehow horribly overengineered and still unflexible. And I don't think it was just lack of experience. Nowdays I prefer a very, very simple and clear approach. No dark empty sp…

Yes simplicity is always important, but it does not equate easiness. The axe of simple to complex is independent of the axe of easy to hard. It may be easy to apply patterns blindly to your codebass and make it complex. Just how it is easy to write naive and simple code that then becomes difficult to work with.

The mark of a good programmer is to balance all of these so that it’s easy to work with the codebase on an ongoing basis. And more often than not it’s similar to the sketching process. At each stage, you get enough feedback to judge the right direction for the next iteration. You do not start with all the details, nor with careless doodling. But one aspect that is often overlooked with artists is how often they practice to get that judgement capability.

Re: Gemini Diffusion

#214

Post by nvtop »

Workaccount2 wrote: post #193
nvtop wrote: post #97

Earlier quoted context omitted.

Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…

Can you have a hybrid model that can do autoregression and diffusion? It doesn't seem like there is something that would fundamentally prevent this. A model with diffusion CoT for rapid "thought" generation, and then autoregression for the answer on the output.

You can absolutely do it, and I think it's a nice idea to try.

Re: Gemini Diffusion

#215

Post by sroussey »

yorwba wrote: post #77
theptip wrote: post #64

Earlier quoted context omitted.

Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…

You could downscale text the same way you downscale images, by averaging token embeddings instead of pixel values. But you don't have to. AFAIK vision transformers don't suffer from sparse gradients that need a resolution hierarchy to overcome, downscaling is just a performance optimization, because processing an image at full resolution is expensive.

So downscaling will summarize?

Re: Gemini Diffusion

#216

Post by cbeach »

beernet wrote: post #175
ealexhudson wrote: post #149

Earlier quoted context omitted.

Perhaps your content quality meter needs a recalibration?

How so? What makes this blog stand out in terms of quality? I prefer a constructive discussion over personal questions, maybe you should, too.

His posts on AI are often very insightful and, unusually for someone so involved in AI, he's not connected to any of the big AI companies. Therefore he is impartial.

Re: Gemini Diffusion

#217

Post by jonahx »

cztomsik wrote: post #100

I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear attention), and created such franken-stein just by post-training. The big wow moment about that is that it sort of implies that most of the useful knowledge is in the FFN, and attention it…

So is the famous "Attention is all you need" wrong?

Re: Gemini Diffusion

#218

Post by jonahx »

spuz wrote: post #115
ManuelKiessling wrote: post #81

Earlier quoted context omitted.

"...what is not in a codebase, and there is meaningful signal in that negative space." Man, I'm writing software for money for decades now, but this fundamental truth never occured to me, at least not consciously and with such clarity. So, thank you!

I am not certain that I agree with this. If there are alternative ways of solving a problem that we're not taken then these should be documented in comments. A mantra I try to tell myself and my colleagues is if information exists in your brain and nowhere else then write down it down _somewhere_. If I tried 5 different libraries before settling on one, then I write in comments which libraries I tried but didn't work…

What are you disagreeing with?

Even if you do this (and it's good practice!), it is, empirically, not done in the vast majority of codebases.

And even if you succeed with the utmost diligence, a vastly greater number of decisions (those you were not even aware of consciously, or took for granted) will remain undocumented but still be quite real in this "negative space" sense.

Re: Gemini Diffusion

#219

Post by lukan »

skydhash wrote: post #213
lukan wrote: post #211

Earlier quoted context omitted.

"I’m talking more about architecting code instead of naively writing them." Yeah, we are talking about code designing. And I got my head filled with all the design patterns back then in university, but my first bigger real world projects were somehow horribly overengineered and still unflexible. And I don't think it was just lack of experience. Nowdays I prefer a very, very simple and clear approach. No dark empty sp…

Yes simplicity is always important, but it does not equate easiness. The axe of simple to complex is independent of the axe of easy to hard. It may be easy to apply patterns blindly to your codebass and make it complex. Just how it is easy to write naive and simple code that then becomes difficult to work with. The mark of a good programmer is to balance all of these so that it’s easy to work with the codebase on an…

"At each stage, you get enough feedback to judge the right direction for the next iteration."

Depends on the project I would say. What do you do, if all of a sudden the requirements change again? Or the plattform evolved/degraded? Then you compromise - and I can better compromise with simple solution. And I would never claim simple equals easy. Rather the opposite. Like you said, it is easy to make complex things. Also I never applied design patterns for the sake of it(even though it might have sounded like it) KISS was part of the theories as well.. but I did value emphasized cleverness too much as I thought that this is the way it is supposed to be done.

My resume is: simple direct solutions are to be prefered and trying to be clever is not very clever.

I rather have 3 lines of code, than one compressed clever one, no one can understand the first time reading it. And the same goes for the bigger design picture.

Re: Gemini Diffusion

#220

Post by airstrike »

jonahx wrote: post #218
spuz wrote: post #115

Earlier quoted context omitted.

I am not certain that I agree with this. If there are alternative ways of solving a problem that we're not taken then these should be documented in comments. A mantra I try to tell myself and my colleagues is if information exists in your brain and nowhere else then write down it down _somewhere_. If I tried 5 different libraries before settling on one, then I write in comments which libraries I tried but didn't work…

What are you disagreeing with? Even if you do this (and it's good practice!), it is, empirically, not done in the vast majority of codebases. And even if you succeed with the utmost diligence, a vastly greater number of decisions (those you were not even aware of consciously, or took for granted) will remain undocumented but still be quite real in this "negative space" sense.

Exactly. I couldn't have said it better.
Post reply on HN