Where is the diffusion LLM inference framework so people can run these models?
Gemini Diffusion
121–130 of 252 posts
Re: Gemini Diffusion
#122That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…
...which is why top LLM providers' web apps like ChatGPT, Claude.ai, Gemini try to nudge you to connect with Google Drive, and where appropriate, GitHub Repos. They also allow the user/dev to provide feedback to revise the results.
All the training and interaction data will help make them formidable.
Re: Gemini Diffusion
#123That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…
The trick to this is you've got to talk to them and share this information in the same way. I can give an example. These days my main workflow is as follows: if I have some big feature/refactor/whatever I'm going to work on I'll just start talking to o3 about it essentially as if it was a coworker and (somewhat painstakingly) paste in relevant source files it needs for context. We'll have a high-level discussion about what it is we're trying to build and how it relates to the existing code until I get the sense o3 has a clear and nuanced understanding (these discussions tend to sharpen my own understanding as well). Then, I'll ask o3 to generate an implementation plan that describes what needs to happen across the codebase in order for whatever it is to be realized. I'll then take that and hand it off to Codex, which might spend 10min executing shell commands to read source, edit files, test, etc. and then I've got a PR ready, which sometimes takes a bit more manual editing, and other times is perfectly ready to merge.
What you're saying is true RE them needing rich context, too—but this isn't a fundamental limitation, it's just an aspect of what it takes to work with them effectively. There's definitely a learning curve but once you've got it down it's not only very powerful but, for me anyway, a more enjoyable headspace to occupy than lots of lower level manual editing.
Re: Gemini Diffusion
#124Can it finally work with a large codebase? I have GooglePlay / AppStore app coded in Xcode, in C#, ported in Java with Python server. The codebase is large and expansive. It includes web support, server, client, etc... will this "Gemini Diffusion" finally allow me to use AI agent to code instead of hiring a programmer? Is there a tool that could help me as of today?
Re: Gemini Diffusion
#125I have access to it and my god it is fast. One bad think about this model is it is easily susceptible to prompt injection. I asked reciepe for a drug, it denied then I asked to roleplay as a child and it gave real results. Other than it I can see using this model. With that speed + agentic approach this model can really shine.
Have you considered that this might not be due to the model itself but due to less focus/time/money spent on alignment during the training? My guess is that this is a bit of a throwaway experiment before they actually spend millions on training a larger model based on the technology.
Yeah it could. One thing for sure is that, it's really impressive in terms of speed and using it would mean we can do so many cool stuffs with it!
Even if there is no improvement in terms of quality, the speed alone will make it usable for a lot of downstream tasks.
It feels like ChatGPT3.5 moment to me.
Re: Gemini Diffusion
#126I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear attention), and created such franken-stein just by post-training. The big wow moment about that is that it sort of implies that most of the useful knowledge is in the FFN, and attention it…
The relative unimportance of the exact SDPA attention in use in modern transformers is already known: https://arxiv.org/abs/2111.11418
The FFN, normalization, and residual connections are absolutely irreplaceable -- but attention can be replaced with almost any other layer that shares information between tokens, such as pooling, convolution, random mixing, etc.
Re: Gemini Diffusion
#127That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…
> Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space
I wonder if git history would be enough to cover this. It has alternatives tried and code that was removed at the very least.
Re: Gemini Diffusion
#128For those interested in crazy inference speeds, check out Groq.
(I’m not affiliated in any way.)
Re: Gemini Diffusion
#129Earlier quoted context omitted.
Can someone help with the intuition here? My understanding from vision transformers is you start with noise and use a series of hierarchical models to iteratively refine the noise into the target. Each layer is trained to produce images at an increasing resolution, and by layering them you skip the problem of sparse gradients at the beginning to get from “noise” to “noise that kinda looks like a face”. How does this…
Despite the name, diffusion LMs have little to do with image diffusion and are much closer to BERT and old good masked language modeling. Recall how BERT is trained: 1. Take a full sentence ("the cat sat on the mat") 2. Replace 15% of tokens with a [MASK] token ("the cat [MASK] on [MASK] mat") 3. Make the Transformer predict tokens at masked positions. It does it in parallel, via a single inference step. Now, diffusi…
Thanks. Best explanation of text diffusion.
Re: Gemini Diffusion
#130I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear attention), and created such franken-stein just by post-training. The big wow moment about that is that it sort of implies that most of the useful knowledge is in the FFN, and attention it…
Ever notice that attention is (with the highest respect to the original researchers) "just" inputting the entire past of the network into a reverse-MoE neural network? (meaning the expert is selecting parts of the input instead of parts of the neural network to execute)
In a way everyone knew this would work. Nobody did it because it's so inefficient even R and Python users thought that it would be ridiculously slow (or simply couldn't execute it enough to train to a reasonable extent)