Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to build a diffusion language model

kuleshov-group.github.io

Post reply on HN
Print view View original

11–20 of 23 posts

Re: How to build a diffusion language model

#11

Post by p1esk »

amelius wrote: post #7
radarsat1 wrote: post #6

Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text. As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert i…

I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.

There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive.

Also, you still need token embeddings (I think you might be confused how that works).

Re: How to build a diffusion language model

#12

Post by amelius »

p1esk wrote: post #11
amelius wrote: post #7

Earlier quoted context omitted.

I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.

There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive. Also, you still need token embeddings (I think you might be confused how that works).

Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we _need_ a bit of additional code to massage the input into tokens and especially why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.

Re: How to build a diffusion language model

#13

Post by faangguyindia »

radarsat1 wrote: post #6

Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text. As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert i…

Diffusion is already being used in drafters

Re: How to build a diffusion language model

#14

Post by butvacuum »

amelius wrote: post #12
p1esk wrote: post #11

Earlier quoted context omitted.

There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive. Also, you still need token embeddings (I think you might be confused how that works).

Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we _need_ a bit of additional code to massage the input into tokens and especially why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.

are you aware of n-grams?

Re: How to build a diffusion language model

#15

Post by mmastrac »

What's missing from these discussions is the real weakness of diffusion models: you can have two positions where two tokens need to be coordinated, but both spots are teetering between two valid states. They don't always collapse to the matching pair.

I only learned this the hard way reimplementing diffusiongemma. I had ideas on how to fix it but no cluster to train and experiment, hah.

Re: How to build a diffusion language model

#16

Post by matrix2596 »

mmastrac wrote: post #15

What's missing from these discussions is the real weakness of diffusion models: you can have two positions where two tokens need to be coordinated, but both spots are teetering between two valid states. They don't always collapse to the matching pair. I only learned this the hard way reimplementing diffusiongemma. I had ideas on how to fix it but no cluster to train and experiment, hah.

have u seen the dspark paper, they add a morkov head for light weight dependency, not sure whether it can be extended to multi step refining

Re: How to build a diffusion language model

#17

Post by thatjoeoverthr »

amelius wrote: post #7
radarsat1 wrote: post #6

Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text. As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert i…

I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.

Embeddings are there to make continuous-space identities so you can run them through a differentiable model. Without this the tokens (no matter your granularity) are pure surrogate identities, and you can’t run a gradient through them. You also hit the curse of dimensionality hard because the model can’t perceive similarity. “Cat” and “kitten” for example are simply different atoms of text, but with embeddings, you can leverage what you learned about “cat” when you encounter “kitten”. Look at “A Neural Probabalistic Language Model” (Bengio, 2003).

You can actually rig up an embedding variant of a Markov chain with just a few tokens of context, and no position coding, transformers, attention, none of it, and only minutes of training time. As long as you have the embedding lookup table trainable it will do some neat stuff.

Re: How to build a diffusion language model

#18

Post by thatjoeoverthr »

amelius wrote: post #12
p1esk wrote: post #11

Earlier quoted context omitted.

There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive. Also, you still need token embeddings (I think you might be confused how that works).

Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we _need_ a bit of additional code to massage the input into tokens and especially why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.

“let the embedding be a latent space that forms naturally when training the network”

The embeddings are produced in concert with the network, to serve the network, and not created as a separate step.

It’s actually very cool

The look-up table is a matrix. Each row is an embedding and each row number is a token ID.

You get a differentiable transformation from token ID to token embedding using a “one hot vector” and a matrix multiplication

If you take the transpose of this matrix, you can convert an internal representation back to the same token form, but treat it as logits and give it to the sampler.

So token embeddings are produced on demand in service of the model, according to the model’s needs.

I found an example of this strategy in a paper as far back as 1980!

In the other reply I recommend the Bengio paper. But do bite the bullet and try it.

Re: How to build a diffusion language model

#20

Post by ux266478 »

pwmglenn wrote: post #19

This should be much more efficient in theory, right? Why dont we see more leading labs adopt this?

Weak "chain-of-thought" abilities, high error rates and a very bad ability to recover from errors. If they produce a non-sequitur somewhere (which they are highly prone to doing), that global refinement spreads it everywhere like a blood infection superhighway.

Diffusion text models are cool, but they're functionally much less reliable than autoregressive transformers... and man that's really saying something. Right now most research on them is trying to figure out what complementary systems they need to be reasonably useful.

Post reply on HN