21–23 of 23 posts
Something I've wondered, maybe I should just do it if I can find some time, but... given DeepSeek's nice results on using rendered text as input, I'm wondering if anyone has given serious research efforts towards image-based diffusion methods for text. As in, instead of all the complexities induced by discrete token generation, just generate the image of the text using standard image diffusion methods, then convert i…
This should be much more efficient in theory, right? Why dont we see more leading labs adopt this?
Earlier quoted context omitted.
Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we _need_ a bit of additional code to massage the input into tokens and especially why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.
“let the embedding be a latent space that forms naturally when training the network” The embeddings are produced in concert with the network, to serve the network, and not created as a separate step. It’s actually very cool The look-up table is a matrix. Each row is an embedding and each row number is a token ID. You get a differentiable transformation from token ID to token embedding using a “one hot vector” and a m…