Earlier quoted context omitted.
What are your thoughts on the HuggingFace incident?
Optimal next token generation disguised as something more.
What else is there?
Continuous Diffusion Language Models (CDLM's)
51–55 of 55 posts
Re: Continuous Diffusion Language Models (CDLM's)
#52Earlier quoted context omitted.
> GPT2 was considered too dangerous to release This is how ridiculous this industry is. Regulation-seeking panic over nothing. Drama in search of a moat. Everything is "too dangerous". GPT2 is going to invent a time machine and break crypto and genetically engineer super rabies. They sell knives, guns, combustible materials, and multi-ton heavy machinery in stores. That's what's actually dangerous.
As I understood it at the time, the rationale was that GPT2 made it too easy to produce content at scale and could easily be used for propaganda, phishing or other kinds of cons. Which turned out to be true.
And which says more about people being stupid and gullible than it does about AI models being smart or dangerous.
I'm tired of allowing stupid and gullible people to determine how the rest of us live, work, play, and create.
Re: Continuous Diffusion Language Models (CDLM's)
#53Earlier quoted context omitted.
GPT2 and 3 work via autoregression. In chat bots, decoders work via autoregression. > people spend years just post-training BERTs in various ways Yes, I was one of them. That’s not called “post-training” it’s called fine-tuning.
Asking out of curiosity, because I have limited experience in that domain. I thought that fine-tuning was changing the weights in the model, not the embedding? Or did I misunderstand?
It’s both. Fine-tuning a BERT model changes its weights, which causes the embeddings to change.
For example you might have one model which embeds a text query and another model which embeds an image. You also have a dataset of image + text captions. Training means updating the weights of those models so that the embedding of the image is close to the embedding of its corresponding caption.
Re: Continuous Diffusion Language Models (CDLM's)
#54> [in 2020/2021] the dominance of autoregression was not as well-established as it is today: GPT-3 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022 I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked. (The only exception I will make is encoder-decoder models…
I suppose what I was trying to say is that in the research community, people were still a lot more willing to entertain alternative modelling paradigms for language at that point, there was much less of a monoculture than there is today. It was really only after ChatGPT that non-autoregressive language modelling came to be seen as a fringe pursuit. Of course that is a highly subjective assessment, and my perspective…
That’s fair, but my impression is that AI is a very industry-driven field. When commercialization fails, industry funding dries up and drags gov funding down too, resulting in an AI winter.
And industry really cares about solving practical problems. So as soon as autoregression “solved” the text generation problem (albeit imperfectly) the field was ready to move on. If text diffusion is not faster, cheaper, or higher quality, there is no ROI.
In fact text diffusion is much faster, so I have used it in the past. And I think it has the potential to be better as well, given a constant time budget. So I think it can be a really promising method moving forward.
Re: Continuous Diffusion Language Models (CDLM's)
#55Earlier quoted context omitted.
What are your thoughts on the HuggingFace incident?
Optimal next token generation disguised as something more.
Well... maybe optimal next token generation is all you need? Reductive as that is. I'm not even sure _why_ that actually matters? It can still result in accidents that impact the real world.
My bet is 2027 we see an agentic worm make global headlines.