101–110 of 297 posts
Earlier quoted context omitted.
why don't you just use chatGPT? from what i know it's running GPT3.5 and it's not that different (at least in terms of code quality)
In my experiments bard is weaker than 3.5, but if it wasn't, than I would prefer the fresh data of bard.
Earlier quoted context omitted.
Optimazing for training could help distillation also.
I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.
Earlier quoted context omitted.
I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.
This was published here in HN last week: https://news.ycombinator.com/item?id=35810663 Don't know if there any public technical reports by any of the big AI companies about this, as its pretty new.
Earlier quoted context omitted.
why don't you just use chatGPT? from what i know it's running GPT3.5 and it's not that different (at least in terms of code quality)
In my experiments bard is weaker than 3.5, but if it wasn't, than I would prefer the fresh data of bard.
With GPT 3.5 and 4, I was able to just paste in the error and it'd do the rest. Bard however tried to tell me what the error could be, and wouldn't do well even when asked to fix the code.
Even GPT 4 though, when asked to go from specs to tests + code, would get stuck in a loop of making one test pass only to make the other pass and vice versa. The program I tried to let it write was a query validator that can test whether a string matches a pattern that uses AND, OR and NOT.
It did well on parsing my specs into tests, but from there on it didn't go very well.
Earlier quoted context omitted.
Optimazing for training could help distillation also.
I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.
Earlier quoted context omitted.
I tried my hand at estimating their parameter count from extrapolating their LAMBADA figures, assuming they all trained on Chinchilla law: https://pbs.twimg.com/media/Fvy4xNkXgAEDF_D?format=jpg&name=... If the extrapolation is not too flawed, it looks like PaLM 2-S might be about 120B, PaLM 2-M 180B, PaLM 2-L 280B. Still, I would expect GPT-4 trained for way longer than Chinchilla, so it could be smaller than even Pa…
They said the smallest PaLM 2 can run locally on a Pixel Smartphone. There's no way it's 120B parameters. It's probably not even 12B.
At I/O, I think they were referencing the scaling law experiments: there are four of them, just like the number of PaLM 2 codenames they cited at I/O (Gecko, Otter, Bison, and Unicorn). The largest of those smaller-scale models is 14.7B, which is too big for a phone too. The smallest is 1B, which can fit in 512MB of RAM with GPTQ4-style quantization.
Either that, or Gecko is the smaller scaling experiment, and Otter is PaLM 2-S.
Earlier quoted context omitted.
I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.
Off the top of my head there's DistilBERT from awhile back. I also recall distilled GPT-2 models from before the GPT-3 times.
Earlier quoted context omitted.
I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.
The name 3.5-turbo sounds to me like it implies distillation. The release notes at the time also hinted at it IIRC.