Once they released its coding ability it became more useful. I use Bard less than ChatGPT still, but it is not useless since it has more modern information.
PaLM 2 Technical Report [pdf]
31–40 of 297 posts
Re: PaLM 2 Technical Report [pdf]
#32So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?
You can sign up for the waitlist at g.co/palm
Re: PaLM 2 Technical Report [pdf]
#33So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?
No API, but Bard is on it.
Re: PaLM 2 Technical Report [pdf]
#34So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?
It should be live on Bard.
Re: PaLM 2 Technical Report [pdf]
#35So, I asked Bard if it's using PaLM 2 and it did confirm it. My initial results are super promising. Highly recommend checking it out again.
Is there any reason to believe it was trained on any amount of technical documentation about itself?
I mean, even if it was, it would be trivial to get it to make stuff up anyway.
Re: PaLM 2 Technical Report [pdf]
#36Once they released its coding ability it became more useful. I use Bard less than ChatGPT still, but it is not useless since it has more modern information.
Is it better than bing or phind though? Why would I use it over bing?
Re: PaLM 2 Technical Report [pdf]
#37Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs. That said, they do mention this: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. [A] smaller but higher quality model significantly improves inference efficiency, reduces serving cost, and enables the model’s downstream…
Optimazing for training could help distillation also.
Re: PaLM 2 Technical Report [pdf]
#38> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs." Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B). Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM…
GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params
The idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast (THIS IS WRONG, SEE CORRECTION BELOW).
These days, the largest models that have been trained optimally (in terms of model size w.r.t. tokens) typically hover around 50B (likely PaLM 2-L size and LLaMa is maxed at 70B). We simply do not have enough pre-training data to optimally train a 1T parameter model. For GPT-4 to be 1 trillion parameters, OpenAI would have needed to:
1) somehow magically unlocked 20x the amount of data (1T tokens -> 20T tokens) 2) somehow engineered an incredibly fast inference engine for a 1T GPT model that significantly better than anything anyone else has built 3) is somehow is able to eat the cost of hosting 1T parameter models
The probability that all the above 3 have happened seem incredibly low.
CORRECTION: The refutation for the size of GPT-4 on the lex fridman podcast was that GPT-4 was 100T parameters (and not directly, they were just joking about it), not 1T, however, the above 3 points still stand.
Re: PaLM 2 Technical Report [pdf]
#39> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs." Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B). Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM…
GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params
ChatGPT 3.5 is likely much smaller than GPT-3’s 175b parameters. Based on the API pricing, I believe 8k context GPT-4 is larger than 175b parameters, but less than 1t.
Re: PaLM 2 Technical Report [pdf]
#40[deleted]