Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

11–20 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#11

Post by thewataccount »

techbruv wrote: post #5

> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs." Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B). Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM…

I've heard Bard was previously 3B parameters but I could never find a good source for it.

I honestly think the end game here is running on consumer devices, 7B and under need ~4GB of ram to actually run which is likely the max reasonable requirement for consumer devices.

That said medium end hardware can do 15B, anything larger then this is currently something only "enthusiasts" can run.

If it is small enough to run on consumer devices then they don't have to pay for the inference compute at that point, and presumably the latency will be improved for consumers.

Re: PaLM 2 Technical Report [pdf]

#12

Post by ilikeatari »

execveat wrote: post #10
ilikeatari wrote: post #9

So, I asked Bard if it's using PaLM 2 and it did confirm it. My initial results are super promising. Highly recommend checking it out again.

It's a language model, FFS. Ask it whether it uses PaLM 1 and it will confirm it as well.

That is fascinating. Is it the same for GPT 3.5 and 4? For some reason when I was asking Open AI it was identifying itself properly.

Re: PaLM 2 Technical Report [pdf]

#13

Post by famouswaffles »

techbruv wrote: post #5

> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs." Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B). Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM…

Those are the numbers for the scaling law tests they did. Not necessarily Palm 2 range.

Re: PaLM 2 Technical Report [pdf]

#14

Post by famouswaffles »

ilikeatari wrote: post #12
execveat wrote: post #10

Earlier quoted context omitted.

It's a language model, FFS. Ask it whether it uses PaLM 1 and it will confirm it as well.

That is fascinating. Is it the same for GPT 3.5 and 4? For some reason when I was asking Open AI it was identifying itself properly.

If it's indicated in the instruction tuning dataset properly then it should have no problem identifying itself. But we don't know if that happened when bard.

Re: PaLM 2 Technical Report [pdf]

#15

Post by famouswaffles »

execveat wrote: post #10
ilikeatari wrote: post #9

So, I asked Bard if it's using PaLM 2 and it did confirm it. My initial results are super promising. Highly recommend checking it out again.

It's a language model, FFS. Ask it whether it uses PaLM 1 and it will confirm it as well.

Not really. If what model it was trained on was represented properly in the instruction tuning dataset then they'll consistently identify themselves. But it's not a given that that was the case for bard.

Re: PaLM 2 Technical Report [pdf]

#16

Post by suddenexample »

ilikeatari wrote: post #9

So, I asked Bard if it's using PaLM 2 and it did confirm it. My initial results are super promising. Highly recommend checking it out again.

Don't need to ask Bard, it was mentioned at I/O and in this tweet: https://twitter.com/Google/status/1656348200263876608?ref_sr...

Re: PaLM 2 Technical Report [pdf]

#17

Post by tempusalaria »

techbruv wrote: post #5

> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs." Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B). Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM…

GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params

Re: PaLM 2 Technical Report [pdf]

#18

Post by tempusalaria »

typon wrote: post #2

No comparisons against GPT-4 except on three benchmarks where PaLM 2 does better on two. Not sure why, but I expected better from Google.

Most of the GPT-4 benchmarks from their report were things like AP tests or leer code scores. Which aren’t benchmarks that can be compared by a different set of researchers as you don’t know the constituent parts of the test to run

Re: PaLM 2 Technical Report [pdf]

#19

Post by josh_cutler »

execveat wrote: post #10
ilikeatari wrote: post #9

So, I asked Bard if it's using PaLM 2 and it did confirm it. My initial results are super promising. Highly recommend checking it out again.

It's a language model, FFS. Ask it whether it uses PaLM 1 and it will confirm it as well.

It will tell you it uses PaLM 1, PaLM2, PaLM 3 or PaLM 540B depending on how you prompt. It will stop acknowledging incremental PaLM models at 5 it seems.

Re: PaLM 2 Technical Report [pdf]

#20

Post by thewataccount »

tempusalaria wrote: post #17
techbruv wrote: post #5

> "We then train several models from 400M to 15B on the same pre-training mixture for up to 1 × 1022 FLOPs." Seems that for the last year or so these models are getting smaller. I would be surprised if GPT-4 had > the number of parameters as GPT-3 (i.e. 175B). Edit: Seems those numbers are just for their scaling laws study. They don't explicitly say the size of PaLM 2-L, but they do say "The largest model in the PaLM…

GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params

Yeah 1 to 2 trillion is the estimates I've heard.

Given the 25 messages / 3 hour limit in chatGPT, I don't think they've found a way to make it cheap to run.

Post reply on HN