Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

201–210 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#201

Post by javierluraschi »

getpost wrote: post #199

I found an exciting feature—a way to submit a large amount of text—larger than you can paste in the Bard dialog window. (It's possible this isn't a new feature. Bard explained it to me this evening.) You can submit links to files in Google Drive. The links have to be publicly accessible. I just pasted the link to my file in Bard chat. Bard can access the contents of the 322K file I pasted the link to. It definitely k…

Seems like the file was in tabular format? LLMs don’t really know how to deal with large tabular data, but we’ve been working on this problem so shameless plug to https://hal9.ai

Re: PaLM 2 Technical Report [pdf]

#202

Post by whoisjuan »

ewoodrich wrote: post #157
whoisjuan wrote: post #63

Earlier quoted context omitted.

Well, I tried it, and this is how dumb it is. I ask it what's the context length it supports. It said that PaLM 2 supports 1024 tokens and then proceeds to say that 1024 tokens equals 1024 words, which is obviously wrong. Then I changed the prompt slightly, and it answered that it supports 512 tokens contradicting its previous answer. That's like early GPT-3.0 level performance, including a good dose of hallucination…

Why is character streaming important if Bard seems to be faster generating a complete answer than ChatGPT?

That's because simple questions in Bard only generate like 200 tokens per answer. The latency is more noticeable for longer answers.

Re: PaLM 2 Technical Report [pdf]

#203

Post by FanaHOVA »

simonw wrote: post #4

"The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data" I really want to know more about the training data. Which web documents, which books, code from where, conversational data from where?

They don't specify, but if you're generally curious you should look into mC4, RedPajama, The Stack, etc as they are the foundation of most training sets.

Re: PaLM 2 Technical Report [pdf]

#204

Post by csomar »

shrimpx wrote: post #162
stygiansonic wrote: post #143

Earlier quoted context omitted.

In that sense, it's very similar to the GPT-4 Technical Report. The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats to companies.

Btw I've arrived at a different interpretation of the "Open" in OpenAI. It's open in the sense that the generic LLM is exposed via an API, allowing companies to build anything they want on top. Companies like Google have been working on language models (and AI more broadly) for years but have hid the generic intelligence of their models, exposing it only via improvements to their products. OpenAI bucked this trend an…

You don't need to "arrive" at an interpretation. Their name is legacy from when they were "open".

Re: PaLM 2 Technical Report [pdf]

#205

Post by getpost »

javierluraschi wrote: post #201
getpost wrote: post #199

I found an exciting feature—a way to submit a large amount of text—larger than you can paste in the Bard dialog window. (It's possible this isn't a new feature. Bard explained it to me this evening.) You can submit links to files in Google Drive. The links have to be publicly accessible. I just pasted the link to my file in Bard chat. Bard can access the contents of the 322K file I pasted the link to. It definitely k…

Seems like the file was in tabular format? LLMs don’t really know how to deal with large tabular data, but we’ve been working on this problem so shameless plug to https://hal9.ai

The file is not in tabular format; that’s why I wanted Bard help, to process a highly irregular file. I was hoping to produce a tabular format.

Re: PaLM 2 Technical Report [pdf]

#206

Post by blahgeek »

getpost wrote: post #199

I found an exciting feature—a way to submit a large amount of text—larger than you can paste in the Bard dialog window. (It's possible this isn't a new feature. Bard explained it to me this evening.) You can submit links to files in Google Drive. The links have to be publicly accessible. I just pasted the link to my file in Bard chat. Bard can access the contents of the 322K file I pasted the link to. It definitely k…

> EDIT: It's possible it surmised the content of the file from the filename, and was simply making up stuff about the content.

I think this is the most probably explanation.

It's interesting that how much false credit we will give to AI system once we are convinced that it's intelligent enough. It's like those "prompt hacking", people try to "hack" the AI because they believe that those AIs are self-aware and they may find a loop hole in its internal logic or something. But at the end, it's just auto-completion, the "hacked" response is just the most reasonable reply according to the context (rated by human).

Re: PaLM 2 Technical Report [pdf]

#207

Post by simonw »

FanaHOVA wrote: post #203
simonw wrote: post #4

"The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data" I really want to know more about the training data. Which web documents, which books, code from where, conversational data from where?

They don't specify, but if you're generally curious you should look into mC4, RedPajama, The Stack, etc as they are the foundation of most training sets.

I've spent quite a bit of time exploring RedPajama! https://simonwillison.net/2023/Apr/17/redpajama-data/

Re: PaLM 2 Technical Report [pdf]

#208

Post by JieJie »

jacooper wrote: post #188
JieJie wrote: post #64

Earlier quoted context omitted.

Bard is really fast. Faster than Bing and Phind.

Yeah fast, but also kinda garbage last time I tried it. Does it even show sources now?

You know, it doesn't that I can see. I recall seeing it with sources in the demo at I/O this morning.

It is smarter than the previous beta, but yes, it's still throws some wild pitches, and without sources. Still a work in progress.

Re: PaLM 2 Technical Report [pdf]

#209

Post by speedgoose »

jacooper wrote: post #34
jxy wrote: post #23

So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?

It should be live on Bard.

Bard is not live though.

> Bard isn't currently supported in your country. Stay tuned!

It has been months…

Re: PaLM 2 Technical Report [pdf]

#210

Post by nabakin »

techbruv wrote: post #38
tempusalaria wrote: post #17

Earlier quoted context omitted.

GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params

The idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast (THIS IS WRONG, SEE CORRECTION BELOW). These days, the largest models that have been trained optimally (in terms of model size w.r.t. tokens) typically hover around 50B (likely PaLM 2-L size and LLaMa is maxed at 70B). We simply do not have enough pre-training data to optimally train a 1T parameter model. Fo…

GPT-2 training cost 10s of thousands

GPT-3 training cost millions

GPT-4 training cost over a hundred million [1]

GPT-4 inferencing is slower than GPT-3 or GPT-3.5

OpenAI has billions of dollars in funding

OpenAI has the backing of Microsoft and their entire Azure infra at cost

There is no way GPT-4 is the same size as GPT-3. Is it 1T parameters? I don't know. No one knows. But I think it is clear GPT-4 is significantly larger than GPT-3.

For fun, if we plot the number of parameters vs training cost we can see a clear trend and I imagine, very roughly predict the amount of parameters GPT-4 has

https://i.imgur.com/rejigr5.png

https://www.desmos.com/calculator/lqwsmmnngc

[1]

> At the MIT event, Altman was asked if training GPT-4 cost $100 million; he replied, “It’s more than that.”

http://web.archive.org/web/20230417152518/https://www.wired....

Post reply on HN