Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

191–200 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#191

Post by sanxiyn »

jxy wrote: post #176
sanxiyn wrote: post #125

Earlier quoted context omitted.

No. Autoregressive models don't have model specific limit to output tokens, it's just when to stop looping.

I guess I only know transformers and how BERT or GPT works, as there would be a limit in the context length. With GPT, you can certainly generate infinite amount of tokens but the previous tokens outside of the maximum context length would be outside of the context window. LLaMa has 2k, GPT-4 has 32k. Are you saying I can give unlimited tokens to PaLM and generate unlimited amount of tokens? So PaLM doesn't have a co…

No, I am not saying that. Since PaLM 2 is a transformer model (they didn't disclose almost anything about the model architecture, but they did disclose that), it has a context length limit. What I am saying is that you can't infer that limit from the limit of maxOutputTokens parameter in the API.

Re: PaLM 2 Technical Report [pdf]

#192

Post by vore »

shrimpx wrote: post #162
stygiansonic wrote: post #143

Earlier quoted context omitted.

In that sense, it's very similar to the GPT-4 Technical Report. The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats to companies.

Btw I've arrived at a different interpretation of the "Open" in OpenAI. It's open in the sense that the generic LLM is exposed via an API, allowing companies to build anything they want on top. Companies like Google have been working on language models (and AI more broadly) for years but have hid the generic intelligence of their models, exposing it only via improvements to their products. OpenAI bucked this trend an…

OpenAI does not need people defending their scummy pivot.

Re: PaLM 2 Technical Report [pdf]

#193

Post by vore »

shrimpx wrote: post #184
ftxbro wrote: post #169

Earlier quoted context omitted.

> Btw I've arrived at a different interpretation of the "Open" in OpenAI. I don't understand why people have to keep trying to wrap their head around the word 'Open' in OpenAI. If you ever saw a commercial like a product has a 'great new taste' but then you tried it and it tasted bad, would you twist yourself into knots trying to understand how you went wrong in your interpretation of 'great'? No that's ridiculous. S…

I think the strong connotation of the word "open" in the software community comes from "open source". If OSS was called "great new source" and a new closed source company called itself GreatNewAI you'd have a similar phenomenon of people taking apart the name.

Understandably, people don’t like it when people take the goodwill of an established term and use it to fulfill their directly opposing goals.

Re: PaLM 2 Technical Report [pdf]

#194

Post by riku_iki »

fzliu wrote: post #140

I don't understand how this can be considered a technical report. No information on model architecture, distributed training methodology, or optimizations. The "Training dataset" section is a pathetic 0.5 pages long. Come on, Google.

they mentioned secret sauce: scaling laws and UL2.

Re: PaLM 2 Technical Report [pdf]

#196

Post by callmekit »

splatzone wrote: post #167
int_19h wrote: post #111

Earlier quoted context omitted.

If you mean asking it what it's running on, it just hallucinates. As others have noted in the comments here, you can get it to say that it runs on PaLM 3 quite easily.

In chat history you can see which model generated each request - for me it’s always LaMDA

I don't see which model generated each request, where exactly do you see this?

Re: PaLM 2 Technical Report [pdf]

#197

Post by getpost »

jp42 wrote: post #62

personal experience - I'm using GPT4 for writing code especially in python. After using bard today, I feel bard is doing quite well considering its free. I will keep using it and if its keep doing well, I will cancel GPT4 $20/month subscription.

Early this evening, I asked Bard if was updated to PaLM 2, and it said it was. I then asked it to write some Python programs, giving it more or less the same prompts I've given GPT4. Bard doesn't seem to be any better than it was a couple weeks ago in the cases I tried, and nowhere near as capable at GPT4. And it goes off the rails quickly. After even a short dialog (~5 statements), it becomes less and less able to stay on track and make coherent corrections to the code.

Re: PaLM 2 Technical Report [pdf]

#198

Post by sanxiyn »

cma wrote: post #101
jumpCastle wrote: post #89

Earlier quoted context omitted.

In my experiments bard is weaker than 3.5, but if it wasn't, than I would prefer the fresh data of bard.

What is its training data cutoff date?

We don't know (both for previous model LaMDA and new model PaLM 2), but it is less important for Bard because Bard has access to live data from Google search.

Re: PaLM 2 Technical Report [pdf]

#199

Post by getpost »

I found an exciting feature—a way to submit a large amount of text—larger than you can paste in the Bard dialog window. (It's possible this isn't a new feature. Bard explained it to me this evening.) You can submit links to files in Google Drive. The links have to be publicly accessible. I just pasted the link to my file in Bard chat.

Bard can access the contents of the 322K file I pasted the link to. It definitely knows about the content of the file. I never said what it was about, but Bard knew it was about butterflies. It knew about content at the beginning of the file, and at the end.

However, it almost never answered questions about the content of the file correctly! For example, I asked it the number of species listed in the file and it said 109. There are 249 numbered species and some that are not numbered. It said the author's name was not in the file, but near the top the file says By . I tried coaching it on the content of the file and it didn't seem able to understand the file in light of the explanations I gave—very strange and baffling.

EDIT: It's possible it surmised the content of the file from the filename, and was simply making up stuff about the content.

Re: PaLM 2 Technical Report [pdf]

#200

Post by re-thc »

adastra22 wrote: post #170
ftxbro wrote: post #169

Earlier quoted context omitted.

> Btw I've arrived at a different interpretation of the "Open" in OpenAI. I don't understand why people have to keep trying to wrap their head around the word 'Open' in OpenAI. If you ever saw a commercial like a product has a 'great new taste' but then you tried it and it tasted bad, would you twist yourself into knots trying to understand how you went wrong in your interpretation of 'great'? No that's ridiculous. S…

You mean when they filled out a form to incorporate their non-profit. Which they later turned into a for-profit company after reaping all the goodwill. The “Open” used to mean something.

All non-profit means is a different tax status. Don't assume they actually don't make money.
Post reply on HN