Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

281–290 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#281

Post by JieJie »

vitorgrs wrote: post #229
JieJie wrote: post #208

Earlier quoted context omitted.

You know, it doesn't that I can see. I recall seeing it with sources in the demo at I/O this morning. It is smarter than the previous beta, but yes, it's still throws some wild pitches, and without sources. Still a work in progress.

According to Google, it will only cite sources if it literally copy-paste answers. So sometimes it says, but is rarely because of course it won't just copy-paste everything.

Yeah, I much prefer Bing's sourcing, though I'm less than pleased with it's sources. Bing likes to find a bunch of "Top 10" posts from content farms that answer whatever question I was asking.

I think LLMs still make up most of their answers, but use whatever links they find to generate context for the answers, so there is a lower possibility it'll generate confabulations. Of course, if the source is low-quality, it's just going to use that to justify a sloppy answer.

Still think it's better than me sorting through content farm posts. I look forward to next year's models that are trained on curated data sifting through well-sourced web sites.

*I like to add, "Only use educational or science journalism sources," to get higher quality links.

Re: PaLM 2 Technical Report [pdf]

#282

Post by flangola7 »

stygiansonic wrote: post #143
fzliu wrote: post #140

I don't understand how this can be considered a technical report. No information on model architecture, distributed training methodology, or optimizations. The "Training dataset" section is a pathetic 0.5 pages long. Come on, Google.

In that sense, it's very similar to the GPT-4 Technical Report. The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats to companies.

> The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats

FTFY

Re: PaLM 2 Technical Report [pdf]

#283

Post by ndr_ »

ilaksh wrote: post #212

Here is their Chat Playground for PaLM 2 https://console.cloud.google.com/vertex-ai/generative/langua... (you have to be logged in to Google Cloud Console I think) Anyone know what parameters are best for code generation? I tried something simple for Node.js and it wasn't horrible but not working. Maybe I used the wron parameters. I tried using 0 for the temperature and turning everything else down like I do with the…

I get this: „ERROR. Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: chat-bison. Please submit a quota increase request.“

Has anyone gotten this fixed?

Re: PaLM 2 Technical Report [pdf]

#285

Post by cubefox »

hbn wrote: post #280
cubefox wrote: post #258

Earlier quoted context omitted.

You can use GPT-4 for free with Bing.

Bing seems dumber than the free tier on OpenAI's chat (I believe it's GPT3.5?). It constantly just falls back to some search results I don't want I don't even bother using it

That's not true. People recognized it was smarter than ChatGPT before GPT-4 was officially revealed.

Re: PaLM 2 Technical Report [pdf]

#286

Post by int_19h »

cubefox wrote: post #266
int_19h wrote: post #116

Earlier quoted context omitted.

Indeed, it's likely that they're running a fairly small model. But this is in and of itself a strange choice, given how ChatGPT became the gateway drug for OpenAI. Why would Google set Bard up for failure like that? Surely they can afford to run a more competent model as a promo, if OpenAI can?

This is just one task it fails at, hardly enough to generalize from.

That's not the only task it fails at, though. Just the one that I found the most interesting when it comes to broader implications because of so many self-contradictions in the output.

Broadly speaking, I haven't seen a single complex example yet where the output was comparable to GPT-4. How close it is to GPT-3.5 is debatable - the overall feeling that I get is that it's better on some tasks and worse on others; this might actually be down to fine-tuning.

Re: PaLM 2 Technical Report [pdf]

#287

Post by adastra22 »

robertlagrant wrote: post #233
adastra22 wrote: post #170

Earlier quoted context omitted.

You mean when they filled out a form to incorporate their non-profit. Which they later turned into a for-profit company after reaping all the goodwill. The “Open” used to mean something.

> Which they later turned into a for-profit company after reaping all the goodwill Did they have a lot of goodwill attached to their company? What did that give them?

When they founded? Yes. The issue was that the big AI players (Google, Facebook, etc.) were keeping their models and training data secret. People (rightly, IMHO) saw this opaque development style as a risk. The OpenAI founders made a big splash by registering as a non-profit and declaring that they were going to do all their model training in public, and share the weights for everyone to use. In other words they were claiming to do something more like what Stability AI is today, except with a stronger legal non-profit organization.

Because of that framing, they poached a lot of very good talent and built one of the best AI teams that has ever been assembled. Then they perverted their corporate structure to be effective for-profit, and renegaded on open access to their trained models, turning into a bog standard service-oriented company.

Re: PaLM 2 Technical Report [pdf]

#288

Post by cubefox »

int_19h wrote: post #286
cubefox wrote: post #266

Earlier quoted context omitted.

This is just one task it fails at, hardly enough to generalize from.

That's not the only task it fails at, though. Just the one that I found the most interesting when it comes to broader implications because of so many self-contradictions in the output. Broadly speaking, I haven't seen a single complex example yet where the output was comparable to GPT-4. How close it is to GPT-3.5 is debatable - the overall feeling that I get is that it's better on some tasks and worse on others; thi…

Makes sense. Others also point out it is not as good as GPT-4 in several benchmarks.

https://news.ycombinator.com/item?id=35895404

They did in fact mostly avoid comparison with GPT-4 in the report. It could of course also be that Bard isn't even running on the largest PaLM 2 model, Unicorn. It seems they would have mentioned that though.

But PaLM 2 seems to be just an intermediate step anyway, since their big new model is "Gemini" (i.e. twins, an allusion to the DeepMind/Brain merger?), which is currently in training, according to Pichai. They also mentioned Bard will switch to Gemini in the future.

Re: PaLM 2 Technical Report [pdf]

#289

Post by Herring »

espadrine wrote: post #6

Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs. That said, they do mention this: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. [A] smaller but higher quality model significantly improves inference efficiency, reduces serving cost, and enables the model’s downstream…

"Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs."

It's kind of weird. In the conclusion, they say

>With PaLM 2, we have independently verified the scaling laws from Hoffmann et al. (2022) at large scales; we have shown that training tokens should grow at roughly the same rate as the number of model parameters.

then a few lines later

>In effect, we find that it is generally more efficient to train a smaller model with more tokens, for a fixed inference and training budget.

Without more architecture details, it's hard to tell what they're going on about.

Re: PaLM 2 Technical Report [pdf]

#290

Post by zarzavat »

Closi wrote: post #214

May be a weird takeaway, but I did find it strange how much the whole report focussed on misgendering as a safety issue. I agree it’s important to get right, but it seems like one of hundreds of safety/alignment issues and that many others are de-emphasised or ignored.

Misgendering in translation is interesting not just because of wokeness but because it is an assuredly AI-complete subproblem in sometimes not AI-complete domain.

For example, let’s say I want to translate “the cat sat on the mat” from English to French. This doesn’t require LLMs, the old bayesian Google Translate could do that just fine.

Now let’s say you want to translate “Carol went to the store. They[3pp] bought some eggs” from a language that doesn’t have gendered 3rd person pronouns, to English which does have gendered pronouns. Now the model needs to know that Carol is a “she”, otherwise you will get the erroneous output “Carol went to the store. He bought some eggs.”

Let’s say we have: “Obama went to the store. [3pp] bought some eggs”. Now the model needs to know whether we are referring to Barack Obama or Michelle Obama so it needs to look back in the context to figure out which Obama which requires comprehension and world knowledge. For example if we precede with “After attending the national security briefing, …” then the model needs to know that: 1) national security briefings are attended by Presidents, 2) Barack Obama was President, in order to deduce that 3) “Obama” here is a “He”.

Getting pronouns right matching human performances requires that model understands language and has some knowledge of the world.

Post reply on HN