131–140 of 297 posts
"The PaLM 2 pre-training corpus is composed of a diverse set of sources: web documents, books, code, mathematics, and conversational data" I really want to know more about the training data. Which web documents, which books, code from where, conversational data from where?
Earlier quoted context omitted.
The report specifically states: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute The largest PaLM model is 540B. So all of PaLM 2 is potentially double-digit parameters. Note though that GPT-3.5 was plausibly not a finetuning of the 175B model, but instead a finetuning of Codex which was based on the 12B version of GPT-3.
How could GPT-3.5 possibly have been a finetuning of the 175B model? They didn't even use the same tokens?
Token embeddings can be trained without changing the other parameters. There is a number of models which add tokens as a finetuning step. Here is recently StarCoder adding ChatML-equivalent tokens: https://huggingface.co/blog/starchat-alpha#a-standard-format...
Earlier quoted context omitted.
Given Reddit's inability to keep their website functioning (unless you use the far superior old.reddit.com) I find it hard to believe they would be able to stop a motivated developer from scraping the whole site.
this is about the time that i expect sites to begin returning intentionally corrupt/incorrect/perhaps outright garbage (subtle or not, probably better subtle so they don't realize it until it's far too late) data in order to intentionally poison enemy wellscraping. where "ethics" dissolve into the inherent raw cannibalistic laws of capitalist ventures. then you can sell them back the TBs they scraped at a 1000x marku…
Earlier quoted context omitted.
Given Reddit's inability to keep their website functioning (unless you use the far superior old.reddit.com) I find it hard to believe they would be able to stop a motivated developer from scraping the whole site.
this is about the time that i expect sites to begin returning intentionally corrupt/incorrect/perhaps outright garbage (subtle or not, probably better subtle so they don't realize it until it's far too late) data in order to intentionally poison enemy wellscraping. where "ethics" dissolve into the inherent raw cannibalistic laws of capitalist ventures. then you can sell them back the TBs they scraped at a 1000x marku…
Earlier quoted context omitted.
I don't think you can do this, it will just make things up. Language models don't have this type of reflection. Google would need to indicate this out of band, like on the page itself, in order for you to be confident about what model you're using.
Agreed. I'm not entirely sure that the person you're replying to is not joking
Many artists and such that I've spoken to about AI work have similar comments about these systems because of the disdain for their existence.
The number of times I hear an argument like "well, they can never taste the tartness of a kiwi and feel the heat of the sun while at the beach" gets quite exhausting. For some reason, many people have this weird notion that this is what AGI means - exactly what humans do, and specifically within the same data domains of humans, but they don't consider working solely outside those domains as a possibility for AGI.
Earlier quoted context omitted.
How could GPT-3.5 possibly have been a finetuning of the 175B model? They didn't even use the same tokens?
Finetuning might not be the best word; sometimes it is a grey line. Token embeddings can be trained without changing the other parameters. There is a number of models which add tokens as a finetuning step. Here is recently StarCoder adding ChatML-equivalent tokens: https://huggingface.co/blog/starchat-alpha#a-standard-format...
You can verify that your Bard instance is using Palm 2 by asking “are you using the palm or palm 2 model?”
I've asked "are you using palm 3": It said: I am using the Palm 3 model. Palm 3 is a large language model... Don't believe it :) Also, In the technical report, It mentions multiple languages, I've asked in Turkish which was supposed to be supported, but wasn't able to answer. Even if its PaLM 2, its hard to trust to the model itself.
It said "I am not using either the Palm 420 language model or the Palm 2 language model. I am using a different language model called Bard, which is a large language model from Google AI."
Perhaps the people at Google saw this and made a manual correction? Hard to say, black boxes and all...
The thing is, once a company creates a proto AGI where the path to a functional AGI is entirely predictable with more compute, they'll keep it a secret. Who would share the fact that the greatest achievement in human history is possible when having it before anyone else gives you a huge competitive advantage?
Earlier quoted context omitted.
I don't think you can do this, it will just make things up. Language models don't have this type of reflection. Google would need to indicate this out of band, like on the page itself, in order for you to be confident about what model you're using.
Agreed. I'm not entirely sure that the person you're replying to is not joking
Come on, Google.