691–700 of 1000 posts
People forget that Larry Page's ultimate goal was to have Google predict what you wanted before you even asked for it. This isn't about hype. This isn't about beating the competition. This is about delivering on a 3 decade dream that's now coming to fruition and Sundar just happens to be the one leading the org as it manifests in the real world. We're going to end up with 10+ vertically integrated AI solutions like t…
Question for AI experts out there: ChatGPT is awesome, but it is obviously hamstrung by the fact that it crawled the web at the end of 2021, so all of its data is essentially "frozen" and it doesn't "know" of any topics that occurred after 2021. Seems to me like the next "holy grail" in large language models is building a model that can be continually updated. If Google can achieve that, I think they could leapfrog O…
Like the earlier responses said, if it is retrieving information from an external data source, that would be a much easier way to update things. REALM ( https://arxiv.org/abs/2002.08909 ) comes to mind as an example of active work in this area. Updating a monolithic LLM seems like a harder problem at the moment, and probably would fall under the umbrella of "continual learning", though that sub-field would have their…
Translation: I received a lot of pressure from investors and Larry/Sergey, so I had to rush out an internal competitor to ChatGPT. It’s not going to be as good as ChatGPT initially, so I’m excusing it ahead by saying that we are using a smaller model than what we really have, in order to make it available to more users. However, we are really making it available only to a small group of users, because we need to cont…
Earlier quoted context omitted.
> So my question is, how hard of a problem is that with LLMs? I get the sense that LLMs are trained on a very large data set all at once, but that it is difficult to incrementally update them with new data. It's the opposite. Getting an LLM to learn the basics of grammar and paragraph-level language structure is the "hard" part. Once you have that, further fine-tuning, specialisation, or other incremental changes are…
>Once you have that, further fine-tuning, specialisation, or other incremental changes are comparatively easy. the problem for Google and OpenAI is that most websites are going to start blocking them in robots.txt if they don't find some way to provide value back for allowing them to scrape and train on their content. Pretty much every other bot or search engine is blocked by default and Cloudflare helps block them t…
Earlier quoted context omitted.
Close enough, I'm just peevish about that word. Teaching is miles away from anything in ML or deep learning practice today.
I was speaking in English and doing so concisely to convey the point about bias transfer. I don’t really care what you call it. I could have said RLHF and blacklisting sources or curating training dataset. I know bias when I see it and the LLM does not come up with it on its own if trained on all data out there because for one thing the world is large with all kinds of opinions. When it refuses to legitimize all opin…
Earlier quoted context omitted.
Grammar is surprisingly easy to learn from unstructured data, to an extent. (source: I spent a lot of my PhD doing this kinda thing.) Continual learning seems to be a tough problem though, from what I'm seeing of my friends working on this problem. Like I said in another comment, just doing gradient updates form new data is fraught with problems. RL has a bunch of techniques to mitigate issues that arise with that, b…
Well, yes, but the tokenization schemes are causing these models to struggle on actually following syntactic rules in poetry, e.g. syllable counts in hiakus, rhymes. I actually wrote an entire paper which gwern cited about how to make LLMs always follow these kind of constraints with no mistakes. https://paperswithcode.com/paper/most-language-models-can-be... If you don't believe this is a problem, try getting ChatGP…
Won’t be long before ads start making their way into answers “you want to know how to make popcorn? Try this amazing air frier from company X!”. The internet is enough of a billboard as it is.
Earlier quoted context omitted.
The question becomes will you trust information that is paid for by advertisers or you will trust information that is paid by you, the user? With ads in link based search engine, you can skip or block them, but if it is a part of a one sentence answer, there is not much you can do about it, so consuming it will be much more frustrating. Of course, there will still be a lot of people who will choose the free informati…
"a growing number", "a lot". Where are those users now? We are a tiny lot, nearly economically inconsequential. Your prediction is optimistic.
Earlier quoted context omitted.
> But I can't see how anyone will be incentivized to create and post content for Google to scrape if they won't have any traffic from Google. The problem right now is that the incentives have caused most of the output online to be garbage.
Exactly. If Google is able to provide better answers than the garbage websites with their SEO hacks, those garbage websites will not get clicks. I could see this improving the incentive system significantly