[deleted]
Llama 2
691–700 of 860 posts
Re: Llama 2
#692[deleted]
Re: Llama 2
#693Earlier quoted context omitted.
I'm confused, fine-tuning is training. How is that not leakage? I'm hesitant to call them researchers, they are employees of a for-profit company trying to meet investor expectations.
1.You train on the kind of problems you want to solve. you don't report numbers that evaluate performance based on examples it trained on. Datasets will typically have splits, one for training and another for testing. 2. Open ai is capped profit. They are also not a publicly traded company. researchers are researchers regardless of who they work for. Training on test data is especially stupid for commercial applicati…
I am suggesting that OpenAI's main product is "LLM that benchmarks the best." From that point, it is completely illogical not to train on at least some of the test data (or data that is very similar to the test data) so that you can fudge the numbers in your favor. You don't want to go too far, but overfitting a tiny bit will make you look like you have a significant edge. When someone says that your product isn't that good, you then point to the benchmarks and say, "objective measures say that you are wrong." This is a tried and true marketing technique.
Hardware companies, which live and die on benchmarks, do this all the time. Meanwhile, it does appear that OpenAI is underperforming consumer expectations, and losing users quite quickly at this point, despite doing incredibly well on benchmarks.
Also, this isn't about profit. It's about market cap and it's about prestige. Those are not correlated to profit.
Re: Llama 2
#694Earlier quoted context omitted.
Apple does not have the capability to train a LLM currently.
Apple has shown time and time again that they have the human capital and money to tackle massive projects discretely. It's already fairly well known that Apple's NLP experts from Siri have been reallocated to some secret project. They are more than capable of training an LLM but given their track record in other segments they probably want to wait for the technology to become more "polished" and give less hallucinate…
Apple is a complete laggard in this space due to years of restrictions on research. They are hiring multiple “AI” roles now and they have the capital and focus to “eventually” catch up — but it is very much a catch-up game.
That said, they seem to prefer catchup waiting till others explore new tech they swoop in an (claim) to perfect it from a usability pov. I have no reason to suspect they won’t do the same here.
Re: Llama 2
#695Earlier quoted context omitted.
Is it possible that some LLM’s are trained on these benchmarks? Which would mean they’re overfitting and are incorrectly ranked? Or am I misunderstanding these benchmarks?…
Presented with no comment :) https://twitter.com/chhillee/status/1635790330854526981?s=46...
Having worked on ML products, there is sometimes debate on whether you should train on the test partition prior to prod deployment - after all, why would you ship a worse model to prod? Obviously you can't tell whether the model is better at generalization compared to an alternate technique, and you also incur some overfit risk. But many industrial problems are solvable through memorization.
Re: Llama 2
#696Key detail from release: > If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under thi…
People keep saying this is commoditize your complement but that's not what this is! Goods A and B are economic complements if, when the price of A goes down, demand for B goes up. LLMs are not complements to social media platforms. There is zero evidence that if "the price of LLMs goes down" then "demand for social media apps go up". This is a case of commoditizing the competition but that's not the same thing. Commo…
Nice analogy and explanation. Another aspect is building a ubiquitous platform and figure out how to monetize later as they (Meta) already have a cash cow.
Zuck is a smart leader. Metaverse was a debacle. But the new world (AI centric) is for real. He is likely focusing on both weakening the stronghold of Google and building a massive community (like Android) around llama. Product ideas (including enterprise focus) will emerge over time.
Re: Llama 2
#697Earlier quoted context omitted.
...is this supposed to make sense? https://www.dropbox.com/scl/fi/5j0mtwdrqm08dxbh84hh6/Screen-...
You're expecting a language model (as opposed to a more general form of ML model) to be numerate? It has no idea that the token "$105,000" has a relationship "is less than" with the token "$140,000". It probably knows that the token "1" has the relationship "is less than" with the token "2" — but that's because it has "1" and "2" as reified concepts, each with many different facts and properties and relationships ass…
> But there's no little numeric model embedded inside the language model that would tell it how many dollars, or be able to compare dollars against dollars
This logic applies to any function an LLM may perform, therefore it cannot perform any function, which is absurd. Indeed there functionally are little circuits inside LLMs; its pretraining amounts to formation of those circuits. [1]
LLaMA tokenizer parses integers as individual tokens and allows the model to see that "$105,000" is [$][1][0][5][,][0][0][0], which makes it more than capable of learning arithmetic for the general case, and with finetuning even the smallest LLaMA-1 can learn to answer better than GPT-4.[2]
It still fails a lot, but so do people, who definitely understand arithmetic on every level from practical to conceptual to algorithmic. The problem is more that it does not have a calculator inside and cannot (by default) use any explicit cache for intermediate values.
1. https://transformer-circuits.pub/2021/framework/index.html
Re: Llama 2
#698Earlier quoted context omitted.
Hard disagree. Google has made it plainly clear that they don't have anything useable in this space. Bard scores below all other commercial model. Google is getting the asses handed to them, badly. I figured that the code red would whip them into shape but the rot runs deep.
It seems you didn't quite hear the argument. I agree with you that the models Google has released to the public are absolutely worthless. That certainly does not mean they don't have extremely performant models at all however. If you actually have worked in the area of NLP for about 10 years, you would recognize how the work from Deepmind is much more novel and innovative than other groups. OpenAI certainly has great…
> but academically Deepmind is one of the best groups around
I think your argument is basically that Google has the potential to create the best models because of superiority in the theory of LLMs, even though we hear of no signs from the board, the ceo, or beta releases or product showcases.
But let’s say you’re right. When do you think we would experience the supremacy of DeepMind in our daily lives?
Re: Llama 2
#699Earlier quoted context omitted.
Enormous complaint about this space: people seemingly never think to include checksums. Drives me wild when there is supposedly all of this concern about the right data and provenance, yet it is not easy to even confirm you have the genuine article.
The checksums are automatically included with the models when you download them using the download.sh script, and verified right after the download completes. This isn't unlike how a lot of packages distribute the SHA256SUMS file next to their downloads over HTTPS, which you can validate yourself. That said it would be nice to announce them somewhere else but if you're already downloading them from Meta directly the…
I suspect the request is to rule out watermarking of the downloads that might suggest watermarking of the model’s outputs.
Re: Llama 2
#700Earlier quoted context omitted.
Hard disagree. Google has made it plainly clear that they don't have anything useable in this space. Bard scores below all other commercial model. Google is getting the asses handed to them, badly. I figured that the code red would whip them into shape but the rot runs deep.
> Google has made it plainly clear that they don't have anything useable in this space. Google hasn't made their best models public because they're too expensive to run for free. > Google is getting the asses handed to them, badly. Bard has 30M active users and isn't even available in large parts of the world. They're in 2nd place - when they were pretty late to the game - that's an odd way to say someone is getting…
Why do you think Google had even bothered with Bard?
And then, given that, why is it worse than the competition?