Conspiracy theory: Is that the reason why GPT-4 is not available as an API? So people wouldn't siphon off it's capabilities?
I think it is available now.
The False Promise of Imitating Proprietary LLMs
51–60 of 90 posts
Re: The False Promise of Imitating Proprietary LLMs
#52Conspiracy theory: Is that the reason why GPT-4 is not available as an API? So people wouldn't siphon off it's capabilities?
1. It is available via API
2. Likely not a conspiracy theory. Newer models don't have logits available and that's almost certainly because they didn't want other labs distilling from them.
Re: The False Promise of Imitating Proprietary LLMs
#53The authors conduct automated, more methodical evaluations of LLMs finetuned to imitate ChatGPT outputs, and find that, despite superficial/informal appearances to the contrary, the base LLMs close little to none of the gap to ChatGPT on tasks that are not heavily supported in the imitation data. It's not good news for the open LLM ecosystem.
This is a very weird type of paper. They take a specific approach, then make arguments about a broad class of approaches that are under constant development. The finding that distilled LLMs must be more specialized than the giant LLMs that train them is unsurprising; nobody at this point expects a 13B parameter model to succeed with the same accuracy at the broad range of tasks supported by what may be a 1T parameter…
That is exactly what people are expecting, and largely because of misleading metrics thrown around to claim ridiculous things like e.g. Vicuna-13b being nearly as good as GPT-3.5. It even shows up in the comments here, and if you go to any tangentially related subreddit, that's the kind of stuff that gets told as "everybody knows" to people setting up a local LLM for the first time.
Re: The False Promise of Imitating Proprietary LLMs
#54"Second, given the large gap between LLaMA and ChatGPT (the latter model is faster, cheaper, and more accurate), " No it's not, llama would be cheaper and likely faster if you ran it on the same scale, actually there've been a few calcs done, that running llama 65b if you're at 100% usage is cheaper than 3.5turbo per token. Also comparing them for accuracy isn't fair comparison, one is a foundational model, one is an…
Isn't it the comparison is like "My home PC server is far cheaper than EC2"?
Re: The False Promise of Imitating Proprietary LLMs
#55nobody with lots of experience with proprietary LMs is surprised
And neither is anyone who has played with these new LLMs, found them so-so, and wondered whether the hype was warranted.
I’m curios as to why you think the hype isn’t warranted. If you go through my history (you don’t have too I’ll sum it up), you’ll see that I’m not impressed by the capabilities of LLM to actually do my work. Not for a lack of trying, but because ChatGPT simply tells too many lies. We’ve yet to get it to really do anything that wasn’t fairly basic, or solved a billion times on the internet anyway. Similarly we’ve stopped using co-pilot because it takes too much time to make it go away when it’s being bad to make up for the good it does.
Or to put it differently in SWE the LLM seem very bad at building things. What they are good at, however, is helping us build things. I’m not sure I’ll ever need to write JSDoc again on anything that isn’t too sensitive to share. Which is a significant efficiency and quality improvement on the work I do. I think of them as Swagger generators, but instead of being for an OpenAPI standard they are for everything. I imagine they’ll become very good at automating testing as another example, which will again be a further improvement on the work a single developer does.
In terms management might understand. I think you can view LLMs similarity to the way we’ve seen frameworks and tooling reduce the team size needed to build an application significantly over the previous 30 years. If you wanted to build a web-portal for asset management in 1999 you’d need a large team to do what a single developer and a good PO can do today. Maybe we won’t see the same reduction manpower, but instead an increase in quality.
Re: The False Promise of Imitating Proprietary LLMs
#56The authors conduct automated, more methodical evaluations of LLMs finetuned to imitate ChatGPT outputs, and find that, despite superficial/informal appearances to the contrary, the base LLMs close little to none of the gap to ChatGPT on tasks that are not heavily supported in the imitation data. It's not good news for the open LLM ecosystem.
Good news for alignment though. This gives me a tiny amount of hope.
So, LLMs aligned with the interests of our corporate overlords and that nebulous "national security" thing that somehow always translates to more surveillance and less due process?
Re: The False Promise of Imitating Proprietary LLMs
#57Conspiracy theory: Is that the reason why GPT-4 is not available as an API? So people wouldn't siphon off it's capabilities?
Not only it is available, but people have been fine-tuning LLaMA using GPT-4 outputs and training data generated by it for a while now; that's why you see models with names like gpt4-x-alpaca on HuggingFace.
Re: The False Promise of Imitating Proprietary LLMs
#58The breathtaking audacity of calling distilling GPT4 'stealing' when GPT4 trained on data it has no proprietary right to.
They put "stealing" in scare quotes, so it's probably not worth getting fired up about.
Re: The False Promise of Imitating Proprietary LLMs
#59From the Conclusion: "Finally, our work raises ethical and legal questions, including whether the open-source community should continue to advance progress by “stealing” what OpenAI and other companies have done, as well as what legal countermeasures companies can take to protect and license intellectual property." Really???
Was this "study" sponsored by "open"ai?
Re: The False Promise of Imitating Proprietary LLMs
#60Earlier quoted context omitted.
By that logic, you also need to accept that no one should ever need to pay you for creating artifacts that are not bound to the physical world solely. I assume you work for free for your employer or in a space that is not “dealing” with data, information, bits, whatsoever.
Hairdressers charge for a service and none of them will assume that they "own" your hair.
I am honestly shocked that this hasn't happened, what with how the world has been going in recent decades.