61–70 of 79 posts
This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
That's why Alpaca is so exciting: it instruction-tunes LLaMA to the point that even the tiny 7B model (the one that fits on a phone) produces useful output: https://simonwillison.net/2023/Mar/13/alpaca/
This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
Earlier quoted context omitted.
But they won’t give us the model… so it’s ultimately meaningless because they’ll just sell out
My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.
This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
That's why Alpaca is so exciting: it instruction-tunes LLaMA to the point that even the tiny 7B model (the one that fits on a phone) produces useful output: https://simonwillison.net/2023/Mar/13/alpaca/
If the cloud of uncertainty around commercial use of derivative weights from LLaMA can be resolved, I think this could be the answer for a lot of domain-specific generative language needs. A model you can fine tune on your own data, and which you host and control, rather than depending on a cloud service not to arbitrarily up prices/close your account/apply unhelpful filters to the output/etc.
Earlier quoted context omitted.
My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.
Why they can train their model on copyrighted data and claiming fair use as they do not outright copy while disallowing training other models on their output? I understand revoking access though.
Person A uses GPT3 to generate training data, and publish it on his blog, without representing it as human generated. Person A does not give permission for it to be used by Alpaca team.
Alpaca team comes along, scrape his blog, and uses it as training data, without permission from person A. Now this is fair use, so there is nothing person A can do to stop it, just like how Github scraped our code for Copilot without permission.
This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
If you are talking about the video that's perfectly fluent English. There are some unusual elements to the story which probably wouldn't be there in a larger model.
I'd invite you to try that with a Markov model or even something like a LSTM based neural network and compare.
This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.
Isn't any LLM mathematically a Markov chain, such that the current state includes the context of the last (finite) n tokens?
Earlier quoted context omitted.
How is that an iPhone 7 is completely current vs give me a branded Android from the same year of release that has even security updates, not even features.
The problem with iPhones is once updates stop there's nothing you can do. The iPhone 7 isn't current, it's stuck on iOS 15 while the newest is 16. And while the pixel 2 (which is only a month younger than the iPhone 7) only got official support up to Android 11; you actually own the device and can easily unlock the boot-loader to upgrade to Android 13.
Earlier quoted context omitted.
My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.
What will OpenAI do, sue? Okay but now it's out there.