>> Responses on par with txt-DaVinci-3 What is "on par" and how is it measured?
They just compared how the responses were rated by human evaluators. No benchmarks
Alpaca: A strong open-source instruction-following model
21–30 of 313 posts
Re: Alpaca: A strong open-source instruction-following model
#22Re: Alpaca: A strong open-source instruction-following model
#23The training data doesn't seem to be great quality...
"instruction": "Calculate the sum of each column in the following table.",
"input": "2 | 3 | 4\n 5 | 6 | 7",
"output": "2+3+4 = 9\n5+6+7 = 18"
Re: Alpaca: A strong open-source instruction-following model
#24https://crfm.stanford.edu/2023/03/13/alpaca.html They say they'll release the weights. But Of course, you'll be beholden to a non-commercial license. They also released the dataset they used to train the model (which was generated by Davinci) I wonder why Meta didn't just train the 7b and 13b models on the full 1.4T tokens. They curves show they weren't even close to converging. Shame really.
I am not a lawyer, but the consensus (around here anyway) seems to be that weights cannot be copyrighted since they are the output of an automated process, so perhaps the license cannot limit how the weights are used. The code can be copyrighted, but the code can also be reimplemented from scratch and used to load the weights.
That seems like wishful thinking. The safe thing to do is to wait until the lawsuits come out and we get a real answer.
Or if you’re not risk averse, then go for it and take one for the team!
Re: Alpaca: A strong open-source instruction-following model
#25Cost to train this wouldn't be very high. A regular CPU can generate or train about 2 words per second. 52,000 training examples of an average of 300 words would take 3 CPU months. And it is fully parallelizable. And you can probably make it go far faster on a GPU.
They say they trained it for $600.
Re: Alpaca: A strong open-source instruction-following model
#26Re: Alpaca: A strong open-source instruction-following model
#27Pretty exciting to see the democratization of this tech. The next wave of innovation seems like it’ll be much less centralized into a handful of big companies. Even more powerful closed door LLMs will appear, but already so much utility out of these that can run on consumer hardware.
Feels like we’ll get to “good enough” in a matter of weeks.
Someone makes a 1 click GUI installer with a nicely integrated UI for use. No command line use at all.
We’ll get model and weight distributors for updated knowledge.
Re: Alpaca: A strong open-source instruction-following model
#28This is why I think we're seeing a Stable Diffusion moment for LLMs: https://simonwillison.net/2023/Mar/11/llama/
Look at the timeline:
24th February 2023: LLaMA is announced, starts being shared with academic partners: https://research.facebook.com/publications/llama-open-and-ef...
2nd March: Someone posts a PR with a BitTorrent link to the models: https://github.com/facebookresearch/llama/pull/73
10th March: First commit to llama.cpp by Georgi Gerganov: https://github.com/ggerganov/llama.cpp/commit/26c084662903dd...
11th March: llama.cpp now runs the 7B model on a 4GB RaspberryPi: https://twitter.com/miolini/status/1634982361757790209
12th March: npx dalai llama: https://cocktailpeanut.github.io/dalai/
13th March (today): llama.cpp on a Pixel 6 phone: https://twitter.com/thiteanish/status/1635188333705043969
And now, Alpaca. It's not even lunchtime yet!
Turned this into a blog post: https://simonwillison.net/2023/Mar/13/alpaca/
Re: Alpaca: A strong open-source instruction-following model
#29They used OpenAI davinci to generate the dataset for instruction tuning. They mention one of the reasons this model is noncommercial is because OpenAI's terms of use prohibit developing models that compete with OpenAI.
They also published the generated dataset.
If one were to use this dataset to tune your own commercial model, would this be OK, or illegal? I would guess OpenAIs ToS no longer applies, as this is now Alpaca's dataset. You are never agreeing to OpenAIs ToS.