Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpaca: A strong open-source instruction-following model

crfm.stanford.edu

Post reply on HN
Print view View original

271–280 of 313 posts

Re: Alpaca: A strong open-source instruction-following model

#271

Post by aortega »

mattkrause wrote: post #128
IanCal wrote: post #106

Earlier quoted context omitted.

I thought I'd put some in as an example. I added "ASSISTANT: " to the end, otherwise it likes to autocomplete the user query part too. This is all with 4 bit quantisation from lambda.cpp, using `--repeat_last_n 256 --repeat_penalty 1.178 --temp 0.7 --top_k 40 --top_p 0`. I've not experimented too much. 7B model: 1) Owl is the common name for birds of prey in the genus Strix (subfamily Strycinae). They have large forw…

It would be even more impressive if it were correct! - Owls are in the order Strigiformes; "true" owls are in the family Strigidae, and only some (earless) owls are in the genus Strix. - As for size, https://largest.org/animals/owls/ has great horned owls as the 5th largest; Wikipedia has them as the 2nd or 3rd largest in North America. - Their legs are long, but the height of an adult person seems like a huge stretc…

LLaMa_65B answer is similar:

SYSTEM: You are an AI research assistant. You use a tone that is technical and scientific.

USER: Hello, who are you?

ASSISTANT: Hi! I am an AI research assistant. How can I help you today?

USER: Tell me something about owls.

ASSISTANT: Owl is the common name for birds of the order Strigiformes. They belong to the class Aves. The other bird orders in the Caprimulgiformes and the Apodiformes are its nearest living relatives.

Re: Alpaca: A strong open-source instruction-following model

#272

Post by espadrine »

kir-gadjello wrote: post #98
azeirah wrote: post #92

Earlier quoted context omitted.

Is there a reason why we aren't using BLOOM's model? It's huge, 160B parameters. It's open, it's free, it's been out there for a while. Is something wrong with it? Using outdated training data? Using outdated training techniques? Or is it just not well known enough -- ie, not in the news at the right time.

BLOOM, while being a useful model for underrepresented languages, underperforms openai's GPT-3 models of similar parameter count on english benchmarks. Which is a pity. And OpenAI's known models (at least the ones for which the architecture is publicly known) https://platform.openai.com/docs/model-index-for-researchers aren't pinnacles of parameter efficiency either: Deepmind's Chinchilla and LLaMA beat them on bench…

> underperforms openai's GPT-3 models of similar parameter count on english benchmarks.

On the flip side, it overperforms it on every other language. It seems clear to me that it is more useful for popular applications (where a large userbase will use their own tongue).

It also has a 7B version. Although it is much weaker than LLaMA, it could be pretrained to match, especially through distillation from LLaMA. By now, it is clear that even Chinchilla is underestimating how much we gain from training more: training is more efficient thanks to new techniques, and even in the paper, some lines are stopped early before reaching Pareto (likely because it ran out of tokens, but LLaMA makes it seem that >1 epoch training is fine).

Re: Alpaca: A strong open-source instruction-following model

#273

Post by aortega »

LeoPanthera wrote: post #194
pmoriarty wrote: post #172

Earlier quoted context omitted.

If it's indistinguishable, then how does it matter who/what it was written by?

Because I am far less likely to trust encyclopedic content written by an AI, which can lie so convincingly that it does not know that it is lying. I have even seen it generating fictional source references that are hard to verify. I just don't trust it.

>can lie so convincingly that it does not know that it is lying.

A human can also lie convincingly.

The problem with AI is that you can do it massively.

Re: Alpaca: A strong open-source instruction-following model

#274

Post by thot_experiment »

valine wrote: post #121
superkuh wrote: post #96

I've played a lot with davinci 3 ($25 of credits worth) and it can do some impressive rhyming and interpretation of concepts as emoji sequences. From the 3 times I've interacted with this fine tuned llama 7B it is clear it cannot do that. I've also run the "vanilla" 7B, 13B, and 30B on my home computer with llama.cpp modified for interactive "chat" mode with various pre-prompt and these can't do it either. I have no…

7B parameters is next to nothing when compared to gpt3. If 7B works as well as it does here, A fine tuned 65B model could very easily achieve chatGPT level performance.

I mean, I'd push back a little bit on that statement but not much, 13B is crazy nice even a 4bit on a 5 year old graphics card. I was able to get it to pretend to be a VM, cat some code into a file, and then pretend it executed the code and it returned correct answers on the first try. I can't wait to try 30B and 65B when I can get access to hardware large enough.

Re: Alpaca: A strong open-source instruction-following model

#275

Post by thot_experiment »

zwaps wrote: post #13
abidlabs wrote: post #11

Cool to see that the entire website, from the agreement to generating the predictions, is one Gradio [1] app written in Python [1] https://gradio.dev/

would be better if it'd work on Firefox tho

I use firefox exclusively and I've been doing a ton of random AI stuff w/ gradio recently and I haven't had any problems so far, what should I be looking out for? Maybe I'm just not noticing.

Re: Alpaca: A strong open-source instruction-following model

#276

Post by bippingchip »

underlines wrote: post #267
2bitencryption wrote: post #46

Earlier quoted context omitted.

Here is one question I have not seen answered yet: All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then further quantizing to int4. Clearly this quantized model still outputs something resembling human language, at the very least. But I haven't seen anyone show what effect this quantizing has on the quality of the output. If the quality of the output is bad, it's un…

See: https://arxiv.org/abs/2210.17323 Q: Doesn't 4bit have worsen output performance than 8bit or 16bit? A: GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16. https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png

This is really interesting, thank you for the reference!

Having worked more with images based NN than language models before, I wonder: are LLM inherently more suited to aggressive quantisation, due to their very large size? I see people suggesting here 4b is pretty good, and 3b should be the target.

I remember ResNets etc can of course also be quantized, and up to 8-6b you get pretty good results with very little effort, with low-ish degradation in performance. Trying to go down to 4b is more challenging, though this paper claims with quantisation aware training 4b is possible indeed, but that means a lot of dedicate training compute needed to get to 4b (not just finetuning post-training): https://arxiv.org/abs/2105.03536

Re: Alpaca: A strong open-source instruction-following model

#277

Post by frozenwind »

From what I understood they used 175 seed tasks as an example for da-vinci which then generated the 52K instructions used to fine-tune LLama.

What I don't understand is how did they achieve giving this context to GPT, since afaik the context window is just 2048 tokens? It seems that it's not fine-tuning, but few-shot.

Re: Alpaca: A strong open-source instruction-following model

#278

Post by throwaway81523 »

dmw_ng wrote: post #72
2bitencryption wrote: post #46

Earlier quoted context omitted.

Here is one question I have not seen answered yet: All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then further quantizing to int4. Clearly this quantized model still outputs something resembling human language, at the very least. But I haven't seen anyone show what effect this quantizing has on the quality of the output. If the quality of the output is bad, it's un…

For 10 billion+ parameter models, the effects of quantization are relatively small, for smaller models like Llama 7B the effect becomes more dramatic, but there is ongoing research on new quantization methods (like GPTQ) that preserve significant performance even on the lower end. Quantization isn't the only technique available for downsizing a model, Llama itself is already the result of sizing the model and input d…

I have heard that the human brain uses the equivalent of around 6 bits. I wonder if that is some kind of optimum reached by evolution.

Re: Alpaca: A strong open-source instruction-following model

#280

Post by paper_good »

rjb7731 wrote: post #154

Seems not too bad, you can tell its been trained on shorter responses. Example #1 Input: explain a neural network in simple terms Alpaca-Output: A neural network is a type of machine learning algorithm that is made up of layers of interconnected nodes (aka neurons). It is inspired by the way the brain works and uses interconnected nodes to learn patterns in data and make predictions. Neural networks can be used to pe…

I tired myself, but it seems Alpaca can't provide contextual conversation. So it's just very simple Q&A. This limit makes Alpaca far away from ChatGPT ability.
Post reply on HN