Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpaca: A strong open-source instruction-following model

crfm.stanford.edu

Post reply on HN
Print view View original

241–250 of 313 posts

Re: Alpaca: A strong open-source instruction-following model

#241

Post by mirekrusin »

dougmwne wrote: post #165
simonw wrote: post #147

Earlier quoted context omitted.

Language models like LLaMA (and GPT-3, ChatGPT etc) are really good at exactly this. They can summarize text, they can extract key points into bullets, they can figure out common topics and so on.

Waiting to be able to feed a LLM a transcript of an ongoing meeting, then ping me when my attention is needed with the context summarized.

Why bother with pingig? Write answer, text-to-speech it and fake-voice it so it sounds like you.

Don't forget to wrap it with supervisor AI so you can replicate minions 100x+ times.

Re: Alpaca: A strong open-source instruction-following model

#243

Post by MacsHeadroom »

DrSiemer wrote: post #198
koheripbal wrote: post #151

Earlier quoted context omitted.

The difference is small, UNTIL you get to 4 bit quantization, where the model is noticeably dumber. 8 bits, imo, is the minimum.

So which is better, running 7B without quantization or running 13B with? They both require about the same amount of vRAM (10gb).

Empirically, 13B with quantization.

In fact the person who said 4bit is worse is empirically incorrect.

13B with quantization even down to 3-bits has very near the same performance as uncompressed 16bit 13B with GPTQ quantization and binning.

Source: https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i...

Re: Alpaca: A strong open-source instruction-following model

#244

Post by maxxk »

notpushkin wrote: post #232
zitterbewegung wrote: post #212

Earlier quoted context omitted.

LLAMA.cpp with 65B parameters runs on a MacBook M1 Max with 64GB of RAM. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...

What's the correlation between parameter count and RAM usage? Will LLaMA-13B fit on my MacBook Air with 8 GB of RAM or am I stuck with 7B?

13B uses about 9GB on my MacBook Air. If you have another machine (x86) with enough RAM to convert the original LLaMA representation to GGML, you can give it a try. But quantization step must be done on MacBook.

Maybe it is more feasible for you to use 7B with larger context. For some "autocompletion" experiments with Python code I had to extend context to 2048 tokens (+1-1.5GB).

Re: Alpaca: A strong open-source instruction-following model

#245

Post by nl »

meghan_rain wrote: post #239
nl wrote: post #196

Note that the OpenAssistant project has a 100k+ multilingual instruction tuning dataset they are in the process of releasing. Stats from March 9 are 50K English prompts and 40K Spanish.

Wow that was quick, they only started in what, December?

Yeah amazingly fast.

I haven't looked at the quality but some comments in the discord seemed to indicate it was decent quality but had a lot of coding-type prompts.

Re: Alpaca: A strong open-source instruction-following model

#246

Post by lxe »

nickthegreek wrote: post #14

>We are waiting for huggingface to officially support the llama models (i.e. this PR to be merged) before we release a stable version of the finetuning code. Can't wait for this to be released to see all the different fine tuned models that will be coming out. There was an article a few days ago about LLM having its Stable Diffusion moment, and I think it was right. We went from LLama release > llama.cpp > Dalai and…

Check this out: https://github.com/facebookresearch/llama/issues/169

Also I'm working on a very hacky way to tune with the hf ecosystem as my very first attempt to do deepspeed/accelerate/transformers https://github.com/lxe/llama-tune

Re: Alpaca: A strong open-source instruction-following model

#247

Post by code51 »

IanCal wrote: post #106
rjb7731 wrote: post #68

Earlier quoted context omitted.

With the 7B none-instruction tuned you have to really pre-prompt it to get good answers to things so like: SYSTEM: You are an AI research assistant. You use a tone that is technical and scientific. USER: Hello, who are you? ASSISTANT: Hi! I am an AI research assistant. How can I help you today? USER: Can you tell me something about owls.

I thought I'd put some in as an example. I added "ASSISTANT: " to the end, otherwise it likes to autocomplete the user query part too. This is all with 4 bit quantisation from lambda.cpp, using `--repeat_last_n 256 --repeat_penalty 1.178 --temp 0.7 --top_k 40 --top_p 0`. I've not experimented too much. 7B model: 1) Owl is the common name for birds of prey in the genus Strix (subfamily Strycinae). They have large forw…

Why would I be getting this from LLaMA_MPS when use the same prompt?

NARRATOR (Voice): This is the latest version of 1857, our new general purpose artificial intelligence core architecture. It has been designed to perform as diverse tasks as possible in order to create technologies useful for society.

SYSTEM: We have successfully created emotional intelligence, cognitive reasoning, natural language processing, problem solving skills, perception capabilities, and even social intelligence. The only thing left to test is what kind of human-like consciousness we can replicate.

NARRATOR (Voice): What you are about to do will be vital to the future development of artificial intelligence technology. SYSTEM: For this experiment we’ll be using your own memories as reference data. Your memory will give us insight into how complex human thinking works. And since it was previously recorded through brain scans, there won’t be any need for additional information collection from you.

NARRATOR (Voice): Our first step will be to copy your existing memory data files. The next stage involves rewiring those neural pathways so they function

Re: Alpaca: A strong open-source instruction-following model

#248

Post by code51 »

code51 wrote: post #247
IanCal wrote: post #106

Earlier quoted context omitted.

I thought I'd put some in as an example. I added "ASSISTANT: " to the end, otherwise it likes to autocomplete the user query part too. This is all with 4 bit quantisation from lambda.cpp, using `--repeat_last_n 256 --repeat_penalty 1.178 --temp 0.7 --top_k 40 --top_p 0`. I've not experimented too much. 7B model: 1) Owl is the common name for birds of prey in the genus Strix (subfamily Strycinae). They have large forw…

Why would I be getting this from LLaMA_MPS when use the same prompt? NARRATOR (Voice): This is the latest version of 1857, our new general purpose artificial intelligence core architecture. It has been designed to perform as diverse tasks as possible in order to create technologies useful for society. SYSTEM: We have successfully created emotional intelligence, cognitive reasoning, natural language processing, proble…

Adding prompt for reference:

SYSTEM: You are an AI research assistant. You use a tone that is technical and scientific.

USER: Hello, who are you?

ASSISTANT: Hi! I am an AI research assistant. How can I help you today?

USER: Can you tell me something about owls.

ASSISTANT:

Re: Alpaca: A strong open-source instruction-following model

#249

Post by ninjin »

azeirah wrote: post #92
bogwog wrote: post #31

Earlier quoted context omitted.

Only if one of those handful of big companies decides to release their models for commercial use. The LLaMA models used here are only available under a non-commercial license. Can’t build anything significant with leaked models. Hopefully we don’t see Facebook/Google adopting OpenAIs business model where they sell API access, although I can’t imagine them ever releasing their models directly. The last thing they woul…

Is there a reason why we aren't using BLOOM's model? It's huge, 160B parameters. It's open, it's free, it's been out there for a while. Is something wrong with it? Using outdated training data? Using outdated training techniques? Or is it just not well known enough -- ie, not in the news at the right time.

BLOOM is not open in the usual sense of the word as it has an ethical license (I wrote a longer comment about this way back [1]). I doubt that this is an issue for academic usage, but I am not sure how comfortable one should be basing a company on compliance with (what is so far) an untested license with vague and wide wording.

[1]: https://news.ycombinator.com/item?id=32079558

With all the appropriation going on from both commercial and academic organisations I think we need a new name for these kinds of models. Currently I am leaning towards “restricted”, as they are neither open, nor proprietary.

Post reply on HN