Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

github.com

Post reply on HN
Print view View original

71–80 of 314 posts

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#71

Post by refulgentis »

tmountain wrote: post #61
2bitencryption wrote: post #44

Having spent quite a bit of time playing around with llama.cpp, alpaca.cpp, loras, and the many other llama-based weights lately, here is my impression: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far. [0] If yo…

This sentence defies lay people: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far.

A Lora is a layer on top of a model, the big deal isn’t that this exists (it’s a Lora for the weakest llama), but the fact they shared their dataset. The stronger llamas trained with this data will produce even better Lora’s and better results.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#72

Post by drexlspivey »

hyperopt wrote: post #18

Does anyone know of any good test suites we can use to benchmark these local models? It would be really interesting to compare all the ones capable of running on consumer hardware so that users can easily choose the best ones to use. Currently, I'm a bit unsure how this compares to the Alpaca model released a few weeks ago.

https://github.com/openai/evals

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#73

Post by sp332 »

m3kw9 wrote: post #51
2bitencryption wrote: post #44

Having spent quite a bit of time playing around with llama.cpp, alpaca.cpp, loras, and the many other llama-based weights lately, here is my impression: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far. [0] If yo…

How does training on just 800k pieces of data need 7b parameters?

Llama 7B was trained on a trillion tokens. The Lora is a small fraction of extra neurons that get integrated into the structure, and those are what get trained on the new data. It's like fine-tuning but takes less RAM and compute than retraining the whole model.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#74

Post by politician »

yieldcrv wrote: post #11

is "number of parameters" going to become a useless metric? or would this be that much better when using a fine-turned version of the 65B parameter Llama model.

Emergent features still require many parameters to stabilize, so there is an incentive to keep experimenting. There’s a blog called “Emergent Features and Int(8)” or close enough that explains the concept.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#75

Post by aunty_helen »

alchemist1e9 wrote: post #60
2bitencryption wrote: post #44

Having spent quite a bit of time playing around with llama.cpp, alpaca.cpp, loras, and the many other llama-based weights lately, here is my impression: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far. [0] If yo…

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet. Can this potentially do that? Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a…

We're a long long long way off that. So check back in two months.

Jokes aside, the limiting factor will be either a technique to pack all of the code into smaller tokens like semantic search (someone else will be able to comment on this as that's the limit of my understanding) or GPU memory for input tokens.

Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#76

Post by alchemist1e9 »

2bitencryption wrote: post #66
alchemist1e9 wrote: post #60

Earlier quoted context omitted.

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet. Can this potentially do that? Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a…

Here's my experience, having used llama+lora 7b, 13b, and 30b, on both cpu and gpu: On gpu, processing the input prompt, even for huge prompts, is almost instant. Meaning, even if your prompt is huge, it will start generating new tokens after your prompt very quickly. On a rented A6000 gpu, using llama+lora 30b, you can use huge prompts and it will start giving a new output right away. On cpu (i.e. the project llama.…

That sounds promising but how about the quality of the output.

I’ve been using OpenAI API with chatblade and been giving it code in various languages and it’s quite surprising how well it describes the purpose and code implementation in english. The english description would be useful and relevant for developers trying to quickly familiarize themselves with a code base.

For a typical 400-600 line file it looks like it would cost around 1-3 cents per file. However loss of privacy isn’t so great.

How do you find output quality of llama + lora for such a task?

Code is research code I’d run it on.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#77

Post by axlee »

refulgentis wrote: post #71
tmountain wrote: post #61

Earlier quoted context omitted.

This sentence defies lay people: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far.

A Lora is a layer on top of a model, the big deal isn’t that this exists (it’s a Lora for the weakest llama), but the fact they shared their dataset. The stronger llamas trained with this data will produce even better Lora’s and better results.

What is a lora or llama? Google gives me nothing.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#79

Post by alchemist1e9 »

aunty_helen wrote: post #75
alchemist1e9 wrote: post #60

Earlier quoted context omitted.

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet. Can this potentially do that? Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a…

We're a long long long way off that. So check back in two months. Jokes aside, the limiting factor will be either a technique to pack all of the code into smaller tokens like semantic search (someone else will be able to comment on this as that's the limit of my understanding) or GPU memory for input tokens. Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentr…

Are you sure?

This is for inference, not training, and I’d do it on blocks of code with less than 4096 tokens. That’s what I have to do with gpt 3.5 turbo via api.

I can do what I want with the API and it is around 1-2 cents per batch of code I run through it, typically 400 lines of average code is around 4k tokens.

Post reply on HN