Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

github.com

Post reply on HN
Print view View original

301–310 of 314 posts

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#301

Post by stolsvik »

2bitencryption wrote: post #66
alchemist1e9 wrote: post #60

Earlier quoted context omitted.

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet. Can this potentially do that? Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a…

Here's my experience, having used llama+lora 7b, 13b, and 30b, on both cpu and gpu: On gpu, processing the input prompt, even for huge prompts, is almost instant. Meaning, even if your prompt is huge, it will start generating new tokens after your prompt very quickly. On a rented A6000 gpu, using llama+lora 30b, you can use huge prompts and it will start giving a new output right away. On cpu (i.e. the project llama.…

As I understand these models, that makes no sense whatsoever. Is it the tokenisation that takes this crazy amount of time? Because each additional token should take exactly the same amount of time as the first.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#302

Post by actionfromafar »

rablackburn wrote: post #187
keyle wrote: post #181

Earlier quoted context omitted.

I feel you! Did you miss the web 3 wave though? I still can't imagine what that was about.

Web 3 seemed silly enough that I didn’t bother really following it. I know that there is probably some very good work going on around blockchain stuff, but NFTs ain’t it. LLM assistants is genuinely just moving _very_ fast, so if you don’t pay attention every day you just miss things. I’m just enjoying having my interactive rubber-duck tbh

I think NFTs may really be on to something, but probably mundane little things like tickets to concerts, not eye-wateringly expensive collectible monkeys.

Like how SMS is now a thing used for all sorts of little stuff, but nobody thinks much about it.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#303

Post by anticensor »

astrange wrote: post #200
sam0x17 wrote: post #30

Earlier quoted context omitted.

According to the US Copyright office's recent ruling, nothing generated by these models can be copyrighted (!), so in theory you should be fine!

They're the first word on this, not the last, so it doesn't mean much. It also clearly leads to absurd results for text models, which can memorize their training data, or echo back your inputs. Or you could pass their output through a human to copyright it.

Not all countries have the same view on this either. A few countries, including the UK, maintain the view that the AI's operator, owner or custodian hold the copyright instead.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#304

Post by sswam »

rlevy wrote: post #274
kolinko wrote: post #223

Earlier quoted context omitted.

I had the same issue, and I just caught up over the weekend. Three books I can recommend to get up to speed: - NumPy basics pdf - first 2-3 chapters - Deep Learning with PyTorch by Voight Godoy [2] - first 2-3 chapters if you had experience with neural networks, or the whole of it if you didn't. With the above, you will get the basics to understand this book about transformers, and the architecture of the models, and…

What’s the third book?

[dead]

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#305

Post by rohfle »

mightytravels wrote: post #54

I feel bad - now someone has to pay 9 cents * 4GB in S3 costs when I downloaded the weights. Cloudflare R2 has free bandwidth and is much faster (my current speed for any S3 download is just 10MBit despite 300MBit connection) - way to few devs use R2.

$1985.74 after one day https://cdn.discordapp.com/attachments/1090427154141020190/1...

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#306

Post by Tepix »

SilentM68 wrote: post #292
Tepix wrote: post #171

Earlier quoted context omitted.

Yes, i haven't seen any fine-tuned LLaMA-65B model so far unfortunately. I guess the cost is a bit high. Perhaps with LoRa someone will do it.

Might want to check out these guys as well: https://cocktailpeanut.github.io/dalai/#/

That's for CPU (and they only provide 7B and 13B).

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#307

Post by alchemist1e9 »

stolsvik wrote: post #300
alchemist1e9 wrote: post #60

Earlier quoted context omitted.

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet. Can this potentially do that? Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a…

Care to share what prompts you run this with? What is the output you desire, and how do you achieve this?

Prompt: You are an expert python programmer. I will give you some python code below. I want you to describe the code and it’s likely purpose.

The answer will be pretty damn good , especially with gpt 4, but it’s still good with gpt 3.5 turbo. Try it out yourself with whatever code you have.

This english description of code can then be searched against. That’s the primary practical application I’m thinking about doing currently.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#308

Post by f_devd »

jhbadger wrote: post #248
tmountain wrote: post #61

Earlier quoted context omitted.

This sentence defies lay people: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far.

As someone who is following this technology while not really an expert (I'm a computational biologist in my day job) LoRA is a way of reducing the number of parameters in a large language model (LLM, the technology behind all these new chatbots) so that it can be run on less powerful hardware (say a laptop or even a phone). The OP is saying that the improvements this chatbot provides isn't so much being more clever a…

LoRA (Low-Rank Adapter) is way to customize/finetune the LLM to a new datasets without needing to retrain the entire network (which makes it better (and in theory easier to do). It doesn't not change the speed significantly afaik

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#309

Post by nl »

jsnell wrote: post #296
nl wrote: post #290

Earlier quoted context omitted.

> For them to make the argument in court that database rights over the database of training data mean they have rights over the model too, they'd need to argue that the model is a derivative work training data. And then it'd mean their model is also a derived work from all the billions of works they scraped to get that data set. It would destroy the business model of the OpenAIs of the world, there is no chance they…

If the uncreative and automated work of training the model is transformative enough to impact the rights of the original content creators, it would also be transformative enough to impact the rights of the database curator. The fair use case is much harder to make here than for search engines since the model will be directly competing with the content creators. And again, how could e.g. OpenAI simultaneously claim th…

I don't necessarily disagree, but it's very unclear what a court would find.

I suggest https://arxiv.org/abs/2303.15715 for a complete overview.

Re: Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

#310

Post by all2 »

nullsense wrote: post #217
tetete wrote: post #124

Earlier quoted context omitted.

With 8 bit training you can do ~13B pram LLM on 3090/4090. https://huggingface.co/blog/trl-peft But it is pretty cheap to rent something at vast.ai or whatever to get 40GB for a final run.

Awesome. Between crypto hype in 2017 and AI hype in 2023 I've acquired a collection of 2x 1080ti, an RTX 3060 and an RTX 4090. All together it's a total of 58GB of VRAM. Is there a way I can pool it all across a distributed cluster of 2 machines for doing anything? I'm assuming it would bottleneck on both network speeds and the slowest GPUs if it's possible at all...

Could you package all that compute into a "virtual" graphics card?
Post reply on HN