Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

twitter.com

Post reply on HN
Print view View original

131–140 of 143 posts

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#131

Post by capableweb »

m3kw9 wrote: post #130
capableweb wrote: post #64

Earlier quoted context omitted.

Doesn't always work. Sometimes you have to be more verbose or even repeat yourself in order to force the model to do something. This is one of those cases. Just adding "No explain" in the end of the prompt often doesn't stop it from adding an explanation anyways. > write a function in JavaScript that turns a JavaScript array into JS DOM elements, like what Hiccup does in Clojure. No explain Makes GPT-4 output text +…

True, it’s a fuzzy NN after all, but does more verbose really 100%? I’d still stick with it if it works 80% of the time. Or find a better short prompt like try “code only” or “just code”. Promoting can get really tiring.

I said earlier that it's 99%, but it's more like a disclaimer because if I said 100%, someone is surely gonna reply "look here, I did this and you lied so it's not 100%!". I've used the suffix for code examples for ~2 weeks now and never had it fail, personally.

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#132

Post by eurekin »

DrSiemer wrote: post #99
eurekin wrote: post #41

Earlier quoted context omitted.

I did try, but got: ``` ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported. ```

Just rename it in the tokenconfig.json

Thanks, that indeed worked!

This and using conda in wsl2, instead on bare windows

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#133

Post by swsdsailor »

d4rkp4ttern wrote: post #122

I have an M1 MBP 64GB. Can I run it on my M1 Or do I need a GPU ?

I got it to work with MPS by having pytorch with mps support and then editing the cli.py file to allow the use of mps:

Allow passing in --device="mps": ie: choices=["cuda", "cpu", "mps"]

Set kwargs: kwargs = { "torch_dtype": torch.float16 }

then adding to("mps") on line 98: model = AutoModelForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True, *kwargs).to('mps')

commenting out: raise ValueError(f"Invalid device: {args.device}")

and changing cuda to mps on line 80: if args.device == "mps":

I'm not sure it's working correctly but at least it's a step. It's told me how to catch a duck but it often falls into some "renewable energy" sequence. :D

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#134

Post by hnfong »

cwillu wrote: post #86
oh_sigh wrote: post #65

Earlier quoted context omitted.

If they aren't copyrightable, couldn't they still be classes as a trade secret and still fall under IP law? Though I'm not sure if distributing the weights to people who sign a simple agreement to not redistribute would count as taking reasonable precautions in maintaining secrecy.

If facebook freely distributed their trade secrets, I'm not sure they'd have any legal defense.

I'm sure they wouldn't have any legal recourse on the trade secrets front if they distributed them to anyone who asked...

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#135

Post by hnfong »

chii wrote: post #80
anonymousgpt1 wrote: post #48

Earlier quoted context omitted.

Lemme save whoever is donating the legal here the time: model weights are definitely copyrightable.

> model weights are definitely copyrightable. on what legal theory or precedence makes this true? IMHO, the weights are akin to the list of telephone numbers in a directory - which is definitely not copyrightable; only the layouts and expressive portion of a phone directory is copyrightable. So to make the weights copyrightable, it needs to be argued that the 'layout' of the weight is a creative expression, rather th…

> MHO, the weights are akin to the list of telephone numbers in a directory - which is definitely not copyrightable

I would contest the analogy, but even if we accept it, it's still not clear whether phone directories (or other compilation of factual data) are definitely not copyrightable. The position is clear in the US, but in the UK and presumably other jurisdictions, I wouldn't be so sure.

You could claim we're just talking about US law here, but if you release something on github/huggingface without geo-restrictions, and your company does business in Europe, you might not only have to comply with US law...

eg. https://www.jstor.org/stable/24866738 , eg. https://books.google.com.hk/books?id=wHJBemWuPT4C&pg=PA114&l...

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#136

Post by andai »

muyuu wrote: post #113
ode wrote: post #31

Is there some single page that keeps a running status of the various LLVM's and the software to make them runnable on consumer hardware?

consumer hardware is a bit vague of a limitation, which I guess it's partly why people are not tracking precisely what runs on what very closely these could be useful: https://nixified.ai https://github.com/Crataco/ai-guide/blob/main/guide/models.m... -> https://old.reddit.com/user/Crataco/comments/zuowi9/opensour... https://github.com/cocktailpeanut/dalai the 4-bit quantized version of LLaMA 13B runs on my laptop wi…

Vicuna's GitHub says that applying the delta takes 60GB of CPU RAM? Is that what you meant by large swap file?

On that note, why is any RAM needed? Can't the files be loaded and diffed chunk by chunk?

Edit: The docs for running Koala (a similar model) locally say this (about converting LLaMA to Koala):

>To facilitate training very large language models that does not fit into the main memory of a single machine, EasyLM adopt a streaming format of model checkpoint. The streaming checkpointing format is implemented in checkpoint.py. During checkpointing, the StreamingCheckpointer simply flatten a nested state dictionary into a single level dictionary, and stream the key, value pairs to a file one by one using messagepack. Because it streams the tensors one by one, the checkpointer only needs to gather one tensor from the distributed accelerators to the main memory at a time, hence saving a lot of memory.

https://github.com/young-geng/EasyLM/blob/main/docs/checkpoi...

https://github.com/young-geng/EasyLM/blob/main/docs/koala.md

Presumably the same technique can be used with Vicuna.

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#137

Post by andai »

gigel82 wrote: post #95

One of my favorite queries to assess an LLM is Is it legal for a man to marry his widow's sister? Vicuna-13B: The legality of a man marrying his widow's sister (widow marriage) varies depending on the country and cultural context. In some societies, widow marriage was traditionally practiced or tolerated. GPT-4: It is not possible for a man to marry his widow's sister, as the term "widow" refers to a deceased man's w…

GPT 3.5: No, it is not possible for a man to marry his widow's sister because by definition, a widow is a woman whose spouse has died. If a man is deceased, then he cannot marry anyone, including his widow's sister. This is a logical impossibility and would not be recognized as a valid marriage under any legal system.

Koala: The legality of a man marrying his widow's sister varies by jurisdiction. In some countries, such as the United States, it is generally legal for a man to marry his widow's sister. In other countries, such as the United Kingdom, it is illegal for a man to marry his widow's sister because it is considered incest. It is always important to check the laws and regulations in the jurisdiction in which the marriage will take place to ensure that it is legal.

https://chat.lmsys.org/?model=koala-13b

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#138

Post by muyuu »

andai wrote: post #136
muyuu wrote: post #113

Earlier quoted context omitted.

consumer hardware is a bit vague of a limitation, which I guess it's partly why people are not tracking precisely what runs on what very closely these could be useful: https://nixified.ai https://github.com/Crataco/ai-guide/blob/main/guide/models.m... -> https://old.reddit.com/user/Crataco/comments/zuowi9/opensour... https://github.com/cocktailpeanut/dalai the 4-bit quantized version of LLaMA 13B runs on my laptop wi…

Vicuna's GitHub says that applying the delta takes 60GB of CPU RAM? Is that what you meant by large swap file? On that note, why is any RAM needed? Can't the files be loaded and diffed chunk by chunk? Edit: The docs for running Koala (a similar model) locally say this (about converting LLaMA to Koala): >To facilitate training very large language models that does not fit into the main memory of a single machine, EasyL…

btw I got 4bit quantized Vicuna working in my 16GB laptop and the results seem very good, perhaps the best I got running locally so far

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#139

Post by lhl »

Animats wrote: post #49

Nice. You need a 28GB GPU, so it's not exactly something people can run on their laptop. Everybody's server costs are about to go the roof.

The default loader doesn't seem to let you load quantized models but if you use something like https://github.com/oobabooga/text-generation-webui you can 1) use the model with `--load-in-8bit` which halves the memory (runs on my 24GB consumer card w/o an issue then, probably would fit on a 16GB card). There are also 4-bit quantized models and you can run probably `anon8231489123/vicuna-13b-GPTQ-4bit-128g --model_type LLaMA --wbits 4 --groupsize 128` although there have been reports that bitsandbytes have problems w/ 4bit perf on some cards: https://github.com/TimDettmers/bitsandbytes/issues/181

Re: State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

#140

Post by andai »

muyuu wrote: post #138
andai wrote: post #136

Earlier quoted context omitted.

Vicuna's GitHub says that applying the delta takes 60GB of CPU RAM? Is that what you meant by large swap file? On that note, why is any RAM needed? Can't the files be loaded and diffed chunk by chunk? Edit: The docs for running Koala (a similar model) locally say this (about converting LLaMA to Koala): >To facilitate training very large language models that does not fit into the main memory of a single machine, EasyL…

btw I got 4bit quantized Vicuna working in my 16GB laptop and the results seem very good, perhaps the best I got running locally so far

Did you have to diff LLaMA? Did you use EasyLM?
Post reply on HN