Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alpaca: A strong open-source instruction-following model

crfm.stanford.edu

Post reply on HN
Print view View original

301–310 of 313 posts

Re: Alpaca: A strong open-source instruction-following model

#301

Post by counttheforks »

dmw_ng wrote: post #295
pradn wrote: post #294

Earlier quoted context omitted.

No, the crucial thing is that a good human will tell you if they don't know something, or if they are simply unsure.

It's adorable seeing this kind of critique in the context of HN, I wonder how many folk knew my heavily upvoted comment above ("For 10 billion+ ...") was from someone who has only been looking at this stuff for a few weeks. ChatGPT is no better or worse than any consultant I've ever met (including myself), or most of the commenters you find here every single day.

It's adorable that you think people assume HN comments are factually correct. I read everything here with extreme skepticism, because I know this is all coming from flawed humans. An computer system giving authoritative text and insisting it is 100% correct is a different story.

Re: Alpaca: A strong open-source instruction-following model

#302

Post by aortega »

kkielhofner wrote: post #296
zitterbewegung wrote: post #290

Earlier quoted context omitted.

How do you make it load on two GPUs or does llama.cpp does it automatically? I have a setup with a threadrippper and a RTX3090 and a Titan RTX. I haven't had the time to set it up so that's why I have been using my Mac.

llama.cpp doesn't use the GPU at all. The genius *.cpp (whisper.cpp, llama.cpp) projects are specifically intended to optimize/democratize otherwise GPU only models to run on CPU/non-GPU (CUDA, ROCm). Technically speaking the released models are capable of running on GPU via standard framework (PyTorch, TensorFlow) support for CPU but in practice without a lot of optimization they are incredibly slow to the point of…

Llama.cpp takes advantage that LLaMa 7B is a tiny, very optimized model. It would run in anything, and very fast. I really doubt you can run the 30B or 65B models at acceptable speed on a CPU at least for a couple years. (I'm ready to eat my words in a couple weeks)

Re: Alpaca: A strong open-source instruction-following model

#303

Post by dragonwriter »

counttheforks wrote: post #301
dmw_ng wrote: post #295

Earlier quoted context omitted.

It's adorable seeing this kind of critique in the context of HN, I wonder how many folk knew my heavily upvoted comment above ("For 10 billion+ ...") was from someone who has only been looking at this stuff for a few weeks. ChatGPT is no better or worse than any consultant I've ever met (including myself), or most of the commenters you find here every single day.

It's adorable that you think people assume HN comments are factually correct. I read everything here with extreme skepticism, because I know this is all coming from flawed humans. An computer system giving authoritative text and insisting it is 100% correct is a different story.

Every bit of text from a computer system also comes from flawed humans.

Re: Alpaca: A strong open-source instruction-following model

#304

Post by zitterbewegung »

kkielhofner wrote: post #296
zitterbewegung wrote: post #290

Earlier quoted context omitted.

How do you make it load on two GPUs or does llama.cpp does it automatically? I have a setup with a threadrippper and a RTX3090 and a Titan RTX. I haven't had the time to set it up so that's why I have been using my Mac.

llama.cpp doesn't use the GPU at all. The genius *.cpp (whisper.cpp, llama.cpp) projects are specifically intended to optimize/democratize otherwise GPU only models to run on CPU/non-GPU (CUDA, ROCm). Technically speaking the released models are capable of running on GPU via standard framework (PyTorch, TensorFlow) support for CPU but in practice without a lot of optimization they are incredibly slow to the point of…

Okay my thread ripper can handle it because it has a 128GB of Ram.

Re: Alpaca: A strong open-source instruction-following model

#305

Post by MacsHeadroom »

tysam_and wrote: post #268
MacsHeadroom wrote: post #243

Earlier quoted context omitted.

Empirically, 13B with quantization. In fact the person who said 4bit is worse is empirically incorrect. 13B with quantization even down to 3-bits has very near the same performance as uncompressed 16bit 13B with GPTQ quantization and binning. Source: https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i...

I looked at the numbers you posted, and am feeling concerned with how aggressively you're commenting towards a number of people on this website. For starters, I started in this field a few years after the 2012 wave started. I've been with it for a while and have seen a lot of trends come and go. One thing that stays the same is that things are always changing. Very few things are set in stone, and due to a few other…

Many good points. I agree with essentially everything you've said, especially regarding relative perplexity.

I'm aware that I was aggressively overselling an unnuanced and overstated position on 4-bit and especially 3-bit performance. That was partially a rhetorical tactic to swing the pendulum the other way, as it were.

And partially it was simply frustration with the number of threads I've seen in the past week of LLaMA drama spreading misinformation about bit precision like "a 16bit 13B model surely outperforms a 4-bit 30B model" which could not be further from the truth. That frustration is my own responsibility to manage and I understand that.

Re: Alpaca: A strong open-source instruction-following model

#306

Post by z3t4 »

uh_uh wrote: post #180
z3t4 wrote: post #91

Earlier quoted context omitted.

Training a model to do math is impossible. If you tell a child that 1+1=2 10+10=20 100+100=200 an "AI" can not figure out that 1000+1000=2000 like a child can.

Has this actually been tried? It would be trivial to generate a very large dataset of simple additions and see if an LLM gets it. Maybe it wouldn't serve an immediate practical purpose but I'm curious.

I did try this, although I'm not an AI expert. It will work great for everything that are in the training set, but if there is a hole the model will have trouble. Lets say we train a model with the numbers 1 and 2. 1+1=2, 1+2=3, 2+2=4 we now give the model 2+1 it will likely guess 2 or 4 because those are the ones that have 2 or 1 in the correct place. A neural network can't do logic. It's good at other things though, for example parsing, a language model could parse human input "two plus one" to 2+1 and then just calculate the answer using a math library.

Re: Alpaca: A strong open-source instruction-following model

#307

Post by nl »

valine wrote: post #254
nl wrote: post #245

Earlier quoted context omitted.

Yeah amazingly fast. I haven't looked at the quality but some comments in the discord seemed to indicate it was decent quality but had a lot of coding-type prompts.

I don’t see any problem with an abundance of coding prompts. The type of person who runs a local LLM is probably using it for coding anyway.

Some will but a quick search for "GPT" on Twitter will show the huge diversity of things people are making LLMs do.

Re: Alpaca: A strong open-source instruction-following model

#308

Post by amoss »

sdenton4 wrote: post #300
amoss wrote: post #256

Earlier quoted context omitted.

Those protections rely on "systematic arrangement", i.e. that some effort has been put into finding, verifying and collection a set of facts / individual pieces of data. It is not obvious that would apply to the set of weights that come out of a training algorithm. (standard I'm not a lawyer, I don't know but I suspect it is unclear enough that it would need to be argued in court).

I assure you that if you scramble the weights of a neutral networks, the model will perform poorly... Training something new cost millions of dollars in accelerator and research salary, and evaluation ('verifying') is a big part of the work.

This is true, but not currently relevant. The cost (difficulty) of training does not indicate how good a fit the model is for copyright protection.

I suspect that part of the arguing this out in court will be making those economic arguments as part of an attempt to extend copyright protection to model weights - but that is a separate issue from if they are protected currently.

Part of the economic argument for copyright is to encourage the investment of time into producing new works, so it may be a persuasive argument for extending protection. Although (relatively) recent history suggests that persuasive argument is not as powerful as giant wads of cold hard cash in extending copyright coverage.

Re: Alpaca: A strong open-source instruction-following model

#310

Post by all2 »

meghan_rain wrote: post #234
all2 wrote: post #129

Earlier quoted context omitted.

The timing of the Facebook leak seems suspect.

What do you mean?

I mean ChatGPT had a lot of attention, so a leak of a competing architecture would shift the attention away from ChatGPT. Which Meta's LLaMA did. And we see it swinging in the other direction with OpenAI announcing GPT-4.
Post reply on HN