What kind of GPU is needed to run the 65B models, can a consumer grade card do the job?
Is it worth it? Or do you use cloud instances?
The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
131–140 of 527 posts
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#132I'm a bit worried the LLaMA leak will make the labs much more cautious about who they distribute models to for future projects, closing down things even more. I've had tons of fun implementing LLaMA, learning and playing around with variations like Vicuna. I learned a lot and probably wouldn't have got so interested in this space if the leak didn't happen.
If the copyright office determines model weights are uncopyrightable (huge if), then one might imagine any institutional leak would benefit everyone else in the space. You might see hackers, employees, or contractors leaking models more frequently. And since models are distilled functionality (no microservices and databases to deploy), they're much easier to run than a constellation of cloud infrastructure.
Even if the weights are copyrighted, running one more epoch of fine-tuning will result in different weights. At a certain point, they'd have to copyright the shapes of the weight vectors.
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#133What kind of GPU is needed to run the 65B models, can a consumer grade card do the job? Is it worth it? Or do you use cloud instances?
It's possible to run with a cpu and then use system memory.
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#134It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?
> Why do these takes around open-source AI remain so popular? I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs. T…
How do you run it locally? llama.cpp + 64GB RAM + 4bit quantized?
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#135It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force?
> Why do these takes around open-source AI remain so popular? I can only speak for myself, but I have a great desire to run these things locally, without network and without anyone being able to shut me out of it and without a running cost except the energy needed for the computations. Putting powerful models behind walls of "political correctness" and money is not something that fits well with my personal beliefs. T…
How have you managed to run the 65B model? Cloud resources, or you have a very kitted-out homelab?
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#136Earlier quoted context omitted.
Sorry for asking a stupid question. How much Theseus do I need to ship before I can copyright it as my own? Is there some threshold for how much of an AI generated work needs to be modified by "human creativity" prior to it being copyrightable?
As far as I can tell, you can claim credit for the output of a tool which produces something at your direction. If you write an algorithm to generate patents, and you execute that algorithm and submit its output to USPTO under your own name, no problem. If you gave credit to the algorithm, their policy is to deny the claim.
Can OpenAI claim copyright on GPT’s outputs?
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#137I'm a bit worried the LLaMA leak will make the labs much more cautious about who they distribute models to for future projects, closing down things even more. I've had tons of fun implementing LLaMA, learning and playing around with variations like Vicuna. I learned a lot and probably wouldn't have got so interested in this space if the leak didn't happen.
Devil's Advocate: The EU comes down hard on any AI company that doesn't work with researchers and institutions in future.
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#138Earlier quoted context omitted.
GPT 4 will admit to not knowing things in many cases where 3.5turbo does not (tested the same prompt), and either will stop there or go off on a "but if it did exist it might go something like this" type continuation. It still hallucinates a lot, but it's not at all clear that this will be all that difficult an issue to solve given the progress.
We generally only hallucinate while dreaming / using our imagination. And we can distinguish those two states. Admitting lack of knowledge is of course good but, for example, if you ask it to write some code that isn’t boilerplate API integrations, it’ll do so happily even when it’s wildly wrong and it can’t tell the difference and that is also the case with GPT4 afaik. Moreover, you can’t solve it through prompt eng…
The point is there's been progress in making it admit when it doesn't know, and we simply don't know how fast that will improve in future version. It may continue to be an issue, or turn out to be easily solved. The improvement with GPT4 does suggest it is at least possible to make it recognise its limits.
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#139> OpenAI published a detailed blog post outlining some of the principles used to ensure safety in their models. The post emphasize in areas such as privacy, factual accuracy Am I the only one amused by the phrase “factual accuracy”? How many stories have we read like the one where it tries to ghost light the guy that this year is actually last year. “Oh, your phone must be wrong too, because there is no way I could b…
I hold a more charitable interpretation. We (the public) have found an important bug in the system, ie. GPT can lie (or "hallucinate"), even if you try to convince it not to lie. The bug is definitely lowering the usefulness of their product, as well as the public option about it. But I'll let the programmer who has never coded a bug cast the first stone. I wouldn't be surprised if they're scrambling internally to mi…
>as well as the public option about it.
assuming you mean the public opinion about it, and I would have to agree that I do hold chatGPT in very low regard because of this. people will argue that it is impressive that they built this thing that can be this impressive, but yeah, it might not be totally accurate. so my response is, what's the point of it then?
say someone was to invent a robot baseball player that can throw the ball 200+mph, or hit about a literal mile but, it can't hit a target with that 200+mph ball or hit the ball without it going foul. yes, the tech would be impressive, but what's the point? yeah yeah, language is hard, but it's just an example of building something impressive that at best almost does the job designed. unless we're saying it's designed this way on purpose??
Re: The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
#140What kind of GPU is needed to run the 65B models, can a consumer grade card do the job? Is it worth it? Or do you use cloud instances?
You can't fit a 65B even at 4bits (~50GB) on a consumer grade GPU. With an average geek's home desktop computer running 65B inference on a CPU with 64GB of system RAM you could expect about 1 token every 2 seconds.