Can someone point me to a ELI5 sequence of steps that shows how someone can install and use LLMs locally and in some way, functionally?
Asking for purposes of educating non-technologists.
Code Llama, a state-of-the-art large language model for coding
211–220 of 525 posts
Re: Code Llama, a state-of-the-art large language model for coding
#212As a complete noob at actually running these models, what kind of hardware are we talking here? Couldn't pick that up from the README. I absolutely love the idea of using one of these models without having to upload my source code to a tech giant.
I've used Ollama to run Llama 2 (all variants) on my 2020 Intel MacBook Pro - it's incredibly easy. You just install the app and run a couple of shell commands. I'm guessing soon-ish this model will be available too and then you'd be able to use it with the Continue VS Code extension. Edited to add: Though somewhat slow, swap seems to have been a good enough replacement for not having the loads of RAM required. Ollam…
Apple Silicon, especially an M1 Max Studio seems to be an interesting machine to hang on to as the models become more and more efficient with using less and less.
If there's nay other opinions or thoughts on this, I'd be very happy to learn as well. I have considered the eGPU route connected to a 1L PC such as a thinkcentre m80/90.
Re: Code Llama, a state-of-the-art large language model for coding
#213Earlier quoted context omitted.
They are behind commercially, very behind. They also don't have the same economic setup and DNA as MS/OpenAI. Large corporate customers don't pay for access to the FB cloud, nor are they likely to -- Ellison has spent years building out Oracle Cloud, and he's on the FB board, for example. And I bet you didn't think of using Oracle's Cloud for your last project. So, your company DNA is free-to-all social based on ad m…
In times like these Facebook/Zuck probably wonders how things would have turned out had they not killed Parse. Had they continued with it, they'd have likely had some semblance of a public cloud today and would be able to sell these models.
Yes. But it also needs a very different org structure to support that. Their internal infra from what I heard is dated (monolithic PHP binary deployment, no federated authorization management etc.). It is doable (FAIR's org structure was very different in the first a few years), but would also be a distraction for a long time.
Very interesting to ponder for sure.
Re: Code Llama, a state-of-the-art large language model for coding
#214Earlier quoted context omitted.
We have three options, IMO: 1. As a species decide to never build another LLM, ever. 2. Change the path of society from the unequal, capitalist one it’s taken the last 2-300 years. 3. Give up I know which I believe in :). Do you disagree?
The problem with 2 is that the people in power, and with immense wealth, remain there because of capitalism. They have the political power, and the resources, to enact the change ... but they also lose the most (unless you count altruism as gain, which of it were true the World would be so different). We could structure things so that LLM, and the generalised AIs to come, benefit the whole of society ... but we know…
Yes but the common man has won before! There has never been a perfect revolution/paradigm shift (personally anti utopia-through-intense-bloodshed, so hesitant to use the former term alone), but there have been many, all of which were against the wishes of those in power.
Plus, if these AIs are enough to change everything, that kinda implies that we've developed flexible, reliable AGI systems. In such a world, everything changes - maybe the calculus of The Powerful Few vs. The Oppressed Masses changes in too! It might even change in our favor, if we're terribly lucky...
Re: Code Llama, a state-of-the-art large language model for coding
#215The highlight IMO > The Code Llama models provide stable generations with up to 100,000 tokens of context. All models are trained on sequences of 16,000 tokens and show improvements on inputs with up to 100,000 tokens. Edit: Reading the paper, key retrieval accuracy really deteriorates after 16k tokens, so it remains to be seen how useful the 100k context is.
> The Code Llama models provide stable generations with up to 100,000 tokens of context.
what is the trick to achieve 100k context? They can't just use 100k wide transformer layer, it is cost prohibitive, right?..
Re: Code Llama, a state-of-the-art large language model for coding
#216Amazing! It's great that Meta is making AI progress. In the meantime, we are still waiting for Google to show what they have (according to their research papers, they are beating others). > User: Write a loop in Python that displays the top 10 prime numbers. > Bard: Sorry I am just an AI, I can't help you with coding. > User: How to ask confirmation before deleting a file ? > Bard: To ask confirmation before deleting…
I don't get comments like this, we can all go and test Bard and see that what you're saying isn't true https://g.co/bard/share/95761dd6d45e
Well look for yourself:
https://g.co/bard/share/e8d14854ccab
The rm answer is now "hardcoded" (aka, manually entered by reviewers), the same with the prime or fibonnaci.
This is why we both see the same code across different accounts (you can make the test if you are curious).
Re: Code Llama, a state-of-the-art large language model for coding
#217Re: Code Llama, a state-of-the-art large language model for coding
#218TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090. [1] https://huggingface.co/TheBloke/CodeLlama-13B-Python-fp16
What kind of cpu/gpu power do you need for quantization or these new gguf formats ?
I haven’t quantized these myself since TheBloke has been the main provider for all the quantized models. But when I did a 8 bit quantization to see how it compares to the transformers library load_in_8bit 4 months ago(?), it didn’t use my GPU but loaded each shard into the RAM during the conversion. I had an old 4C/8T CPU and the conversion took like 30 mins for a 13B.
Re: Code Llama, a state-of-the-art large language model for coding
#219Earlier quoted context omitted.
They are behind commercially, very behind. They also don't have the same economic setup and DNA as MS/OpenAI. Large corporate customers don't pay for access to the FB cloud, nor are they likely to -- Ellison has spent years building out Oracle Cloud, and he's on the FB board, for example. And I bet you didn't think of using Oracle's Cloud for your last project. So, your company DNA is free-to-all social based on ad m…
I would add that having open source gen AI will enable the creation of content for metaverse / AR / VR, which will improve the chances that all of that will take off.
Right, exactly this. Ratcheting the costs down two orders of magnitude in both dollar and expertise/human costs is going to make huge changes. You better believe FB is thinking about this hard.
Re: Code Llama, a state-of-the-art large language model for coding
#220Can someone point me to a ELI5 sequence of steps that shows how someone can install and use LLMs locally and in some way, functionally? Asking for purposes of educating non-technologists.
There are several different ways, but the easiest way in my (clearly biased) opinion is just got to ollama.ai, download it, and start playing around. It works out of the box w/ newer Macs, but there are versions for Linux and Windows in the works.