61–70 of 85 posts
If you’re someone who wants to fine-tune Llama 2 on Google Colab, I have a couple live coding streams I did this past week where I fine tune Llama on my own dataset Here’s the stream - https://www.youtube.com/live/LitybCiLhSc?feature=share One is with LoRa and the other QLoRa I also do a breakdown on each fine-tuning method. I wanted to make these since I myself have had issues running LLMs locally and Colab is the c…
If you’re someone who wants to fine-tune Llama 2 on Google Colab, I have a couple live coding streams I did this past week where I fine tune Llama on my own dataset Here’s the stream - https://www.youtube.com/live/LitybCiLhSc?feature=share One is with LoRa and the other QLoRa I also do a breakdown on each fine-tuning method. I wanted to make these since I myself have had issues running LLMs locally and Colab is the c…
Cool stream! Subbed!
Earlier quoted context omitted.
> “The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago” Hmm. If LLMs turned out like JS frameworks, that would mean that in ten years people will be saying: “Maybe we don’t really need all this expensive ceremony, honestly this could be done with vanilla if/else heuristics…?”
I can imagine a bloated world where 500B param models are used for tasks where 7B param modes perform adequately. At that time, there could be complaints on hacker news about messaging apps with autocomplete models that take up gigabytes.
The main critiques outside of data privacy I've read are related to energy consumption, but even then, it's...not compelling? I read an article[0] that estimated the training of ChatGPT (3.5) to emit as much C02 as more than 3 round-trip flights between SF and NYC. That's not good! But also, really highlights that if we're to reduce emissions, there's clearly bigger targets than the largest ML models in the world.
[0]: https://themarkup.org/news/2023/07/06/ai-is-hurting-the-clim...
What's the cheapest way to run e.g. LLaMa2-13B and have it served as an API? I've tried Inference Endpoints and Replicate, but both would cost more than just using the OpenAI offering.
In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
Not sure about this. atm, the cost of any cloud GPU (spot or not) far exceeds the cost of OpenAI's API. I'd be glad to be proven wrong because I, too, want to run L2 (the 70b model).
Also, buying a GPU, even 4090, is not feasible for most people. And it's not just about GPU—you'd have to build a PC for it to work, and there's the hidden maintenance cost of running desktop Linux (to use GPTQ for instance). It's not surprising that most users prefer someone else (OpenAI) to do it for them.
Earlier quoted context omitted.
I can imagine a bloated world where 500B param models are used for tasks where 7B param modes perform adequately. At that time, there could be complaints on hacker news about messaging apps with autocomplete models that take up gigabytes.
I could see a world where people turn to an LLM for a task where the hand-rolled solution could be a simple state machine or some nested switch statements. The irony would be that the LLM could write you that code, but if you don’t know to ask…
In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
In my tests LLaMa2-13B is useable for information extraction tasks and LLaMA2-70B is almost as good as GPT-4 (for IE). These models are the real thing. We can fine-tune LLaMAs, unlike OpenAI's models. Now we can have privacy, control and lower prices. We can introduce guidance, KV caching and other tricks to improve the models. The enthusiasm around it reminds me of JavaScript framework wars of 10 years ago - tons of…
> ... and lower prices. Not sure about this. atm, the cost of any cloud GPU (spot or not) far exceeds the cost of OpenAI's API. I'd be glad to be proven wrong because I, too, want to run L2 (the 70b model). Also, buying a GPU, even 4090, is not feasible for most people. And it's not just about GPU—you'd have to build a PC for it to work, and there's the hidden maintenance cost of running desktop Linux (to use GPTQ fo…
Sure, you can run something comparable to OpenAI's flagship product at home, but it's moderately expensive and slightly inconvenient so people will still pay for the convenience.