Any good resources or suggestions for system/pre-prompt for general coding or when targeting a specific language? ie, using the CodeLlama and working on typescript, ruby, rust, elixir etc.. is there a universal prompt that gives good results or would you want to adjust the prompt depending on the language you're targeting?
Meta AI releases Code Llama 70B
281–290 of 311 posts
Re: Meta AI releases Code Llama 70B
#282Earlier quoted context omitted.
Disney didn't spend millions on a whim. It's just the reality of box office that even millions in investment are no guarantee for returns.
Financially, they have underperformed significantly over longer period of time (10 years): For shareholders, this subpar performance has destroyed value. Disney stock has underperformed the stocks of Disney’s self-selected proxy peers and the broader market over every relevant period during the last decade and during the tenure of each non-management director. Furthermore, it has underperformed since Bob Iger was fir…
Share price isn't the be all end all.
Disney has steamrolled Hollywood for the last decade, bringing in by far the biggest global box office revenue in 7 consecutive years out of 8. They have more billion dollar box office movies than every other studio co mbined. This kind of dominance was unheard of in the history of Hollywood.
Setting box office aside, Disney revenue has tripled since Iger took over and is twice as much as it should be adjusted for inflation.
The idea that the company has underperformed for the last 10 years or that they spend millions "on a whim" is a joke. And using share price as some justification is even more absurd, share price was double what it was today just in 2021.
Re: Meta AI releases Code Llama 70B
#283Earlier quoted context omitted.
P40 is essentially a faster 1080 with 24GB ram. For many tasks (including LLMs) it's easy to be memory bandwidth bottlenecked and if you are they are more evenly matched. (newer hardware has more bandwidth, sure but not in a cost proportional manner). I find that my hosts using 9x P40 do inference on 70b models MUCH MUCH faster than a e.g. a dual 7763 and cost a lot less. ... and can also support 200B parameter model…
Interesting. Technically, P40 is rated at an impressive 347.1GB/sec memory bandwidth, and 4060, at a slightly lower 272GB/sec. For bandwidth-limited workloads, the P40 still wins. The 4090 is about 3-4x that, but as you point out, is not cost-competitive. What do you use to fit 9x P40 cards in one machine, supply them with 2-3kW of power, and keep them cooled? Best I've found are older rackmount servers, and the ones…
I technically have 10 plus a 100GBE card in them, but due to nvidia chipset limitations using more than 9 in a single task is a pain. (Also, IIRC one of the slots is only 8x in my chassis)
Supermicro has made a couple 5U chassis that will take 10x double width cards and provide adequate power and cooling. SYS-521GE-TNRT is one such example (I'm not sure off the top of my head which mine are, they're not labeled on the chassis, but they may be that).
They pricey new but they show up on ebay for 1-2k. The last ones I bought I paid $1800, I think the earlier set I paid $1500/ea-- around the time that ethereum gpu mining ended ( I haven no clue why someone was using chassis like these for gpu mining, but I'm glad to have benefited! ).
Re: Meta AI releases Code Llama 70B
#284Earlier quoted context omitted.
I hope someone will soon post a quantized version that I can run on my macbook pro.
Ollama has released the quantized version. https://ollama.ai/library/codellama:70b https://x.com/ollama/status/1752034686615048367?s=20 Just need to run `ollama run codellama:70b` - pretty fast on macbook.
ollama run codellama:70b
pulling manifest
pulling 1436d66b6
1.1 GB/ 38 GB 24 MB/s 25m21s
Re: Meta AI releases Code Llama 70B
#285Credit where credit is due, Meta has had a fantastic commitment towards open source ML. You love to see it.
Redistributable and free to use weights does not make a model open source (even if it's really nice, with very few people having access to that kind of training power).
Re: Meta AI releases Code Llama 70B
#286Earlier quoted context omitted.
I hope someone will soon post a quantized version that I can run on my macbook pro.
Ollama has released the quantized version. https://ollama.ai/library/codellama:70b https://x.com/ollama/status/1752034686615048367?s=20 Just need to run `ollama run codellama:70b` - pretty fast on macbook.
Do you know how much vram is required?
Re: Meta AI releases Code Llama 70B
#287Earlier quoted context omitted.
Financially, they have underperformed significantly over longer period of time (10 years): For shareholders, this subpar performance has destroyed value. Disney stock has underperformed the stocks of Disney’s self-selected proxy peers and the broader market over every relevant period during the last decade and during the tenure of each non-management director. Furthermore, it has underperformed since Bob Iger was fir…
Share price isn't the be all end all. Disney has steamrolled Hollywood for the last decade, bringing in by far the biggest global box office revenue in 7 consecutive years out of 8. They have more billion dollar box office movies than every other studio co mbined. This kind of dominance was unheard of in the history of Hollywood. Setting box office aside, Disney revenue has tripled since Iger took over and is twice a…
> The idea that the company has underperformed for the last 10 years ... is a joke
Did you even read the Triad Partners quote from their letter? It's their words, not mine.
Re: Meta AI releases Code Llama 70B
#288Earlier quoted context omitted.
A 70B model is quite accessible; just rent a data center GPU hourly. There are easy deployment services that are getting better all the time. Smaller models can be derived from the big ones to run on a MacBook running Apple Silicon. While the compute won’t be a match for Nvidia hardware, a MacBook can pack 128GB of RAM and run enormous models - albeit slowly.
Ok, well now that we’ve downvoted me below the visibility threshold, I was being sincere. And Altman did say that. I am not a hater. So. Maybe we could help other people figure out why VRAM is maxing out. I think it has to do with various new platforms leaking memory. In my case, I suspect ollama and diffusers are not actually evicting VRAM. nvidia-smi shows it in one case, but I haven’t figured it out yet. Hey, my p…
Haha, for posterity, if anyone was interested in the things I mentioned. My specific frustrations came from leaving out torch.no_grad when running inference.
I used copilot to refactor that, and it just didn’t put no_grad back, and I did not notice.
I was uselessly recalculating all of my weights to /dev/null and waste heat.
I daemonized my process to control it more tightly, but I still see way above expected vram allocation.
Just keep in mind. It’s not like the models are going to get smaller, past the quantization limit. What, is there a quantization of retrievable information to 0 bits? ;)
Re: Meta AI releases Code Llama 70B
#289Earlier quoted context omitted.
Personally I was testing with TogetherAI because I don't have the specs for a local 70b. Using quantized versions helps (Ollama's downloads 4-bit by default, you can get down to 2), but it would still require a higher-end Mac. Highly recommend Together, it runs quite quickly and is $0.9/million tokens
are there any docs on setting up togetherAI with continue.dev? would be interested in checking that out as an alternative to OpenAI for experimenting with larger models that won't run/run well on a m1 max.
Definitely, here is a brief reference page for the Together provider: https://continue.dev/docs/reference/Model%20Providers/togeth..., and a higher-level explanation of model configuration here: https://continue.dev/docs/model-setup/overview
Re: Meta AI releases Code Llama 70B
#290Earlier quoted context omitted.
Share price isn't the be all end all. Disney has steamrolled Hollywood for the last decade, bringing in by far the biggest global box office revenue in 7 consecutive years out of 8. They have more billion dollar box office movies than every other studio co mbined. This kind of dominance was unheard of in the history of Hollywood. Setting box office aside, Disney revenue has tripled since Iger took over and is twice a…
> The idea that the company has underperformed for the last 10 years ... is a joke Did you even read the Triad Partners quote from their letter? It's their words, not mine.
There is no quote that says that.
"Earnings per share (“EPS”) in the most recent fiscal year were lower than the EPS generated by Disney a decade ago"
is not the same as "underpeforming for a decade".
all that says is that EPS is currently low, not that it has been low and reducing/stagnant for a decade.