Hypothetically speaking, if Falcon 40B could out-perform GPT-3.5, would that force OpenAI to open-source GPT-3.5?
Falcon 40B LLM (which beats Llama) now Apache 2.0
91–100 of 143 posts
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#92Hypothetically speaking, if Falcon 40B could out-perform GPT-3.5, would that force OpenAI to open-source GPT-3.5?
It would as open source improvements would start to exceed performance of 3.5 for specific use-cases. At the very least they would have to make it fine-tunable.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#9340B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
I briefly looked at prices a few days ago, I think you can rent an 80gb GPU for about $2.50 an hour. Then you just pay while you're using it. Someone else might be better able to confirm the pricing, but in any case you don't need to purchase the hardware.
I'm less interested in using someone else's computer (not as much, but similar to how i'm disinterested in an API from someone like OpenAI), would rather pay the upfront hardware cost than worry about how many tokens i generate (kind of hinders creativity and excitment about it).
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#9440B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
Some people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000
Woah, that's a cool direction. Thank you! I'll explore this.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#95Earlier quoted context omitted.
I think they are talking about VRAM
Anything that runs on GPU can be run on CPU, the only question is how much slower it's gonna be.
I've been running vicunda-13B on a workstation that can be aquired from ebay for about $500. It's slow compared to online services, but probably slightly faster than text to speech would recite its output, so plenty.
Falcon 40B is probably too much for it, but apparently there's similar cheap hardware that could work.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#9640B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
Stupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole damn model into GPU RAM at once and then complaining when it can't, instead of swapping portions of the model to CPU RAM, or hell, even SSD? Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"
That’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow.
For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#97Oh wow. That’s great news.
Falcon seemed good till I read the license fine print about pre approvals and what not. This seems to fix that
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#98Earlier quoted context omitted.
> Look at Stable Diffusion 1.5 and LLaMA: They are thriving, but the original implementations are ancient history, and Meta/StabilityAI/RunawayML have done precisely nothing. I mean, that’s true of SD 1.5 in the sense that what the original creators have done since is new versions (SD 2.0, 2.1, and currently SDXL, which is apparent another SD2-architecture model, and DeepFloyd.) 2.1 has also seen some community uptak…
I dunno. Theres some experimentation with 2.1, but the consensus seems to be that it produces inferior output to 1.5 outside of some niches, and thats before taking the 768x768 1.5 finetunes into account. Deepfloyd is niche. SDXL is indeed interesting, especially if its happy with 4/8 bit quant... we will see about that. Nevertheless StabilityAI seems kinda disconnected from all the innovations going on in the commun…
It's a faulty consensus that came from people comparing outputs from the base model with fine tunes of the SD1.5 model. SD2.1 actually is a far superior model once fine-tuned.
I agree that it's a shame that StabilityAI seem to struggle so much to actually leverage their community (ideally with much more open development)... One could say they're a little too "full of themselves" and think they know better than everyone else.
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#99Hypothetically speaking, if Falcon 40B could out-perform GPT-3.5, would that force OpenAI to open-source GPT-3.5?
Did Stable Diffusion "force" OpenAI or Google to open-source their imaging models? No. Don't see what the mechanism would be for that
Re: Falcon 40B LLM (which beats Llama) now Apache 2.0
#100Not specific to this model, but beyond the large players (OpenAI, Cohere, etc) are there any free hosted versions of the open(ish) LLMs? Even the smaller 7B parameter ones? I'm prototyping out a project and using OpenAI for now, but it feels like there has to be a hosted alternative somewhere. I spent some time today exploring HuggingFace's Inference API but if the model is sufficiently large (> 10gb), HF requires yo…