111–120 of 143 posts
Am I the only one who finds this very sketchy? They had the whole license things, there's been some loud complaining by the HF CTO on social media that this model is not getting enough attention, and there are also press releases about how Falcon tops the "leaderboard": https://www.morningstar.com/news/business-wire/2023052900504... I've never seen this kind of "strategy" with an ML model before. Maybe I'm seeing som…
4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!
The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?
But if you're willing to spend $1500 on two used RTX 3090, it's the sweet spot in terms of the ability to run large models right now. Everything beyond that is much more expensive.
Am I the only one who finds this very sketchy? They had the whole license things, there's been some loud complaining by the HF CTO on social media that this model is not getting enough attention, and there are also press releases about how Falcon tops the "leaderboard": https://www.morningstar.com/news/business-wire/2023052900504... I've never seen this kind of "strategy" with an ML model before. Maybe I'm seeing som…
maybe attention IS all you need
Seriously though, I think it's mostly the opposite. How much advertising did Georgi Gerganov do for ggml / llama.cpp and it's super popular. Maybe other people are just being more subtle, but I feel generally merit stands out on it's own and advertising is a poor substitute.
Am I the only one who finds this very sketchy? They had the whole license things, there's been some loud complaining by the HF CTO on social media that this model is not getting enough attention, and there are also press releases about how Falcon tops the "leaderboard": https://www.morningstar.com/news/business-wire/2023052900504... I've never seen this kind of "strategy" with an ML model before. Maybe I'm seeing som…
maybe attention IS all you need
https://towardsdatascience.com/attention-is-all-you-need-dis...
Earlier quoted context omitted.
Very cool but how does this compare to GPT-4 (before it was nerfed)? I feel like the best benchmark atm is the orig gpt-4 version.
GPT-4 wasn’t nerfed. https://news.ycombinator.com/item?id=36155267
Earlier quoted context omitted.
TIL Oracle has VPS offerings with a free tier. Are they any good? Is the free-tier time limited? This use-case is alright for a toy I guess - which is the extent that I was originally expecting these things to be useful for.
They're okay. This isn't the place for a full review of their offerings (especially considering everyone's mixed feelings on Oracle), but I'm confident that it's better than most 1core/$5 deals you'll find elsewhere. > Are they any good? Yep, free tier allows you to spec up to 24gb of RAM without paying, which is cool. The bottleneck is really the disk speed, but that's not an issue with mmaped models. There's enough…
Earlier quoted context omitted.
That’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow. For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.
Is it memory/disk bandwidth bound or latency bound?
Am I the only one who finds this very sketchy? They had the whole license things, there's been some loud complaining by the HF CTO on social media that this model is not getting enough attention, and there are also press releases about how Falcon tops the "leaderboard": https://www.morningstar.com/news/business-wire/2023052900504... I've never seen this kind of "strategy" with an ML model before. Maybe I'm seeing som…
What PR noise they make is secondary to that in my mind.