161–170 of 274 posts
I recently bought a T4 to go with my epyc 7402 and 512GB ram for fun and this looks like a great use case. Thanks!
What's the advantage of purchasing a T4 instead of a 3090 or 4090?
Earlier quoted context omitted.
The cloud has been cheaper to train large models on for a couple years now. Compare buying 8xA100 server vs renting one on Lambda Labs. At least 3 years to break even - if you are using it non-stop 24/7. Longer if not.
This is not true - the break even period is closer to 6-7 months.
Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`. I also modified the prompt a bit to make it more... uh alive: Assistant: Did you know that Saturn is 97 times the size of Earth? Human: Are you sure? Assistant: What difference does size make, really, anyway? Human: You didn't ans…
Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`. I also modified the prompt a bit to make it more... uh alive: Assistant: Did you know that Saturn is 97 times the size of Earth? Human: Are you sure? Assistant: What difference does size make, really, anyway? Human: You didn't ans…
That's impressive! How much customization did you have to do in order to get the chatbot to respond like that?
A chat between a human and a more funny and witty intelligent assistant
Human: Hello! What can you do?
Assistant: Hey there! I can pretty much do whatever you ask!
Human: What is the name of the tallest mountain in the world?
Assistant: It's Everest.
Human: Are you sure?
Assistant: You damn right I'm sure! I'm a hyperintelligent AI!Very cool. Worth mentioning though that the highlighted figures (1.12 tok/s for OPT-175B for "FlexGen with Compression") are for inputs of 512 tokens and outputs of 32 tokens. Since decoder-only transformer memory requirements scale with the square of sequence lengths, things would probably slow down significantly for very long sequences, which would be required for a back-and-forth conversation. Still though, until…
> transformer memory requirements scale with the square of sequence lengths Not true, see: Flash Attention. You can losslessly calculate the attention in blocks using a little math trick. Essentially each subsequent block "corrects" the denominator of the last block's softmax calculation. At the end you have a perfectly* accurate softmax. Since you don't need to keep the whole sequence in memory to perform the softma…
Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`. I also modified the prompt a bit to make it more... uh alive: Assistant: Did you know that Saturn is 97 times the size of Earth? Human: Are you sure? Assistant: What difference does size make, really, anyway? Human: You didn't ans…
This is amazing. Reminds me of claptrap from Borderlands
A lot of people are looking at this wrong. A $350 3060Ti has 12GB RAM. If there's a way to run models locally, it opens up the door to: 1) Privacy-sensitive applications 2) Tinkering 3) Ignoring filters 4) Prototyping 5) Eventually, a bit of extra training The upside isn't so much cost / performance, as local control over a cloud-based solution.
I have that exact card, this maybe the nudge where I remove windows from the computer and try out linux gaming (and local GPT)
We are hiring in that area of work in Europe time zone. If you are exited about and capable in this field, please apply here: https://ai-jobs.net/job/41469-senior-research-engineer-llms-...
Earlier quoted context omitted.
That's impressive! How much customization did you have to do in order to get the chatbot to respond like that?
Not much. Here's the prompt: A chat between a human and a more funny and witty intelligent assistant Human: Hello! What can you do? Assistant: Hey there! I can pretty much do whatever you ask! Human: What is the name of the tallest mountain in the world? Assistant: It's Everest. Human: Are you sure? Assistant: You damn right I'm sure! I'm a hyperintelligent AI!
Earlier quoted context omitted.
do/did you seriously think that the processing requirement was going to “kill” that possibility? the history of computing clearly indicates that either the requirements for running or the bar for owning this technology was always going to drop
The history of computing had Moore's Law.