81–90 of 146 posts
Earlier quoted context omitted.
20 years?
In 20 years I'll still be alive and enjoying myself. Does 20 years seem long to you?
20y ago we had the GeForce Ti4400, current graphics cards now come with 100x the vram and 50-60x the bandwidth.
Earlier quoted context omitted.
Hmmm, 2-3 high end GPUs implies it's likely not very far off from mainstream. Maybe runnable on the average device within 10-20 years... perhaps even sooner if the model/software can be optimized?
I’d be surprised if there weren’t any algo breakthroughs before that to make these several times faster (10x?). (such as are all of the weights really needed all of the time, or could you load different ones based on recent context?) If in 10 years on top of that you’d have that 10x faster hardware as well, you might be running GPT-3s as just a subcomponent of games/apps.
A component is a self-contained unit of functionality, often in the form of a class or object, that represents a piece of an overall system or application. A subcomponent, on the other hand, is a smaller, simpler component that is used as part of a larger component. A subcomponent is typically used to break down a larger component into smaller, more manageable pieces, and is often used to create reusable building blocks for larger components. In other words, a subcomponent is a component that is used by another component.
Earlier quoted context omitted.
> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…
fp4 ?= float point of 4 bits??? I was already mind blown by floats of 8b, how can you fit any float precision in 4b?
Q: How would I make for loop in python?
A: I can help you create an AI chat bot. It would talk to you like a human. (additional text that is not relevant to the prompt)
It is just me or this does not seem right?
Earlier quoted context omitted.
A few 4090Ti GPUs might be good enough, if they do indeed end up with 48GB of VRAM.
According to wikipedia ChatGPT is built on GPT-3 which has 175B parameters requiring 800GB of memory, so a few 4090s will fall short.
Someone will come along selling pre-builts for this purpose, if that's not already available for their current research/enterprise cards.
Earlier quoted context omitted.
> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…
How badly is bad? What sort of output are we talking? I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh. Can it form sentences or are those small models completely unusable for anything? I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite fi…
Sadly, they really offer almost no value.
For the effort, you’re better off with an NLP framework like spacy.
You can play with the small neo gpt models on hugging face, eg. https://huggingface.co/EleutherAI/gpt-neo-125M
…but, the tldr is they’re cute to play with, but practically, the content they can generate is short, inconsistent and full of errors.
So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…
> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…
Enough system memory to fit 84GB isn't all that expensive...
Earlier quoted context omitted.
My family told me that ChatGPT came up from the pulpit AT CHURCH Me? I made a few comments like a scared luddite when ChatGPT solved two of my outstanding engineering problems instantly. I got better. But this is exactly right. The world in general now knows about AI and ML. It’s a pivot point. When something scares a seasoned engineer for a minute, and anyone can now make use of this… write it down in your diary as…
I wonder what kind of engineering problems you’ve talked with ChatGPT. Because its limited understanding of spatial problems, I found it hard to use it for mechanical engineering. How did you do it?
ChatGPT is currently best at things programmers would think about. You’re correct about spatial reasoning. But try stuff like this:
“Write a python program that calculates the static forces on a cantilevered ledge 15 feet long, with a support beam”
Haha it took the longest I’ve ever seen. You may have a point. It’s really good at writing code though.
Caution. I tried my example with matlab instead of python, and I think I may have set a server rack on fire ;)
Earlier quoted context omitted.
fp4 ?= float point of 4 bits??? I was already mind blown by floats of 8b, how can you fit any float precision in 4b?
For weights, the order of magnitude is the important part. And the sign bit. So you can get pretty good coverage with only 16 values.
fp4 with 1-3-0 would mean 27 values if the first bit were interpreted as binary. But--and an engineer should check me on this cause to me a transistor is a distant abstraction--I think you could double that to 54 values if you were clever with the sign bit and arithmetic circuitry. Maybe push it to 42 if only some of my intuition is wrong.
Earlier quoted context omitted.
> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…
Pretty cool! Honestly even if it were to take a few minutes per response, that's likely sufficient for many use cases. I'd get value out of that if it allowed bypassing a paywall. I'm curious how these models end up being monetized/supported financially, as they sound expensive to run at scale. The required disk space seems the biggest barrier for local.