61–70 of 70 posts
Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I was never interested in gaming, crypto or ML), but I found GPT very useful and I'd prefer to run a local version instead of keep feeding OpenAI.
Have you checked Gpt4all or Faraday.dev ?
Earlier quoted context omitted.
I think the point is that LOC is not a terribly useful metric in that everything is 1 LOC at the highest level of abstraction. The business proposition here is that you don't need to write the LOCs for the underlying layers, they do it. The pitch here is that it's not as straightforward for large GPU clusters.
Sure, I get that. I've definitely seen demos of "Do X in Y LoC" that do X but offload all of the hard work of Y to some libray. This is not that. This is intended to be a demo that shows you what you can do with one Cerebras module. And, the result is that, by writing 565 LoC yourself, you can train and run an LLM the size of GPT-3. In that sense, 565 LoC is a perfectly fair number. It doesn't count PyTorch, numpy, t…
Cerebras' point still stands though, even if you can get the LoC count down significantly nowadays, it's still a major PITA to debug those systems, deal with node crashing, tweak the architecture and the data-loading pipeline to have high GPU utilization, optimize network bottlenecks etc. Scaling vertically first like Cerebras is doing surely makes that much easier.
On a tangentially related note, this is imho where OpenAI has built it's moat: training and inference stack that they have refined over the last 6 years. They have good researchers, but so does MS, Google and Meta. But no one else has the ability to train such large models with such ease. Same for the inference stack, being able to run GPT-3.5/4 in prod at the scale at which they are doing it is no joke, and I'm 100% convinced this is why Gemini is still not widely available a year after 3.5 came out.
Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I was never interested in gaming, crypto or ML), but I found GPT very useful and I'd prefer to run a local version instead of keep feeding OpenAI.
On windows you can get it via lmstudio.ai for example
Here is ur running on a MacBook M2 Air, we have smaller, more performant models coming
For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…
Hey investor, we want chatgpt 5. The issue is not lines of code.
Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I was never interested in gaming, crypto or ML), but I found GPT very useful and I'd prefer to run a local version instead of keep feeding OpenAI.
I recommend downloading and running OpenHermes inside LM Studio. https://lmstudio.ai/
In LM Studio, search for OpenHermes. Pick the Q5_K_M version (this is the best quality/speed trade off). Then go to the chat tab.
On the chat tab, set the context length to 4096 (or up to 16k if you want longer context) and set the number of CPU cores you have under "Hardware Settings."
Select the model from the drop down and start chatting!
For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…
> I'm a small investor in Cerebras. Hey investor, we want chatgpt 5. The issue is not lines of code.
Lines of code is not strictly speaking a bottleneck for the next generation of models, but it ties with other objectives that are: researcher productivity, hardware efficiency, and model verifiability. GPT-5 might be another case of simply scaling up the existing transformer models, but the next step-function change of model quality are going to involve a lot more R&D about the right architectural primitives to take before scaling it up. And in that case lines of code do matter - because they allow simple concepts to be robustly tested, optimized, and iterated against. Doing that against a 50k monolith is a much harder task.
Andrew Karpathy is truly a gem and super grateful he still publishes videos showing his art.
Cerebras showing their distributed architecture on that same piece of code is impressive.
All of AI is search for a god algorithm. An algorithm so simple it could be written on an A4 piece of paper in 12px font size - but with enough data and compute it can more intelligent than entire cities of humans combined.
NanoGPT is a glimpse of that.
For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…
This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-m…
We don't have overcomplicated distributed training infra. They are pretty much as complicated as needed.
Scaling vertically (having a bigger chip) is very hard. There are tons of tradeoff when making the chip, and it's overall an insanely complex problem. That's why Cerebras is a 8 years old company and yet you would be hard pressed to find anyone using them still.
And even if you give me a Cerebras chip that works perfectly, it will still be much easier for me to buy two of those chips and link them together in distributed training mode, than it will be for Cerebras to build a chip that is 2x the size.
The scale of the current generation of clusters to train models the size of GPT-4 are in the range of 25,000+ GPUs with 80GB of memory each, so no matter your chip size, complicated distributed infra is a necessity. Even assuming everything on Cerebras' marketing page is fully accurate, you would still need to distribute the training over 500+ of those massive chips to replicate a 25k GPU cluster.
Earlier quoted context omitted.
To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.
New to the space — I have been using HF for both training and inference. Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to
Inference is where it falls short really, and solutions like vLLM are much much faster.