Run DeepSeek R1 Dynamic 1.58-bit
unsloth.ai
Run DeepSeek R1 Dynamic 1.58-bit
1–10 of 346 posts
Re: Run DeepSeek R1 Dynamic 1.58-bit
#2[deleted]
Re: Run DeepSeek R1 Dynamic 1.58-bit
#3Would be great if the next generation of base models was designed to be inferred with 128GB of VRAM while 8bit quantized (which would fit in the consumer hardware class).
For example, I imagine a strong MoE base with 16 billion active parameters and 6 or 7 experts would keep a good performance while being possible to run on 128GB RAM macbooks.
Re: Run DeepSeek R1 Dynamic 1.58-bit
#4> Unfortunately if you naively quantize all layers to 1.58bit, you will get infinite repetitions in seed 3407: “Colours with dark Colours with dark Colours with dark Colours with dark Colours with dark” or in seed 3408: “Set up the Pygame's Pygame display with a Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's Pygame's”.
This is really interesting insight (although other works cover this as well). I am particularly amused by the process by which the authors of this blog post arrived at these particular seeds. Good work nonetheless!
Re: Run DeepSeek R1 Dynamic 1.58-bit
#5In general, how do you run these big models on cloud hardware? Do you cut them up layer-wise and run slices of layers on individual A100/H100s?
Re: Run DeepSeek R1 Dynamic 1.58-bit
#6Thanks for the run instructions, unsloth. Deepseek is so new it's been breaking most of my builds.
Re: Run DeepSeek R1 Dynamic 1.58-bit
#7Would be great if the next generation of base models was designed to be inferred with 128GB of VRAM while 8bit quantized (which would fit in the consumer hardware class). For example, I imagine a strong MoE base with 16 billion active parameters and 6 or 7 experts would keep a good performance while being possible to run on 128GB RAM macbooks.
Would be great, but unfortunately i think intelligence at that compute scale will be limit by hardware not its model. Though at hardware limit I would expect it to be roughly human level especially if optimized for a particular domain.
Re: Run DeepSeek R1 Dynamic 1.58-bit
#8In general, how do you run these big models on cloud hardware? Do you cut them up layer-wise and run slices of layers on individual A100/H100s?
Was wondering the same, but for HPC clusters :)
Re: Run DeepSeek R1 Dynamic 1.58-bit
#9Would be great if the next generation of base models was designed to be inferred with 128GB of VRAM while 8bit quantized (which would fit in the consumer hardware class). For example, I imagine a strong MoE base with 16 billion active parameters and 6 or 7 experts would keep a good performance while being possible to run on 128GB RAM macbooks.
Would be great, but unfortunately i think intelligence at that compute scale will be limit by hardware not its model. Though at hardware limit I would expect it to be roughly human level especially if optimized for a particular domain.
I remember that Llama 3 was trained on data curated by Llama 2 and it resulted in a model with a significant performance boost (even though it was trained by a previous generation model of the same size).
Maybe using a strong reasoning model such as R1 the next generation, even more performance can be extracted from smaller models.
Re: Run DeepSeek R1 Dynamic 1.58-bit
#10In general, how do you run these big models on cloud hardware? Do you cut them up layer-wise and run slices of layers on individual A100/H100s?
You could do that, and add pipelining to improve speed.