Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Can I run this LLM? (locally)

can-i-run-this-llm-blue.vercel.app

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Show HN: Can I run this LLM? (locally)

#31

Post by reaperman »

do_not_redeem wrote: post #28

> Can I Run DeepSeek R1 > Yes, you can run this model! Your system has sufficient resources (16GB RAM, 12GB VRAM) to run the smaller distilled version (likely 7B parameters or less) of this model. Last I checked DeepSeek R1 was a 671B model, not a 7B model. Was this site made with AI?

It says “smaller distilled model” in your own quote which, generously, also implies quantized.

Here[0] are some 1.5B and 8B distilled+quantized derivatives of DeepSeek. However, I don’t find a 7B model, that seems totally made-up from whole cloth. Also, I personally wouldn’t call this 8B model “DeepSeek”.

0: https://www.reddit.com/r/LocalLLaMA/comments/1iskrsp/quantiz...

Re: Show HN: Can I run this LLM? (locally)

#32

Post by CharlesW »

> One of the most frequent questions one faces while running LLMs locally is: I have xx RAM and yy GPU, Can I run zz LLM model ?

In my experience, LM Studio does a pretty great job of making this a non-issue. Also, whatever heuristics this site is based on are incorrect — I'm running models on a 64GB Mac Studio M1 Max that it claims I can't.

Re: Show HN: Can I run this LLM? (locally)

#33

Post by do_not_redeem »

sudohackthenews wrote: post #30
do_not_redeem wrote: post #28

> Can I Run DeepSeek R1 > Yes, you can run this model! Your system has sufficient resources (16GB RAM, 12GB VRAM) to run the smaller distilled version (likely 7B parameters or less) of this model. Last I checked DeepSeek R1 was a 671B model, not a 7B model. Was this site made with AI?

> > smaller distilled version Not technically the full R1 model, it’s talking about the distillations where Deepseek trained Qwen and Llama models based on R1 output

Then how about DeepSeek R1 GGUF:

> Yes, you can run this model! Your system has sufficient resources (16GB RAM, 12GB VRAM) to run this model.

No mention of distillations. This was definitely either made by AI, or someone picking numbers for the models totally at random.

Re: Show HN: Can I run this LLM? (locally)

#34

Post by jsheard »

do_not_redeem wrote: post #28

> Can I Run DeepSeek R1 > Yes, you can run this model! Your system has sufficient resources (16GB RAM, 12GB VRAM) to run the smaller distilled version (likely 7B parameters or less) of this model. Last I checked DeepSeek R1 was a 671B model, not a 7B model. Was this site made with AI?

> Was this site made with AI?

OP said they "vibe coded" it, so yes.

https://en.m.wikipedia.org/wiki/Vibe_coding

Re: Show HN: Can I run this LLM? (locally)

#35

Post by abujazar »

Nice concept – but unfortunstely I found it to be incorrect in all of the examples I tried with my Mac.

It'd also need to be much more precise in hardware specs and cover a lot more models and their variants to be actually useful.

Grading the compatibilty is also an absolute requirement – it's rarely an absolute yes or no, but often a question of available GPU memory. There's a lot of other factors too which don't seem to be considered.

Re: Show HN: Can I run this LLM? (locally)

#36

Post by monocasa »

do_not_redeem wrote: post #28

> Can I Run DeepSeek R1 > Yes, you can run this model! Your system has sufficient resources (16GB RAM, 12GB VRAM) to run the smaller distilled version (likely 7B parameters or less) of this model. Last I checked DeepSeek R1 was a 671B model, not a 7B model. Was this site made with AI?

Is it maybe because DeepSeek is a MoE and doesn't require all parameters for a given token?

That's not ideal from a token throughput perspective, but I can see min working set of weight memory gains if you can load pieces into vram for each token.

Re: Show HN: Can I run this LLM? (locally)

#39

Post by rkagerer »

abujazar wrote: post #35

Nice concept – but unfortunstely I found it to be incorrect in all of the examples I tried with my Mac. It'd also need to be much more precise in hardware specs and cover a lot more models and their variants to be actually useful. Grading the compatibilty is also an absolute requirement – it's rarely an absolute yes or no, but often a question of available GPU memory. There's a lot of other factors too which don't se…

I found it to be incorrect in all of the examples I tried

Are you sure it's not powered by an LLM inside?

Re: Show HN: Can I run this LLM? (locally)

#40

Post by throwaway314155 »

monocasa wrote: post #36
do_not_redeem wrote: post #28

> Can I Run DeepSeek R1 > Yes, you can run this model! Your system has sufficient resources (16GB RAM, 12GB VRAM) to run the smaller distilled version (likely 7B parameters or less) of this model. Last I checked DeepSeek R1 was a 671B model, not a 7B model. Was this site made with AI?

Is it maybe because DeepSeek is a MoE and doesn't require all parameters for a given token? That's not ideal from a token throughput perspective, but I can see min working set of weight memory gains if you can load pieces into vram for each token.

It still wouldn't fit in 16 GB memory. Further there's too much swapping going on with MoE models to move expert layers to and from gpu without bottlenecks.
Post reply on HN