Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run?

#32

Post by pajeets »

kkielhofner wrote: post #28
pajeets wrote: post #6

Earlier quoted context omitted.

need a 3090 at least for that

llama.cpp and others can run purely on CPU[0]. Even production grade serving frameworks like vLLM[1]. There are a variety of other LLM inference implementations that can run on CPU as well. [0] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su... [1] - https://docs.vllm.ai/en/v0.6.1/getting_started/cpu-installat...

wait this is crazy

what model can i run on 1TB and how many tokens per second ?

for instance Nvidia Nemotron Llama 3.1 quantized at what speed ? ill get a GPU too but not sure how much VRAM I need for the best value for your buck

Re: Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run?

#33

Post by kkielhofner »

pajeets wrote: post #32
kkielhofner wrote: post #28

Earlier quoted context omitted.

llama.cpp and others can run purely on CPU[0]. Even production grade serving frameworks like vLLM[1]. There are a variety of other LLM inference implementations that can run on CPU as well. [0] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su... [1] - https://docs.vllm.ai/en/v0.6.1/getting_started/cpu-installat...

wait this is crazy what model can i run on 1TB and how many tokens per second ? for instance Nvidia Nemotron Llama 3.1 quantized at what speed ? ill get a GPU too but not sure how much VRAM I need for the best value for your buck

> what model can i run on 1TB

With 1TB of RAM you can run nearly anything available (405B essentially being the largest ATM). Llama 405B in FP8 precision fits in H100x8 which is 640GB VRAM. Quantization is a very deep and involved well (far too much for an HN comment).

I'm aware it "works" but I don't bother with CPU, GGUF, even llama.cpp so I can't really speak to it. They're just not even remotely usable for my applications.

> tokens per second

Sloooowwww. With 405B it could very well be seconds per token but this is where a lot of system factors come in. You can find benchmarks out there but you'll see stuff like a very high spec AMD EPYC bare metal system with very fast DDR4/5, tons of memory channels, etc doing low single-digit tokens per second with 70B.

> ill get a GPU too but not sure how much VRAM I need for the best value for your buck

Most of my experience is top-end GPU so I can't really speak to this. You may want to pop in at https://www.reddit.com/r/LocalLLaMA/ - there is much more expertise there for this range of hardware (CPU and/or more VRAM limited GPU configs).

Re: Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run?

#35

Post by kjok »

I’ve a proposal. I’ve created a Hypervisor that can launch lightweight and efficient VMs, similar to containers. Workloads in these VMs behave like isolated processes (isolated from other VMs) You can pack and sell 10x more VMs. Unlike traditional VMs, you don’t have to worry about turning them off so as to free resources. Would love to work together with you if you’re interested in using it.

Re: Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run?

#38

Post by throw4950sh06 »

etcd wrote: post #36
pajeets wrote: post #21

Earlier quoted context omitted.

gonna need quantum computing once you breach in to the mid 20s

I hear that quantum computing can run all the tabs in parallel.

All possible tabs exist in an open/not open state and the wave function collapses once you open a tab. Instant internet using your multiverse modem!
Post reply on HN