Viewing profile — colorant

Hardware Requirements: - 380GB CPU memory for DeepSeek V3/R1 671B INT4 model - 128GB CPU memory for Qwen3MoE 235B INT4 model - 1-2 ARC A770 or B580 - 500GB Disk space

With ~1000 input, the TTFT is ~10 seconds

The ipex-llm implementation extends llama.cpp and includes additonal CPU-GPU hybrid optimizations for sparse MoE

Currently >8 token/s; there is a demo in this post: https://www.linkedin.com/posts/jasondai_run-671b-deepseek-r1...

Yes, but the context length will be limited due to VRAM constraint

Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)

This is based on llama.cpp

>8TPS at this moment on a 2-socket 5th Xeon (EMR)

See this section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...

Also see the demo from Jason Dai's post: https://www.linkedin.com/posts/jasondai_with-the-latest-ipex...

Yes, you are right. Unfortunately HN somehow truncated my original URL link.

https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic... Requirements (>8 token/s): 380GB CPU Memory 1-8 ARC A770 500GB Disk

llamafile cannot use Intel GPU (including integrated GPU on your PC)

There is ipex-llm support for Ollama on Intel GPU ( https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic... )

BigDL-LLM is a library for running LLM (language language model) on your local laptop using INT4 with very low latency on CPU. (It is built on top of the excellent work of llama.cp…