Ask HN: What's the best hardware to run small/medium models locally?

I don't want to derail the OP's question, but would the same kind of system to run an LLM on also be suitable for an image generator like Stable Diffusion or does it work through different methods?

I think there are a couple of basic questions need answered before we can find a good solution:

1) What are you trying to do?

2) What's your budget?

Generically saying, "run inference" is like... you can do that on your current thinkpad, if you want a small enough model. If you want to run 7B or 13B or 34B models for document or sentiment analysis, or whatever, then you can move to the budget question.

When I was faced with this question, I bought the cheapest 4060 Ti with 16GB I could find. It does "okay". Here's an example run:

  Llama.generate: prefix-match hit
  
  llama_print_timings:        load time =     627.53 ms
  llama_print_timings:      sample time =     415.30 ms /   200 runs   (    2.08 ms per token,   481.58 tokens per second)
  llama_print_timings: prompt eval time =     162.12 ms /    62 tokens (    2.61 ms per token,   382.44 tokens per second)
  llama_print_timings:        eval time =    8587.32 ms /   199 runs   (   43.15 ms per token,    23.17 tokens per second)
  llama_print_timings:       total time =    9498.89 ms
  Output generated in 9.79 seconds (20.43 tokens/s, 200 tokens, context 63, seed 1836128893)

I'm using the text-generation-webui to provide the OpenAI API interface. It's pretty easy to hit:

  import os
  import openai
  url = "http://localhost:7860/v1"
  openai_api_key = os.environ.get("OPENAI_API_KEY")
  client = openai.OpenAI(base_url=url, api_key=openai_api_key)
  result = client.chat.completions.create(
      model="wizardlm_wizardcoder-python-13b-v1.0",
      messages = [
          {"role":"system", "content":"You are a helpful AI agent. You are honest and truthful"},
          {"role":"user", "content": "What is the best approach when writing recursive functions?"},
      ]
  print(result)

But again, it just depends on what you want to do.

Somewhat related; how to run an uncensored model locally? I run llamafile (llamafile-server-0.1-llava-v1.5-7b-q4 and mistral-7b-instruct-v0.1-Q4_K_M-server) ones on my macbook m1 and they run file (fast enough for playing), but they both seem neutered quite a bit. It's hard to get them off the rails and mistral (the above one) actually barfs really quickly just repeating the same letter (fffffff usually) where it should've said fuck. Now i'm not looking for something that writes porn or whatnot, but the online models are so pc, it's getting on my nerves.

I think so. A big nvidia gpu will run both.

4060Ti w/ 16GB VRAM or 3090 w/ 24 GB VRAM

Of course with those you'll also have to spend some money on motherboard, ram, SSD, PSU, CPU, ect.

I think the best bang for the buck is probably a Mac studio with as much ram as you can afford.

I bought an RTX A2000 (12GB VRAM), and it's fine for 7B models and some 13B models with 4 bit quantization, but I kind of regret not getting something with more VRAM.