Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Ollama – Run LLMs on your Mac

github.com

Post reply on HN
Print view View original

1–10 of 101 posts

Show HN: Ollama – Run LLMs on your Mac

#1

Post by jmorgan »

Hi HN

A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container runtime and image registry side), the recent rise in open source language models made us think something similar needed to exist for large language models too.

While not exactly the same as running linux containers, running LLMs shares quite a few of the same challenges. There are "base layers" (e.g. models like Llama 2), specific configuration to run correctly (parameters, temperature, context window sizes etc). There's also embeddings that a model can use at runtime to look up data – we don't support this yet but it's something we're looking at doing soon.

It's an early project, and there's still lots to do!

Show HN: Ollama – Run LLMs on your Mac
github.com

Re: Show HN: Ollama – Run LLMs on your Mac

#2

Post by zenlikethat »

I've been playing with this lately and it's been loads of fun (on OSX in particular, on Windows/WSL I don't think it's rigged up with GPTQ yet).

Of note is the experimental "Modelfile" that allows you to ship around an AI character or "stack" like a Docker image. I can really see the future around this shaping up to be really exciting, since I'll be able to hand you something much more replicable in terms of model, prompt, etc, and once we are able to chain calls and bundle embeddings, LoRa, etc., it seems like it could be really powerful.

Re: Show HN: Ollama – Run LLMs on your Mac

#5

Post by Patrick_Devine »

willietran wrote: post #3

Whoa, this is really neat. Does this mean that I can essentially try any LLM on my local machine?

Yep. Right now we've packaged llama2, vicuna, wizardlm, and orca. The idea is to make it crazy easy to get started though. You do need quite a bit of RAM (16GB should work for the smaller models, 32MB+ for the bigger ones), and for now a newer Mac. We're working versions for Windows and Linux too though.

EDIT: We don't let you run stuff from HF, but we are trying to repackage the popular models. The plan is to let you upload your own in the future to share them.

Re: Show HN: Ollama – Run LLMs on your Mac

#8

Post by jmorgan »

zenlikethat wrote: post #7

One Q for the maintainers, in https://github.com/jmorganca/ollama/blob/main/library/modelf... does that imply you can override TEMPLATE directly? I am experimenting with adding some dynamic context between user instructions

You can, although this is definitely "hard mode" when writing Modelfiles.

Easier is to override SYSTEM which allows will customize the system prompt.

SYSTEM "You are an expert in writing python."

If you do want modify the template, you can start by looking at the "base images" https://github.com/jmorganca/ollama/blob/main/library/modelf...

For example with Llama 2 as a base model, you can do this, but YMWV:

  FROM llama2
  TEMPLATE """
  {{- if .First }}
  >
  A conversation between three friends
  >
  {{- end }}

  Friend 1: hello
  Friend 2: hey
  Friend 3: {{ .Prompt }}
  """
edit: we're looking at how to make it easier so you don't need {{ .First }} to provide the system prompt on first message

Re: Show HN: Ollama – Run LLMs on your Mac

#10

Post by willietran »

Patrick_Devine wrote: post #5
willietran wrote: post #3

Whoa, this is really neat. Does this mean that I can essentially try any LLM on my local machine?

Yep. Right now we've packaged llama2, vicuna, wizardlm, and orca. The idea is to make it crazy easy to get started though. You do need quite a bit of RAM (16GB should work for the smaller models, 32MB+ for the bigger ones), and for now a newer Mac. We're working versions for Windows and Linux too though. EDIT: We don't let you run stuff from HF, but we are trying to repackage the popular models. The plan is to let yo…

Awesome! Thanks for this. Trying this out now.
Post reply on HN