Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Ollama – Run LLMs on your Mac

github.com

Post reply on HN
Print view View original

11–20 of 101 posts

Re: Show HN: Ollama – Run LLMs on your Mac

#11

Post by mchiang »

magundu wrote: post #9

It is really amazing. With this command “ollama run llama2”, I can run llama2 instantly. Do you know how much RAM and disk space needed to run this llama2 model?

By default the `llama2` model is the 7B model, and it's recommended you have at least 16GB of memory to run it.

Regarding the disk space, the model itself is 3.8GB.

Re: Show HN: Ollama – Run LLMs on your Mac

#13

Post by jmorgan »

magundu wrote: post #9

It is really amazing. With this command “ollama run llama2”, I can run llama2 instantly. Do you know how much RAM and disk space needed to run this llama2 model?

For disk size, you can see how big models are locally by running "ollama list"

  NAME          SIZE   MODIFIED      
  llama2:latest 3.8 GB 3 hours ago
Now, if you make new models that are "FROM llama2" for example, Ollama will re-use this "base model" so you don't store it twice.

Re: Show HN: Ollama – Run LLMs on your Mac

#15

Post by jmorgan »

politelemon wrote: post #12

Surprised to see no Linux support, considering that it's a project that manages... docker.

While we haven't built Linux binaries yet (waiting to make sure we have _fast_ GPU support before publishing a Linux and Windows binaries) it does run on Linux. For example it's used to create and publish the "library" images (e.g. "llama2") on a linux server today

If you have the Go toolchain installed, there are some steps from building it from source here: https://github.com/jmorganca/ollama#building

Re: Show HN: Ollama – Run LLMs on your Mac

#17

Post by jmorgan »

mikeravkine wrote: post #16

Maybe mention that GGML is under the hood. I see you've forked the library, did you pick up the GQA changes needed for llama2-70b? Also seems like you are picking quants on people's behalf, which ones?

This is super helpful! Currently it runs 4-bit quantization models (with names often ending in q4_0). Sorry this isn't more obvious. Will fix that and soon update the hosted models to support different quantization levels, so you can choose including k quant models. More to come there.

It does use GGML under the hood. Will definitely make sure that's more obvious for compatibility. There's about to be an update as well that includes those recent changes. We do vendor the source code since it made building with Go a lot easier, but definitely don't want to be "forking" it (vs supporting it) – it's a brilliant piece of software!

Post reply on HN