Show HN: Ollama – Run LLMs on your Mac

Under the hood it does a few things I'll shed some light on (at a high level!):

1. Running the model: it's built on the open-source (and amazing) llama.cpp project for running quantized (i.e. compressed) models like Llama 2 (launched yesterday) that will fit in memory on even a commodity Mac. It's similar to their "server" example as a starting point.

2. Downloading and storing models: models are distributed in a way that ensures their integrity and re-usability as much as possible (since they are large files!). For this we use a similar approach as Docker (https://github.com/distribution/distribution)

3. Creating custom models: models can be extended with this new idea we're experimenting with: a Modelfile. What this will do is effectively add "layers" to a model so you can distribute model data together and keep them self-contained. This builds on what I mentioned in 2 – our hope is this will make it easier to extend models like Llama 2 to your own use cases (e.g. a character).

Error: Post "http://127.0.0.1:11434/api/generate": EOF

Try running the orca model (default is 3B), and it requires much less memory.

``` ollama run orca ```

Looks cool. I downloaded it, moved it to the applications directory and opened it. It's asking for Administrator access to install the command line. Why?

I feel like I've seen a project just like this pop up almost every other day here on HN. Sorry for my ignorance, but how is this different when compared to other "LLM on your mac CLI" solutions nowadays? (please bear in mind I'm not up to speed with these sibling projects)

Is it mostly in the `Modelfile` approach with expandable models and quick setup?