Show HN: Ollama – Run LLMs on your Mac • Hacker News Board

Re: Show HN: Ollama – Run LLMs on your Mac

#91

Post by d0gbread » Fri, Jul 21, 2023, 11:00 PM UTC

sdesol wrote: post #45

Hey congrats. Your project is showing a hockey stick type growth https://devboard.gitsense.com/jmorganca/ollama which highlights how hot this space is right now. Full Disclosure: This is my tool.

Awesome tool. Always thought about building something like this. One of my thoughts was to take all that data and distill it into a composite "trust" score of some kind, to help make decisions on whether to lean on it or not.

Re: Show HN: Ollama – Run LLMs on your Mac

#92

Post by sdesol » Sat, Jul 22, 2023, 12:21 AM UTC

d0gbread wrote: post #91

sdesol wrote: post #45

Hey congrats. Your project is showing a hockey stick type growth https://devboard.gitsense.com/jmorganca/ollama which highlights how hot this space is right now. Full Disclosure: This is my tool.

Awesome tool. Always thought about building something like this. One of my thoughts was to take all that data and distill it into a composite "trust" score of some kind, to help make decisions on whether to lean on it or not.

Yes creating an "open source score" is something that I might do. The challenge is to obviously not turn it into a negative thing. For example, a low score could should be seen as being inviting for new contributors.

Re: Show HN: Ollama – Run LLMs on your Mac

#93

Post by paddim8 » Sat, Jul 22, 2023, 8:14 AM UTC

bilalq wrote: post #72

It probably wasn't intentional, but the naming of this is a great play off the Arabic word "علماء", pronounced as Ulamā[1]. It translates to "scholars", or "the learned ones". Pretty fitting. Nowadays, the word is most often used in the context of religious knowledge, but isn't limited to that. [1]: https://en.wikipedia.org/wiki/Ulama

In Swedish it's quite inappropriate if you ignore the last two letters

Re: Show HN: Ollama – Run LLMs on your Mac

#94

Post by brianjking » Sat, Jul 22, 2023, 9:08 PM UTC

iamdamian wrote: post #59

I've been wanting to try out local LLMs for a while but have always had the feeling that I would end up wasting my time. I imagined that I would have to spend a perfectly good Saturday futzing around with lots of settings and environments and trying to free up hundreds of GB of hard drive space for a model or wouldn't have the right hardware to properly run one locally. I also imagined that this probably wouldn't wor…

https://gpt4all.io/index.html is another great and simple way to get started.

Re: Show HN: Ollama – Run LLMs on your Mac

#95

Post by neontomo » Sun, Jul 23, 2023, 2:39 AM UTC

I like this a lot, unfortunately I can only run Orca due to 8GB Macbook Air.

This opens a lot of doors to using LLMs in art projects for me without worrying about API calls.

Re: Show HN: Ollama – Run LLMs on your Mac

#96

Post by Traveler42 » Sun, Jul 23, 2023, 4:00 AM UTC

jpeeler wrote: post #90

mchiang wrote: post #20

Earlier quoted context omitted.

On the mac, we have enabled Metal support.

What is the best way to confirm that the GPU is being utilized? I can open Activity Monitor and see the usage spike, but surely there's a better/direct way.

Try asitop. For me it doesn't indicate anything but CPU is in use.

https://github.com/tlkh/asitop

Re: Show HN: Ollama – Run LLMs on your Mac

#97

Post by Traveler42 » Sun, Jul 23, 2023, 4:09 AM UTC

If I use asitop, it doesn't show any use of the GPU. Should it?

Is there some flag needed to enable acceleration on MPS?

Re: Show HN: Ollama – Run LLMs on your Mac

#98

Post by Traveler42 » Sun, Jul 23, 2023, 6:14 AM UTC

Traveler42 wrote: post #97

If I use asitop, it doesn't show any use of the GPU. Should it? Is there some flag needed to enable acceleration on MPS?

I had downloaded and built it locally. If you download the version from the web site, it is a lot faster...

Re: Show HN: Ollama – Run LLMs on your Mac

#99

Post by Traveler42 » Sun, Jul 23, 2023, 5:04 PM UTC

Traveler42 wrote: post #97

If I use asitop, it doesn't show any use of the GPU. Should it? Is there some flag needed to enable acceleration on MPS?

Ok now it uses MPS. I had to carefully make sure that I have Go for arm64 set up properly. "brew bundle install go" mysteriously didn't work but "brew install go" did. I haven't looked into why.

Now my local build uses the GPU similar to the downloaded version!

Re: Show HN: Ollama – Run LLMs on your Mac

#100

Post by mxm2 » Tue, Jul 25, 2023, 10:56 AM UTC

Hey, great work. But how do I actually delete a model, if it doesn't work on my machine? the rm command doesn't actually delete the files it seems, as if I then do the run command again, it doesn't need to download again...