Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile – The easiest way to run LLMs locally on your Mac

ppaolo.substack.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Llamafile – The easiest way to run LLMs locally on your Mac

#11

Post by woleium »

dang wrote: post #8
senthil_rajasek wrote: post #4

See also, Llamafile is the new best way to run a LLM on your own computer (simonwillison.net) https://news.ycombinator.com/item?id=38489533 And https://news.ycombinator.com/item?id=38464057

Thanks! Macroexpanded: Llamafile is the new best way to run a LLM on your own computer - https://news.ycombinator.com/item?id=38489533 - Dec 2023 (45 comments) Llamafile lets you distribute and run LLMs with a single file - https://news.ycombinator.com/item?id=38464057 - Nov 2023 (286 comments)

Do you think it would be useful to explain how to macroexpand whenever you do it so the folks you are responding to can learn and do it themselves next time?

(myself included)

Re: Llamafile – The easiest way to run LLMs locally on your Mac

#13

Post by wokwokwok »

superkuh wrote: post #9
wokwokwok wrote: post #7

Why? It's unsafe and it takes all the choice and control away from you. You should, instead: 1) Build a local copy of llama.cpp (literally clone https://github.com/ggerganov/llama.cpp and run 'make'). 2) Download the model version you actually want from hugging face (for example, from https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGU... , with the clearly indicated required RAM for each variant) 3) Run the…

It only takes away choice if you use the demo files with the models baked in. There are versions of this under the Releases->Assets that are only the actual llama.cpp OS portable binaries that you pass the model file path to as normal. Compiling llama.cpp is relatively easy. Compiling llama.cpp for GPU support is a bit harder. I think it's nice this OS portable binaries of llama.cpp applications like main, server, an…

> Compiling llama.cpp is relatively easy. Compiling llama.cpp for GPU support is a bit harder.

It is not.

For Mac using M1 as per this specific post, you run “make”.

If you have an older Intel Mac and have to run using cpu, you run “make”.

> Downloading applications off the internet is not that weird. After all, it's the recommended way to install Rust, etc.

Downloading applications from trusted sources is not that weird.

Eg. Rust, from the rust organisation.

Downloading and running user uploaded binaries is a security nightmare.

Re: Llamafile – The easiest way to run LLMs locally on your Mac

#14

Post by paolop »

wokwokwok wrote: post #13
superkuh wrote: post #9

Earlier quoted context omitted.

It only takes away choice if you use the demo files with the models baked in. There are versions of this under the Releases->Assets that are only the actual llama.cpp OS portable binaries that you pass the model file path to as normal. Compiling llama.cpp is relatively easy. Compiling llama.cpp for GPU support is a bit harder. I think it's nice this OS portable binaries of llama.cpp applications like main, server, an…

> Compiling llama.cpp is relatively easy. Compiling llama.cpp for GPU support is a bit harder. It is not. For Mac using M1 as per this specific post, you run “make”. If you have an older Intel Mac and have to run using cpu, you run “make”. > Downloading applications off the internet is not that weird. After all, it's the recommended way to install Rust, etc. Downloading applications from trusted sources is not that w…

While in general I agree with your security concerns, here the links are from very trusted sources (Mozilla Internet Ecosystem and Mozilla's innovation group) and the user is well known (present on X too with a large following).

Re: "simplicity", sure for you and I it's simple to compile llama.cpp, but it's like asking a regular user to compile their applications themselves. It's not that simple for them, and should not be required if we want to make AI and OSS AI in particular more mainstream.

Re: Llamafile – The easiest way to run LLMs locally on your Mac

#15

Post by wokwokwok »

paolop wrote: post #14
wokwokwok wrote: post #13

Earlier quoted context omitted.

> Compiling llama.cpp is relatively easy. Compiling llama.cpp for GPU support is a bit harder. It is not. For Mac using M1 as per this specific post, you run “make”. If you have an older Intel Mac and have to run using cpu, you run “make”. > Downloading applications off the internet is not that weird. After all, it's the recommended way to install Rust, etc. Downloading applications from trusted sources is not that w…

While in general I agree with your security concerns, here the links are from very trusted sources (Mozilla Internet Ecosystem and Mozilla's innovation group) and the user is well known (present on X too with a large following). Re: "simplicity", sure for you and I it's simple to compile llama.cpp, but it's like asking a regular user to compile their applications themselves. It's not that simple for them, and should…

The command to run is:

Open terminal

curl -LO https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/ma...

chmod 755 …

./…

Open localhost:8080 in browser

To make this accessible to a broader cohort you would package it into an app and put it somewhere with provenance, eg. A well known GitHub account or App Store.

The solution, as shown, doesn’t solve either of the problems you’ve said are problems it attempts to solve.

It is a bad solution.

Re: Llamafile – The easiest way to run LLMs locally on your Mac

#16

Post by dang »

woleium wrote: post #11
dang wrote: post #8

Earlier quoted context omitted.

Thanks! Macroexpanded: Llamafile is the new best way to run a LLM on your own computer - https://news.ycombinator.com/item?id=38489533 - Dec 2023 (45 comments) Llamafile lets you distribute and run LLMs with a single file - https://news.ycombinator.com/item?id=38464057 - Nov 2023 (286 comments)

Do you think it would be useful to explain how to macroexpand whenever you do it so the folks you are responding to can learn and do it themselves next time? (myself included)

I just say "macroexpanded" as a fun metaphor. The Arc code that generates the formatted references is at https://news.ycombinator.com/item?id=35723423.

The rest consists of going through HN Search results and the relevant threads with the help of a lot of keyboard shortcuts (https://news.ycombinator.com/item?id=35668525).

One of these years I do want to make all this available!

Re: Llamafile – The easiest way to run LLMs locally on your Mac

#17

Post by paolop »

wokwokwok wrote: post #15
paolop wrote: post #14

Earlier quoted context omitted.

While in general I agree with your security concerns, here the links are from very trusted sources (Mozilla Internet Ecosystem and Mozilla's innovation group) and the user is well known (present on X too with a large following). Re: "simplicity", sure for you and I it's simple to compile llama.cpp, but it's like asking a regular user to compile their applications themselves. It's not that simple for them, and should…

The command to run is: Open terminal curl -LO https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/ma... chmod 755 … ./… Open localhost:8080 in browser To make this accessible to a broader cohort you would package it into an app and put it somewhere with provenance, eg. A well known GitHub account or App Store. The solution, as shown, doesn’t solve either of the problems you’ve said are problems it attempts to s…

Totally agreed it's not yet ideal - absolutely. But I feel we are expanding the pie of users with this step, which is just an intermediate step. Do you want to work on that packaging ;-)?

Re: Llamafile – The easiest way to run LLMs locally on your Mac

#19

Post by paolop »

gapchuboy wrote: post #18
aldarisbm wrote: post #12

Why does this keep popping up on here?

Because, people on hackernews are interested more in the prompt engineering. Convenience and satisfaction 》5 minutes of git pull and make

Agreed - and ultimately, you start removing the need to have a git app and git knowledge to pull and compile... it's not just 5 minutes, but you open up the market to way more people. Now, ideally it should just be as installing an app, but it's a good step in that direction.
Post reply on HN