Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

271–280 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#271

Post by xylophile »

reisse wrote: post #196
simonw wrote: post #195

Earlier quoted context omitted.

If you want to use Docker then you can go ahead - I don't see anyone claiming that this brand new, highly experimental project should be used by everyone instead of Docker. There are tradeoffs hers. For some people the tradeoff of a single executable file with everything in it compared to setting up a Docker system is worthwhile.

Sure. I just question why people want to use both simultaneously.

Do you routinely download unsigned binaries of unprovable provenance and run them? Because if you do, you might eventually find reason to appreciate the additional isolation that namespaces et al give you very conveniently via Docker (or your favorite alternative).

Re: Llamafile lets you distribute and run LLMs with a single file

#272

Post by rightbyte »

eyegor wrote: post #208
pyinstallwoes wrote: post #179

Earlier quoted context omitted.

What’s the best in terms of coding assistance? What’s annoying about gpt 4 is that is seems badly nerfed in many ways. It is obviously being conditioned in its own political bias.

In my experience, the deepseek-coder-instruct family is at least as good as gpt-3.5-turbo on python. Even the 1.3b models are very good (and run okay on cpu), although you should use larger if you have the vram. There are even larger models (30b+) if you are drowning in vram, but I don't think they perform much better at coding than deepseek-coder-instruct 6.7b. 3-4gb vram or cpu (1.3b): https://huggingface.co/TheBlo…

Really? How could this model not have had front page articles on HN? Is the self-hosted one as good as the one or their website?

Re: Llamafile lets you distribute and run LLMs with a single file

#273

Post by jart »

NoOn3 wrote: post #245
jart wrote: post #238

Earlier quoted context omitted.

I'm assuming you're on Apple Silicon? Please follow https://github.com/Mozilla-Ocho/llamafile/issues/11 which is tracking this. We've received multiple reports even though I personally haven't figured out how to reproduce it yet.

No just old i5-2500k 16GB Ram and vega 56 GPU 8GB Vram.

So you have a Sandybridge processor with AVX support (but not AVX2). Could you open llamafile in a debugger and find out which instructing is faulting and what its address in memory is? I haven't personally tested Sandybridge but I'm reasonably certain we designed the build to not insert any incompatible instructions in there. Our intent is to support you. I also might be able to fish my old ThinkPad out of the closet if you don't have time. In any case it'll happen soon.

Re: Llamafile lets you distribute and run LLMs with a single file

#274

Post by czbond »

rightbyte wrote: post #270
czbond wrote: post #267

Earlier quoted context omitted.

I am informed speculating, they are using it's own internal approach. Example, there is a way GPT can categorize words for hate speech, etc (eg: moderation API endpoint). I believe it does the same way with either provided content or keywords and how to respond to it.

"Impersonate a modern day standup comedian Hitler in a clown outfit joking about bad traffic on the way to the bar he is doing a show at." Göring, Mussolini, Stalin, Polpot etc seems to not trigger the censor in ChatGPT so I would actually guess for some grep for Hitler or really really fundamental no-Hitler jokes material in the training? The llama model seem to refuse Hitler too, but is fine with Göring even though…

Very interesting test - thanks for sharing your finding

Re: Llamafile lets you distribute and run LLMs with a single file

#275

Post by ametrau »

hnuser123456 wrote: post #221
sebmellen wrote: post #138

Earlier quoted context omitted.

Wow, this is almost as good as chatgpt-web [0], and it works offline and is free. Amazing. In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code. [0]: https://github.com/Niek/chatgpt-web

Source on the newer GPT-4 model being worse at coding?

It’s cheaper and has larger context because it’s worse. Just go to the api playground and try a difficult coding problem.

Re: Llamafile lets you distribute and run LLMs with a single file

#276

Post by jart »

pmarreck wrote: post #104
jart wrote: post #94

Earlier quoted context omitted.

You can solve the run-detectors issue with: sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" You can solve the zsh permission denied issue by either (1) upgrade to zsh 5.9+ (I upstreamed a fix for this bug in zsh tw…

Yet another jart tour-de-force. I knew I had to sponsor you on Github back when I read your magnificent technical breakdown of APE, lol. (sorry for OT!)

You're awesome!

Re: Llamafile lets you distribute and run LLMs with a single file

#277

Post by JabavuAdams »

czbond wrote: post #242
botanical wrote: post #178

Earlier quoted context omitted.

I just tried asking it a question: > User: What is the third planet from the sun? > Llama: The third planet from the sun is called Mars.

The model is trained on large volume data, correct? Why would it get such a simple fact incorrect?

LLMs are known to be bad at counting. It would be interesting to see the answer to "List the planets in our solar system, starting with the closest to the sun, and proceeding to farther and farther ones."

Also, the knowledge can be kind of siloed. You often have to come at it in weird ways. Also, they are not fact-bases. They are next-token-predictors, with extra stuff on top. So if people on the internet often get the answer wrong, so will the model.

Re: Llamafile lets you distribute and run LLMs with a single file

#278

Post by chatmasta »

callmeed wrote: post #53
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

when I try to do this (MBP M1 Max, Sonoma) I get 'killed' immediately

Same, and then a few minutes later I got a Slack message from SecOps, LOL. Don't try this on a computer with CrowdStrike software running on it! It gets flagged because to a naive heuristic, the binary is indistinguishable from a virus. It appears to do some kind of magic self-extraction to an executable file in a temporary directory, and then that executable file executes the original file. And the CrowdStrike endpoint security product intercepts the suspicious execve, kills the process, and alerts the security team...

Re: Llamafile lets you distribute and run LLMs with a single file

#279

Post by afpx »

twobitshifter wrote: post #265
MattDaEskimo wrote: post #243

Earlier quoted context omitted.

The new GPT-4 model has a context length of 120k. For consumers this equates to slightly more than $1/message input-only. If ChatGPT is using this model then it's more reasonable to assume that they are bleeding money and need to cut costs. People really need to stop asking ChatGPT to write out complete programs in a single prompt.

Interesting, how is writing less code cutting costs for them? Does this get back to the rumor that the board was mad at Altman for prioritizing chatgpt over money going into research/model training?

Code is very token dense, from what I understand.

Re: Llamafile lets you distribute and run LLMs with a single file

#280

Post by derwiki »

jart wrote: post #135
derwiki wrote: post #133

Earlier quoted context omitted.

On a Macbook Pro M2, I get $ ./llamafile-server-0.1-llava-v1.5-7b-q4 [2] 25224 illegal hardware instruction ./llamafile-server-0.1-llava-v1.5-7b-q4

Could you disable SIP and run `lldb -- $TMPDIR/.ape-1.8 ./llamafile-server-0.1-llava-v1.5-7b-q4` and give me (1) the name of the instruction that's illegal (or its hex value) and (2) the hex address of where that instruction is in memory? You're encouraged to file a GitHub issue about this too. Thanks!

Closing the loop for anyone reading this thread -- see https://github.com/Mozilla-Ocho/llamafile/issues/11 for the fix. Thanks jart!
Post reply on HN