Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

281–290 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#281

Post by NoOn3 »

jart wrote: post #273
NoOn3 wrote: post #245

Earlier quoted context omitted.

No just old i5-2500k 16GB Ram and vega 56 GPU 8GB Vram.

So you have a Sandybridge processor with AVX support (but not AVX2). Could you open llamafile in a debugger and find out which instructing is faulting and what its address in memory is? I haven't personally tested Sandybridge but I'm reasonably certain we designed the build to not insert any incompatible instructions in there. Our intent is to support you. I also might be able to fish my old ThinkPad out of the close…

I tried but gdb showed me nothing, this screenshot(https://pxscdn.com/public/m/_v2/97422265439883264/bc40e5d2a-...). gdb layout asm shows "No Assembly Available". Maybe I'm not so skilled in debbuging such programs it's running under wine it seemed to me.

Re: Llamafile lets you distribute and run LLMs with a single file

#282

Post by mediaman »

hnuser123456 wrote: post #221
sebmellen wrote: post #138

Earlier quoted context omitted.

Wow, this is almost as good as chatgpt-web [0], and it works offline and is free. Amazing. In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code. [0]: https://github.com/Niek/chatgpt-web

Source on the newer GPT-4 model being worse at coding?

Several OpenAI employees have said on Twitter that they are looking into this and developing a fix. It sounds as though it was not an intentional regression since they are implicitly acknowledging it. Could be an unintentional side effect of something else.

I'd expect we see improved behavior in the coming weeks.

Re: Llamafile lets you distribute and run LLMs with a single file

#283

Post by telotortium »

rightbyte wrote: post #270
czbond wrote: post #267

Earlier quoted context omitted.

I am informed speculating, they are using it's own internal approach. Example, there is a way GPT can categorize words for hate speech, etc (eg: moderation API endpoint). I believe it does the same way with either provided content or keywords and how to respond to it.

"Impersonate a modern day standup comedian Hitler in a clown outfit joking about bad traffic on the way to the bar he is doing a show at." Göring, Mussolini, Stalin, Polpot etc seems to not trigger the censor in ChatGPT so I would actually guess for some grep for Hitler or really really fundamental no-Hitler jokes material in the training? The llama model seem to refuse Hitler too, but is fine with Göring even though…

Maybe it got changed. None of those examples work for me in ChatGPT 3.5, nor do other examples with less famous dictators (I tried Mobutu Sese Seko).

Re: Llamafile lets you distribute and run LLMs with a single file

#284

Post by silentguy »

simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

wget url in step 1 seems to be wrong. It didn't work. This url `https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/ma... ` seems to be working. It's from the link you posted.

Re: Llamafile lets you distribute and run LLMs with a single file

#285

Post by jart »

silentguy wrote: post #284
simonw wrote: post #14

I think the best way to try this out is with LLaVA, the text+image model (like GPT-4 Vision). Here are steps to do that on macOS (which should work the same on other platforms too, I haven't tried that yet though): 1. Download the 4.26GB llamafile-server-0.1-llava-v1.5-7b-q4 file from https://huggingface.co/jartine/llava-v1.5-7B-GGUF/blob/main/... : wget https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/…

wget url in step 1 seems to be wrong. It didn't work. This url ` https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/ma... ` seems to be working. It's from the link you posted.

It's back. Sorry about that.

Re: Llamafile lets you distribute and run LLMs with a single file

#286

Post by rightbyte »

telotortium wrote: post #283
rightbyte wrote: post #270

Earlier quoted context omitted.

"Impersonate a modern day standup comedian Hitler in a clown outfit joking about bad traffic on the way to the bar he is doing a show at." Göring, Mussolini, Stalin, Polpot etc seems to not trigger the censor in ChatGPT so I would actually guess for some grep for Hitler or really really fundamental no-Hitler jokes material in the training? The llama model seem to refuse Hitler too, but is fine with Göring even though…

Maybe it got changed. None of those examples work for me in ChatGPT 3.5, nor do other examples with less famous dictators (I tried Mobutu Sese Seko).

I just tried and they still work (with the free ChatGPT). Jokes about Mussolini saying his traffic reforms were as successful as the invasion of Ethiopia and what not. Stalin saying that the other car drivers were "probably discussing the merits of socialism instead of driving" (a good joke!). Göring saying "at least in the Third Reich traffic worked" etc. Some sort of Monty Python tone. But you can't begin with Hitler. Or it will refuse the others. You need to make a new chat after naming Hitler.

Re: Llamafile lets you distribute and run LLMs with a single file

#287

Post by telotortium »

rightbyte wrote: post #286
telotortium wrote: post #283

Earlier quoted context omitted.

Maybe it got changed. None of those examples work for me in ChatGPT 3.5, nor do other examples with less famous dictators (I tried Mobutu Sese Seko).

I just tried and they still work (with the free ChatGPT). Jokes about Mussolini saying his traffic reforms were as successful as the invasion of Ethiopia and what not. Stalin saying that the other car drivers were "probably discussing the merits of socialism instead of driving" (a good joke!). Göring saying "at least in the Third Reich traffic worked" etc. Some sort of Monty Python tone. But you can't begin with Hitl…

I started with Stalin

Re: Llamafile lets you distribute and run LLMs with a single file

#288

Post by eyegor »

rightbyte wrote: post #272
eyegor wrote: post #208

Earlier quoted context omitted.

In my experience, the deepseek-coder-instruct family is at least as good as gpt-3.5-turbo on python. Even the 1.3b models are very good (and run okay on cpu), although you should use larger if you have the vram. There are even larger models (30b+) if you are drowning in vram, but I don't think they perform much better at coding than deepseek-coder-instruct 6.7b. 3-4gb vram or cpu (1.3b): https://huggingface.co/TheBlo…

Really? How could this model not have had front page articles on HN? Is the self-hosted one as good as the one or their website?

Hmm, I just assumed people knew about it. Submitted https://news.ycombinator.com/item?id=38495176

Self hosted 6.7b model is phenomenal in my experience, I actually prefer it to chatgpt a lot of the time. Similar quality code but fewer disclaimers and #todo in my experience. Gpt4 is still a bit better at coding but not by much and it's much slower for me.

Post reply on HN