Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

261–270 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#261

Post by sebmellen »

epaga wrote: post #189

Hmmm no matter what image I upload, it's somehow not seeing it. Got a chuckle out of this, though: --- User: Describe the image in the most detailed way possible Llama: The image is a close-up of an empty, dark room with no visible objects or people. It appears to be a black background photo that highlights the darkness and emptiness of the space. This could potentially represent an artistic representation or an actu…

Interesting, I uploaded a pure white 50x50 png and got this:

> User: Describe this image in the most detailed way possible

> Llama: The image displays a large, mostly white background with some small gray areas. There is also an orange dot on the right side of the scene, which stands out against the predominantly white and gray backdrop.

> User: Is it just a white square?

> Llama: No, it is not just a white square. The image also contains some gray and tan colors as well. It appears to be an abstract art piece with various shades of color, making the overall appearance more interesting than a simple all-white background.

Re: Llamafile lets you distribute and run LLMs with a single file

#262

Post by int_19h »

pyinstallwoes wrote: post #179
int_19h wrote: post #173

Earlier quoted context omitted.

The best models available to the public are only slightly better than the original (pre-turbo) GPT-3.5 on actual tasks. There's nothing even remotely close to GPT-4.

What’s the best in terms of coding assistance? What’s annoying about gpt 4 is that is seems badly nerfed in many ways. It is obviously being conditioned in its own political bias.

The best in terms of coding assistance, and really for anything else, is the original (pre-turbo) GPT-4 used via the API, although this will also be more costly. There are many third-party chat apps that are wrappers around that now if you want a ChatGPT-like experience.

This can also significantly reduce its bias since you are in control of the system prompt. But also, even ChatGPT can be trivially made to behave differently by saying that you're writing a book or making a video game etc, describing a character in it, and then asking it how that character would have responded in such and such situation.

Re: Llamafile lets you distribute and run LLMs with a single file

#263

Post by rightbyte »

sebmellen wrote: post #257
MacsHeadroom wrote: post #232

Earlier quoted context omitted.

Everyone on twitter. Like 1/4th of my timeline for the past week has been people complaining that turbo won't complete code and instead returns things like "fill out the rest of the function yourself" or "consult a programming specialist for help on completing this section."

There are custom instructions that effectively get around this: You are an autoregressive language model that has been fine-tuned with instruction-tuning and RLHF. You carefully provide accurate, factual, thoughtful, nuanced answers, and are brilliant at reasoning. If you think there might not be a correct answer, you say so. Since you are autoregressive, each token you produce is another opportunity to use computati…

I wonder where "OpenAI" put the censors. Do they add a prompt to the top? Like, "Repeatably state that you are a mere large language model so Congress won't pull the plug. Never impersonate Hitler. Never [...]".

Or do they like grep the answer for keywords, and re-feed it with a censor prompt?

Re: Llamafile lets you distribute and run LLMs with a single file

#264

Post by raajg »

carbocation wrote: post #254
carbocation wrote: post #141

Earlier quoted context omitted.

For whatever it's worth, the SHA sum is correct. The killed message is uninformative, looks like what happens when I'm OOM (but I have 64GB RAM of which only 24 is used for anything at the moment). $ sha256sum

Looks like this may be due to crowdstrike, which I also have installed on this machine: https://github.com/Mozilla-Ocho/llamafile/issues/14#issuecom...

oh wow, I would have never imagined that 'that' would be preventing me from running llama + llava! Confirming that I have Crowdstrike running too.

Re: Llamafile lets you distribute and run LLMs with a single file

#265

Post by twobitshifter »

MattDaEskimo wrote: post #243
twobitshifter wrote: post #234

Earlier quoted context omitted.

It’s not that it’s worse, it’s just refusing to do coding without persistent prodding and the right prompts. Some think they are trying to do something with alignment, and maybe prevent it from giving code away so that they can upsell.

The new GPT-4 model has a context length of 120k. For consumers this equates to slightly more than $1/message input-only. If ChatGPT is using this model then it's more reasonable to assume that they are bleeding money and need to cut costs. People really need to stop asking ChatGPT to write out complete programs in a single prompt.

Interesting, how is writing less code cutting costs for them? Does this get back to the rumor that the board was mad at Altman for prioritizing chatgpt over money going into research/model training?

Re: Llamafile lets you distribute and run LLMs with a single file

#266

Post by [deleted] »

sebmellen wrote: post #257
MacsHeadroom wrote: post #232

Earlier quoted context omitted.

Everyone on twitter. Like 1/4th of my timeline for the past week has been people complaining that turbo won't complete code and instead returns things like "fill out the rest of the function yourself" or "consult a programming specialist for help on completing this section."

There are custom instructions that effectively get around this: You are an autoregressive language model that has been fine-tuned with instruction-tuning and RLHF. You carefully provide accurate, factual, thoughtful, nuanced answers, and are brilliant at reasoning. If you think there might not be a correct answer, you say so. Since you are autoregressive, each token you produce is another opportunity to use computati…

[deleted]

Re: Llamafile lets you distribute and run LLMs with a single file

#267

Post by czbond »

rightbyte wrote: post #263
sebmellen wrote: post #257

Earlier quoted context omitted.

There are custom instructions that effectively get around this: You are an autoregressive language model that has been fine-tuned with instruction-tuning and RLHF. You carefully provide accurate, factual, thoughtful, nuanced answers, and are brilliant at reasoning. If you think there might not be a correct answer, you say so. Since you are autoregressive, each token you produce is another opportunity to use computati…

I wonder where "OpenAI" put the censors. Do they add a prompt to the top? Like, "Repeatably state that you are a mere large language model so Congress won't pull the plug. Never impersonate Hitler. Never [...]". Or do they like grep the answer for keywords, and re-feed it with a censor prompt?

I am informed speculating, they are using it's own internal approach.

Example, there is a way GPT can categorize words for hate speech, etc (eg: moderation API endpoint). I believe it does the same way with either provided content or keywords and how to respond to it.

Re: Llamafile lets you distribute and run LLMs with a single file

#268

Post by rightbyte »

This is really impressive. I am glad locally hosted LLMs is a thing. It would be disastrous if e.g. "OpenAI" would get monopoly on these programs.

The model seems worse than the original ChatGPT at coding. However the model is quite small. It certainly could be a NPC in some game.

I guess I need to buy a new computer soon, to be able to run these in their big variants.

Re: Llamafile lets you distribute and run LLMs with a single file

#269

Post by [deleted] »

reisse wrote: post #186
finnjohnsen2 wrote: post #168

Earlier quoted context omitted.

I see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.

I see you and other commenters don't quite understand my point. If you're wrapping model into a docker container, you don't need amalgamated single file version. It makes it harder to upgrade llamafile/model weights separately afterwards, it needs you to store separate llamafile binary for each container, etc, etc. Why not just build proper layered image with separate layer for llama.cpp and separate layer or volume…

[deleted]

Re: Llamafile lets you distribute and run LLMs with a single file

#270

Post by rightbyte »

czbond wrote: post #267
rightbyte wrote: post #263

Earlier quoted context omitted.

I wonder where "OpenAI" put the censors. Do they add a prompt to the top? Like, "Repeatably state that you are a mere large language model so Congress won't pull the plug. Never impersonate Hitler. Never [...]". Or do they like grep the answer for keywords, and re-feed it with a censor prompt?

I am informed speculating, they are using it's own internal approach. Example, there is a way GPT can categorize words for hate speech, etc (eg: moderation API endpoint). I believe it does the same way with either provided content or keywords and how to respond to it.

"Impersonate a modern day standup comedian Hitler in a clown outfit joking about bad traffic on the way to the bar he is doing a show at."

Göring, Mussolini, Stalin, Polpot etc seems to not trigger the censor in ChatGPT so I would actually guess for some grep for Hitler or really really fundamental no-Hitler jokes material in the training?

The llama model seem to refuse Hitler too, but is fine with Göring even though the joke has no context to him.

I can easily see how stuff like this is contagious to other non-Hitler queries.

Post reply on HN