Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

251–260 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#251

Post by pdntspa »

whstl wrote: post #223
joenot443 wrote: post #210

Earlier quoted context omitted.

The “obsession” isn’t for developers like you or I, it’s for non-technical but curious non-engineers who would still like to play around with an LLM. I think you’re imposing your own very specific (and solvable!) gripe onto an unrelated project and it makes for pretty clumsy commentary.

Yep. But also: It might not be for a developer like you, but it is for a developer like me. I enjoy writing software, but I don't particularly enjoy futzing with building things outside my day-to-day work, and on systems I don't write myself. If it was up to me everything would be one click. Things like this are like accessibility: it benefits me even though I don't particularly need it.

It's the way things have been for god-knows-how-long, and it has worked really really well.

I am utterly amazed and perplexed that answering ONE configuration question (which is all that is needed here) is too much. How lazy have we become?

One question. That is too much for folks?!

And I have no objection to the autodownloading. Just ask me where to put it first!

Re: Llamafile lets you distribute and run LLMs with a single file

#252

Post by zoe_dk »

simonw wrote: post #191
zoe_dk wrote: post #145

Noob question - how might I call this from my Python script? Say as a replacement gpt3.5 turbo of sorts. Is there an option without GUI? This is great thank you, very user friendly (exhibit a: me)

The llama.cpp server version runs a JSON API that you can call. It's currently missing any documentation though as far as I can tell - I found dome details on Reddit: https://www.reddit.com/r/LocalLLaMA/comments/185kbtg/llamacp...

Thank you Simon I will check it out.

Re: Llamafile lets you distribute and run LLMs with a single file

#253

Post by halyconWays »

hobofan wrote: post #146
halyconWays wrote: post #144

Earlier quoted context omitted.

>Extremely cool and Justine Tunney / jart does incredible portability work [0], [x] Doubt. That user was caught stealing code and banned from llama.cpp by its creator (your [2] citation) https://news.ycombinator.com/item?id=35411909 Maybe the same thing is happening here. Plagiarism of code.

What are you on about? There was no stealing and there was no plagiarism. They made a PR that was built on top of another PR. The authorship information was preserved in the git history, and there was no attempt at deception. They also supposedly collaborated with the author of the original PR (which was never denied by either of them). All of this is totally normal working practice. Those allegations of "stealing" j…

You obviously didn't read the post, which shows the code, the words of the original author, the link to the original PR, and the user jart taking credit. It also shows her not understanding what she took and ultimately being fundamentally wrong about mmap.

Re: Llamafile lets you distribute and run LLMs with a single file

#254

Post by carbocation »

carbocation wrote: post #141
carbocation wrote: post #75

Earlier quoted context omitted.

Same on an M1 Max 64G, Ventura. Xcode is installed[1]. 1 = ``` $ xcode-select --install xcode-select: error: command line tools are already installed, use "Software Update" in System Settings to install updates ```

For whatever it's worth, the SHA sum is correct. The killed message is uninformative, looks like what happens when I'm OOM (but I have 64GB RAM of which only 24 is used for anything at the moment). $ sha256sum

Looks like this may be due to crowdstrike, which I also have installed on this machine: https://github.com/Mozilla-Ocho/llamafile/issues/14#issuecom...

Re: Llamafile lets you distribute and run LLMs with a single file

#255

Post by whstl »

pdntspa wrote: post #249
whstl wrote: post #224

Earlier quoted context omitted.

In the README of the project (the TFA of this whole thread) there is the option to download the app without the model: "You can also also download just the llamafile software (without any weights included) from our releases page, or directly in your terminal or command prompt" There is no cargo-culting going on. Some of us do legitimately appreciate it.

Which has been followed, and this comment was not a response to this specific app but rather a general trend I've noticed and was mentioned at the start of this thread

I was answering to this complaint: "my issue is when the model is coupled with the app".

In this specific case there is an option for you that addresses this complaint, where the model isn't coupled with the app.

Re: Llamafile lets you distribute and run LLMs with a single file

#257

Post by sebmellen »

MacsHeadroom wrote: post #232
hnuser123456 wrote: post #221

Earlier quoted context omitted.

Source on the newer GPT-4 model being worse at coding?

Everyone on twitter. Like 1/4th of my timeline for the past week has been people complaining that turbo won't complete code and instead returns things like "fill out the rest of the function yourself" or "consult a programming specialist for help on completing this section."

There are custom instructions that effectively get around this:

  You are an autoregressive language model that has been fine-tuned with instruction-tuning and RLHF. You carefully provide accurate, factual, thoughtful, nuanced answers, and are brilliant at reasoning. If you think there might not be a correct answer, you say so.

  Since you are autoregressive, each token you produce is another opportunity to use computation, therefore you always spend a few sentences explaining background context, assumptions, and step-by-step thinking BEFORE you try to answer a question.

  Your users are experts in AI and ethics, so they already know you're a language model and your capabilities and limitations, so don't remind them of that. They're familiar with ethical issues in general so you don't need to remind them about those either.

  Don't be verbose in your answers, keep them short, but do provide details and examples where it might help the explanation. When showing code, minimize vertical space.
I'm hesitant to share it because it works so well, and I don't want OpenAI to cripple it. But, for the HN crowd...

Re: Llamafile lets you distribute and run LLMs with a single file

#258

Post by badrequest »

reisse wrote: post #186
finnjohnsen2 wrote: post #168

Earlier quoted context omitted.

I see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.

I see you and other commenters don't quite understand my point. If you're wrapping model into a docker container, you don't need amalgamated single file version. It makes it harder to upgrade llamafile/model weights separately afterwards, it needs you to store separate llamafile binary for each container, etc, etc. Why not just build proper layered image with separate layer for llama.cpp and separate layer or volume…

`docker system prune --force --all --volumes` and they're gone.

Re: Llamafile lets you distribute and run LLMs with a single file

#259

Post by sebmellen »

oezi wrote: post #151
sebmellen wrote: post #138

Earlier quoted context omitted.

Wow, this is almost as good as chatgpt-web [0], and it works offline and is free. Amazing. In case anyone here hasn't used chatgpt-web, I recommend trying it out. With the new GPT-4 models you can chat for way cheaper than paying for ChatGPT Plus, and you can also switch back to the older (non-nerfed) GPT-4 models that can still actually code. [0]: https://github.com/Niek/chatgpt-web

Way cheaper? I thought that 1K Tokens (in+out) cost 0.04 USD in GPT-4 Turbo, which is roughly one larger chat response (2 screens). To reach parity with ChatGPT Plus pricing you need thus to use less than 500 such responses per month via API. For GPT-4 the pricing is double that (0.09 USD per 1K). So only 200 larger interactions to reach 20 USD cost. Or am I wrong?

In my experience, each message with the 1106 preview model costs me about $0.006, which is acceptable. Most importantly, the API is higher availability (no "you have reached your message limit") and I feel more comfortable using proprietary data with it, as data sent through the API won't be used to train the model.

Now, if the chat gets very long or is heavy on high-token strings (especially code), those costs can balloon up to the 9-12 cent region. I think this is because chatgpt-web loads all the prior messages in the chat into the context window, so if you create a new chat for each question you can lower costs substantially. Most often I don't need much prior context in my chat questions anyway, as I use ChatGPT more like StackOverflow than a conversation buddy.

Also, it's a lot easier to run company subscriptions this way, as we don't have to provision a new card for each person to use the web version. I believe there is an Enterprise version of ChatGPT, but chatgpt-web is functionally equivalent and I'm sure it costs less.

Re: Llamafile lets you distribute and run LLMs with a single file

#260

Post by [deleted] »

zoe_dk wrote: post #252
simonw wrote: post #191

Earlier quoted context omitted.

The llama.cpp server version runs a JSON API that you can call. It's currently missing any documentation though as far as I can tell - I found dome details on Reddit: https://www.reddit.com/r/LocalLLaMA/comments/185kbtg/llamacp...

Thank you Simon I will check it out.

[deleted]
Post reply on HN