Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile Returns

blog.mozilla.ai

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Llamafile Returns

#11

Post by thangalin »

Tips:

    # Avoid issues when wine is installed.
    sudo su -c 'echo 0 > /proc/sys/fs/binfmt_misc/status'
And:

    # Capture the entirety of the instructions to obtain the input length.
    readonly INSTRUCT=$(
      join ${PATH_PREFIX_SYSTEM} ${PATH_PROMPT_SYSTEM} ${PATH_PREFIX_SYSTEM}
      join ${PATH_SUFFIX_USER} ${PATH_PROMPT_USER} ${PATH_SUFFIX_USER}
      join ${PATH_SUFFIX_ASSIST} "/dev/null" ${PATH_SUFFIX_ASSIST}
    )

    (
      echo ${INSTRUCT}
    ) | ./llamafile \
      -m "${LINK_MODEL}" \
      -e \
      -f /dev/stdin \
      -n 1000 \
      -c ${#INSTRUCT} \
      --repeat-penalty 1.0 \
      --temp 1.5 \
      --silent-prompt > output.txt

Re: Llamafile Returns

#12

Post by michaelgiba »

I’m glad to see llamafile being resurrected. A few things I hope for:

1. Curate a continuously extended inventory of prebuilt llamafiles for models as they are released 2. Create both flexible builds (with dynamic backend loading for cpu and cuda) and slim minimalist builds 3. Upstreaming as much as they can into llama.cpp and partner with the project

Re: Llamafile Returns

#13

Post by michaelgiba »

michaelgiba wrote: post #12

I’m glad to see llamafile being resurrected. A few things I hope for: 1. Curate a continuously extended inventory of prebuilt llamafiles for models as they are released 2. Create both flexible builds (with dynamic backend loading for cpu and cuda) and slim minimalist builds 3. Upstreaming as much as they can into llama.cpp and partner with the project

Crazier ideas would be: - extend the concept to also have some sort of “agent mode” where the llamafiles can launch with their own minimal file system or isolated context - detailed profiling of main supported models to ensure deterministic outputs

Re: Llamafile Returns

#14

Post by njbrake »

michaelgiba wrote: post #13
michaelgiba wrote: post #12

I’m glad to see llamafile being resurrected. A few things I hope for: 1. Curate a continuously extended inventory of prebuilt llamafiles for models as they are released 2. Create both flexible builds (with dynamic backend loading for cpu and cuda) and slim minimalist builds 3. Upstreaming as much as they can into llama.cpp and partner with the project

Crazier ideas would be: - extend the concept to also have some sort of “agent mode” where the llamafiles can launch with their own minimal file system or isolated context - detailed profiling of main supported models to ensure deterministic outputs

Love the idea!

Re: Llamafile Returns

#16

Post by benatkin »

> As the local and open LLM ecosystem has evolved over the years, time has come for llamafile to evolve too. It needs refactoring and upgrades to incorporate newer features available in llama.cpp and develop a refined understanding of the most valuable features for its users.

It seems people have moved on from Llamafile. I doubt Mozilla AI is going to bring it back.

This announcement didn't even come with a new code commit, just a wish. https://github.com/mozilla-ai/llamafile/commits/main/

Re: Llamafile Returns

#17

Post by setheron »

rvz wrote: post #8
bsenftner wrote: post #7

Earlier quoted context omitted.

People are so uninformed, they don't know you are Mozilla AI's star employee.

s/are/was I don't know if you were informed but you realize that jart is no longer at Mozilla anymore and now at Google Inc?

jart, you are back at Google?

Re: Llamafile Returns

#18

Post by setheron »

bsenftner wrote: post #7
jart wrote: post #2

Really exciting to see Mozilla AI starting up and I can't wait to see where the next generation takes the project!

People are so uninformed, they don't know you are Mozilla AI's star employee.

I agree that the work they put out is A+ Whenever I see content produced by jart, it's always amazing.

Re: Llamafile Returns

#19

Post by chrismorgan »

thangalin wrote: post #11

Tips: # Avoid issues when wine is installed. sudo su -c 'echo 0 > /proc/sys/fs/binfmt_misc/status' And: # Capture the entirety of the instructions to obtain the input length. readonly INSTRUCT=$( join ${PATH_PREFIX_SYSTEM} ${PATH_PROMPT_SYSTEM} ${PATH_PREFIX_SYSTEM} join ${PATH_SUFFIX_USER} ${PATH_PROMPT_USER} ${PATH_SUFFIX_USER} join ${PATH_SUFFIX_ASSIST} "/dev/null" ${PATH_SUFFIX_ASSIST} ) ( echo ${INSTRUCT} ) | ./…

> # Avoid issues when wine is installed.

> sudo su -c 'echo 0 > /proc/sys/fs/binfmt_misc/status'

Please don’t recommend this. If binfmt_misc is enabled, it’s probably for a reason, and disabling it will break things. I have a .NET/Mono app installed that it would break, for example—it’s definitely not just Wine.

If binfmt_misc is causing problems, the proper solution is to register the executable type. https://github.com/mozilla-ai/llamafile#linux describes steps.

I made myself a package containing /usr/bin/ape and the following /usr/lib/binfmt.d/ape.conf:

  :APE:M::MZqFpD::/usr/bin/ape:
  :APE-jart:M::jartsr::/usr/bin/ape:

Re: Llamafile Returns

#20

Post by dolmen »

Cosmocc and Cosmopolitan are remarkable technical achievements and llamafile made me discover them.

The llamafile UX (CLI interface and web server with chat to quickly interact with the model) is great and make easy to download and play with a local LLM.

However I fail to see use cases where I would build a solution built on a llamafile. If I want to play with multiple models, I don't need to have the binary attached to the model data. If I want to play with a model on multiple operating systems, I'm fine downloading the llamafile tool binary for the platform separately from the model data (in fact, on Windows one have to download the llamafile.exe separately anyway because of a limit of the OS for executable files).

So Cosmopolitan is great tech, the llamafile command (the "UX for a model" part) is great, but I'm not convinced by the value of Cosmopolitan applied here.

Post reply on HN