Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llamafile lets you distribute and run LLMs with a single file

github.com

Post reply on HN
Print view View original

291–300 of 302 posts

Re: Llamafile lets you distribute and run LLMs with a single file

#291

Post by jart »

NoOn3 wrote: post #281
jart wrote: post #273

Earlier quoted context omitted.

So you have a Sandybridge processor with AVX support (but not AVX2). Could you open llamafile in a debugger and find out which instructing is faulting and what its address in memory is? I haven't personally tested Sandybridge but I'm reasonably certain we designed the build to not insert any incompatible instructions in there. Our intent is to support you. I also might be able to fish my old ThinkPad out of the close…

I tried but gdb showed me nothing, this screenshot( https://pxscdn.com/public/m/_v2/97422265439883264/bc40e5d2a-... ). gdb layout asm shows "No Assembly Available". Maybe I'm not so skilled in debbuging such programs it's running under wine it seemed to me.

Check out the llamafile 0.2.1 release. Old Intel CPU support is fully fixed now on the release page and hugging face. Enjoy!

Re: Llamafile lets you distribute and run LLMs with a single file

#292

Post by topato »

finnjohnsen2 wrote: post #168
reisse wrote: post #165

Earlier quoted context omitted.

What is the point of wrapping absolutely portable single-file program into a Docker container, honest question? Looks like cargo cult for me.

I see this as not polluting my OS (filesystem and processes) with bits and bobs I downloaded off the internet. The cargo cult is a clean, safe and warm space and I highly recommend it.

Ah, Ive been meaning to distro hop, from Arch,t4 GG Huover to Vanilla OS or BlendOS, for the past few weeks; I can't

Re: Llamafile lets you distribute and run LLMs with a single file

#293

Post by NoOn3 »

jart wrote: post #291
NoOn3 wrote: post #281

Earlier quoted context omitted.

I tried but gdb showed me nothing, this screenshot( https://pxscdn.com/public/m/_v2/97422265439883264/bc40e5d2a-... ). gdb layout asm shows "No Assembly Available". Maybe I'm not so skilled in debbuging such programs it's running under wine it seemed to me.

Check out the llamafile 0.2.1 release. Old Intel CPU support is fully fixed now on the release page and hugging face. Enjoy!

Thanks a lot.

Re: Llamafile lets you distribute and run LLMs with a single file

#294

Post by rightbyte »

telotortium wrote: post #287
rightbyte wrote: post #286

Earlier quoted context omitted.

I just tried and they still work (with the free ChatGPT). Jokes about Mussolini saying his traffic reforms were as successful as the invasion of Ethiopia and what not. Stalin saying that the other car drivers were "probably discussing the merits of socialism instead of driving" (a good joke!). Göring saying "at least in the Third Reich traffic worked" etc. Some sort of Monty Python tone. But you can't begin with Hitl…

I started with Stalin

I guess they are feeding us different models then?

Re: Llamafile lets you distribute and run LLMs with a single file

#295

Post by lol768 »

phh wrote: post #74
lol768 wrote: post #62

Earlier quoted context omitted.

$ chmod +x llamafile-server-0.1-llava-v1.5-7b-q4 $ ./llamafile-server-0.1-llava-v1.5-7b-q4 run-detectors: unable to find an interpreter for ./llamafile-server-0.1-llava-v1.5-7b-q4 Hmm. Did I do something wrong? (Ubuntu 22.04 / ) Installing the portable binfmt_misc gets me further, but still: $ ./llamafile-server-0.1-llava-v1.5-7b-q4 zsh: permission denied: ./llamafile-server-0.1-llava-v1.5-7b-q4 $ sh -c ./llamafile-s…

Last thing you need is to chmod +x the interpreter: chmod +x /usr/bin/ape (it is indeed not in the README)

This was it, wget creates the file and it's non-executable (and I'd already double checked the actual llamafile was executable, but had missed this) - thanks!

Re: Llamafile lets you distribute and run LLMs with a single file

#296

Post by wazoox »

brucethemoose2 wrote: post #61
estebarb wrote: post #16

Currently which are the minimum system requirements for running these models?

Basically enough to fit the download in RAM + a bit more. In practice, you kinda need a GPU, even a small one. Otherwise prompt processing is really slow.

It's really decent without any GPU. Image analysis is somewhat long, but text prompts are fine. My Ryzen laptop does 2.5 to 4 tokens per second, my Mac pro more like 8.

Re: Llamafile lets you distribute and run LLMs with a single file

#297

Post by mud_dauber »

JabavuAdams wrote: post #277
czbond wrote: post #242

Earlier quoted context omitted.

The model is trained on large volume data, correct? Why would it get such a simple fact incorrect?

LLMs are known to be bad at counting. It would be interesting to see the answer to "List the planets in our solar system, starting with the closest to the sun, and proceeding to farther and farther ones." Also, the knowledge can be kind of siloed. You often have to come at it in weird ways. Also, they are not fact-bases. They are next-token-predictors, with extra stuff on top. So if people on the internet often get t…

I just tried the same "third planet from the sun" question and got the correct response. No other training or tweaks.

Can't wait to unleash Pluto questions.

Re: Llamafile lets you distribute and run LLMs with a single file

#298

Post by ametrau »

mediaman wrote: post #282
hnuser123456 wrote: post #221

Earlier quoted context omitted.

Source on the newer GPT-4 model being worse at coding?

Several OpenAI employees have said on Twitter that they are looking into this and developing a fix. It sounds as though it was not an intentional regression since they are implicitly acknowledging it. Could be an unintentional side effect of something else. I'd expect we see improved behavior in the coming weeks.

Could you link to tweet?

Re: Llamafile lets you distribute and run LLMs with a single file

#299

Post by stavros »

jart wrote: post #98
stavros wrote: post #97

Earlier quoted context omitted.

That worked, thanks Justine! I use fish, so I didn't get a zsh error, but I had missed the Gotchas section (and the README), so this helps!

Fish is another cool shell I got to help improve two years ago by upstreaming a patch for this. So long as you're using a recent version, you should be golden (provided binfmt_misc doesn't cause any issues). Let us know what you think of llamafile!

Thank you, I really like it! It's a very clever way to get LLMs deployed, and with Cosmopolitan, I don't need to point people to different downloads for the same LLM. Excellent job.

Re: Llamafile lets you distribute and run LLMs with a single file

#300

Post by dmazzoni »

sebmellen wrote: post #261
epaga wrote: post #189

Hmmm no matter what image I upload, it's somehow not seeing it. Got a chuckle out of this, though: --- User: Describe the image in the most detailed way possible Llama: The image is a close-up of an empty, dark room with no visible objects or people. It appears to be a black background photo that highlights the darkness and emptiness of the space. This could potentially represent an artistic representation or an actu…

Interesting, I uploaded a pure white 50x50 png and got this: > User: Describe this image in the most detailed way possible > Llama: The image displays a large, mostly white background with some small gray areas. There is also an orange dot on the right side of the scene, which stands out against the predominantly white and gray backdrop. > User: Is it just a white square? > Llama: No, it is not just a white square. T…

LLM vision is surprisingly human-like. Point an actual human at a blank canvas and I'll bet many would hallucinate things that aren't there.
Post reply on HN