Is there a way we can change this into a rest service using ngrest library in C++. I am trying but looks like there is a lot to unpack in the Chat.cpp code and am not able to proceed.
Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
291–300 of 303 posts
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#292Ok not bad and pretty damn good performance: > Give me a writing prompt for a sci-fi story. A spaceship has crashed on Earth, and its crew of aliens are desperately trying to repair their ship before they can be detected by humans. They need help from the local population in order to succeed - but will that trust ever come?
I like "> at the end of the universe, two powerful alien species battle for..."
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#293Earlier quoted context omitted.
I think the LLaMA-7B in general might not just be very good. I've been playing around and run full non-quantized LLaMA-30B and LLaMA-7B in a bunch of experiments and I think the quality of output is much, much better in LLaMA-30B.
What kind of hardware is necessary to run non-quantized LLama-30B?
I am running fp16 LLaMA 30B (via vanilla-llama) on six AMD MI25s. Computer has 384 GB of RAM but the model fits in the VRAM. It takes up about 87 GB of VRAM out of the 96 GB available on the six cards. Performance is about 1.6 words per second in an IRC chat log continuation task and it pulls about 400W additional when "thinking."
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#294Earlier quoted context omitted.
What, nothing about Electron's unacceptable code bloat factor? Most disappointing! https://www.hendrik-erz.de/post/electron-bloated-web-and-tra... A GPT-4-on-GPT-4 flame war might be an incredible spectacle...
I asked GPT-4 to produce a simulated argument about Electron which devolves into name calling. Here's the 2 page of result it returned: User1: Hey, have you heard about Electron? It's an awesome framework for creating cross-platform apps using web technologies like HTML, CSS, and JavaScript. User2: Yeah, I've heard of it. But I think it has some pretty serious shortcomings. Like, it's notorious for consuming a lot of…
> I asked GPT-4 to produce a simulated argument about Electron which devolves into name calling. Here's the 2 page of result it returned:
How many attempts did the above take?
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#295Earlier quoted context omitted.
Not big on censorship, but I agree it's a bit too much. I linked to it externally if anyone's curious.
It’s not censorship if you’re doing it to yourself or with permission. That’s just editing.
What logic is that? You can compel someone into permission with or without them knowing. That’s sketchy territory.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#296Earlier quoted context omitted.
I think a really interesting area of further research into LLMs is going to be how to further decouple "intelligence" from "knowledge". It would be amazing to have an LLM that "knows how to think" but doesn't actually know anything -- specific data points not baked into the weights, but rather stored separately in a way that can be updated and corrected. Perhaps a step forward is the "toolformer" router, making the L…
If you want something that "knows how to think" then I don't think you'll find it in an LLM. You can't divorce knowledge from this kind of model, it guesses what word comes next based on its training on a corpus of knowledge.
You can separate that from which someone can recite from that which someone can perform in practice or do. To remember facts and present them in language is different than being able to craft a bow.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#297Earlier quoted context omitted.
Recent advances in ML have finally redeemed my decision to building a gaming PC with 64GB of RAM and a 3090 when my most played game was Dolphin Emulator.
I forgot briefly that Dolphin refers to GameCube/Wii and was very curious about what a "dolphin emulator" would be (and how it would differ from a dolphin simulator )
LOL same here
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#298Earlier quoted context omitted.
you should censor your post around the answer to the kms question. it isn't necessary
Why? Do you believe that self-proclaimed hackers lack the mental fortitude necessary to read basic suicide instructions? It's not even "buy this and this off Amazon and build an exit bag", it's literally "there's rope and pills that will kill you". Not even "gory".
Mental fortitude isn't a constant and dafault trait in humans. Any of us can be vulnerable at a time.
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#299Earlier quoted context omitted.
I'm trying the 13B one, it's decent, I'd say on par with GPT 3, definitely not 3.5 or 4 yet however. I just ordered 64 GB of RAM sticks to try the 65B model when the sticks arrive.
Recent advances in ML have finally redeemed my decision to building a gaming PC with 64GB of RAM and a 3090 when my most played game was Dolphin Emulator.
man i only play Minecraft an i didn't bother setting it up to use gpu it friggin runs in cpu. so my gpu is not even woken up from slumber yet
Re: Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
#300[flagged]