Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPU-Accelerated LLM on an Orange Pi

blog.mlc.ai

Post reply on HN
Print view View original

21–30 of 84 posts

Re: GPU-Accelerated LLM on an Orange Pi

#21

Post by regularfry »

villgax wrote: post #17

I'm already getting 1.5tok on Ubuntu running on Android via UserLand w/ Llama.cpp(v2-Q4). Don't really see acceleration. If anything I need to see my phone do something actually useful at let's say 7-10toks

Human speech is in the 2-4 tokens per second range, I think that's about where my frustration limit is.

Re: GPU-Accelerated LLM on an Orange Pi

#22

Post by regularfry »

chank wrote: post #12
omneity wrote: post #4

Earlier quoted context omitted.

How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM. It’s less likely to hallucinate this way.

this concept already exists and is in practice at many companies that require knowledge driven results. https://arxiv.org/abs/2005.11401

There was another paper out recently that adds to this: https://arxiv.org/pdf/2308.04430.pdf. Looks like a more flexible approach to document storage, and it outperforms retrieval in context.

They trained up their own LLM, but from the text it seems like it might be possible to use any LLaMA-style LM without retraining. Not sure though, need to give it a proper look.

Re: GPU-Accelerated LLM on an Orange Pi

#23

Post by asdgioanionio »

kvetching wrote: post #2

I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything. Perfect for preppers / survivalists. Store it in a faraday cage along with a solar generator.

Depending on an LLM for survival is a good way to end up dead.

English Wikipedia will fit on an SD card. It's more valuable and more practical.

Re: GPU-Accelerated LLM on an Orange Pi

#24

Post by adrian_b »

jan_Sate wrote: post #8

Orange Pi 5 has an NPU. I wonder if it'd be any faster than using GPU.

It might have been more work to convert the model for the RK3588 NPU, even if Rockchip provides an SDK and an automated conversion tool that should help (the SDK includes a simulator for the NPU, so the converted model can be tried on a PC before being deployed on a board like Orange Pi):

https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html

Re: GPU-Accelerated LLM on an Orange Pi

#25

Post by regularfry »

jacquesm wrote: post #11
yu3zhou4 wrote: post #9

Earlier quoted context omitted.

It would be great to see such project implemented, I wonder how good would it perform

Yes, especially having fact checked output of LLMs would be a nice step in the right direction. Throwing out the hallucinated bits and keeping the good stuff would make LLMs a lot more applicable.

https://arxiv.org/pdf/2308.04430.pdf is interesting from that point of view. They've tackled it from the perspective of avoiding copyright content in training, but including it in inference but I think it ought to mean less hallucination because they also (claim to) solve the attribution problem.

Re: GPU-Accelerated LLM on an Orange Pi

#26

Post by unforeseen9991 »

asdgioanionio wrote: post #23
kvetching wrote: post #2

I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything. Perfect for preppers / survivalists. Store it in a faraday cage along with a solar generator.

Depending on an LLM for survival is a good way to end up dead. English Wikipedia will fit on an SD card. It's more valuable and more practical.

"I am confident those red berries you are describing are perfectly fine to eat"

Re: GPU-Accelerated LLM on an Orange Pi

#27

Post by jacquesm »

regularfry wrote: post #25
jacquesm wrote: post #11

Earlier quoted context omitted.

Yes, especially having fact checked output of LLMs would be a nice step in the right direction. Throwing out the hallucinated bits and keeping the good stuff would make LLMs a lot more applicable.

https://arxiv.org/pdf/2308.04430.pdf is interesting from that point of view. They've tackled it from the perspective of avoiding copyright content in training, but including it in inference but I think it ought to mean less hallucination because they also (claim to) solve the attribution problem.

Nice one, thank you. Added to my 'read later today' list, the abstract looks very interesting.

Re: GPU-Accelerated LLM on an Orange Pi

#28

Post by sunshine-o »

I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet.

A bit like you can self host your apps at home on a Umbrel box.

I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?

Re: GPU-Accelerated LLM on an Orange Pi

#29

Post by omneity »

jacquesm wrote: post #11
yu3zhou4 wrote: post #9

Earlier quoted context omitted.

It would be great to see such project implemented, I wonder how good would it perform

Yes, especially having fact checked output of LLMs would be a nice step in the right direction. Throwing out the hallucinated bits and keeping the good stuff would make LLMs a lot more applicable.

My hypothesis is that including information in the LLM’s prompt to support its answer changes the task roughly from text generation, very hallucination prone, to text summarization, or reformulation with some reasoning, and this is less likely to hallucinate.

That was my personal experience in general with ChatGPT as well as LLaMa1/2.

Re: GPU-Accelerated LLM on an Orange Pi

#30

Post by thehumanmeat »

sunshine-o wrote: post #28

I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box. I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?

See tinybox https://tinygrad.org/
Post reply on HN