Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPU-Accelerated LLM on an Orange Pi

blog.mlc.ai

Post reply on HN
Print view View original

11–20 of 84 posts

Re: GPU-Accelerated LLM on an Orange Pi

#11

Post by jacquesm »

yu3zhou4 wrote: post #9
omneity wrote: post #4

Earlier quoted context omitted.

How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM. It’s less likely to hallucinate this way.

It would be great to see such project implemented, I wonder how good would it perform

Yes, especially having fact checked output of LLMs would be a nice step in the right direction. Throwing out the hallucinated bits and keeping the good stuff would make LLMs a lot more applicable.

Re: GPU-Accelerated LLM on an Orange Pi

#12

Post by chank »

omneity wrote: post #4
jacquesm wrote: post #3

Earlier quoted context omitted.

> knows almost everything It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.

How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM. It’s less likely to hallucinate this way.

this concept already exists and is in practice at many companies that require knowledge driven results. https://arxiv.org/abs/2005.11401

Re: GPU-Accelerated LLM on an Orange Pi

#13

Post by bravura »

jacquesm wrote: post #7
omneity wrote: post #4

Earlier quoted context omitted.

How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM. It’s less likely to hallucinate this way.

That would make a lot more sense. That way at least you have a chance to check up on the output, lest your first meal of 'Hedysarum alpinum' ends up being your last.

The key aspect to this being a good solution is actually building a corpus representing as much possible reference knowledge needed in scenario. The idea that the answer is Wikipedia way underestimates the scope.

The Wikipedia patch doesn’t make much sense to me.

What percent of the important questions being asked in this doomsday scenario actually have their answer in Wikipedia?

If 50% of the time you are left trusting raw LLaMa, then you don’t really have a decent solution.

I do appreciate the sentiment tho that future or finetuned LLMS might fit on an RPi or whatever, and be good enough.

Re: GPU-Accelerated LLM on an Orange Pi

#14

Post by tysam_and »

jdfedgon wrote: post #5

Nice achievement. How many users would realistically be able to use it at the same time when running on such a device? I am interested in its scalability.

Not many, since it's slow to begin with.

You'll get a log_2-based scaling efficiency with nearly any batchsize increase, pending some limitations (memory, etc).

That should be enough at least to roughly sketch it out.

Re: GPU-Accelerated LLM on an Orange Pi

#15

Post by tysam_and »

pjmlp wrote: post #6

Best part is that they are using TVM.

This was the part that intrigued me. After years of silent begging to PyTorch to support ROCm, suddenly models in general supporting so many different platforms feels overwhelming, and really good.

I'm crediting llama.cpp of all things for being the boost to really up the ante on open source model compilation.

Whatever it is, at least, many of these open source things feel like they just 'happen', as an eventuality, but in order for that to happen it takes a lot of work from a lot of people! Really happy to see the dream of this particular kind of democratization opening widely! :)

Re: GPU-Accelerated LLM on an Orange Pi

#16

Post by eis »

jacquesm wrote: post #11
yu3zhou4 wrote: post #9

Earlier quoted context omitted.

It would be great to see such project implemented, I wonder how good would it perform

Yes, especially having fact checked output of LLMs would be a nice step in the right direction. Throwing out the hallucinated bits and keeping the good stuff would make LLMs a lot more applicable.

Isn't that a bit of a holy grail though? If your software can fact check the output of LLMs and prevent hallucinations then why not use that as the AI to get the answers in the first place?

Re: GPU-Accelerated LLM on an Orange Pi

#19

Post by jacquesm »

eis wrote: post #16
jacquesm wrote: post #11

Earlier quoted context omitted.

Yes, especially having fact checked output of LLMs would be a nice step in the right direction. Throwing out the hallucinated bits and keeping the good stuff would make LLMs a lot more applicable.

Isn't that a bit of a holy grail though? If your software can fact check the output of LLMs and prevent hallucinations then why not use that as the AI to get the answers in the first place?

Because you - hopefully - have a check against something that is on average of higher quality than the combined input of an LLM.

I'm not sure if this can work or not but it would be nice to see a trial, you could probably do this by hand if you wanted to by breaking up the answer from an LLM into factoids and then to check each of those individually, and to assign a score to them based on the amount of supporting evidence for the factoid. I'd love that as a plug-in to a browser too.

Re: GPU-Accelerated LLM on an Orange Pi

#20

Post by hnuser123456 »

bravura wrote: post #13
jacquesm wrote: post #7

Earlier quoted context omitted.

That would make a lot more sense. That way at least you have a chance to check up on the output, lest your first meal of 'Hedysarum alpinum' ends up being your last.

The key aspect to this being a good solution is actually building a corpus representing as much possible reference knowledge needed in scenario. The idea that the answer is Wikipedia way underestimates the scope. The Wikipedia patch doesn’t make much sense to me. What percent of the important questions being asked in this doomsday scenario actually have their answer in Wikipedia? If 50% of the time you are left trust…

I think the theory is that an LLM can integrate the knowledge of Wikipedia and become something greater than the sum of its parts by applying reasoning that is explained in one article to situations in other topics where the reasoning might not be so well explained. Then you can ask naive questions in new scenarios where you might not have the background knowledge (or simply the mental energy) to figure out a right answer to a situation on your own but it powers through for you. AFAIK current LLMs are not this abstract. If one type of reasoning is more often applied in one scenario and another type of reasoning is applied to another, they don't have any context beyond the words and they know what topics humans usually jump to from other given topics.
Post reply on HN