51–60 of 84 posts
I had to make a minor modification to the code to get the Rust compiler happy, just add a `.as_slice()` when the compilation fails. I'll submit a PR if it's not fixed already.
I had to make a minor modification to the code to get the Rust compiler happy, just add a `.as_slice()` when the compilation fails. I'll submit a PR if it's not fixed already.
Orange Pi 5 has an NPU. I wonder if it'd be any faster than using GPU.
Earlier quoted context omitted.
> knows almost everything It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.
How about a local wikipedia dump, with precalculated embeddings? Then you can perform a similarity search first and feed the results to the LLM. It’s less likely to hallucinate this way.
There exists (at least) a project to train and query an LLM on local documents: privateGPT - https://github.com/imartinez/privateGPT
It should provide links to the the source with the relevant content, to check the exact text:
> You'll need to wait 20-30 seconds (depending on your machine) while the LLM model consumes the prompt and prepares the answer. Once done, it will print the answer and the 4 sources it used as context from your documents
You will have noticed, in that first sentence, that it may not be practical, especially on an Orange Pi.
Any idea how the GPU compares to the Nvidia's Jetson series?
It is difficult to find information about the performance of the ARM GPUs that can be compared with that of NVIDIA/AMD/Intel. However it seems that the Mali-G610 MC4 is in about the same range as the cheaper models of the old Jetson Xavier. The newer Jetson Orin models have much faster Ampere GPUs with between 1024 and 2048 FP32 ALUs. Nevertheless, the various Jetson Orin models have a price between 4 times and 13 ti…
> Any small computer with AMD Phoenix offers a much better GPU performance per dollar than any NVIDIA Orin.
> The use of NVIDIA Orin is justified only when one needs a device that is qualified for an automotive environment.
Nvidia Orin will use significantly less energy though.
I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything. Perfect for preppers / survivalists. Store it in a faraday cage along with a solar generator.
Orange Pi 5 has an NPU. I wonder if it'd be any faster than using GPU.
How many tokens per second do you think we can get out of this 6TFlops NPU?
Unbatched token generation is basically RAM bandwidth limited, as the entire model has to be cycled through for each token. I bet theoretical performance is similar to the GPU, albeit with much lower power consumption.
I've been thinking of this. It's just fascinating to me to have a small device that you can converse with and knows almost everything. Perfect for preppers / survivalists. Store it in a faraday cage along with a solar generator.
> knows almost everything It really doesn't. It doesn't even know what it knows and what it doesn't know. Without ways to check up on whether what it told you is true or not you may well end up in more trouble than where you were before.
This, without taking into account reasoning and consistence. And already this notion that I picked randomly is not without issues: dollars how computed? And, it is not difficult to state that Columbus reached the Caribbeans in 1492; more complex to "decide" the year of the siege of Troy out of the many dates proposed.
But already at the simplified level of determined clear notions: if LLMs are told that "A is B", and in absence of inconsistency in the training corpus, what is the failure rate (i.e. then outputting something critically different)?
> ways to check up
Some LLMs work as search engines, outputting not just their tentative answer but linked references. A reasonably safe practice at this stage is to use LLMs that way: ask then use the output to check the reference.
I'm surprised we haven't seen dedicated boxes to self host your uncensored & private LLM yet. A bit like you can self host your apps at home on a Umbrel box. I wonder if the NVIDIA Jetson serie would be the hardware that makes the most sense?
Smartphones aside, little Ryzen 6000 boxes would be OK.
Used DDR5 laptops with a little discrete GPU would be even better. I have one with a broken screen that may be dedicated to this very task.
Earlier quoted context omitted.
It is difficult to find information about the performance of the ARM GPUs that can be compared with that of NVIDIA/AMD/Intel. However it seems that the Mali-G610 MC4 is in about the same range as the cheaper models of the old Jetson Xavier. The newer Jetson Orin models have much faster Ampere GPUs with between 1024 and 2048 FP32 ALUs. Nevertheless, the various Jetson Orin models have a price between 4 times and 13 ti…
Thanks for the answer. > Any small computer with AMD Phoenix offers a much better GPU performance per dollar than any NVIDIA Orin. > The use of NVIDIA Orin is justified only when one needs a device that is qualified for an automotive environment. Nvidia Orin will use significantly less energy though.
Not really.
A Ryzen 7 7840U has a GPU with 768 FP32 ALUs @ 2.7 GHz and a NPU that can do 10 TOPS and it has a default TDP of 28 W.
The top Jetson AGX Orin models consume up to 60 W or 75 W, but they are so expensive that it does not make sense to compare them with a computer with 7840U and 32 GB of LPDDR5x-7500 that costs 3 times less.
A comparison that makes more sense is with a Jetson Orin NX 16GB (still significantly more expensive), which has a GPU with 1024 FP32 ALUs @ 0.918 GHz and it has a default TDP of 25 W.
For graphics tasks, Jetson Orin NX would be several times slower than an AMD Phoenix, due to its low GPU clock frequency and much slower CPU cores. The same is true for any programs executed on the CPU cores.
On the other hand, for AI inference, Jetson Orin has very fast tensor cores, so it can be many times faster than an AMD GPU or an ARM GPU, i.e. Jetson Orin NX 16 GB is claimed to be able to do 100 TOPS, so if this is the main intended application it can be worthwhile. Nevertheless, the usefulness of the Jetson Orin models for AI inference is diminished by the fact that their price increases very steeply when more memory is desired.