81–90 of 95 posts
Starting from https://cloud.google.com/tpu/docs/system-architecture-tpu-vm what are you looking for?
System Architecture of these chips with detailed Functional Units and how they are used by the AI algorithm Instruction/Data streams.
Earlier quoted context omitted.
Yeah it’s performance vs cost is honestly nuts. 6 TOPS is a pretty solid NPU, but I don’t know what their software is like. Programming those accelerators is often difficult, especially if you’re a small time customer. Curious if anyone can weigh in on their SW usability. A quick search for their user level tools showed examples/documentation in Chinese(?)
Yeah it's the first under $100 system I've tried, (and I'm a fan of the genre) that's truly a desktop replacement in terms of being a snappy responsive desktop even with lots of browser tabs, etc. I do know they have their own special sauce to talk to the NPU. I was discouraged from making the effort myself because their special sauce to talk to the VPU has barely any ffmpeg support, it p much only uses gstreamer, an…
By VPU are you talking about stuff like ISP, video encoder/decoder, or something else?
Among embedded processors I’ve seen touting vision acceleration, gstreamer support is fairly widespread. I bit the bullet to learn it because my role requires it. Maybe it’s Stockholm Syndrome talking, but I’ve somehow grown to like gstreamer. The learning curve was awkward. I struggled with documentation and learned more by analyzing some examples and trial-and-error.
Earlier quoted context omitted.
For neural nets it's actually the opposite - half-precision bfloat16 is enough. You need large range, but not much accuracy. Yes, the exact numbers are going to vary, but just giving a data point to indicate the magnitude of the numbers. If you want to quibble there's CPU SIMD too.
For gaming do matter those double precision. And we were talking about a certain GPU, which is used for gaming, not AI. Hence why the AI chips exists in the first place - dedicated hardware for dedicated tasks (or ASIC for short)
Earlier quoted context omitted.
>* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's This does not seem to be true in reality. The H100 is about 2.5x 'better' for AI than the A100 (obviously depends exactly what you are doing), and they released about 2 years apart. That is roughly in line with Moore's law.
The difference seems to be: more units for parallel calculation, but the speed of calculation in itself doesn't double anymore. In other words: Moore's law has stopped for raw speed and perhaps other areas, but is still alive in other areas. This has some weird consequences: Some models can't be processed in smaller chips (because swapping in parts is too slow to be useful), but after a threshold is crossed, suddenly…
Earlier quoted context omitted.
>* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's This does not seem to be true in reality. The H100 is about 2.5x 'better' for AI than the A100 (obviously depends exactly what you are doing), and they released about 2 years apart. That is roughly in line with Moore's law.
The difference seems to be: more units for parallel calculation, but the speed of calculation in itself doesn't double anymore. In other words: Moore's law has stopped for raw speed and perhaps other areas, but is still alive in other areas. This has some weird consequences: Some models can't be processed in smaller chips (because swapping in parts is too slow to be useful), but after a threshold is crossed, suddenly…
Earlier quoted context omitted.
Yeah it's the first under $100 system I've tried, (and I'm a fan of the genre) that's truly a desktop replacement in terms of being a snappy responsive desktop even with lots of browser tabs, etc. I do know they have their own special sauce to talk to the NPU. I was discouraged from making the effort myself because their special sauce to talk to the VPU has barely any ffmpeg support, it p much only uses gstreamer, an…
Yeah I’m very surprised to see a single board computer at less than $100 for a processor like that. Hard to tell what their actual 1ku price is, but if the random Alibaba I found for $20 is right, then that price for the overall board is absurd. By VPU are you talking about stuff like ISP, video encoder/decoder, or something else? Among embedded processors I’ve seen touting vision acceleration, gstreamer support is f…
Oh no. The prospect of gstreamer being the only way... Oh no.
Maybe there's a zsh plugin or smth that autocompletes sane defaults? AAAAAAAAHHHHHH there surely isn't, anyone merciful enough to make one would just use ffmpeg instead...
Earlier quoted context omitted.
There have been some really strange instruction sets conceived. As a student at Stanford we had a time-share system that was home-grown (as I remember). It had opcodes to reverse bits in a bitstring! And odder things. Somebody needed that for some research project I guess. And then repurposed the damn thing for timeshare. It was pretty sad timeshare as I recall. The only machine(s?) available to a population of what?…
> It had opcodes to reverse bits in a bitstring Bit reversal is used in Fast Fourier Transforms. Its not entirely surprising to me that you'd have specialized hardware for that operation. Ref: https://en.wikipedia.org/wiki/Bit-reversal_permutation#Appli...
https://medium.com/@adi.fu7/ai-accelerators-part-i-intro-822...
Earlier quoted context omitted.
Nothing that fancy. Usually matrix-matrix and matrix-scalar multiplication.
This is indeed the bread-and-butter, but there is use of all sorts of standard linear algebra algorithms. You can check various xla-related (accelerated linear algebra) folders in tensorflow or torch folders in pytorch to see the list of what is used [1],[2] [1] https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b... [2] https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee...
Earlier quoted context omitted.
For gaming do matter those double precision. And we were talking about a certain GPU, which is used for gaming, not AI. Hence why the AI chips exists in the first place - dedicated hardware for dedicated tasks (or ASIC for short)
The NVIDIA cards are all dual-use for gaming and compute/ML. Some features like the RTX 4070's Tensor Cores (incl. bfloat16) are there primarily for ML, and other features like ray tracing are there for gaming.