Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD Alveo V70 AI inference accelerator card

xilinx.com

Post reply on HN
Print view View original

31–40 of 94 posts

Re: AMD Alveo V70 AI inference accelerator card

#31

Post by zamadatix »

scottlamb wrote: post #29
novaRom wrote: post #18

Earlier quoted context omitted.

10 fps should be fast enough to provide input tensors for real time inference with small scale transformers / convolutional nets.

Maybe. Running inference at 10 fps is probably plenty. But that doesn't mean you only have to do 10 fps of H.264/H.265 decoding. I think the most common scenario is for the input video to be e.g. 30 fps with mostly P frames that each depend on the prior frame in a chain. In that case, you need to decode almost [1] 30 fps to get 10 fps of evenly spaced frames to process. [1] You could skip the last P frame before an I…

If your source is 96 YouTube videos sure, if it's 96 CCTV cameras it's different.

Re: AMD Alveo V70 AI inference accelerator card

#32

Post by calaphos »

novaRom wrote: post #15
capableweb wrote: post #8

Earlier quoted context omitted.

TOPS - Trillions of Operations Per Second, used as a benchmark to figure out the performance of the accelerator. In my experience, mostly a marketing number, higher TOPS doesn't actually mean it'll be faster than something with a lower TOPS. As always, you need to do your own benchmarks with your use case in mind.

What kind of operations is not clear. Wether it's a simple logic operation or a FMA is big difference.

I assume it's int8 operations (FMA would count as 2). At least that's the case with FLOPs, TOPs for AI accelerators is basically the same measure, with the changed acronym reflecting the non float data format.

Re: AMD Alveo V70 AI inference accelerator card

#33

Post by hyuen »

I won't even take a look at the numbers unless they show a PyTorch model running on it, the problem is the big disconnect between HW and SW, realistically, have you ever seen any off-the shelf model running on something other than NVidia?

Re: AMD Alveo V70 AI inference accelerator card

#35

Post by scottlamb »

zamadatix wrote: post #31
scottlamb wrote: post #29

Earlier quoted context omitted.

Maybe. Running inference at 10 fps is probably plenty. But that doesn't mean you only have to do 10 fps of H.264/H.265 decoding. I think the most common scenario is for the input video to be e.g. 30 fps with mostly P frames that each depend on the prior frame in a chain. In that case, you need to decode almost [1] 30 fps to get 10 fps of evenly spaced frames to process. [1] You could skip the last P frame before an I…

If your source is 96 YouTube videos sure, if it's 96 CCTV cameras it's different.

Still depends. As it happens, I'm developing my own open source NVR software, [1] so I know a bit about this. Some cameras are fairly good about this, supporting the following features:

* "Temporal SVC", in which the frame dependencies are structured so you can discard down to 1/2 or 1/4th of the nominal frame rate and still decode the remainder.

* Three output streams, which you could configure for say forensics (high-bandwidth/high-resolution/high-fps), inference (mid-bandwidth/mid-resolution/low-fps), and viewing multiple streams / over mobile networks (low-bandwidth/low-resolution/mid-fps).

* On-camera ML tasks too. (Although I haven't seen one that lets you upload your own model.)

But other cameras are less good. E.g. some Reolinks [2] only support two streams, and the "sub" stream is fixed at 640x352, which is uncomfortably low. Your inference network may not take more resolution than that, but even if not, you might want to crop down to the area of interest (where there's motion and/or where the user has configured an alert) to improve quality. (You probably wouldn't pair that cheap Reolink camera with this expensive inference card, but the point stands in general.)

Even the "better" cameras' timestamp handling is awful, so it's hard to reliably match up the main stream, sub stream, analytics output, and wall clock time. Given that limitation it'd be desirable to just use the main stream for everything but the on-NVR transcoding's likely unaffordable.

[1] https://github.com/scottlamb/moonfire-nvr

[2] https://github.com/scottlamb/moonfire-nvr/wiki/Cameras:-Reol...

Re: AMD Alveo V70 AI inference accelerator card

#36

Post by Narew »

hyuen wrote: post #33

I won't even take a look at the numbers unless they show a PyTorch model running on it, the problem is the big disconnect between HW and SW, realistically, have you ever seen any off-the shelf model running on something other than NVidia?

It's for inference only not training. In this use case, there is lots of device that's not Nvidia. For server you have Google tpu, for more close to public there is the Apple Neural Engine for example.

Re: AMD Alveo V70 AI inference accelerator card

#37

Post by scottlamb »

h2odragon wrote: post #25
inetsee wrote: post #19

Earlier quoted context omitted.

I have no problem imagining a security camera application needing to monitor quite a few video channels.

Certainly. I'm suspecting that doing much of anything with all 96 channels would really need more RAM, for most users.

On the inference accelerator? IIUC, the RAM is just to hold the model and whatever state it needs during a particular inference operation. I'm not an expert on ML but AFAIK 16 GiB is plenty. I suppose it'd also need to hold onto reference frames for the video decoding, but at 1080p with e.g. YUV420 (12 bits per pixel), you can hold a lot of those in 16 GiB. edit: e.g., 4 references for each of the 96 streams would take ~1 GiB.

Even on the host, 16 GiB is fine for say an NVR. They don't need to keep a lot of state in RAM (or for that matter to do a lot of on-CPU computation either). I can run an 8-stream NVR on a Raspberry Pi 2 without on-NVR analytics. That's about its limit because the network and disk are on the same USB2 bus, but there's spare CPU and RAM.

Re: AMD Alveo V70 AI inference accelerator card

#38

Post by kombine »

hyuen wrote: post #33

I won't even take a look at the numbers unless they show a PyTorch model running on it, the problem is the big disconnect between HW and SW, realistically, have you ever seen any off-the shelf model running on something other than NVidia?

That. I work on the research side and I am still waiting for non-NVIDIA hardware for training deep models.

Re: AMD Alveo V70 AI inference accelerator card

#39

Post by kombine »

This is inference only. AMD should invest into the full AI stack starting from training. For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Honestly, I don't know why AMD aren't doing that already, they are best positioned to do that in the industry landscape.

Re: AMD Alveo V70 AI inference accelerator card

#40

Post by frozenport »

hyuen wrote: post #33

I won't even take a look at the numbers unless they show a PyTorch model running on it, the problem is the big disconnect between HW and SW, realistically, have you ever seen any off-the shelf model running on something other than NVidia?

Google's TPU comes to mind
Post reply on HN