AMD Alveo V70 AI inference accelerator card • Hacker News Board

Re: AMD Alveo V70 AI inference accelerator card

#41

Post by Hooray_Darakian » Tue, Jan 10, 2023, 5:59 PM UTC

kombine wrote: post #39

This is inference only. AMD should invest into the full AI stack starting from training. For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Honestly, I don't know why AMD aren't doing that already, they are best positioned to do that in the industry landscape.

> AMD should invest into the full AI stack starting from training.

https://www.amd.com/en/graphics/servers-solutions-rocm-ml

> For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware.

Why is a high end product a requirement for entry level research?

Re: AMD Alveo V70 AI inference accelerator card

#42

Post by wmf » Tue, Jan 10, 2023, 5:59 PM UTC

kombine wrote: post #39

This is inference only. AMD should invest into the full AI stack starting from training. For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Honestly, I don't know why AMD aren't doing that already, they are best positioned to do that in the industry landscape.

MI200, MI250, and MI300 should work for training. They don't have an exact equivalent to the 4090 but that may be setting the bar too high. Nobody can deliver everything that Nvidia has but better.

Re: AMD Alveo V70 AI inference accelerator card

#43

Post by modeless » Tue, Jan 10, 2023, 6:02 PM UTC

kombine wrote: post #39

This is inference only. AMD should invest into the full AI stack starting from training. For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Honestly, I don't know why AMD aren't doing that already, they are best positioned to do that in the industry landscape.

The hardware is not their major problem. They have been failing super hard at the software side of machine learning for a solid decade now.

It seems like pure management incompetence to me. They need to invest a whole lot more in software, integrating their stuff directly into pytorch/TF/XLA/etc and making sure it works on consumer cards too. The investment would be paid back tenfold. The market is crying out for more competition for Nvidia and there's huge money to be made on the datacenter side but it all needs to work on the consumer side too.

Re: AMD Alveo V70 AI inference accelerator card

#44

Post by wmf » Tue, Jan 10, 2023, 6:04 PM UTC

YakBizzarro wrote: post #20

wyldfire wrote: post #11

AMD XDNA – Versal AI Core / 2nd-gen AIE-ML tiles Are these programmable by the end-user? The "software programmability" section describes "Vitis AI" frameworks supported. But can we write our own software on these? Is this card FPGA-based? EDIT: [1] more info on the AI-engine tiles: scalar cores + "adaptable hardware (FPGA?)" + {AI+DSP}. [1] https://www.xilinx.com/products/technology/ai-engine.html

Interesting. It seems then that the xdna architecture in the Ryzen 4070 is nothing more than a port of the existing Xilinx Versal cores (fpga+ai engine)

That's what AMD said.

Re: AMD Alveo V70 AI inference accelerator card

#45

Post by jzwinck » Tue, Jan 10, 2023, 6:21 PM UTC

Hooray_Darakian wrote: post #41

kombine wrote: post #39

This is inference only. AMD should invest into the full AI stack starting from training. For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Honestly, I don't know why AMD aren't doing that already, they are best positioned to do that in the industry landscape.

> AMD should invest into the full AI stack starting from training. https://www.amd.com/en/graphics/servers-solutions-rocm-ml > For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Why is a high end product a requirement for entry level research?

Because high end research uses a fleet of them, not just one.

Re: AMD Alveo V70 AI inference accelerator card

#46

Post by p1esk » Tue, Jan 10, 2023, 6:27 PM UTC

How much memory does it have?

Re: AMD Alveo V70 AI inference accelerator card

#47

Post by hhh » Tue, Jan 10, 2023, 6:42 PM UTC

h2odragon wrote: post #14

Douglas Adams said we'd have robots to watch TV for us. That seems to be the designed use case for this. 16gb RAM / 96 video channels ... I haven't done any of that work but it feels like they expect that "96" not to be fully used in practice.

I have models in production that currently monitor ~400 cameras with an addition of 2-3 cameras/month. If it were cheap enough, it would be useful for our use case (Quality Control). We generally pull from cameras roughly 6400 pixels per region of interest, of which one instance may have 4-30 RoIs across N cameras.

Re: AMD Alveo V70 AI inference accelerator card

#48

Post by hhh » Tue, Jan 10, 2023, 6:45 PM UTC

hyuen wrote: post #33

I won't even take a look at the numbers unless they show a PyTorch model running on it, the problem is the big disconnect between HW and SW, realistically, have you ever seen any off-the shelf model running on something other than NVidia?

I've ran models on Apple HW, Raspberry Pi, random CPUs, NVIDIA GPUs, TPUs. Waiting to get my hands on Tenstorrent gear.

Re: AMD Alveo V70 AI inference accelerator card

#49

Post by hedgehog » Tue, Jan 10, 2023, 6:49 PM UTC

kombine wrote: post #39

This is inference only. AMD should invest into the full AI stack starting from training. For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Honestly, I don't know why AMD aren't doing that already, they are best positioned to do that in the industry landscape.

Big question is why? Would competing for entry level researchers buy them much?

Re: AMD Alveo V70 AI inference accelerator card

#50

Post by kombine » Tue, Jan 10, 2023, 6:52 PM UTC

Hooray_Darakian wrote: post #41

kombine wrote: post #39

This is inference only. AMD should invest into the full AI stack starting from training. For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Honestly, I don't know why AMD aren't doing that already, they are best positioned to do that in the industry landscape.

> AMD should invest into the full AI stack starting from training. https://www.amd.com/en/graphics/servers-solutions-rocm-ml > For this they need a product comparable to NVIDIA 4090, so that entry level researchers could use their hardware. Why is a high end product a requirement for entry level research?

4090 (or 3090, 1080Ti and so on) is a high-end consumer GPU, but at the same time it is an entry level GPU for AI researchers. Don't forget that workstation cards (RTX 8000) let alone server-grade GPUs such as A100 are an order of magnitude more expensive.