Exciting news on the hardware bus side.
Now waiting for AMD Power9 processors.
OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
21–30 of 59 posts
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#22Earlier quoted context omitted.
Intel has 99% of the server market.
The people involved or wanting to use this buy so many chips from intel that if intel doesn't get on board, it's going to likely turn out badly for intel. I note that Facebook and Intel are missing, which makes me wonder if they are off in a corner somewhere doing their own thing.
or it could turn badly for the others, it will not gain attraction if intel products are not supported
Inertia is a very strong decisive factor, especially when you need to make sure that 30+ year-old code still work like it's the case in HPC
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#23No Intel ? Without them joining on board this might not be as useful.
It is precisely because Intel has developed its own next generation buses, and is licencing them very restrictively, that CAPI exists. Intel has QPI (Quickpath) for internal and Omnipath for external.
nVidia would have loved to have a GPU with a QPI interconnect, but Intel wouldn't let them because they have their own GPU ambitions in Xeon Phi. So they came up with NvLink, which is kind-of PCIe but faster. They don't have any switch asics as yet so they are limited to fully connected topologies. Details on NvLink (without NDA) are scant, but I don't think it has the ability to be multi-node (it really is a bus).
Intel are now making Xeon Phi with on board Omnipath (more like Infiniband), which is curious.
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#24Hmmm, interesting. I wonder what this means for the new Intel Xeon Phi Knights Landing? I liked the approach of many cores on one bootable chip, all having a reasonable amount of local memory, and high bandwidth interconnects: no need to offload data to a peripheral (GPU) device. However with this standard the currently limited bandwidth between peripherals and the main cpu will improve a lot.
To me it is obvious why Intel is not joining this party.
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#25Hopefully this effort will get us rid of PCIe for good, unlike the version of CAPI available on POWER8.
Still, many devices weren't able to fully utilize PCIe bandwidth and also latency. Turn around time is 1us to 1.5us for typical PCIe IP.
IMO, I'm a little bit pessimistic for OpenCAPI to reduce this latency much, unless it directly connects inside the chip.
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#26No Intel ? Without them joining on board this might not be as useful.
[deleted]
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#27Earlier quoted context omitted.
It depends on how low latency they mean by 'low latency'. If it can't drive fat gaming GPUs to full utilization, PCIe will be around for a while still. Also, Intel isn't joining up, so PCIe is absolutely sticking around.
In the spec, they say the maximum acceptable network delay is 200ns. The smallest network delay is 5ns.
It is really hard to believe that OpenCAPI can achieve this short latency with 25Gbps per lane interface and off-chip connectivity.
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#28Given AMD's involvement, what is the difference between this and coherent hypertransport?
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#29Earlier quoted context omitted.
In the spec, they say the maximum acceptable network delay is 200ns. The smallest network delay is 5ns.
It is really hard to believe that OpenCAPI can achieve this short latency with 25Gbps per lane interface and off-chip connectivity.
Xilinx offers 25gbps single-lanes that can bond up to 4x to get IEEE 802.3-2012 spec compliance for free* with their suite. Sure, you're going to need to control those trace impedences and your board won't be something coming out of OSH Park, but those are definitely attainable speeds for the consumer (e.g., in the single-thousands of dollars; not 800k Cisco VXR tier-1 infrastructure).
You can configure it in CAUI-10 (10 lanes x 10.3125G) or CAUI-4 (4 lanes x 25.78125G), either way, it's been production-ready for quite some time now. (The docs have numbers, but trust me, you can get full throughput within that 200 ns).
There's even production Agilent off-the-shelf test equipment out there that can fully sample at those speeds (none of that over-sampling tomfoolery, we're talking live, Bill O'Reilly style).
In 1989, UltraSPARC had similar facilities (SBus) to push 100MBit between other Sun machines, so I mean, not too insane comparatively.
* Free with purchase of Virtex® UltraScale™ and Kintex® UltraScale FPGA required haha.
Re: OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
#30Hmmm, interesting. I wonder what this means for the new Intel Xeon Phi Knights Landing? I liked the approach of many cores on one bootable chip, all having a reasonable amount of local memory, and high bandwidth interconnects: no need to offload data to a peripheral (GPU) device. However with this standard the currently limited bandwidth between peripherals and the main cpu will improve a lot. To me it is obvious why…
well, i think its a bit more about market segments and interoperability than anything else. currently overall systems from ibm, and, and intel are fundamentally incompatible. the PCI-E bus that knights landing hangs off of is a qualitatively different thing than the kinds of memory-coherent inter-cpu busses that are being addressed with the CAPI proposal. Intel has their own proprietary QPI. AMD has a quasi-open hyper transport (still?).
If this is done properly it means you could make generic motherboards, and generic memory controllers, and all sorts of different accelerators and mix and match them from various vendors. So its no surprise that the smaller players in the market and trying to gang together and the larger player is trying to keep lock-in.
Knights Landing as it stands would already integrate in systems better if it were using an inter-cpu bus than a peripheral bus as well as GPUs, FPGAs, and certainly RDMA/memory window systems like Mellanox.
Inherent distrust of standards aside, this could be a great win for people putting together bespoke systems in interesting configurations (i.e. Google), I don't think there is any downside in theory for Intel except more competition.