Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ultra Ethernet Specification v1.0 [pdf]

ultraethernet.org

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Ultra Ethernet Specification v1.0 [pdf]

#31

Post by rincebrain »

throw0101d wrote: post #29
jauntywundrkind wrote: post #17

Earlier quoted context omitted.

Intel's Omni-Path was very similar but a little different from Infiniband. Anyone remember any details on how; I'm forgetting? Omni-Path was sold off to Cornelis Networks in 2020. Cornelis just released new 400Gbps products last week, their CN5000 line, with 48 and 576 port switches. https://www.crn.com/news/components-peripherals/intel-spins-...

> Intel's Omni-Path was very similar but a little different from Infiniband. While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days): * https://www.dell.com/support/kbdoc/en-us/000019215/install-a...

Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.)

Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space.

So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.

Re: Ultra Ethernet Specification v1.0 [pdf]

#32

Post by timnetworks »

throw0101d wrote: post #15
jpgvm wrote: post #14

Earlier quoted context omitted.

UltraEthernet predates the AI boom. I don't blame them for adding the AI buzzwords, for this to succeed people need to think about it in the same vein as IB/RoCE/etc and a lot of the decision makers are unfortunately not technical enough to understand what this is or does. Ultimately it's just a different (IMO slightly more refined) look at how to support RDMA on Ethernet vs RoCE which is a more ham fisted implementa…

> UltraEthernet predates the AI boom. UEC was formed/announced in July 2023: * https://ultraethernet.org/leading-cloud-service-semiconducto... ChatGPT was launched in November 2022: * https://en.wikipedia.org/wiki/ChatGPT * https://en.wikipedia.org/wiki/GPT-4

The writing was on the wall by 2015-2017, earlier still if you're the smart/attentive type.

Re: Ultra Ethernet Specification v1.0 [pdf]

#33

Post by crote »

phonon wrote: post #30
crote wrote: post #26

Earlier quoted context omitted.

Desktop computer are barely on 2.5GBASE-T, and there's still 5G in-between. I reckon it'll be 5-10 years until we'll see widespread 10G rollout, and it'll easily last another 10 before we need something better. The bigger issue is wireless access points. High-end ones are already on 10G today, and unlike desktops PoE is quite critical here. 25GBASE-T and 40GBASE-T are dead-on-arrival: the spec exists, but nobody ever…

The Realtek RTL8127 chip for 10GbE will cost $10, draw 1.95 W and is designed for motherboards. So I think you're a bit pessimistic on timelines... https://www.tomshardware.com/networking/realteks-usd10-tiny-...

Key word being "widespread".

10GBASE-T dates back to 2006, and we've been seeing it on the odd prosumer workstation / elite gamer motherboard for quite a while now. But those markets aren't going to lead to a full office rewiring.

The entry-level market is more interesting: when will we see it on motherboards that retail for $100? When will we see it on the average Dell Optiplex? When will your ISP router come with 10G ports?

Will some people be using 10G before that? Definitely, and I'll be one of them. Will it be something to consider for the average office environment? Nah.

Re: Ultra Ethernet Specification v1.0 [pdf]

#34

Post by throw0101d »

rincebrain wrote: post #31
throw0101d wrote: post #29

Earlier quoted context omitted.

> Intel's Omni-Path was very similar but a little different from Infiniband. While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days): * https://www.dell.com/support/kbdoc/en-us/000019215/install-a...

Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.) Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space. So at this point, I believe NVIDIA's Mellanox devices are th…

> […] seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.

I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).

Re: Ultra Ethernet Specification v1.0 [pdf]

#35

Post by MrMorden »

bgnn wrote: post #25
lousken wrote: post #21

This is cool, but I am more curious about what happens to all the cat 6a cabling in offices, where do we go after 10GBASE-T

There's already 25G BASE-T over CAT6 available. I worked on and taped out a 25G PHY in 2019. Cable reach is 30m instead of 100m though. There's work going on 40G for 10m reach.

If by "available" you mean that there's a standard. There may be PHY designs that exist but they're not available for purchase, probably because there's no identifiable market. Any potential customer who for some reason wanted to build an AP1000 reactor unit for their datacenter and a second unit for that datacenter's PHYs would need to design their own networking equipment. Which might use 40G but at this point it's questionable — that's the sort of slow equipment you might see in a rack-scale on-premises installation.

Re: Ultra Ethernet Specification v1.0 [pdf]

#36

Post by jpgvm »

timnetworks wrote: post #32
throw0101d wrote: post #15

Earlier quoted context omitted.

> UltraEthernet predates the AI boom. UEC was formed/announced in July 2023: * https://ultraethernet.org/leading-cloud-service-semiconducto... ChatGPT was launched in November 2022: * https://en.wikipedia.org/wiki/ChatGPT * https://en.wikipedia.org/wiki/GPT-4

The writing was on the wall by 2015-2017, earlier still if you're the smart/attentive type.

Yeah I may have underestimated how recent the formation of UEC was but IMO the efforts started around the period you are indicating with a lot of the libfabric guys proposing stuff like this.

Re: Ultra Ethernet Specification v1.0 [pdf]

#37

Post by rincebrain »

throw0101d wrote: post #34
rincebrain wrote: post #31

Earlier quoted context omitted.

Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.) Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space. So at this point, I believe NVIDIA's Mellanox devices are th…

> […] seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks. I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).

One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, and Infiniband support on Windows might now be up to merely 'iffy' instead of "requires a cold boot after you try using RDMA". So after we tried using IB once and got burned by being promised the software would be there Soon(tm), we went with then-10GbE for the portion we managed.

Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport.

But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.

Re: Ultra Ethernet Specification v1.0 [pdf]

#38

Post by bgnn »

MrMorden wrote: post #35
bgnn wrote: post #25

Earlier quoted context omitted.

There's already 25G BASE-T over CAT6 available. I worked on and taped out a 25G PHY in 2019. Cable reach is 30m instead of 100m though. There's work going on 40G for 10m reach.

If by "available" you mean that there's a standard. There may be PHY designs that exist but they're not available for purchase, probably because there's no identifiable market. Any potential customer who for some reason wanted to build an AP1000 reactor unit for their datacenter and a second unit for that datacenter's PHYs would need to design their own networking equipment. Which might use 40G but at this point it's…

The engineering samples are available and if a network equipment manufacturer or a hyperscaler chooses to use it, it can be in production quite fast.

It's a bad idea though. This is the reason it's not adopted in any product.

Re: Ultra Ethernet Specification v1.0 [pdf]

#39

Post by jauntywundrkind »

phonon wrote: post #30
crote wrote: post #26

Earlier quoted context omitted.

Desktop computer are barely on 2.5GBASE-T, and there's still 5G in-between. I reckon it'll be 5-10 years until we'll see widespread 10G rollout, and it'll easily last another 10 before we need something better. The bigger issue is wireless access points. High-end ones are already on 10G today, and unlike desktops PoE is quite critical here. 25GBASE-T and 40GBASE-T are dead-on-arrival: the spec exists, but nobody ever…

The Realtek RTL8127 chip for 10GbE will cost $10, draw 1.95 W and is designed for motherboards. So I think you're a bit pessimistic on timelines... https://www.tomshardware.com/networking/realteks-usd10-tiny-...

Agreed.

Change is coming! I feel like 5Gbit is mostly useful for switches, that a lot of other products are gonna jump direct to 10GBase-T.

25GBase-T or 40GBase-T is probably going to be a bit more energy intensive for a while, and 30m is ok but not great range if you're trying to span an office.

Ideally I'd really like to see optics costs fall, for consumers to start picking it up. Maybe the future is SFP everywhere and perhaps that makes sense (swap out transciever if one fails! Direct Attach Copper maybe or Base-T for convenience). But I really hope we see some real inroads, some of this separation between consumer & commercial go away.

I got a 150' 40Gbps fiber optic display port cable for ~$50, over half a decade ago. Unidirectional (I believe) but still just totally reset my expectations of what stuff really has to cost! Seemed too good to be true but still using it! We're spending more and more energy and effort trying to squeeze bits down a difficult copper channel, but optics right there, getting easier and easier.

I wish Intel hadn't given up on on-package interconnects (available on package on Xeon & Phi, 2016-2019). High end at the time. But now PCIe, USB, and DisplayPort all have optical capable specs, and the energy cost and distance of copper keeps making less and less sense. We gotta go forward, and the break from scant tiny consumer bandwidth to something much different feels like a potential dam burst.

I hope Intel's still doing some optical stuff with the recent shake ups. Loved their embarrassingly parallel RISC-V beast with 16 optical links. And it was a year ago they demoed a new all optical switch (rip Intel/Barefoot Tofino, you wonderful p4 programmable beast) with copious 4Tb/s PCIe switching. https://www.theregister.com/2023/09/01/intel_graph_analytics... https://newsroom.intel.com/artificial-intelligence/intel-unv...

Re: Ultra Ethernet Specification v1.0 [pdf]

#40

Post by lousken »

crote wrote: post #26
lousken wrote: post #21

This is cool, but I am more curious about what happens to all the cat 6a cabling in offices, where do we go after 10GBASE-T

Desktop computer are barely on 2.5GBASE-T, and there's still 5G in-between. I reckon it'll be 5-10 years until we'll see widespread 10G rollout, and it'll easily last another 10 before we need something better. The bigger issue is wireless access points. High-end ones are already on 10G today, and unlike desktops PoE is quite critical here. 25GBASE-T and 40GBASE-T are dead-on-arrival: the spec exists, but nobody ever…

as someone with a homelab i disagree, i dont want to throw away my cat 6a cabling here, however i would like to see a bit more than i currently have with my 10Gbps, so maybe a middleground?

I would assume that max distance wouldn't be 100meters, but even if 30, that's more than plenty for lots of use cases

And as you mentioned, poe devices will be a huge pain that will have to be resolved

Post reply on HN