Live data from Hacker News

Reliable 25 Gigabit Ethernet via Thunderbolt

kohlschuetter.github.io

21–30 of 133 posts

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#21
post #17

Note that you can do point-to-point network links directly with thunderbolt (and usb4). https://support.apple.com/guide/mac-help/ip-thunderbolt-conn... etc

Yes! However, I got around 15 Gbps with a Thunderbolt-only setup (TB3/TB4) = only 75% of the Ethernet setup.

You'd also mostly be limited to short cables (1-2m) and a ring topology.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#22

Neat, but the thermal design is absolutely terrible. Sticking that heatsink inside the aluminum case without any air circulation is awful.

Yeah, it's because the network card adapter's heatsink is sandwiched between two PCBs. Not great, not terrible, works for me. The placement is mostly determined by the design of the OCP 2.0 connector. OCP 3.0 has a connector at the short edge of the card, which allows exposing/extending the heat sink directly to the outer case. If somebody has the talent, designing a Thunderbolt 5 adapter for OCP 3.0 cards could be a…

A Flex PCB connecting the OCP2 connector would allow to put the converter board behind the NIC board, allowing the NIC board to be exposed to the aluminum case to use the case itself as a heatsink (would need a split case so the NIC board can be screwed to one side of the case, pressing the main chip against it via a thermal pad).

As a stop-gap, I'd see if there was any way to get airflow into the case - I'd expect even a tiny fan would do much more than those two large heatsinks stuck onto the case (since the case itself has no thermal connection to the chip heatsink).

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#24

Thunderbolt is basically external PCIe, so this is not so surprising. High speed NICs do consume a relatively large amount of power. I have a feeling I've seen that logo on the board before.

I don't know how to measure the direct power impact on a MacBook Pro (since it's got a battery), but the typical power consumption of these cards is 9 W, not much more than Aquantia 10 GBit cards. Also, if you remember where you saw that logo, please let me know!

Plus 1-2.5w per active cable. You need the heatsinks as the cx4 cards expect active airflow, and active transceivers as well.

I have a 10gbit dual port card in a Lenovo mini pc. There is no normal way to get any heat out of there so I put a 12v small radial fan in there as support. It works great at 5v: silent and cool. It is a fan though so might not suit your purpose.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#25

Neat, but the thermal design is absolutely terrible. Sticking that heatsink inside the aluminum case without any air circulation is awful.

Yeah, it's because the network card adapter's heatsink is sandwiched between two PCBs. Not great, not terrible, works for me. The placement is mostly determined by the design of the OCP 2.0 connector. OCP 3.0 has a connector at the short edge of the card, which allows exposing/extending the heat sink directly to the outer case. If somebody has the talent, designing a Thunderbolt 5 adapter for OCP 3.0 cards could be a…

I'd be more worried about cooling the transceivers properly.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#26

Earlier quoted context omitted.

Yeah, it's because the network card adapter's heatsink is sandwiched between two PCBs. Not great, not terrible, works for me. The placement is mostly determined by the design of the OCP 2.0 connector. OCP 3.0 has a connector at the short edge of the card, which allows exposing/extending the heat sink directly to the outer case. If somebody has the talent, designing a Thunderbolt 5 adapter for OCP 3.0 cards could be a…

A Flex PCB connecting the OCP2 connector would allow to put the converter board behind the NIC board, allowing the NIC board to be exposed to the aluminum case to use the case itself as a heatsink (would need a split case so the NIC board can be screwed to one side of the case, pressing the main chip against it via a thermal pad). As a stop-gap, I'd see if there was any way to get airflow into the case - I'd expect e…

My goal was to get a fanless setup (for a quiet office).

If that's not a requirement just get the Raiden Digit Light One, which does have a fan (and otherwise the same network card).

If I could design an adapter PCB myself, I would go straight to OCP 3.0, which allows for a much simpler construction, and TB5 speeds.

Alternatively, there are DELL CX422A rNDC cards (R887V) that appear to have an OCP 2.0 connector but a better heatsink design.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#27
post #6
post #2

Now I just have to contrive the circumstances where this is useful to me. :)

I don't know about the Ethernet part but it bothers me that even wifi has become faster than the wired USB port on our phones. All I want to do is copy over all the photos and videos from my phone to my computer but I have to baby sit the process and think whether I want to skip or retry a failed copy. And it is so slow. USB 2.0 slow. I guess everybody has given up on the idea of saving their photos and videos over U…

I feel like this is an artifact from the late 2010s when the talk was of removing the port completely from phones, where that was being touted alongside swapping speakers with haptic screen audio as a way to make them completely waterproof.

As wireless charging never quite reached the level hoped – see AirPower – and Google/Apple seemingly bought and never did anything with a bunch of haptic audio startups, I figure that idea died....but they never cared enough to make sure the USB port remained top end.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#28
I'm surprised you are only getting 20gbit/s. I did not expect PCIe to be be the limiting factor here. I've got a 100gbit cx4 card currently in a PCIe3 X4 slot (for reasons, don't judge) and it easily maxes that out. I would have expected the 25g cx4 cards to be at least able to get everything out of it. RDMA is required to achieve that in a useful way though.

Edit: forgot is isn't "true" PCIe but tunneled.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#29
post #13

Earlier quoted context omitted.

Wouldn’t this be useful for clustering Macs over TB5? Wasn’t the maximum bandwidth over USB-cables 5Gbps? With a switch, you could cluster more than just 4 Mac Studios and have a couple terabytes for very large models to work with.

I was hoping somebody would suggest that (and eventually try it out). With TB5, and deep pockets, you might probably also benchmark it against a setup with dedicated TB5 enclosures (e.g., Mercury Helios 5S). TB5 has PCIe 4.0 x4 instead of PCIe 3.0 x4 -- that should give you 50 GbE half-duplex instead of 25 GbE. You would need a different network card though (ConnectX-5, for example). Pragmatically though, you could a…

Too bad Jeff Geerling returned his Mac Studios to Apple. Would be lovely to see how 5x faster RDMA impacts the performance.
Post reply on HN