Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

jeffgeerling.com

Post reply on HN
Print view View original

21–30 of 121 posts

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#21

Post by davkan »

nutjob2 wrote: post #9

Somewhat off-topic: has anyone implemented USB C based networking? So for instance you have two or more machines that have USB 4 ports at 20 to 80 Gbps and you plug in the appropriate USB C cables to connect them. What do you need to make the connections appear as a (point to point) part of a regular TCP/IP network? Seems much cheaper than Ethernet if you already have the ports.

I used to have a thunderbolt ring network for a ceph backend on my k8s cluster. On Linux load the thunderbolt_net kernel module and they show up the same as any other Ethernet interfaces. Just directly connect them and configure the network settings on each end using your normal tools.

Latency was an order of magnitude higher than twisted pair but the throughput was greater than 10Gbps and I didn’t have to spend on 10GbE capable servers and network equipment. I vaguely remember that they had to be proper Thunderbolt not just USB-C but that could have changed it’s been quite a while.

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#22

Post by olavgg »

dlcarrier wrote: post #16
areoform wrote: post #4

Earlier quoted context omitted.

https://developer.apple.com/documentation/technotes/tn3205-l... MacOS has supported RDMA over thunderbolt since March 19th.

Is RDMA something worth looking into for connecting two random Linux computers? I've been extremely underwhelmed by the stability and performance of my NFS shares.

Yes, nfs with rdma will increase your performance significantly.

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#23

Post by geerlingguy »

wrs wrote: post #8
areoform wrote: post #4

Earlier quoted context omitted.

https://developer.apple.com/documentation/technotes/tn3205-l... MacOS has supported RDMA over thunderbolt since March 19th.

That is just RDMA itself and seems to be aimed directly at distributed inference (no surprise there!). Now they need to update the SMB driver to make use of it.

Yeah, raw RDMA is one thing, making sure other protocols can use it is another... hopefully Apple gets there, but right now SMB with RDMA is probably a tiny, tiny niche.

I do wonder what Apple's custom-built servers are running for networking. I wish they still made Mac Pro-level chassis specifically for better IO. Thunderbolt 5 is nice, but far inferior to a PCIe x16 slot (which can hit 400 Gbps networking) nowadays.

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#24

Post by esseph »

dlcarrier wrote: post #16
areoform wrote: post #4

Earlier quoted context omitted.

https://developer.apple.com/documentation/technotes/tn3205-l... MacOS has supported RDMA over thunderbolt since March 19th.

Is RDMA something worth looking into for connecting two random Linux computers? I've been extremely underwhelmed by the stability and performance of my NFS shares.

NFS 4.1 / 4.2?

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#25

Post by ssl-3 »

nutjob2 wrote: post #9

Somewhat off-topic: has anyone implemented USB C based networking? So for instance you have two or more machines that have USB 4 ports at 20 to 80 Gbps and you plug in the appropriate USB C cables to connect them. What do you need to make the connections appear as a (point to point) part of a regular TCP/IP network? Seems much cheaper than Ethernet if you already have the ports.

With USB 4, very strictly speaking: It's iffy. USB 4 doesn't require hosts to be able to switch roles and talk to other hosts. It can support this, but support is optional.

But for all certified Thunderbolt 4 and 5 ports, this kind of networking support is a requirement. (It is optional with Thunderbolt 3.)

The rest is up to the operating system.

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#26

Post by gregsadetsky »

SlavikCA wrote: post #10

What are typical use cases for needing 25Gbps? Author talks about 4k video, but AI tells me that for "4K production mastering (ProRes 422 HQ, 4:2:2, 10-bit)" the rate is under 2Gbps. I'm not saying it's not needed, - just curious.

We refurbish laptops and hence distribute .iso / ipsw images to various refurbishing stations. (or also, think of PXE/network boot situations). More images x more laptops refurbished at once = more bandwidth.

Obviously it's possible to move this to a NAS, not use a mac, etc. etc. - but just wanted to chime in with my little use case of moving multi-gig-files around. :-)

I was just looking for a thunderbolt-sfp solution. Super timely!

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#27

Post by geerlingguy »

QuantumNomad_ wrote: post #14
SlavikCA wrote: post #10

What are typical use cases for needing 25Gbps? Author talks about 4k video, but AI tells me that for "4K production mastering (ProRes 422 HQ, 4:2:2, 10-bit)" the rate is under 2Gbps. I'm not saying it's not needed, - just curious.

What if it’s uncompressed footage at 60fps? And dealing with many shots. And on top of that maybe working with multiple projects in parallel.

Normal 4K editing workflows are fine with even 2.5 Gbps wired networking (wired to make latency like when scrubbing not-horrid)... 10 Gbps more necessary if editing ProRes RAW and doing multicam (4-8 streams simultaneously).

I'm in a never-ending pursuit of cutting time when transferring large files though — AI models is one thing, but I have 100-200 GB projects I transfer across for backup or import (using an external TB5 drive capable of 3-5 GB/sec), and if I can do that in 1 or 2 minutes per project instead of 3 or 4, that's a noticeable speedup.

Definitely overkill for 98% of my workflow though. 10 Gbps is fine.

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#28

Post by gorkish »

pzmarzly wrote: post #2

One of the YouTube comments pointed out the issue is most likely the lack of support for SMB Direct (RDMA) in macOS. Would be nice to repeat the test on a Windows/Linux laptop.

This is a ConnextX4 on a thunderbolt bridge. This is well trodden territory. Unless some hero of open source has come to bat recently, MacOS doesn't support RDMA with this card or over Ethernet at all for that matter. It's only supported over Thunderbolt.

His iperf3 results show that it's able to go full beans on a single 25g port. A cross platform SMB file copy is kind of a meaningless benchmark in this context.

IMO a pretty cool product would be a network switch providing native thunderbolt interfaces to multiple upstream hosts. That wouldn't magically fix this RDMA issue, though. This is just a separate thought. Seems like the kind of thing Mikrotik should make.

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#29

Post by frollogaston »

teaearlgraycold wrote: post #3

It’s crazy that the Ethernet chips consume more power (presumably - because of the heat) than the processor in the computer. How is driving 25Gb/s such a hot task compared to driving multiple 40Gb/s Thunderbolt ports?

Maybe the NIC does a lot? In past experience, a whole Xeon CPU couldn't keep up with a 10gig NIC if it was sending/receiving min-size UDP packets. Even with Intel DPDK iirc.

Re: Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio

#30

Post by gorkish »

olavgg wrote: post #22
dlcarrier wrote: post #16

Earlier quoted context omitted.

Is RDMA something worth looking into for connecting two random Linux computers? I've been extremely underwhelmed by the stability and performance of my NFS shares.

Yes, nfs with rdma will increase your performance significantly.

If your storage devices are native PCIe and your network is actually fast enough and your nics and switches have the proper hardware and driver support and your switches are properly configured and your nfs client and server configurations are tweaked just so, and you hold your face in just the right position

The perf is amazing

Post reply on HN