Live data from Hacker News

Reliable 25 Gigabit Ethernet via Thunderbolt

kohlschuetter.github.io

111–120 of 133 posts

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#111

Earlier quoted context omitted.

"because I can, and it is fun." The best answer! I am most of the way done with upgrading most of my homelab to 100G from 10G, but there really isn't a practical reason for it. 100G has dropped in price so much as datacenters are all about 400/800G now.

Nice! Cool to hear from a fellow admirer of overkill-lan setups ;) Which cards do you prefer for 100G, and what is the situation with dacs/optics?

I wouldn't really call 100Gbit overkill, if you compare it to modern disk drives is about where we should be relative to shared storage/NAS/etc infrastructure people used to run. So yes, being able to share my /home directory across a few dozen machines at my house without a huge perf impact vs using a local drive seems a pretty reasonable use case. Sure its faster than my WAN access, but who cares?

Frankly, 10Gbit is fully 25 years old with, 10GbaseT being 20 years old this year.

Thats ridiculously ancient technology. There is/was a 25/40GbaseT spec too (now 10 years old), which basically no one implemented because like ECC ram (and tape drives, and seem to be trying to do it with harddrives and GPUs) the MBA's have taken over parts of the computer industry and decided that they can milk huge profit margins from technologies which are incrementally more difficult because smaller users just don't matter to their bottom lines. The only reason those MBAs are allowing us to have it now, is because a pretty decent percentage of us can now get 5Gbit+ internet access and our wifi routers can do 1Gbit+ wireless, and the weak link is being able to attach the two.

I did a bit of back of the napkin math/simulation about a possible variable rate Ethernet (ex like NBbaseT, where it has multiple speeds and selects faster one based on line conditions), and concluded that 80+Gbit using modern PHY/DSP's and high symbol rate, multiple bands, techology which is dirt cheap thanks to wifi/bt/etc on fairly short cable distances (ex 30-50M) on CAT8 is entirely possible. And this isn't even fantasy, short cat7 runs are an entire diffrent ballpark from a phone pair, and these days mg.fast/etc have shown 10Gbit+ over that junk.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#112

Earlier quoted context omitted.

The linux implementation is quite poor. Among other issues, your answer is linux treats it as a TCP/IP link- so packets don't have any offloading for checksums, etc. it's also incomplete (ex- you have to physically unplug and replug the cord every time one side loses link- even a reboot). This is my firsthand experence trying to get some tablet motherboards to link up and work as a proxmox cluster w/ TB3 as the link…

I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.

AWESOME!! Do you mind linking references? I put the project aside because of these issues.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#113
post #87

Earlier quoted context omitted.

Yep, these cards need a fan (or any kind of directed air flow). Where did you get "several ms of latency" figure from? I have not measured external card, but may be I should do it... Because cards themselves have latency in range of microseconds, not millis.

I haven't tested this particular Thunderbolt SFP adapter, but my experience with a TP-Link 1Gbps USB adapter is that it adds about 4ms of latency. Far from being unusable and similar to WiFi perhaps, but worse than PCIe cards that should be <1ms.

I think there's definitely something with that specific setup. For me, pinging between two cheap Realtek 2.5 GbE USB dongles (one is on a Mac one is on a 7 year old Intel Atom Synology) is still sub-ms (hovering around 0.7-0.8ms) so it's not an inherent problem to USB dongles.

USB itself can have a lot of issues anywhere in the chain. I have a Thunderbolt dock where half of the USB ports adds latency and reduced throughput just because the USB chipset that powers them is terrible (it has two separate USB chipsets from different brands). Switch to a different port on the exact same dock and it's fine.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#114

Earlier quoted context omitted.

The new low-power Realtek chipsets will definitely push 10 GbE forward because the chipset won't be much more expensive to integrate and run than the 2.5Gbps packages. It all comes down to performance per Watt, the availability of cheap switching gear, and the actual utility in an office / home environment. For 10 Gbps, cabling can be an issue. Existing "RJ45"-style Cat 6 cables could still work, but maybe not all of…

But why is the gear progressing so very slowly? Why a 25 year gap between reasonable power 1Gbps and 10Gbps? And while not every cat6 will do 10, it would still be worth a shot, and devices aren't using 5 instead they're using even less. Not to mention that cat8 will happily do 40Gbps as long as you can get from your switch to your end devices in 30 meters.

Perhaps because internet speeds have been under 1gbps most places until recently and the average person doesn't care about file transfers over LAN.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#115
I've had a lot of problems with even 10GbE via Thunderbolt 3/4. Bandwidth-wise it works fine, but latency and jitter are issues. This means that stuff like high-speed cameras that need to be synchronized over Ethernet using Precision Time Protocol (PTP) tend to simply fail with these devices.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#116

I've had a lot of problems with even 10GbE via Thunderbolt 3/4. Bandwidth-wise it works fine, but latency and jitter are issues. This means that stuff like high-speed cameras that need to be synchronized over Ethernet using Precision Time Protocol (PTP) tend to simply fail with these devices.

I’d heard similar complaints re: TB networking latency & jitter. Did some investigations and tuning on a pair of machines with USB4 ports connected via short TB5-rated cables. Eventually got the thunderbolt links to consistently beat the ether ones on both latency and jitter. And not just switched Ethernet either - even a direct Ethernet P2P link lost out to TB, though the difference there was small.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#118
Is this satire?

> All other 25 GbE adapter solutions I’ve found so far ... have a spinning fan. ... the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. ... Other than that, the PX seems to do the job

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#119
post #29

Earlier quoted context omitted.

Too bad Jeff Geerling returned his Mac Studios to Apple. Would be lovely to see how 5x faster RDMA impacts the performance.

25 Gbps isn't all that much. It would be good, but would be below the 40+ Gbps I was getting on the TB5 ring network. I think where it would show more significant speed up is on the AMD Strix Halo cluster. Except I haven't been able to get RDMA over Thunderbolt on there to work, so it'd be apples to oranges comparing ConnectX to Thunderbolt on Linux.

Oddly enough, that’s exactly what I’ve been benchmarking - different ways of linking Strix Halo machines - with respect to throughput & latency.

Posted a little bit re: the TB side of things on the Framework and Level1Techs forums but haven’t pulled everything together yet because the higher-speed Ethernet and Infiniband data is still being collected.

So far my observations re: TB is that, on Strix Halo specifically, while latency can be excellent there seem to be some limits on throughput. My tests cap out at ~11Gbps unidir (Tx|Rx), ~22Gbps bidi (Tx+Rx). Which is wierd because the USB4 ports are advertised at 40Gbps bidi, the links report as 2x20Gbs, and are stable with no errors/flapping - so not a cabling problem.

The issue seems rather specific to TB networking on Strix Halo using the USB4 links between machines.

Emphasis to exclude common exceptions - other platforms eg Intel users getting well over 20Gbps; other mini PCs eg MS-1 Max USB4v2; local network eg I’ve measured loopback >100Gbps; or external storage where folk are seeing 18Gbps+ / numbers that align with their devices.

Emd goal is to get hard data on all reasonably achievable link types. Already have data on TB & lower-speed Ethernet (switched & P2P), currently doing setup & tuning on some Mellanox cards to collect data for higher-speed Ethernet and IB. P2P-only for now; 100GbE switching is becoming mainstream but IB switches are still rather nutty.

Happy to collaborate with any other folk interested in this topic. Reach out to (username at pm dot me).

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#120

Earlier quoted context omitted.

I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.

AWESOME!! Do you mind linking references? I put the project aside because of these issues.

Yea I'll dig up the link I used - there was a great reference about getting the thunderbolt working after reboots/etc.
Post reply on HN