Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reliable 25 Gigabit Ethernet via Thunderbolt

kohlschuetter.github.io

Post reply on HN
Print view View original

71–80 of 133 posts

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#71

Post by butvacuum »

admash wrote: post #70
kohlschuetter wrote: post #21

Earlier quoted context omitted.

Yes! However, I got around 15 Gbps with a Thunderbolt-only setup (TB3/TB4) = only 75% of the Ethernet setup. You'd also mostly be limited to short cables (1-2m) and a ring topology.

Any idea why that would be the case?

The linux implementation is quite poor. Among other issues, your answer is linux treats it as a TCP/IP link- so packets don't have any offloading for checksums, etc. it's also incomplete (ex- you have to physically unplug and replug the cord every time one side loses link- even a reboot).

This is my firsthand experence trying to get some tablet motherboards to link up and work as a proxmox cluster w/ TB3 as the link between nodes.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#72

Post by usagisushi »

kohlschuetter wrote: post #56
usagisushi wrote: post #46

Earlier quoted context omitted.

JFYI, for measuring power draw, you might be able to use `macmon`[0] to see the total system power consumption. The values reported by the internal current sensor seem to be quite accurate. [0] https://github.com/vladkens/macmon

Very nice tip, thank you! I measure around +11W idle. While running a speed test, I read ca. +15W.

Thanks for the measurements! 15W under load definitely justifies those massive heatsinks.

I’m looking forward to your writeup on the RTL8127AF as well. Your blog is awesome!

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#73

Post by drnick1 »

I used to have an SFP28 Mellanox card in my home server, but went back to a simple 2.5G Ethernet port for the LAN side. The Mellanox card ran hot and needed an extra fan near it to dissipate the heat. It was cool but there was no real benefit other than occasionally when transferring some large files.

Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors like the one presented here are unreliable and add several ms of latency.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#74

Post by fc417fc802 »

drawfloat wrote: post #27
mcny wrote: post #6

Earlier quoted context omitted.

I don't know about the Ethernet part but it bothers me that even wifi has become faster than the wired USB port on our phones. All I want to do is copy over all the photos and videos from my phone to my computer but I have to baby sit the process and think whether I want to skip or retry a failed copy. And it is so slow. USB 2.0 slow. I guess everybody has given up on the idea of saving their photos and videos over U…

I feel like this is an artifact from the late 2010s when the talk was of removing the port completely from phones, where that was being touted alongside swapping speakers with haptic screen audio as a way to make them completely waterproof. As wireless charging never quite reached the level hoped – see AirPower – and Google/Apple seemingly bought and never did anything with a bunch of haptic audio startups, I figure…

I'd usually be against losing ports and user serviceable stuff but if the device could actually be properly sealed up (ie no speakers, mics, charge ports, etc) that would be legitimately useful.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#75

Post by sponaugle »

omgtehlion wrote: post #67
kasey_junk wrote: post #54

Earlier quoted context omitted.

If you don’t mind me asking, what are you using these for? Saturating these seems like it would have reasonably few workloads outside of like cdn or multi-tenant scenarios. Curious what my lack of imagination is hiding here.

Officially: to access NAS, get raw market-data files (tens to hundreds of gigabytes a day), not needed on laptop every day, but only once in a while to fix or analyze something. Really: because I can, and it is fun. I upgraded my home lan to 10G, because used 10G hardware is cheap (and now 25G enters the same price range).

"because I can, and it is fun." The best answer! I am most of the way done with upgrading most of my homelab to 100G from 10G, but there really isn't a practical reason for it. 100G has dropped in price so much as datacenters are all about 400/800G now.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#76

Post by kohlschuetter »

drnick1 wrote: post #73

I used to have an SFP28 Mellanox card in my home server, but went back to a simple 2.5G Ethernet port for the LAN side. The Mellanox card ran hot and needed an extra fan near it to dissipate the heat. It was cool but there was no real benefit other than occasionally when transferring some large files. Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors lik…

For reference, I'm seeing pings from my Mac to my Linux boxes (Lenovo Tiny5) at well under 1ms, not much worse than between them directly. But yeah, your mileage may vary.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#77

Post by sponaugle »

butvacuum wrote: post #71
admash wrote: post #70

Earlier quoted context omitted.

Any idea why that would be the case?

The linux implementation is quite poor. Among other issues, your answer is linux treats it as a TCP/IP link- so packets don't have any offloading for checksums, etc. it's also incomplete (ex- you have to physically unplug and replug the cord every time one side loses link- even a reboot). This is my firsthand experence trying to get some tablet motherboards to link up and work as a proxmox cluster w/ TB3 as the link…

I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#79

Post by omgtehlion »

sponaugle wrote: post #75
omgtehlion wrote: post #67

Earlier quoted context omitted.

Officially: to access NAS, get raw market-data files (tens to hundreds of gigabytes a day), not needed on laptop every day, but only once in a while to fix or analyze something. Really: because I can, and it is fun. I upgraded my home lan to 10G, because used 10G hardware is cheap (and now 25G enters the same price range).

"because I can, and it is fun." The best answer! I am most of the way done with upgrading most of my homelab to 100G from 10G, but there really isn't a practical reason for it. 100G has dropped in price so much as datacenters are all about 400/800G now.

Nice! Cool to hear from a fellow admirer of overkill-lan setups ;)

Which cards do you prefer for 100G, and what is the situation with dacs/optics?

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#80

Post by omgtehlion »

drnick1 wrote: post #73

I used to have an SFP28 Mellanox card in my home server, but went back to a simple 2.5G Ethernet port for the LAN side. The Mellanox card ran hot and needed an extra fan near it to dissipate the heat. It was cool but there was no real benefit other than occasionally when transferring some large files. Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors lik…

Yep, these cards need a fan (or any kind of directed air flow).

Where did you get "several ms of latency" figure from? I have not measured external card, but may be I should do it... Because cards themselves have latency in range of microseconds, not millis.

Post reply on HN