Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reliable 25 Gigabit Ethernet via Thunderbolt

kohlschuetter.github.io

Post reply on HN
Print view View original

121–130 of 133 posts

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#121

Post by clawsyndicate »

kasey_junk wrote: post #54
omgtehlion wrote: post #43

Ha! Been running these for years on both linux and windows (on lenovo x1 laptops). Using cheap chinese thunderbolt-to-nvme adapters + nvme-to-pcie boards + mellanox cx4 cards (recently got one cx5 and a solarflare x2). Pic of a previous cx3 (10 gig on tb3) setup: https://habrastorage.org/r/w780/getpro/habr/upload_files/d3c... 10gig can saturate full speed, 25G in my experience rarely reaches same 20G as the author ob…

If you don’t mind me asking, what are you using these for? Saturating these seems like it would have reasonably few workloads outside of like cdn or multi-tenant scenarios. Curious what my lack of imagination is hiding here.

[flagged]

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#122

Post by sponaugle »

nodesocket wrote: post #102
sponaugle wrote: post #77

Earlier quoted context omitted.

I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.

I figured this was Jeff’s CTO Laboratory. I enjoy your channel. Are you kicking yourself for pulling the trigger on MS-01s now that MS-02 Ultras released?

Yea! Those MS-O2s look great, so I may need to upgrade! I did get a couple of DGX Sparks to play with.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#123

Post by sponaugle »

omgtehlion wrote: post #79
sponaugle wrote: post #75

Earlier quoted context omitted.

"because I can, and it is fun." The best answer! I am most of the way done with upgrading most of my homelab to 100G from 10G, but there really isn't a practical reason for it. 100G has dropped in price so much as datacenters are all about 400/800G now.

Nice! Cool to hear from a fellow admirer of overkill-lan setups ;) Which cards do you prefer for 100G, and what is the situation with dacs/optics?

I'm using ConnectX5s for most, and some ConnectX4s in the older servers. Both of those cards have really come down in price in the used/ebay market. I have been playing around with some different optics - I have a bunch of CWDM4s which are very inexpensive and use a single SingleMode pair.... but of course they run hot so if you have them in servers without good air flow you might have problems.

I'm using mostly fiber just because the servers are connected to Cisco 9305 with 72 100g ports.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#124

Post by sponaugle »

StillBored wrote: post #111
omgtehlion wrote: post #79

Earlier quoted context omitted.

Nice! Cool to hear from a fellow admirer of overkill-lan setups ;) Which cards do you prefer for 100G, and what is the situation with dacs/optics?

I wouldn't really call 100Gbit overkill, if you compare it to modern disk drives is about where we should be relative to shared storage/NAS/etc infrastructure people used to run. So yes, being able to share my /home directory across a few dozen machines at my house without a huge perf impact vs using a local drive seems a pretty reasonable use case. Sure its faster than my WAN access, but who cares? Frankly, 10Gbit i…

Agreed - the big thing is 100g is much much cheaper now as so much 100g gear is coming out of datacenters. So many of those older ConnectX4s and 5s, plus lots of switches and optics. 100g really is the new 10g for homelabs.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#125

Post by omgtehlion »

sponaugle wrote: post #123
omgtehlion wrote: post #79

Earlier quoted context omitted.

Nice! Cool to hear from a fellow admirer of overkill-lan setups ;) Which cards do you prefer for 100G, and what is the situation with dacs/optics?

I'm using ConnectX5s for most, and some ConnectX4s in the older servers. Both of those cards have really come down in price in the used/ebay market. I have been playing around with some different optics - I have a bunch of CWDM4s which are very inexpensive and use a single SingleMode pair.... but of course they run hot so if you have them in servers without good air flow you might have problems. I'm using mostly fibe…

Thanks! That cisco sounds like a hell of equipment. Is it loud? And how much power does it draw?

And thanks for pointing at CWDM4, these are quite cheap on ebay now

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#126

Post by prmoustache »

whatever1 wrote: post #82

Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster. Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.

> Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.

wifi is not faster.

However ethernet is not as critical as it used to be, even at the office. People like the conveniency of having laptops they can move around. Unless you are working from home, having a dedicated office space is now seen as a waste of space. If the speed of the wifi is good enough when you are in a meeting room or in your kitchen, there is no reason to plug your laptop when you move back in another place, especially if most connections are to the internet and not the local network. In the workplace, most NAS have been replaced by onedrive / gdrive, at home NAS use has always been limited to a niche population: nerds/techies, photographers, music or video producers...

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#127

Post by fulafel »

Gigachad wrote: post #94
fulafel wrote: post #85

Earlier quoted context omitted.

Wireless happened, I'd think. People started using wifi and cellular data for everything, so applications had to adapt to this lowest common denominator, and consumer broadband demand for faster-than-wifi speeds isn't there. Plus operators put all their money into cellular infra leaving no money to update broadband infra.

Wifi now can pretty realistically beat 2.5gbit/s while most Ethernet is still gigabit. It just seems strange to live in a world where the average laptop will get a faster connection speed over wifi than plugged in to Ethernet.

Ethernet is following suit to 2.5G which is otherwise a nonsensical step for ethernet speeds, I think this is further evidence that everything just follows wifi now.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#128

Post by addandsubtract »

kohlschuetter wrote: post #50
throwaway2037 wrote: post #45

This is an outstanding blog post. Initially, the title did little to captivate me, but the blog post was so well written that I got nerd-sniped. Who knew this little adapter was so fascinating! I wonder if the manufacturer is buying the Mellanox cards used from data center tear-downs. The author claims they can be had for only 20 USD online. That seems too good to be true! Small thing: I just checked Amazon.com: http…

Thank you! I think, tragically, the blog post has caused this price increase. The offers on Amazon are most likely all drop shippers trying to gauge a price that works for them. You might have better luck ordering directly from China for a fraction of the price: https://detail.1688.com/offer/836680468489.html

AliExpress sells them, too, which is much more accessible for anyone living in the EU.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#129

Post by sponaugle »

omgtehlion wrote: post #125
sponaugle wrote: post #123

Earlier quoted context omitted.

I'm using ConnectX5s for most, and some ConnectX4s in the older servers. Both of those cards have really come down in price in the used/ebay market. I have been playing around with some different optics - I have a bunch of CWDM4s which are very inexpensive and use a single SingleMode pair.... but of course they run hot so if you have them in servers without good air flow you might have problems. I'm using mostly fibe…

Thanks! That cisco sounds like a hell of equipment. Is it loud? And how much power does it draw? And thanks for pointing at CWDM4, these are quite cheap on ebay now

Oh they are both loud and eat a ton of power. I think the 9305 is at least 800 watts at idle. That is the biggest downside of the retired datacenter gear... yo u really need a dedicated room with power, cooling, and sound isolation.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#130

Post by icelancer »

pdrayton wrote: post #116
icelancer wrote: post #115

I've had a lot of problems with even 10GbE via Thunderbolt 3/4. Bandwidth-wise it works fine, but latency and jitter are issues. This means that stuff like high-speed cameras that need to be synchronized over Ethernet using Precision Time Protocol (PTP) tend to simply fail with these devices.

I’d heard similar complaints re: TB networking latency & jitter. Did some investigations and tuning on a pair of machines with USB4 ports connected via short TB5-rated cables. Eventually got the thunderbolt links to consistently beat the ether ones on both latency and jitter. And not just switched Ethernet either - even a direct Ethernet P2P link lost out to TB, though the difference there was small.

Interesting - I'll try some other methods
Post reply on HN