Live data from Hacker News

Reliable 25 Gigabit Ethernet via Thunderbolt

kohlschuetter.github.io

61–70 of 133 posts

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#61

nitpicking but why would someone type `sudo su` vs `sudo -i`

Muscle memory for folks who have been doing it since before -i was an option. I still instinctively type `sudo su -` because it worked consistently on older deployments. When you have to operate a fleet of varying ages and distributions, you tend to quickly learn [if only out of frustration] what works everywhere vs only on the newer stuff.

`sudo su - ` also seems easier for me to type than `sudo -i -u `

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#62

This is an outstanding blog post. Initially, the title did little to captivate me, but the blog post was so well written that I got nerd-sniped. Who knew this little adapter was so fascinating! I wonder if the manufacturer is buying the Mellanox cards used from data center tear-downs. The author claims they can be had for only 20 USD online. That seems too good to be true! Small thing: I just checked Amazon.com: http…

I believe the author is talking about the OCP (2.0) network card itself, that these adapters internally. The OCP nics are quite cheap compared to pcie - here’s 100GBE for 100! https://ebay.us/m/HMQAph

This 100GbE card is an OCP 2.0 type 2 adapter, which will _probably_ not work with the PX PCB since that NIC has two of these mezzanine connectors, and PX only one.

What also may not work are Dell rNDC cards. They look like they have OCP 2.0 type 1 connectors, but may not quite fit (please correct me if I'm wrong). They do however have a nice cooling solution, which could be retrofitted to one of the OCP 2.0 cards.

I've also ordered a Chelsio T6225-OCP cards out of curiosity. These should fit in the PX adapter but require a 3rd-party driver on macOS (which then supports jumbo frames, etc.)

What also fits physically is a Broadcom BCM957304M3040C, but there are no drivers on macOS, and I couldn't get the firmware updated on Linux either.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#63

This is an outstanding blog post. Initially, the title did little to captivate me, but the blog post was so well written that I got nerd-sniped. Who knew this little adapter was so fascinating! I wonder if the manufacturer is buying the Mellanox cards used from data center tear-downs. The author claims they can be had for only 20 USD online. That seems too good to be true! Small thing: I just checked Amazon.com: http…

I saw the blog post last week and immediately bought the last one on that Amazon listing for the original price... hopefully they restock soon! I'm going to try a couple other fan assisted cooling options, as I'd like to keep the setup reasonably compact. I just ran fiber to my desk and I have a more expensive QNAP unit that does 10G SFP+, but this will let me max out the connection to my NAS.

Be sure to test this adapter on iPad Pro, just for kicks (yes it works!)

Although I managed to panic the kernel a couple of times without the extra heatsinks on...

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#64
post #37

> reduces temperatures by at least 15 Kelvin, bringing the ambient enclosure temperature below 40 °C, I had to do a double-take when it mentioned Kelvin since That is physically impossible.

Isn't "reduces temperatures by 15 Kelvin" the same as "reduces temperatures by 15 Celsius"?

Yes, Kelvin is only a linear offset from Celsius. (273.15 for anyone who doesn’t already know).

It’s a little bit funny/coy to use it mixed with Celsius.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#65
post #47

Earlier quoted context omitted.

Wifi is fast but the latency is terrible and the reliability is even worse. It can go up and down like a yo-yo. USB is far more predictable even if it is a bit slower.

I have a cluster of 4 RPi Zero Ws and network reliability is not great. Since it is for the chaos, it’s fine, but it’s very common to have a node be offline at any given time. Even worse, the control plane is exposed, but for something that runs 3 Hercules mainframe emulation and two Altairs with MP/M, it’s fine.

I have a bunch of HA wifi connected sensors, I see them drop off and reconnect all the time it is most annoying.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#66

Earlier quoted context omitted.

JFYI, for measuring power draw, you might be able to use `macmon`[0] to see the total system power consumption. The values reported by the internal current sensor seem to be quite accurate. [0] https://github.com/vladkens/macmon

Speaking of hardware, the RTL8159 (10Gbps) hit the market late last year and is said to consume only about 2–3W. It apparently runs very cool compared to older chips. (Though it would need to be bonded to reach 25Gbps ;-)

I got me one of these adapters (RTL8127AF TXA403, with SFP+ cage); I haven't properly benchmarked it yet.

There's no driver support on macOS, and for Linux you'd need a bleeding edge kernel. Just trying to physically connect it (along with a connected SFP28 transceiver) to my Mac's Thunderbolt port using an external PCIe-to-TB adapter, macmon tells me a power draw of around 4.3 W, so it's not significantly less for half the bandwidth, but the card doesn't get hot at all.

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#67

Ha! Been running these for years on both linux and windows (on lenovo x1 laptops). Using cheap chinese thunderbolt-to-nvme adapters + nvme-to-pcie boards + mellanox cx4 cards (recently got one cx5 and a solarflare x2). Pic of a previous cx3 (10 gig on tb3) setup: https://habrastorage.org/r/w780/getpro/habr/upload_files/d3c... 10gig can saturate full speed, 25G in my experience rarely reaches same 20G as the author ob…

If you don’t mind me asking, what are you using these for? Saturating these seems like it would have reasonably few workloads outside of like cdn or multi-tenant scenarios. Curious what my lack of imagination is hiding here.

Officially: to access NAS, get raw market-data files (tens to hundreds of gigabytes a day), not needed on laptop every day, but only once in a while to fix or analyze something.

Really: because I can, and it is fun. I upgraded my home lan to 10G, because used 10G hardware is cheap (and now 25G enters the same price range).

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#68

Earlier quoted context omitted.

I believe the author is talking about the OCP (2.0) network card itself, that these adapters internally. The OCP nics are quite cheap compared to pcie - here’s 100GBE for 100! https://ebay.us/m/HMQAph

This 100GbE card is an OCP 2.0 type 2 adapter, which will _probably_ not work with the PX PCB since that NIC has two of these mezzanine connectors, and PX only one. What also may not work are Dell rNDC cards. They look like they have OCP 2.0 type 1 connectors, but may not quite fit (please correct me if I'm wrong). They do however have a nice cooling solution, which could be retrofitted to one of the OCP 2.0 cards. I…

That’s a good point to note! I think the stacking height would matter, but in theory the single connector is still 8x pcie and should link without the upper 8x lanes connected.

Spec for reference, I’m not 100% sure. https://docs.nvidia.com/nvidia-connectx-5-ethernet-adapter-c...

Re: Reliable 25 Gigabit Ethernet via Thunderbolt

#70
post #17

Note that you can do point-to-point network links directly with thunderbolt (and usb4). https://support.apple.com/guide/mac-help/ip-thunderbolt-conn... etc

Yes! However, I got around 15 Gbps with a Thunderbolt-only setup (TB3/TB4) = only 75% of the Ethernet setup. You'd also mostly be limited to short cables (1-2m) and a ring topology.

Any idea why that would be the case?
Post reply on HN