Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Add 10 GbE to your system with an M.2 2280 module

cnx-software.com

Post reply on HN
Print view View original

171–180 of 259 posts

Re: Add 10 GbE to your system with an M.2 2280 module

#171

Post by zamadatix »

aix1 wrote: post #157
ericd wrote: post #81

Earlier quoted context omitted.

Yeah, the mikrotiks recommend populating only every other slot if you're using RJ45 transceivers to avoid overheating.

Is the power consumption (and heat dissipation) a function of bit rate? E.g. would the same 10GBASE-T transceiver consume less power when running at, say, 2.5Gb/s than at 10Gb/s? Would love to understand this a bit better. (edit: corrected the units.)

Yes (or at least I've noticed it on mine) but worth noting 99% of 10G RJ45 SFP+ transceivers only support 10G and nothing else. Typically it's only fixed copper interfaces that support negotiating different speeds.

The MikroTik adapters are a bit special in that they are more a 2 port transparent bridge where the inside facing portion of the module always runs at 10g and the outside facing portion auto-negotiates. This allows 10G only switch interfaces to support 10/100/1000/2500/5000/10000 clients. In a MikroTik switch it reports back the negotiated speed and the switch can shape traffic to the appropriate bandwidth instead of letting the adapter do it (which I assume is just policed but could be wrong, never tested). I have a 100G switch which is backwards compatible with 40G and allows breakouts so with a QSA can support SFP+ modules... putting this MikroTik module in I can plug a 10 megabit half duplex device into a 100 gigabit port!

Re: Add 10 GbE to your system with an M.2 2280 module

#172

Post by omgtehlion »

loosescrews wrote: post #5

I have had a lot of trouble with the Marvell AQtion controllers under Linux. They are supposed to work and are plug and play on modern kernels, but I was never able to resolve a bug where the controller stopped working after a few hours. The only resolution I found was rebooting, which made the controller not very useful. The Intel 10GbE controllers are more expensive, but much better in my experience. For anyone loo…

If you do not need a cable, there are m.2 to 4x PCIe (open slot, so you can insert 8x and 16x cards) for $5.

Re: Add 10 GbE to your system with an M.2 2280 module

#173

Post by zamadatix »

I think a lot are missing that this is extremely low profile and has a separated port in contrast to a more typical PCIe riser adapter and HHHL card. Much more suited for hacking in embedded builds, 1u builds where a gpu already takes the horizontal slot, or even laptops if you're brave enough. I don't think it's meant to replace/compete a standard m.2 to PCIe riser on an HTPC or NAS that most are used to seeing.

Re: Add 10 GbE to your system with an M.2 2280 module

#174

Post by toast0 »

gjulianm wrote: post #57
walrus01 wrote: post #37

Earlier quoted context omitted.

10GbE and 100GbE on fiber is quite cheap and easy now - but 99% of consumers and people doing ordinary stuff have no capability or interest in doing fiber. You can terminate cat5e or cat6 with $25 in hand tools... I think what's new is the prevalence now of 2.5 and 5GBaseT ethernet chips that are cheap enough companies are starting to build them into any $125+ ATX motherboard. At short lengths even old crappy cat5e h…

Even 100GbE is hardly seen on company datacenters. Yes, it's cheaper than before, but still more expensive than 10G, and that's extra cost multiplied by all the devices that need to have the improved hardware to take advantage. Plus, most servers won't saturate a 10G link without tweaks on the setup. For 100G it's even worse, I think it will take a long time to see them on datacenters outside of core links or for com…

> Plus, most servers won't saturate a 10G link without tweaks on the setup.

That doesn't seem right. When I got my first 10G server, it was running dual Xeon E5-2690 (either v1 or v2), and I don't recall needing to tweak much of anything. That was mostly a single large file downloaded over http, so not super hard to tweak anyway, but server chips are a lot better now than sandy/ivy bridge. It could only get 9gbps out with https, but the 2690v4 could do 2x10G with https because aes acceleration.

Re: Add 10 GbE to your system with an M.2 2280 module

#176

Post by aix1 »

zamadatix wrote: post #171
aix1 wrote: post #157

Earlier quoted context omitted.

Is the power consumption (and heat dissipation) a function of bit rate? E.g. would the same 10GBASE-T transceiver consume less power when running at, say, 2.5Gb/s than at 10Gb/s? Would love to understand this a bit better. (edit: corrected the units.)

Yes (or at least I've noticed it on mine) but worth noting 99% of 10G RJ45 SFP+ transceivers only support 10G and nothing else. Typically it's only fixed copper interfaces that support negotiating different speeds. The MikroTik adapters are a bit special in that they are more a 2 port transparent bridge where the inside facing portion of the module always runs at 10g and the outside facing portion auto-negotiates. Th…

Super interesting re Mikrotik adapters. Those (S+RJ10) are exactly what I run, so very relevant, thank you!

Re: Add 10 GbE to your system with an M.2 2280 module

#177

Post by DannyBee »

jabl wrote: post #137
DannyBee wrote: post #129

Earlier quoted context omitted.

The main thing, i think, is that outside of DAC, most faster speeds are used over fiber. The cabling starts to become an issue otherwise. It's also hot and power hungry over copper. Having gotten into the home fiber side of it for various reasons, it's clear on the fiber side there is plenty of innovation/cost lowering/etc.

I mostly agree, but optical transceivers are still expensive. Been waiting for dirt cheap silicon photonics for a couple of decades now. But it seems they are on track to be a thing, though initially for high-speed networking not home/office type stuff. Maybe one day, sigh.

I'm not sure how you define expensive. https://www.fs.com/c/10g-sfp-plus-63

20 bucks over MMF, 27 bucks over SMF doesn't seem expensive.

If you want to do it over single fiber rather than duplex, it's 40 bucks: https://www.fs.com/c/bidi-sfp-plus-64

25Gbps is 39 bucks over MMF, 59 bucks over SMF. 40Gbps is also 39 bucks on MMF (more expensive on SMF)

I don't think any of these are very expensive.

The cards are also ~same price between SFP+ and 10GBaseT from places like startech (outside of that, the 10gbaset ones are actually often much more expensive)

Re: Add 10 GbE to your system with an M.2 2280 module

#178

Post by ericpauley »

_flux wrote: post #142
kortilla wrote: post #108

Earlier quoted context omitted.

A lot of that sounds pretty incredibly wasteful and I don’t think it should be something to strive for more of. Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong. Azure charges (like the other major cloud providers) by egress data. I feel bad for whoever is paying the bill for 20gbps going out. In general, people in ML shoul…

> Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong. Well the images need to be in some order, and if you are fine tuning the very initial parts of the image, this can happen. But it seems a better transfer algorithm could be used for those individual layers. It rather seems like if they don't match, everything's transferred…

base64-encoding data only poses a 1% overhead[1] when the data is subsequently gzipped, so this is hardly an issue in practice.

[1] head -c 100000 /dev/urandom | base64 | gzip | wc -c

Re: Add 10 GbE to your system with an M.2 2280 module

#179

Post by junon »

kkielhofner wrote: post #147
junon wrote: post #115

Earlier quoted context omitted.

> If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong When I worked at Vercel during the days we supported Docker hosting (back when it was still ZEIT), we saw that even among many tenants on the same machine, the "layers" thing wasn't really that beneficial - we still had to fetch a ton of stuff over and over again. So this is great in theory, but it doesn't really…

I believe Docker has even throttled/blocked corporate users doing excessive layer pulls from Hub (and suggested they sign up for X plan). Reasonable, IMHO.

That's a recent thing, as I understand it. After Hub switched to their enterprise plan setup.

ZEIT had a pretty aggressive cache anyway, for the short time it was implemented (before ZEIT moved away from docker hosting, at least).

Re: Add 10 GbE to your system with an M.2 2280 module

#180

Post by [deleted] »

_flux wrote: post #142
kortilla wrote: post #108

Earlier quoted context omitted.

A lot of that sounds pretty incredibly wasteful and I don’t think it should be something to strive for more of. Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong. Azure charges (like the other major cloud providers) by egress data. I feel bad for whoever is paying the bill for 20gbps going out. In general, people in ML shoul…

> Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong. Well the images need to be in some order, and if you are fine tuning the very initial parts of the image, this can happen. But it seems a better transfer algorithm could be used for those individual layers. It rather seems like if they don't match, everything's transferred…

[deleted]
Post reply on HN