Add 10 GbE to your system with an M.2 2280 module • Hacker News Board

Re: Add 10 GbE to your system with an M.2 2280 module

#201

Post by sigstoat » Wed, Jan 19, 2022, 5:47 PM UTC

soneil wrote: post #78

martinald wrote: post #9

It's interesting to see just how slow faster ethernet standards have been. Feels like I've had gigabit ethernet at a pretty low cost for nearly 20 years now, but faster than that has been pretty esoteric outside of the datacentre. I guess there really isn't much demand for faster than gigabit speeds even now (outside of servers?)

These are naive takes, accounting only for linespeed and nothing more, but give a useful rule of thumb: - An 8x cdrom narrowly beats 10meg ethernet. - 1x dvdrom narrowly beats 100meg ethernet. - ATA133 narrowly beats 1gbit ethernet.

> - 1x dvdrom narrowly beats 100meg ethernet.

that doesn't fit with my recollection of reading/transferring DVDs, or with https://en.wikipedia.org/wiki/Optical_storage_media_writing_...

> - ATA133 narrowly beats 1gbit ethernet.

the electrical interface/protocol, sure. i don't think any ATA133 drive made could actually saturate its interface, or a gigabit link.

Re: Add 10 GbE to your system with an M.2 2280 module

#202

Post by mrb » Wed, Jan 19, 2022, 5:55 PM UTC

YXNjaGVyZWdlbgo wrote: post #99

pdimitar wrote: post #93

Earlier quoted context omitted.

In my case I'd like to transfer from/to my NAS where I centralize a number of data pieces. I work inside a workstation but occasionally on a laptop as well. Having 10GbE will help me when I'm on my workstation. It's not fatal, definitely, but it adds up to lost productivity.

Is your NAS NVME based if you are using spinning metal you won't reach speeds to saturate 10GbE especially with small file sizes?

I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.

Re: Add 10 GbE to your system with an M.2 2280 module

#203

Post by ashtuchkin » Wed, Jan 19, 2022, 6:06 PM UTC

_flux wrote: post #142

kortilla wrote: post #108

Earlier quoted context omitted.

A lot of that sounds pretty incredibly wasteful and I don’t think it should be something to strive for more of. Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong. Azure charges (like the other major cloud providers) by egress data. I feel bad for whoever is paying the bill for 20gbps going out. In general, people in ML shoul…

> Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong. Well the images need to be in some order, and if you are fine tuning the very initial parts of the image, this can happen. But it seems a better transfer algorithm could be used for those individual layers. It rather seems like if they don't match, everything's transferred…

json is used only for manifests; actual layers are .tar.gz.

Re: Add 10 GbE to your system with an M.2 2280 module

#204

Post by [deleted] » Wed, Jan 19, 2022, 6:14 PM UTC

Gigachad wrote: post #97

vladvasiliu wrote: post #22

Earlier quoted context omitted.

> But I guess most people don't transfer files in their local networks anymore and use their network purely for internet access. I think this is clearly the case. Most new laptops don't even bother with a wired network port. I've got a new "pro" HP laptop the other day, and it only comes with some cheap Wi-Fi card. And it's not an "entry-level" laptop, and it's thick enough for an RJ-45 plug to physically fit. I also…

It’s all usb c now. My iPad Pro has 10 gbit Ethernet support over the usb C port.

[deleted]

Re: Add 10 GbE to your system with an M.2 2280 module

#205

Post by pdimitar » Wed, Jan 19, 2022, 6:16 PM UTC

jjoonathan wrote: post #145

YXNjaGVyZWdlbgo wrote: post #99

Earlier quoted context omitted.

Is your NAS NVME based if you are using spinning metal you won't reach speeds to saturate 10GbE especially with small file sizes?

If somebody knows why they should pay a premium for 10GbE, they also know how bad spinning rust is.

Put 8 high-quality HDDs in a stripe & mirror config. Thus you can read from 4 simultaneously. They each top at 260MB/s. That amounts to little over 1GB/s -- which is 10GbE.

Re: Add 10 GbE to your system with an M.2 2280 module

#206

Post by csdvrx » Wed, Jan 19, 2022, 6:27 PM UTC

mrb wrote: post #202

YXNjaGVyZWdlbgo wrote: post #99

Earlier quoted context omitted.

Is your NAS NVME based if you are using spinning metal you won't reach speeds to saturate 10GbE especially with small file sizes?

I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.

How did you bench your ZFS pool?

Re: Add 10 GbE to your system with an M.2 2280 module

#207

Post by csdvrx » Wed, Jan 19, 2022, 6:29 PM UTC

omgtehlion wrote: post #169

virtuallynathan wrote: post #134

Earlier quoted context omitted.

100GbE NICs are <$500, and 100GbE switch ports are $100-150.

while 10G is now at sub-$30 range, totally accessible to everyone )

Could you recommend a switch and some NICs, along with a thunderbolt adapter?

Re: Add 10 GbE to your system with an M.2 2280 module

#208

Post by liketochill » Wed, Jan 19, 2022, 6:34 PM UTC

sigstoat wrote: post #197

erik wrote: post #84

Earlier quoted context omitted.

Another factor: data centres moved to fiber. And fiber is less physically robust and not great for desktop connections or plugging in to a laptop. 10GBASE-T exists, but it turns out pushing 10gbit/s over 100m of twisted pair requires chips that are hot and power hungry. Again, not great for desktops or laptops. And because it’s not used in data centres, there are no economies of scale or trickle down effects. Gigabit…

> And fiber is less physically robust and not great for desktop connections i buy armored multimode patch cables for, i dunno, 10 or 20% more per unit length, and they seem indestructible in my residential short-distance use cases. > Gigabit Ethernet and wifi being “good enough” i think this explains it all. when the average user prefers wireless to wired gigabit, we know how much bandwidth they actually need, and it…

It takes special connectors to survive being disconnected and reconnected daily and cables left disconnected with out the risk of an accidental laser eye surgery

Re: Add 10 GbE to your system with an M.2 2280 module

#209

Post by Teknoman117 » Wed, Jan 19, 2022, 6:51 PM UTC

PragmaticPulp wrote: post #140

jiggawatts wrote: post #80

Some people are saying that they don't feel the need for > 1 Gbps in a home network. As a counter-point, I'm regularly limited by 1 Gbps PHY limits. During the 2 years of the pandemic I've been working from home with gigabit Internet. Why gigabit? Because that's the "speed limit" of the Ethernet PHY in the fibre broadband box, and also my laptop's ethernet port. My laptop can write to disk at multiple GB per second,…

As a counterpoint: I had 10G networking and went back to 1G last time I reorganized. Haven’t really missed the 10G for now . I ditched the 10G because the 10G switch was hot and noisy and had a fan, whereas my 1G equipment was silent and low power. This could be overcome by putting the switch in another room, obviously, but I haven’t missed it enough to go through the trouble yet. > During the 2 years of the pandemic…

I ended up buying one of these for my 10g home network. It's completely fanless. The 1G Ubiquity EdgeRouter-X I have runs way hotter.

https://mikrotik.com/product/crs309_1g_8s_in

Re: Add 10 GbE to your system with an M.2 2280 module

#210

Post by omgtehlion » Wed, Jan 19, 2022, 7:10 PM UTC

csdvrx wrote: post #207

omgtehlion wrote: post #169

Earlier quoted context omitted.

while 10G is now at sub-$30 range, totally accessible to everyone )

Could you recommend a switch and some NICs, along with a thunderbolt adapter?

Switches: when I last looked, there were cheap mikrotik switches and tp-link, if you want to buy new. Otherwise search on eBay/craigslist/etc.

Network cards: intel, mellanox, solarflare. Intel has a lot of drivers for all OSes, mellanox is most stable electrically (you can provide it with separate power, no need for fancy adapters, just plug any 12v in), solarflare is cheapest on ebay if you buy 2-3-generations old cards. Actually, any 10G network card will do, just check if they have drivers for your OS.

Thunderbolt: there are ready-made eGPU cases, they are costly but easier to handle. If you want to be cheap: wait for sale of any thunderbolt3-to-nvme adapter. PCBs for all of them are made by the same factory (you can find it on alibaba/taobao) they differ only by box and branding. Based on intel JHL6340

edit: I have got this, it was on-sale at the time https://www.aliexpress.com/item/4000200151507.html

edit2: just checked eBay, the cheapest solarflare "S7120" (it is SFN7022F actually) is $15 (!) + free shipping, wow. Though you'll need some sfps, they start from $6 (FTLX8571)