Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Add 10 GbE to your system with an M.2 2280 module

cnx-software.com

Post reply on HN
Print view View original

221–230 of 259 posts

Re: Add 10 GbE to your system with an M.2 2280 module

#221

Post by omgtehlion »

csdvrx wrote: post #219
omgtehlion wrote: post #210

Earlier quoted context omitted.

Switches: when I last looked, there were cheap mikrotik switches and tp-link, if you want to buy new. Otherwise search on eBay/craigslist/etc. Network cards: intel, mellanox, solarflare. Intel has a lot of drivers for all OSes, mellanox is most stable electrically (you can provide it with separate power, no need for fancy adapters, just plug any 12v in), solarflare is cheapest on ebay if you buy 2-3-generations old c…

How much did you pay when your tb3-to-nvme was on sale? It's not an urgent purchase :)

$55 + shipping (less than $5)

Re: Add 10 GbE to your system with an M.2 2280 module

#222

Post by chipsa »

spacedcowboy wrote: post #185
jiggawatts wrote: post #80

Some people are saying that they don't feel the need for > 1 Gbps in a home network. As a counter-point, I'm regularly limited by 1 Gbps PHY limits. During the 2 years of the pandemic I've been working from home with gigabit Internet. Why gigabit? Because that's the "speed limit" of the Ethernet PHY in the fibre broadband box, and also my laptop's ethernet port. My laptop can write to disk at multiple GB per second,…

I use a map-maker app for creating game levels, the data is stored on a central server when it's saved. A 4kx4k map layer section, in RGBA is 64MB. Each layer on the map has multiple types of mask applied (brushed alpha, brushed noise-function,...) so there's another 64MB of XXXX data per layer. A map can have (say) 10 layers on it, so we're up to 1GB, and then there's the data for ephemeral layers ("stamped" images…

> Each layer on the map has multiple types of mask applied (brushed alpha, brushed noise-function,...) so there's another 64MB of XXXX data per layer.

Wait, why are the masks RGBA instead of grayscale?

Re: Add 10 GbE to your system with an M.2 2280 module

#223

Post by derekdahmer »

Zandikar wrote: post #218
jiggawatts wrote: post #80

Some people are saying that they don't feel the need for > 1 Gbps in a home network. As a counter-point, I'm regularly limited by 1 Gbps PHY limits. During the 2 years of the pandemic I've been working from home with gigabit Internet. Why gigabit? Because that's the "speed limit" of the Ethernet PHY in the fibre broadband box, and also my laptop's ethernet port. My laptop can write to disk at multiple GB per second,…

I've been ranting about this for a while tbh. Network interfaces used to be one of the fastest interfaces on a computer. Now they're one of if not the slowest save for USB2 or bluetooth. 1GbE networking is one of the slowest interfaces on modern computers. - The slowest USB3 spec is 5Gbps. USB3 is a mess, but you can easily far exceed 1Gbps with the absolute worst of it. - The current fastest USB spec is USB4, and is…

Ethernet is by far the longest connection out of all of those, supporting up to 100 meters for 10G over copper and over 300 meters for fiber [1] which is pretty impressive. No other commodity interface comes close to that kind of range at that speed.

Wifi 6 officially has a real world limit of 700mbps [2] and drops to half that if you're more than a few feet from the router. Passive USB 3 and thunderbolt cables have a max length of 2 meters [3]. The rest of the interfaces you've listed like PCIe all operate on the scale of inches and plug straight into the motherboard.

Plus it really isn't very expensive to upgrade to 10G given the use case is pretty niche for home use. You can get a budget SFP+ Switch and 2 NICs for ~$300.

[1] https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-SR

[2] https://www.theregister.com/2019/12/05/wifi6_700mbps_speeds/

[3] https://www.newnex.com/usb-cable-maximum-length-limits.php

Re: Add 10 GbE to your system with an M.2 2280 module

#224

Post by wmf »

ksec wrote: post #216
zinekeller wrote: post #104

Earlier quoted context omitted.

Thank Intel for single-handedly changing the 2.5Gbps landscape by integrating 2.5Gb Ethernet on their chipsets, allowing no-brainer OEM integration. Unfortunately, unless ISP router OEMs (Nokia, Zyxel, Huawei et al.) do their part (the reason being that very few people actually bothers to buy a separate router), we will not see the economies of scale necessary to fully finish 2.5Gb Ethernet.

But is kind of sad they skip 5Gbps. I thought it was the best compromise for consumer between 1 and 10 Gbps Ethernet.

5 Gbps will probably come to the consumer market around 2030.

Re: Add 10 GbE to your system with an M.2 2280 module

#225

Post by martijnvds »

t0mas88 wrote: post #192
martijnvds wrote: post #7

Earlier quoted context omitted.

Lots of consumer motherboards don't have a lot of PCIe slots wider than x1 (which is not enough for 10GigE), but they often have multiple NVMe capable M.2 slots, that this card would work with. In my home server for instance, which is built on a consumer μATX board, the wider PCIe slots are filled with a GPU and HBA, which leaves no way to add 10GigE without spending a lot of money on a new motherboard and CPU - or f…

For a home server you probably don't need PCIe 16x for the GPU? If you even need a GPU at all?

The GPU is useful for quick on the fly transcoding of video. They only make x16 cards (or slow underpowered x1 ones).

Putting the HBA in the second x16 ("electrically x8") slot makes both work at x8 speeds, as those lanes are "shared".

Re: Add 10 GbE to your system with an M.2 2280 module

#226

Post by mrb »

csdvrx wrote: post #206
mrb wrote: post #202

Earlier quoted context omitted.

I have a Linux NAS with 6 7200RPM HDDs in a raidz2 ZFS pool, and the local fs read speed on large files is 7 times faster than GbE, about ~930 MB/s. So while this wouldn't saturate a 10GbE link, it would still be greatly beneficial to multiply the network read speed by 7-fold by upgrading from GbE to 10GbE.

How did you bench your ZFS pool?

With dd. And normally when I do this I drop the caches with "echo 3 > /proc/sys/vm/drop_caches" but this doesn't work with ZFSOnLinux as it has a separate ARC cache that doesn't get cleared by that command (and I don't believe it's possible to clear it, other than by exporting and reimporting the zpool.)

So I read a 10GB chunk of a file that I know is not cached:

  $ dd if=/some/large/file of=/dev/null bs=1M count=10240
  10240+0 records in
  10240+0 records out
  10737418240 bytes (11 GB, 10 GiB) copied, 11.3378 s, 947 MB/s
The 947 MB/s figure printed by dd matches the speed reported by a "zpool iostat tank 1" running during the benchmarking, which confirms I'm not reading from the cache. When I repeat the same command, dd completes almost immediately reporting a speed of about 6-8 GB/s on my machine, while the zpool iostat command shows zero read bandwidth.

Re: Add 10 GbE to your system with an M.2 2280 module

#227

Post by spacedcowboy »

chipsa wrote: post #222
spacedcowboy wrote: post #185

Earlier quoted context omitted.

I use a map-maker app for creating game levels, the data is stored on a central server when it's saved. A 4kx4k map layer section, in RGBA is 64MB. Each layer on the map has multiple types of mask applied (brushed alpha, brushed noise-function,...) so there's another 64MB of XXXX data per layer. A map can have (say) 10 layers on it, so we're up to 1GB, and then there's the data for ephemeral layers ("stamped" images…

> Each layer on the map has multiple types of mask applied (brushed alpha, brushed noise-function,...) so there's another 64MB of XXXX data per layer. Wait, why are the masks RGBA instead of grayscale?

8 bits per mask, more than one mask, packed into a 32-bit value.

1st one is typically a brush-in one - basically just Alpha brushed in from various alpha-brushes over the map

2nd can be a noise function, where the 1st alpha mask is multiplied by the second to get a brushed noise, and the noise might be useful for randomly-shaped patches of different-colored grass in a scene

3rd might be a shadow mask, where any objects "stamped" on top apply a default shadow at a given angle, and that default shadow is written into this channel. Then the map-maker can alter those if (for example) a shadow hits a wall.

Then there's the shader mask - so you can select a given shader for a given area of the map. These'll all be batched up later, but if you want flowing water, brush in where you want the water shader to be applied. Same for smoke, lava, etc. etc.

Re: Add 10 GbE to your system with an M.2 2280 module

#228

Post by justsomehnguy »

fulafel wrote: post #28
glogla wrote: post #14

Earlier quoted context omitted.

NAS with SSD can go way faster than the ~ 120 MB/s 1GbE can offer. For NAS with HDD, the benefit is not as big but still there. Also, the main competition is WiFi which is significantly slower than 1GbE in most cases.

Spinning HDDs are faster than this as well, 250-300 MB/s sequential read seems common per drive, and in a home setup you might have a couple of those in a RAID-1 giving 500-600 MB/s read BW.

> couple of those in a RAID-1 giving 500-600 MB/s

If you are talking about only two drives than no. Multiple drives can give you more throughput, but at that point it is RAID-10.

Re: Add 10 GbE to your system with an M.2 2280 module

#229

Post by justsomehnguy »

aseipp wrote: post #163
seized wrote: post #160

Earlier quoted context omitted.

Those same people should also know that spinning rust can go very fast as well... My 8 and 12 drive spinning rust pools can read at 500MB/sec easily and get close to GB/sec. But the 10Gbit network is there for the flash pool.

Yeah, but... The noise! I just can't handle spinning rust anymore for this alone even if the $/GB ratio is still so much better. Consumer SATA SSDs are plentiful enough that I'm just going all in on flash when I revamp my server and do an 8 drive (+1 hotswap) RAID10 build or whatever. I don't need +60TB or anything and 8-9 drives is about the limit you can find on high end consumer mobos I think...

> The noise

It is not required for the NAS to be in the same room where you sleep.

With a proper drives (and maybe some APM tinkering) you can make a very quiet one. Also using 2.5" drives is an option too.

Though if you can throw money at the problem than of course you can just buy a bunch of TB SSD drives.

Re: Add 10 GbE to your system with an M.2 2280 module

#230

Post by justsomehnguy »

spacedcowboy wrote: post #185
jiggawatts wrote: post #80

Some people are saying that they don't feel the need for > 1 Gbps in a home network. As a counter-point, I'm regularly limited by 1 Gbps PHY limits. During the 2 years of the pandemic I've been working from home with gigabit Internet. Why gigabit? Because that's the "speed limit" of the Ethernet PHY in the fibre broadband box, and also my laptop's ethernet port. My laptop can write to disk at multiple GB per second,…

I use a map-maker app for creating game levels, the data is stored on a central server when it's saved. A 4kx4k map layer section, in RGBA is 64MB. Each layer on the map has multiple types of mask applied (brushed alpha, brushed noise-function,...) so there's another 64MB of XXXX data per layer. A map can have (say) 10 layers on it, so we're up to 1GB, and then there's the data for ephemeral layers ("stamped" images…

OpenVPN has support for compression. Makes me wonder if it would work in your case.
Post reply on HN