Live data from Hacker News

10Gb/s Ethernet: what I did to get it working in my home

gilesthomas.com

61–70 of 180 posts

Re: 10Gb/s Ethernet: what I did to get it working in my home

#61

I'm extremely happy after upgrading my network to 10gbit copper ethernet. It was much more expensive than I thought it should be, but worth it even if I only max it occasionally. Now I can easily fully saturate my 10gbit ethernet doing a first Time Machine backup or transferring files to my M.2 SSD NAS which saves me waiting rime and is satisfying to watch. It's wild to me that 10gbit isn't the norm by now and tech p…

Heh it's honestly wild to me anyone needs over a gig. My work has a one gig fiber line supporting hundreds of employees and usage generally remains below 10%. The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.

1Gb Internet service seems low these days, much less 1Gb LAN. I have 3Gb Google Fiber service and actually get 2+ for individual downloads from some internet services like Steam. Even at 2Gb it's annoying to wait tens of minutes for 100+ GiB games to download. If I go on vacation I come home with 10s of GiBs of photos and videos on multiple devices that start syncing with cloud storage.

During the day I need to pull large data files from the work VPN so it's nice that that can happen at full speed even when Steam and movie streaming are also at full throttle. Combine that with backups and moving various files back and forth to my NAS and I'm very happy to have 10Gb local wiring.

Re: 10Gb/s Ethernet: what I did to get it working in my home

#62

That's pretty wild. I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.

And here I am sitting in Brooklyn and haven't had one apartment that has had fiber as an option. I get to pay Spectrum $90/month for "400/20" and in reality get 100/10.

Re: 10Gb/s Ethernet: what I did to get it working in my home

#63
post #26

Unfortunately the blog didn't link to the SFP+ module they're using, but everyone should know there's effectively 2 different generations of 10gbit sfp+ to ethernet^H10BASE-T modules. The old gen, labeled as 30 meters, draws ~3 W, and gets extremely hot (to the point it'll usually cause link flaps), and the newer gen, usually labeled as 100m or 80m, draws ~1.5 W, and runs much, much cooler. Example of the new gen: ht…

BCM84891L. I like these modules (select 80 or 100 m in the drop down): https://www.luleey.com/product/10gbase-t-sfp-to-rj45-copper-...

Using this module, I was able to get a stable 10 gig over a 75 feet long, 20 year old run of Cat 5e.

Re: 10Gb/s Ethernet: what I did to get it working in my home

#64
post #39
post #22

Earlier quoted context omitted.

Cripes. When possible just do fiber and DACs. Faster and much cooler than 10Gbit. 10Gbit uses an absurd amount of power per port thus the need for all that cooling.

yeah, the article though says... "The apartment has structured cabling -- each room has one or more RJ45 sockets in the wall," ... Which is the main problem most folks face. wish the standard was "conduit" instead of "bake-this-years-tech-into-the-wall" which doesn't always last...

Folks have been saying that conduit is the way, and the fiber is the future, and all kinds of things like that for decades so far.

But the simple truth for all those decades is this: When there's already cat-whatever cable in the wall, it generally still works.

Decently-installed conduit (ie, actually-usable conduit) adds a ton of time and expense, which is why it is very seldom used for data circuits in residential structures.

The cable that exists is a lot better than the conduit that doesn't. And copper ethernet is bog-standard like MP3 is: It isn't the best in any technical sense at all, but everything supports it. Universal compatibility is pretty nice.

---

So the ongoing cost of copper 10gbe is electricity. Someone else here in the comments says that a copper 10GBe SFP+ module can use ~3 Watts, or that a newer one can use about 1.5 Watts.

We can be generous by using the larger figure of 3 Watts, and 8 devices..

With 4 ports, eight 10-gig endpoints @ 3 Watts each, and $0.19 per kWh [delivered]: That's $3.28 per month, or about $400 per decade.

If we assume 1.5 Watt endpoints, then that number halves.

If we subtract the power consumption of fiber SFP+ modules (or media converters or whatever) to make the number a relative comparison instead of an absolute, then that figure goes down further.

Not so bad, compared to conduit.

Re: 10Gb/s Ethernet: what I did to get it working in my home

#65
post #35
post #26

Unfortunately the blog didn't link to the SFP+ module they're using, but everyone should know there's effectively 2 different generations of 10gbit sfp+ to ethernet^H10BASE-T modules. The old gen, labeled as 30 meters, draws ~3 W, and gets extremely hot (to the point it'll usually cause link flaps), and the newer gen, usually labeled as 100m or 80m, draws ~1.5 W, and runs much, much cooler. Example of the new gen: ht…

Thanks! 10Gb Eth is insane for exactly this reason (optical SFP+ modules are way cheaper and more reliable)

Yeah but home users already often have 75-100 foot runs of cat 5e in the walls, and those work fine for 10G-baseT.

Re: 10Gb/s Ethernet: what I did to get it working in my home

#66
Regarding the proxmox cluster, I would advise not to set up a cluster unless you really need to, it adds complexity in order of shutting down the server, HA, Quorum votes, ceph monitor, among many others, and if one server goes offline, or even the wrong order, it will impact the others and sometimes some tough time in recovery, or data loss.

Re: 10Gb/s Ethernet: what I did to get it working in my home

#67

Earlier quoted context omitted.

Heh it's honestly wild to me anyone needs over a gig. My work has a one gig fiber line supporting hundreds of employees and usage generally remains below 10%. The high expense of 10gig is, in part, because it isn't widely necessary and the people buying it are willing to pay extra.

1Gb Internet service seems low these days, much less 1Gb LAN. I have 3Gb Google Fiber service and actually get 2+ for individual downloads from some internet services like Steam. Even at 2Gb it's annoying to wait tens of minutes for 100+ GiB games to download. If I go on vacation I come home with 10s of GiBs of photos and videos on multiple devices that start syncing with cloud storage. During the day I need to pull…

This is one of those things where I just have to express that a lot of the HN crowd is entirely divorced from the reality the rest of the world experiences. ;)

Nearly nobody has multigig anything in the home, a probably surprisingly large percentage of business networking is 1gig LAN or less. And most people would not notice the difference if they did.

I am glad it works for you, but everyone else most certainly doesn't need it. (Yet.)

Personally, I do try for mostly gigabit in my home, because I do selfhost, but I have a ~800 Mbps download service (200 Mbps upload, it's asymmetric) that was only 500 Mbps when I signed up. And to be honest most of my patch cables are CAT5e because I'm cheap. I do make sure to run CAT6 through walls though because I don't want to ever have to do it again.

Also, I used to have Astound, and I feel so much sympathy for Google Fiber customers, you have no idea what's coming. If you thought Google had a reputation for bad customer service... just wait!

Re: 10Gb/s Ethernet: what I did to get it working in my home

#68

Earlier quoted context omitted.

It's wild to me Time Machine works on your network. Are you just doing "first backups" over and over again, or have you somehow achieved the very rare state where Time Machine can run for, say, a week at a time without falling over? Sorry, this is snarky and off topic, but I'm nostalgic for the days when Time Machine "just worked".

I can't remember the exact phrasing, but are you talking about the error message that essentially says: "The Tardis is broken. Your backup has diverged into an entirely separate timeline, and I have no way of reconciling it. You may now sacrifice an entire weekend to do an initial backup again."? I've been on a lucky streak for several years now, where I haven't gotten that one on any of my devices. "Preparing backup…

> "Preparing backup..." taking an unreasonable amount of time is a regular occurrence,

TM heavily throttles disk I/O used for backing up in order to ensure that normal user activity isn't affected. That makes it appear that TM is dramatically slower than you would expect which greatly annoys me. This becomes obvious after you run this command which will make both the preparing and transferring phases go closer to the theoretical speed you'd expect:

sudo sysctl debug.lowpri_throttle_enabled=0

Re: 10Gb/s Ethernet: what I did to get it working in my home

#69
> The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to).

This is not quite correct.

The primary problem is cross-talk. Copper wire itself will carry the relevant frequencies up to 100m without issue but even with balanced pairs the balancing is not perfect and the "dirty paper precoding" is not perfect so some cross-talk will occur. How long you can go with Cat-5e depends on how well the wire is twisted, how many wires are bundled together, are there any loops or tight bends, and other factors. Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.

Bottom line is: for almost any normal home or apartment any quality Cat-5e cable properly terminated will carry 10GBase-T without issue. In fact if you have problems I would first re-terminate the cable before assuming you need to run new cable. Cat-6 or 6A just isn't necessary.

As a PSA: beware of "CCA". I've noticed Amazon and eBay are absolutely flooded with cheap chinese electrical and networking cable that shows nice shiny copper in the pictures but is actually "copper clad aluminum". If they mention anything at all they code it as "CCA" cable without explaining what that means.

CCA cable cannot, by definition, be ethernet cable. I won't get into the full technical details but the standard was amended to clarify that only pure copper wires are acceptable for ethernet. Personally I would not dare use CCA for anything. It has lower performance, lower current-carrying capability for the same wire diameter (inherent in aluminum), and introduces the risk of oxidation and loosening of connections as people will treat them as copper connections when aluminum needs special installation procedures and connections to avoid them coming loose over time. For electrical connections especially this not only can but absolutely will lead to a fire over time if not treated with the appropriate care. All it takes is a little bit of mechanical action scraping off the thin copper layer and you now have an effectively aluminum wire - a time bomb ticking away.

Re: 10Gb/s Ethernet: what I did to get it working in my home

#70
post #8

That's pretty wild. I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.

10Gbps is enough bandwidth for 500 concurrent Netflix streams in 4K/UHD (15Mbps) AND 500 concurrent video calls (4Mbps). Home users don’t need more bandwidth to improve their internet experiences, they need lower latency, less congestion and less loss. https://help.netflix.com/en/node/306 https://learn.microsoft.com/en-us/microsoftteams/prepare-net...

> Netflix streams in 4K/UHD (15Mbps)

proper 4k, like the one you watch from a blu-ray, will have peaks of 150mbps.

the 4k we see on streaming services is awfully overcompressed.

and yeah, you can see the difference, it's day and night.

Post reply on HN