Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Hardware Ethernet Switch project, part 1

serd.es

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Open Hardware Ethernet Switch project, part 1

#21

Post by jmcguckin »

azonenberg wrote: post #5
Aurornis wrote: post #3

This is a seriously impressive project and it's been fun to follow along. I almost hesitate to say anything, but the constant scope creep (that he readily admits in the blog) is starting to feel like a real obstacle to getting anything done. It's fun to see all of the theorizing about 96-port switches with $11K of FPGAs, but it would be fantastic if something like the simple 4-port switch design could be spun out as…

The plan is 96 ports total for my own use, split across two 48-port 1U switches. And that's something that is now well within reach: I have the power boards assembled and ready to go, I have the FPGAs and PHYs and (not populated) line card boards in hand. (EDIT: clarification, the plan had been 96 ports total since at least 2015, since that's what I had in existing Cisco switching. The major scope creep over that tim…

If i wanted to do a motherboard that held 16 or 24 raspberry-pi compute modules, could i connect them together using the onboard gig ethernet without using any magnetics? Just by using the gige output from the module going to a commercial switch chip.

Re: Open Hardware Ethernet Switch project, part 1

#22

Post by fxtentacle »

jmcguckin wrote: post #21
azonenberg wrote: post #5

Earlier quoted context omitted.

The plan is 96 ports total for my own use, split across two 48-port 1U switches. And that's something that is now well within reach: I have the power boards assembled and ready to go, I have the FPGAs and PHYs and (not populated) line card boards in hand. (EDIT: clarification, the plan had been 96 ports total since at least 2015, since that's what I had in existing Cisco switching. The major scope creep over that tim…

If i wanted to do a motherboard that held 16 or 24 raspberry-pi compute modules, could i connect them together using the onboard gig ethernet without using any magnetics? Just by using the gige output from the module going to a commercial switch chip.

In general, that should work. Here's an example board with 4x CM4 modules and a RTL8367N switching chip: https://github.com/will127534/Miniature-CM4-Cluster

Re: Open Hardware Ethernet Switch project, part 1

#23

Post by azonenberg »

azonenberg wrote: post #20
minetest2048 wrote: post #19

Earlier quoted context omitted.

> (I ended up writing a bare metal sshd because I couldn't find one that supported no-malloc, no-OS operation) Is there a write up for this? This sounds interesting

The code is at https://github.com/azonenberg/staticnet but I've intentionally avoided over-publicizing it since it hasn't had any kind of third party security review. As of now it's functional enough I'm willing to deploy it on a lab network but wouldn't trust it open to an untrusted network. I work in embedded security and have tried to avoid any of the most gross footguns, deliberately simplifying the implementatio…

For context of how lightweight this is, an -O3 release build of my entire firmware on the management processor right now (including the sshd, hardware drivers, TCP/IP stack, the CLI itself, all of the code to query the supported set of sensors, etc) uses 109 kB of flash and 84 kB of SRAM. The -Og debug build is smaller at 86 kB flash usage.

It compiles in five seconds from a clean build tree on my workstation.

Sure, this isn't as feature-rich as OpenSSH or even Dropbear and is missing a lot of the fancy features you get on Linux, but it's tiny and fast. Good luck getting buildroot or something to give you a 100 kB kernel+userspace image that builds in five seconds.

And it's fast: "time -p ssh testbed show ver" returns in 70 ms on a debug build. That's faster than some x86 Debian + OpenSSH machines I've benchmarked against. And I'm on a 500 MHz single-core Cortex-M7.

Re: Open Hardware Ethernet Switch project, part 1

#24

Post by azonenberg »

fxtentacle wrote: post #22
jmcguckin wrote: post #21

Earlier quoted context omitted.

If i wanted to do a motherboard that held 16 or 24 raspberry-pi compute modules, could i connect them together using the onboard gig ethernet without using any magnetics? Just by using the gige output from the module going to a commercial switch chip.

In general, that should work. Here's an example board with 4x CM4 modules and a RTL8367N switching chip: https://github.com/will127534/Miniature-CM4-Cluster

You're going to at least want to AC couple through some capacitors to avoid problems with common mode / RX bias issues between the different PHYs. But if it's on-mobo you don't need to have as much of a fault voltage rating compared to standard Ethernet which IIRC specs 1 kV isolation on the transformers.

Re: Open Hardware Ethernet Switch project, part 1

#25

Post by 486sx33 »

azonenberg wrote: post #5
Aurornis wrote: post #3

This is a seriously impressive project and it's been fun to follow along. I almost hesitate to say anything, but the constant scope creep (that he readily admits in the blog) is starting to feel like a real obstacle to getting anything done. It's fun to see all of the theorizing about 96-port switches with $11K of FPGAs, but it would be fantastic if something like the simple 4-port switch design could be spun out as…

The plan is 96 ports total for my own use, split across two 48-port 1U switches. And that's something that is now well within reach: I have the power boards assembled and ready to go, I have the FPGAs and PHYs and (not populated) line card boards in hand. (EDIT: clarification, the plan had been 96 ports total since at least 2015, since that's what I had in existing Cisco switching. The major scope creep over that tim…

[dead]

Re: Open Hardware Ethernet Switch project, part 1

#26

Post by Palomides »

zokier wrote: post #7

I wonder if the author considered at any point using Sparx-5 switch asic from Microchip? Those are available in single quantities for not too crazy price ($121 for 128G variant), and they are Linux based. Of course I understand that having custom switch engine is far more satisfying to do.

I really, really would love to see someone make Sparx-5 switch. It's weird that I can't even find any commercially available/closed source switches with them (anyone know of any?).

Re: Open Hardware Ethernet Switch project, part 1

#27

Post by Bluecobra »

azonenberg wrote: post #5
Aurornis wrote: post #3

This is a seriously impressive project and it's been fun to follow along. I almost hesitate to say anything, but the constant scope creep (that he readily admits in the blog) is starting to feel like a real obstacle to getting anything done. It's fun to see all of the theorizing about 96-port switches with $11K of FPGAs, but it would be fantastic if something like the simple 4-port switch design could be spun out as…

The plan is 96 ports total for my own use, split across two 48-port 1U switches. And that's something that is now well within reach: I have the power boards assembled and ready to go, I have the FPGAs and PHYs and (not populated) line card boards in hand. (EDIT: clarification, the plan had been 96 ports total since at least 2015, since that's what I had in existing Cisco switching. The major scope creep over that tim…

It might help to take a look at Arista/Metamako 10G L1 switches for some inspiration. These are 48x ports and use the FPGA for specific network applications (including switching). You might be able to find some of the older models on the cheap on eBay.

I’ve opened up a few and the board itself seems fairly simplistic. I do recall a giant copper heatsink though.

Re: Open Hardware Ethernet Switch project, part 1

#28

Post by yjftsjthsd-h »

acyou wrote: post #12

"I was tipped off by a friend to a batch of Kintex UltraScale+’s, specifically the XCKU5P, on AliExpress for a mere $55 each. He had tested one from the seller and they appeared to be legitimate, although likely salvaged/reballed from some scrapped equipment." Must be nice to know enough to be sure that your sketchy hardware isn't backdoored. I cannot imagine buying random network hardware from a sketchy source. Thou…

> You do not want random bad actor open source contributors adding backdoors.

I think the proprietary network hardware companies have thoroughly demonstrated that it doesn't need to be open source to get back doors.

Re: Open Hardware Ethernet Switch project, part 1

#29

Post by luma »

azonenberg wrote: post #14
luma wrote: post #13

Earlier quoted context omitted.

In this space, it's the closed source firmware blobs that nobody can inspect that I'd be a lot more worried about.

There are no blobs I'm aware of anywhere that are actually running in the system. The STM32s have a small boot "ROM" burned into a write-protected region of flash but I have it jumpered so I boot from main user flash, not the bootloader. I did a quick silicon overview of them (just out of curiosity... they came new from Digikey so I have no reason to believe they're fishy). STM32H735: top metal only https://siliconpr…

Oh for sure, I was posting in response to the OP concerned about open source supply chain attacks. I think your work here is great and makes loads of sense specifically because of security. I trust an open supply chain a lot more than I do a closed one.

Re: Open Hardware Ethernet Switch project, part 1

#30

Post by azonenberg »

Bluecobra wrote: post #27
azonenberg wrote: post #5

Earlier quoted context omitted.

The plan is 96 ports total for my own use, split across two 48-port 1U switches. And that's something that is now well within reach: I have the power boards assembled and ready to go, I have the FPGAs and PHYs and (not populated) line card boards in hand. (EDIT: clarification, the plan had been 96 ports total since at least 2015, since that's what I had in existing Cisco switching. The major scope creep over that tim…

It might help to take a look at Arista/Metamako 10G L1 switches for some inspiration. These are 48x ports and use the FPGA for specific network applications (including switching). You might be able to find some of the older models on the cheap on eBay. I’ve opened up a few and the board itself seems fairly simplistic. I do recall a giant copper heatsink though.

The board-level architecture is going to be super simple as big FPGA designs go:

* XCKU5P in the middle

* 12x GTYs routed to 4x Samtec ARF6 connector for the line cards

* 2x GTYs routed to 2x SFP28 uplinks

* RGMII to back panel management PHY

* Parallel SRAM bus to STM32H735 management processor

* A bunch of Murata MYMGK modules for power conversion off the 12V rail

* STM32L431 in QFN48 or more likely BGA100 depending on IO requirements as a PMIC and to manage reset sequencing etc

This will be fully FPGA based, no separate switch ASIC, and I want to do all of the hardware design. I'm not sure there is much I can learn from somebody else's FPGA switch design at the board level - it's basically just going to be a bunch of transceivers hooked up to SFPs and some power distribution. All the magic happens inside the FPGA.

Post reply on HN