Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing “6-pack”: the first open hardware modular switch

code.facebook.com

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Introducing “6-pack”: the first open hardware modular switch

#91

Post by charlesnw »

So this is very interesting.

As someone who runs a 6509 (almost fully populated with line cards and specialized controllers) as my HOME core switch (with a second one on the way for full redundancy), an open modular switch is very cool.

(I'm currently hacking on OpenFlow -> NETCONF bridging, to bring typical SDN capabilities to legacy Cisco environments). Hence my rather..... extensive home network.

This is very similar to what Cisco is doing with Nexus and the "fabric extender" TOR replacement kit. I'll have to see if Facebook has any of this stuff in GIT and stand it up in a VM environment and play with it (I already do a bunch of OpenFlow stuff on OpenWRT and am looking at implementing an open southbound API on FPGA on the parallela board).

Re: Introducing “6-pack”: the first open hardware modular switch

#92

Post by seany »

wmf wrote: post #64
btoptical wrote: post #63

These big switch boxes typically end up being about thermal management and this box looks like thermal design was an afterthought. Also I'm not sure the power entry design is really all that smart. Generally speaking if you're in the market for a 640Gb/s switch (or 3.8Tb/s switch), does your data center really not have access to 48V power? The AC/DC conversion wastes power and space. The QSFP's are "spaced for optima…

does your data center really not have access to 48V power? No, they really don't. I've never heard of 48V outside of telco.

I have 48v in both my home rack, and the offices rack.

Re: Introducing “6-pack”: the first open hardware modular switch

#93

Post by skuhn »

tiernano wrote: post #80
kijiki wrote: post #68

Earlier quoted context omitted.

Basically anything on here: http://cumulusnetworks.com/support/linux-hardware-compatibil... Some easy ways to buy are: https://bm-switch.com/ http://whiteboxswitch.com/ http://ipfabrics.net/ http://www.penguincomputing.com/products/network-switches/ - nolan co-founder/CTO Cumulus

Are there any that are, say, 48 port GigE? 10G and 40G are a bit much for home use...

There are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and $700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain support in order to continue using the product (legally). I can't imagine getting enough utility from a switch for this to make sense for home use.

It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly low end 1 gig switch (single power supply, limited L3 capability), and you can absolutely negotiate Juniper / Cisco / etc. down to be close to or even below the pricing of the Cumulus solution. That is tougher for 10 gig or 40 gig equipment, which is where the value proposition kicks in for Cumulus.

Maybe Cumulus's approach is enough of a value add for it to make sense to pay a premium, but everyone I talk to is interested in cost savings first and better manageability a distant second. You'll still have to have Juniper / Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they don't have a full range of switch models.

Re: Introducing “6-pack”: the first open hardware modular switch

#94

Post by kijiki »

mino wrote: post #86
kijiki wrote: post #68

Earlier quoted context omitted.

Basically anything on here: http://cumulusnetworks.com/support/linux-hardware-compatibil... Some easy ways to buy are: https://bm-switch.com/ http://whiteboxswitch.com/ http://ipfabrics.net/ http://www.penguincomputing.com/products/network-switches/ - nolan co-founder/CTO Cumulus

Hey Nolan, I love Cumulus' approach and I bet you guys will go very far! I thought about adopting Cumulus for a large telco project, but for the moment it doesn't seem a good fit as we do lots of openflow, L3 and custom application development. Actually, have a look at it. It's also blue (maybe we should sue facebook :D): http://ripe69.ripe.net/archives/video/177/

I'll take a look, thanks for the link.

We don't do OpenFlow, but L3 and custom apps are our main focus.

That said, often times the subset of things that can be done w/ OpenFlow in actual existing hardware are things we can do natively. Feel free to email me if you want to discuss!

nolan@cumulusnetworks.com

Re: Introducing “6-pack”: the first open hardware modular switch

#95

Post by kijiki »

skuhn wrote: post #93
tiernano wrote: post #80

Earlier quoted context omitted.

Are there any that are, say, 48 port GigE? 10G and 40G are a bit much for home use...

There are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and $700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain support in order to continue using the product (legally). I can't imagine getting enough utility from a switch for this to make sense for home use. It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly…

Stay tuned for low-end 1G platforms running Cumulus Linux.

- nolan

Re: Introducing “6-pack”: the first open hardware modular switch

#96

Post by kijiki »

MichaelGG wrote: post #76
kijiki wrote: post #67

Earlier quoted context omitted.

To clarify, the only part that is closed is "switchd", which is a userspace program that watches the kernel data structures (route tables, neighbor tables, bridges, ports, vlans, etc) and programs the hardware to match. It links against proprietary silicon vendor SDKs, and programs registers whose description were given to us under NDA. Without this part, everything works the same, but is of course not hardware accel…

What is the flexibility with "open" switches? To get linerate switching, I'm guessing you're still limited by the hardware? Is the benefit that you can more easily setup routing tables (instead of depending on the switch vendor's capabilities), vlans, etc. just by creating them in userspace then pushing them over to the hardware part? Or can you actually get fairly low level, like implementing your own algorithms for…

You are limited by the hardware, and what our code supports programming into it.

The big advantages are reusing config management tools like puppet/chef/ansible/etc, and monitoring tools like collectd/graphite/nagios/etc.

Also, it is super easy to run services on the switches. For example, you can easily run isc-dhcpd on each ToR, instead of DHCP relaying back to one mega DHCP server. Distributing services like this scales better, and reduces the blast radius of service failure.

I've been experimenting with the idea of a transparent caching TFTP proxy server running on the top of rack switch, to make PXE scale better to large clusters.

The important thing is that anyone who has the know-how to write a transparent caching TFTP proxy server for Linux can just go ahead and do that on a Cumulus Linux switch! You don't need to come to us and convince us that it is a good idea and then wait for us to actually implement it. Compare that to asking for features from a traditional switch vendor...

- nolan co-founder/CTO Cumulus

Re: Introducing “6-pack”: the first open hardware modular switch

#97

Post by kijiki »

DiabloD3 wrote: post #70
e12e wrote: post #56

Earlier quoted context omitted.

Ah, of course. I was misled by wikipedia[1], but it should've been obvious that the same connection could be used for both inifinband and ethernet. So, indeed, there are copper interconnects: http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Cop... Actually seems the price is finally coming down a bit (compared to what I remember these used to cost, years ago -- but maybe I've just upped my budget ;-). [1] htt…

The cable you're looking for is called a direct connect cable. "Fake" optical modules on both sides, plugs it together with a permanently wired copper cable.

DAC is a common term for this. Direct Attached Copper.

It is a SFP+ plug on each end (but without all the optical magic), connected with twin-ax cable, which is like coax but with 2 signal paths, one for each direction.

Post reply on HN