Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

rtfm.net

Post reply on HN
Print view View original

51–58 of 58 posts

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#51

Post by stephen_g »

rdl wrote: post #23

In general for small hardware I go with two options. Qualcomm-Atheros AR9331 based systems, with 64MB ram and 16MB flash (kind of hard to get in that config cheaply). You can get decent performance, potentially hang a 5.8/2.4ghz radio on it too, and power it off a single laptop USB port (200mA normal power use, and with stuff, still less than 500mA.) Can put second ethernet, wifi, 3G, etc. there. There's no current c…

I've been looking at Intel's DPDK recently [1], and it looks really interesting for really fast packet processing in the 'beefier' kind of systems you describe.

It would be a bit higher cost, but could mean you could make a router like the EdgeRouter but with good performance without the proprietary accelerator (I believe that Intel's drivers are open source).

1. Data Plane Development Kit, which lets you skip the kernel IP stack (which takes thousands of CPU cycles to process) and do packet processing in userland taking just tens to hundreds of cycles per packet. http://dpdk.org/

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#52

Post by weland »

rdl wrote: post #31
nodata wrote: post #30

Earlier quoted context omitted.

Reworked how?

Someone who is less bad at surface mount soldering than I am can take a 702/703, pop the 2MB flash and 8-16M RAM chips off, and replace with 16/64. There are people in China who do this commercially, and some eBay stores. I've bricked (well, melted) 2 of them so far doing it myself.

It sounds pretty hard to do using nothing but a soldering iron. The nice thing about the C2 is that you have all the I/O pins nicely exposed and there's a fairly helpful community built around them. It's not necessarily impressive in terms of documentation, but it's certainly better than something that's pretty much reverse-engineered.

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#53

Post by rdl »

stephen_g wrote: post #51
rdl wrote: post #23

In general for small hardware I go with two options. Qualcomm-Atheros AR9331 based systems, with 64MB ram and 16MB flash (kind of hard to get in that config cheaply). You can get decent performance, potentially hang a 5.8/2.4ghz radio on it too, and power it off a single laptop USB port (200mA normal power use, and with stuff, still less than 500mA.) Can put second ethernet, wifi, 3G, etc. there. There's no current c…

I've been looking at Intel's DPDK recently [1], and it looks really interesting for really fast packet processing in the 'beefier' kind of systems you describe. It would be a bit higher cost, but could mean you could make a router like the EdgeRouter but with good performance without the proprietary accelerator (I believe that Intel's drivers are open source). 1. Data Plane Development Kit, which lets you skip the ke…

I wonder how low the price of an Atom C2358 based system with 4x Intel Gig-E and either mini-PCI or some other solution for wireless would be. That would probably be what I'd build a "high end" home router, or low-end business router, on -- you'd still be able to hit PoE 802.af 2009 (25W).

DPDK supports Atom. This Atom has both virtualization and AES-NI, so you could do some interesting security things (the fireeye-style "run malware in a VM", and linerate crypto).

~$120 in parts means a retail price around $500. That doesn't seem at all unreasonable to me, but there's always the "sell it for $250 and make $5-10/mo MRC" option. I think Cisco/Linksys tried that but it was pretty weak.

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#54

Post by pktgen »

stephen_g wrote: post #51
rdl wrote: post #23

In general for small hardware I go with two options. Qualcomm-Atheros AR9331 based systems, with 64MB ram and 16MB flash (kind of hard to get in that config cheaply). You can get decent performance, potentially hang a 5.8/2.4ghz radio on it too, and power it off a single laptop USB port (200mA normal power use, and with stuff, still less than 500mA.) Can put second ethernet, wifi, 3G, etc. there. There's no current c…

I've been looking at Intel's DPDK recently [1], and it looks really interesting for really fast packet processing in the 'beefier' kind of systems you describe. It would be a bit higher cost, but could mean you could make a router like the EdgeRouter but with good performance without the proprietary accelerator (I believe that Intel's drivers are open source). 1. Data Plane Development Kit, which lets you skip the ke…

> 1. Data Plane Development Kit, which lets you skip the kernel IP stack (which takes thousands of CPU cycles to process) and do packet processing in userland taking just tens to hundreds of cycles per packet. http://dpdk.org/

I wish OS developers saw this as a problem. There is no reason kernel stacks should be so slow for tasks where all processing is done in the kernel. (For packets destined to userspace, you've got the syscall overhead to deal with.)

I recently tested the Linux network stack's PPS performance with an Intel X520 10GbE NIC. I used Debian testing, with the 3.12 kernel. My destination machine was an i7-3930K at stock speed. I wrote a simple kernel module adding an NF_IP_PRE_ROUTING hook returning NF_DROP with no processing, which would be the simplest possible code path. For a packet generator, I used another older machine with another X520, using the "pfsend" tool included with PF_RING, and the card in PF_RING DNA mode. That was easily able to saturate the link at line rate (14.8M PPS).

The result: the kernel was only able to sustain about 2.8M PPS.

I then loaded the DNA driver on the destination machine, used the included "pfcount" tool, and no packet drops - it was receiving the full 14M PPS.

I tested DPDK recently and had similar results.

I also modified the Linux ixgbe driver ixgbe_clean_rx_irq() function, and added a step in between the "fetch packets from RX ring" and "put packet in SKB and send to network stack" functions. Even when I added a bunch of useless comparisons for each packet, I was able to get ~12-13M PPS. I could get line rate by just dropping and not doing any processing.

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#55

Post by rdl »

pktgen wrote: post #54
stephen_g wrote: post #51

Earlier quoted context omitted.

I've been looking at Intel's DPDK recently [1], and it looks really interesting for really fast packet processing in the 'beefier' kind of systems you describe. It would be a bit higher cost, but could mean you could make a router like the EdgeRouter but with good performance without the proprietary accelerator (I believe that Intel's drivers are open source). 1. Data Plane Development Kit, which lets you skip the ke…

> 1. Data Plane Development Kit, which lets you skip the kernel IP stack (which takes thousands of CPU cycles to process) and do packet processing in userland taking just tens to hundreds of cycles per packet. http://dpdk.org/ I wish OS developers saw this as a problem. There is no reason kernel stacks should be so slow for tasks where all processing is done in the kernel. (For packets destined to userspace, you've g…

I wonder if CloudFlare does any of this.

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#56

Post by pktgen »

rdl wrote: post #55
pktgen wrote: post #54

Earlier quoted context omitted.

> 1. Data Plane Development Kit, which lets you skip the kernel IP stack (which takes thousands of CPU cycles to process) and do packet processing in userland taking just tens to hundreds of cycles per packet. http://dpdk.org/ I wish OS developers saw this as a problem. There is no reason kernel stacks should be so slow for tasks where all processing is done in the kernel. (For packets destined to userspace, you've g…

I wonder if CloudFlare does any of this.

They do. They use Solarflare's OpenOnload: http://blog.cloudflare.com/a-tour-inside-cloudflares-latest-...

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#57

Post by rdl »

pktgen wrote: post #56
rdl wrote: post #55

Earlier quoted context omitted.

I wonder if CloudFlare does any of this.

They do. They use Solarflare's OpenOnload: http://blog.cloudflare.com/a-tour-inside-cloudflares-latest-...

Those look nice, but presumably Intel NICs would be an order of magnitude cheaper, especially if built into the chipset.

Re: FreeBSD 10.0 on Ubiquiti EdgeRouter Lite

#58

Post by pktgen »

rdl wrote: post #57
pktgen wrote: post #56

Earlier quoted context omitted.

They do. They use Solarflare's OpenOnload: http://blog.cloudflare.com/a-tour-inside-cloudflares-latest-...

Those look nice, but presumably Intel NICs would be an order of magnitude cheaper, especially if built into the chipset.

Definitely. I am not sure what metrics Solarflare beat others (presumably including Intel) on, besides software support (OpenOnload implements the BSD sockets API and doesn't require software changes, while DPDK and others implement their own). In my testing, the X520s have been capable of both sending and receiving at line-rate.

It's possible the Solarflare NICs could be better at some metric like latency, by a matter of microseconds (only speculating here), but I can't see how they'd consider that worth the extra money for HTTP servers.

Post reply on HN