Seeking input on planned router build

Offloading HDDs and other functions to remote NAS or servers is increasingly popular
Post Reply
Jay_S
*Lifetime Patron*
Posts: 715
Joined: Fri Feb 10, 2006 2:50 pm
Location: Milwaukee, WI

Seeking input on planned router build

Post by Jay_S » Thu Feb 05, 2015 9:45 am

Hi all,

I've been casually "shopping" for router hardware for about a year. I'm in no big rush and was hoping something used would fall into my lap. I want to play around with router distros (pfsense and sophos specifically). HCLs usually recommend Intel NICs, and I won't argue with that. Luckily used dual-port intel NICs are inexpensive.

I can get older HP dc7900 SFF desktops for under $100. They're usually E7x00 or E8x00 cure 2 dual machines with 1 - 4GB of RAM. I'm very familiar with them (we use them at work). My local craisglist has three of them for sale right now, Newegg always has refurbs, and they're well represented on ebay. They include Intel's 82567V gigabit controller. The SFF desktops include half-height PCIe x16, x4, and x1 slots. Used dual-port PCIe x4 NICs are around $25 on ebay.

I've offered $75 for one of the CL models with 4GB of RAM.

Adding an additional NIC, let's say I'm at $100 all-in. That buys me a used dc7900 with 2.66 GHz dual core CPU, 4GB of RAM, and 2-3 Intel NICs. That doesn't sound too bad! But they idle around 35-40 watts & this is a 5 year old machine (with 5+ year old capacitors).

Planned deployment includes firewalling, ClamAV and Snort IPS. The deep packet inspection features wreck throughput. Thanks to America's "broadband", I don't need much! (10 Mbps today.)

I'm trying to balance costs, power and efficiency, in that order. Cost is paramount because I'm just in the "playing around" stage now.

Compared to two other options:
1) new Intel Atoms (Avoton & Rangeley) are more power efficient. But they cost a ton.
2) older Intel Atoms are cheap and efficient, but underpowered when DPI features are enabled.

Questions: Compared to the dc7900 above,
1) can I buy more power at approximately equal efficiency and cost?
2) can I buy more efficiency at approximately equal power and cost?
3) can I buy more reliability at approximately equal power, cost & efficiency?

washu
Posts: 571
Joined: Thu Nov 19, 2009 10:20 am
Location: Ottawa

Re: Seeking input on planned router build

Post by washu » Thu Feb 05, 2015 11:10 am

I would suggest looking at the new consumer level atoms, something like a Gigabyte J1900N is under $100 new and comes with two NICs built in. Another option if you can find one, is the GA-C1037 which is also under $100 new with two NICs and has a dual core Ivy bridge based CPU. Either of those should have lots of grunt for your purposes even with DPI and use much less power. The J1900N likely better if your software can scale to 4 cores, the C1037 better if not.

Jay_S
*Lifetime Patron*
Posts: 715
Joined: Fri Feb 10, 2006 2:50 pm
Location: Milwaukee, WI

Re: Seeking input on planned router build

Post by Jay_S » Thu Feb 05, 2015 11:55 am

pfsense doesn't usually (always?) receive timely driver updates from FreeBSD. Some Realtek controllers are supported, but almost everyone recommends sticking to mature Intel controllers. Whether that's just fanboyism or not, I'm not experienced enough to judge.

The GA-C1037 boards have Realtek 8111x NICs. pfsense forums suggest support depends on which version of the 8111 Gigabyte decides to ship with. And sadly, no PCI express slots.

The bay trail atoms seem like a gamble but for different reasons (BIOS, according to reviews), and they still have Realtek NICs. But they do have a PCIe slot, so that's progress!

Either of the above, fully built out would cost well over $100. Right now I'm aiming for ~ $100. If I ever get serious, I'd spend for something like Supermicro's Atom C2550 barebones. After reading SPCR's review of it's big brother, the 1U PSU doesn't scare me as much as it used to.

<sigh> Just prior to getting interested in a router build, I gave away an old C2D optiplex. Would have been a perfect test mule.

washu
Posts: 571
Joined: Thu Nov 19, 2009 10:20 am
Location: Ottawa

Re: Seeking input on planned router build

Post by washu » Thu Feb 05, 2015 12:40 pm

The problems with realtek NICs are way overblown. Yes, they do use more CPU than better cards but for 99% of uses they are just fine. If you were trying to to DPI at full Gbit then high end NICs are needed. At 10 Mbit not so much, your system probably wont even notice the tiny bit of extra CPU. And in 99.99% of cases, a PCIe realtek is better than a PCI (no e or x) Intel. I've seen people make that mistake many times.

I have the older version of the GA-C1037, the GA-1007. It's pretty much the same board only at 1.5 instead of 1.8 GHz. I run stock FreeBSD on it and have no problems with the realtek NICs. They can push full 1 Gbit/sec with no problems.

pFsense is a great firewall distro, but it is based on an older version of FreeBSD. It will have just as much problem recognizing a new Intel NIC as a new Realtek.

If you are really worried about cost vs power then just get an old used Atom. Cheap, low power and the NIC is pretty much guaranteed to work with pFsense, realtek or not. Unless you know you are going to increase your network speed significantly in the near future they should be fine for 10 mbit even with DPI. I've run Snort on machines way slower with higher network speeds. Just don't make your ruleset insane.

If you are really serious about doing the PC-as-a-router in the long term look around for a cheap switch that supports VLANs. Then you don't have to worry about multiple NICs anymore.

CA_Steve
Moderator
Posts: 7650
Joined: Thu Oct 06, 2005 4:36 am
Location: St. Louis, MO

Re: Seeking input on planned router build

Post by CA_Steve » Thu Feb 05, 2015 12:48 pm

Your electric rate is ~$0.103/kWh? That's about $0.9/W/yr for an always on device. Figure the C2D system idles at 20W higher level than the J1900. So, what you save in upfront hardware costs derates by $18/yr.

Jay_S
*Lifetime Patron*
Posts: 715
Joined: Fri Feb 10, 2006 2:50 pm
Location: Milwaukee, WI

Re: Seeking input on planned router build

Post by Jay_S » Thu Feb 05, 2015 2:21 pm

Thanks all,
washu wrote:The problems with realtek NICs are way overblown. Yes, they do use more CPU than better cards but for 99% of uses they are just fine. If you were trying to to DPI at full Gbit then high end NICs are needed. At 10 Mbit not so much, your system probably wont even notice the tiny bit of extra CPU.
<snip>
pFsense is a great firewall distro, but it is based on an older version of FreeBSD. It will have just as much problem recognizing a new Intel NIC as a new Realtek.
My impression is that some reatek NICs simply do not work at all. I'm not concerned about a few % CPU use or maximum NIC throughput (after all, I'm limited by my 10Mbps WAN). But I want the NICs to 'just work' out of the box. Intel's 82571 (found on the dual-port PCIe NICs I'm watching) is on the FreeBSD 10.1 HCL. The HCL also lists support for Realtek's 8111, but doesn't detail which 8111 versions are supported. June 2013 RTL8111f is supported but 8111g is not. Oct 2014, both 8111g and 8111f are supported. Which driver is in pfsense 2.2? Admittedly, I'm out of my league sifting through this stuff.

For the record, I actually love the little gigabyte GA-C1037UN. Assume the NICs are perfect - I don't think I could buy it, a case, RAM, storage & PSU for under $100.

washu wrote:...a PCIe realtek is better than a PCI (no e or x) Intel. I've seen people make that mistake many times.
I have a PCI Intel NIC. I bought it ages ago because I couldn't get WOL working on an old motherboard's onboard NIC (Asrock 939Dual-SATA2, Realtek 8201CL, ULi chipset). With the PCI Intel NIC, WOL worked perfectly but it couldn't exceed 300Mbps (w/real file x-fers or iozone), even though iperf hit 800+Mbps. Strongly suggests the PCI bus was the likely bottleneck. I think the SATA controller was even on the PCI bus! That motherboard was simultaneously really cool and really goofy.

washu wrote:If you are really worried about cost vs power then just get an old used Atom. Cheap, low power and the NIC is pretty much guaranteed to work with pFsense, realtek or not.
Yeah but cheaper than $100? It really seems that, based on what's available to me, a used C2D is the value leader. Used HP 8000's may be an even better deal because DDR3 is cheaper in higher densities. Again, I'm targeting $100 because I'm not even sure I'll stick with any of these software router distros.

CA_Steve wrote:Your electric rate is ~$0.103/kWh?
My rate is a little more: $0.13111/kWh. Assuming a 20W difference = $22.97 / year delta in electrical costs. I could double my initial budget and assuming 1/2 the power consumption I break even at ~ 4.3 years * 24/7.

CA_Steve
Moderator
Posts: 7650
Joined: Thu Oct 06, 2005 4:36 am
Location: St. Louis, MO

Re: Seeking input on planned router build

Post by CA_Steve » Thu Feb 05, 2015 3:20 pm

I remember my e8400 gaming system idling in the 50W range, discounting the discrete gfx card. I'm bringing up a Zotac Zbox Nano C1320 now and it's idling at 6W. food for thought.

dancingsnails
Posts: 21
Joined: Thu Jan 07, 2010 12:05 am
Location: San Francisco

Re: Seeking input on planned router build

Post by dancingsnails » Thu Feb 05, 2015 3:33 pm

I'm quite happy with my Ubiquiti edgerouter lite for $99. I haven't tried installing clamav on it, but it's Debian based, so it should be possible once you enable the right repositories. It's a nice (fast) router, if you don't mind dropping to the CLI sometimes. It uses 6 watts or so.

jhahn@ubnt:~$ sudo apt-get install clamav
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
clamav-base clamav-freshclam libclamav6 libjson0
Suggested packages:
clamav-docs apparmor libclamunrar6
The following NEW packages will be installed:
clamav clamav-base clamav-freshclam libclamav6 libjson0
0 upgraded, 5 newly installed, 0 to remove and 31 not upgraded.
Need to get 2217 kB of archives.
After this operation, 4275 kB of additional disk space will be used.
Do you want to continue [Y/n]? n
Abort.
jhahn@ubnt:~$

washu
Posts: 571
Joined: Thu Nov 19, 2009 10:20 am
Location: Ottawa

Re: Seeking input on planned router build

Post by washu » Thu Feb 05, 2015 7:36 pm

Jay_S wrote:My impression is that some reatek NICs simply do not work at all. I'm not concerned about a few % CPU use or maximum NIC throughput (after all, I'm limited by my 10Mbps WAN). But I want the NICs to 'just work' out of the box. Intel's 82571 (found on the dual-port PCIe NICs I'm watching) is on the FreeBSD 10.1 HCL. The HCL also lists support for Realtek's 8111, but doesn't detail which 8111 versions are supported. June 2013 RTL8111f is supported but 8111g is not. Oct 2014, both 8111g and 8111f are supported. Which driver is in pfsense 2.2? Admittedly, I'm out of my league sifting through this stuff.
You are not making a fair comparison. The Intel 82571 is a fine NIC but it is pretty old. It has had years to be supported. You are comparing it to new realteks. You could get a standalone realtek nic of similar age and it would be just as well supported.

A new MB comes with new NICs. Realtek or Intel, they might not be supported until FreeBSD gets updated.

Intel makes things a bit easier by giving new NIC chip revisions new designations while realtek often just bumps the revision. It is two ways of saying the same thing.

pFsense 2.2 is basically FreeBSD 10.1, so that HCL applies.

Post Reply