February 22, 2018

Splitting the IPS from the routing function with pfSense

By Eric Viseur Computer networks, Home server, Information security 0 Comments

Performance issues with my previous setup

On a previous post, I explained how I set up a virtual router between my home network and my ISP box. That setup was later enriched with the Suricata IPS, running using the NFQUEUE bindings to check all traffic that went through my router.

Although that setup was functional, it also prove to have very bad performance. I ended up with a bandwidth of roughly 7-8 Mbps towards the Internet, while my provider gives me a solid 65 to 80 Mbps most of the time. Well, now, that is frustrating. It turns out I’m facing the combination of several issues:

NFQUEUE has performance limitations;
All my virtual machines are running on a KVM host based on CentOS 7, using virtio-net network interfaces. While that works well, that also has performance limitations, and virtio has gone leaps ahead in terms of performance since CentOS 7 was released;
It appears (I didn’t save the source, sorry) that running a virtual machine with several vCPU’s on KVM results in very bad performance.

After giving it a bit of tought, I decided to split my setup between a front and a back router. My exisiting router, without the IPS and NAT functions, will act as the back router. I will add a front router to fulfill these functions. And since I’m both curious and sometimes lazy, I decided to go with pfSense for this one. I used this FreeBSD derivative in the past for other purposes and was very happy with the outcome.

A new architecture for better throughput

This new architecture clearly separates the roles. I could have done the NAT on the back router and have the front router act as a bridge, but it didn’t feel natural and doesn’t fit with the pfSense philosophy. It expects to have a LAN and a WAN interface and automatically sets up routing between them. Diverting from that would have resulted in an unnecessarily complex firewall policy.

So, let’s get going!

In this article I’ll bypass all the discussion about setting up virtual switches and so on, and focus on the IPS aspects. Everything described here happens in virtual machines running over KVM in my personal case. A physical port is connected to the ISP Lan and another one to the Endpoints network, for my WiFi access point.

Setting up the pfSense virtual machine

pfSense installs without much trouble over KVM with virtio devices. Just remember, as per [1], to disable all hardware checksum offloading to maximize performance. As I don’t intend to locally store many logs on that VM, I gave it a 8 GB virtual drive. For now, it appears to be more than enough.

As I plan to keep the DNS on my back router, I disabled the DNS Forwarder/Resolver services on pfSense and pointed it to my internal DNS server.

Front/back routing setup

Before thinking about deploying an IPS, we need to setup a routing chain. At this point, you should keep your existing back router in place and focus on the front router configuration. Translated to my schematic above, that means has both its previously existing connection to the ISP Lan (default route) and a connection to the Inter-routers network. Set up the necessary firewall rules, if any, so that the endpoints can communicate with the front router web interface. We need this access to configure the front router.

Moving the NAT to the front router

First, we want to set up outbound NAT – the equivalent of the MASQUERADE rule on iptables.

On pfSense, it all happens in Firewall > NAT > Outbound. I kept it simple and disabled all automatic rules generation (Manual outbound NAT rule generation mode). I know it affects performance every now and then but I don’t like all that UPnP crap that creates port mappings behind your back. Create the equivalant of an IPtables MASQUERADE rule here:

I didn’t bother with subnets filtering here because it all happens in the actual firewall rules.

The port forwarding rules are defined in, duh, Firewall > NAT > Port Forward. This one is pretty straightforward.

Now, we need to get acquinted with the firewall itself. pfSense using the OpenBSD firewall, that FreeBSD integrates too: pf (Packet Filter). It has three major differences versus iptables:

It is implicitly stateful. In other words, if you create a rule to allow new connections on a given interface and port, it will automatically track the connection and allow all the packets related to it without the need of a rule to track the ESTABLISHED,RELATED packets.
It is a last match firewall, opposite to iptables’ first match principle. It is possible to make a rule match even if it is not the last one with the QUICK keyword.
There is no equivalent of the FORWARD chain. Packets that just go through the router are also considered incoming packets, e.g. what goes into the INPUT chain of iptables.

The first difference is very interesting for a router firewall, especially with many different subnets: it can dramatically increase performance and tables readability by reducing the amount of rules to create and process.

I haven’t made a decision yet, but I might end up migrating my back router to FreeBSD/pfSense too for this very reason. My current firewall policy has over 100 rules and is sometimes VERY difficult to understand after a few weeks of not touching it. And I’m planning to add some more soon…

Firewall rules

With just the NAT rule, we have a working router, but a completely insecure one: nothing is filtered. pfSense natively does some stuff but this is far from sufficient. Go to the Firewall > Rules page.

This is a front router whose main job is to allow protected access towards the Internet. This means I want packets related to end users web browsing to be processess as soon as possible, e.g. a QUICK rule in the pf vocabulary. These rules can be created under the Floating tab.

The first rule allow all connections originating from the InternalSubnets alias, and with a destination other than the router itself (e.g.: any packet towards the internet) to go through. Since pf is natively stateful, the answer from the internet will also be allowed without any additional rule. The InternalSubnets alias has to be manually created contains all IPv4 subnets that are allowed to access the Internet.

The second rule is just necessary for some unusual monitoring stuff I’m testing and not mandatory at all.

With this, we’ve restricted who may access the Internet, and made sure it goes fast. At that point, I recommend you run a speedtest. If you have a good virtualisation platform, you’ll find out the performance impact is barely observable.

Under the WAN tab, we find the rules autogenerated from the NAT portmappings, and that’s it. Nothing more is necessary for my setup. Maybe you’ll need something else ! Finally, under the LAN tab, I added some rules for my monitoring server. By default, pfSense blocks everything on the LAN interface except the HTTPS and SSH access.

Alright, now, time to do what we actually installed pfSense for: the IPS!

Setting up Snort

Sadly, I had to let Suricata go. My virtualization setup is based on CentOS 7 with KVM. It turns out these boys don’t like Suricata over FreeBSD: I had multiple connection drops, even if Suricata wasn’t running in inline mode. I wasn’t happy, but hey, that’s life. Snort lacks some interesting features but is still a beast an as IPS.

Past the package installation, there are a few things to take care of:

You should create the Snort instance on the LAN interface. On the WAN side, the NAT is already done and you can’t tell which host is the offending one in your LAN.
I wanted packets offending enabled rules to be blocked, but my clients to retain connectivity, e.g. not end up on the Snort block list. To do this:
- In the firewall, create an alias that contains all the private subnets as well as the WAN subnet;
- Create a Snort pass list that contains that alias;
- When enabling BLOCKING mode, define that Snort pass list as the whitelist. This way, if an host communicates with a malicious host, the packet will be dropped and the malicious host will be blocked, but your internal hosts will always retain connectivity.

The rest is highly self-explanatory. The pfSense guys did a good job! Note you will get a lot of false positives in the beginning, so don’t forget to use the disable SID feature.