Guide for setting up Multi-WAN (Load balancing, failover, etc) on pfSense 2.0
As with previous multi-wan guides, this setup enables pfSense to load balance or fail over traffic from your LAN to multiple internet connections (WANs). With load balancing, traffic from the LAN is shared out on a round robin basis across the available WANs. With failover, traffic will go out the highest priority WAN until it goes down, then the next is used. pfSense monitors each WAN connection, using either the gateway IP or an IP address you provide, and if the monitor fails it will remove that WAN from use.
In most setups, there are only three parts that need to be done
Before starting, make sure you have all of your WAN-type interfaces enabled. For static IP WANs, make sure they all have a gateway set.
Make sure you can ping your gateway/monitor IP to confirm that each WAN is actually online and working before proceeding. You should be able to see this under Status > Gateways if you have gateways defined. If they're green, you should be OK.
Ensure you have a gateway entry for each WAN interface (Check System > Routing, on the Gateways tab)
Static IP WANs will have a normal gateway entry, DHCP/PPPoE/etc will have a dynamic gateway entry.
For every gateway there are some settings that can change their behavior slightly with respect to multi-wan usage. Most people can leave these set at the defaults, but others may need to alter them slightly based on the quality of their WAN.
By default pfSense will ping your gateway to determine the quality of your WAN. In some cases, that is not an accurate measure. For instance, if your WAN gateway is actually a device that is local to you, and not on the other side of your ISP circuit, then your actual WAN link could be down and pinging your gateway would never show it. Also, if your ISP gateway is up but your ISP experiences upstream failures, those cannot be detected by pinging your gateway.
You can enter a custom IP address to monitor here that will be used to determine the WAN quality. You can use a public website, Google public DNS, or any IP on the internet that responds to pings. The downside is that should that IP ever go offline, or suffer a failure of its own, your WAN could be marked down when it's really up.
By default all WANs on the same tier are considered equal when doing load balancing. If your WANs are different speeds, the weight parameter lets you give the system some bias toward a faster link. If you had a 50Mbit line and a 10Mbit line you probably would not want to share them equally, as it would often leave the 50Mbit line underloaded and the 10Mbit line overloaded. You can give your 50MBit line a weight of 5 so that you get a 5:1 ratio of usage to prefer the faster WAN.
Every WAN is different in how it operates "normally". Some WANs have low latency and no loss and are great, others perform normally even when there is some loss registered on the line or higher latency. You can use these fields to dial in link-appropriate values for what is actually an alarm state for your WAN gateways. On some lossy cable lines, increasing the loss percentage to 20 or more may be fine. On slow DSL or satellite links, a few hundred ms of latency is fine. You have to watch your own quality graphs to get an idea of what is good/normal for any given WAN.
Gateway groups (System > Routing, Groups tab) are just what their name implies. They group together gateways to act in a coordinated fashion. They can do load balancing, failover, or a mixture of the two.
A common practice for a two-WAN setup is to make three gateway groups for a multi-wan configuration: one that load balances, and two for failover, one preferring each WAN. This could be expanded for any number of WANs, just make one group that prefers each of them and fails over to some ordering of other WANs. This will let you selectively put traffic on each WAN as well as load balance.
In a gateway group, you assign each gateway to a tier to determine when it is used. The lower tier numbers are preferred. If any two gateways are on the same tier, they will load balance. If they are on different tiers, they will do failover preferring the lower tier. If the tier is set to "Never" then the gateway is not considered part of this group.
When two gateways are on the same tier, they will load balance. This means that on a per-connection basis, traffic is routed over each WAN in a round-robin manner. If any gateway on the same tier goes down, it is removed from use and the other gateways on the tier continue to operate normally.
When two gateways are on different tiers, the lower tier gateway(s) are preferred. If a lower tier gateway goes down, it is removed from use and the next highest tier gateway is used.
Because of the tier system, you can have any number of combinations of load balancing and failover that you like, such as One WAN that if it goes down fails to two load balancing WANs that if both go down fail to three load balancing WANs, and so on. The only limit is that there are only 5 tiers so such configurations can only go 5 levels deep.
Defining gateway groups is only part of the story. You must assign traffic to these gateways using the Gateway setting on firewall rules.
On your Firewall > Rules screen, on the tab for the internal interface you want to use the gateway group with, you can either edit your existing pass rules and add the gateway setting, choosing the gateway you want, or add a new rule to match only certain traffic you want to direct into the gateway group. Remember that rules are processed from the top down, and once a rule is matched, processing stops.
You can direct certain traffic to one WAN with a failover group, match some other traffic for another WAN, and let your catchall rule go to the load balancer.
When a firewall rule directs traffic into the gateway, it bypasses the firewall's normal routing table. Policy route negation is just a rule that passes traffic to other local or VPN-connected networks that does not have a gateway set. By not setting a gateway on that rule it will bypass the gateway group and use the firewall's routing table. These rules should be at the top of the ruleset -- or at least above any rules using gateways.
You should have at least one DNS server reachable on each WAN. This can be accomplished by editing your DNS servers under System > General and picking an interface for each DNS server. Make sure that the DNS server chosen for a given WAN will work there (i.e. it's public or from that ISP). The system's DNS forwarder will query all DNS servers simultaneously, so it should not be affected by a WAN failure.
If you have the DNS servers hardcoded on the clients, this limitation isn't relevant, however services on the firewall itself will still need DNS and could become slow or fail waiting for DNS if there is not a reachable DNS server.
In 2.0 it is possible to load balance/failover services from the local firewall such as OpenVPN, DNS requests, and Squid. That is out of the scope of this document, but may be covered elsewhere. Check the forum.