Personal tools

Multi-WAN Version 1.2.x

From PFSenseDocs

(Redirected from MultiWanVersion1.2)
Jump to: navigation, search

This community-contributed guide leaves out some important information and considerations. The best source of multi-WAN information is in the pfSense book.

Introduction

This setup enables pfSense to load balance traffic from your LAN to multiple internet connections (WANs).

Traffic from the LAN is shared out on a round robin basis across the available WANs.

pfSense monitors each WAN connection, using an IP address you provide, and if the monitor fails, a failover configuration is used, this typically just feeds all traffic down the other connection(s).

This example sets up 2 WANs, but 3 or more can be used by simply extending what this page describes.

Note that currently most pfSense add-on packages do NOT support multi WAN and all their traffic will use the WAN connection.

Overview

Networks and computers in a multi WAN installation

This guide helps you setup pfSense to support a local network (the LAN) and 2 connections to the internet (WAN and WAN2). Most traffic is shared out between the 2 WAN connections, but specific rules are also setup for some types of traffic to only use 1 connection (for example https), where load balancing can cause problems.

pfSense runs in a small system that uses 3 network interface cards (NICs), 1 for each of the WANs and 1 for the LAN.

pfSense can also be run in a virtual machine for testing and lightweight use, although this is not as secure or robust as a physical machine implementation.

The guide also shows how to setup access from the internet to servers on the internal network, and has guides to the setup for some specific applications.

Note that if you install servers connected to DMZ1 or DMZ2, these are not protected by pfSense, and will have to be internet hardened.

Before you start

You must have completed the basic pfSense installation.

Target network setup

This guide assumes the following network setup; you can easily do something different, but you will need to translate network addresses appropriately if you do.

  1. Your ISPs have assigned a single IP address for each internet connection (which could be dynamic) and you are using your modem / routers in router mode (some guidance on other variants of this are included in the details below).
  2. DMZ 1 is going to use the subnet 192.168.0.0/24
    This means that DMZ 1 uses IP addresses between 192.168.0.0 and 192.168.0.254.
  3. DMZ 2 is going to use the subnet 192.168.1.0/24
    This means that DMZ 2 uses IP addresses between 192.168.1.0 and 192.168.1.254
  4. The LAN uses subnet 192.168.10.0/24
    This means that the internal network uses IP addresses between 192.168.10.0 and 192.168.10.254

You should pick up the 3 interface cards. Note that if you have DHCP turned off on your WAN1 modem router, there will be a long pause here while pfSense tries to pick up an IP address.

Finishing pfSense console setup

The console will eventually give a prompt pfSense console setup. Select option 2 and setup up the LAN interface as follows:

LAN IP Address
192.168.10.254
subnet bit count
24 (for a class C space) - this will allow up 250 computers to be used
DHCP
y
DHCP start address
192.168.10.10
DHCP end address
192.168.10.200

You should now be able to plug a PC into the network, and it will be allocated an IP address and you will be able to access pfSense web interface (although not much else yet).

Setting up your modems / routers

Router mode setup

Modem / router setup for load balancing in router mode

If you have CABLE/DSL modems that are bridge routers you may want to use them in router mode. The client ID (PPPoE) is installed on the modem/router and the modem/router maps the Public IP it receives to a Private IP on the modem/router LAN interface. How to do this is specific to each modem/router.

settingWAN (WAN1) modem / routerOPT1 (WAN2) modem / router
LAN IP address192.168.0.254192.168.1.254
Subnet mask255.255.255.0255.255.255.0
DHCPonon
DHCP address range192.168.0.10 - 192.168.0.100192.168.1.10 - 192.168.1.100

Once you have set up the modem/routers you can test them by plugging a PC into their network, and accessing your favourite web site.

Or you can wait until the basic pfSense configuration is in place, and test through pfSense.

Note if you are *cheating* by running multiple subnets on one physical network, you must have DHCP turned off on all but 1 subnet.

Bridge mode setup

Modem / router setup for load balancing in bridge and router mode

If you have a fixed IP address from your ISP you can also use bridged mode for some or all of your connections. (If you do not have a fixed address it makes life complicated in pfSense)

In bridged mode, the modem becomes a transparent (in IP terms) device, and your internet IP address is allocated to the pfSense interface. This makes life a bit simpler as it means there is one less NAT going on.

You can usually set up at least WAN1 to work in bridge mode (if your modem / router allows it). as this connections allows PPPoE or bigpond account information to be configured in pfSense.

If you do this, your ISP assigned address will replace the 192.168.x.y address (from the router mode setup above) in the later sections of the setup.

Using the pfSense Wizard

  • Go to http://192.168.10.254 (or the address you gave pfSense if different)
  • Select System - Setup Wizard from the menu

General parameters screen

  • hostname
    • leave as pfsense
  • domain
    • as you like - I use me.local at home
  • Primary DNS server
    • a DNS address from WAN1 DNS list
  • Secondary DNS server
    • a DNS address from WAN2 DNS list
  • Allow DNS server list to be overridden by DHCP/PPP on WAN
    • Unchecked - if this is checked you won't see the right DNS server list when you set up load balancing pools
  • Click next

Note: it is important to use one from each (or use a public DNS service) or you will loose internet access when one or other connections fails.

date, time and time zone screen

  • time server DNS name
    • its a good idea to select a local service - either the one your ISP provides, or a local pool.ntp.org address (for example uk.pool.ntp.org if you are in the UK, or one in your time zone).
  • Timezone
    • pick the right entry from the time zone. Note pfSense can provide an NTP service so all your local machines pick up time from pfSense.
  • click Next

WAN configuration

If have set your WAN modem router to DHCP, you can leave this set to DHCP, otherwise:

  • Selected type
    • Static
  • IP address
    • 192.168.0.1 /24 (or an address in your DMZ1 subnet)
  • Gateway
    • 192.168.0.254 (or the address you gave your fist modem / router

If you are using a plain modem then you can set up your ISP account information here, I can't find a wiki page about this, but there several threads in the forums that discuss this.

LAN configuration This was set up through the console so shouldn't need changing

Change your password and reboot

Put in a sensible password, then let pfSense reboot.

After Wizard general setup

These settings make it easier to access machines on your local network - you can access them by name, and if you are running Windoze you will not suffer at the vagiaries of WINS.

  • Go into 'Services' - 'DNS Forwarder', turn on
    • Register DHCP leases in DNS forwarder
    • Register DHCP static mappings in DNS forwarder

Initial setup for Load balancing

Finishing the interfaces setup

Now it is time to finish setting up the interfaces and make sure they are setup OK.

Setting up the OPT 1 interface

Optional 1 (WAN2) set up for a MultiWAN configuration

From the pfSense menu select Interfaces - OPT1 and set up as follows:

enable Optional 1 interface
checked
Type
Static - assumes you are not using an address assigned by your ISP
MAC address and MTU
do not usually need to be set - see info on screen
Bridge with
None
IP address
192.168.1.1 /24 - or an appropriate address in DMZ 2 if you used a different subnet
Gateway
192.168.1.254 - or whatever address you gave modem / router 2 (or your ISP has assigned, if no routing being used)

Checking interfaces

Interfaces set up for a MultiWAN configuration

From the pfsense menu select Interfaces - Assign and you should get an screen like the one of the right. Note your hex numbers (The MAC addresses) will be different.

Now to check that pfSense can see your modem routers you use Diagnostics - Ping. With WAN 1 selected, enter the IP address of your modem / router - 192.168.0.254 if you are using the guide values in this document.

If you are using using a modem / router without NAT, the check first that the WAN link is up and ping the DNS server address that you recorded earlier.

FTP helper: Check also that FTP helper is only enabled for the LAN interface. That is it should be disabled on all WAN interfaces


Setting up Load Balancing pools

Overview

how the various Pools and gateways are related, and how they can be used

This setup uses 3 pools

  1. One pool for load balanced use when both WANS are working
  2. One pool which prefers WAN 1, for use when WAN 2 has failed
  3. One pool which prefers WAN 2, for use when WAN 1 has failed

These pools use the 2 gateways that are already established (by the interfaces WAN and WAN 2) to load balance and support failover when a WAN link fails

Selecting a Monitor IP address

pfSense monitors each WAN connection by pinging the monitor address you specify. If the ping fails, the link is marked down and the appropriate failover configuration is used (actually if the ping fails it retries a few times to be sure, this avoids false indications of the connection going down).

Note that pfSense automatically sets up to route traffic to your monitor IP only down the link it is monitoring, so don't use a popular web site as this will force all its traffic down 1 link. Better to use a router or server in your ISP's network.

Good addresses to use your ISP's DNS server (1 from each ISP). The web interface makes it easy to pick these when setting up the pools later.

Other good monitor addresses are the default gateway your modem has assigned (if it responds to ping!), your ISP's webmail server, or a router within your ISP's network - you can find one of these by using traceroute to a public service, be careful though, larger ISPs will have networks that dynamically adapt so a router you see now may not be there an hour later!

Setting up the pools

Setup for the first (load balancing) pool

We are going to set up 3 pools in Services - Load Balancer

Note that each pool has 2 monitors set up, when complete the 1st pool should correspond to the screenshot on the right.

SettingPool 1Pool 2Pool 3
Pool nameLoadBalanceWAN1FailsToWAN2WAN2FailsToWAN1
DescriptionRound Robin load balancing WAN 2 preferred when WAN 1 failsWAN 1 preferred when WAN 2 fails
TypeGatewayGatewayGateway
BehaviorLoad BalancingFailoverFailover
PortUnusedUnusedUnused
1st Monitor IPDNS server 1DNS server 2DNS server 1
1st Interface nameWANWAN2WAN
2nd Monitor IPDNS server 2DNS server 1DNS server 2
2nd Interface nameWAN 2WANWAN2
3 pools set up ready for load balancing

This finals screenshot shows the summary you should end up with.


Setting up DNS for Load Balancing

Make sure that you have a DNS server from each ISP in the General Settings. This will ensure that you have DNS service in case one ISP goes down. You will also need to setup Static Routes for each DNS server. In this example if the DNS is on the WAN link then the static route for that DNS server will have 192.168.0.254 as the gateway. If the DNS server is on the other ISP (ie OPT1) then the static route will have have 192.168.1.254 as the gateway.

Sticky Connections

pfSense Version 1.2 introduced Sticky connections, which can be used as part of a MultiWan setup. Where Sticky connections are used, some of the firewall rules previously used are no longer required; this is noted in the information below. 'Sticky connections' are a very good where there are many active systems / users, or where your WAN connections are fast, they are not so useful for small number of users on slower connections (as the multiple requests involved in fetching a single web page will not be shared across the available connections.

Basic Firewall Rules

These are the rules you need to add to support access from your LAN to the internet. Later sections describe the rules you need to support incoming access from the internet to machines on your LAN, this includes how to support peer to peer applications.

First 3 rules

If you do not need to access any of your systems from the internet, and you use sticky connections, then these are probably the only rules you will need.

Set these rules up in Firewall - Rules, and then click the LAN tab.

RuleLoad BalanceDMZ 1DMZ 2
Position in rule listLastTopTop(-1)
ActionPassPassPass
DisabledUncheckedUncheckedUnchecked
InterfaceLANLANLAN
Protocolanyanyany
SourceLAN subnetLAN subnetLAN subnet
Source OSanyanyany
Destinationanynetwork:
192.168.0.0 / 24
WAN2 subnet
Lognoyes temporarily (see below)yes temporarily (see below)
Schedulenonenonenone
GatewayLoadBalancedefaultdefault
Description Everything else gets shared out Make sure DMZ 1 traffic goes to right interface Make sure DMZ 2 traffic goes to WAN2 DMZ

Rule logging

It is always a good idea to put a new rule in with logging turned on, then check by generating some appropriate traffic, that the rule is working, then turn logging off once you know it is having the right effect.

Rule explanation - Load Balance

This rule must always be the last rule in the rule list. It catches anything else that is not special in any way, and load balances the traffic. Any rule that comes after this rule will never trigger, so may as well not be there!

Rule explanation - DMZ 1 and DMZ 2

These rules make sure that any traffic to the modem / router, (or other machines that are connected to this subnet if you are not using bridge mode), go down the right WAN connection. Without these rules you will find strange things happening when you try to access your modem / router.

These rules should always be top of the rule list as you do not want earlier rules to route this traffic elsewhere.

Testing these rules

Testing the DMZ rules
Use a web browser to access the administration interface on your modem / router. Then use Status - System Logs, Firewall tab to check if the rule has fired.
Testing the load balancing rule
Access any site on the internet, then check the firewall log (as above) to see if the rule fired.

Don't forget to turn off logging on the rules once you have checked them.

Testing failover

Now you should make sure that failover is working.

  • Switch off (or unplug) one modem / router
  • Check the pfSense Load balancer status screen ('Status' - 'Load Balancer')
    it should show (within a few seconds) that one link has failed.
    • if it shows that both links have failed, it probably means you have your monitor IP's the wrong way round. Use a trace route from PC on the LAN to trace the route to each monitor IP address and if it is using the wrong WAN link, re-setup the WAN links the right way round.
  • Now try accessing a internet site, it should appear without any problems.
    If it fails, then check the load balancer status (see above). If one link is still marked up, check that it is not a DNS failure

Setting up for protocols that don't like load balancing

Some sites (for example banking sites) get upset when requests from a single session come from different IP addresses. To avoid this, protocols that are likely to suffer from load balancing are setup to favour 1 connection.

Note that use of the sticky bit (see above) should avoid this issue. If you are not using sticky bit, you definitely need this.

For each protocol that needs to be handled this way you need a rule on the LAN interface; the sample below is for https (port 443). The values marked in bold are the ones that change for different protocols.

These rules need to be above the final load balancing rule, and below the rules for DMZ access.

ParameterValue
ActionPass
Disabledunchecked
InterfaceLAN
ProtocolTCP
Source: notunchecked
Source: typeLAN subnet
Source OSAny
Destination: notunchecked
Destination: typeany
Destination port rangeHTTPS
Logchecked initially; uncheck when known to be working
GatewayWAN1FailsToWAN2 - or WAN2FailsToWAN1 as you prefer
DescriptionRoute https through one working connection

Other entries you are likely to need are SSH and POP3. For these just replace HTTPS in bold above with the protocol you requre, and amend the description.

Further Rules for handling outgoing traffic

Depending on usage there are likely to be other rules you will need for outgoing traffic.

Setting up rules to access specific ISPs

If you send traffic to hosts on a specific ISP (such as SMTP email) you may have to make sure that traffic goes to the right ISPs WAN connection. ISPs block mail being sent if it does not come from one of their customer's lines, so if you try to send mail through the wrong connection it will be rejected. If your WAN connections are from different ISPs and you send mail using SMTP you will need to do this. If you only use webmail (your email interface is a web browser, such as hotmail), you do not need this.

The simplest way to handle this is to route all SMTP traffic to one ISP - of course if you send SMTP mail through both ISPs you will need to handle this a different way.

For this type of use, the rule is setup to use only 1 WAN connection. This means that if the connection goes down, the traffic cannot pass, but as it would fail if it picked up the other connection this is the right behaviour.

The example below is for SMTP, change the bold parameters for other traffic

These rules should go in above both DMZ and preferred traffic rules

ParameterValue
ActionPass
Disabledunchecked
InterfaceLAN
ProtocolTCP usually
Source: notunchecked
Source: typeLAN subnet
Source OSAny
Destination: notunchecked
Destination: typeany
Destination port rangeSMTP
Logchecked initially; uncheck when known to be working
Gateway192.168.0.254 or 192.168.1.254 or the appropriate gateway address for this traffic
DescriptionRoute SMTP to the ISP that handles it
This article is part of the HOWTO series.