IPsec Troubleshooting

From PFSenseDocs

Contents 1 Common Errors 1.1 Mismatched Local/Remote Subnets 1.2 Failed pfkey align 1.3 pfkey Delete 1.4 REGISTER message 1.5 Stuck/Broken Phase 1 2 IPsec Packet Loss (with multiple tunnels) 3 IPsec Debugging 4 Shrew Soft VPN Client Debugging 5 Packet Loss with Certain Protocols 6 Some Hosts Work, Others Do Not 7 Dropping Tunnels on ALIX/embedded

Common Errors

Mismatched Local/Remote Subnets

Feb 20 10:33:41  racoon: ERROR: failed to pre-process packet.
Feb 20 10:33:41  racoon: ERROR: failed to get sainfo.

This can result from mismatched subnet masks in the IPsec tunnel definitions. Check to be sure that the local and remote subnet masks match up on each side, typically they should be "/24" and not "/32".

Failed pfkey align

racoon: ERROR: libipsec failed pfkey align (Invalid sadb message)

This should be fixed as of pfSense 1.2.1. If you still see this message after that, check to make sure that your Phase 2 timeouts match up on both ends of the tunnel. Some people still see this periodically with no ill effect. It shows up at intervals equal to the Phase 2 timeout, but nowhere near the actual expiration time.

pfkey Delete

ERROR: pfkey DELETE received

You might see this message repeatedly as Phase 2 is renegotiated between two endpoints (for multiple subnets). The tunnels still work, but traffic may be delayed while the tunnel is switched/reestablished. (more research needed for possible solutions)

REGISTER message

racoon: INFO: unsupported PF_KEY message REGISTER

This is a normal log message. It is not indicative of any problem.

Stuck/Broken Phase 1

Client:

racoon: ERROR: none message must be encrypted

Server:

racoon: ERROR: can't start the quick mode, there is no ISAKMP-SA

Or also:

racoon: INFO: request for establishing IPsec-SA was queued due to no phase1 found.

Try to stop and restart racoon on the client/opposite side. This can turn up if one side still thinks Phase 1 is good/active, and the other side thinks it is gone.

IPsec Packet Loss (with multiple tunnels)

If you have multiple/parallel IPsec tunnels and you have packet loss to one or more of the tunnels, it may be due to duplicate identifiers. Check your tunnel definitions, and be sure that for each tunnel you have defined between two given routers, there is a unique identifier.

Example:

• Tunnel 1: Site A:10.0.0.0/24 <-> Site B:192.168.0.0/24
  • Identifier: User FQDN, site-a1.example.com
• Tunnel 2: Site A:10.0.0.0/24 <-> Site B:192.168.10.0/24
  • Identifier: User FQDN, site-a2.example.com

Without unique identifiers for each tunnel between two given routers, the ipsec daemon cannot distinguish between the traffic for each tunnel, and will likely drop/lose packets.

IPsec Debugging

You can run racoon in debug mode in the foreground to view details on negotiation. First kill racoon, then launch it from a SSH session by running:

racoon -d -v -F -f /var/etc/racoon.conf

Or leave out the -F to run it in the background but log details to the IPsec log.

Shrew Soft VPN Client Debugging

Open the Trace app. Stop the IKE Service, and go to File, Options. Change the log output level to debug and click OK. Start the IKE Service and attempt to connect.

Packet Loss with Certain Protocols

If you experience packet loss only when using specific protocols (SMB, RDP, etc), you may need to reduce your WAN MTU. IPsec does not handle fragmented packets very well, and a reduced MTU will ensure that the packets traversing the tunnel are all of a size which can be transmitted whole. A good starting point would be 1300, and if that works, slowly increase the MTU until you find the breaking point, then back off a little from there.

Some Hosts Work, Others Do Not

If some hosts can communicate across a VPN tunnel and others cannot, it typically means that for some reason the packets from that client system are not being routed to pfSense. This could happen for a number of reasons, but the two most common are:

• Incorrect gateway on client system: pfSense needs to be the gateway, or your gateway must have a static route for tunnel traffic which forwards those packets to pfSense
• Incorrect subnet mask on the client system: If your VPN subnets are close, say 192.168.0.x and 192.168.1.x, ensure that the subnet mask is 255.255.255.0 on the client systems. If one of them has an incorrect mask, such as 255.255.0.0, it will try to reach the remote systems locally and not send the packets out via the gateway.

Dropping Tunnels on ALIX/embedded

If you experience dropped tunnels during periods of high IPsec throughput on an ALIX or other embedded hardware, you may need to disable DPD on the tunnel. When the CPU on an ALIX is tied up with sending IPsec traffic, it may not take the time to respond to a DPD request on the tunnel. As a consequence, the tunnel will fail a DPD check and be disconnected.