By default, pfSense rewrites the source port on all outgoing packets. Many OS's do a poor job of source port randomization, if they do it at all. This makes IP spoofing easier, and makes it possible to fingerprint hosts behind your firewall from their outbound traffic. Rewriting the source port eliminates these potential (but unlikely) security vulnerabilities.
However, this breaks some applications. There are built in rules when Advanced Outbound NAT is disabled that don't do this for UDP 500 (ISAKMP for IPsec VPN traffic) this traffic will almost always be broken by rewriting the source port.
Note that 1.2.3 and older releases will not rewrite the source port on SIP, UDP 5060 traffic, by default though 2.0 and newer do. In many cases you must enable advanced outbound NAT and not rewrite the source port on this traffic, such as where multiple phones must connect on a single public IP. To do so you need to not check the static port box.
You may use other protocols, like some games amongst other things, that do not work properly when the source port gets rewritten. To disable this functionality, you need to use the static port option. Click Firewall -> NAT, and the Outbound tab. Click "Manual Outbound NAT rule generation (Advanced Outbound NAT (AON))" and click Save. You will then see a rule at the bottom of the page labeled "Auto created rule for LAN". Click + to copy that rule. Change the rule so it only covers the source IP of your device that needs static port, and any other settings you need. Check the "static port" box on that page, and click Save. Move the rule to the top of the list. Apply changes and this behavior will be disabled.