Check Point Firewall Guide Performance Optimization: The Dual Default Gateway Problem

Is your Check Point Firewall connected to your core internal router on a dedicated VLAN/segment with no other systems present? In other words, is your firewall connected to your core internal router with a transit network used solely to forward traffic between the firewall and core router like this:

Figure 3-4: A Private Firewall- Core Transit Network

Or do you have something like this:

Figure 3‑5: Non-private Transit Network between Firewall and Core Route

I may have some bad news for you; this latter network architecture is infamous for causing network performance issues. Ideally the VLAN between the firewall’s internal interface and your internal core router is simply used to transit traffic between the two entities. When there are other systems present on that transit VLAN, those systems have two possible gateways for routing and are in a kind of a “no man’s land” in regard to routing.;So what should those systems’ default gateway be? In most scenarios the default gateway is set to be the internal core router and things seem to work. However this setup will cause one of the following to occur:

• Traffic initiated from these systems to the Internet and/or firewall DMZs will flow asymmetrically to and from the firewall
• The core router will issue ICMP Redirect packets to dynamically update the routing tables of the no-man’s-land systems to encourage them to send Internet/DMZ traffic directly to the firewall and avoid asymmetry

Let me guide you through these two scenarios. The former is a minor issue while the latter can be a major issue from a performance perspective.

Asymmetric Path Issue

In the first scenario let’s suppose one of the no man’s land systems (a user workstation) between the firewall and the Internal core router needs to send traffic to a system located on the internal network. The destination IP is determined to be nonlocal, so the workstation sends the packet to its default route (the internal core router) in the forward direction (Stage 1). The packet is sent to the destination inside the network by the core router, and on the return path the core router sends the reply directly to the workstation (Stage 2). Everything was symmetric and handled properly; no problems here:

Figure 3‑6: Symmetric Traffic Path from Server to Internal Network

Now let’s suppose a system in no-man’s-land initiates a connection to the Internet:

Figure 3‑7: Asymmetric Traffic Path to the Internet for Serve

Once again the destination IP is determined to be nonlocal, so the workstation sends the packet to its default gateway-the internal core router (Stage 1). Notice that the packet went towards the internal network and not towards the Internet, at least initially.Upon receipt let’s assume the core router is not configured to send ICMP redirects. In this case the core router will “hairpin” or “u turn” the packet right back out the same interface it was received towards the firewall (Stage 2). Let’s assume the firewall accepts the packet and forwards it to the Internet (Stage 3).;The reply returns to the firewall on its external interface (Stage 4). The firewall then sends it directly to the no-man’s-land system (Stage 5). Notice how the forward path and the return path were not completely identical, which is by definition asymmetric. From the firewall’s enforcement perspective the connection was symmetric, since the traffic crossed the same firewall internal interface on both the forward and return path.

Normally this situation will not cause a dramatic performance issue, but in general it is desirable to avoid asymmetric pathing in a network. As an example of why, let’s suppose the Internal core router is overloaded due to numerous backups running between several different VLANs on the internal networks. A user on the no man’s land workstation initiates an FTP to the Internet and GETs a large file. The main flow of traffic (and largest packets) in this case is in the return direction from the Internet to the workstation and the core router is not involved with routing this heavy traffic. There is a stream of ACKs being sent by the system through the core router, back to the firewall and to the Internet in the forward direction, but these are small infrequent packets that are not dramatically impacted. In this case the user sees very good file transfer performance and is not directly impacted by the heavy load on the core router.

However in the same FTP connection session the user attempts a file PUT operation.Now the heaviest flow of traffic is in the forward direction from the user’s workstation to the Internet, which must flow through the overloaded core router and back through the firewall. The user now sees ridiculously bad file transfer performance due to the overloaded core router losing and/or inordinately delaying the heavy flow of traffic. This is a textbook example of why it is desirable to avoid asymmetric pathing in the network; it can cause strange, inconsistent performance issues such as these.

ICMP Redirect Issue

The second possibility in our Dual Default Gateway discussion is far less obvious and will almost certainly cause intermittent performance issues and even brief outages in some extreme cases. Let’s step through it.

Figure 3‑8: ICMP Redirect from Core Router to Force Symmetric Path for Serve

The workstation in question initiates a connection to a system on the Internet. Let’s suppose the destination IP address is 129.82.102.32. Once again the destination IP is determined to be nonlocal, so the system sends the packet to its default gateway the core router (Stage 1). Notice that the packet went towards the internal network and not towards the Internet, at least initially. Upon receipt let’s assume the core router is configured to send ICMP Redirects. Upon consulting its routing table, the core router notices that the workstation sent the packet to the “wrong” default gateway. If the core router is configured to issue ICMP redirects it forwards the packet to the firewall AND sends a special ICMP message called a redirect (ICMP Type 5) to the workstation basically saying “hey you went the wrong way, instead of using me as the gateway for this destination host address send it to the firewall instead” (Stage 2). Upon receipt of this special ICMP packet, the workstation dynamically adds a temporary static, host based route to its routing table for address 129.82.102.32 with its next hop being the firewall internal interface address. All future packets to 129.82.102.32 are now sent directly to the firewall (Stage 3), thus assuring network path symmetry. Now FTP performance to this Internet destination from the workstation will be completely unaffected by the performance state of the internal core router.

This sounds great, but danger lurks.There are many things that can go wrong here and all of them will impact the network performance of the user’s workstation. First off, how do we know if this situation is present in our network? On the user’s workstation, initiate some traffic to a few Internet sites, then run a netstat -s command and under the ICMP section look for a counter called “Redirects”. If it is nonzero and increments every time a unique Internet site is visited, the situation discussed is present and you should not rely on this mechanism for a properly functioning network.

However this situation may be present in your network but everything seems to be working just fine! You are lucky, but eventually you will have this hurt you at some point; it is not a question of if but when. Why? Here is a sampling of a few things that can go wrong:

• The router is rate limited on the number of ICMPs it is allowed to send per time interval. If it cannot send ICMPs fast enough to keep up with demand, asymmetry will return at seemingly random intervals and potentially degraded performance with it.
• ICMP messages are not sent reliably. There is no mechanism to detect if they were lost and resend them.
• If a workstation is communicating with a large number of destination IP addresses on the Internet, the workstation’s routing table can fill up with hundreds or thousands of routing table entries, thus degrading route lookup performance on the host.

It is risky from a performance perspective to have a Dual Default Gateway scenario as has been described here and it may be impacting your network’s performance right now without you even being aware of it. Unless you want to start hard coding static routes on all workstations located in no-man’s-land, you’ll need to create an isolated VLAN solely for transit between the internal core router and the firewall as follows:

1. In an outage window move the VLAN implemented between the core router and firewall and its IP addressing scheme to somewhere inside the network. ;No changing of IP addressing or default gateways will be required on the no man’s land workstations being moved.
2. Next create a new private VLAN Plug the firewall internal interface into this VLAN and give it a new IP addressing scheme (anything from the private RFC1918 ranges is fine, as long as it does not conflict with your internal networks or any business partners you are connected to). Allocate a new IP address for the internal router on this new network as well and plug it in.
3. Change the default route on the core router to point to the firewall’s new internal address.
4. On the firewall, change the next hop for all internally facing routes (routes for which the next hop is the internal core router) to the core router’s new IP address on the private VLAN.

Timothy Hall is the author of Max Power: Check Point Firewall Performance Optimization. He is an Information Security Professional with over 20 years of experience, and has worked with Check Point firewall products since 1997. If you want to contribute as well, click here.