Check Point Alert of the Week: Policy installation resulted in high CPU load, cluster may failover

This is a real life sample alert from Indeni

Description:

A new policy has been recently installed. During the policy installation it appears that there has been a CPU usage level of more than 70% for a period of at least 30 seconds. This may result in a failover from the currently active cluster member to a standby one. A failover during high CPU loads may result in certain network traffic being blocked temporarily.

Manual Remediation Steps:

In order to avoid the failover and the possible loss of connectivity, set the kernel parameter fwha_freeze_state_machine_timeout to 30 or 60 seconds on all cluster members.
To set a kernel parameter you can use the “fw ctl set int” command. If you would like for the change to survive a reboot, you should place the new value in the fwkern.conf file (for SPLAT or GAiA) or by using modzap (for IPSO).
indeni recommends consulting with your technical support provider prior to changing a kernel parameter.

Read SK32488 for more information.

How does this alert work?

indeni tracks the CPU usage on firewalls as well as logs when policy installations are done. Once an installation is complete, indeni looks back at the CPU usage. If it’s above 70% (by default) on a ClusterXL cluster member, an alert is issued.