Home

FortiGate IPS Configuration: The Complete Guide to Deploying Intrusion Prevention

.

In today's threat landscape, where zero-day exploits and sophisticated malware constantly evolve, a robust firewall alone is insufficient. Intrusion Prevention Systems (IPS) act as a vital real-time threat detection and blocking layer, inspecting traffic to identify and stop attacks that exploit application and system vulnerabilities. FortiGate's integrated IPS solution combines signature-based detection, anomaly-based analysis, and behavioral heuristics to provide comprehensive protection. This guide distills essential information from official Fortinet documentation and expert resources to provide a complete roadmap for configuring and optimizing FortiGate IPS, ensuring your network is shielded from both known and emerging threats.

Understanding Core Concepts and Deployment Modes

Before configuration, understanding where and how the IPS operates is crucial. FortiGate IPS can be deployed in several operational modes, each fitting different network architectures.

  • Layer 3 (NAT/Route Mode): The most common deployment, where the FortiGate operates as a router between network segments (e.g., LAN and WAN). All traffic between these segments is routed through the FortiGate, where IPS policies can be applied. It requires IP configuration on each interface.
  • Virtual Wire Mode: The FortiGate is deployed transparently between two network segments (like between a router and a core switch) without performing routing or NAT. This requires no changes to existing IP schemes and is simpler to deploy than transparent mode, while still allowing for granular, VLAN-based policies.
  • Transparent Mode: The FortiGate operates as a Layer 2 bridge within the same broadcast domain. All interfaces in a Virtual Domain (VDOM) belong to the same forwarding domain. While offering deep inspection without IP changes, it requires careful design to avoid network loops.

For high-availability scenarios, the FortiGate Clustering Protocol (FGCP) in Active-Passive or Active-Active clusters ensures uninterrupted IPS inspection during a device failure, with session synchronization maintaining protection continuity.

Step-by-Step Guide: Creating and Configuring an IPS Sensor

An IPS sensor is a container for signatures and filters that you apply to firewall policies. Configuring one is the primary administrative task.

1. Basic Sensor Creation (GUI)

Navigate to Security Profiles > Intrusion Prevention and click Create New. Essential settings include:

  • Name: Assign a unique, descriptive name (e.g., IPS-External-Facing-Servers).
  • Block Malicious URLs: Enables a local database to block connections to URLs known for drive-by download exploits. This database is updated via FortiGuard.
  • Botnet C&C: Defines action for traffic to known botnet command-and-control servers. Options are Disable, Monitor (log only), or Block.

2. Adding Signatures and Filters

This is the core of your sensor's intelligence. You can add entries of type Signature (individual) or Filter (group-based).

Configuration Aspect Signature-Based Entry Filter-Based Entry
Best For Applying actions to specific, individual signatures. Efficiently grouping signatures by common attributes (e.g., all high-severity attacks against web servers).
Key Settings Action (Allow, Monitor, Block, Reset, Quarantine), Status, Packet Logging, Rate-Based Settings. Action, Status, and defining the filter by attributes: Target, Severity, Protocol, OS, Application.
Action Options Block: Drops the packet. Reset: Drops packet and tears down the TCP session. Quarantine: Blocks traffic and can quarantine the source IP (requires FortiAnalyzer). Monitor: Allows traffic but logs the event.

Rate-Based Settings (for Signature entries) are a powerful tool to reduce noise and target sustained attacks. You can set a threshold (rate-count) and duration (rate-duration). For example, you can configure a signature to only trigger its "Block" action after being matched 10 times within 30 seconds from the same source IP (rate-track), preventing single false positives from causing outages.

Advanced Configuration and Optimization Options

To fine-tune IPS performance and capabilities, several global and sensor-specific options are available, primarily via the CLI.

1. Performance and Hardware Acceleration

  • IPS Engine & Algorithm: For multi-processor units, you can adjust the number of concurrent IPS engines (engine-count). The search algorithm can be tuned for speed (high) or memory efficiency (low), with engine-pick as the default.
  • NTurbo & IPSA: These features offload processing to specialized network (NP) and content (CP) processors. 
    config ips global     set np-accel-mode {none | basic}  # For NTurbo on NP processors     set cp-accel-mode {none | basic | advanced} # For IPSA on CP8/CP9 processors end
    The advanced mode for IPSA is available on models with CP9 Security Processing Units (SPUs).

2. Database and Coverage

  • Extended IPS Database: Provides broader signature coverage. On non-CP9 SPU models, enabling this delivers a "slim" version for performance balance. CP9 SPU models and VMs with ≥8 vCPUs get the full database. 
    config ips global     set database extended end
  • OT Threat Definitions: Protects Industrial Control Systems (ICS/SCADA). Requires an OT Security Service license and is excluded by default (set exclude-signatures ot).

3. Resilience and Handling Overload

  • Fail-Open: Determines behavior if the IPS engine is overloaded. If disabled (default), traffic is dropped. If enabled, traffic bypasses IPS inspection to maintain network availability at the cost of temporary lost protection.
  • IPS Buffer Size: Adjusts the socket buffer size (socket-size). Increasing it can prevent fail-open scenarios during traffic bursts but uses more memory.

Deployment Architecture and Best Practices

Effective IPS deployment requires strategic planning.

  • Placement: Deploy IPS sensors at critical network boundaries:
    • Internet Perimeter: On policies governing inbound traffic to DMZ servers and outbound traffic from internal users.
    • Internal Segmentation: Between different trust zones, like between the corporate network and the data center or guest Wi-Fi.
  • Traffic Inspection: For maximum efficacy, combine IPS with SSL/TLS inspection to decrypt and scan encrypted traffic, which is a common vector for hidden threats.
  • Tuning Workflow: Start with a monitor-only policy for new sensors. Analyze logs in FortiAnalyzer to identify false positives and legitimate traffic matching signatures before enforcing blocking actions. Use exempt IPs for critical, trusted assets that may trigger benign alerts.
  • High Availability (HA): In an FGCP Active-Active cluster, enable load-balance-all for IPS-inspected traffic to distribute the processing load across all cluster units.

Troubleshooting and Maintenance

  • False Positives: Use the log details to identify the triggering signature. Refine the sensor by disabling or setting the signature to "Monitor," adjusting rate-based thresholds, or adding exempt IPs.
  • Performance Issues: If throughput is lower than expected, verify that hardware acceleration (NTurbo/IPSA) is enabled and supported for your traffic flow. Consider enabling the "slim" extended database on lower-end models.
  • Updates: Regularly update your IPS signature database through the FortiGuard service. Keep FortiOS updated to ensure access to the latest protocol decoders and engine improvements.

Frequently Asked Questions (FAQ)

What is the fundamental difference between a signature and a filter in a FortiGate IPS sensor?

A signature is a single, predefined pattern that matches a specific exploit or attack. A filter is a collection of attributes (like severity, target OS, or protocol) that automatically includes all signatures matching those criteria. Filters are efficient for applying broad policies (e.g., block all "Critical" severity attacks), while signatures are for precise control.

Should I enable the "Block Malicious URLs" feature?

Yes, in most cases. It provides an additional, lightweight layer of protection against web-based drive-by exploits by checking connections against a dynamically updated local database of known bad URLs, complementing the full IPS signature analysis.

When would I use the "Quarantine" action versus "Block"?

Use Quarantine when you want to not only block the attack but also automatically isolate the offending source IP address for a period, preventing further attack attempts. This typically requires integration with FortiAnalyzer. Use Block to simply drop the malicious packet/session without tracking the source IP further.

How does Fail-Open mode work, and when should I enable it?

When the IPS engine is critically overloaded, the fail-open setting decides the fate of new sessions. If disabled (default), traffic is dropped—prioritizing security. If enabled, traffic bypasses IPS inspection entirely—prioritizing availability. Enable this only in environments where continuous uptime is more critical than a temporary lapse in IPS protection, understanding the associated risk.

My FortiGate model has a CP9 SPU. Why is this important for IPS?

A CP9 SPU is a dedicated content processor designed for intensive security tasks. It allows your FortiGate to run the full extended IPS signature database and use the advanced IPSA acceleration mode, significantly boosting IPS throughput and capacity without impacting the main CPU.