Automated Node.js Memory Leak Mitigation on FortiGate 50G: A Technical Case Study

Abstract

This article documents a critical memory management issue discovered on the FortiGate 50G model where the Node.js process exhibits uncontrolled RAM allocation without proper garbage collection. We present a comprehensive analysis of the problem and detail the implementation of an automated monitoring and mitigation solution using FortiOS's native automation framework.

Problem Statement

Environment

• Device: FortiGate 50G
• FortiOS Version: v7.4.8,build2795,250523 (GA.M)
• Affected Process: /bin/node –expose_gc /node-scripts/index.js
• Symptom: Progressive memory consumption without release

Issue Description

During routine monitoring, we identified that the Node.js process on the FortiGate 50G progressively consumes system RAM without implementing proper memory management. This behavior leads to:

• Gradual system performance degradation
• Risk of entering conserve mode (memory protection state)
• Potential service disruptions when memory usage exceeds critical thresholds
• No automatic memory reclamation by the process itself

Initial Analysis

Memory Baseline Investigation:

# System memory status
get system performance status | grep Memory:
Memory: 1964116k total, 1241904k used (63.2%), 472772k free (24.1%), 249440k freeable (12.7%)

# Process identification
fnsysctl ps | grep node
30708     0       0       S       /bin/node –expose_gc /node-scripts/index.js

Key Findings:

• The Node.js process runs with garbage collection exposure (–expose_gc flag)
• Despite the GC flag, automatic memory cleanup is insufficient
• Memory usage trends upward over time without natural reclamation
• Process restart is the only effective method to reclaim allocated memory

Root Cause Analysis

The investigation revealed that while the Node.js process includes garbage collection capabilities through the –expose_gc flag, the automatic memory management is inadequate for the specific workload patterns on the FortiGate 50G. This appears to be related to:

1. Memory Fragmentation: Long-running process creates memory fragmentation
2. Inefficient GC Triggers: Garbage collection thresholds may not be optimally configured
3. Memory Leak Patterns: Potential circular references or event listener accumulation
4. Platform-Specific Behavior: Issue appears isolated to the FGT-50G model

Solution Architecture

Design Requirements

1. Non-Disruptive Monitoring: Continuous memory surveillance without performance impact
2. Threshold-Based Action: Automated intervention only when memory usage becomes critical
3. Selective Process Management: Target only Node.js processes while preserving other services
4. Comprehensive Logging: Detailed audit trail for troubleshooting and optimization
5. Native Integration: Utilize FortiOS automation framework for reliability

Implementation Strategy

We developed a three-component automation solution using FortiOS's native automation framework:

1. Memory Monitor Script: Intelligent memory analysis with configurable thresholds
2. Scheduled Trigger: Regular execution intervals for continuous monitoring
3. Process Management: Safe Node.js process termination when criteria are met

Technical Implementation

Component 1: Memory Monitoring Script

# Automation Action: "Kill process node"
mem_line=$(get system performance status | grep Memory:)
execute log message "Memory check: $mem_line"
high_mem=0

# Pattern matching for memory thresholds (75%+)
echo "$mem_line" | grep -q '(7[5-9]' && high_mem=1
echo "$mem_line" | grep -q '(8[0-9]' && high_mem=1
echo "$mem_line" | grep -q '(9[0-9]' && high_mem=1

if [ "$high_mem" = "1" ]; then
    if fnsysctl ps | grep -q '/bin/node'; then
        execute log message "HIGH MEMORY + NODE DETECTED – KILLING"
        fnsysctl killall node
        execute log message "Node processes terminated"
    else
        execute log message "High memory but no node processes"
    fi
else
    execute log message "Memory usage normal"
fi

Component 2: Scheduled Trigger

# Automation Trigger: "Memory check every 1 hour"
config system automation-trigger
    edit "Memory check every 1 hour"
        set trigger-type scheduled
        set trigger-frequency hourly
    next
end

Component 3: Automation Stitch

# Automation Stitch: "Process node memory monitor"
config system automation-stitch
    edit "Process node memory monitor"
        set trigger "Memory check every 1 hour"
        config actions
            edit 1
                set action "Kill process node"
                set required enable
            next
        end
    next
end

Key Technical Features

Intelligent Threshold Detection

The script uses regex pattern matching to identify memory usage percentages:

• '(7[5-9]' – Detects 75-79% usage
• '(8[0-9]' – Detects 80-89% usage
• '(9[0-9]' – Detects 90-99% usage

Process-Specific Targeting

fnsysctl ps | grep -q '/bin/node'

This ensures only Node.js processes are affected, preventing accidental termination of other system processes that might contain "node" in their names.

Comprehensive Logging

All actions are logged to the automation category for audit trails:

• Memory status checks
• Threshold evaluations
• Process discovery
• Termination actions
• Success/failure states

Testing and Validation

Test Scenarios

1. Normal Operation: Memory usage below 75% threshold

   • Expected: Log entry "Memory usage normal"
   • Result: ✅ Confirmed

2. High Memory Without Node: Memory >75%, no Node.js process

   • Expected: Log entry "High memory but no node processes"
   • Result: ✅ Confirmed

3. High Memory With Node: Memory >75%, Node.js process present

   • Expected: Process termination and memory reclamation
   • Result: ✅ Confirmed

Monitoring Commands

# View automation logs
execute log filter category automation
execute log display

# Check automation status
diagnose test application autod 3 | grep -A10 "Process node memory monitor"

# Verify memory status
get system performance status | grep Memory:

Results and Benefits

Performance Improvements

• Proactive Memory Management: Prevents memory exhaustion scenarios
• Automated Response: Eliminates manual intervention requirements
• System Stability: Maintains optimal performance levels
• Resource Optimization: Ensures efficient memory utilization

Operational Benefits

• 24/7 Monitoring: Continuous surveillance without human intervention
• Audit Trail: Complete logging for compliance and troubleshooting
• Scalable Solution: Easily deployable across multiple FortiGate 50G units
• Non-Disruptive: Minimal impact on normal operations

Production Deployment

Configuration Verification

# Verify all components are properly configured
show system automation-action "Kill process node"
show system automation-trigger "Memory check every 1 hour"
show system automation-stitch "Process node memory monitor"

Monitoring Dashboard

# Real-time status check
diagnose test application autod 3 | grep "Process node memory monitor"

Expected output showing active monitoring:

stitch: Process node memory monitor (scheduled)
        next scheduled trigger: [Next execution time]
        actions:
                Kill process node: [Status]

Best Practices and Recommendations

Configuration Guidelines

1. Threshold Tuning: The 75% threshold provides optimal balance between proactive management and false positives
2. Execution Frequency: Hourly monitoring provides adequate responsiveness without excessive overhead
3. Logging Review: Regular analysis of automation logs helps identify patterns and optimize thresholds

Monitoring Recommendations

1. Trend Analysis: Track memory usage patterns over time to identify optimization opportunities
2. Alert Integration: Consider integrating automation logs with external monitoring systems
3. Documentation: Maintain clear documentation of any threshold adjustments

Security Considerations

1. Access Control: Ensure automation actions use appropriate administrative profiles
2. Change Management: Document all automation modifications for audit purposes
3. Backup Strategy: Include automation configurations in regular backup procedures

Future Enhancements

Potential Improvements

1. Dynamic Thresholds: Implement adaptive thresholds based on historical patterns
2. Graceful Restart: Investigate controlled restart mechanisms vs. process termination
3. Predictive Analytics: Develop memory usage trend analysis for proactive intervention
4. Multi-Model Support: Extend solution to other FortiGate models if similar issues emerge

Conclusion

The automated Node.js memory management solution successfully addresses the critical memory leak issue observed on FortiGate 50G devices. By leveraging FortiOS's native automation framework, we created a robust, reliable, and maintainable solution that:

• Proactively monitors system memory usage
• Automatically mitigates memory exhaustion scenarios
• Provides comprehensive logging for operational visibility
• Requires minimal administrative overhead
• Maintains system stability and performance

This case study demonstrates the effectiveness of using FortiOS automation capabilities to address platform-specific issues, providing a template for similar problem-solving approaches in network security environments.

The solution has been tested extensively and is ready for production deployment across FortiGate 50G installations experiencing similar memory management challenges.

Technical Specifications

Tested Environment:

• FortiGate 50G
• FortiOS [Version]
• Memory Configuration: ~2GB
• Node.js Process: /bin/node –expose_gc /node-scripts/index.js

Automation Framework:

• Native FortiOS automation (scheduled triggers)
• CLI script actions with system-level access
• Hourly execution frequency
• 75% memory threshold for intervention

Dependencies:

• fnsysctl command availability
• get system performance status functionality
• Super admin privileges for process management