Enhancing Security in AWS EKS with File Integrity Monitoring

Why?

At CTO2B, we understand that security and stability are paramount for organizations striving to maintain their competitive edge and protect their assets. As a company constantly seeking better, more secure, and automated infrastructure management solutions, we recognize that the cost of security breaches and disruptions can be staggering.

According to a study by IBM, the average price of a data breach in 2024 was around  $5.17 million [link], which continues to rise annually. Furthermore, incidents like the ransomware attack on Colonial Pipeline [link], which resulted in a $4.4 million payout, highlight the potential financial and reputational damage caused by security vulnerabilities.

To mitigate such risks, organizations must adhere to stringent security frameworks such as WASP (Web Application Security Project), ISO 27001, SOC (Service Organization Control), and PCI DSS (Payment Card Industry Data Security Standard). These frameworks provide guidelines for securing infrastructure and ensuring compliance with best practices.

One of the more challenging requirements to implement across these frameworks is File Integrity Monitoring (FIM). FIM is a critical security process that involves monitoring designated files—such as operating system files, configuration files, data files, and application files—for changes that may indicate a cyber attack. This monitoring process helps detect unauthorized modifications, which could signify a breach or malware infection.

Ensuring effective file integrity monitoring can be particularly complex in AWS Managed Elastic Kubernetes Service (EKS). EKS is designed to simplify the management of Kubernetes clusters, but this abstraction can sometimes obscure direct access to underlying EC2 instances. Consequently, implementing FIM in these instances takes a lot of work, requiring additional strategies and tools to ensure compliance and security.

In this blog post, we will explore the necessity of file integrity monitoring in AWS EKS, the challenges involved, and how to effectively implement these checks to enhance the security of your infrastructure.

How

File Integrity Monitoring (FIM) Concept

File Integrity Monitoring (FIM) is a security process that involves continuously observing designated files for changes that could indicate a cyber attack. This includes monitoring operating system files, configuration files, data files, and application files. The primary objective of FIM is to detect unauthorized modifications, which could signify a breach or malware activity.

FIM is typically implemented using either agent-based or agentless solutions. These solutions create a baseline of the files’ state, which includes capturing the file hash digests. Over time, the system compares current file hashes against the baseline to identify any discrepancies or changes.

• Agent-Based Solutions: Agent-based FIM solutions involve installing software agents directly on the systems that require monitoring. These agents have direct access to file systems and can efficiently track real-time changes. They provide detailed insights and granular control over which files to monitor, allowing for tailored configurations specific to an organization’s needs.
• Agentless Solutions: Agentless FIM solutions operate without installing software on the target systems. Instead, they leverage network protocols and APIs to remotely access and monitor file systems. While this can simplify deployment and reduce system overhead, it may introduce limitations in real-time monitoring capabilities and granular control.

Despite the complexities involved, FIM remains a vital component of a comprehensive security strategy, providing early detection of potential threats and unauthorized changes.

AWS Config and AWS Systems Manager (SSM)

Two robust services in AWS that facilitate file integrity monitoring and compliance are AWS Config and AWS Systems Manager (SSM). These services provide a comprehensive framework for tracking resource changes and maintaining security and governance across your infrastructure.

• AWS Config: is a managed service that monitors AWS resource changes and configurations for security and compliance. It continuously audits resource configurations, compares them against internal rules, and stores the data in an S3 bucket for analysis. However, AWS Config struggles with Infrastructure as a Service (IaaS) offerings, like EC2 instances or on-premises environments. To overcome this, it uses the SSM Agent to gather data from these environments.
• AWS Systems Manager (SSM): AWS Systems Manager offers a comprehensive suite of security, configuration, and fleet management tools. Within this suite, the Systems Manager Inventory is the specific service that aids file tracking. It provides a snapshot of the current state of monitored file systems, capturing vital information about the system’s configuration and inventory.

The primary limitation of the Systems Manager Inventory is its focus on providing a current snapshot rather than tracking changes over time. This is where AWS Config becomes instrumental. By integrating these services, organizations can achieve a more holistic view of file integrity:

• SSM Agent: The SSM agent continuously collects file metadata from its installed system. It bridges the gap between real-time file monitoring and historical change tracking.
• Systems Manager Inventory: The collected metadata is sent to the Systems Manager Inventory, which provides a visualization of the current state of the monitored file systems. This gives a detailed view of the present configuration and helps manage the system inventory efficiently.
• AWS Config: Simultaneously, the SSM agent sends the same metadata to AWS Config, which tracks changes over time. This enables organizations to monitor file modifications continuously, ensuring that any unauthorized changes can be detected promptly.

By leveraging AWS Config alongside AWS Systems Manager, organizations can implement a robust file integrity monitoring solution that offers real-time visibility and historical change tracking, enhancing their security and compliance posture.

Implementation

Implementing file integrity monitoring in an AWS EKS environment using AWS Config and AWS Systems Manager involves several key steps. Here, we’ll walk through the process of enabling these services to monitor file changes effectively and extend the functionality using a custom AWS Config Lambda rule.

Step 1: Enable AWS Config

The first step is to enable AWS Config to track changes in file data. AWS Config can be configured to collect metadata about file changes through the Integration of Systems Manager (SSM).

1. Access AWS Config: Log in to your AWS Management Console and navigate the AWS Config service.
2. Create a Configuration Recorder: Ensure you have a configuration recorder set up. This component records configuration changes to your AWS resources.
3. Specify SSM: When setting up AWS Config, specify that it collects data using the SSM:FileData attribute. This setting ensures that AWS Config will gather metadata about files the SSM agent monitors.
4. Set Up Rules and Delivery Channel: Define rules in AWS Config to evaluate file data changes and set up a delivery channel to specify where the configuration history and snapshots should be stored, typically an S3 bucket.

For detailed instructions, refer to the official AWS guide on Setting Up AWS Config with SSM Data Collection.

Step 2: Enable SSM Inventory

Once AWS Config is set up, you must configure the SSM Inventory to collect data specifically from the /etc directory, which often contains critical configuration files.

1. Access AWS Systems Manager: In the AWS Management Console, navigate to the AWS Systems Manager service.
2. Create an SSM Inventory: You can create an inventory by choosing the “Inventory” option under the Systems Manager’s “Fleet Management” section.
3. Define Inventory Policy: Define a policy that specifies the files and directories to be inventoried. Ensure that the policy includes the /etc directory for file integrity monitoring.
4. Deploy the SSM Agent: Ensure that the SSM agent is installed and running on all EC2 instances in your EKS cluster. The agent will collect and send file data from the specified directory to the Systems Manager Inventory.
5. Schedule Inventory Collection: Set a schedule for how frequently the inventory should be collected. This can help maintain an up-to-date view of the file system state.

Refer to the AWS Systems Manager Inventory Setup Guide for a step-by-step guide.

Step 3: Create a Custom AWS Config Lambda Rule

To enhance monitoring capabilities, you can create a custom AWS Config Lambda rule. This rule allows you to define specific logic for evaluating resource configurations.

1. Create a Lambda Function
   • Access AWS Lambda: In the AWS Management Console, navigate to the AWS Lambda service.
   • Create a New Function: Click “Create function” and choose the “Author from scratch” option.
   • Configure Function Settings: Provide a name for your function, such as `FileIntegrityCheckLambda`. Choose Python as the runtime environment and configure any necessary IAM roles to allow the function access to AWS Config data.
2. Deploy Lambda Code

Use the following Python code to define your Lambda function logic. This code will be executed each time the AWS Config rule is triggered.  

import json
import boto3

def lambda_handler(event, context):
    # Parse the AWS Config event
    invoking_event = json.loads(event[‘invokingEvent’])
    configuration_item = invoking_event[‘configurationItem’]
    configuration_item_diff = invoking_event[‘configurationItemDiff’]

    resource_id = configuration_item[‘resourceId’]

    if configuration_item[‘resourceType’] == ‘AWS::SSM::FileData’ and configuration_item_diff:
        if configuration_item_diff[‘changeType’] == ‘UPDATE’:
            changes = []

            for n, item in configuration_item_diff[‘changedProperties’].items():
                if ‘updatedValue’ in item:
                    for k,v in item[‘updatedValue’].items():
                        changes.append(“%s/%s” % (v[‘InstalledDir’], k))

            files = “, “.join(changes)
            message = (
                f”File change detected:\n”
                f”Instance ID: {resource_id}\n”
                f”Files: {files}”
            )

            # @TODO Implement handling of notification
            print(message)

    return {
        ‘statusCode’: 200,
        ‘body’: json.dumps(‘File modification check completed’)
    }

3. Enable the Custom AWS Config Rule
   • Access AWS Config: Return to the AWS Config service and navigate to the “Rules” section.
   • Create a Custom Rule: Click “Add rule,” select “Add custom Lambda rule,” and provide the necessary details. Assign a name and description and specify the Lambda function you created.
   • Set Trigger Conditions: Configure the rule based on specific events or changes detected in the file metadata collected by SSM.
   • Monitor Rule Execution: Once the rule is active, AWS Config will evaluate changes according to your Lambda function logic, and the results will be displayed in the AWS Config dashboard.

By following these steps, you can effectively integrate file integrity monitoring into your AWS EKS environment, leveraging AWS Config and SSM to ensure compliance with security policies.

Conclusion

Implementing file integrity monitoring in AWS EKS using AWS Config and AWS Systems Manager provides a powerful mechanism to detect unauthorized changes and enhance security. By leveraging these AWS services, you can ensure continuous compliance with security frameworks and protect your infrastructure from potential threats.

With the custom AWS Config Lambda rule, you can easily define the logic to evaluate file changes and configure alerts for any modifications detected across your EKS node fleet. This approach allows for flexibility in monitoring and provides the capability to respond swiftly to potential security incidents.

However, manually setting up and managing these configurations can be time-consuming and complex, especially in large-scale environments. This is where the CTO2B DevOps automation platform comes into play. Our platform offers these capabilities, streamlining the process and reducing the operational overhead of manual setup.

Utilizing the CTO2B platform allows you to take advantage of automated configuration, monitoring, and alerting for file integrity across all your EKS nodes. This ensures that your infrastructure remains secure and compliant with minimal effort, allowing your team to focus on strategic initiatives rather than operational tasks.

In conclusion, integrating file integrity monitoring in your AWS EKS environment is essential for maintaining security and compliance. By leveraging AWS services and the CTO2B DevOps automation platform, you can easily and efficiently achieve robust monitoring and protection for your infrastructure.