Collecting VPC Flow Logs using the Lambda Function

VPC Flow Logs gives you information on the IP traffic to and from network interfaces in your VPC. This can be wielded as a security measure to monitor the traffic flowing to your instance. You can configure the VPC Flow logs to be published to Amazon S3 buckets from which Site24x7 collects it for monitoring. Learn more about log management with Site24x7.

Sample Flow Log Syntax

A flow log record is a space-separated string that has the following format:

<version> <account-id> <interface-id> <srcaddr> <dstaddr> <srcport> <dstport> <protocol> <packets> <bytes> <start> <end> <action> <log-status>

Create a Log Profile

To collect the VPC Flow logs you will first need to create a Log Profile. Navigate to Admin > AppLogs > Log Profile > Add Log Profile, and follow the instructions below:

1. Profile Name: Enter a name for your Log Profile.
2. Log Type: Choose VPC Flow logs. If you haven't enabled VPC Flow logs in your AWS account, please follow the instructions given here.
3. Log Source: Choose Amazon Lambda.
4. Timezone: Select a timezone for your logs.
5. Click Save.
6. Configure the Lambda function as described here.

Create a Flow Log to publish to Amazon S3 Bucket

• Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
• In the navigation pane, choose Your VPCs.
• Select one or more VPCs and then choose Actions, Create flow log.
• Filter: Specify the type of IP traffic data to log. Choose All to log accepted and rejected traffic, Rejected to record only rejected traffic, or Accepted to record only accepted traffic. Destination: Choose Send to an Amazon S3 bucket.
• For S3 bucket ARN, specify the Amazon Resource Name (ARN) of an existing Amazon S3 bucket. You can include a subfolder in the bucket ARN.

• For an existing user or role, ensure that Write permission is given for the Flow Logs. Else, create a new user or role and provide the necessary permissions.
• Choose Create.

Configure the Lambda Function

• Choose Lambda from the Services drop-down list, and choose Create Function. Select Author from scratch, define a name for the function, and choose Python 3.7 as the Runtime. 

• Permissions: You can choose an existing IAM role or create a new role from the AWS Policy Template. From the Policy Template drop-down select Amazon S3 Object Read-only permission, and enter a role name. You also have the option to create a new user role and extend permission to other services as well.
• Add triggers: Scroll down to choose S3 Bucket. Any log file added to the S3 bucket will be sent to Site24x7 by the Lambda Function. 
• Configure Triggers
  • Bucket:Enter the name of the S3 bucket from which logs will be collected.
  • Event type: Choose All object create events.
  • Click Add.
  In the window that opens, click on the Lambda Function as shown:
• Scroll to the editor, and place the code provided in the link below:

           https://github.com/site24x7/applogs-aws-lambda/blob/master/s3/s3-sender.py

• After entering the code, navigate to the Site24x7 web client, select Admin > Applogs > Log Profile, then select the created Log Profile, and copy the code that appears on the screen as the input for the variable logtypeConfig.

• Paste this code under Environment Variable with the field name logTypeConfig in the AWS console.

VPC Flow logs dashboard

AppLogs creates an exclusive dashboard for every Log Type, and shows a few widgets by default. Here's a list of the widgets available in the VPC Flow logs dashboard:

• Total Bytes Transferred
• Average Bytes Transferred
• Average Packets Transferred
• Network Action Stats
• Top IP addresses with highest number of rejected requests
• Top IP addresses using UDP transfer protocol
• Top 10 byte transfers by source
• Bytes transfers by source and destination IP addresses

Related log types

• CloudFront logs
• CloudTrail logs
• S3 logs