S3 Logs

Collecting S3 logs using SQS

Site24x7's AppLogs now collects the log files stored in the S3 buckets in your Amazon environment and automatically sends it to Site24x7 for monitoring. We make full use of the SQS service to look out for events like a new log file addition and to retrieve log files. Below, you can learn more on how to monitor the logs in your S3 buckets.

Alternatively, you can also collect S3 logs using Lambda Function as described here.

Learn more about log management with Site24x7. 

Getting started

1. Log in to your Site24x7 account.
2. Download and install the Site24x7 Server Monitoring agent (Windows | Linux). 

Define the Log Type: 

A Log Type is a clear definition of the format in which an application writes logs. Different applications (such as IIS, Cassandra, Apache, MySQL) may write logs in different formats. Defining them as Log Types groups logs from different applications to simplify access and assist in efficient searching. Once you define a Log Type for your logs stored in your S3 bucket, list it under a Log Profile and start managing your logs by performing search queries. Learn how to create a Log Type.

Create a Log Profile: 

A Log Profile enables you to associate log types to a particular set of servers. Once a Log Profile is created, the corresponding logs are automatically added to the set of supported log types.   

To create a Log Profile, navigate to AppLogs > Log Profile > Add Log Profile, and follow the instructions below:

1. Profile Name: Enter a name for your Log Profile.

2. Log Type: Choose the Log Type of the S3 logs you would like to associate with this profile. 

3. Log Source: Choose Amazon S3 bucket as the log source.

4. AWS monitor: Choose an AWS monitor in which you've allowed Read permission for S3 buckets and SQS queue. Click on the + to associate a new AWS monitor.

5. S3 bucket name: Select the name of the S3 Bucket from which logs should be collected.

6. S3 Prefix: Mention the name of a folder if you've a specific folder from which the logs have to be collected.

7. SQS Queue name: Select the SQS Queue to which the message would be sent when a new log file is added.

8. Associate this log profile with these servers: Select the servers to which you would like to associate this log profile with and upload the logs from S3.

Please read on to learn about creating an SQS queue and providing access to Site24x7.

Creating an SQS queue:

1. Go to the AWS console and select SQS from the Services drop-down.

2. Create a New Queue or select an existing queue.

3. A default region will be selected automatically.The SQS queue needs to be in the same region as the S3 bucket.You can check the S3 bucket region as highlighted in the image below. You can also change the region of the SQS from the drop down menu.

Adding permissions to the SQS queue

1. After creating the queue, select it from the table, navigate to the Permissions tab and click on Edit Policy Document(Advanced).

2. An editor window will open.Here, you can copy and paste the JSON below:

{
"Version": "2008-10-17",
"Id": "PolicyExample",
"Statement":[
{
"Sid": "send-message",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "SQS:SendMessage",
"Resource":<SQS ARN>,
"Condition": {
"ArnLike": {
"aws:SourceArn":"arn:aws:s3:*:*:<S3 bucket name>"
}
}
}
]
}

You'll need to replace:

• <S3 bucket name> with your S3 bucket name.

• <SQS ARN> with the number found under the Details tab once you've highlighted the SQS queue.

Configure your S3 bucket to send events to the SQS queue

1. Select S3 from the Services drop down in the AWS console.

2. Select the bucket you put in the SQS policy, and choose the Properties tab.

3. Click on the Events tab under Advanced settings, choose Add notification, and select the field All object create events. You will also need to select SQS queue under the section Send to, and choose the name of the SQS queue to which the events are to be sent.

Grant permissions to Site24x7 to delete message from your SQS

1. After a message is read from the SQS, it would be considered best practise to delete it, else, the queue would be flooded with older messages.To delete data from your SQS queue, you will need to provide permission to Site24x7 by modifying the permissions for an existing user/role. To do this, go to your AWS dashboard and select IAM from the Security, Identity, & Compliance section:

2. From your IAM dashboard, select Users or Roles - which ever was used to configure the AWS account in Site24x7. 

3. If your AWS account in Site24x7 is configured with User, then select the user from the users list and under the Permissions tab, click on Add Inline Policies, and select click here. If you've configured the account with Role, select the appropriate role and follow the same steps as described above in this step. Choose the services, their corresponding actions and resources and in the editor window, give a name to your custom policy and paste the following:

{
"Version": "2012-10-17",
"Statement":[
{
"Sid": "DeleteMessage",
"Effect": "Allow",
"Action": "sqs:DeleteMessage",
"Resource": "<SQS ARN>"
}
]
}

Replace,

• <SQS ARN>: with the number found under the Details tab once you've highlighted the SQS queue.

4. Name the policy, for example, Site24x7-DeleteMessage-policy, and click Apply Policy.

Once you've applied the policy, you can go ahead and create a Log Profile for S3 and start monitoring it from the Site24x7 console.

Related log types

• S3 logs using Lambda Function
• CloudFront logs
• CloudTrail logs
• VPC Flow logs