Upload Logs from Fluentd

Fluentd is an open-source log collector that helps you build a unified logging layer. You may have different log types in your application infrastructure—with Fluentd, you can consolidate various log types under a single roof and track them easily.

You can push your logs using Fluentd to Site24x7 AppLogs to track, analyze, and manage various log types. Follow the steps below.  

Table of contents

• Install the Site24x7-Fluentd output plugin
• Create a Log Profile in Site24x7
• Match and send events to Site24x7 AppLogs
• Troubleshooting instructions

Install the Site24x7-Fluentd output plugin

• Open the Command Prompt and install the plugin using the below command if you are using FluentD agent:

  $ gem install fluent-plugin-site24x7

• Use the below command if you are using the td-agent:

  $ /usr/sbin/td-agent-gem install fluent-plugin-site24x7

Add a Log Profile in Site24x7 AppLogs

1. Log in to your Site24x7 account > Admin > AppLogs > Log Profile and click Add Log Profile in the top-right corner.
2. Click the Choose the Log Type drop-down to select your log type (example: Apache Access logs). The default log pattern will be applied. You can also add a custom log pattern.
3. Choose Log Collectors as the Log Source.
   
   
4. Click Save. 
5. After saving your log profile, navigate to Admin > AppLogs > Log Profile, and select your newly created Log Profile from the list.
6. Copy the input variable text given and enter it in the log collector after the parameter log_type_config.

Match Events to Site24x7 AppLogs

1. After installing the plugin and creating a log profile in Site24x7 AppLogs, add the below configuration in the Fluentd configuration file to match events and send them to Site24x7: Note: If you had installed the td-agent, you can locate the configuration file in this path: /etc/td-agent/td-agent.conf

   # Match events tagged with "site24x7.**" and send them to Site24x7 <match site24x7.**> 
   @type site24x7
   @id site24x7_agent
   log_type_config 
   #Paste the key copied from Site24x7 AppLogs in the <your_log_type_config> 
   
   <buffer>
   @type memory
   flush_thread_count 4
   flush_interval 3s
   chunk_limit_size 5m
   chunk_limit_records 500
   </buffer> 
   </match> 
   
    
   

   Example:
   
2. Optional parameters:
   

   Property	Description	Default Value
   max_retry	Number of times to resend failed uploads.	3
   retry_interval	Time interval to sleep between retries with exponential increase on each retry.	2 seconds
   http_idle_timeoutthis	Timeout (in seconds) during which HTTP-persistent connection will stay open without traffic.	5 seconds
   http_read_timeout	Timeout (in seconds) during which the socket connects until the connection breaks.	30 seconds
   http_proxy	Your proxy URL Optional HTTP proxy: http_proxy 'http://user:passs@mproxy.host:proxy.port'	nil

3. Restart the Fluentd agent to push events tagged with Site24x7 to Site24x7 AppLogs

Troubleshooting instructions:

I don't see logs from Fluentd in Site24x7 AppLogs. How do I fix this?

If you are not receiving the logs in Site24x7, it could be because:

• The entered log pattern in the configured log profile is incorrect. 
• Some additional parameters have been pushed that weren't present in the configured log pattern.

To troubleshoot these issues, follow the below given instructions:

1. Enable debug logs in Fluentd. You can add the following entry to the Fluentd configuration file and restart the Fluentd agent: 
   <system>
   log_level debug
   </system>
2. Check for the "/var/log/td-agent/td-agent.log" log file using the message "pattern not matched" and identify which log types you have not seen data for in Site24x7 AppLogs. 
3. Reconfigure the Log Profile for the identified log types, and update the log_type_config value in your Fluentd configuration file.

If you run across any other errors, kindly contact support@site24x7.com with the error logs.