Derived Field Support

A log event contains varied fields in different formats. Using Site24x7 AppLogs derived field support, you can create custom parsing rules for log fields to extract useful information. For example, log data can contain mixed messages, and you may want to pull some information from the message data. In this case, you can create a derived field by adding a RegEx rule to the message data to extract the necessary information.

Getting started

1. Log in to your Site24x7 account.
2. Download and install the Site24x7 server monitoring agent (Windows | Linux).
3. Go to Admin > AppLogs > Log Types > Add Log Type.

You can refer to this Adding a Log Type help document for more information.

Example: Extracting the domain and SSL time from the message field

In the following sample logs, the first log line contains a message that has a domain name and an SSL handshake time.

Sample log

[Thu Aug 12 14:52:23 IST 2022|DEBUG|39]:SSL Handshake Time https://zylker.com :42

[Thu May 12 14:37:51 IST 2022|DEBUG|30]: Object.wait break after up notification received...

[Thu May 12 14:37:51 IST 2022|DEBUG|42]: Up Notification received... breaking the waiting thread....

Here is the corresponding log pattern for the sample log above:

Log pattern

[$Datetime:date:EEE MMM dd HH:mm:ss z yyyy$|$LogLevel$|$Time$]:$Message$

Using derived field support, you can create custom parsing rules to derive the domain name and the SSL handshake time from the message field of the log data.

Steps to create derived fields

1. In the Sample Output table, hover over the Message field name to find the   icon, and click it. You can also click the Edit Field Configurations link.
2. In the Message - Field Configurations window that opens, fill in the Derived Fields section on the right side. If you have clicked the Edit Field Configurations link, choose the required field from the left pane and fill in the Derived Fields section.
   1. Display Name: Enter the display name.
   2. RegEx: Add a customized parsing rule to parse the field. In this example, we're adding the following RegEx rule: SSL\sHandshake\sTime\s(?<Domain>.*)\s:(?<SSLTime>\d+)
   3. The customized field names (Domain and SSLTime) are automatically added and displayed on the screen. Choose the correct data types for the customized field. Currently, AppLogs supports string and number data types. You can choose different units, such as percent, IOPS, counts, bytes, or kilobytes, from the drop-down next to the Number drop-down.
   4. Click Add Rule to add the custom parsing rule.
   5. You can add, edit, or delete rules.
3. Click Apply to apply the field configuration settings.
4. Click Save, and associate it to a Log Profile

Example: Extracting the compressed and actual lengths of a ZIP file from the message field

The following sample log contains a message about extracting a ZIP file of varied lengths.

Sample log

[Thu Aug 12 14:52:23 IST 2022|DEBUG|39]: HttpConnection : After successful gzip extraction downloaded response(22015) was extracted to 80132 of content-length -1

Here is the corresponding log pattern for the sample log above:

Log pattern

[$Datetime:date:EEE MMM dd HH:mm:ss z yyyy$|$LogLevel$|$Time$]:$Message$

Follow the steps mentioned above to create a derived field, and use the RegEx rule below to get the Compressed and ActualLength fields.

RegEx rule

HttpConnection.response\((?<Compressed>.*)\)\swas\sextracted\sto\s(?<ActualLength>\d+).

The customized field names (Compressed and ActualLength) are automatically added and displayed on the screen.

Example: Extracting a user email and IP address from the message field

The following sample log contains a message that has the user's email address and IP address.

Sample log

Jun 7 09:22:38 10.8.0.36 : %ASA-4-722051: Group <GroupPolicyVPNCert> User <demo123@gmail.com> IP <127.0.0.1> IPv4 Address <127.0.0.1> IPv6 address <::> assigned to session

Here is the corresponding log pattern for the sample log above:

Log pattern

$Datetime:date$ $MachineIp$ : $Message$

Follow the steps above to create a derived field, and use the RegEx rule below to get the user and IP fields.

RegEx rule

User\s<(?<User>.*)>\sIP\s<(?<IP>.*?)>
The customized field names (User and IP) are automatically added and displayed on the screen.

You can add more than one rule for the same field. For example, you can combine all the rules above in the same message field.

In the dashboard below, you can see how the derived fields (ssltime and compressed) extract and visualize meaningful information (Maximum SSL Time, Average Download Size, Average SSL Time, and Maximum Compressed). 

You can use the query below to get the logs with an ssltime greater than 100.
logtype="EumDemo" and message_ssltime>100

While querying the derived field, you should use the fieldname_derivedfieldname format to query the data. For example, message_ssltime represents the message field name and the ssltime derived field name in the query above.

Below is a sample query that uses the group by field on the derived field.
logtype="EumDemo" groupby message_domain

Related links

AppLogs saved search
AppLogs alerts
Multiple log pattern support