Hashing and Masking Log Data

Hashing and masking help you hide sensitive data while sending your logs to Site24x7 AppLogs.

Masking log data

Before sending your logs to Site24x7, you can use a mask rule to hide sensitive information and prevent it from being sent to Site24x7. You have to configure the expressions that you want to mask as a capture group in the regex. You can provide a mask string or choose to use the default (***).

The log license is calculated after applying the masking or hashing filter. If the masking reduces the size of the log, the smaller size will be used when tracking the total size of the logs scanned.

Consider the below example of an Apache access log:

209.85.238.199 - - [26/Oct/2021:10:05:15 +0000] "GET presentations/details?apiKey=877avjkj329082j30sf83s1&type=ppt HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)"

Here, you want to mask the apiKey parameter value (877avjkj329082j30sf83s1) in the request URI.

You can use apiKey=(.*)& expression so that the log that's forwarded to Site24x7 will look like:

209.85.238.199 - - [26/Oct/2021:10:05:15 +0000] "GET presentations/details?apiKey=***&type=ppt HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)"

If you want to mask complete data in the field, then use (.*) in the expression.
Do not unnecessarily match on more logs than needed. For example, don't use the following expression as it matches more than necessary:
GET\s[\w\/]*\?apiKey=(.*)&
You can specify multiple capture groups. Note that if multiple capture groups are specified in one filter, each value will be masked in the same way. So if you create one filter for users' email addresses and mobile numbers, both will be replaced with the same mask string.

Hashing log data

Hashing is similar to masking; however, the expression is replaced with a MD5 hash code, and the data is completely hidden before being sent to Site24x7. Each unique value will have a unique hash code. You have to configure the expressions that you want to hash as a capture group in the regex.

For example, consider the log line:

209.85.238.199 - - [26/Oct/2021:10:05:15 +0000] "GET user/details?apiKey=877avjkj329082j30sf83s1&email=david@zylker.com HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)"

Here, you want to hash the email parameter value (david@zylker.com) in the request URI.

You can use &email=(.*)\s expression and the log that's forwarded to Site24x7 will look like:

209.85.238.199 - - [26/Oct/2021:10:05:15 +0000] "GET user/details?apiKey=877avjkj329082j30sf83s1&email=706e02761ac7b8e758695db3a69e2fc1 HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)"

In addition, you can specify multiple match groups. If multiple match groups are specified, each value will be hashed uniquely.
Expression: &email=(.*)&mobile=(.*)&

Hashing and logging from the Site24x7 web client

To apply hash and mask rules to your log data, follow the steps below:

Log in to Site24x7.
Go to Admin > AppLogs > Log Type.
Click on the desired log type.
Go to Field Configurations from the Edit Log Type screen that pops up.
Enable Masking: Toggle to Yes to enable masking. Provide the mask expression for the data to be masked as a capture group in the regex and the mask string.
Enable Hashing: Toggle to Yes to enable hashing. Provide the hash expression and include the data to be hashed as a capture group in the regex.
Click Apply.
Click Save.

Site24x7 will start to receive hashed and masked log data as per your configurations.

Masking and hashing logs