How to optimize Redis with JedisPool

17-May-2022 05:06 PM by vijayakumar

Jedis Pool Optimization

Do you love reading about how various performance-related issues are traced out and fixed? If so, here's our story of how we fixed the Redis performance issues we were facing by optimizing JedisPool. We hope this will help you strengthen your IT environment.

Our Redis performance issues and efforts to find the cause

A spike in the response time

Site24x7 uses Redis in-memory data structures to reduce hits to our back-end databases. We started finding occasional spikes in our application response time. Upon investigation, we found that the Redis cache calls were failing, and the servlets started hitting the database, which led to a spike in the overall application response time. This happened randomly for a few seconds, then the application returned to normal with zero Redis failures.

Redis BigKeys

As our traffic to Redis was almost constant throughout the day, we ruled out scaling-related issues. Then, we started monitoring the Redis performance metrics and found that whenever there was a drop in Redis connections, we observed a high CPU alert. We were then able to correlate high CPU usage with the Redis failures, but then the reason for the high CPU usage remained a mystery.

Our suspicions pointed towards the BigKeys and CPU-intensive commands such as the SMEMBERS key. The Redis command line provides an option to scan for the BigKeys (redis-cli --bigkeys) and view the list of BigKeys based on data types. As the next step to fix the issue, we deleted the BigKeys greater than 1MB. We also tried using the Redis Slow Log to list all the commands that could slow down the system, but there were no CPU-intensive commands listed.

Redis Performance Issue

Jedis connections

At this juncture, we were a bit puzzled, as we couldn't find any BigKeys, slow commands in the Slow Log, Redis latency, or any issues in Redis response time. But, we still observed CPU spikes and dropped connections. As creating new connections is an I/O-intensive operation, we started observing the pattern of new connections. To our surprise, we found that the number of connections created per second was equal to the number of commands executed. We were surprised by this revelation because we were already using JedisPool, a collection of reusable Jedis connections, to reuse the connections and had not changed our JedisPool configuration.

Upon further research, we learned that JedisPool offers many options for setting the configuration based on the application's needs. But we were using the default configuration. This worked fine when we had fewer application servers. As we scaled up the application servers, the default configuration resulted in the inefficient usage of JedisPool.

What is Jedis?

Jedis is one of the most commonly used Java client libraries for Redis and also one of the recommended clients on the official Redis client list. You can download the latest build here.

Since individual Jedis instances aren't thread-safe, using them in a multi-threaded environment will lead to various errors or inconsistencies. Also, creating a new connection for every Redis query isn't an ideal option as that will create an I/O overhead, leading to performance issues. To overcome these challenges, we decided to try JedisPool, a collection of thread-safe Jedis instances.

Low maxIdle connections

By default, both the maxPoolSize and maxIdle values related to JedisPool are 8. As our product grew, we needed more connections, so we increased maxPoolSize to 150 while maxIdle remained unchanged. So with this configuration, JedisPool could create a maximum of 150 connections at any given time. Still, it could only hold 8 idle connections in the pool at a time, and the remaining connections were dropped.

This nullified the purpose of using JedisPool, as new connections were created every second. This occurred from every single application server. Creating connections increased the Redis CPU usage, which in turn resulted in Redis dropping the connections. As all fingers pointed towards high CPU usage in Redis clusters due to a high number of connections to Redis per second as the major reason, we zeroed in on JedisPool optimization, which can be done by fine-tuning the JedisPool configuration.

JedisPool configuration

JedisPool configuration is based on GenericObjectPoolConfig from Apache Commons Pool 2. By setting the GenericObjectPoolConfig parameter to a particular value, Redis performance tuning can be achieved. The following sample shows how to initialize and use JedisPool:

JedisPool jedisPool = new JedisPool(jedisPoolConfig, redisHost, redisPort, timeout);

try (Jedis jedis = jedisPool.getResource()) {

     //Execute necessary commands

    //jedis.set("foo", "bar");

}catch (Exception e) {

    LOGGER.log(Level.SEVERE, e.getMessage(), e);
}

Setting proper parameters for JedisPoolConfig is crucial, as this determines how a new connection will be created.

JedisPool optimization through tweaking critical parameters

Tweaking the connection pool parameters based on the load and other environment-related factors is the best way to achieve better performance and resolve performance issues. Here are a few of the crucial parameters that we tried tweaking:

maxTotal

The maxTotal parameter specifies the maximum number of connections that can be created from the pool at any given time. Since Jedis instances are single-threaded, this setting will affect the concurrent requests between your application and Redis.

The default value is 8, which is probably too low for a heavy application. On the other hand, setting a high value for this parameter will lead to CPU and memory overhead. It is crucial to figure out the maxTotal value that is suitable for your environment. Here are a few major pointers to consider when identifying your ideal maxTotal value:

Determine the expected number of concurrent connections to Redis.
Zero in on an average response time to execute a Redis command.
Set the maximum number of connections supported by a Redis server.

By default, version 2.6 of the Redis server supports 10,000 client connections. The product of the number of application instances and maxTotal must always be lower than the maximum number of connections supported by Redis.

Let's consider the following example:

Maximum client connections supported by a Redis server = 10,000
Average response time of a Redis query = 2 ms
Expected number of queries per second (QPS) per Redis instance = 30,000

The number of Redis queries handled by a single Jedis instance per second is calculated by dividing one second by the response time, which in this scenario = 500 (1,000 ms / 2 ms = 500).

The required maxTotal is calculated by dividing the expected QPS by the number of queries handled by a single Jedis instance per second. Thus, maxTotal = 60 (30,000 / 500).

This is a theoretical value. You can use a monitoring tool like APM Insight to obtain the metrics mentioned above. It is good to reserve a few extra connections for an unpredictable load.

It would be best to keep in mind that the total number of connections reserved across all instances should not exceed the maximum number of connections supported by Redis. For example, if you have 100 instances, the total number of connections would be 6,000 (100 * 60), which is less than 10,000. Whereas if you have 200 instances, the total would be 12,000 connections, which might cause your commands to fail.

maxIdle

The maxIdle parameter specifies the maximum number of idle Jedis instances that will be maintained in the pool without being evicted from it. This is the actual number of connections required by the application. The maxTotal value includes the total number of maxIdle plus the surplus connection required for an unpredictable load.

If the value of maxIdle is too small for your application, then new Jedis connections will be created frequently to serve the requests. Creating new connections frequently will increase the CPU usage of the Redis cluster and will negatively impact the application. If the value is too large for your application, the reserved connections will be wasted, and other application instances may not get enough resources to communicate with Redis.

By setting maxIdle to a proper value, you can ensure that maxTotal achieves the best performance in most scenarios since no new connections will be created once the pool size reaches maxIdle.

minIdle

The minIdle parameter specifies the minimum number of idle connections that will always be maintained in the pool. The default value is zero, which means all the connections that are idle for a period longer than minEvictableIdleTimeMillis will be evicted from the pool.

So when the load surges, many new connections will be created, which may affect the application's performance. We can avoid this by setting a higher minIdle value depending on the load. For applications with fluctuating load, a minimum of (maxIdle / 2) should be set, and it can go up to maxIdle.

blockWhenExhausted

This Boolean setting specifies whether the client should wait for a free connection once the pool is exhausted. The default value is true. Only when the blockWhenExhausted value is true will the maxWaitMillis parameter be effective.

maxWaitMillis

This parameter specifies the maximum time that the client must wait when no connection is available. After this, an exception error message saying "Could not get resource from pool" will be displayed. This setting will take effect only when blockWhenExhausted is set to true. The default value is -1 ms, which means it will wait indefinitely for connection from the pool. We recommend setting the same value as your Jedis connection timeout.

minEvictableIdleTimeMillis

This parameter specifies the minimum amount of time a connection can be idle before eviction. The default value is 60 seconds, which is suitable for most applications. You can alter this value based on your environment and load.

The values provided here will suit most applications. However, it is best to monitor the actual usage and decide the values for these parameters. You can either depend on any of the free monitoring tools available on the market, like Site24x7, or log the JedisPool usage periodically to understand the load and then determine the proper values.

I hope our Redis optimization suggestions will be of use to you. We'd love to hear your feedback. Please watch this space for more blogs and tips from Site24x7 engineers

Comments (0)