From Redis to KeyDB with Nginx as load balancer and .Net Core

Part - 3 Two KeyDB instances running in active replica mode behind a Nginx load-balancer and .Net Core

This will be a short series of articles describing what the research led to. Before continuing further let me first share what this series would be and what it would not be.

It would be a quick walkthrough a simple setup of an ASP.NET Core web API application that is going to use a single Redis instance for caching and we are going to replace it with two instances of KeyDB running in Active Replica setup behind a Nginx load balancer. Along the way I will share a hint or two that you might find useful. Our whole setup is for Windows OS.

• Part 1 - What we will do is to wrap up a quick web API application using Redis store in a Docker image.
• Part 2 - In the second part we are going to try to switch from Redis to KeyDB running in a docker container. After we make some tests and see how this works we are going to move from a single KeyDB instance to two instances in Docker containers.
• Part 3 - The third part would be to put the two KeyDB instances behind a load-balancer for which Nginx will come into aid.
• Part 4 - And the final fourth part would be to implement retry logic using Polly.

Nginx

There is a really simple and understandable information what actually Nginx is given by Wikipedia:

NginX, is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. The software was created by Igor Sysoev and publicly released in 2004. Nginx is free and open-source software, released under the terms of the 2-clause BSD license. A large fraction of web servers use NGINX, often as a load balancer.

In our series we will use it as a load-balancer. Load balancing is the distribution of traffic across multiple servers, the load balancer receives the traffic and distributes it to the servers based on different rules and criteria. In our case we will have a Nginx load balancer that will distribute the traffic from two KeyDB instances running in active replica mode.

Having such a distributed setup will help us with redundancy - in case of one of our cache databases goes down the other will take the load until the problem is solved. Having our KeyDB servers in active replica mode means that no matter which server receives the data first it will be replicated to the others too. When a server that has been down is restored back in the active replica mode it will be synced with the other servers in the setup.

Now that we have an idea of what we are going to implement we will start with setting up Nginx first. I would also like to mention that the whole setup including the Nginx is as we said in the beginning for Windows OS.

You can download the latest version for Windows from this link. After you have downloaded it and unzipped it in a folder of your choice navigate to that folder and start the application.

To start Nginx I will use the command prompt with elevated admin rights. First I will navigate to where the .exe file is and then start it.

C:\> cd C:\nginx-1.20.1
C:\nginx-1.20.1>start nginx

In my case it is located in C:\nginx-1.20.1. Keep in mind that by default it will try to start on port 80 and there is good chance that this port is already occupied by some other process. If that’s the case you may receive some warning like.

To check if the server has started is to try to locate it among the process by executing the following command:

C:\nginx-1.20.1>tasklist /fi "imagename eq nginx.exe"

If that is the case you can either stop whatever process occupies port 80 or navigate to C:\nginx-1.20.1\conf and modify nginx.conf file to some other port. I will use 8085.

After changing the port that the server listens on and saving the changes you can repeat the commands from above and your result will probably look something like this.

You will see two processes - the master process and the worker process. Detailed information could be found in the official documentation here.

After we have the server up and running we will start to configure it to play the role of a load balancer. One thing that we will need for the whole walkthrough to work is your IP address. To obtain it you can simply type the command ipconfig in the cmd and hit enter. Your IP will be the IPv4 Address address.

Once we have our IP address we will edit the nginx.conf file to make the setup we need.

A bit of explanation. Line 5 has to be stream, the default value is http and we have to change it. Line 6 in the upstream section are added the servers that will be load balanced, in our case the running in docker KeyDB instances which are port forwarded to local host - that is why we have the 127.0.0.1 which is localhost with the port that we are using in the containers. We are naming the stream myproject, you can choose whatever name you like. For line 11 where you will specify your IP address, it has to specific, something like localhost (as in this setup we are running Nginx locally) will not work, followed by the port number that we have chosen for Nginx. Line 12 proxy_pass receives the name of the upstream section (in our case myproject). And finally proxy_protocol off; is redundant but is left here to illustrate that the protocol must be off (it is off by default).

When the changes to the config file are implemented the Nginx server needs to become aware of them. We could stop and start it or what I will do is to use nginx -s reload command which will start a new worker with new configuration and will gracefully shut down the old one.

What we will do now is go to the application and will change the appsettings.json to point towards the load balancer, the IP plus the port 8085 in our case.

In the WeatherForecastController.cs we will modify the cache key to differentiate that it comes from the load balanced setup.

Again in the first call we see that the property cached is false.

We will reload the browser so that another call to the controller is made and this time the data will come from the cache.

We will also check the two KeyDB instances to verify that the cached data is propagated there.

In order to prove that our setup is redundant we will stop one of the KeyDB instances and restart it again. After restarting it we will have to instruct it to be active replica of the other KeyDB server.

First we will delete the cache that we just have added with the command DEL loadbalance-weather-forecast.

DEL loadbalance-weather-forecast

What will be our next steps

• Stop keydb-2 with docker stop keydb-2
• Call again the controller and check keydb-1 to verify that the data is cached
• Start keydb-2 and verify that it does not have the cache key loadbalance-weather-forecast
• Make keydb-2 again part of the replication and see the data update with the latest from keydb-1

There is one issue that we have to keep in mind here. When starting the application and trying to get the cache the load balancer makes a connection to the first instance in the upstream section of the config file if it is available. If not it will connect to the next available.

In our case we had the first connection up and running and it is keydb-1. We showed that stopping and starting keydb-2 works perfectly, but if we stop keydb-1 then we have a problem. The problem is that we no longer will be connected to the cache cluster.

If such a situation is not handled properly our application will throw an exception. Fortunately there is a easy way to tackle this problem with some retry logic. In the next article we will see how to handle this with Polly.

Part 4 - And the final fourth part would be to implement retry logic using Polly.