Configure HAProxy and Keepalived for vIP

Authors: Roman Diaz and Aleksandra Nowotna

This page describes how to install and configure HAProxy and Keepalived to get a virtual IP working with a Galera cluster (datanode/services nodes) and Emmett cluster (monitoring nodes) that are already running.

Galera cluster

Change the MySQL port

The first change we need to make is to change the port that MySQL is listening on. By default MySQL uses 3306, but haproxy will use port 3306, so we need to use another port for MySQL, such as 3307.

We need to make this change on all 3 services nodes.

1. Edit /etc/my.cnf.d/server.cnf

2. Add the following:

   [mysqld] port=3307

3. Restart the service with service mariadb restart.
   To quickly check that MySQL is listening on the right port, you can use this command:

   netstat -tunap | grep mysql

    

Create an haproxy user for each node

In MySQL, create an haproxy user for each node. You can create the users on one of the clusters and they will replicate on the other nodes.

To create the nodes use the following command format

The important thing is to create one user for each node with the IP of the node. For example:

You don’t need to give these users any privileges, haproxy will just use them to check the connection.

Install and configure haproxy

Do the following installation and configuration steps on each services node.

1. Use the following command to install haproxy:

2. Go to the following folder

3. Copy the default configuration file

4. Create a new configuration file called haproxy.cfg

5. Add the following configuration and save the file
   
   

   Here you can see that haproxy is listening on port 3306, which was previously used by MySQL and it uses a round robin algorithm to balance the three nodes.
   
   You must add the IP of the node that you are configuring, as well as the rest of the nodes, and for the port, when you are configuring a node, you need to add the local MySQL port, which is 3307, and for the other nodes, the haproxy port, which is 3306.
   
   

After you have configured all 3 nodes, on each node, start the service with the following commands

Check haproxy

One way to check that haproxy is running properly is to check a socket that the service creates on startup. To check the socket, we recommend the use of socat.

1. If you don’t have socat, install it:

2. Then run the following commands:

An example of the output from socat is given in the following screenshot.

Open

In this screenshot, there are 3 nodes and they are UP.

You can also run the following command:

Check that the command returns something different each time, which shows that it is balancing properly.

Install and configure keepalived

Do the following installation and configuration steps on each services node:

1. Install keepalived:

2. Go to the configuration folder

3. Copy the default configuration file

4. Create a new configuration file

The configuration file contents will vary across the nodes:

On the main node of the Galera cluster, add the following configuration for the main node for keepalived:

Here you need to specify the virtual IP, which in my example configuration is 10.10.203.220. Also, keepalived works with a main/secondary configuration, and this is the main configuration. Add this main configuration to the main node of the galera cluster, so it is also the main node for keepalived.

Then on each of the secondary nodes, make the following changes:

• router_id → Each node must have its own identifier. For example, the main node is LBL01, so the others can be LBL02, LBL03, and so on if you have more nodes.

• state → The main node has the MASTER state and the secondary nodes should have the BACKUP state.

• priority → The priority of the main node is 101, and on the secondary nodes it should be 100.

After you configure the nodes, on each node, activate and start the service:

To quickly check that you have configured everything correctly, run the following command

Open

Check the IP address assigned to your network interface

This should show that your network interface (e.g. eth0) has the correct IP assigned to it.

You can also run the following command from a server that is not one of the services nodes, but which is running a mysql client and which has access to the service nodes over the network:

As you can see, this is very similar to the command to check that haproxy is working, but this time we are connecting to the virtual IP, which is supplied by keepalived, to check keepalived.

Changes to API/UI

To make the Abiquo API use the virtual IP, log in to the server and do the following steps:

1. Go to the configuration folder:

2. Edit the api.xml file

3. Remove configuration that uses the services nodes:
   driverClassName="com.mysql.jdbc.Driver" url="jdbc:mysql://ip1.example.com:3306,ip2.example.com:3306,ip3.example.com:3306/kinton?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8"/>

4. Add the virtual IP configuration:
   driverClassName="com.mysql.jdbc.Driver" url="jdbc:mysql://"VirtualIP":3306/kinton?autoReconnect=true&useUnicode=true&characterEncoding=UTF-8"/>

5. Restart the tomcat service

Changes to Reporting

If you are using Abiquo Reporting (Install Abiquo reports | Configure Abiquo data source in Jasper reports server )

1. Repeat the step to run install.sh, but this time, send the virtual IP as the host value:

This will point JasperReports at the virtual IP and it won’t depend on one of the nodes in the cluster only.

Monitoring

This section describes a configuration with two monitoring nodes. The Abiquo servers access the Monitoring server through a virtual IP. It is possible to extend the configuration to use three nodes.

On both monitoring nodes:

1. Install haproxy with yum install haproxy -y

2. Backup default haproxy configuration file with:
   mv /etc/haproxy/haproxy.cfg /etc/haproxy/backup_haproxy.cfg

3. Create a new configuration file with: touch /etc/haproxy/haproxy.cfg

4. Edit the file and paste in the following configuration

5. Change the IP addresses in backend monitoring-cluster section to your monitoring nodes:

6. In the above configuration, haproxy is listening port to 36639 (bind), so open the firewall on this port with firewall-cmd --zone=public --permanent --add-port=36639/tcp and reboot the firewall service with: systemctl restart firewalld

7. When you have configured all of the above on all nodes, activate and start the service with:
   systemctl enable haproxy and systemctl start haproxy

8. To check if haproxy is running, use socket.

   1. Install it with yum install socat -y

   2. Run the following in the terminal (copy all 3 lines and paste them at the same time):

   3. The output should be similar to the following
      There are two nodes and their status is up.

9. Install keepalived with: yum install keepalived -y

10. Back up the configuration file with: mv /etc/keepalived/keepalived.conf /etc/keepalived/backup_keepalived.conf

11. Create a new configuration file with: touch /etc/keepalived/keepalived.conf

On the main monitoring node:

1. Copy and paste the following configuration, change the IP address in the virtual_ipaddress section to the virtual IP address in your environment.

On the second monitoring node:

1. Copy and paste the following configuration, change the IP address in the virtual_ipaddress section  to the one you have prepared (the same as on the main monitoring node in the keepalived configuration):
   
   

On both monitoring nodes:

1. Enable and start the service with: systemctl enable keepalived and systemctl start keepalived

2. Check if the configuration is correct with: ip addr sh eth0
   The output should be similar to the following.

    

3. If the firewall is active on both nodes, we have to add VRRP to the rules

4. Restart the firewall with systemctl restart firewalld

Check the failover

Check if the failover is working properly:

1. Open main monitoring node console from vCenter or another way that does not require a network connection on the interface configured for keepalived

2. Disable network interface assigned to keepalived with: ifconfig eth0 down

3. Check the logs on the main monitoring node if the state of the server has been updated properly:

   You should see something similar to this transition to a FAULT state:

4. On the second monitoring node check the logs with the same command:

   Here you should see similar transition from BACKUP state to MASTER:

5. Go back to main monitoring node console and enable the network interface back again with:

   Check the logs again and on the main node you should see transition back to MASTER state:

6. On the second node you should see it went to the BACKUP state as it was before:

API and Remote Services

On API server and Remote Services servers:

1. Edit abiquo.properties file at /opt/abiquo/config/abiquo.properties

2. Change the following properties to the Virtual IP address and haproxy listening port:
   
   

3. Restart abiquo-tomcat service: systemctl restart abiquo-tomcat

Related tutorials:

• https://www.thenoccave.com/2013/12/mariadb-galera-cluster-ha-proxy-keepalived-centos-6/

• https://fromdual.com/making-haproxy-high-available-for-mysql-galera-cluster