Configure HAProxy and Keepalived for vIP

Authors: Roman Diaz and Aleksandra Nowotna

This page describes how to install and configure HAProxy and Keepalived to get a virtual IP working with a Galera cluster (datanode/services nodes) and Emmett cluster (monitoring nodes) that are already running.

HAproxy is a load balancer and keepalived provides a virtual IP for failover functionality

Galera cluster

The services nodes are a configuration of 3 servers that are each running mariadb, rabbitmq, redis, and galera

Change the MySQL port

The first change we need to make is to change the port that MySQL is listening on. By default MySQL uses 3306, but haproxy will use port 3306, so we need to use another port for MySQL, such as 3307.

We need to make this change on all 3 services nodes.

Edit /etc/my.cnf.d/server.cnf
Add the following:
```
[mysqld]
port=3307
```
Restart the service with service mariadb restart.
To quickly check that MySQL is listening on the right port, you can use this command:
```
netstat -tunap | grep mysql
```

Create an HAproxy user for each node

In MySQL, create an HAproxy user for each node. You can create the users on one of the clusters and they will replicate on the other nodes.

To create the nodes use the following command format

CREATE USER 'haproxy'@'nodex.example.com ';

The important thing is to create one user for each node with the IP of the node. For example:

CREATE USER 'haproxy'@'10.10.203.215';
CREATE USER 'haproxy'@'10.10.203.216';
CREATE USER 'haproxy'@'10.10.203.217';

You don’t need to give these users any privileges, HAproxy will just use them to check the connection.

Install and configure HAproxy

Do the following installation and configuration steps on each services node.

Use the following command to install HAproxy:
```
yum install haproxy -y
```
Go to the following folder
```
cd /etc/haproxy/
```
Copy the default configuration file
```
mv haproxy.cfg haproxy.cfg.bak
```
Create a new configuration file called haproxy.cfg

Add the following configuration and save the file

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats.sock mode 600 level admin
    stats timeout 2m

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    tcp
    log                     global
    option                  httplog
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout check           10s
    maxconn                 3000

listen haproxy-monitoring *:8090
  mode    http
  stats   enable
  stats   show-legends
  stats   refresh           5s
  stats   uri               /
  stats   realm             Haproxy\ Statistics
  stats   auth              monitor:AdMiN123
  stats   admin             if TRUE

#---------------------------------------------------------------------
# frontend galera-cluster
#---------------------------------------------------------------------
frontend haproxy-galera-1
    bind             *:3306
    default_backend  galera-cluster

#---------------------------------------------------------------------
# round robin balancing galera-cluster backend
#---------------------------------------------------------------------
backend galera-cluster
    balance     roundrobin
    server  NodeServices1 ip1.example.com:3307 check
    server  NodeServices2 ip2.example.com:3306 check
    server  NodeServices3 ip3.example.com:3306 check

Here you can see that haproxy is listening on port 3306, which was previously used by MySQL and it uses a round robin algorithm to balance the three nodes.

You must add the IP of the node that you are configuring, as well as the rest of the nodes, and for the port, when you are configuring a node, you need to add the local MySQL port, which is 3307, and for the other nodes, the haproxy port, which is 3306.

After you have configured all 3 nodes, on each node, start the service with the following commands

chkconfig haproxy on
service haproxy start

Check HAproxy

One way to check that HAproxy is running properly is to check a socket that the service creates on startup. To check the socket, we recommend the use of socat.

If you don’t have socat, install it:
```
yum install socat -y
```

Then run the following commands:

socat /var/lib/haproxy/stats.sock readline
prompt
show stat

An example of the output from socat is given in the following screenshot.

In this screenshot, there are 3 nodes and they are UP.

You can also run the following command:

mysql -u "user" -h "IpdelNodo" -p -e "show variables like 'wsrep_node_name';"

Check that the command returns something different each time, which shows that it is balancing properly.

Install and configure Keepalived

Do the following installation and configuration steps on each services node:

Install Keepalived:
```
yum install keepalived -y
```
Go to the configuration folder
```
cd /etc/keepalived/
```
Copy the default configuration file
```
mv keepalived.conf keepalived.conf.bak
```
Create a new configuration file
```
vim keepalived.conf
```

The configuration file contents will vary across the nodes:

On the main node of the Galera cluster, add the following configuration for the main node for Keepalived:

global_defs {
        router_id LBL01
}
vrrp_sync_group SyncGroup01 {
        group {
                FloatIP1
        }
}
vrrp_script check_haproxy {
        script "killall -0 haproxy"
        interval 2
        weight 2
}
vrrp_instance FloatIP1 {
        state MASTER
        interface eth0
        virtual_router_id 10
        priority 101
        advert_int 1
        virtual_ipaddress {

                vip.example.com

        }
        track_script {
                check_haproxy
        }
}

Here you need to specify the virtual IP, which in the example configuration is 10.10.203.220. Also, Keepalived works with a main/secondary configuration, and this is the main configuration.

Add the main configuration to the main node of the galera cluster, so it is also the main node for Keepalived.

Then on each of the secondary nodes, make the following changes:

router_id → Each node must have its own identifier. For example, the main node is LBL01, so the others can be LBL02, LBL03, and so on if you have more nodes.
state → The main node has the MASTER state and the secondary nodes should have the BACKUP state.
priority → The priority of the main node is 101, and on the secondary nodes it should be 100.

After you configure the nodes, on each node, activate and start the service:

chkconfig keepalived on
service keepalived start

To quickly check that you have configured everything correctly, run the following command

ip addr sh eth0

Check the IP address assigned to your network interface

This should show that your network interface (e.g. eth0) has the correct IP assigned to it.

You can also run the following command from a server that is not one of the services nodes, but which is running a mysql client and which has access to the service nodes over the network:

mysql -u "user" -h "VirtualIP" -p -e "show variables like 'wsrep_node_name';"

As you can see, this is very similar to the command to check that HAproxy is working, but this time we are connecting to the virtual IP, which is supplied by Keepalived, to check Keepalived.

Changes to API and UI

To make the Abiquo API use the virtual IP, log in to the server and do the following steps:

Go to the configuration folder:

cd /opt/abiquo/tomcat/conf/Catalina/localhost/

Edit the api.xml file

Check the default configuration in this format.

<Context>
    <Resource 
        name="jdbc/abiquoDB" 
        auth="Container" 
        type="javax.sql.DataSource" 
        factory="com.zaxxer.hikari.HikariJNDIFactory"
        dataSourceClassName="org.mariadb.jdbc.MariaDbDataSource"
        connectionTimeout="120000"
        maximumPoolSize="100"
        username="${server.database.username}"
        password="${server.database.password}"
        dataSource.url="jdbc:mariadb://${server.database.host}:${server.database.port}/kinton"
        />
</Context>

Remove the dataSource.url value and replace it with the virtual IP value
```
dataSource.url="jdbc:mariadb://${virtual.ip}:${database.port}/kinton"
```
Restart the tomcat service
```
service abiquo-tomcat restart
```

Monitoring

This section describes a configuration with two monitoring nodes. The Abiquo servers access the Monitoring server through a virtual IP. It is possible to extend the configuration to use three nodes.

On both monitoring nodes:

Install haproxy with yum install haproxy -y
Backup default haproxy configuration file with:
mv /etc/haproxy/haproxy.cfg /etc/haproxy/backup_haproxy.cfg
Create a new configuration file with: touch /etc/haproxy/haproxy.cfg
Edit the file and paste in the following configuration

Change the IP addresses in backend monitoring-cluster section to your monitoring nodes:

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats.sock mode 600 level admin
    stats timeout 2m

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    tcp
    log                     global
    option                  httplog
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout check           10s
    maxconn                 3000

listen haproxy-monitoring *:8090
  mode    http
  stats   enable
  stats   show-legends
  stats   refresh           5s
  stats   uri               /
  stats   realm             Haproxy\ Statistics
  stats   auth              monitor:AdMiN123
  stats   admin             if TRUE

#---------------------------------------------------------------------
# frontend monitoring-cluster
#---------------------------------------------------------------------
frontend haproxy-monitoring-1
    bind             *:36639
    default_backend monitoring-cluster

#---------------------------------------------------------------------
# round robin balancing monitoring-cluster backend
#---------------------------------------------------------------------
backend monitoring-cluster
    balance     roundrobin
    server  MonNode1 10.10.51.21:36638 check
    server  MonNode2 10.10.51.22:36638 check

In the above configuration, haproxy is listening port to 36639 (bind), so open the firewall on this port with firewall-cmd --zone=public --permanent --add-port=36639/tcp and reboot the firewall service with: systemctl restart firewalld
When you have configured all of the above on all nodes, activate and start the service with:
systemctl enable haproxy and systemctl start haproxy
To check if HAproxy is running, use socket.
1. Install it with yum install socat -y
2. Run the following in the terminal (copy all 3 lines and paste them at the same time):
```
socat /var/lib/haproxy/stats.sock readline
prompt
show stat
```
3. The output should be similar to the following
  There are two nodes and their status is up.
Install Keepalived with: yum install keepalived -y
Back up the configuration file with: mv /etc/keepalived/keepalived.conf /etc/keepalived/backup_keepalived.conf
Create a new configuration file with: touch /etc/keepalived/keepalived.conf

On the main monitoring node:

Copy and paste the following configuration, change the IP address in the virtual_ipaddress section to the virtual IP address in your environment.

global_defs {
        router_id MNT01
}
vrrp_sync_group SyncGroup01 {
        group {
                FloatIP1
        }
}
vrrp_script check_haproxy {
        script "killall -0 haproxy"
        interval 2
        weight 2
}
vrrp_instance FloatIP1 {
        state MASTER
        interface eth0
        virtual_router_id 11
        priority 101
        advert_int 1
        virtual_ipaddress {
                10.10.51.66
        }
        track_script {
                check_haproxy
        }
}

On the second monitoring node:

Copy and paste the following configuration, change the IP address in the virtual_ipaddress section to the one you have prepared (the same as on the main monitoring node in the keepalived configuration):

global_defs {
        router_id MNT02
}
vrrp_sync_group SyncGroup01 {
        group {
                FloatIP1
        }
}
vrrp_script check_haproxy {
        script "killall -0 haproxy"
        interval 2
        weight 2
}
vrrp_instance FloatIP1 {
        state BACKUP
        interface eth0
        virtual_router_id 11
        priority 100
        advert_int 1
        virtual_ipaddress {
                10.10.51.66
        }
        track_script {
                check_haproxy
        }
}

On both monitoring nodes:

Enable and start the service with: systemctl enable keepalived and systemctl start keepalived
Check if the configuration is correct with: ip addr sh eth0
The output should be similar to the following.

If the firewall is active on both nodes, we have to add VRRP to the rules

firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent

Restart the firewall with systemctl restart firewalld

Check the failover

Check if the failover is working properly:

Open main monitoring node console from vCenter or another way that does not require a network connection on the interface configured for keepalived
Disable network interface assigned to keepalived with: ifconfig eth0 down

Check the logs on the main monitoring node if the state of the server has been updated properly:

tailf -500  /var/log/messages | grep -i keepalived

You should see something similar to this transition to a FAULT state:

Keepalived_vrrp[20215]: Kernel is reporting: interface eth0 DOWN
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) Entering FAULT STATE
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) removing protocol VIPs.
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) Now in FAULT state

On the second monitoring node check the logs with the same command:

tailf -500  /var/log/messages | grep -i keepalived

Here you should see similar transition from BACKUP state to MASTER:

Keepalived_vrrp[7058]: VRRP_Instance(FloatIP1) Transition to MASTER STATE
Keepalived_vrrp[7058]: VRRP_Instance(FloatIP1) Entering MASTER STATE
Keepalived_vrrp[7058]: VRRP_Instance(FloatIP1) setting protocol VIPs.
Keepalived_vrrp[7058]: Sending gratuitous ARP on eth0 for 10.10.51.66
Keepalived_vrrp[7058]: VRRP_Instance(FloatIP1) Sending/queueing gratuitous ARPs on eth0 for 10.10.51.66

Go back to main monitoring node console and enable the network interface back again with:

ifconfig eth0 up

Check the logs again and on the main node you should see transition back to MASTER state:

Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) Entering BACKUP STATE
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) forcing a new MASTER election
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) Transition to MASTER STATE
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) Entering MASTER STATE
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) setting protocol VIPs.
Keepalived_vrrp[20215]: Sending gratuitous ARP on eth0 for 10.10.51.66
Keepalived_vrrp[20215]: VRRP_Instance(FloatIP1) Sending/queueing gratuitous ARPs on eth0 for 10.10.51.66

On the second node you should see it went to the BACKUP state as it was before:

Keepalived_vrrp[7058]: VRRP_Instance(FloatIP1) Received advert with higher priority 103, ours 102
Keepalived_vrrp[7058]: VRRP_Instance(FloatIP1) Entering BACKUP STATE
Keepalived_vrrp[7058]: VRRP_Instance(FloatIP1) removing protocol VIPs.Now the failover seems to be working correctly.

API and Remote Services

On API server and Remote Services servers:

Edit abiquo.properties file at /opt/abiquo/config/abiquo.properties

Change the following properties to the Virtual IP address and haproxy listening port:

abiquo.watchtower.host    = 10.10.51.66 
#virtual IP set in keepalived
abiquo.watchtower.port    = 36639 
#bind port set in haproxy

Restart abiquo-tomcat service: systemctl restart abiquo-tomcat