Upgrade Abiquo

1. Introduction

This section describes how to upgrade from Abiquo 6.1.x to an Abiquo 6.2.x version. 

• To upgrade from earlier versions, first see: 

  • Upgrade Abiquo from 4.7 to 5.2.x

  • Upgrade Abiquo from 5.2 to 5.4.x

  • Upgrade Abiquo from 5.4.x to 6.1.x

For your convenience, we have also prepared the following concise upgrade document that groups the commands by servers.
However, we encourage you to use the current Upgrade Abiquo document as your main reference:

• Upgrade Abiquo 6.1.x to 6.2.x distributed by servers

Please contact Abiquo Support for further information.

Major version upgrade table

Minor version upgrade table

2. Warnings and notices

• Before you begin, contact Abiquo Platform and Customer Success department to obtain your credentials for Abiquo YUM repositories.

• The upgrade will overwrite lang_en_US_labels.json. Back up your customized labels or translations before the upgrade, and restore them afterwards.

3. Prevent cloud users from performing VM operations

1. In the UI in the Infrastructure view, select each physical machine and click Disable

2. Using the API, set the state of each physical machine to DISABLED

4. Check for operations in progress on the platform

Before you shut down the platform servers, check that no operations are in progress.

1. Check that the Abiquo RabbitMQ queues are empty on the Abiquo Monolithic Server, Abiquo Server or Datanode server

   The number of messages in all queues must be 0.

   # rabbitmqctl list_queues messages name

   # rabbitmqctl list_queues messages name
   Listing queues ...
   0	abiquo.am.notifications
   0	abiquo.bpm.notifications
   0	abiquo.datacenter.requests.ADatacenter.bpm
   0	abiquo.datacenter.requests.ADatacenter.virtualfactory
   0	abiquo.ha.tasks
   0	abiquo.nodecollector.notifications
   0	abiquo.pcrsync.messages
   0	abiquo.pcrsync.parking-expect-no-consumers
   0	abiquo.scheduler.fast.requests
   0	abiquo.scheduler.requests
   0	abiquo.scheduler.slow.requests
   0	abiquo.tracer.traces
   0	abiquo.virtualfactory.notifications
   0	abiquo.virtualmachines.definitionsyncs
   0	abiquo.vsm.eventsynk
   ...done.

2. On the V2V Server, check for any active conversions by checking for the V2V or Mechadora processes

   $ ps aux | grep v2v
   $ ps aux | grep mechadora

When user VM operations are blocked and all of the above checks show that no tasks are running, it is safe to halt the platform.

5. Stop platform services

This section describes how to stop platform services on all servers.

To stop platform services:

1. Stop the API on the API server or monolithic server

   systemctl stop abiquo-tomcat

2. Stop the UI on the API server or monolithic server or dedicated UI server

   systemctl stop httpd

3. Stop RabbitMQ on the API server or dedicated datanode server

   systemctl stop rabbitmq-server

4. Stop remote services server

   systemctl stop abiquo-tomcat

5. V2V Server
   You do not need to stop anything because the BPM remote service is run on-demand only
   

6. Stop monitoring server

   systemctl stop abiquo-delorean
   systemctl stop abiquo-emmett
   systemctl stop kairosdb
   systemctl stop cassandra

7. On the monitoring server, check if Cassandra is really dead

   ps auwx | grep cassandra

   Get the process number for Cassandra (the first number in the output of the previous command), and kill it. In this example, Cassandra is process 12345.

   kill -9 12345

All processes on platform servers should now be halted.

6. Back up the main platform elements

To perform a basic backup of the platform, run the following backups:

# On the Abiquo API Server or UI server, back up the UI
tar cvfz /var/www/html/ui.tgz-$(date +%Y%m%d_%H%M%S) /var/www/html/ui

# Copy the api.xml file to back up the database connection details
cp /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml /opt/abiquo/api.xml-$(date +%Y%m%d_%H%M%S)

# On ALL servers, back up the /opt/abiquo folder
tar cvfz /opt/abiquo.tgz-$(date +%Y%m%d_%H%M%S) /opt/abiquo

# On Abiquo Monolithic or API Server, Remote Services, or Datanode, back up Redis
redis-cli save
cp -a /var/lib/redis /var/lib/redis-$(date +%Y%m%d_%H%M%S)

# On DB server, back up the Abiquo MySQL DBs
mysqldump --routines --triggers kinton            > kinton.sql-$(date +%Y%m%d_%H%M%S)
mysqldump --routines --triggers kinton_accounting > kinton_accounting.sql-$(date +%Y%m%d_%H%M%S)

# On the Monitoring server, back up Watchtower DB
mysqldump --routines --triggers watchtower            > watchtower.sql-$(date +%Y%m%d_%H%M%S)

# If you are using SAML authentication with multiple IdPs, on ALL Abiquo API Servers, back up this file
cp 
/opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml-$(date
 +%Y%m%d_%H%M%S)

7. Make snapshots and backups of all platform servers

This is a major upgrade, so we recommend that you make a snapshot and/or a full backup of your platform servers.

8. Remove Docker and unmount shared folders

1. On your RS server(s) uninstall Docker

   rpm -e --nodeps docker
   rpm -e --nodeps docker-client
   rpm -e --nodeps docker-common

2. If you have multiple instances of the NARS Remote Service in your datacenter, check on your RS servers for an NFS shared folder for the NARS results (with the Abiquo configuration property abiquo.nars.async.fileresults.path):

   $ mount | grep -i "nfs\|results" | grep -v sunrpc
   10.60.37.87:/abq-pro-nfs on /opt/abiquo/results-nars type nfs4 (rw,nosuid,nodev,noexec,relatime,sync,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.60.37.77,local_lock=none,addr=10.60.37.87,user)

3. Unmount the shared folder. You will remount it after the upgrade

   sudo umount /opt/abiquo/results-nars

Prepare yum repositories for 6.2 or 6.2.x

Check that you have the repository URL and credentials

On all Abiquo Servers, such as Abiquo API, Remote Services, Watchtower monitoring.

Install release package

1. To upgrade to a version with a patch number of zero, for example, version 6.2.0

   1. Remove the previous version Abiquo release packages.

      yum remove 'abiquo-release-*'

   2. Find the abiquo-release-ee package for your version in the list of available versions

   3. Install the release package.
      For Abiquo 6.2.0, the command would be similar to the following:

      yum localinstall http://usr:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-release-ee-6.2.0-AAAAAAAA_BBBBBBB.el7.noarch.rpm

      For Abiquo 6.2.1, the release package would be in the format:

      http://usr:passwd@mirror.abiquo.com/el7/6.2/updates/x86_64/abiquo-release-ee-6.2.1-BBBBBBBBBBBB.el7.noarch.rpm

   4. For major versions, disable the updates repo.

      yum-config-manager --disable abiquo-updates

Prepare yum repositories for 6.2.x (minor version)

1. Check that you have the repository URL and credentials

2. To upgrade to the latest version with a patch number that is not zero, for example, version 6.2.x, enable both repositories:

   yum-config-manager --enable abiquo-base
   yum-config-manager --enable abiquo-updates

3. Optionally, add your username and password to the Abiquo repos

   yum-config-manager --save --setopt=abiquo-*.username=MYUSERNAME
   yum-config-manager --save --setopt=abiquo-*.password=MYPASSWORD

   Don't forget to use a backslash to escape any shell special characters. For more details, see https://www.oreilly.com/library/view/learning-the-bash/1565923472/ch01s09.html

4. Clean yum and make cache

   yum clean all && yum makecache

5. If you did not make snapshots of all servers already, then you could do this now.

9. Upgrade packages on ALL servers

These instructions are for monolithic, distributed, and HA environments.

1. Update all packages.

   yum clean all && yum makecache && yum update 

   The MariaDB update requires a manual install. But first download and run the gpmigrator tool

2. If you earlier unmounted shared folders from your Remote Services server, mount this folder again (and any other folders!)

   sudo mount -a

3. The Abiquo services must run as the tomcat user (not root), so set the required permissions and enforce the use of the package configuration files.
   If you have a MONOLITHIC SERVER do all of the following steps on the Monolithic server

   1. On Abiquo server and remote services

      chown -R tomcat /opt/abiquo

   2. On the remote services with appliance manager and the V2V server (i.e. servers that mount the NFS repository)

      chown -R tomcat /opt/vm_repository
      chmod -R a+r /opt/vm_repository

   3. On the V2V server, check the /etc/sysconfig/abiquo/ec2-api-tools file exists.
      The file must contain the following configuration. If the file does not exist, create it and add this configuration.

      For CentOS 7

      EC2_HOME=/opt/aws

10. Run the guest password migrator

1. Download and install the gpmigrator package

   yum localinstall http://user:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-gpmigrator-6.2.0-20240205_155724.el7.noarch.rpm

2. Run the tool to migrate the guest passwords of the current VMs to a new column that will be added in 6.2.0.

   1. The tool requires access to the kinton database and to the API Redis

   2. Usage:

      --jdbc-url VAL    : Replace VAL with Kinton JDBC URL including user and password
      --output-file VAL : Replace VAL with the output file (by default: 'guestpasswords.sql') 
      --redis-host VAL  : Redis host to connect
      --redis-port N    : Redis port to connect (by default: 6379)

      For example:

      java -jar gpmigrator.jar --jdbc-url="jdbc:mariadb://localhost:3306/kinton?user=root&password=root" --redis-host=localhost

3. Save the guestpasswords.sql file (or whatever you called it) to run after you update the Abiquo database

11. Upgrade MariaDB for Abiquo 6.2.0

1. On all database servers, including the Watchtower server, upgrade MariaDB

   rpm --nodeps -ev MariaDB-server-10.4.10-1.el7.centos.x86_64
   yum install MariaDB-server
   service mariadb start
   # OR for datanode: galera_new_cluster 
   mysql_upgrade 

12. Update the Abiquo databases

1. Check that your hostname is in your DNS or in your /etc/hosts file

2. Upgrade the Abiquo API databases

   abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

3. If the database update fails with a message similar to the following: 

   Liquibase update Failed: Validation Failed:
   1 change sets check sum
   src/X.X.X-XX/kinton-X.X.X.xml::ABICLOUDPREMIUM-XXXX-XxxxxxxxXXxXxxxxxxXxxxxx::XXXXXXXXX is now: 7:ee2fa6e058ec76c7abf801567898917d
   For more information, use the --logLevel flag

   Clear the checksums and run the update again

   abiquo-db clearCheckSums
   abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

4. For Abiquo 6.2.0, run the gpmigrator SQL file output (with the appropriate user and credentials)

   mysql kinton < guestpasswords.sql

5. To upgrade the Abiquo Watchtower database, for the monitoring appliance, run the command below:

   watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

13. Upgrade additional elements

For every version, on EVERY Remote Services servers, update the Redis database script with the new VM definitions.

cd /usr/share/doc/abiquo-redis/redis/4.2.3
bash ./00-old-vsm-definitions.sh

For Abiquo 6.2.0+, update the database connector configurations

1. On the Abiquo API Server, edit /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml

2. The latest version of the file should contain new options as shown below.
   Update the file and/or variables marked with ${} for your environment.

   <Context>
       <Resource 
           name="jdbc/abiquoDB" 
           auth="Container" 
           type="javax.sql.DataSource" 
           factory="com.zaxxer.hikari.HikariJNDIFactory"
           dataSourceClassName="org.mariadb.jdbc.MariaDbDataSource"
           connectionTimeout="120000"
           maximumPoolSize="100"
           username="${server.database.username}"
           password="${server.database.password}"
           dataSource.url="jdbc:mariadb://${server.database.host}:${server.database.port}/kinton"
           />
   </Context>

3. If you have a DB cluster, change the datasource.url to point to your Galera cluster.
   This will not affect the multi-master configuration of the cluster

   jdbc:mariadb:loadbalance://address=(host=${DBMAIN})(port=3306)(type=master),
           address=(host=${DBR1})(port=3306)(type=slave),
           address=(host=${DBR2})(port=3306)(type=slave)/kinton

4. On the Watchtower server, edit the emmett.conf and delorean.conf files at:

   /etc/abiquo/watchtower/delorean.conf
   /etc/abiquo/watchtower/emmett.conf

   1. From the connection details, remove jdbc:mysql: and replace with jdbc:mariadb:

   2. Add your Galera configuration if your monitoring DB is in a cluster

14. Configure Abiquo after the upgrade

1. Before you start the Abiquo tomcat server, add Abiquo configuration properties to the abiquo.properties file.
   By default the abiquo.properties file is found in the /opt/abiquo/config/ folder.
   See Changes to Abiquo configuration properties

2. Configure the user interface. The default UI location is /var/www/html/ui.
   Optional: Add custom labels and translations in the lang_xx_XX_custom.json files in the lang folder
   Add custom configuration to client-config-custom.json. See Configure Abiquo UI and Changes to UI configuration
   If your API is not in the same domain as the UI, set the API endpoint pointing to your Abiquo API server:

   {
       "config.endpoint": "http://myAPIserver/api"
   }

3. SAML: If you are using SAML with multiple IDPs, restore the configuration as described at Restore SAML security beans after upgrade.

4. Abiquo 6.2.0 updates Java and other packages, so restore your certificates. To do this, you can follow the steps at Update certificates in your Abiquo environment.

15. Start Abiquo server and services

To start the Abiquo platform servers and services, do these steps:

• On Abiquo server or datanode, restart RabbitMQ

  systemctl start rabbitmq-server

• On the Abiquo server, restart the HTTP daemon to refresh the user interface files, and bring up the Tomcat server.

  service httpd start
  service abiquo-tomcat start

• On the Remote services server, start the Tomcat server

  service abiquo-tomcat start

• On the Monitoring server: start the Cassandra service

  sudo service cassandra start

• WAIT about 5 minutes until the service is up and running

• Start the KairosDB service

  sudo service kairosdb start

• Start the other services in this order

  sudo service abiquo-emmett start
  sudo service abiquo-delorean start

• On the V2V server: restart the Tomcat server:

  service abiquo-tomcat restart

Finally, in Abiquo, UI

1. Re-enable the physical machines!

2. In Abiquo 6.2.0, to activate the cost usage reports, regenerate billing

After the upgrade, to prevent glitches in the UI, clear your browser cache