Upgrade Abiquo distributed to v5.2.x

Owned by MS (Unlicensed)
Feb 04, 2024
17 min read


This section describes how to upgrade from Abiquo 4.7.0 or above to an Abiquo 5.2.x version. Please contact Abiquo Support for further information.



Major upgrade

The upgrade to Abiquo 5.2.0 is a major upgrade with upgrades of third-party software packages that are critical platform dependencies

You must make snapshots of ALL servers before you start the upgrade



  • Abiquo YUM repositories are no longer open, please contact Abiquo Support to obtain your credentials

  • This upgrade process starts from Abiquo 4.7 or above. To upgrade from earlier versions, please contact Abiquo Support



Prepare branding

  • If you are upgrading from Abiquo 4.7.x, the new UI in Abiquo 5.0 requires new branding. See Abiquo branding guide





Prevent cloud users from performing VM operations

  1. In the UI in the Infrastructure view, select each physical machine and click Disable

  2. Using the API, set the state of each physical machine to DISABLED





Check for operations in progress on the platform

Before you shut down the platform servers you should check that no operations are in progress on the platform.

  1. Check that the Abiquo RabbitMQ queues are empty on the Abiquo Monolithic Server, Abiquo Server or Datanode server

    The number of messages in all queues must be 0.

    # rabbitmqctl list_queues messages name

     

    # rabbitmqctl list_queues messages name Listing queues ... 0 abiquo.am.notifications 0 abiquo.bpm.notifications 0 abiquo.datacenter.requests.ADatacenter.bpm 0 abiquo.datacenter.requests.ADatacenter.virtualfactory 0 abiquo.ha.tasks 0 abiquo.nodecollector.notifications 0 abiquo.pcrsync.messages 0 abiquo.pcrsync.parking-expect-no-consumers 0 abiquo.scheduler.fast.requests 0 abiquo.scheduler.requests 0 abiquo.scheduler.slow.requests 0 abiquo.tracer.traces 0 abiquo.virtualfactory.notifications 0 abiquo.virtualmachines.definitionsyncs 0 abiquo.vsm.eventsynk ...done.

     

  2. On the V2V Server, check for any active conversions by checking for the V2V or Mechadora processes

    $ ps aux | grep v2v $ ps aux | grep mechadora

     

When user VM operations are blocked and all of the above checks show that no tasks are running, it is safe to halt the platform.



Back up the main platform elements

To perform a basic backup of the platform, run the following backups:

Before you begin, stop platform services, and check you have enough space on your destination systems.

 

 

 

Disable old billing dashboards and delete last bills

In Abiquo 5.2, the billing dashboard feature is now in the core Abiquo platform.

If you are using the billing dashboard scripts from previous versions, disable them and delete last bills before you continue with the upgrade.

  1. On the server where you were running the billing packages, remove the Cron jobs to run the billing scripts.



  2. Delete the last bills from the previous version of the billing dashboards (from the kinton.billing_consolidation and kinton.billing_register tables). Remove the bills for Azure and Amazon providers, for the number of months covered by the abiquo.enterprise.property.billing.monthoffset property, which has a default value of 2 months. Later the new billing dashboards feature will regenerate these bills.





Stopplatform services

This section describes how to stop platform services on all servers.

 

To stop platform services:

  1. Stop the API on the API server or Monolithic server

  2. Stop the UI on the API server or Monolithic server or dedicated UI server

  3. Stop Remote services server

  4. Stop the database on the Monolithic server or Database server or Datanode server.

    For a datanode configuration, you will also need to stop the Galera cluster. For more details, see Stop and start HA configuration

  5. Stop RabbitMQ (on the monolithic server or API Server or Datanode)

  6. V2V server -
    You do not need to stop anything because the BPM remote service is run on-demand only

  7. Stop monitoring server

  8. On the monitoring server, check if Cassandra is really dead

    Get the process number for Cassandra (the first number in the output of the previous command), and kill it. In this example, Cassandra is process 12345.

All processes on platform servers should now be halted.

 



Make snapshots and backups of all platform servers

The upgrade to Abiquo 5.2.x is a major upgrade with upgrades of third-party software packages that are critical platform dependencies

You MUST make snapshots of all servers in your platform before you upgrade to Abiquo 5.2.x.



Prepare yum repositories on all servers

Check that you have the repository URL and credentials

  • To upgrade to a version with a patch number of zero, for example, version 5.2.0

  • OR To upgrade to a version with patch number that is not zero, for example, version 5.2.1, enable both repositories:

  • Optionally add your username and password to the Abiquo repos

Don't forget to use a backslash to escape any shell special characters. For more details, see Learning the bash Shell, Second Edition

  • Clean yum and make cache

  • If you did not make snapshots of all servers already, then do this now



Upgrade distributed API and DB servers

This section describes the upgrade for a standard API and DB server.

These steps are for a Distributed API Server with the database on the API server.

Abiquo will upgrade to new versions of RabbitMQ and MySQL that require a manual upgrade path of dependencies.

 

  1. Remove the Erlang and Galera packages.

     

  2. Install new RabbitMQ and MariaDB server

    Before you confirm, check that the packages will be installed from the abiquo-base repository

    1. rabbitmq-server: 3.8.2.1

    2. MariaDB-server: 10.4.10.1

  3. Upgrade Abiquo to 5.2.x

    Before you confirm, check the following packages will be installed from the abiquo-base repository:

    1. jdk: 11.0.6u10

    2. redis: 5.0.7.1

  4. After the update, check versions.

    Check that the symbolic link to the latest version of Java points to Java 11 on Monolithic Server or Datanode

    Check that the correct Tomcat version will be used on the Monolithic Server

    On the Monolithic Server in the JDK folder, check certificate migration for AM download. The API Server certificate should be listed for Java 11 with its FQDN. You may need to enter the storepass option to supply the password and you can use the alias option to search for the hostname or FQDN of your server

    For example

     

  5. Enable and start the new services

     

  6. Upgrade MySQL

    The mysql_upgrade step may detect errors that trigger messages such as the following:

    These messages are expected and this same mysql_upgrade process will automatically fix these errors in its next stages.

    Check that the mysql_upgrade process completes correctly.

     

  7. Check that your hostname is in your DNS or in your /etc/hosts file

  8. Upgrade the Abiquo API databases

    If the liquibase update fails with a message similar to the following: 

    Do the following steps

    1. Clear the database checksums

       

    2. Retry the above abiquo-db update command.

     

  9. Change file owners to tomcat user

     

 

This section contains an alternative upgrade procedure for standalone Database servers and a Distributed API server without DB



Upgrade standalone database servers

This step is for standalone MariaDB database servers.

Abiquo will upgrade to a new version of MySQL that requires a manual upgrade path of dependencies.

 

  1. Remove the Galera package

  2. Install new MariaDB server

    Before you confirm, check that the packages will be installed from the abiquo-base repository

    1. MariaDB-server: 10.4.10.1

  3. Upgrade Abiquo to 5.2.x

    Before you confirm, check the following packages will be installed from the abiquo-base repository:

    1. jdk: 11.0.6u10

    2. redis: 5.0.7.1

  4. After the update, check versions.

    Check that the symbolic link to the latest version of Java points to Java 11

     

  5. Enable the services that were reinstalled

     

  6. Upgrade MySQL

    The mysql_upgrade step may detect errors that trigger messages such as the following:

    These messages are expected and this same mysql_upgrade process will automatically fix these errors in its next stages.

    Check that the mysql_upgrade process completes correctly.

     

  7. Check that your hostname is in your DNS or in your /etc/hosts file

  8. Upgrade the Abiquo API databases. This script is found on the API Server

     

    If the liquibase update fails with a message similar to the following: 

     

    Do the following steps

    1. Clear the database checksums

    2. Retry the above abiquo-db update command.

 



Upgrade distributed API servers

These steps are for an API server in a distributed environment that has a separate database server.

Abiquo will upgrade to new versions of RabbitMQ that require a manual upgrade path of dependencies.

 

  1. Remove the Erlang packages

  2. Install new RabbitMQ

    Before you confirm, check that the packages will be installed from the abiquo-base repository

    1. rabbitmq-server: 3.8.2.1

  3. On the API server, upgrade Abiquo to 5.2.x

    Before you confirm, check the following packages will be installed from the abiquo-base repository:

    1. jdk: 11.0.6u10

    2. redis: 5.0.7.1

  4. Check versions of Java and Tomcat

    Check that the symbolic link to the latest version of Java points to Java 11

    Check that the correct Tomcat version will be used

    On the API Server in the JDK folder, check certificate migration for AM download. The API Server certificate should be listed for Java 11 with its FQDN. You may need to enter the storepass option to supply the password and you can use the alias option to search for the hostname or FQDN of your server

    For example

  5. Enable and start RabbitMQ on the API server

  6. On the API Server, change file permissions so the Abiquo services can run as the tomcat user (not root) and allow access to the configuration files

     

 

 

Upgrade remote services servers

These steps are for remote services servers in a distributed or HA environment.

Upgrade EVERY remote services server with these steps.

  1. On the remote services servers, upgrade Abiquo to 5.2.x

    Before you confirm, check the following packages will be installed from the abiquo-base repository:

    1. jdk: 11.0.6u10

    2. redis: 5.0.7.1

  2. After the update, check versions of Java and Tomcat

    Check that the symbolic link to the latest version of Java points to Java 11

    Check that the correct Tomcat version will be used

    On the remote services server in the JDK folder, check certificate migration to allow AM download

    The API server certificate should be listed for Java 11 with its FQDN. You may need to enter the storepass option to supply the password and you can use the alias option to search for the hostname or FQDN of your server

    For example

     

  3. On all remote services servers using SSL

    Edit the Tomcat server configuration file:

    Find the location of the certificate files for Java 8

    Change it to the location for Java 11 (e.g. remove "/jre")

     

  4. On all remote services servers, change file owners to tomcat user

     

  5. On the remote services with appliance manager and the V2V server (i.e. servers that mount the NFS repository), change file owners to Tomcat user





Upgrade monitoring server

This step is for Watchtower monitoring servers in all installations.

  1. To continue using Java 8 (for Cassandra), remove the JDK through the package manager. If you use yum, it will be delete all the dependencies and you will have to reinstall them

     

  2. Check the jdk to install is version 8

     

  3. Check that Java is correctly installed:

    1. If you execute this command and get that error, please execute the command from the next bullet:

       

    2. Fix it by executing this command, then selecting the option "java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/bin/java)":

       

  4. As for the Abiquo DB steps, remove Galera, which will also remove MariaDB

     

  5. Clean yum and make cache

     

  6. Install MariaDB

     

  7. Update Abiquo

     

  8. Check that the Java package is now "javajdk" to use Java 1.8.0u144 with Cassandra:

     

  9. If you have a HA datanode that runs the watchtower database, skip directly to Step 10 to update watchtower database

  10. Enable and start the services that were reinstalled

     

  11. Upgrade MySQL

    The mysql_upgrade step may detect errors that trigger messages such as the following message (note: this example is from the Abiquo database server).

     

    These messages are expected and this same mysql_upgrade process will automatically fix these errors in its next stages.

    Check that the mysql_upgrade process completes correctly.

  12. Update the Abiquo Watchtower database

     

  13. Edit the file /opt/kairosdb/conf/kairosdb.properties to update the name of the following variable and to remove the port from it:

    1. Remove the line kairosdb.datastore.cassandra.host_list=192.168.888.999:9160.
      And add this line kairosdb.datastore.cassandra.cql_host_list=192.168.888.999.
      Please note the new cql_ prefix for host_list

    2. Add kairosdb.datastore.cassandra.datapoint_ttl = 15768000 (ref: Internal JIRA SUP-333):

  14. Edit the file /etc/cassandra/conf/cassandra.yml OR /etc/cassandra/default.conf/cassandra.yaml (whichever exists) to remove a variable:

    1. Remove the line starting with kairosdb.datastore.cassandra.datapoint_ttl and save and close the file.

 

Upgrade steps from 4.7.x to 5.0.x

Start with your original version and perform the steps to the final version.

Run Redis database scriptUpgrade steps from 4.7.x to 5.0.x

On each Remote Services server:

 

Post-upgrade steps:

  • Change the tutorials' softlinks in the frontend servers. If there is a softlink /var/www/html/ui/config/tutorials pointing to /opt/abiquo/config/tutorials, change it to point to /opt/abiquo/tutorials instead.

  • If your Apache redirect addresses end in trailing slash characters ("/"), remove the trailing slashes





Upgrade steps from 5.0.x to 5.1.x

Start with your original version and perform the steps to the final version.

Run script to update job names on Veeam server

When upgrading from 5.0.x or 5.1.x to 5.1.2 or above, if you are using Veeam, run the following application that will update the job names on the Veeam server to allow duplicate VM names.

The instructions for running the tool are in the following file.

To run the tool, do these steps:

  1. Edit the veeam.properties file and set the MySQL jdbc properties. Replace the examples with the values for the Abiquo database

     

  2. Obtain the Veeam version and credentials to pass as arguments to the script

    • veeam95u4 Run for usage with Veeam server 9.5u4 or veeam10 Run for usage with Veeam server 10

    • -ip IP address of Veeam server

    • -u, --user veeamServerUser

    • -p, --password veeamServerPassword

  3. The other argument defines a dry-run mode to test the script

    • -no-dry, --no-dry-run If true, update Veeam job names. If false, only show what will the names will look like after the change

  4. Run the tool in dry-run mode (with the argument "--no-dry-run false" and check the Veeam jobs that it will update.

     

  5. Run the tool in production mode to update the Veeam server

     

  6. After you run the tool, edit the veeam.properties file, and remove the temporary properties with the database credentials.

 



 

Run mandatory database scripts on Redis database and Watchtower SQL database

When upgrading from any version lower than 5.1.2, do the following steps once (e.g. during upgrade to 5.1.2 OR upgrade to 5.2.x)

On the Abiquo Server, run the following tool that will generate two scripts to run on the Redis database and Watchtower SQL database.

These scripts change the use of the Abiquo VM Identifier (name attribute) as the UID to the ID of the VM as the new UID.

The instructions for running the tool are in the following file:

To run the tool and scripts, do these steps:

  1. On the Abiquo Server, run the tool to generate the scripts, supplying the database credentials

    You should find the following scripts in the output directory (by default the current directory)

    • redisdelta, which is Bash script

    • watchtowerdelta.sql

  2. For each Remote Services server, copy the redisdelta Bash script to the server.

  3. On each Remote Services server, run the Bash script. Note that this script requires Bash version 4, which is the default version on Abiquo servers

     

  4. Copy the SQL script to the Watchtower Monitoring Server.

  5. Run the SQL script

     

Run Redis database script

On each Remote Services server:

 






Upgrade steps from 5.1.x to 5.2.x

Start with your original version and perform the steps to the final version.

When upgrading from 5.1.0 or 5.1.1 to 5.1.2 or above, follow the steps from the Upgrade steps from 5.0.x to 5.1.x block.







 





Upgrade steps for 5.2 versions

These steps apply to upgrades starting from version 5.2.0 and above

 

 

Configure Abiquo after the upgrade

  1. Before you start the Abiquo Tomcat server, add Abiquo configuration properties to the abiquo.properties file.
    By default the abiquo.properties file is found in the /opt/abiquo/config/ folder.
    See Changes to Abiquo configuration properties

  2. Configure the user interface. The default UI location is /var/www/html/ui.
    Optional: Add custom labels and translations in the lang_xx_XX_custom.json files in the lang folder
    Add custom configuration to client-config-custom.json. See Configure Abiquo UI
    If your API is not in the same domain as the UI, set the API endpoint pointing to your Abiquo API server:



  3. Reporting changes: To upgrade the Abiquo Reports database for the upgrade to Abiquo 4.7.x+, contact Abiquo Support for the file and procedure.





Start Abiquo servers and services

To start the Abiquo platform servers and services, do these steps:

  1. On the Abiquo server, restart the HTTP daemon to refresh the user interface files, and bring up the Tomcat server.

  2. On the Remote services server, start the Tomcat server

  3. On the Monitoring server do these steps.

    1. Edit the file /opt/kairosdb/conf/kairosdb.properties to update the name of the following variable and to remove the port from it:

      1. Replace the line kairosdb.datastore.cassandra.host_list=192.168.888.999:9160.

      2. With this line kairosdb.datastore.cassandra.cql_host_list=192.168.888.999. Please note the new cql_ prefix for host_list.

    2. Edit the file /etc/cassandra/conf/cassandra.yml OR /etc/cassandra/default.conf/cassandra.yaml (whichever exists) to remove a variable:

      1. Remove the line starting with kairosdb.datastore.cassandra.datapoint_ttl and save and close the file.

    3. Edit the file /opt/kairosdb/conf/kairosdb.properties to add a new variable (ref: Internal JIRA SUP-333):

      1. Add kairosdb.datastore.cassandra.datapoint_ttl = 15768000

  4. On the Monitoring server, start the Cassandra service

  5. WAIT about 5 minutes until the service is up and running

  6. Start the KairosDB service

  7. Start the other services in this order

  8. On the V2V server: restart the Tomcat server:

     

  9. In Abiquo, re-enable the physical machines!

  10. Clear your browser cache to prevent glitches in the UI







Copyright © 2006-2024, Abiquo Holdings SL. All rights reserved