Page Comparison

...

Before you shut down the platform servers you should check that no operations are in progress .Check for operations in progress on the platform

...

.

1. Check

...

Back up the main platform elements

Back up the main platform elements

...

Disable old billing dashboards and delete last bills

In Abiquo 5.2, the billing dashboard feature is now in the core Abiquo platform.

If you are using the billing dashboard scripts from previous versions, disable them and delete last bills before you continue with the upgrade.

On the server where you were running the billing packages, remove the Cron jobs to run the billing scripts.

# rm /etc/cron.d/abiquo-billing-ec2
# rm /etc/cron.d/abiquo-billing-arm

...

Delete the last bills from the previous version of the billing dashboards (from the kinton.billing_consolidation and kinton.billing_register tables). Remove the bills for Azure and Amazon providers, for the number of months covered by the abiquo.enterprise.property.billing.monthoffset property, which has a default value of 2 months. Later the new billing dashboards feature will regenerate these bills.

Stopplatform services

Stop platform services

...

Make snapshots and backups of all platform servers

The upgrade to Abiquo 5.2.x is a major upgrade with upgrades of third-party software packages that are critical platform dependencies

You MUST make snapshots of all servers in your platform before you upgrade to Abiquo 5.2.x.

Prepare yum repositories on all servers

Prepare yum repositories on all servers for v5.2

...

Upgrade distributed API and DB servers

This section describes the upgrade for a standard API and DB server.

Upgrade Distributed API and DB server to v5.2

...

This section contains an alternative upgrade procedure for standalone Database servers and a Distributed API server without DB

...

Upgrade standalone database servers

Upgrade standalone Database servers to v5.2

...

Upgrade distributed API servers

Upgrade Distributed API Server without DB to v5.2

...

Upgrade remote services servers

Upgrade remote services servers to v5.2

...

Upgrade monitoring server

Upgrade monitoring server to v5.2

...

Upgrade steps from 4.7.x to 5.0.x

Start with your original version and perform the steps to the final version.

...

Upgrade steps from 4.7.x to 5.0.x

...

Upgrade steps from 5.0.x to 5.1.x

Start with your original version and perform the steps to the final version.

...

Upgrade steps from 5.0.x to 5.1.x for Veeam

...

Upgrade steps from 5.0.x to 5.1.2 or 5.2.x for VM names

...

Upgrade steps from 5.1.x to 5.2.x

Start with your original version and perform the steps to the final version.

When upgrading from 5.1.0 or 5.1.1 to 5.1.2 or above, follow the steps from the Upgrade steps from 5.0.x to 5.1.x block.

...

...

...

1. that the Abiquo RabbitMQ queues are empty on the Abiquo Monolithic Server, Abiquo Server or Datanode server

   The number of messages in all queues must be 0.

   Code Block
   # rabbitmqctl list_queues messages name

   Code Block

   # rabbitmqctl list_queues messages name Listing queues ... 0 abiquo.am.notifications 0 abiquo.bpm.notifications 0 abiquo.datacenter.requests.ADatacenter.bpm 0 abiquo.datacenter.requests.ADatacenter.virtualfactory 0 abiquo.ha.tasks 0 abiquo.nodecollector.notifications 0 abiquo.pcrsync.messages 0 abiquo.pcrsync.parking-expect-no-consumers 0 abiquo.scheduler.fast.requests 0 abiquo.scheduler.requests 0 abiquo.scheduler.slow.requests 0 abiquo.tracer.traces 0 abiquo.virtualfactory.notifications 0 abiquo.virtualmachines.definitionsyncs 0 abiquo.vsm.eventsynk ...done.

2. On the V2V Server, check for any active conversions by checking for the V2V or Mechadora processes

   Code Block
   $ ps aux | grep v2v $ ps aux | grep mechadora

When user VM operations are blocked and all of the above checks show that no tasks are running, it is safe to halt the platform.

...

Back up the main platform elements

To perform a basic backup of the platform, run the following backups:

# On DB server, back up the Abiquo MySQL DBs
mysqldump --routines --triggers kinton            > kinton.sql-$(date +%Y%m%d_%H%M%S)
mysqldump --routines --triggers kinton_accounting > kinton_accounting.sql-$(date +%Y%m%d_%H%M%S)

# On the Monitoring server, back up Watchtower DB
mysqldump --routines --triggers watchtower            > watchtower.sql-$(date +%Y%m%d_%H%M%S)

# On the Abiquo API Server or UI server, back up the UI
tar cvfz /var/www/html/ui.tgz-$(date +%Y%m%d_%H%M%S) /var/www/html/ui

# If you are using SAML authentication with multiple IdPs, on ALL Abiquo API Servers, back up this file
cp 
/opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml-$(date
 +%Y%m%d_%H%M%S)

# On ALL servers, back up the /opt/abiquo folder
tar cvfz /opt/abiquo.tgz-$(date +%Y%m%d_%H%M%S) /opt/abiquo

# On Abiquo Monolithic or API Server, Remote Services, or Datanode, back up Redis
redis-cli save
cp -a /var/lib/redis /var/lib/redis-$(date +%Y%m%d_%H%M%S)

...

Disable old billing dashboards and delete last bills

In Abiquo 5.2, the billing dashboard feature is now in the core Abiquo platform.

If you are using the billing dashboard scripts from previous versions, disable them and delete last bills before you continue with the upgrade.

1. On the server where you were running the billing packages, remove the Cron jobs to run the billing scripts.

   Code Block
   # rm /etc/cron.d/abiquo-billing-ec2 # rm /etc/cron.d/abiquo-billing-arm

   
   

2. Delete the last bills from the previous version of the billing dashboards (from the kinton.billing_consolidation and kinton.billing_register tables). Remove the bills for Azure and Amazon providers, for the number of months covered by the abiquo.enterprise.property.billing.monthoffset property, which has a default value of 2 months. Later the new billing dashboards feature will regenerate these bills.

...

Stopplatform services

This section describes how to stop platform services on all servers.

To stop platform services:

1. Stop the API on the API server or Monolithic server

   Code Block
   systemctl stop abiquo-tomcat

2. Stop the UI on the API server or Monolithic server or dedicated UI server

   Code Block
   systemctl stop httpd

3. StopRemote services server

   Code Block
   systemctl stop abiquo-tomcat

4. Stop the database on the Monolithic server or Database server or Datanode server.

   Code Block
   systemctl stop mariadb

   For a datanode configuration, you will also need to stop the Galera cluster. For more details, see Stop and start HA configuration

5. Stop RabbitMQ (on the monolithic server or API Server or Datanode)

   Code Block
   systemctl stop rabbitmq-server

6. V2V server -
   You do not need to stop anything because the BPM remote service is run on-demand only

7. Stop monitoring server

   Code Block
   systemctl stop abiquo-delorean systemctl stop abiquo-emmett systemctl stop kairosdb systemctl stop cassandra

8. On the monitoring server, check if Cassandra is really dead

   Code Block
   ps auwx | grep cassandra

   Get the process number for Cassandra (the first number in the output of the previous command), and kill it. In this example, Cassandra is process 12345.

   Code Block
   kill -9 12345

All processes on platform servers should now be halted.

...

Make snapshots and backups of all platform servers

The upgrade to Abiquo 5.2.x is a major upgrade with upgrades of third-party software packages that are critical platform dependencies

You MUST make snapshots of all servers in your platform before you upgrade to Abiquo 5.2.x.

...

Prepare yum repositories on all servers

Prepare yum repositories on all servers for v5.2

...

Upgrade distributed API and DB servers

This section describes the upgrade for a standard API and DB server.

These steps are for a Distributed API Server with the database on the API server.

Abiquo will upgrade to new versions of RabbitMQ and MySQL that require a manual upgrade path of dependencies.

1. Remove the Erlang and Galera packages.

   Code Block
   yum remove erlang* galera yum clean all && yum makecache

2. Install new RabbitMQ and MariaDB server

   Code Block
   yum install rabbitmq-server MariaDB-server

   Before you confirm, check that the packages will be installed from the abiquo-base repository

   1. rabbitmq-server: 3.8.2.1

   2. MariaDB-server: 10.4.10.1

3. Upgrade Abiquo to 5.2.x

   Code Block
   yum update

   Before you confirm, check the following packages will be installed from the abiquo-base repository:

   1. jdk: 11.0.6u10

   2. redis: 5.0.7.1

4. After the update, check versions.

   Check that the symbolic link to the latest version of Java points to Java 11 on Monolithic Server or Datanode

   Code Block
   # cd /usr/java # ll latest -> /usr/java/jdk-11.0.6+10

   Check that the correct Tomcat version will be used on the Monolithic Server

   Code Block

   # cd /opt/abiquo/tomcat/bin # sh version.sh Using CATALINA_BASE: /opt/abiquo/tomcat Using CATALINA_HOME: /opt/abiquo/tomcat Using CATALINA_TMPDIR: /opt/abiquo/tomcat/temp Using JRE_HOME: /usr Using CLASSPATH: /opt/abiquo/tomcat/bin/bootstrap.jar:/opt/abiquo/tomcat/bin/tomcat-juli.jar NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED Server version: Apache Tomcat/9.0.33 Server built: Mar 11 2020 09:31:38 UTC Server number: 9.0.33.0 OS Name: Linux OS Version: 3.10.0-1160.15.2.el7.x86_64 Architecture: amd64 JVM Version: 11.0.6+10 JVM Vendor: AdoptOpenJDK

   On the Monolithic Server in the JDK folder, check certificate migration for AM download. The API Server certificate should be listed for Java 11 with its FQDN. You may need to enter the storepass option to supply the password and you can use the alias option to search for the hostname or FQDN of your server

   Code Block
   # cd /usr/java/jdk-11.0.6+10/lib/security # keytool -list -keystore cacerts -alias (hostname or fqdn)

   For example

   Code Block
   sudo keytool -list -keystore cacerts -alias abiquo.example.com

5. Enable and start the new services

   Code Block
   systemctl enable mariadb systemctl enable rabbitmq-server systemctl start mariadb systemctl start rabbitmq-server

6. Upgrade MySQL

   Code Block
   mysql_upgrade

   The mysql_upgrade step may detect errors that trigger messages such as the following:

   Code Block
   kinton_accounting.ACCOUNT_PERIOD_USAGE_VW Error : Column count of mysql.proc is wrong. Expected 21, found 20. Created with MariaDB 100129, now running 100410. Please use mysql_upgrade to fix this error

   These messages are expected and this same mysql_upgrade process will automatically fix these errors in its next stages.

   Check that the mysql_upgrade process completes correctly.

7. Check that your hostname is in your DNS or in your /etc/hosts file

8. Upgrade the Abiquo API databases

   Code Block
   abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

   If the liquibase update fails with a message similar to the following: 

   Code Block
   Liquibase update Failed: Validation Failed: 1 change sets check sum src/X.X.X-XX/kinton-X.X.X.xml::ABICLOUDPREMIUM-XXXX-XxxxxxxxXXxXxxxxxxXxxxxx::XXXXXXXXX is now: 7:ee2fa6e058ec76c7abf801567898917d For more information, use the --logLevel flag

   Do the following steps

   1. Clear the database checksums

      Code Block
      abiquo-db clearCheckSums

   2. Retry the above abiquo-db update command.

9. Change file owners to tomcat user

   Code Block
   chown -R tomcat /opt/abiquo

This section contains an alternative upgrade procedure for standalone Database servers and a Distributed API server without DB

...

Upgrade remote services servers

These steps are for remote services servers in a distributed or HA environment.

Upgrade EVERY remote services server with these steps.

1. On the remote services servers, upgrade Abiquo to 5.2.x

   Code Block
   yum update

   Before you confirm, check the following packages will be installed from the abiquo-base repository:

   1. jdk: 11.0.6u10

   2. redis: 5.0.7.1

2. After the update, check versions of Java and Tomcat

   Check that the symbolic link to the latest version of Java points to Java 11

   Code Block
   # cd /usr/java # ll latest -> /usr/java/jdk-11.0.6+10

   Check that the correct Tomcat version will be used

   Code Block

   # cd /opt/abiquo/tomcat/bin # sh version.sh Using CATALINA_BASE: /opt/abiquo/tomcat Using CATALINA_HOME: /opt/abiquo/tomcat Using CATALINA_TMPDIR: /opt/abiquo/tomcat/temp Using JRE_HOME: /usr Using CLASSPATH: /opt/abiquo/tomcat/bin/bootstrap.jar:/opt/abiquo/tomcat/bin/tomcat-juli.jar NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED Server version: Apache Tomcat/9.0.33 Server built: Mar 11 2020 09:31:38 UTC Server number: 9.0.33.0 OS Name: Linux OS Version: 3.10.0-1160.15.2.el7.x86_64 Architecture: amd64 JVM Version: 11.0.6+10 JVM Vendor: AdoptOpenJDK

   On the remote services server in the JDK folder, check certificate migration to allow AM download

   The API server certificate should be listed for Java 11 with its FQDN. You may need to enter the storepass option to supply the password and you can use the alias option to search for the hostname or FQDN of your server

   Code Block
   # cd /usr/java/jdk-11.0.6+10/lib/security # keytool -list -keystore cacerts -alias (hostname or fqdn)

   For example

   Code Block
   sudo keytool -list -keystore cacerts -alias abiquo.example.com

3. On all remote services servers using SSL

   Edit the Tomcat server configuration file:

   Code Block
   /opt/abiquo/tomcat/conf/server.xml

   Find the location of the certificate files for Java 8

   Code Block
   keystoreFile="/usr/java/default/jre/lib/security/cacerts"

   Change it to the location for Java 11 (e.g. remove "/jre")

   Code Block
   keystoreFile="/usr/java/default/lib/security/cacerts"

4. On all remote services servers, change file owners to tomcat user

   Code Block
   chown -R tomcat /opt/abiquo

5. On the remote services with appliance manager and the V2V server (i.e. servers that mount the NFS repository), change file owners to Tomcat user

   Code Block
   chown -R tomcat /opt/vm_repository chmod -R a+r /opt/vm_repository

   
   

...

Upgrade monitoring server

This step is for Watchtower monitoring servers in all installations.

1. To continue using Java 8 (for Cassandra), remove the JDK through the package manager. If you use yum, it will be delete all the dependencies and you will have to reinstall them

   Code Block
   rpm -e --nodeps jdk

2. Check the jdk to install is version 8

   Code Block
   rpm -qa | grep jdk

3. Check that Java is correctly installed:

   1. If you execute this command and get that error, please execute the command from the next bullet:

      Code Block
      $ java -bash: java: command not found

   2. Fix it by executing this command, then selecting the option "java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/bin/java)":

      Code Block
      $ sudo alternatives --config java There are 3 programs which provide 'java'. Selection Command ----------------------------------------------- 1 /usr/java/jdk1.8.0_144/bin/java + 2 /usr/java/jdk-11.0.6+10/bin/java * 3 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/bin/java) Enter to keep the current selection[+], or type selection number: 3

4. As for the Abiquo DB steps, remove Galera, which will also remove MariaDB

   Code Block
   yum remove galera

5. Clean yum and make cache

   Code Block
   yum clean all && yum makecache

6. Install MariaDB

   Code Block
   yum install MariaDB-server

7. Update Abiquo

   Code Block
   yum update

8. Check that the Java package is now "javajdk" to use Java 1.8.0u144 with Cassandra:

   Code Block
   # rpm -qa 'javajdk*' javajdk-1.8.0u144-6691.el7.x86_64

9. If you have a HA datanode that runs the watchtower database, skip directly to Step 10 to update watchtower database

10. Enable and start the services that were reinstalled

    Code Block
    systemctl enable mariadb systemctl start mariadb

11. Upgrade MySQL

    Code Block
    mysql_upgrade

    The mysql_upgrade step may detect errors that trigger messages such as the following message (note: this example is from the Abiquo database server).

    Code Block
    kinton_accounting.ACCOUNT_PERIOD_USAGE_VW Error : Column count of mysql.proc is wrong. Expected 21, found 20. Created with MariaDB 100129, now running 100410. Please use mysql_upgrade to fix this error

    These messages are expected and this same mysql_upgrade process will automatically fix these errors in its next stages.

    Check that the mysql_upgrade process completes correctly.

12. Update the Abiquo Watchtower database

    Code Block
    watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

...

Upgrade steps from 4.7.x to 5.0.x

Start with your original version and perform the steps to the final version.

# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

...

Upgrade steps from 5.0.x to 5.1.x

Start with your original version and perform the steps to the final version.

/usr/share/doc/abiquo-model/scripts/veeam-job-modifier/README.md

/usr/share/doc/abiquo-model/scripts/wt-db-redis-db-migration-tool/README.md

# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

...

Upgrade steps from 5.1.x to 5.2.x

Start with your original version and perform the steps to the final version.

When upgrading from 5.1.0 or 5.1.1 to 5.1.2 or above, follow the steps from the Upgrade steps from 5.0.x to 5.1.x block.

...