Page Comparison

...

Before you shut down the platform servers you should check that no operations are in progress .

...

on the platform.

1. Check

...

Back up the main platform elements

...

Stop platform services

...

Make snapshots and backups of all platform servers

This is a major upgrade, so we recommend that you make a snapshot and/or a backup of your platform servers.

Prepare yum repositories for 5.4

...

Prepare yum repositories for 5.4.x

...

Upgrade packages on ALL servers

These instructions are for monolithic, distributed, and HA environments.

Check in your RS server(s) if there is an NFS shared folder for the NARS results (related to the system property abiquo.nars.async.fileresults.path):

$ mount | grep -i "nfs\|results" | grep -v sunrpc
10.60.37.87:/abq-pro-nfs on /opt/abiquo/results-nars type nfs4 (rw,nosuid,nodev,noexec,relatime,sync,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.60.37.77,local_lock=none,addr=10.60.37.87,user)

Unmount the previous shared folder if you have it mounted:

sudo umount /opt/abiquo/results-nars

Update Abiquo packages:

yum clean all && yum makecache && yum update 'abiquo-*'

Mount again the shared folder from the second step.

sudo mount -a

The Abiquo services must run as the tomcat user (not root), so set the required permissions and enforce the use of the package configuration files.
If you have a MONOLITHIC SERVER do all of the following steps on the Monolithic server

On Abiquo Server and Remote Services

chown -R tomcat /opt/abiquo

On the Remote Services with Appliance Manager and the V2V Server (i.e. servers that mount the NFS repository)

chown -R tomcat /opt/vm_repository
chmod -R a+r /opt/vm_repository

On the V2V server, check the /etc/sysconfig/abiquo/ec2-api-tools file exists.
The file must contain the following configuration. If the file does not exist, create it and add this configuration.

For CentOS 7

EC2_HOME=/opt/aws

Update the Abiquo databases

...

Check that your hostname is in your DNS or in your /etc/hosts file

Upgrade the Abiquo API databases

abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

If the liquibase update fails with a message similar to the following: 

Liquibase update Failed: Validation Failed:
1 change sets check sum
src/X.X.X-XX/kinton-X.X.X.xml::ABICLOUDPREMIUM-XXXX-XxxxxxxxXXxXxxxxxxXxxxxx::XXXXXXXXX is now: 7:ee2fa6e058ec76c7abf801567898917d
For more information, use the --logLevel flag

Do the following steps:

Clear the database checksums

abiquo-db clearCheckSums

...

Retry the above abiquo-db update command.

To upgrade the Abiquo Watchtower database, on the monitoring appliance, run the command below:

watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

...

Reporting changes: To upgrade the Abiquo Reports Server for the upgrade to Abiquo 5.3.x, contact Abiquo Support for more information.

• The Abiquo Reports Server now runs with JasperReports v7.8.0

Perform the provider ID migration in an upgrade through 5.3 only

For an upgrade that passes through Abiquo 5.3, perform the provider ID migration using the PIM tools.

Please note that the PIM migration tools use Java version 11.

Run the PIM planner

...

Run the PIM migrator

...

Upgrade additional elements

By default, you will find the upgrade scripts (e.g. for Redis definitions) on the Abiquo Server under the folder: /usr/share/doc/abiquo-redis/redis/

For each upgrade step, run the Redis database script to remove old VSM definitions, which can be found in the 4.2.3 subfolder

On Remote Services server:

# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

 If you have custom billing queries, please update them now.

...

...

...

...

Configure Abiquo after the upgrade

...

Before you start the Abiquo tomcat server, add Abiquo configuration properties to the abiquo.properties file.
By default the abiquo.properties file is found in the /opt/abiquo/config/ folder.
See Changes to Abiquo configuration properties

Configure the user interface. The default UI location is /var/www/html/ui.
Optional: Add custom labels and translations in the lang_xx_XX_custom.json files in the lang folder
Add custom configuration to client-config-custom.json. See Configure Abiquo UI
If your API is not in the same domain as the UI, set the API endpoint pointing to your Abiquo API server:

{
    "config.endpoint": "http://myAPIserver/api"
}

...

SAML: If you are using SAML with multiple IDPs, restore the configuration as described at Restore SAML security beans after upgrade.

...

Reporting changes: To upgrade the Abiquo Reports Server for the upgrade to Abiquo 5.3.x, contact Abiquo Support.

Start Abiquo server and services

...

1. that the Abiquo RabbitMQ queues are empty on the Abiquo Monolithic Server, Abiquo Server or Datanode server

   The number of messages in all queues must be 0.

   Code Block
   # rabbitmqctl list_queues messages name

   Code Block

   # rabbitmqctl list_queues messages name Listing queues ... 0 abiquo.am.notifications 0 abiquo.bpm.notifications 0 abiquo.datacenter.requests.ADatacenter.bpm 0 abiquo.datacenter.requests.ADatacenter.virtualfactory 0 abiquo.ha.tasks 0 abiquo.nodecollector.notifications 0 abiquo.pcrsync.messages 0 abiquo.pcrsync.parking-expect-no-consumers 0 abiquo.scheduler.fast.requests 0 abiquo.scheduler.requests 0 abiquo.scheduler.slow.requests 0 abiquo.tracer.traces 0 abiquo.virtualfactory.notifications 0 abiquo.virtualmachines.definitionsyncs 0 abiquo.vsm.eventsynk ...done.

2. On the V2V Server, check for any active conversions by checking for the V2V or Mechadora processes

   Code Block
   $ ps aux | grep v2v $ ps aux | grep mechadora

When user VM operations are blocked and all of the above checks show that no tasks are running, it is safe to halt the platform.

...

Back up the main platform elements

To perform a basic backup of the platform, run the following backups:

# On DB server, back up the Abiquo MySQL DBs
mysqldump --routines --triggers kinton            > kinton.sql-$(date +%Y%m%d_%H%M%S)
mysqldump --routines --triggers kinton_accounting > kinton_accounting.sql-$(date +%Y%m%d_%H%M%S)

# On the Monitoring server, back up Watchtower DB
mysqldump --routines --triggers watchtower            > watchtower.sql-$(date +%Y%m%d_%H%M%S)

# On the Abiquo API Server or UI server, back up the UI
tar cvfz /var/www/html/ui.tgz-$(date +%Y%m%d_%H%M%S) /var/www/html/ui

# If you are using SAML authentication with multiple IdPs, on ALL Abiquo API Servers, back up this file
cp 
/opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml-$(date
 +%Y%m%d_%H%M%S)

# On ALL servers, back up the /opt/abiquo folder
tar cvfz /opt/abiquo.tgz-$(date +%Y%m%d_%H%M%S) /opt/abiquo

# On Abiquo Monolithic or API Server, Remote Services, or Datanode, back up Redis
redis-cli save
cp -a /var/lib/redis /var/lib/redis-$(date +%Y%m%d_%H%M%S)

...

Stop platform services

This section describes how to stop platform services on all servers.

To stop platform services:

1. Stop the API on the API server or monolithic server

   Code Block
   systemctl stop abiquo-tomcat

2. Stop the UI on the API server or monolithic server or dedicated UI server

   Code Block
   systemctl stop httpd

3. Stop remote services server

   Code Block
   systemctl stop abiquo-tomcat

4. V2V Server
   You do not need to stop anything because the BPM remote service is run on-demand only

5. Stop monitoring server

   Code Block
   systemctl stop abiquo-delorean systemctl stop abiquo-emmett systemctl stop kairosdb systemctl stop cassandra

6. On the monitoring server, check if Cassandra is really dead

   Code Block
   ps auwx | grep cassandra

   Get the process number for Cassandra (the first number in the output of the previous command), and kill it. In this example, Cassandra is process 12345.

   Code Block
   kill -9 12345

All processes on platform servers should now have halted.

...

Make snapshots and backups of all platform servers

This is a major upgrade, so we recommend that you make a snapshot and/or a backup of your platform servers.

...

Prepare yum repositories for 5.4

Check that you have the repository URL and credentials

1. To upgrade to a version with a patch number of zero, for example, version 5.4.0

   1. Remove the previous version Abiquo release packages.

      Code Block
      yum remove 'abiquo-release-*'

   2. Locate the corresponding abiquo-release-ee package in the list of available versions

   3. Install the release package. For Abiquo 5.4.0, the command would be similar to the following

      Code Block
      yum localinstall http://user:passwd@mirror.abiquo.com/el7/5.4/os/x86_64/abiquo-release-ee-5.4.0-XXXX.el7.noarch.rpm

   4. Disable the updates repo. See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-managing_yum_repositories

Prepare yum repositories for 5.4.x

1. Check that you have the repository URL and credentials

2. To upgrade to the latest version with a patch number that is not zero, for example, version 6.0.x or version 6.1.x, enable both repositories:

   Code Block
   yum-config-manager --enable abiquo-base yum-config-manager --enable abiquo-updates

3. Optionally add your username and password to the Abiquo repos

   Code Block
   yum-config-manager --save --setopt=abiquo-*.username=MYUSERNAME yum-config-manager --save --setopt=abiquo-*.password=MYPASSWORD

   Don't forget to use a backslash to escape any shell special characters. For more details, see https://www.oreilly.com/library/view/learning-the-bash/1565923472/ch01s09.html

4. Clean yum and make cache

   Code Block
   yum clean all && yum makecache

5. If you did not make snapshots of all servers already, then you could do this now.

...

Upgrade packages on ALL servers

These instructions are for monolithic, distributed, and HA environments.

1. Check in your RS server(s) if there is an NFS shared folder for the NARS results (related to the system property abiquo.nars.async.fileresults.path):

   Code Block
   language bash
   $ mount | grep -i "nfs\|results" | grep -v sunrpc 10.60.37.87:/abq-pro-nfs on /opt/abiquo/results-nars type nfs4 (rw,nosuid,nodev,noexec,relatime,sync,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.60.37.77,local_lock=none,addr=10.60.37.87,user)

   
   

2. Unmount the previous shared folder if you have it mounted:

   Code Block
   language bash
   sudo umount /opt/abiquo/results-nars

   
   

3. Update Abiquo packages:

   Code Block
   yum clean all && yum makecache && yum update 'abiquo-*'

   
   

4. Mount again the shared folder from the second step.

   Code Block
   language bash
   sudo mount -a

   
   

5. The Abiquo services must run as the tomcat user (not root), so set the required permissions and enforce the use of the package configuration files.
   If you have a MONOLITHIC SERVER do all of the following steps on the Monolithic server

   1. On Abiquo Server and Remote Services

      Code Block
      chown -R tomcat /opt/abiquo

      
      

   2. On the Remote Services with Appliance Manager and the V2V Server (i.e. servers that mount the NFS repository)

      Code Block
      chown -R tomcat /opt/vm_repository chmod -R a+r /opt/vm_repository

      
      

   3. On the V2V server, check the /etc/sysconfig/abiquo/ec2-api-tools file exists.
      The file must contain the following configuration. If the file does not exist, create it and add this configuration.

      For CentOS 7

      Code Block
      EC2_HOME=/opt/aws

      
      

Update the Abiquo databases

1. Check that your hostname is in your DNS or in your /etc/hosts file

2. Upgrade the Abiquo API databases

   Code Block
   abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

   
   

3. If the liquibase update fails with a message similar to the following: 

   Code Block
   Liquibase update Failed: Validation Failed: 1 change sets check sum src/X.X.X-XX/kinton-X.X.X.xml::ABICLOUDPREMIUM-XXXX-XxxxxxxxXXxXxxxxxxXxxxxx::XXXXXXXXX is now: 7:ee2fa6e058ec76c7abf801567898917d For more information, use the --logLevel flag

   Do the following steps:

   1. Clear the database checksums

      Code Block
      abiquo-db clearCheckSums

      
      

   2. Retry the above abiquo-db update command.

   
   

4. To upgrade the Abiquo Watchtower database, on the monitoring appliance, run the command below:

   Code Block
   watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

   
   

5. Reporting changes: To upgrade the Abiquo Reports Server for the upgrade to Abiquo 5.3.x, contact Abiquo Support for more information.

   • The Abiquo Reports Server now runs with JasperReports v7.8.0

Perform the provider ID migration in an upgrade through 5.3 only

For an upgrade that passes through Abiquo 5.3, perform the provider ID migration using the PIM tools.

Please note that the PIM migration tools use Java version 11.

Run the PIM planner

The PIM planner connects to the Abiquo kinton database and extracts the data for the PIM migrator tool.

• The PIM planner will process each deployed VM to:

  • Compute the provider ID

  • Define the rename of any backup jobs or groups

  • Define the rename of any firewalls and load balancers

The PIM planner doesn't make any changes to the platform but you must run it after you upgrade the database to Abiquo 5.3.

What is the output of the PIM planner?

• A log in standard output (you should redirect it to a file)

• A migration plan in ZIP format called migration-plan.data

PIM planner requirements

The PIM planner requires access to the MariaDB kinton schema via the JDBC URL. 

• It encrypts the database credentials for runtime use only; it does not save or export credentials

Before you run the PIM planner

1. Start the Abiquo upgrade to 5.3 as usual including

   1. Stop the Abiquo platform

   2. Create backups including VM snapshots, Database dump, and Redis dump

2. Upgrade the database

3. STOP the upgrade

Run the PIM planner

To run the PIM planner do these steps.

1. Run the Abiquo upgrade to 5.3 and stop after the database upgrade

2. On the Abiquo API Server, install the Abiquo PIM tools package

   Code Block
   yum install abiquo-pim-tools

   By default, it should install in the /opt/abiquo/pim-tools folder

3. Run the PIM planner against your database.

   We strongly reccommend that you redirect the log in the standard output of the PIM planner to a file.

   For example:

   Code Block
   java -jar /opt/abiquo/pim-tools/pimplanner.jar --jdbc-url="jdbc:mysql://my.database.server:3306/kinton?autoReconnect=true&user=root&password=root" | tee output_pimplanner.log

4. Check the output file from the PIM planner
    If there are any warnings or errors, resolve them before you continue. 
   If you have any doubts, please contact Abiquo Support.

5. For ALL Remote Services servers

   1. Copy the migration-plan.data file to each Remote Services server

   Next, follow the instructions to run the PIM migrator on each Remote Services server

PIM planner options

...

Run the PIM migrator

The PIM migrator performs the provider ID migration in the compute, backup, and networking providers via the Remote Services servers.

• It updates the VMs, backups, firewalls, and load balancers in the providers and in Redis

• It also tests the migration in dry run mode

What does it output?

• An SQL file to run on the kinton database

• A log in standard output (you should redirect it to a file)

PIM migrator requirements

The PIM migrator requires the following.

• The migration plan from the PIM planner

• Access to:

  • the Redis instance of the Remote Service server.

  • the hypervisors in the datacenter

Before you run the PIM migrator

Do these steps.

1. Start the Abiquo upgrade to 5.3 as usual including

   1. Stop the Abiquo platform

   2. Create backups including snapshots, Database dump, and Redis dump

2. Upgrade the database

3. STOP the upgrade

4. Run the PIM planner and obtain the migration plan

Run the PIM migrator

Do these steps to run the PIM migrator.

Log in to ALL remote services servers and do these steps on EACH server. 

1. Install the PIM tools

   Code Block
   yum install abiquo-pim-tools

   The default install folder is /opt/abiquo/pim-tools

2. Obtain the datacenter-id of the Remote Services server from the value of the abiquo.datacenter.id property in the abiquo.properties file.

   Code Block
   abiquo.datacenter.id=abq_dc1

   In this case, the value of the datacenter-id parameter will be "abq_dc1"

3. Run the PIM migrator in dry run mode, which is the default mode that doesn't make any changes.

   The "-noseed" parameter is required. The default value is false, which means the migrator will use the platform's default seed file. You can specify a seed file with the "-seed" parameter. 

   We recommend that you save the log in standard output to a file. And we recommend that you give the output files names that will identify the Remote Services server.

   For example 

   Code Block
   java -jar /opt/abiquo/pim-tools/pimmigrator.jar -dc=abq_dc1 -redishost=localhost -plan=migration-plan.data -noseed -output=update_DC1.sql | tee pimmigrator_dry_run_DC1.log

4. Check the output file. If there are any errors or warnings, resolve them.
   If necessary, contact Abiquo Support.

5. Run the PIM migrator in update mode, by setting the "no dry run" option to true.

   Code Block
   java -jar /opt/abiquo/pim-tools/pimmigrator.jar -nodry -dc=abq_dc1 -redishost=localhost -plan=migration-plan.data -noseed -output=update_DC1.sql | tee pimmigrator_DC1.log

6. Copy the SQL file from the remote services server to the Abiquo database server.

   For example

   Code Block
   scp update_DC1.sql root@my.database.server:~/

After you run the migrator on ALL Remote Services servers, run the SQL upgrades on the database server as described in the next step. 

Update the Abiquo database server

1. Update the Abiquo database with all of the update.sql files. For example

   Code Block
   mysql kinton < update_DC1.sql mysql kinton < update_DC2.sql ...

Now continue with the Abiquo upgrade

PIM migrator options

PIM migrator notes

The migrator does not process the following providers.

• Hyper-V

• Networker

• Google Cloud Platform (see note below)

• Firewalls and load balancers that are not in NSX

For Google Cloud Platform, if you have VMs deployed before the upgrade, after you apply the PIM tools, do these steps:

1. Delete the VMs from the platform only 

   • in the API, delete them with the "logicaldelete" parameter set to true. See Delete a VM API doc 

2. Delete the virtual datacenters

3. Onboard the resources again

...

Upgrade additional elements

By default, you will find the upgrade scripts (e.g. for Redis definitions) on the Abiquo Server under the folder: /usr/share/doc/abiquo-redis/redis/

For each upgrade step, run the Redis database script to remove old VSM definitions, which can be found in the 4.2.3 subfolder

On Remote Services server:

# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

 If you have custom billing queries, please update them now.

# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

Configure Abiquo after the upgrade

1. Before you start the Abiquo tomcat server, add Abiquo configuration properties to the abiquo.properties file.
   By default the abiquo.properties file is found in the /opt/abiquo/config/ folder.
   See Changes to Abiquo configuration properties

2. Configure the user interface. The default UI location is /var/www/html/ui.
   Optional: Add custom labels and translations in the lang_xx_XX_custom.json files in the lang folder
   Add custom configuration to client-config-custom.json. See Configure Abiquo UI
   If your API is not in the same domain as the UI, set the API endpoint pointing to your Abiquo API server:

   Code Block
   { "config.endpoint": "http://myAPIserver/api" }

   
   

3. SAML: If you are using SAML with multiple IDPs, restore the configuration as described at Restore SAML security beans after upgrade.

4. Reporting changes: To upgrade the Abiquo Reports Server for the upgrade to Abiquo 5.3.x, contact Abiquo Support.

Start Abiquo server and services

To start the Abiquo platform servers and services, do these steps:

1. On the Abiquo server, restart the HTTP daemon to refresh the user interface files, and bring up the Tomcat server.

   Code Block
   service httpd start service abiquo-tomcat start

2. On the Remote services server, start the Tomcat server

   Code Block
   service abiquo-tomcat start

3. On the Monitoring server do these steps.

   1. Edit the file /opt/kairosdb/conf/kairosdb.properties to update the name of the following variable and to remove the port from it:

      1. Replace the line kairosdb.datastore.cassandra.host_list=192.168.888.999:9160.

      2. With this line kairosdb.datastore.cassandra.cql_host_list=192.168.888.999. Please note the new cql_ prefix for host_list.

   2. Edit the file /etc/cassandra/conf/cassandra.yml OR /etc/cassandra/default.conf/cassandra.yaml (whichever exists) to remove a variable:

      1. Remove the line starting with kairosdb.datastore.cassandra.datapoint_ttl and save and close the file.

   3. Edit the file /opt/kairosdb/conf/kairosdb.properties to add a new variable (ref: Internal JIRA SUP-333):

      1. Add kairosdb.datastore.cassandra.datapoint_ttl = 15768000

4. On the Monitoring server, start the Cassandra service

   Code Block
   sudo service cassandra start

5. WAIT about 5 minutes until the service is up and running

6. Start the KairosDB service

   Code Block
   sudo service kairosdb start

7. Start the other services in this order

   Code Block
   sudo service abiquo-emmett start sudo service abiquo-delorean start

8. On the V2V server: restart the Tomcat server:

   Code Block
   service abiquo-tomcat restart

9. In Abiquo, re-enable the physical machines!

10. Clear your browser cache to prevent glitches in the UI