Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

For your convenience, we have also prepared the following concise upgrade document that groups the commands by servers.
However, we encourage you to use the current Upgrade Abiquo document as your main reference:

Please contact Abiquo Support for further information.

...

From version

To version

Upgrade type

Notes

6.1.x

6.2.x

StandardMajor

Follow the process in this document (or at Upgrade Abiquo 6.1.x to 6.2.x distributed by servers

  • Upgrade Abiquo packages

  • Run the guestpasswordmigrator

  • Upgrade mariadb on the main DB and watchtower DB

  • Apply the guestpasswords.sql file to the main DB

  • For Galera clusters, update the cluster details for the database connection

  • To display cost-usage reports, run regenerate billing

...

  • Before you begin, contact Abiquo Platform and Customer Success department to obtain your credentials because for Abiquo YUM repositories require a login.

Warning

Major upgrade

The upgrade to Abiquo 6.2.0 is a major upgrade with upgrades of third-party software packages that are critical platform dependencies

You must make snapshots of ALL servers before you start the upgrade

  • The upgrade will overwrite lang_en_US_labels.json. If you have Back up your customized labels or translations , back them up before the upgrade, and restore them afterwards.

Note

SAML upgrade

  • The upgrade will overwrite the SAML login configuration for multiple IdPs. If you use SAML with multiple IdPs, back Back up the configuration files before the upgrade, and add the multiple IdPs again afterwards accordingly. See Restore SAML security beans after upgrade.

  • For Abiquo 6.0.0+, you must configure the SAML maximum authentication age property or SAML will not initialize

...

3. Prevent cloud users from performing VM operations

  1. In the UI in the Infrastructure view, select each physical machine and click Disable

  2. Using the API, set the state of each physical machine to DISABLED

...

Code Block
languagebash
# On DB server, back up the Abiquo MySQLAPI DBsServer mysqldumpor --routines --triggers kinton            > kinton.sqlUI server, back up the UI
tar cvfz /var/www/html/ui.tgz-$(date +%Y%m%d_%H%M%S) mysqldump --routines --triggers kinton_accounting > kinton_accounting.sql-$(date +%Y%m%d_%H%M%S)

# On the Monitoring server, back up Watchtower DB
mysqldump --routines --triggers watchtower            > watchtower.sql/var/www/html/ui

# Copy the api.xml file to back up the database connection details
cp /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml /opt/abiquo/api.xml-$(date +%Y%m%d_%H%M%S)

# On theALL Abiquoservers, APIback Serverup or UI server, back up the UI
the /opt/abiquo folder
tar cvfz /var/www/html/uiopt/abiquo.tgz-$(date +%Y%m%d_%H%M%S) /var/www/html/uiopt/abiquo

# IfOn youAbiquo areMonolithic usingor SAMLAPI authenticationServer, withRemote multiple IdPsServices, on ALL Abiquo API Serversor Datanode, back up thisRedis
fileredis-cli save
cp -a /optvar/abiquolib/tomcatredis /webappsvar/apilib/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml-$(date
redis-$(date +%Y%m%d_%H%M%S)

# On DB server, back up the Abiquo MySQL DBs
mysqldump --routines --triggers kinton            > kinton.sql-$(date +%Y%m%d_%H%M%S)
mysqldump # Copy the api.xml file to back up the database connection details
cp /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml /opt/abiquo/api.xml--routines --triggers kinton_accounting > kinton_accounting.sql-$(date +%Y%m%d_%H%M%S)

# On ALLthe Monitoring serversserver, back up the /opt/abiquo folder
tar cvfz /opt/abiquo.tgzWatchtower DB
mysqldump --routines --triggers watchtower            > watchtower.sql-$(date +%Y%m%d_%H%M%S) /opt/abiquo

# On Abiquo Monolithic or API Server, Remote Services, or Datanode

# If you are using SAML authentication with multiple IdPs, on ALL Abiquo API Servers, back up Redisthis redis-clifile
save
cp -a /var/lib/redis /var/lib/redis-$(date +%Y%m%d_%H%M%S/opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml-$(date
 +%Y%m%d_%H%M%S)

...

7. Make snapshots and backups of all platform servers

This is a major upgrade, so we recommend that you make a snapshot and/or a full backup of your platform servers.

...

8.

...

Check that you have the repository URL and credentials

On all Abiquo Servers, such as Abiquo API, Remote Services, Watchtower monitoring.

Install release package

To upgrade to a version with a patch number of zero, for example, version 6.2.0

...

Remove the previous version Abiquo release packages.

Code Block
yum remove 'abiquo-release-*'

...

Find the abiquo-release-ee package for your version in the list of available versions

Install the release package.
For Abiquo 6.2.0, the command would be similar to the following:

Code Block
yum localinstall http://usr:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-release-ee-6.2.0-20240205_155724.el7.noarch.rpm

For Abiquo 6.2.1, the release package would be in the format:

Code Block
http://usr:passwd@mirror.abiquo.com/el7/6.2/updates/x86_64/abiquo-release-ee-6.2.1-BBBBBBBBBBBB.el7.noarch.rpm

...

For major versions, disable the updates repo. See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-managing_yum_repositories

Remove Docker and unmount shared folders

  1. On your RS server(s) uninstall Docker

    Code Block
    rpm -e --nodeps docker
    rpm -e --nodeps docker-client
    rpm -e --nodeps docker-common
  2. If you have multiple instances of the NARS Remote Service in your datacenter, check on your RS servers for an NFS shared folder for the NARS results (with the Abiquo configuration property abiquo.nars.async.fileresults.path):

    Code Block
    languagebash
    $ mount | grep -i "nfs\|results" | grep -v sunrpc
    10.60.37.87:/abq-pro-nfs on /opt/abiquo/results-nars type nfs4 (rw,nosuid,nodev,noexec,relatime,sync,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.60.37.77,local_lock=none,addr=10.60.37.87,user)
  3. Unmount the shared folder. You will remount it after the upgrade

    Code Block
    languagebash
    sudo umount /opt/abiquo/results-nars

...

9. Prepare yum repositories for 6.2 or 6.2.x

...

Check that you have the repository URL and credentials

...

On all Abiquo Servers, such as Abiquo API, Remote Services, Watchtower monitoring.

Install release package

  1. To upgrade to a version with a patch number that is not of zero, for example, version 6.2.x, enable both repositories:.0

    1. Remove the previous version Abiquo release packages.

      Code Block
      yum
    -config-manager --enable abiquo-base yum-config-manager --enable abiquo-updates
  2. Optionally add your username and password to the Abiquo repos

    Code Block
    yum-config-manager --save --setopt=abiquo-*.username=MYUSERNAME
    yum-config-manager --save --setopt=abiquo-*.password=MYPASSWORD

    Don't forget to use a backslash to escape any shell special characters. For more details, see https://www.oreilly.com/library/view/learning-the-bash/1565923472/ch01s09.html

  3. Clean yum and make cache

    Code Block
    yum clean all && yum makecache
  4. If you did not make snapshots of all servers already, then you could do this now.

9. Upgrade packages on ALL servers

These instructions are for monolithic, distributed, and HA environments.

  1. Check in your RS server(s) if there is an NFS shared folder for the NARS results (related to the Abiquo configuration property abiquo.nars.async.fileresults.path):

    Code Block
    languagebash
    $ mount | grep -i "nfs\|results" | grep -v sunrpc
    10.60.37.87:/abq-pro-nfs on /opt/abiquo/results-nars type nfs4 (rw,nosuid,nodev,noexec,relatime,sync,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.60.37.77,local_lock=none,addr=10.60.37.87,user)
  2. Unmount the previous shared folder if you have it mounted:

    Code Block
    languagebash
    sudo umount /opt/abiquo/results-nars
  3. Also on your RS server(s) uninstall Docker

    Code Block
    yum remove docker
  4. Update all packages:

    Code Blockyum
    1.  remove 'abiquo-release-*'
    2. Find the abiquo-release-ee package for your version in the list of available versions

    3. Install the release package.
      For Abiquo 6.2.0, the command would be similar to the following:

      Code Block
      yum localinstall http://usr:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-release-ee-6.2.0-AAAAAAAA_BBBBBBB.el7.noarch.rpm

      For Abiquo 6.2.1, the release package would be in the format:

      Code Block
      http://usr:passwd@mirror.abiquo.com/el7/6.2/updates/x86_64/abiquo-release-ee-6.2.1-BBBBBBBBBBBB.el7.noarch.rpm
    4. For major versions, disable the updates repo.

      Code Block
      yum-config-manager --disable abiquo-updates

Prepare yum repositories for 6.2.x (minor version)

  1. Check that you have the repository URL and credentials

  2. To upgrade to the latest version with a patch number that is not zero, for example, version 6.2.x, enable both repositories:

    Code Block
    yum-config-manager --enable abiquo-base
    yum-config-manager --enable abiquo-updates
  3. Optionally, add your username and password to the Abiquo repos

    Code Block
    yum-config-manager --save --setopt=abiquo-*.username=MYUSERNAME
    yum-config-manager --save --setopt=abiquo-*.password=MYPASSWORD

    Don't forget to use a backslash to escape any shell special characters. For more details, see https://www.oreilly.com/library/view/learning-the-bash/1565923472/ch01s09.html

  4. Clean yum and make cache

    Code Block
    yum clean all && yum makecache && yum update 

    (warning) The MariaDB update requires a manual install. We will update it manually later

  5. Mount the shared folder from the second step again.

    Code Block
    languagebash
    sudo mount -a
  6. The Abiquo services must run as the tomcat user (not root), so set the required permissions and enforce the use of the package configuration files.
    If you have a MONOLITHIC SERVER do all of the following steps on the Monolithic server

  7. On Abiquo server and remote services

    Code Block
    chown -R tomcat /opt/abiquo
  8. On the remote services with appliance manager and the V2V server (i.e. servers that mount the NFS repository)
  9. If you did not make snapshots of all servers already, then you could do this now.

...

10. Upgrade packages on ALL servers

These instructions are for monolithic, distributed, and HA environments.

  1. Update all packages.

    Code Block
    yum clean all && yum makecache && yum update 

    (warning) The MariaDB update requires a manual install. But first download and run the gpmigrator tool

  2. If you earlier unmounted shared folders from your Remote Services server, mount this folder again (and any other folders!)

    Code Block
    languagebash
    sudo mount -a
  3. The Abiquo services must run as the tomcat user (not root), so set the required permissions and enforce the use of the package configuration files.
    If you have a MONOLITHIC SERVER do all of the following steps on the Monolithic server

    1. On Abiquo server and remote services

      Code Block
      chown -R tomcat /opt/vm_repository
      chmod -R a+r /opt/vm_repository
      On the
      abiquo
    2. On the remote services with appliance manager and the V2V server (i.e. servers that mount the NFS repository)

      Code Block
      chown -R tomcat /opt/vm_repository
      chmod -R a+r /opt/vm_repository
    3. On the V2V server, check the /etc/sysconfig/abiquo/ec2-api-tools file exists.
      The file must contain the following configuration. If the file does not exist, create it and add this configuration.

      For CentOS 7

      Code Block
      EC2_HOME=/opt/aws

...

...

11. Run the guest password migrator

  1. Download and install the guestpasswordmigrator gpmigrator packageRun the tool

    Code Block
    yum localinstall http://user:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-gpmigrator-6.2.0-20240205_155724.el7.noarch.rpm
  2. Run the tool to migrate the guest passwords of the current VMs to a new column that will be added in 6.2.0.

    1. The tool requires access to the kinton database and to the API Redis

    2. Usage:

      Code Block
      --jdbc-url VAL    : Replace VAL with Kinton JDBC URL including user and password
      --output-file VAL : Replace VAL with the output file (by default: 'guestpasswords.sql') 
      --redis-host VAL  : Redis host to connect
      --redis-port N    : Redis port to connect (by default: 6379)

      For example:

      Code Block
      java -jar gpmigrator.jar --jdbc-url="jdbc:mariadb://localhost:3306/kinton?user=root&password=root" --redis-host=localhost
  3. Save the guestpasswords.sql file and run it (or whatever you called it) to run after you update the Abiquo database with abiquo-db.

...

12. Upgrade MariaDB for Abiquo 6.2.0

  1. On all database servers, including the Watchtower server, upgrade MariaDB

    Code Block
    rpm --nodeps -ev MariaDB-server-10.4.10-1.el7.centos.x86_64
    yum install MariaDB-server
    service mariadb start
    # OR for datanode: galera_new_cluster 
    mysql_upgrade 

...

13. Update the Abiquo databases

  1. Check that your hostname is in your DNS or in your /etc/hosts file

  2. Upgrade the Abiquo API databases

    Code Block
    abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update
  3. If the Liquibase database update fails with a message similar to the following: 

    Code Block
    Liquibase update Failed: Validation Failed:
    1 change sets check sum
    src/X.X.X-XX/kinton-X.X.X.xml::ABICLOUDPREMIUM-XXXX-XxxxxxxxXXxXxxxxxxXxxxxx::XXXXXXXXX is now: 7:ee2fa6e058ec76c7abf801567898917d
    For more information, use the --logLevel flag
    Do the following steps

    Clear the

    database

    checksums and run the update again

    Code Block
    abiquo-db clearCheckSums
  4. Retry the above abiquo-db update command.

  5. (warning) In MariaDB, for Abiquo 6.2.
    
    abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update
  6. (warning)For Abiquo 6.2.0, run the gpmigrator SQL file output from the guest password migrator script (with the appropriate user and credentials)

    Code Block
    mysql kinton < guestpasswords.sql
  7. To upgrade the Abiquo Watchtower database, on for the monitoring appliance, run the command below:

    Code Block
    watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] update

...

...


14. Upgrade additional elements

By default, you will find the upgrade scripts (e.g. for Redis definitions) on the Abiquo Server under the folder: /usr/share/doc/abiquo-redis/redis/

For each upgrade step, run the Redis database script to remove old VSM definitions, which can be found in the 4.2.3 subfolder

On remote services server:

Code Block
# cd /usr/share/doc/abiquo-redis/redis/
# bash ./4.2.3/00-old-vsm-definitions.sh

For Abiquo 6.2.0+, update the database connector configurations

...

On the Abiquo API Server, edit /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml

...

If your database is on the Abiquo Server or a single server

  1. For the database connector, remove jdbc:mysql://dbhost and replace with jdbc:mariadb:${DBHOST}

If you are using a separate database cluster:

...

For Abiquo 6.2.0+, update the database connector configurations

  1. On the Abiquo API Server, edit /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml

  2. The latest version of the file should contain new options as shown below.
    Update the file and/or variables marked with ${} for your environment.

    Code Block
    <Context>
        <Resource 
            name="jdbc/abiquoDB" 
            auth="Container" 
            type="javax.sql.DataSource" 
            factory="com.zaxxer.hikari.HikariJNDIFactory"
            dataSourceClassName="org.mariadb.jdbc.MariaDbDataSource"
            connectionTimeout="120000"
            maximumPoolSize="100"
            username="${server.database.username}"
            password="${server.database.password}"
            dataSource.url="jdbc:mariadb://${server.database.host}:${server.database.port}/kinton"
            />
    </Context>

  3. If you have a DB cluster, change the datasource.url to point to your Galera cluster.
    This will not affect the multi-master configuration of the cluster

    Code Block
    jdbc:mariadb:loadbalance://address=(host=${DBMAIN})(port=3306)(type=master),
            address=(host=${DBR1})(port=3306)(type=slave),
            address=(host=${DBR2})(port=3306)(type=slave)/kinton
  4. On the Watchtower server, edit the emmett.conf and delorean.conf files at:

    Code Block
    /etc/abiquo/watchtower/delorean.conf
    /etc/abiquo/watchtower/emmett.conf
    1. From the connection details, remove jdbc:mysql: and replace with jdbc:mariadb:

    2. Add your Galera configuration if your monitoring DB is in a cluster

...

15. Configure Abiquo after the upgrade

  1. Before you start the Abiquo tomcat server, add Abiquo configuration properties to the abiquo.properties file.
    By default the abiquo.properties file is found in the /opt/abiquo/config/ folder.
    See Changes to Abiquo configuration properties

  2. Configure the user interface. The default UI location is /var/www/html/ui.
    Optional: Add custom labels and translations in the lang_xx_XX_custom.json files in the lang folder
    Add custom configuration to client-config-custom.json. See Configure Abiquo UI and Changes to UI configuration
    If your API is not in the same domain as the UI, set the API endpoint pointing to your Abiquo API server:

    Code Block
    {
        "config.endpoint": "http://myAPIserver/api"
    }

  3. SAML: If you are using SAML with multiple IDPs, restore the configuration as described at Restore SAML security beans after upgrade.

  4. Abiquo 6.2.0 updates Java and other packages, so restore your certificates. To do this, you can follow the steps at Update certificates in your Abiquo environment.

...

...

16. Start Abiquo server and services


To start the Abiquo platform servers and services, do these steps:

  • On Abiquo server or datanode, restart RabbitMQ

    Code Block
    systemctl start rabbitmq-server
  • On the Abiquo server, restart the HTTP daemon to refresh the user interface files, and bring up the Tomcat server.

    Code Block
    service httpd start
    service abiquo-tomcat start
  • On the Remote services server, start the Tomcat server

    Code Block
    service abiquo-tomcat start
  • On the Monitoring server: start the Cassandra service

    Code Block
    sudo service cassandra start
  • WAIT about 5 minutes until the service is up and running

  • Start the KairosDB service

    Code Block
    sudo service kairosdb start
  • Start the other services in this order

    Code Block
    sudo service abiquo-emmett start
    sudo service abiquo-delorean start
  • On the V2V server: restart the Tomcat server:

    Code Block
    service abiquo-tomcat restart

...