Upgrade Abiquo Monolithic to v5.2.x
- MS (Unlicensed)
This section describes how to upgrade from Abiquo 4.7.0 or above to 5.2.x. Please contact Abiquo Support for further information.
Major upgrade
The upgrade to Abiquo 5.2.0 is a major upgrade with upgrades of third-party software packages that are critical platform dependencies
You must make snapshots of ALL servers before you start the upgrade
- Abiquo YUM repositories are no longer open, please contact Abiquo Support to obtain your credentials
- This upgrade process starts from Abiquo 4.7 or above. To upgrade from earlier versions, please contact Abiquo Support
1. Prepare branding
- If you are upgrading from Abiquo 4.7.x, the new UI in Abiquo 5.0 requires new branding. See Abiquo branding guide
2. Prevent cloud users from performing VM operations
- In the UI in the Infrastructure view, select each physical machine and click Disable
- Using the API, set the state of each physical machine to DISABLED
3. Check for operations in progress on the platform
Before you shut down the platform servers you should check that no operations are in progress.
Before you shut down the platform servers you should check that no operations are in progress on the platform.
Check that the Abiquo RabbitMQ queues are empty on the Abiquo Monolithic Server, Abiquo Server or Datanode server
The number of messages in all queues must be 0.
# rabbitmqctl list_queues messages name
# rabbitmqctl list_queues messages name Listing queues ... 0 abiquo.am.notifications 0 abiquo.bpm.notifications 0 abiquo.datacenter.requests.ADatacenter.bpm 0 abiquo.datacenter.requests.ADatacenter.virtualfactory 0 abiquo.ha.tasks 0 abiquo.nodecollector.notifications 0 abiquo.pcrsync.messages 0 abiquo.pcrsync.parking-expect-no-consumers 0 abiquo.scheduler.fast.requests 0 abiquo.scheduler.requests 0 abiquo.scheduler.slow.requests 0 abiquo.tracer.traces 0 abiquo.virtualfactory.notifications 0 abiquo.virtualmachines.definitionsyncs 0 abiquo.vsm.eventsynk ...done.
On the V2V Server, check for any active conversions by checking for the V2V or Mechadora processes
$ ps aux | grep v2v $ ps aux | grep mechadora
When users' VM operations are blocked and all of the above checks show that no tasks are running, it is safe to halt the platform.
4. Disable old billing dashboards and delete last bills
In Abiquo 5.2, the billing dashboard feature is now in the core Abiquo platform.
If you are using the billing dashboard scripts from previous versions, disable them and delete last bills before you continue with the upgrade.
On the server where you were running the billing packages, remove the Cron jobs to run the billing scripts.
# rm /etc/cron.d/abiquo-billing-ec2 # rm /etc/cron.d/abiquo-billing-arm
Delete the last bills from the previous version of the billing dashboards (from the kinton.billing_consolidation and kinton.billing_register tables), see commands below. Remove the bills for Azure and Amazon providers, for the number of months covered by the abiquo.enterprise.property.billing.monthoffset system property, which has a default value of 2 months. Later the new billing dashboards feature will regenerate these bills:
SELECT COUNT(*) FROM kinton.billing_consolidation; SELECT COUNT(*) FROM kinton.billing_register;
5. Back up the main platform elements
To perform a basic backup of the platform, run the following backups:
Before you begin, stop platform services, and check you have enough space on your destination systems.
# On DB server, back up the Abiquo MySQL DBs mysqldump --routines --triggers kinton > kinton.sql-$(date +%Y%m%d_%H%M%S) mysqldump --routines --triggers kinton_accounting > kinton_accounting.sql-$(date +%Y%m%d_%H%M%S) # On the Abiquo API Server or UI server, back up the UI tar cvfz /var/www/html/ui.tgz-$(date +%Y%m%d_%H%M%S) /var/www/html/ui # If you are using SAML authentication with multiple IdPs, on ALL Abiquo API Servers, back up this file cp /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml-$(date +%Y%m%d_%H%M%S) # On ALL servers, back up the /opt/abiquo folder tar cvfz /opt/abiquo.tgz-$(date +%Y%m%d_%H%M%S) /opt/abiquo # On Abiquo Monolithic or API Server, Remote Services, or Datanode, back up Redis redis-cli save cp -a /var/lib/redis /var/lib/redis-$(date +%Y%m%d_%H%M%S)
If you are using HTTPS-SSL between your Tomcat servers, for each Tomcat server:
From the /opt/abiquo/tomcat/conf folder, back up the server.xml files. You will need to restore and modify them after the yum upgrade.
6. Stop platform services
This section describes how to stop platform services on all servers.
if there are operations in progress, DO NOT STOP the platform services because this can cause serious issues with your platform. See Check for operations in progress on the platform and wait for all operations to complete
To stop platform services:
Stop the API on the API server or monolithic server
systemctl stop abiquo-tomcat
Stop the UI on the API server or monolithic server or dedicated UI server
systemctl stop httpd
Stop remote services server
systemctl stop abiquo-tomcat
Stop the database on the monolithic server or database server.
systemctl stop mariadb
For a datanode configuration, you will also need to stop the Galera cluster. For more details, see Stop and start HA configuration
Stop RabbitMQ (on the monolithic server or API Server or Datanode)
systemctl stop rabbitmq-server
- V2V server -
You do not need to stop anything because the BPM remote service is run on-demand only Stop monitoring server
systemctl stop abiquo-delorean systemctl stop abiquo-emmett systemctl stop kairosdb systemctl stop cassandra
On the monitoring server, check if Cassandra is really dead
ps auwx | grep cassandra
Get the process number for Cassandra (the first number in the output of the previous command), and kill it. In this example, Cassandra is process 12345.
kill -9 12345
All processes on platform servers should now be halted.
7. Make snapshots and backups of all platform servers
The upgrade to Abiquo 5.2.x is a major upgrade with upgrades of third-party software packages that are critical platform dependencies
You MUST make snapshots of all servers in your platform before you upgrade to Abiquo 5.2.x.
8. Prepare yum repositories on all servers
Check that you have the repository URL and credentials
To upgrade to a version with a patch number of zero, for example, version 5.2.0
OR To upgrade to a version with patch number that is not zero, for example, version 5.2.1, enable both repositories:
yum-config-manager --enable abiquo-base yum-config-manager --enable abiquo-updates
Optionally add your username and password to the Abiquo repos
yum-config-manager --save --setopt=abiquo-*.username=MYUSERNAME yum-config-manager --save --setopt=abiquo-*.password=MYPASSWORD
Don't forget to use a backslash to escape any shell special characters. For more details, see https://www.oreilly.com/library/view/learning-the-bash/1565923472/ch01s09.html
Clean yum and make cache
yum clean all && yum makecache
If you did not make snapshots of all servers already, then do this now
Major upgrade
The upgrade to Abiquo 5.2.x is a major upgrade with upgrades of third-party software packages that are critical platform dependencies
You must make snapshots of ALL servers before you start the upgrade
9. Upgrade monolithic server
These steps are for a Monolithic Abiquo Server, with API and Remote Services on a single server.
Abiquo will upgrade to new versions of RabbitMQ and MySQL that require a manual upgrade path of dependencies.
These steps are for a server that contains RabbitMQ AND MySQL
Remove the Erlang and Galera packages.
yum remove erlang* galera yum clean all && yum makecache
Install new RabbitMQ and MariaDB server
yum install rabbitmq-server MariaDB-server
Before you confirm, check that the packages will be installed from the abiquo-base repository
- rabbitmq-server: 3.8.2.1
- MariaDB-server: 10.4.10.1
Upgrade Abiquo to 5.2.x
yum update
Before you confirm, check the following packages will be installed from the abiquo-base repository:
- jdk: 11.0.6u10
- redis: 5.0.7.1
After the update, check versions
Enable and start the new services
systemctl enable mariadb systemctl enable rabbitmq-server systemctl start mariadb systemctl start rabbitmq-server
Upgrade MySQL (you might need to specifiy the user and password in the command line):
mysql_upgrade
The mysql_upgrade step may detect errors that trigger messages such as the following:
kinton_accounting.ACCOUNT_PERIOD_USAGE_VW Error : Column count of mysql.proc is wrong. Expected 21, found 20. Created with MariaDB 100129, now running 100410. Please use mysql_upgrade to fix this error
These messages are expected and this same mysql_upgrade process will automatically fix these errors in its next stages.
Check that the mysql_upgrade process completes correctly.
Check that your hostname is in your DNS or in your /etc/hosts file
Upgrade the Abiquo API databases
abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update
If the liquibase update fails with a message similar to the following:
Liquibase update Failed: Validation Failed:
1 change sets check sum
src/X.X.X-XX/kinton-X.X.X.xml::ABICLOUDPREMIUM-XXXX-XxxxxxxxXXxXxxxxxxXxxxxx::XXXXXXXXX is now: 7:ee2fa6e058ec76c7abf801567898917d
For more information, use the --logLevel flagDo the following steps
Clear the database checksums
abiquo-db clearCheckSums
- Retry the above abiquo-db update command.
On the monolithic server, change file owners to tomcat user
chown -R tomcat /opt/abiquo
On the monolithic server running Remote Services that mount the NFS repository (AM, V2V), change file owners to tomcat user
chown -R tomcat /opt/vm_repository chmod -R a+r /opt/vm_repository
10. Upgrade monitoring server
This step is for Watchtower monitoring servers in all installations.
To continue using Java 8 (for Cassandra), remove the JDK through the package manager. If you use yum, it will be delete all the dependencies and you will have to reinstall them
rpm -e --nodeps jdk
Check the jdk to install is version 8
rpm -qa | grep jdk
- Check that Java is correctly installed:
If you execute this command and get that error, please execute the command from the next bullet:
$ java -bash: java: command not found
Fix it by executing this command, then selecting the option "java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/bin/java)":
$ sudo alternatives --config java There are 3 programs which provide 'java'. Selection Command ----------------------------------------------- 1 /usr/java/jdk1.8.0_144/bin/java + 2 /usr/java/jdk-11.0.6+10/bin/java * 3 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/bin/java) Enter to keep the current selection[+], or type selection number: 3
As for the Abiquo DB steps, remove Galera, which will also remove MariaDB
yum remove galera
Clean yum and make cache
yum clean all && yum makecache
Install MariaDB
yum install MariaDB-server
Update Abiquo
yum update
Check that the Java package is now "javajdk" to use Java 1.8.0u144 with Cassandra:
# rpm -qa 'javajdk*' javajdk-1.8.0u144-6691.el7.x86_64
If you have a HA datanode that runs the watchtower database, skip directly to Step 10 to update watchtower database
Enable and start the services that were reinstalled
systemctl enable mariadb systemctl start mariadb
Upgrade MySQL
mysql_upgrade
The mysql_upgrade step may detect errors that trigger messages such as the following message (note: this example is from the Abiquo database server).
kinton_accounting.ACCOUNT_PERIOD_USAGE_VW Error : Column count of mysql.proc is wrong. Expected 21, found 20. Created with MariaDB 100129, now running 100410. Please use mysql_upgrade to fix this error
These messages are expected and this same mysql_upgrade process will automatically fix these errors in its next stages.
Check that the mysql_upgrade process completes correctly.
Update the Abiquo Watchtower database
watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] update
11. Upgrade steps from 4.7.x to 5.0.x
Start with your original version and perform the steps to the final version.
12. Upgrade steps from 5.0.x to 5.1.x
Start with your original version and perform the steps to the final version.
13. Upgrade steps from 5.1.x to 5.2.x
Start with your original version and perform the steps to the final version.
When upgrading from 5.1.0 or 5.1.1 to 5.1.2 or above, follow the steps from the Upgrade steps from 5.0.x to 5.1.x block.
Run Redis database script
These scripts can always be found on the Abiquo Server.
On each Remote Services server (including Monolithic sever) or Datanode server run the Redis scripts.
# cd /usr/share/doc/abiquo-redis/redis/ # bash ./4.2.3/00-old-vsm-definitions.sh
14. Upgrade steps for 5.2 versions
These steps apply to upgrades starting from version 5.2.0 and above.
Abiquo 5.2.0 or later to latest 5.2.x version
Run Redis database script
These scripts can always be found on the Abiquo Server.
On each Remote Services server (including Monolithic sever) or Datanode server run the Redis scripts# cd /usr/share/doc/abiquo-redis/redis/ # bash ./4.2.3/00-old-vsm-definitions.sh
15. Configure Abiquo after the upgrade
Before you start the Abiquo tomcat server, add Abiquo configuration properties to the abiquo.properties file.
By default the abiquo.properties file is found in the /opt/abiquo/config/ folder.
See Changes to Abiquo configuration propertiesConfigure the user interface. The default UI location is /var/www/html/ui.
Optional: Add custom labels and translations in the lang_xx_XX_custom.json files in the lang folder
Add custom configuration to client-config-custom.json. See Configure Abiquo UI
If your API is not in the same domain as the UI, set the API endpoint pointing to your Abiquo API server:{ "config.endpoint": "http://myAPIserver/api" }- Reporting changes: To upgrade the Abiquo Reports database for the upgrade to Abiquo 4.7.x+, contact Abiquo Support for the file and procedure.
16. Start Abiquo servers and services
To start the Abiquo platform servers and services, do these steps:
On the Abiquo server, restart the HTTP daemon to refresh the user interface files, and bring up the Tomcat server.
service httpd start service abiquo-tomcat start
On the remote services server, start the Tomcat server
service abiquo-tomcat start
On the monitoring server: edit the file /opt/kairosdb/conf/kairosdb.properties to update the name of this variable and to remove the port from it:
Replace the line kairosdb.datastore.cassandra.host_list=192.168.888.999:9160 .
With this line kairosdb.datastore.cassandra.cql_host_list=192.168.888.999 . Please note the new cql_ preffix for this property.
On the monitoring server: edit the file /etc/cassandra/conf/cassandra.yml OR /etc/cassandra/default.conf/cassandra.yaml (whichever exists) to remove a variable:
Remove the line starting with kairosdb.datastore.cassandra.datapoint_ttl and save and close the file.
On the monitoring server: edit the file /opt/kairosdb/conf/kairosdb.properties to add a new variable (ref: Internal JIRA SUP-333):
kairosdb.datastore.cassandra.datapoint_ttl = 15768000
On the monitoring server: start the Cassandra service
sudo service cassandra start
WAIT about 5 minutes until the service is up and running
Start the KairosDB service
sudo service kairosdb start
Start the other services in this order
sudo service abiquo-emmett start sudo service abiquo-delorean start
On the V2V server: restart the Tomcat server:
service abiquo-tomcat restart
Copyright © 2006-2024, Abiquo Holdings SL. All rights reserved