Upgrade Abiquo 6.1.x to 6.2.x distributed by servers

These instructions are still being updated for the Abiquo 6.2 upgrade

This section describes how to upgrade from Abiquo 6.1.x to Abiquo 6.2.x.
Please contact Abiquo Support for further information.

Major upgrade

The upgrade to Abiquo 6.2.0 is a major upgrade with upgrades of third-party software packages that are critical platform dependencies

You must make snapshots of ALL servers before you start the upgrade

This is a non-official upgrade document from which you can easily cut and paste the commands for each to prepare your own upgrade document.

The official upgrade document, that you must read and refer to for your upgrade is still Upgrade Abiquo

For Abiquo YUM repositories, please contact Abiquo Support to obtain your credentials

The upgrade will overwrite lang_en_US_labels.json. If you have customized labels or translations, back them up before the upgrade and restore them afterwards accordingly.
The upgrade will overwrite the SAML login configuration for multiple IdPs. If you use SAML with multiple IdPs, back up the configuration files before the upgrade and add the multiple IdPs afterwards accordingly. See Restore SAML security beans after upgrade

1. Prevent cloud users from performing VM operations

In the UI in the Infrastructure view, select each physical machine and click Disable
Using the API, set the state of each physical machine to DISABLED

2. API servers

On each API Server do these steps.

If you have a separate DB server, do the DB upgrade steps on that server.

# Log in as root
sudo su -


# Check that the Abiquo RabbitMQ queues are empty (API or Datanode).
# The number of messages in all queues must be 0.
rabbitmqctl list_queues messages name


# Stop  services
systemctl stop httpd
systemctl stop abiquo-tomcat
# (API or Datanode)
systemctl stop rabbitmq-server


# Back up the UI
tar cvfz /var/www/html/ui.tgz-$(date +%Y%m%d_%H%M%S) /var/www/html/ui

# Copy the api.xml file to back up the database connection details
cp /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml /opt/abiquo/api.xml-$(date +%Y%m%d_%H%M%S)

# Back up the /opt/abiquo folder
tar cvfz /opt/abiquo.tgz-$(date +%Y%m%d_%H%M%S) /opt/abiquo
  
# Back up Redis
redis-cli save
cp -a /var/lib/redis /var/lib/redis-$(date +%Y%m%d_%H%M%S)

# On DB Server, back up the Abiquo MySQL DBs
mysqldump --routines --triggers kinton            > kinton.sql-$(date +%Y%m%d_%H%M%S)
mysqldump --routines --triggers kinton_accounting > kinton_accounting.sql-$(date +%Y%m%d_%H%M%S)

# IF you are using SAML with multiple IdPs, back up security beans files
cp /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml /opt/abiquo/tomcat/webapps/api/WEB-INF/classes/springresources/security/saml/security-saml-generated-beans.xml-$(date
 +%Y%m%d_%H%M%S)

# Install latest yum release package for your target version using link
yum remove 'abiquo-release-*'
# Find the correct link at https://abiquo.atlassian.net/wiki/spaces/doc/pages/311363475/Abiquo-release-ee+packages+list
# For 6.2
yum localinstall https://user:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-release-ee-6.2.0-AAAAAAAA_BBBBBB.el7.noarch.rpm

# Prepare yum repository for your target version with yum-utils
yum-config-manager --enable abiquo-base
yum-config-manager --enable abiquo-updates

 # Perform update
yum clean all && yum makecache

yum update
# 6.2.0: - Note: MariaDB-server requires a manual update after running the gpmigrator in the next steps


# 6.2.0: - Download and install the guestpasswordmigrator package
yum localinstall http://user:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-gpmigrator-6.2.0-20240205_155724.el7.noarch.rpm

# Run the tool to prepare to migrate the guest passwords of the current VMs to a new column that will be added in 6.2.0.
# The tool requires access to the kinton database and to the API Redis
cd /opt/abiquo/utilities/
java -jar gpmigrator.jar --jdbc-url="jdbc:mariadb://localhost:3306/kinton?user=root&password=root" --redis-host=localhost
# Save the guestpasswords.sql file to run after you update the Abiquo database

# 6.2.0: - On all database servers, upgrade MariaDB 
rpm --nodeps -ev MariaDB-server-10.4.10-1.el7.centos.x86_64
yum install MariaDB-server

service mariadb start
# OR for datanode: galera_new_cluster

mysql_upgrade

# Update the Abiquo databases
abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update


# IF the database upgrade fails because of checksums
abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] clearCheckSums
abiquo-db [-h DB hostname] [-P DB port] [-u user] [-p password] update


# 6.2.0: - Run guestpasswords.sql file
mysql kinton < guestpasswords.sql


# Post-upgrade step for API
chown -R tomcat /opt/abiquo


# Run Redis database script
# On each Remote Services server (including Monolithic sever) 
cd /usr/share/doc/abiquo-redis/redis/
bash ./4.2.3/00-old-vsm-definitions.sh


# 6.2.0: - Edit the tomcat configuration file and update the database connector configuration 
# Edit /opt/abiquo/tomcat/conf/Catalina/localhost/api.xml
# The latest version of the file should contain new options
# Update the file and/or variables marked with ${} for your environment
<Context>
    <Resource 
        name="jdbc/abiquoDB" 
        auth="Container" 
        type="javax.sql.DataSource" 
        factory="com.zaxxer.hikari.HikariJNDIFactory"
        dataSourceClassName="org.mariadb.jdbc.MariaDbDataSource"
        connectionTimeout="120000"
        maximumPoolSize="100"
        username="${server.database.username}"
        password="${server.database.password}"
        dataSource.url="jdbc:mariadb://${server.database.host}:${server.database.port}/kinton"
        />
</Context>
# For a DB cluster, change the datasource.url to point to your Galera cluster
# This will not affect the multi-master configuration of the cluster
    dataSource.url="jdbc:mariadb:loadbalance://address=(host=dbmain)(port=3306)(type=master),
        address=(host=dbr1)(port=3306)(type=slave),
        address=(host=dbr2)(port=3306)(type=slave)/kinton"
        

# Add the Java keytool path to your $PATH variable:
export PATH=$PATH:/usr/java/jdk-21+35/bin/

# Check your cacerts certificates
keytool -v -list -cacerts | grep ${ENV_NAME}

# If necessary, restore your certificates following the instructions
# See https://abiquo.atlassian.net/wiki/spaces/doc/pages/609648729/Update+certificates+in+your+Abiquo+environment

Additional API and UI upgrade steps

Add translations for new UI labels to lang_xx_XX_custom.json. See Abiquo UI client language configuration
Optionally add new configuration to client-config-custom.json. See Configure Abiquo UI
In your UI configuration, check that the API endpoint is pointing to your Abiquo API server:
```
{
    "config.endpoint": "http://<API_IP_ADDRESS>/api"
}
```
Add new API properties to the abiquo.properties file found in the /opt/abiquo/config/ folder. See Changes to Abiquo Configuration properties

3. Remote services servers

On each remote services server do these steps.

# Log in as root
sudo su -
 
 
# On V2V servers, check for active conversions
# ps aux | grep v2v
# ps aux | grep mechadora
 

# Back up the /opt/abiquo folder
tar cvfz /opt/abiquo.tgz-$(date +%Y%m%d_%H%M%S) /opt/abiquo
# If you are using remote services with TLS, back up the connector
cp /opt/abiquo/tomcat/conf/server.xml /opt/abiquo/server.xml--$(date +%Y%m%d_%H%M%S) 

  
# Back up Redis
redis-cli save
cp -a /var/lib/redis /var/lib/redis-$(date +%Y%m%d_%H%M%S)


# Stop services
systemctl stop abiquo-tomcat


# If you have multiple NARS remote services in your datacenter, check for a results-nars or similar shared folder 
# as defined by the system property abiquo.nars.async.fileresults.path):
mount | grep -i "nfs\|results" | grep -v sunrpc
10.60.37.87:/abq-pro-nfs on /opt/abiquo/results-nars type nfs4 (rw,nosuid,nodev,noexec,relatime,sync,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.60.37.77,local_lock=none,addr=10.60.37.87,user)
# Unmount the previous shared folder if you have it mounted
umount /opt/abiquo/results-nars


# Install latest yum release package for your target version using link
yum remove 'abiquo-release-*'

# Remove docker
rpm -e --nodeps docker
rpm -e --nodeps docker-client
rpm -e --nodeps docker-common

# Find the correct link at https://https://abiquo.atlassian.net/wiki/spaces/doc/pages/311363475/Abiquo-release-ee+packages+list
# For 6.2
yum localinstall https://user:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-release-ee-6.2.0-AAAAAAAA_BBBBBB.el7.noarch.rpm

# Prepare yum repository for your target version with yum-utils
# yum install yum-utils if you do not have it yet
yum-config-manager --enable abiquo-base
yum-config-manager --enable abiquo-updates
  

# Perform update
yum clean all && yum makecache

yum update

# If you have a NARS shared folder, remount it (note this will also mount any other folders you have unmounted)
mount -a


# Post upgrade steps for RS
chown -R tomcat /opt/abiquo
chown -R tomcat /opt/vm_repository
chmod -R a+r /opt/vm_repository


# Refresh VM definitions
cd /usr/share/doc/abiquo-redis/redis/
bash ./4.2.3/00-old-vsm-definitions.sh


# If your remote services use TLS, restore the connection data server.xml 
# See https://abiquo.atlassian.net/wiki/spaces/doc/pages/311377433/Configure+Abiquo+Tomcat+with+HTTPS+for+Remote+RS


# Add the Java keytool path to your $PATH variable:
export PATH=$PATH:/usr/java/jdk-21+35/bin/

# Check certificates in your Java key store
keytool -v -list -keystore /opt/abiquo/tomcat/conf/${KEYSTORE_NAME}.jks

# If necessary, restore your certificates following the instructions
# See https://abiquo.atlassian.net/wiki/spaces/doc/pages/609648729/Update+certificates+in+your+Abiquo+environment

Additional RS upgrade steps

Add new RS properties to the abiquo.properties file found in the /opt/abiquo/config/ folder. See Changes to Abiquo Configuration Properties

4. Monitoring servers

On each Watchtower monitoring server do these steps.

# Log in as root and stop services
sudo su -
systemctl stop abiquo-emmett
systemctl stop abiquo-delorean
systemctl stop kairosdb
systemctl stop cassandra
 

# Really kill Cassandra by getting its process number (e.g. XXXXX) and killing it
ps auwx | grep cassandra
kill -9 XXXXX


# Install latest yum release package for your target version using link
yum remove 'abiquo-release-*'
# Find the correct link at https://https://abiquo.atlassian.net/wiki/spaces/doc/pages/311363475/Abiquo-release-ee+packages+list
# For 6.2
yum localinstall https://user:passwd@mirror.abiquo.com/el7/6.2/os/x86_64/abiquo-release-ee-6.2.0-AAAAAAAA_BBBBBB.el7.noarch.rpm
# Prepare yum repository for your target version 
yum-config-manager --enable abiquo-base
yum-config-manager --enable abiquo-updates

 
# Perform update
yum clean all && yum makecache
yum update


# 6.2.0: - On monitoring database server, upgrade MariaDB 
rpm --nodeps -ev MariaDB-server-10.4.10-1.el7.centos.x86_64
yum install MariaDB-server
service mariadb start
mysql_upgrade

# Clear the checksums of watchtower database
watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] clearCheckSums
# Update watchtower database
watchtower-db [-h DB hostname] [-P DB port] [-u user] [-p password] update


# 6.2.0: - Edit the emmett.conf and delorean.conf files at:
/etc/abiquo/watchtower/delorean.conf
/etc/abiquo/watchtower/emmett.conf
# Replace "jdbc:mysql" with "jdbc:mariadb"
# And the IP address as for your monitoring database server or cluster

5. Start Abiquo server and services

Start all servers in the correct order.

# On API or datanode 
systemctl start rabbitmq-server


# On API Server
service httpd restart
service abiquo-tomcat start
 

# On Remote Services including V2V
service abiquo-tomcat start
 

# On the Monitoring Server, start the Cassandra service
sudo service cassandra start
 

# WAIT about 5 minutes until Cassandra is up and running
# Start the KairosDB service
sudo service kairosdb start
 

# Start the other services in this order
sudo service abiquo-emmett start
sudo service abiquo-delorean start

6. In Abiquo UI

Re-enable the physical machines
In Abiquo 6.2.0, to activate Cost usage reports, regenerate billing

7. Reporting server

In Abiquo 6.2.0 the Reporting feature has been removed. If you are using on-premise billing, you can retire the Reporting server.