Configure VM monitoring and metrics
Introduction to monitoring and metrics
The Abiquo VM monitoring and metrics feature enables you to obtain a rapid and convenient overview of VM performance through the Abiquo cloud console. Abiquo supports monitoring and metrics of VMs on private cloud including ESXi with vCenter and KVM, and public cloud including AWS and Azure.
This feature supports the following metrics:
Built-in metrics from each hypervisor and public cloud plugin; the monitor manager (VSM remote service) fetches these metrics periodically
Custom metrics with two options:
user configures and populates custom metrics using the Abiquo API
user configures custom metrics in collectd and the Abiquo collectd plugin populates them
The following diagram shows a basic monitoring and metrics configuration.
In addition to metrics, the Monitoring Server also manages:
Metric alarms that detect when metrics reach threshold levels
Alerts that activate when a set of alarms are triggered. Alerts can notify users or trigger actions on VMs, such as autoscaling
Install monitoring
In a test environment, install the Monitoring OVA that is a CentOS server with the monitoring server components (Watchtower server, KairosDB, and Cassandra).
For full instructions on how to install it, see OVA deployment. In a production environment, a monitoring server would typically have 4 CPUs and 16 GB RAM.
For a larger production environment, you can install separate servers:
Watchtower Server
KairosDB
Cassandra, with one of these configurations in order of scale:
On Kairos DB server
Separate server
Cluster
Cassandra servers require:
JRE 7
Sufficient disk space to store monitoring data: ideally, 1 TB of spinning disks or 3.5 TB of SSD
Post-install configuration
After you complete the install, perform the following configuration.
Edit the file /etc/cassandra/conf/cassandra.yml OR /etc/cassandra/default.conf/cassandra.yaml.
Check Cassandra documentation for more details.
After you add this configuration, save and close the file.Set batch_size_warn_threshold_in_kb: 50
Set batch_size_fail_threshold_in_kb: 70
If there is a line starting with kairosdb.datastore.cassandra.datapoint_ttl remove it.
Configure seeds with the KairsoDB IP (even if it's on the same server)
Set Broadcast address
Edit the file /opt/kairosdb/conf/kairosdb.properties and configure as required, also making these recommended changes:
Set monitoring data retention time: KairosDB manages the data in Cassandra, so you can set the time to live for Cassandra datapoints.
By default, the property is not set and Cassandra will keep datapoints forever, which will require a LOT of hard disk space
You can find the datapoints in the /var/lib/cassandra/data/kairosdb folder
When the platform deletes a metric, it will delete the datapoints in KairosDB
Example: kairosdb.datastore.cassandra.datapoint_ttl = 15768000
Reference: https://github.com/kairosdb/kairosdb/blob/v0.9.4/src/main/resources/kairosdb.properties#L91-L95
Check that the Cassandra CQL host list property is in the correct format. For example: kairosdb.datastore.cassandra.cql_host_list=192.168.888.999
You may need to add cql_ prefix and remove a port number
The number of datapoints to send to Cassandra in large environments. Set the Cassandra batch_size properties.
For best performance, set kairosdb.queue_processor.batch_size=10000
See https://github.com/super-collider/docker-openjdk8-kairosdb/blob/master/kairosdb.properties#L193-L204
You can configure KairosDB logs to store data for 7 days only
Firewall:
On the Watchtower server, enable the Abiquo Server and ALL the Remote Services servers (except separate V2V services) to access TCP 36638
On the Abiquo Server, allow Watchtower to access RabbitMQ
In multi-server environments:
On the KairosDB server, allow Abiquo Watchtower to access TCP 8080
On the Cassandra servers, allow KairosDB to access Cassandra on the configured ports
Synchronize server clocks: you MUST configure and synchronize NTP across all Abiquo servers and all hypervisors before you enable metrics.
Monitoring will not send alerts if you do not synchronize the time on ALL servers, including the Watchtower server and hypervisors (ESXi and vCenter servers)On remote services (Monolithic and Remote Services, but not V2V services), edit the abiquo.properties file to add the following properties.
# Enable/disable monitoring and alarms abiquo.monitoring.enabled = true # Address where watchtower is located abiquo.watchtower.host=IP_OF_YOUR_WATCHTOWER_INSTALLATION # If you are using a monitoring cluster, enter the IP of the master host # Port where watchtower is listening abiquo.watchtower.port=36638
Remember to restart your Abiquo Tomcat service.
Add the Watchtower servers to your platform monitoring system. See Monitor platform servers
Optional configuration for KVM hypervisor
In the AIM properties file in the [stats] section to manage statistics for metrics and monitoring:
collectFreqSeconds = 60
refreshFreqSeconds = 30
database = /var/lib/abiquo-aim.db
See the AIM agent for further details.
To support metric dimensions in KVM through AIM, in Abiquo properties, set abiquo.vsm.measures.pusher.frequency. See vsm properties in Abiquo configuration properties
Within Abiquo, to configure fetch and display of metrics by default
Go to Configuration → General
Select the Enable virtual machine monitoring by default option. The new value will apply to all VMs created after you change the value.
Create a VM and go to Monitoring. Check that the Fetch metrics checkbox is selected
Within Abiquo, assign users privileges to view and manage metrics, alarms, and alerts as required
Monitoring architecture
Abiquo Watchtower uses two types of storage: MySQL (the watchtower schema on the Abiquo MySQL database server) and KairosDB with Cassandra. For messaging, it uses the Abiquo RabbitMQ instances used by the Abiquo API server with queues for Evaluation and Notification.
Abiquo Watchtower has two main modules, called "abiquo-emmett" and "abiquo-delorean".
Emmett manages metrics, alarms, and alerts. It receives metric data and obtains alarm details, and requests alarm evaluation
Delorean evaluates alarms and updates status changes. Then it evaluates the related alerts, and updates status changes
Abiquo API notifies subscribers and creates events
How to use VM monitoring, metrics, alarms and alerts
Note that the monitoring configuration files are stored under the /etc/abiquo/watchtower path.
The Abiquo UI displays the VM monitoring information, and enables you to manage alarms and alerts. See the following pages:
The monitoring system is also available through the API. For example, see:
https://wiki.abiquo.com/api/latest/VirtualMachinesResource.html#get-metric-statistics-on-a-virtual-machine and the following sections.
https://wiki.abiquo.com/api/latest/VirtualMachinesResource.html#create-an-alarm-on-a-virtual-machine and the following sections.
The functionality to push custom metrics is not available through the UI.
See Custom metrics resources and API documentation pages for each data entity that supports custom metrics.
For example, for a VM:
Similar methods are available for other resources.
Use collectd to gather metrics
Abiquo supports collectd for custom metrics collection on Linux VMs. The Abiquo-collectd-plugin automatically pushes the values gathered by collectd to the Abiquo API. There is a specific Abiquo API endpoint that the plugin uses to perform the push. As with other custom metrics, the Abiquo API then pushes the collectd metrics to the Abiquo Monitoring Server. You can use this method for hypervisors and providers where Abiquo does not support built-in metrics.
To install and configure collectd
In Abiquo, the tenant administrator or user configures the collectd application (authentication and permissions) using OAuth.
The user must have the “Allow user to push own metrics” privilege and assign this privilege to the application.Users install collectd on their VMs, and then install the Abiquo collectd plugin (from source or using the Abiquo Chef cookbook).
The user configures their metrics in collectd.
The Abiquo collectd plugin automatically pushes all collectd metrics to the Abiquo API in PUTVAL JSON format.
Collectd links
Collectd website: http://collectd.org/
Abiquo-collectd plugin: https://github.com/abiquo/collectd-abiquo
Install Abiquo-collectd plugin with Abiquo Chef cookbook: https://github.com/abiquo/collectd-abiquo-cookbook
Abiquo API VM resource: VirtualMachinesResource
Custom metrics display in the UI
The Abiquo UI enables users to display custom metrics for the following entities:
Virtual appliance
Scaling group
Virtual machine
Hide custom metrics for virtual appliances
To hide custom metrics for virtual appliances, the administrator can edit the client-config-default.json
file and set the following UI property.
"client.monitoring.button.alwaysdisplayed": false
This will prevent the display of:
The metrics button on the virtual appliance card
The virtual appliance metrics popup
The virtual appliance metrics page in the Virtual appliance metrics screen. The VM metrics page will still display.
For more details see Configure Abiquo UI.
Copyright © 2006-2024, Abiquo Holdings SL. All rights reserved