Configure VM monitoring and metrics

Introduction to monitoring and metrics

The Abiquo VM monitoring and metrics feature enables you to obtain a rapid and convenient overview of VM performance through the Abiquo cloud console. Abiquo supports monitoring and metrics of VMs on private cloud including ESXi with vCenter and KVM, and public cloud including AWS and Azure. Abiquo also supports metrics from infrastructure.

This feature supports the following metrics:

• Built-in metrics from each hypervisor and public cloud plugin; the monitor manager (VSM remote service) fetches these metrics periodically

• Custom metrics with two options:

  • user configures and populates custom metrics using the Abiquo API

  • user configures custom metrics in collectd and the Abiquo collectd plugin populates them

The following diagram shows a basic monitoring and metrics configuration.

Open

Abiquo monitoring and metrics architecture

In addition to metrics, the Monitoring server also manages:

• Metric alarms that detect when metrics reach threshold levels

• Alerts that activate when a set of alarms are triggered. Alerts can notify users or trigger actions on VMs, such as autoscaling

Install monitoring

In a test environment, install the Monitoring OVA that is a CentOS server with the monitoring server components (Watchtower server, KairosDB, and Cassandra). For full instructions on how to install it, see OVA deployment.

In a small production environment, a monitoring server would typically have 4 CPUs and 16 GB RAM.

For a large production environment, you can install a cluster of monitoring servers. See Distributed HA cluster environment. In a new distributed scalable environment, you can install a cluster with one server.

The monitoring components are:

1. Watchtower server

2. KairosDB

3. Cassandra, where you can also use these configurations in order of scale:

   1. On Kairos DB server

   2. Separate server

   3. Cluster

Post-install configuration

After you complete the install, perform the following configuration.

1. Edit the /etc/cassandra/conf/cassandra.yml or /etc/cassandra/default.conf/cassandra.yaml file. Check Cassandra documentation for more details.
   Add the following configuration, then save and close the file.

   1. Set batch_size_warn_threshold_in_kb: 50

   2. Set batch_size_fail_threshold_in_kb: 70

   3. If there is a line starting with kairosdb.datastore.cassandra.datapoint_ttl remove it.

   4. Configure seeds with the KairsoDB IP (even if it's on the same server)

   5. Set the Broadcast address

2. Edit the file /opt/kairosdb/conf/kairosdb.properties and configure as required, also making these recommended changes:

   1. Set monitoring data retention time: KairosDB manages the data in Cassandra, so you can set the time to live for Cassandra datapoints.

      • By default, the property is not set and Cassandra will keep datapoints forever, which will require a LOT of hard disk space

        • You can find the datapoints in the /var/lib/cassandra/data/kairosdb folder

        • When the platform deletes a metric, it will delete the datapoints in KairosDB

      • Example: kairosdb.datastore.cassandra.datapoint_ttl = 15768000

      • Reference: https://github.com/kairosdb/kairosdb/blob/v0.9.4/src/main/resources/kairosdb.properties#L91-L95

   2. Check that the Cassandra CQL host list property is in the correct format. For example: kairosdb.datastore.cassandra.cql_host_list=192.168.888.999 

      • You may need to add cql_ prefix and remove a port number

   3. The number of datapoints to send to Cassandra in large environments. Set the Cassandra batch_size properties.
      For best performance, set kairosdb.queue_processor.batch_size=10000
      See https://github.com/super-collider/docker-openjdk8-kairosdb/blob/master/kairosdb.properties#L193-L204

3. You can configure KairosDB logs to store data for 7 days only

4. Firewall:

   1. On the Watchtower server, enable the Abiquo Server and ALL the Remote Services servers (except separate V2V services) to access 36638/tcp

   2. On the Abiquo Server (or Datanode), allow Watchtower to access RabbitMQ

   3. In multi-server environments:

      1. On the KairosDB server, allow Abiquo Watchtower to access 8080/tcp

      2. On the Cassandra servers, allow KairosDB to access Cassandra on the configured ports (by default these are 9042/tcp and 9160/tcp)

      3. From the Watchtower, allow access to the Watchtower MySQL database on port 3306/tcp.

      4. Allow access to Emmett on port 7000/tcp

5. Synchronize server clocks: you MUST configure and synchronize NTP across all Abiquo servers and all hypervisors before you enable metrics.
   Monitoring will not send alerts if you do not synchronize the time on ALL servers, including the Watchtower server and hypervisors (ESXi and vCenter servers)

6. On remote services (Monolithic and Remote Services, but not V2V services), edit the abiquo.properties file to add the following properties.

   # Enable/disable monitoring and alarms abiquo.monitoring.enabled = true   # Address where watchtower is located abiquo.watchtower.host=IP_OF_YOUR_WATCHTOWER_INSTALLATION # If you are using a monitoring cluster, enter the IP of the master host # Port where watchtower is listening abiquo.watchtower.port=36638

   Remember to restart your Abiquo Tomcat service to activate the properties.

7. Add the Watchtower servers to your platform monitoring system. See Monitor platform servers

8. Within Abiquo, to configure fetch and display of metrics by default

   1. Go to Configuration → General

   2. Select the Enable virtual machine monitoring by default option. The new value will apply to all VMs created after you change the value.

   3. Create a VM and go to Monitoring. Check that the Fetch metrics checkbox is selected

9. Within Abiquo, assign users privileges to view and manage metrics, alarms, and alerts as required

If you are using TLS for your platform servers, now complete the configuration in Watchtower TLS guide.

Monitoring architecture

Abiquo Watchtower uses two types of storage: MySQL (the watchtower schema on the Abiquo MySQL database server) and KairosDB with Cassandra. For messaging, it uses the Abiquo RabbitMQ instances used by the Abiquo API server with queues for Evaluation and Notification.

Abiquo Watchtower has two main modules, called abiquo-emmett and abiquo-delorean.

• Emmett manages metrics, alarms, and alerts. It receives metric data and obtains alarm details, and requests alarm evaluation

• Delorean evaluates alarms and updates status changes. Then it evaluates the related alerts, and updates status changes

• Abiquo API notifies subscribers and creates events

Open

Abiquo monitoring and metrics alarm and alert processing

How to use VM monitoring, metrics, alarms and alerts

Note that the monitoring configuration files are stored under the /etc/abiquo/watchtower path.

The Abiquo UI displays the VM monitoring information, and enables you to manage alarms and alerts. See the following pages:

• VM monitoring and metrics

• Manage cloud alarms

• Manage alerts

The monitoring system is also available through the API. For example, see:

• VirtualMachinesResource.html#enable-fetch-of-metrics-for-a-virtual-machine

• VirtualMachinesResource.html#get-metric-statistics-on-a-virtual-machine and the following sections.

• VirtualMachinesResource.html#create-an-alarm-on-a-virtual-machine and the following sections.

• AlertsResource.html

The functionality to push custom metrics is not available through the UI.
See Custom metrics resources and API documentation pages for each data entity that supports custom metrics.

For example, for a VM:

• VirtualMachinesResource.html#push-metric-datapoints-on-a-virtual-machine

• VirtualMachinesResource.html#push-collectd-values-on-a-virtual-machine

Similar methods are available for other resources.

Use collectd to gather metrics

Abiquo supports collectd for custom metrics collection on Linux VMs. The Abiquo-collectd-plugin automatically pushes the values gathered by collectd to the Abiquo API. There is a specific Abiquo API endpoint that the plugin uses to perform the push. As with other custom metrics, the Abiquo API then pushes the collectd metrics to the Abiquo Monitoring Server. You can use this method for hypervisors and providers where Abiquo does not support built-in metrics.

To install and configure collectd

1. In Abiquo, the tenant administrator or user configures the collectd application (authentication and permissions) using OAuth.
   The user must have the Allow user to push own metrics privilege and assign this privilege to the application.

2. Users install collectd on their VMs, and then install the Abiquo collectd plugin.

3. The user configures their metrics in collectd.

4. The Abiquo collectd plugin automatically pushes all collectd metrics to the Abiquo API in PUTVAL JSON format.

Collectd links

• Collectd website: https://collectd.org/

• Abiquo-collectd plugin: https://github.com/abiquo/collectd-abiquo

• Abiquo API VM resource: VirtualMachinesResource

Custom metrics display in the UI

Hide custom metrics for virtual appliances

To hide custom metrics for virtual appliances, the administrator can edit the client-config-default.json file and set the following UI property.

This will prevent the display of:

• The metrics button on the virtual appliance card

• The virtual appliance metrics popup

• The virtual appliance metrics page in the Virtual appliance metrics screen. The VM metrics page will still display. 

For more details see Configure Abiquo UI.