Introduction

The Abiquo accounting service is a usage metering system. It was originally designed to enable Abiquo customers to charge their users on a pay per use basis for private cloud resources. The Abiquo accounting service records how much of each resource a tenant uses in a virtual datacenter or datacenter. For example, if a user deploys 5 VMs for a total of 25 hours each, then by default the system will record 125 VM hours during the accounting period. The resources to account are fully configurable and you can also create your own custom resources and queries for managed services. You can use the Abiquo billing integration to create a file for your billing system.

This is a basic technical overview of the Abiquo accounting system:

1. Abiquo records when users reserve or use resources in the kinton_accounting DB.
2. The administrator configures which resources are accounted and the accounting periods used for it.
3. Some accounting procedures are run periodically to consolidate/update resource usage and purge old records.

Accounting information backup

The accounting master tables are in kinton_accounting DB. Include these tables in your backup procedures:

• accounting_event_ips
• accounting_event_pm
• accounting_event_repository
• accounting_event_storage
• accounting_event_vlan
• accounting_event_vm
• accounting_event_ds_storage
• accounting_event_vm_on
• accounting_event_vm_off
• accounting_event_backup
• accounting_event_firewall
• accounting_event_loadbalancer

Accounting parameters

To retrieve the current accounting parameters, run the query below in kinton_accounting DB:

SELECT * FROM accounting_parameters;

+---------+-------------------------------------+------------+
| idParam | paramName                           | paramValue |
+---------+-------------------------------------+------------+
|       1 | AccountingEnabled                   | 1          |
|       2 | DeleteRegEventsUseSPParam           | 0          |
|       3 | DeleteRegEventsDeleteHours          | 26280      |
|       4 | MaximumPeriodsToFirstInit           | 1          |
|       5 | MaximumPeriodsToProcess             | 24         |
|       6 | AccountPeriodSize                   | HOUR       |
|       7 | AccountPeriodGranularity            | HOUR       |
|       8 | LoggingLevel                        | INFO       |
|       9 | VirtualMachine-vcpu                 | 1          |
|      10 | VirtualMachine-vram                 | 1          |
|      11 | VirtualMachine-vhd                  | 1          |
|      12 | VirtualMachine-vtierhd              | 1          |
|      13 | VirtualMachine-backup               | 1          |
|      14 | ExternalStorage                     | 1          |
|      15 | IPAddress                           | 1          |
|      16 | VLAN                                | 1          |
|      17 | VirtualMachine-hypervisorType       | 1          |
|      18 | Firewall                            | 1          |
|      19 | LoadBalancer                        | 1          |
|      20 | VirtualMachine-vcpuon               | 1          |
|      21 | VirtualMachine-vcpuoff              | 1          |
|      22 | VirtualMachine-vramon               | 1          |
|      23 | VirtualMachine-vramoff              | 1          |
|      24 | VirtualMachine-hp                   | 1          |
|      25 | VirtualMachine-hpon                 | 1          |
|      26 | VirtualMachine-hpoff                | 1          |
|      27 | VirtualMachine-haHosted             | 0          |
|      28 | ReservedPhysicalMachine-cpu         | 0          |
|      29 | ReservedPhysicalMachine-ram         | 0          |
|      30 | RepositoryStorage                   | 0          |
|      31 | VirtualMachine-antiAffinity         | 0          |
|      32 | Consolidation-time-sensitivity-secs | 30         |
|    1419 | draas_protection                    | 1          |
+---------+-------------------------------------+------------+

To check the value of an accounting parameter, use the query below:

SELECT * FROM accounting_parameters WHERE paramName="PARAM_NAME";

SELECT * FROM accounting_parameters WHERE paramName="VirtualMachine-hypervisorType";
+---------+-------------------------------+------------+
| idParam | paramName                     | paramValue |
+---------+-------------------------------+------------+
|      17 | VirtualMachine-hypervisorType | 0          |
+---------+-------------------------------+------------+
1 row in set (0.00 sec)

Description of resources

Abiquo stores this data about the use of these resources in the Accounting events tables. Then Administrators can configure which resources are accounted in the Accounting event detail table.

Resource	Description	Unit	Billing Type	Association
Cores	vCPUs in VMs created by users	vCPU	Consumption	VM
Cores ON	vCPUs in VMs deployed and running	vCPU	Consumption	VM
Cores OFF	vCPUs in VMs deployed and stopped	vCPU	Consumption	VM
RAM	RAM in VMs created by users	Megabytes	Consumption	VM
RAM ON	RAM in VMs deployed and running	Megabytes	Consumption	VM
RAM OFF	RAM in VMs deployed and stopped	Megabytes	Consumption	VM
Local Storage	Disk space connected to the hypervisor as datastores for VM disks, including secondary hard disks	Bytes	Consumption	VM
External Storage	Disk space for use as external volumes created by users on VMs	Bytes	Reservation	VM/Independent
Public IPs	IP addresses with Internet access for use by VMs	No. of IP addresses	Reservation	VM/Independent
VLANs	Private VLANs for VMs Unmanaged, external and public network VLANs are not accounted A private VLAN is automatically created when a user creates a VDC	No. of VLANs	Reservation	VDC
Firewall	Firewalls created by the user	No. of Firewalls	Reservation	VDC
Load Balancer	Load balancers created by the user	No. of Load Balancers	Reservation	VDC
Hypervisor Type	The hypervisor type for the VM	HV type	Consumption	VM
Cost Code	Optional cost codes for virtual image templates	Cost code	Consumption	VM
Backup	Backup policies applied to VMs deployed	Backup policy	Consumption	VM
Hardware profile	VM Hardware profile	Hardware profile	Consumption	VM
Hardware profile ON	VM Hardware profile in the ON state	Hardware profile	Consumption	VM
Hardware profile OFF	VM Hardware profile in the OFF state	Hardware profile	Consumption	VM
DR protected VM	VM is protected by the disaster recovery service (DRAAS)	True or false	Consumption	VM
HA Hosted VM	VM deployed on an Abiquo high-availability rack	True or false	Consumption	VM
Reserved Physical Machine Cores	CPU of each server that has been explicitly reserved for an enterprise. Recorded even if the server is not being used.	vCPU	Reservation	Enterprise
Reserved Physical Machine RAM	RAM of each server that has been explicitly reserved for an enterprise. Recorded even if the server is not being used.	Megabytes	Reservation	Enterprise
Repository Storage	Repository space is used when: Customers create a VM instance Customers upload disk images into the template repository Abiquo converts instances and templates to other hypervisor formats Abiquo accounts for space used by instances and templates (cases 1 and 2). If you wish to account conversions, create a custom metric (case 3).	Bytes used by each template	Consumption	Enterprise
Anti-affinity VM	VM deployed in an anti-affinity layer separate from other VMs in the same layer	True or false	Consumption	VM
Tiered local datastore storage	Local storage on hypervisor datastores by datastore tiers (including primary and secondary non-persistent hard disks)	Bytes used by each disk	Consumption	VM
Custom	See Custom Accounting Resource Definitions	User defined	User defined	User defined

Key

• - Resource accounted

•  - Deprecated
• Blue shading - disabled by default
• Pink shading - custom resource

Enable or disable accounting of resources

Abiquo always tracks all resources in the accounting master tables, but Administrators can configure which resources will be consolidated for billing and reporting in the accounting_event_detail table.

To select which resources Abiquo will record in this table, enable/disable the corresponding accounting parameters by setting the corresponding accounting parameter value.

UPDATE accounting_parameters SET paramValue=1 WHERE paramName="VirtualMachine-antiAffinity";

Accounting time parameters

In the accounting_event_detail table, Abiquo stores resource usage information by account period size and account period granularity. Always set these parameters before allowing users to work with your system.

Account period size parameter

For each accounting period, the update procedure checks the Accounting event tables for resources that have been reserved and/or used in the last period by granularity and records the usage data in the accounting_event_detail table.

The minimum period of consumption of a resource is always one accounting period. If a resource is active for an interval of less than one accounting period, the system automatically rounds consumption up to one accounting period. But please note that the value stored will depend on the granularity.

Resources may be turned on and off during an accounting period. Abiquo stores all these events in the Accounting master tables. However, accounting_event_detail only contains the maximum value for an accounting period.

You can set the accounting period size parameter in the accounting_parameters table.

UPDATE accounting_parameters SET paramValue="WEEK" WHERE paramName="AccountPeriodSize";

Accounting period granularity

For each accounting period, the Update procedure checks the master tables for resources reserved and/or consumed in the last period by granularity  and records the usage data in the Accounting event detail table.

The value stored for each accounting period can be divided into smaller time units using the granularity. However, the granularity does not affect the minimum usage.

The minimum period of usage is always determined by the accounting period:

• If accounting period = DAY and granularity = HOUR, then for a single CPU VM, Abiquo would record 24 resource units for a day record. So this could represent a charge for 24 hours of vCPU usage.
• If accounting period = DAY and granularity = DAY, then Abiquo will record 1 resource unit. So this could represent a charge for 1 day of vCPU usage.

You can set the granularity parameter in the accounting_parameters table.

The following table shows compatibles periods and granularities. Abiquo uses calendar months, so week granularity is not compatible with month.

UPDATE accounting_parameters SET paramValue="DAY" WHERE paramName="AccountPeriodGranularity";

Change the accounting period parameters in an existing system

To change the accounting period, follow these steps: 

1. Backup the accounting_event_detail table.
2. Delete the existing content of the accounting_event_detail table.
3. Change the accounting parameters to the desired values. Set MaximumPeriodsToFirstInit to ensure old data is accounted correctly.
4. Monitor the content of accounting_event_detail to ensure the generated data meets your needs. If not, then repeat from step 2.

Accounting periodic procedures

The accounting system runs the accounting procedures below periodically to:

• UpdateAccounting processes the data in the accounting master tables to consolidate the resource usage in the accounting_event_detail table.
  
• DeleteOldRegisteredEvents deletes data in the accounting master tables older than the DeleteRegEventsDeleteHours accounting parameter (26280 hours or 3 years by default)

These procedures are configured in the /etc/cron.d/abiquo-accounting cron job. Note that DeleteOldRegisteredEvents does not use its legacy parameter to set the maximum record age by default.

To view the cron job configuration for the accounting procedures, run the command:

cat /etc/cron.d/abiquo-accounting  

# Runs every hour
0 * * * * root mysql -uroot -hlocalhost -e "CALL kinton.UpdateAccounting();"
# Runs every Sunday at 12:00 and deletes records older than a week.
0 12 * * 0 root mysql -uroot -hlocalhost -e "CALL kinton.DeleteOldRegisteredEvents(168);"

Set the frequency of procedure run to delete old records

To change the maximum record age of the accounting master table records, set the DeleteRegEventsDeleteHours parameter in accounting_parameters table as desired.

It is also possible to use the stored procedure parameter as maximum record age by enabling the DeleteRegEventsUseSPParam parameter in accounting parameters, but it is not recommended to use this method.

UPDATE accounting_parameters SET paramValue="17520" WHERE paramName="DeleteRegEventsDeleteHours";

Retrieve Accounting data

Use the following queries as templates for creating queries to view aggregate resource usage as required. These queries should retrieve information from the replicated accounting database.

Previously the accounting system used views, which have been maintained for legacy applications. See Abiquo accounting architecture.

SELECT
  v.startTime AS startTime,
  v.endTime AS endTime,
  v.idAccountingResourceType AS idAccountingResourceType,
  v.resourceType AS resourceType,
  SUM(v.resourceUnits) AS resourceUnits,
  v.idEnterprise AS idEnterprise,
  v.idVirtualDataCenter AS idVirtualDataCenter,
  v.enterpriseName AS enterpriseName,
  v.virtualDataCenter AS virtualDataCenter
FROM
  kinton_accounting.HOURLY_USAGE_MAX_VW v
WHERE
  # APPLY CUSTOM QUERY FILTERS HERE, E.G. BY DATE RANGE AS BELOW
  v.startTime >= '2012-04-16' AND v.endTime < '2012-04-17'
GROUP BY
  v.startTime,
  v.idAccountingResourceType,
  v.idEnterprise, v.idVirtualDataCenter;

SELECT
  CAST(a.startTime AS DATE) AS startTime,
  CAST(a.startTime AS DATE) AS endTime,
  a.idAccountingResourceType AS idAccountingResourceType,
  a.resourceType AS resourceType,
  SUM(a.resourceUnits) AS resourceUnits,
  a.idEnterprise AS idEnterprise,
  a.idVirtualDataCenter AS idVirtualDataCenter,
  a.enterpriseName AS enterpriseName,
  a.virtualDataCenter AS virtualDataCenter
FROM
  kinton_accounting.HOURLY_USAGE_MAX_VW a
WHERE
  # APPLY CUSTOM QUERY FILTERS HERE, E.G. BY DATE RANGE AS BELOW
  a.startTime >= '2012-04-16' AND a.endTime < '2012-04-17'
GROUP BY
  CAST(a.startTime AS DATE),
  a.idAccountingResourceType,
  a.idEnterprise, a.idVirtualDataCenter;

SELECT
  CAST((a.startTime - INTERVAL (DAYOFMONTH(a.startTime) - 1) day) AS DATE) AS startTime,
  LAST_DAY(a.startTime) AS endTime,
  a.idAccountingResourceType AS idAccountingResourceType,
  a.resourceType AS resourceType,
  SUM(a.resourceUnits) AS resourceUnits,
  a.idEnterprise AS idEnterprise,
  a.idVirtualDataCenter AS idVirtualDataCenter,
  a.enterpriseName AS enterpriseName,
  a.virtualDataCenter AS virtualDataCenter
FROM
  kinton_accounting.HOURLY_USAGE_MAX_VW a
WHERE
  # APPLY CUSTOM QUERY FILTERS HERE, E.G. BY DATE RANGE AS BELOW
  a.startTime >= '2012-04-01' AND a.endTime < '2012-05-01'
GROUP BY
  CAST((a.startTime - INTERVAL (dayofmonth(a.startTime) - 1) day) AS DATE),
  a.idAccountingResourceType,
  a.idEnterprise, a.idVirtualDataCenter;

Accounting consistency checks

You can use the Abiquo accounting consistency check to monitor your accounting system.

See Abiquo accounting consistency check