Troubleshooting Configuration Problems

This document provides a list of quick answers to issues caused by incorrect configuration.

Events Problems

Events are not available

Symptoms

No events are available in Abiquo.

Solution

Check the default_outbound_api_user in Abiquo. You may need to configure some of the M properties in the abiquo.properties file on the server where M will run. By default, M runs on the Abiquo Server.  The  API location is required  and  if you change the   M user or password, you must update the M user account details . See Troubleshooting Events and Outbound API Configuration

Only Events Related to Roles Are Displayed

Symptoms

No events except those related to roles are displayed.

Solution

Verify that you correctly configured the network when you installed Abiquo. If you configured one of the eth addresses using DHCP, it has to deliver a hostname and gateway address. To verify this is properly configured, access the /etc/hosts file and check if your hostname appears in it; if not, add it.

Infrastructure Problems

Abiquo not connecting to database (Bad handshake)

Symptoms

Attempting to connect to MySQL or MariaDB fails with message: "Error 1043: Bad handshake" when trying to connect with support for pre-4.1 hashed password.

Solution

The parameter "old_passwords" must be set to "0" in "/etc/my.cnf" file.

If compatibility with pre-4.1 hashed passwords is needed in your environment a downgrade to version 5.1.56 is the solution until the bug fix is shipped with a newer version.

Remote Services Error Despite Green Lights on Checks

Symptoms

When creating a new Physical Data Center, Abiquo gives an error that some of the remote services aren't properly configured although every check is green.

Solution

1. Review the NFS configuration in the file /opt/abiquo/config/abiquo.properties.
2. Restart the Tomcat server "service abiquo-tomcat restart".
3. Check there is a file in the directory /opt/vm_repository/ named ".abiquo_repository".

• See VM repository folder and Troubleshooting NFS Server configuration in Abiquo

NetApp Storage Access Failure

Symptoms

• Cannot access NetApp storage device

Solution

• Edit the storage device in Infrastructure view and check NetApp credentials

Appliance Library Problems

Downloaded Virtual Machine Templates Do Not Appear in the Appliance Library

Symptoms

• The server has been installed or restarted recently.
• Images appear as downloaded in the remote repository.
• No images appear in the Appliance Library.

Solution

Check that the NFS repository is correctly configured.

• See VM repository folder and Troubleshooting NFS Server configuration in Abiquo

Appliance Manager Problems

Appliance Manager is not working

Symptoms

• Appliance Manager is not working
• NFS server is located on the same machine as the remote services.
• Error message: "Can't validate ''abiquo.appliancemanager.repositoryLocation'' 127.0.0.1:/opt/vm_repository. It's a local repository"

Solution

Disable the check for mounted repositories by setting the property "abiquo.appliancemanager.checkMountedRepository" to false on /opt/abiquo/config/abiquo.properties. By default this property is set to true.

Virtual Datacenters Problems

General Storage Troubleshooting on ESX and ESXi

Symptoms

An external volume is not attached or operational when used in managed storage, generic iSCSI storage volumes or persistent virtual machines, but there is no notification in the GUI or API.

Cause

If a volume has been removed or replaced in the storage device, Abiquo may try to attach a volume that does not exist or is a completely different to one previously configured. The volume will not be attached or operational, but there will be no notification in the GUI or API.

Abiquo avoids performing a full HBA rescan of storage on ESX or ESXi for performance reasons. Instead it checks for a raw device mapping for a given IQN and LUN. If the raw device mapping is not correct, this problem may occur.

Solution

This problem can be resolved by manually performing a full HBA rescan of all ESX or ESXi hosts that had raw device mappings for the devices involved.

Cannot Initialize the Extended Disks Error in ESXi

Symptoms

• Deploy in ESXi. The VM is not deployed and the message "Cannot initialize the extended disks" is displayed

Problems

VCenter is managing the ESXi machine. There is already an existing VM with the same name. VCenter renames the new VM but then Abiquo cannot find it.

Solution

Remove the orphaned VM with the same name.

Not Enough Resources Available

Symptoms

When deploying, you receive the notification: "Not enough resources available".

Solution

The problem is there are not physical machines with enough resources (HD, CPU, RAM) to deploy the virtual machine.
There are 2 possible solutions:

• Undeploy some virtual machines
• Make sure you have a physical machine with enough resources to be able to deploy

Persistent Virtual Machine Process Does Not Work Because of Missing iSCSI Client

Symptoms

Check for the existence of a message like "command not found" in the log.

Solution

The V2V or Remote Services machine or the target Hypervisor (if the error is at deploy time) does not have the iSCSI client installed. Check that the machine has the open-iscsi package installed.

Persistent Process Does Not Work Because of Missing iSCSI Server

Symptoms

Check for a message like "could not send targets"

Solution

Communication between machines may be broken. A network error could have occurred because of a firewall, a timeout, etc. The iSCSI server may be down (stopped or broken). Check that the iSCSI service is up and running. Then check that you can telnet from the BPM machine to the storage machine, to the host port.
For more information, see storage servers.

If you suspect that the client service is down, run this command with one of the options on server:

/etc/init.d/iscsid [restart | stop | status]

In Nexenta:

To check if services are enabled:

root@nexenta: svcs svc:/network/iscsi/target:default svc:/system/stmf:default
STATE          STIME    FMRI
online         7:56:29  svc:/system/stmf:default
online         7:56:30  svc:/network/iscsi/target:default

To enable services in Nexenta (remember that you should log in to Nexenta as admin. Then change privileges by using su (and not su-))

root@nexenta: svcadm enable svc:/system/stmf:default
root@nexenta: svcadm enable svc:/network/iscsi/target:default

Persistent Process Does Not Work Because of Error in Cloning

Symptoms

• Check for a message like "disk doesn't have enough space"
• The conversion to RAW finished, the volume is created but the image is not dumped to the volume, resulting in the message:

[!] deploy: Error cloning
/opt/vm_repository/1/abiquo-repository.abiquo.com/s3direct/m0n0wall/formats/951c42ea-bd2c-4b4e-bef5-24c0f2f2394b-snapshot-m0n0wall-1.3b18-i386-flat.vmdk-RAW
to 10.60.1.29:iqn.2010-09.com.abiquo.lvm-qa:89020746-8b8c-4f04-bbce-829459f8892a\],
User[noname]

Solution

Delete volumes to free space in the storage pool.

Hyper-V Does Not Deploy Images

Symptoms

An error occurred when calling the deploy operation in a Hyper-V machine. This error appears in the log:

13:02:15.374 INFO  c.a.v.machine.impl.AbsHyperVMachine - Cloning the virtual disk from: \\10.60.1.72\opt\vm_repository\1\rs.bcn.abiquo.com\centos_vhd\cent
os.vhd to C:\\ABQ_50a782d5-e535-425d-a552-545eb699d849.vhd 
13:02:15.601 ERROR c.a.v.utils.hyperv.HyperVUtils - Monitoring job result error:
        ErrorCode: 32768 ErrorDescription: 'The system failed to convert '\\10.60.1.72\opt\vm_repository\1\rs.bcn.abiquo.com\centos_vhd\centos.vhd'. Error
 Code: The network name cannot be found.
13:02:15.614 ERROR c.a.v.machine.impl.AbsHyperVMachine - Failed to deploy machine :
org.jinterop.dcom.common.JIException: An error was occurred when calling an operation in a Hyper-V machine

Solution

Although the URI is correct, Hyper-V accesses the filesystem using Samba. In Samba, the directory is aliased as vm_repository. To change this directory go to the server, /opt/abiquo/config (remember that this path could be in a chroot system). Then edit the abiquo.properties file and enter the following property containing the IP address of the CIFS server:

    abiquo.virtualfactory.hyperv.repositoryLocation = //<SERVER_IP>/vm_repository/

and restart the Tomcat server.

Hyper-V Does Not Create an Instance

Symptoms

• Only happens in Hyper-V
• This error appears in Events:
  • Error bundling VirtualAppliance: number
  • An error occurred when calling an operation in a Hyper-V machine
• This error appears in the log:

ErrorCode: 32769 ErrorDescription: 'The system failed to convert 'C:\\localRepository\\e13fcbec-2475-46c1-9f90-66aa1f2850be.vhd'. Error Code: General
access denied error14:54:17.420 ERROR c.a.v.v.i.VirtualapplianceresourceDeployer - An error occurred: org.jinterop.dcom.common.JIException: An error was occurred
when calling an operation in a Hyper-V machineErrorCode: 32769. ErrorDescription:
'The system failed to convert 'C:\\localRepository\\e13fcbec-2475-46c1-9f90-66aa1f2850be.vhd'. Error Code: General access denied error

Solution

Check the smb.conf file is correctly configured. This file is configured during installation. For more information, see VM repository folder.

Hyper-V Does Not Deploy Persistent Virtual Machine with Storage

Symptoms

• Persistent VM is properly generated
• When deploying a persistent VM or a VM with storage, this message is displayed in the log: Connection refused.

Solution

The physical machine does not have the right port. Change it to the correct one. You can check this port in the hypervisortype table in the DB.

Cannot Deploy in XenServer

Symptoms

When deploying, notification in tracer: "SR_BACKEND_FAILURE_47The SR is not available [opterr=no such directory /var/run/sr-mount/fe1a9f24-3699-a02d-e226-dfc59db6b596]" .

Solution

NFS mount point is unreachable by XenServer. Connect to XenCenter and delete the Abiquo repositories (ONLY the Abiquo repositories) of the XenServer machines that are failing. To delete them you must Detach and Forget.
Once deleted, try to deploy again. The Abiquo repository should appear again in XenServer and the VM should deploy successfully.