Mitel Virtualization Diagnostics Tool

The intended use of the Virtualization Diagnostic tool is to pinpoint performance and voice quality issues found when running Mitel applications in a virtual environment. The tool is especially helpful for customers who do not have control of the underlying infrastructure but are interested in determining the cause of problems.

The Diagnostic tool is a component of the Mitel Virtualization Framework (MVF) and includes a "Mitel Virtualization" screen that appears within the MSL Server Manager. The screen enables you to obtain an overview of the virtual machine and MVF properties, manage storage monitoring, receive a diagnostic overview, configure a connection to the vCenter server or ESXi hypervisor, and run the diagnostic tool to generate a variety of log files containing statistical, performance and configuration data.

Supported Applications

To employ the Diagnostics tool, you require the following:

  • Operating System: MSL 10.0 or higher

  • VMware environment: vSphere 4.1 or higher

  • Mitel Virtual Framework: MVF 2.0 or higher

Reviewing the Virtual Machine Properties

The "Virtual Machine Properties" table displays information concerning the Virtual Machine and Mitel Virtual Framework. The information is presented in two columns:

  • Current Dimensions: Lists the configuration at the time that the current Mitel Virtualization page was loaded. Refreshing the page resets the settings.

  • First Boot Dimensions: Lists the configuration after the Mitel Open Virtual Appliance (OVA) package has been installed and the settings configured, but before the virtual machine has been powered on for the first time.

To review the virtual machine properties:

  1. Under Administration, click Virtualization.

  2. Under Virtual Machine Properties, review the following settings:

    Setting Description
    MVF Version The version number of the Mitel Virtualization Framework (MVF), a software package that enable Mitel applications to run in a virtual infrastructure. MVF has the capacity to support multiple operating systems and hypervisor products.
    Virtualization Agent Version (VMware Tools) The version number of VMware Tools, a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine.
    Hypervisor Version The version number of the VMware ESXi hypervisor that hosts one or more virtual machines and their "guest" operating systems.
    vCPU count The number of virtual Central Processing Units (vCPUs) configured on this virtual machine.
    Memory (MB) The amount of virtual physical memory available for use by the operating system on this virtual machine.
    Disk size (GB) The virtual disk size available for use by the operating system on this virtual machine.
    NIC count The number of virtual network interface cards configured on this virtual machine.
    CPU Reservation (MHz) The guaranteed minimum allocation of CPU resources for this virtual machine.
    Memory Reservation (MB) The guaranteed minimum allocation of memory resources for this virtual machine.
    CPU Limit (MHz) The upper limit of CPU resources that can be allocated to this virtual machine. This limit is expressed in concrete units (Megahertz) and cannot be exceeded.
    Memory Limit (MB) The upper limit of for memory resources that can be allocated to this virtual machine. This limit is expressed in concrete units (Megabytes) and cannot be exceeded.
    vCPU Speed (MHz)

    The speed of the virtual CPU, which is dependant on the speed of your underlying processor. So if you have a 12 cores and a processor speed of 3.36GHz, that means a virtual machine with a single vCPU running a single threaded application can consume 3.36GHz.

    The setting defines what a single vCPU will consume, not the aggregated amount among multiple vCPUs on a single virtual machine. Accordingly, if you have two vCPUs this figure should be doubled.

Managing Storage Monitoring

Use this tool to detect to degrading storage conditions and take corrective actions.

To manage the storage monitoring settings:

  1. Under Administration, click Storage Monitoring.

  2. Under Storage Monitoring, enter the following settings:

    Setting Description
    File System Monitoring

    Use this setting to specify whether file system monitoring is enabled or disabled. If the feature is enabled (the default), the system will check for disk I/O errors every five seconds. If any errors are detected, a warning notification is sent to the "admin" email address configured on the Email Settings screen.

    The following errors are monitored:

    • File system errors: Errors related to storage degradation

    • CPU Starvation: When the monitoring process is not dispatched within a specified time (default is 5 seconds),

    • High I/O Latency: When I/O operations exceed the configured duration time.
    Reboot on Read-Only State
    If this setting is enabled (the default), the system will automatically reboot whenever it enters read-only state. After the system reboots, all disk I/O errors will be cleared and the system will be in read-write state.
    Note:

    • File System Monitoring must be enabled before this feature can be employed.

    • Read-only state occurs when there are I/O errors on the virtual machine disk drives, and is intended to protect the file system from damage.

  3. Click Save.

Reviewing the Diagnostic Overview

The Virtualization Diagnostic tool constantly monitors the system in order to report on three alarm conditions and the state of the last nightly analysis.

To review the virtualization diagnostics overview:

  1. Under Administration, click Virtualization.

  2. Under Diagnostic Overview, review the following settings:
    Condition Description States
    Hypervisor Version Indicates whether or not the version of VMware ESXi Hypervisor is supported. The Hypervisor is also known as the Virtual Machine Monitor (VMM).

    • Supported - Your ESXi version is supported and no changes are required.

    • Unsupported - Your ESXi version is not supported and you must switch to a supported version in order to restore monitoring functionality. For example, if you are running ESXi 4.0 or earlier, you must upgrade to version 4.1 or later.
    Current Dimensions Indicates whether the currently configured application resource dimensions are supported.

    • Supported - Your configuration is supported and no changes are required.

    • Unsupported - Your configuration is not supported due to a setting (vCPU count, Memory, Disk size, or NIC count) being out of boundaries. To resolve any performance issues, do the following:

    1. Revert to the default configuration for your deployment. For details, see Default Configurations.

    2. Contact Mitel Product Support for assistance.
    AMC Connectivity Indicates whether the Virtual Machine can connect to the Mitel Application Management Center (AMC) for licensing purposes.

    • Connected - Your Virtual Machine can connect to the AMC.

    • Error - Your Virtual Machine cannot connect to the AMC. Check the networking configuration and Application Resource ID (ARID). See the Mitel Standard Linux Installation and Maintenance Guide for more information.
    Last Nightly Analysis Indicates the date and time that the last nightly analysis was completed, and whether any problems occurred while it was being run. Upon successful completion, the nightly analysis generates the following log file: NIGHTLY-REPORT-YYYY-MM-DD.txt YYYY/MM/DD & Problems (if any)

Configuring the Virtualization Diagnostics Credentials

To enable the Virtualization Diagnostics tool to collect statistics for the virtual machine and the host, and then use the statistics to generate log files, you must enter credentials for the vCenter server or ESXi hypervisor.

The information collected depends on the credentials entered:

  • Admin login to vCenter - full range of features and statistics.

  • Read-only login to vCenter - subset of features and statistics.

  • Read-only login to ESXi - subset of features and statistics.

  • No credentials - Allocation and Reservation & Limits information only.

Note: For optimum results, enter credentials for the vCenter. Entering credentials for the ESXi may result in connectivity problems if settings are changed on the hypervisor.

Enter New Credentials

To enter the virtualization diagnostics credentials:

  1. Under Administration, click Virtualization.

  2. Under Virtualization Diagnostics, enter the following settings:
    Setting Description
    FQDN or IP address Enter the Fully Qualified Domain Name or IP address of the vCenter or ESXi hypervisor.
    Username Enter the username required to access the vCenter or ESXi hypervisor.
    Password Enter the password required to access vCenter or ESXi hypervisor.
    Nightly Analysis Time

    Specify the one-hour period during which the nightly analysis will be run each day. Select hours between 0-1 and 23-24.

    Upon successful completion, the nightly analysis generates the following log file: NIGHTLY-REPORT-YYYY-MM-DD.txt.

  3. Click Save.

Once a connection is established, the system will obtain performance statistics for the virtual machine and the host, and you may click the Run Diagnostics button in order to manually generate log files and an online report. For more information, see Manually Generated Log Files.

Note: For a newly installed system, wait for it to collect statistics for at least 15 minutes before clicking the Run Diagnostics button.

Remove Current Credentials

To remove the virtualization diagnostics credentials:

  1. Under Administration, click Virtualization.

  2. Under Virtualization Diagnostics, click Remove.

You may now enter new credentials.

Note: Without credentials, the system will not collect statistics or generate log files for virtualization diagnostics.

Reviewing the Log Files

The system generates log files containing performance and configuration data plus statistical events.

To view and/or download the log files:

Automatically Generated Log Files

The following log files are generated automatically by the system on a periodic basis.
Report Name Description
NIGHTLY-REPORT-YYYY-MM-DD.txt This report contains the previous day's detailed performance and configuration information, and is generated daily in the Nightly Analysis Time you have specified. The system retains seven reports, deleting the oldest file after seven days.
VM-STATS-YYYY-MM-DD.csv This report contains virtual machine statistics for the previous week. The system polls for new data every 15 minutes and deletes existing data after seven days.
HOST-STATS-YYYY-MM-DD.csv This report contains host system statistics for the previous week. The system polls for new data every 15 minutes and deletes existing data after seven days.
ALL-CONFIG-YYYY-MM-DD.csv This report contains all CPU, performance and network configuration statistics concerning the host and virtual machine for the previous week. The system polls for new data every 15 minutes and deletes existing data after seven days.

Manually Generated Log Files

A number of log files are created when you request them.

To manually generate the log files and an online report:

  1. Under Administration, click Virtualization.

  2. Under Virtualization Diagnostics, click Run Diagnostics.

Note:

  • For a newly installed system, allow it to collect statistics for at least 15 minutes before you click the Run Diagnostics button.

  • If you repeatedly click the Run Diagnostics button, you may exceed the storage capacity of the host server's hard drive.

Report Name Description
USER-REPORT-YYYY-MM-DD-HH-MM-SS.txt

This report is similar to the NIGHTLY-REPORT-YYYY-MM-DD.txt report, but contains detailed performance and configuration information for the previous week (rather than a single day), collected from the moment you click the Run Diagnostics button. The report file is retained for seven days and then deleted.

In the event you cannot resolve a problem by yourself, Mitel Product Support will request that you obtain this log file and send it to them. For details, see View/Download Log Files

USER-SUMMARY.tmp

This report is an abbreviated version of USER-REPORT-YYYY-MM-DD-HH-MM-SS.txt report. It contains performance and configuration overviews for each day of the previous week.

This report is presented in two formats:

  • Displayed on the Mitel Virtualization screen. This report is retained until you navigate away from the screen.

  • Recorded in the log files. This file is retained until the Run Diagnostics button is clicked again.
VM-EVENTS-YYYY-MM-DD.csv The report contains 15 days' activity regarding the operation of the Virtual Machine. This file is retained until the Run Diagnostics button is clicked again.

Log File Contents

Although the log files are primarily intended for use by Mitel Product support, you may use them to troubleshoot basic issues with the following issues:

  • Performance Problems: The system analyzes performance data and if it detects five consecutive "out of bounds" events, an problem will be reported. For example, if the virtual machine waits longer than two seconds to be serviced by the host, five times in a row, the system will report a "CPU Ready" error. Note that system events are registered every twenty seconds.

  • Configuration Problems: The system checks configuration data and statistical events on an ongoing basis. If a problem is found, and error is logged immediately.

See Analysis Tuning Parameters for detailed information concerning the system settings which control the generation of log file problems.
Performance Problems Description
CPU Ready (seconds) The virtual machine has exceeded the maximum amount of time that it can wait to be run on the physical CPU(s). The default is 2 seconds.
CPU Usage (percent) The virtual machine has exceeded its CPU capacity limit, which is expressed as a percentage of the total amount available. For example, with a limit of 50%, if the virtual machine has four CPUs with 2 GHz processors, and you are running an application that requires 6 GHz (75% of capacity), the limit has been exceeded by 25%. The default is 50%.
Disk Latency (seconds) The virtual machine has exceeded the maximum amount of time permitted for a SCSI command to be issued by the guest operating system to the virtual machine hard disk. The default is 0.02.
Network Usage (MB) The virtual machine has exceeded the maximum network utilization (combined transmit and receive rates, in Megabytes per second). The default is 50.0 MB.
Memory Swapped (MB) The virtual machine has exceeded the maximum amount of memory, in Megabytes, that can be swapped into memory from disk. The default is 0 MB.
Memory Use (MB) The virtual machine has exceeded the maximum amount of memory capacity that it can use, expressed as a percentage of the total amount available. For example, if the virtual machine has 4 GHz of memory, and you are running an application that requires 3 GHz (75% of capacity), an event will be registered. The default value is 50%.
Number of Packets Dropped (average) The virtual machine has exceeded the maximum number of received packets that can be dropped at the network interface. The default value is 0.
Disk Usage (MB) The virtual machine has exceeded the maximum amount of data, in Megabytes per second, that can be read from the virtual machine hard disk. The default value is 30 MB.
Configuration Detections Description (Yes/No)
High VM-to-host CPU ratio If "Yes" is displayed, the ESXi host has exceeded the virtual CPU to host CPU ratio, which is 0.79 by default. For example, if five virtual machines with 4 GHz vCPUs are powered on, and the host has 8 physical/16 logical cores, then the ratio is 4 + 4 + 4 + 4 + 4 ÷ 16 = 1.25. Since 1.25 exceeds 0.79, a potential configuration issue is detected.
High VM-to-host Memory ratio If "Yes" is displayed, the ESXi host has exceeded the virtual memory to host memory ratio, which is 1.20 by default. For example, if five VMs are powered on, each using 2 GHz of memory, and the host has 8 GHz of physical memory, then the ratio is 2 + 2 + 2 + 2 + 2 ÷ 8 = 1.25, which will cause an event to be registered.
Snapshots Present If "Yes" is displayed, the system checks to determine if snapshots are supported on the virtual machine. Because snapshots create considerable disk I/O load, use of this feature may degrade the voice quality of calls.
Low CPU Speed (MHz) If "Yes" is displayed, the maximum speed of the virtual CPU, which is dependant on the speed of the underlying processor on the ESXi host, has been exceeded.
No Hyperthreading (Ignore if running on non-Intel processor)

If "Yes" is displayed, the system checks to determine if hyperthreading is enabled on the ESXi host.

Note: This parameter can only report on Intel processors that support hyperthreading. It cannot report on AMD or other non-Intel processors.
vMotion occurred If "Yes" is displayed, the system checks to determine if vMotion is enabled on the ESXi host.
Low CPU Reservation (MHz) If "Yes" is displayed, the guaranteed minimum allocation of CPU resources for this virtual machine has been exceeded.
Low Memory Reservation (MB) If "Yes" is displayed, the guaranteed minimum allocation of memory resources for this virtual machine has been exceeded.
Parent topic: Administration
↑