QTIP Design Specifications

Dashboard

The dashboard gives user an intuitive view of benchmark result.

Purpose

The basic element to be displayed is QPI a.k.a. QTIP Performance Index. But it is also important to show user

  1. How is the final score calculated?
  2. Under what condition is the test plan executed?
  3. How many runs of a performance tests have been executed and is there any deviation?
  4. Comparison of benchmark result from different PODs or configuration

Templates

Different board templates are created to satisfy the above requirements.

Composition

QTIP gives a simple score but there must be a complex formula behind it. This view explains the composition of the QPI.

Condition

The condition of a benchmark result includes

  • System Under Test
    • Hardware environment
    • Hypervisor version
    • Operation System release version
    • System Configuration
  • Test Tools
    • Release version
    • Configuration
  • Test Facility
    • Laboratory
    • Engineer
    • Date

Conditions that do NOT have an obvious affect on the test result may be ignored, e.g. temperature, power supply.

Stats

Performance tests are actually measurement of specific metrics. All measurement comes with uncertainty. The final result is normally one or a group of metrics calculated from many repeats.

For each metric, the stats board shall consist of a diagram of all measured values and a box of stats:

^                                                  +------------+
|                                                  |  count: ?  |
|                                                  |average: ?  |
|                                                  |    min: ?  |
|                   X                              |    max: ?  |
| XXXX          XXXX X              XXXXX          |            |
|X    XX      XX      XX XXX     XXX     XX        |            |
|       XXXXXX          X   XXXXX          XX      |            |
|                                                  |            |
|                                                  |            |
|                                                  |            |
|                                                  |            |
|                                                  |            |
+--------------------------------------------->    +------------+

The type of diagram and selection of stats shall depend on what metric to show.

Comparison

Comparison can be done between different PODs or different configuration on the same PODs.

In a comparison view, the metrics are displayed in the same diagram. And the parameters are listed side by side.

Both common parameters and different parameters are listed. Common values are merged to the same cell. And user may configure the view to hide common rows.

A draft design is as following:

^
|
|
|
|           XXXXXXXX
|         XXX      XX+-+ XXXXXXXXXX
|      XXX          +XXXX         XXXXX
+-+XX X         +--+    ++            XXXXXX     +-+
| X+-+X   +----+          +-+              +----+X
|X    +--+                   +---+         XXXXXX X
|                                 +-------+        X
|
|
+----------------------------------------------------->

+--------------------+----------------+---------------+
| different param 1  |                |               |
|                    |                |               |
+-----------------------------------------------------+
| different param 2  |                |               |
|                    |                |               |
+-------------------------------------+---------------+
| common param 1     |                                |
|                    |                                |
+-------------------------------------+---------------+
| different param 3  |                |               |
|                    |                |               |
+-------------------------------------+---------------+
| common param 2     |                                |
|                    |                                |
+--------------------+--------------------------------+
                                         +------------+
                                         | HIDE COMMON|
                                         +------------+

Time line

Time line diagram for analysis of time critical performance test:

+-----------------+-----------+-------------+-------------+-----+
|                 |           |             |             |     |
+----------------->           |             |             |     |
|                 +----------->             |             |     |
|                 ? ms        +------------->             |     |
|                             ? ms          +------------>+     |
|                                           ? ms          ? ms  |
|                                                               |
+---------------------------------------------------------------+

The time cost between checkpoints shall be displayed in the diagram.

Compute QPI

The compute QPI gives user an overall score for system compute performace.

Summary

The compute QPI are calibrated a ZTE E9000 server as a baseline with score of 2500 points. Higher scores are better, with double the score indicating double the performance. The compute QPI provides three different kinds of scores:

  • Workload Scores
  • Section Scores
  • Compute QPI Scores

Baseline

ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory.

Workload Scores

Each time a workload is executed QTIP calculates a score based on the computer’s performance compared to the baseline performance.

Section Scores

QTIP uses a number of different tests, or workloads, to measure performance. The workloads are divided into five different sections:

Section Detail Indication
Integer Integer workloads measure the integer instruction performace of host or vm by performing Dhrystone test. All app relies on integer performance
Floating point Floating point workloads measure the floating pointperfo rmance by performing Whetstone test. Floating point performance is especially important in video games,digital content creation applications.
Memory Memory workloads measure memory bandwidth by performing RamSpeed test. Software working with cipher large amounts data relies on SSL Performace.
DPI DPI workloads measure deep-packet inspection speed by performing nDPI test. Software working with network packet anlysis relies on DPI performance.
SSL SSL Performance workloads measure cipher speeds by using the OpenSSL tool. Software working with cipher large amounts data relies on SSL Performace

A section score is the geometric mean of all the workload scores for workloads that are part of the section. These scores are useful for determining the performance of the computer in a particular area.

Compute QPI Scores

The compute QPI score is the weighted arithmetic mean of the five section scores. The compute QPI score provides a way to quickly compare performance across different computers and different platforms without getting bogged down in details.

Integration with Yardstick

Problem description

For each specified QPI [1], QTIP needs to select a suite of test cases and collect required test results. Based on these results, Qtip calculates the score.

Proposed change

Qtip has a flexible architecture [2] to support different mode: standalone and agent. It is recommended to use agent mode to work with existing test runners. Yardstick will act as a runner to generate test result and trigger Qtip agent on the completion of test.

Work Items in Yardstick

  1. Create a customized suite in Yardstick

Yardstick not only has many existing suites but also support customized suites. Qtip could create a suite named Qtip-PoC in Yardstick repo to verify workflow of Qtip agent mode.

  1. Launch Qtip in Yardstick

Whether to launch Qtip will be determined by checking the existence of OS environment variable QTIP. If it exists, Qtip will be launched by using Yardstick CLI yardstick plugin install [3].

  1. Yardstick interacts with Qtip

See Yardstick-Qtip+integration for details.

Work Items in Qtip

  1. Provide an API for Yardstick to post test result and environment info

After completing test execution, Yardstick will post test result and enviroment info with JSON format via QTIP API. See Yardstick-Qtip+integration for details.

  1. Parse yardstick test result

When Qtip agent receive Yarstick test result and enviroment info, Qtip agent will extract metrics which is definded in metric spec configuration file. Based on these metrics, Qtip agent will caculate QPI.

  1. Provide an API for querying QPI

Qtip will provide an API for querying QPI. See Yardstick-Qtip+integration for details.

Implementation

Assignee(s)

Primary assignee:
wu.zhihui
Other contributors
TBD

Testing

The changes will be covered by new unit test.

Documentation

TBD