QTIP Release Notes

Danube

This document provides the release notes for Danube of QTIP.

Version history

Date	Ver.	Author	Comment
2017-03-30	Danube 1.0	Yujun Zhang
2017-05-04	Danube 2.0	Yujun Zhang
2017-07-14	Danube 3.0	Yujun Zhang

Important notes

QTIP is totally reworked in Danube release. The legacy benchmarks released in Brahmaputra (compute, network and storage) are deprecated.

Summary

QTIP Danube release introduces QPI, a.k.a. QTIP Performance Index, which is calculated from metrics collected in performance tests.

A PoC of compute performance benchmark plan is provided as a sample use case.

Available benchmark plans can be listed, shown and executed from command line or over API.

Release Data

Project	QTIP
Repo/commit-ID	qtip/danube.3.0
Release designation	Tag update only
Release date	2017-07-14
Purpose of the delivery	OPNFV quality assurance

Version change

New in Danube 3.0

• No change in QTIP itself
• Validated on OPNFV Danube latest release

New in Danube 2.0

• Bug fix in regex of ssl

Module version changes

The following Python packages are used in this release:

ansible==2.1.2.0
click==6.7
connexion==1.1.5
Jinja2==2.9.5
numpy==1.12.1
paramiko==2.1.2
pbr==2.0.0
prettytable==0.7.2
six==1.10.0
PyYAML==3.12

It is considered as a baseline for future releases.

Reason for version

Features additions

• Compute QPI (QTIP Performance Index) specification and benchmarking plan
• Command line interface
• API server

Framework evolution

The following components are implemented and integrated

• Native runner
• File loader
• Ansible driver
• Logfile collector
• Grep parser
• Console reporter

See JIRA for full change log

Deliverables

Software

• QTIP Docker image (tag: danube.3.0)
• QTIP Docker image (tag: danube.2.0)
• QTIP Docker image (tag: danube.1.0)

Documentation

• Installation & Configuration
• User Guide
• Developer Guide

Known Limitations, Issues and Workarounds

Limitations

• The compute benchmark plan is hard coded in native runner
• Baseline for Compute QPI is not created yet, therefore scores are not available

Known issues

• QTIP-230 - logger warns about socket /dev/log when running in container

Test Result

QTIP has undergone QA test runs with the following results:

TEST-SUITES	Results:
qtip-verify-danube	94/94 passed
qtip-os-nosdn-kvm-ha-zte-pod3-daily-danube	passed
qtip-os-nosdn-nofeature-ha-zte-pod3-daily-danube	passed
qtip-os-odl_l2-nofeature-ha-zte-pod1-daily-danube	passed

Brahmaputra

NOTE: The release note for OPNFV Brahmaputra is missing. This is a copy of the README.

QTIP Benchmark Suite

QTIP is a benchmarking suite intended to benchmark the following components of the OPNFV Platform:

1. Computing components
2. Networking components
3. Storage components

The efforts in QTIP are mostly focused on identifying

1. Benchmarks to run
2. Test cases in which these benchmarks to run
3. Automation of suite to run benchmarks within different test cases
4. Collection of test results

QTIP Framework can now be called: (qtip.py).

The Framework can run 5 computing benchmarks:

1. Dhrystone
2. Whetstone
3. RamBandwidth
4. SSL
5. nDPI

These benchmarks can be run in 2 test cases:

1. VM vs Baremetal
2. Baremetal vs Baremetal

Instructions to run the script:

1. Download and source the OpenStack adminrc file for the deployment on which you want to create the VM for benchmarking
2. run python qtip.py -s {SUITE} -b {BENCHMARK}
3. run python qtip.py -h for more help
4. list of benchmarks can be found in the qtip/test_cases directory
5. SUITE refers to compute, network or storage

Requirements:

1. Ansible 1.9.2
2. Python 2.7
3. PyYAML

Configuring Test Cases:

Test cases can be found within the test_cases directory. For each Test case, a Config.yaml file contains the details for the machines upon which the benchmarks would run. Edit the IP and the Password fields within the files for the machines on which the benchmark is to run. A robust framework that would allow to include more tests would be included within the future.

Jump Host requirements:

The following packages should be installed on the server from which you intend to run QTIP.

1: Heat Client 2: Glance Client 3: Nova Client 4: Neutron Client 5: wget 6: PyYaml

Networking

1: The Host Machines/compute nodes to be benchmarked should have public/access network 2: The Host Machines/compute nodes should allow Password Login

QTIP support for Foreman

{TBA}