QTIP Design Specifications¶

QTIP Architecture Design¶

In Danube, QTIP releases its standalone mode, which is also know as solo:

The runner could be launched from CLI (command line interpreter) or API (application programming interface) and drives the testing jobs. The generated data including raw performance data and testing environment are fed to collector. Performance metrics will be parsed from the raw data and used for QPI calculation. Then the benchmark report is rendered with the benchmarking results.

The execution can be detailed in the diagram below:

QTIP Command Line Interface¶

Abstract¶

QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI’s) such as compute, storage and network. A plan consists of one or more QPI’s, depending upon how the end user would want to measure performance. CLI is designed to help the user, execute benchmarks and view respective scores.

Framework¶

QTIP CLI has been created using the Python package Click, Command Line Interface Creation Kit. It has been chosen for number of reasons. It presents the user with a very simple yet powerful API to build complex applications. One of the most striking features is command nesting.

As explained, QTIP consists of metrics, QPI’s and plans. CLI is designed to provide interface to all these components. It is responsible for execution, as well as provide listing and details of each individual element making up these components.

Design¶

CLI’s entry point extends Click’s built in MultiCommand class object. It provides two methods, which are overridden to provide custom configurations.

class QtipCli(click.MultiCommand):

    def list_commands(self, ctx):
        rv = []
        for filename in os.listdir(cmd_folder):
            if filename.endswith('.py') and \
                filename.startswith('cmd_'):
                rv.append(filename[4:-3])
        rv.sort()
        return rv

    def get_command(self, ctx, name):
        try:
            if sys.version_info[0] == 2:
                name = name.encode('ascii', 'replace')
            mod = __import__('qtip.cli.commands.cmd_' + name,
                             None, None, ['cli'])
        except ImportError:
            return
        return mod.cli

Commands and subcommands will then be loaded by the get_command method above.

Extending the Framework¶

Framework can be easily extended, as per the users requirements. One such example can be to override the builtin configurations with user defined ones. These can be written in a file, loaded via a Click Context and passed through to all the commands.

class Context:

    def __init__():

        self.config = ConfigParser.ConfigParser()
        self.config.read('path/to/configuration_file')

    def get_paths():

        paths = self.config.get('section', 'path')
        return paths

The above example loads configuration from user defined paths, which then need to be provided to the actual command definitions.

from qtip.cli.entry import Context

pass_context = click.make_pass_decorator(Context, ensure=False)

@cli.command('list', help='List the Plans')
@pass_context
def list(ctx):
    plans = Plan.list_all(ctx.paths())
    table = utils.table('Plans', plans)
    click.echo(table)

Which framework has been used and why
How to extend to more api

Compute QPI¶

The compute QPI gives user an overall score for system compute performace.

Summary¶

The compute QPI are calibrated a ZTE E9000 server as a baseline with score of 2500 points. Higher scores are better, with double the score indicating double the performance. The compute QPI provides three different kinds of scores:

Workload Scores
Section Scores
Compute QPI Scores

Baseline¶

ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory.

Workload Scores¶

Each time a workload is executed QTIP calculates a score based on the computer’s performance compared to the baseline performance.

Section Scores¶

QTIP uses a number of different tests, or workloads, to measure performance. The workloads are divided into five different sections:

Section	Detail	Indication
Integer	Integer workloads measure the integer instruction performace of host or vm by performing Dhrystone test.	All app relies on integer performance
Floating point	Floating point workloads measure the floating pointperfo rmance by performing Whetstone test.	Floating point performance is especially important in video games,digital content creation applications.
Memory	Memory workloads measure memory bandwidth by performing RamSpeed test.	Software working with cipher large amounts data relies on SSL Performace.
DPI	DPI workloads measure deep-packet inspection speed by performing nDPI test.	Software working with network packet anlysis relies on DPI performance.
SSL	SSL Performance workloads measure cipher speeds by using the OpenSSL tool.	Software working with cipher large amounts data relies on SSL Performace

A section score is the geometric mean of all the workload scores for workloads that are part of the section. These scores are useful for determining the performance of the computer in a particular area.

Compute QPI Scores¶

The compute QPI score is the weighted arithmetic mean of the five section scores. The compute QPI score provides a way to quickly compare performance across different computers and different platforms without getting bogged down in details.