============================================================
Performance report for TestPMD implementation (only phy2phy deployment is supported)
============================================================

.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, Intel Corporation, AT&T and others.

Introduction
============
The objective of the OPNFV project titled **"Characterise vSwitch Performance
for Telco NFV Use Cases"**, is to evaluate a virtual switch to identify its
suitability for a Telco Network Function Virtualization (NFV) environment. As
well as this, the project aims to identify any gaps or bottlenecks in order to
drive architectural changes to improve virtual switch performance and
determinism. The purpose of this document is to summarize the results of the
tests carried out on the virtual switch in the Network Function Virtualization
Infrastructure (NFVI) and, from these results, provide evaluations and
recommendations for the virtual switch. Test results will be outlined in
details-of-LTR_, preceded by the document-identifier_ and the scope_ and
references_).

This document is currently in draft form.

.. _document-identifier:

Document identifier
-------------------
The document id will be used to uniquely identify versions of the LTR. The
format for the document id will be:
OPNFV\_vswitchperf\_LTR\_rel\_STATUS, the status is one of: DRAFT, REVIEWED,
CORRECTED or FINAL. The document id for this version of the LTR is:
OPNFV\_vswitchperf\_LTR\_Brahmaputra\_DRAFT.

.. _scope:

Scope
-----
The scope of this report is to detail the results of the tests that have been
performed on the virtual switch. This report will also evaluate the results of
these tests and, based on these evaluations, provide recommendations on the
suitability of the virtual switch for use in a Telco NFV environment.

.. _references:

References
----------
`OPNFV_vswitchperf_LTD_Brahmaputra_REVIEWED
<http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html>`__

.. _details-of-LTR:

Details of the Level Test Report
================================
This section provides a test-results-overview_. Also included are the rationale_
and the conclusions_.

.. _test-results-overview:



Test ID: PVP_TPUT
--------------------------

Test Environment
~~~~~~~~~~~~~~~~
Below is the environment that the test was performed in:

* OS: CentOS Linux 7 Core
* Kernel Version: 4.4.50-rt62-1703161746nfv
* NIC(s):
    * Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    * Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
* Board: Intel Corporation S2600WT2R [2 sockets]
* CPU:  Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
* CPU cores: 44
* Memory: 65939100 kB
* Virtual Switch Set-up: pvp
* vswitchperf: GIT tag: 7c4a2cdcdd99f309dccfe0a24e829bbf3afa95f9
* Traffic Generator: IxNet, Version: 8.01.1029.6, GIT tag: None
* vSwitch: none, Version: None, GIT tag: None
* VNF: QemuPciPassthrough, Version: 2.5.0, GIT tag: a8c40fa2d667e585382080db36ac44e216b37a1c
* VM images:
    * /home/jenkins/vloop-vnf-ubuntu-14.04_20160804.qcow2
* VM loopback apps:
    * testpmd, Version: 16.07.0, GIT tag: 20e2b6eba13d9eb61b23ea75f09f2aa966fa6325

Below are test details:

* Test ID: pvp_tput
* Description: LTD.Throughput.RFC2544.PacketLossRatio
* Deployment: pvp
* Traffic type: rfc2544
* Bidirectional : 

Test results for packet size: 64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A detailed summary of the main results is outlined below.

Results/Metrics Collected
^^^^^^^^^^^^^^^^^^^^^^^^^

The following are the metrics obtained during this test:

========================== ==================================
        Metric                          Result
========================== ==================================
tx_rate_fps                                           Unknown
throughput_rx_fps                                11346751.923
tx_rate_mbps                                          Unknown
throughput_rx_mbps                                   5809.537
tx_rate_percent                                        38.125
throughput_rx_percent                                  38.125
frame_loss_percent                                      0.000
min_latency_ns                                       4540.000
max_latency_ns                                      33900.000
avg_latency_ns                                      11632.500
type                                                  rfc2544
packet_size                                                64
traffic_type                                              udp
test_execution_time                                  00:16:49
guest_loopback_app                                    testpmd
========================== ==================================

Statistics collected
^^^^^^^^^^^^^^^^^^^^

The following system statistics were collected during testcase execution:

========================== ==================================
Process: qemu-system-x86
-------------------------------------------------------------
      Statistic                        Value
========================== ==================================
UID                                                         0
PID                                                     43678
%usr                                                     0.00
%system                                                  0.22
%guest                                                  99.66
%CPU                                                    99.80
CPU                                                         -
minflt/s                                                 0.00
majflt/s                                                 0.00
VSZ                                                   2689736
RSS                                                     76244
%MEM                                                     0.12
kB_rd/s                                                  0.00
kB_wr/s                                                  0.00
kB_ccwr/s                                                0.00
========================== ==================================


Test results for packet size: 128
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A detailed summary of the main results is outlined below.

Results/Metrics Collected
^^^^^^^^^^^^^^^^^^^^^^^^^

The following are the metrics obtained during this test:

========================== ==================================
        Metric                          Result
========================== ==================================
tx_rate_fps                                           Unknown
throughput_rx_fps                                 9575604.083
tx_rate_mbps                                          Unknown
throughput_rx_mbps                                   9805.419
tx_rate_percent                                        56.688
throughput_rx_percent                                  56.688
frame_loss_percent                                      0.000
min_latency_ns                                       4800.000
max_latency_ns                                      41460.000
avg_latency_ns                                      14901.500
type                                                  rfc2544
packet_size                                               128
traffic_type                                              udp
test_execution_time                                  00:16:49
guest_loopback_app                                    testpmd
========================== ==================================

Statistics collected
^^^^^^^^^^^^^^^^^^^^

The following system statistics were collected during testcase execution:

========================== ==================================
Process: qemu-system-x86
-------------------------------------------------------------
      Statistic                        Value
========================== ==================================
UID                                                         0
PID                                                     43678
%usr                                                     0.00
%system                                                  0.22
%guest                                                  99.66
%CPU                                                    99.80
CPU                                                         -
minflt/s                                                 0.00
majflt/s                                                 0.00
VSZ                                                   2689736
RSS                                                     76244
%MEM                                                     0.12
kB_rd/s                                                  0.00
kB_wr/s                                                  0.00
kB_ccwr/s                                                0.00
========================== ==================================


Test results for packet size: 512
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A detailed summary of the main results is outlined below.

Results/Metrics Collected
^^^^^^^^^^^^^^^^^^^^^^^^^

The following are the metrics obtained during this test:

========================== ==================================
        Metric                          Result
========================== ==================================
tx_rate_fps                                           Unknown
throughput_rx_fps                                 4699183.002
tx_rate_mbps                                          Unknown
throughput_rx_mbps                                  19247.854
tx_rate_percent                                           100
throughput_rx_percent                                     100
frame_loss_percent                                      0.000
min_latency_ns                                       5620.000
max_latency_ns                                      48600.000
avg_latency_ns                                      40739.500
type                                                  rfc2544
packet_size                                               512
traffic_type                                              udp
test_execution_time                                  00:16:49
guest_loopback_app                                    testpmd
========================== ==================================

Statistics collected
^^^^^^^^^^^^^^^^^^^^

The following system statistics were collected during testcase execution:

========================== ==================================
Process: qemu-system-x86
-------------------------------------------------------------
      Statistic                        Value
========================== ==================================
UID                                                         0
PID                                                     43678
%usr                                                     0.00
%system                                                  0.22
%guest                                                  99.66
%CPU                                                    99.80
CPU                                                         -
minflt/s                                                 0.00
majflt/s                                                 0.00
VSZ                                                   2689736
RSS                                                     76244
%MEM                                                     0.12
kB_rd/s                                                  0.00
kB_wr/s                                                  0.00
kB_ccwr/s                                                0.00
========================== ==================================


Test results for packet size: 1024
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A detailed summary of the main results is outlined below.

Results/Metrics Collected
^^^^^^^^^^^^^^^^^^^^^^^^^

The following are the metrics obtained during this test:

========================== ==================================
        Metric                          Result
========================== ==================================
tx_rate_fps                                           Unknown
throughput_rx_fps                                 2394601.729
tx_rate_mbps                                          Unknown
throughput_rx_mbps                                  19616.577
tx_rate_percent                                           100
throughput_rx_percent                                     100
frame_loss_percent                                      0.000
min_latency_ns                                       6380.000
max_latency_ns                                      53100.000
avg_latency_ns                                      42665.500
type                                                  rfc2544
packet_size                                              1024
traffic_type                                              udp
test_execution_time                                  00:16:49
guest_loopback_app                                    testpmd
========================== ==================================

Statistics collected
^^^^^^^^^^^^^^^^^^^^

The following system statistics were collected during testcase execution:

========================== ==================================
Process: qemu-system-x86
-------------------------------------------------------------
      Statistic                        Value
========================== ==================================
UID                                                         0
PID                                                     43678
%usr                                                     0.00
%system                                                  0.22
%guest                                                  99.66
%CPU                                                    99.80
CPU                                                         -
minflt/s                                                 0.00
majflt/s                                                 0.00
VSZ                                                   2689736
RSS                                                     76244
%MEM                                                     0.12
kB_rd/s                                                  0.00
kB_wr/s                                                  0.00
kB_ccwr/s                                                0.00
========================== ==================================


Test results for packet size: 1518
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A detailed summary of the main results is outlined below.

Results/Metrics Collected
^^^^^^^^^^^^^^^^^^^^^^^^^

The following are the metrics obtained during this test:

========================== ==================================
        Metric                          Result
========================== ==================================
tx_rate_fps                                           Unknown
throughput_rx_fps                                 1625466.218
tx_rate_mbps                                          Unknown
throughput_rx_mbps                                  19739.662
tx_rate_percent                                           100
throughput_rx_percent                                     100
frame_loss_percent                                      0.000
min_latency_ns                                       6860.000
max_latency_ns                                      47340.000
avg_latency_ns                                      38604.500
type                                                  rfc2544
packet_size                                              1518
traffic_type                                              udp
test_execution_time                                  00:16:49
guest_loopback_app                                    testpmd
========================== ==================================

Statistics collected
^^^^^^^^^^^^^^^^^^^^

The following system statistics were collected during testcase execution:

========================== ==================================
Process: qemu-system-x86
-------------------------------------------------------------
      Statistic                        Value
========================== ==================================
UID                                                         0
PID                                                     43678
%usr                                                     0.00
%system                                                  0.22
%guest                                                  99.66
%CPU                                                    99.80
CPU                                                         -
minflt/s                                                 0.00
majflt/s                                                 0.00
VSZ                                                   2689736
RSS                                                     76244
%MEM                                                     0.12
kB_rd/s                                                  0.00
kB_wr/s                                                  0.00
kB_ccwr/s                                                0.00
========================== ==================================



Anomalies
~~~~~~~~~~
No anomalies were detected during the course of this test.

Testing Activities/Events
~~~~~~~~~~~~~~~~~~~~~~~~~
pidstat is used to collect the process statistics, as such some values such as
%CPU and %USER maybe > 100% as the values are summed across multiple cores. For
more info on pidstat please see: http://linux.die.net/man/1/pidstat.

Known issues: Some reported metrics have the value "unkown". These values are
marked unknown as they are not values retrieved from the external tester
(traffic generator). They were incorrectly derived in a way that made assumptions
about packet sizes, as such they have been deprecated from vsperf and marked as
unknown. They will be resolved in the next release.

.. There must be blank lines around to ensure correct formatting.

.. This work is licensed under a Creative Commons Attribution 4.0 International License.
   http://creativecommons.org/licenses/by/4.0

.. _rationale:

Rationale for decisions
=======================
The tests conducted do not have pass/fail/conditional-pass criteria. The test
is simply conducted and the results are reported.

.. _conclusions:

Conclusions and recommendations
===============================
The test results are stable. The vsperf CI jobs that were used to obtain the
results can be found at https://wiki.opnfv.org/wiki/vsperf_results.

General
========

Glossary
--------
- NFV - Network Function Virtualization
- Mbps - 1,000,000bps

Document change procedures and history
--------------------------------------
=============================================== ================= =============
             Document ID                            Author        Date Modified
=============================================== ================= =============
`OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT` Christopher Nolan 23/01/2015
`OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT` Christopher Nolan 28/01/2015
=============================================== ================= =============

