VSPERF Results

1. VSPERF Test Scenarios

Predefined Tests suitable for automated execution with CI:

Test Definition
phy2phy_tput PacketLossRatio for Phy2Phy
back2back BackToBackFrames for Phy2Phy
phy2phy_tput_mod_vlan PacketLossRatioFrameModification for Phy2Phy
phy2phy_cont Phy2Phy blast vswitch at x% TX rate and measure throughput
pvp_cont PVP blast vswitch at x% TX rate and measure throughput
pvvp_cont PVVP blast vswitch at x% TX rate and measure throughput
phy2phy_scalability Scalability0PacketLoss for Phy2Phy
pvp_tput PacketLossRatio for PVP
pvp_back2back BackToBackFrames for PVP
pvvp_tput PacketLossRatio for PVVP
pvvp_back2back BackToBackFrames for PVVP
phy2phy_cpu_load CPU0PacketLoss for Phy2Phy
phy2phy_mem_load Same as CPU0PacketLoss but using a memory intensive app

Deployment topologies:

  • Phy2Phy: Physical port -> vSwitch -> Physical port.
  • PVP: Physical port -> vSwitch -> VNF -> vSwitch -> Physical port.
  • PVVP: Physical port -> vSwitch -> VNF -> vSwitch -> VNF -> vSwitch -> Physical port.

Loopback applications in the Guest:

Supported traffic generators:

  • Spirent Testcenter
  • Ixia: IxOS and IxNet.
  • Xena
  • MoonGen
  • Dummy
  • T-Rex

2. OPNFV Test Results

VSPERF CI jobs are run daily and sample results can be found at https://wiki.opnfv.org/display/vsperf/Vsperf+Results

Testcase names shown in the dashboard are combination of orignal testcase name from VSPERF framework and indication of used vswitch.

Example:

Testcase phy2phy_tput is executed for three vSwitch types: OvsDpdkVhost, OvsVanilla and VppDpdkVhost. In this case, following testcase names will be used in the dashboard: phy2phy_tput_ovsdpdkvhost, phy2phy_tput_ovsvanilla and phy2phy_tput_vppdpdkvhost.

In case of RFC2544 Throughput test, the recorded metric is FPS (frames per second) without packet loss. For RFC2544 Back2Back test, the recorded metric is back-to-back value (number of frames) without packet loss.

The loopback application in the VNF used for PVP and PVVP scenarios was DPDK testpmd.

Guest interface types are vhost-user for OvsDpdkVhost and VppDpdkVhost and virtio-net for OvsVanilla.

3. Performance report for OvsDpdkVhost

3.1. Introduction

The objective of the OPNFV project titled “Characterise vSwitch Performance for Telco NFV Use Cases”, is to evaluate a virtual switch to identify its suitability for a Telco Network Function Virtualization (NFV) environment. As well as this, the project aims to identify any gaps or bottlenecks in order to drive architectural changes to improve virtual switch performance and determinism. The purpose of this document is to summarize the results of the tests carried out on the virtual switch in the Network Function Virtualization Infrastructure (NFVI) and, from these results, provide evaluations and recommendations for the virtual switch. Test results will be outlined in details-of-LTR, preceded by the document-identifier and the scope and references).

This document is currently in draft form.

3.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTR. The format for the document id will be: OPNFV_vswitchperf_LTR_rel_STATUS, the status is one of: DRAFT, REVIEWED, CORRECTED or FINAL. The document id for this version of the LTR is: OPNFV_vswitchperf_LTR_Brahmaputra_DRAFT.

3.1.2. Scope

The scope of this report is to detail the results of the tests that have been performed on the virtual switch. This report will also evaluate the results of these tests and, based on these evaluations, provide recommendations on the suitability of the virtual switch for use in a Telco NFV environment.

3.2. Details of the Level Test Report

This section provides a test-results-overview. Also included are the rationale and the conclusions.

3.2.1. Test ID: PHY2PHY_TPUT

3.2.1.1. Test Environment

Below is the environment that the test was performed in:

  • OS: CentOS Linux 7 Core

  • Kernel Version: 4.4.116-1.el7.elrepo.x86_64

  • NIC(s):
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

  • Board: Intel Corporation S2600WT2R [2 sockets]

  • CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

  • CPU cores: 44

  • Memory: 65917716 kB

  • Virtual Switch Set-up: p2p

  • vswitchperf: GIT tag: 96de8654952a606d28d56c057ba871b5553e4176

  • Traffic Generator: IxNet, Version: 8.01.1029.6, GIT tag: None

  • vSwitch: OvsDpdkVhost, Version: 2.8.1, GIT tag: 5c67e9cfc1eaab19187720c857841be6c8046c40

  • DPDK Version: 17.08.0, GIT tag: 02657b4adcb8af773e26ec061b01cd7abdd3f0b6

Below are test details:

  • Test ID: phy2phy_tput
  • Description: LTD.Throughput.RFC2544.PacketLossRatio
  • Deployment: p2p
  • Traffic type: rfc2544_throughput
  • Bidirectional : True
3.2.1.2. Test results for packet size: 64

A detailed summary of the main results is outlined below.

3.2.1.2.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 23546936.935
tx_rate_mbps Unknown
throughput_rx_mbps 12056.032
tx_rate_percent 79.117
throughput_rx_percent 79.117
frame_loss_percent 0.000
min_latency_ns 4660.000
max_latency_ns 38800.000
avg_latency_ns 12970.500
type rfc2544_throughput
packet_size 64
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:13:18
start_time 2018-11-02 17:27:47
stop_time 2018-11-02 17:41:06
3.2.1.2.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.14
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43017
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50696
RSS 6724
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.3. Test results for packet size: 128

A detailed summary of the main results is outlined below.

3.2.1.3.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 16891353.401
tx_rate_mbps Unknown
throughput_rx_mbps 17296.746
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 13660.000
max_latency_ns 125400.000
avg_latency_ns 99489.000
type rfc2544_throughput
packet_size 128
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:13:18
start_time 2018-11-02 17:27:47
stop_time 2018-11-02 17:41:06
3.2.1.3.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.14
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43017
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50696
RSS 6724
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.4. Test results for packet size: 512

A detailed summary of the main results is outlined below.

3.2.1.4.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 4699204.712
tx_rate_mbps Unknown
throughput_rx_mbps 19247.942
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 13140.000
max_latency_ns 43220.000
avg_latency_ns 31519.000
type rfc2544_throughput
packet_size 512
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:13:18
start_time 2018-11-02 17:27:47
stop_time 2018-11-02 17:41:06
3.2.1.4.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.14
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43017
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50696
RSS 6724
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.5. Test results for packet size: 1024

A detailed summary of the main results is outlined below.

3.2.1.5.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 2394616.983
tx_rate_mbps Unknown
throughput_rx_mbps 19616.702
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12580.000
max_latency_ns 36960.000
avg_latency_ns 26658.000
type rfc2544_throughput
packet_size 1024
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:13:18
start_time 2018-11-02 17:27:47
stop_time 2018-11-02 17:41:06
3.2.1.5.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.14
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43017
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50696
RSS 6724
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.6. Test results for packet size: 1518

A detailed summary of the main results is outlined below.

3.2.1.6.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 1625475.359
tx_rate_mbps Unknown
throughput_rx_mbps 19739.773
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12320.000
max_latency_ns 35780.000
avg_latency_ns 25296.000
type rfc2544_throughput
packet_size 1518
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:13:18
start_time 2018-11-02 17:27:47
stop_time 2018-11-02 17:41:06
3.2.1.6.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.14
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43017
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50696
RSS 6724
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.7. Anomalies

No anomalies were detected during the course of this test.

3.2.1.8. Testing Activities/Events

pidstat is used to collect the process statistics, as such some values such as %CPU and %USER maybe > 100% as the values are summed across multiple cores. For more info on pidstat please see: http://linux.die.net/man/1/pidstat. Please note that vsperf recalculates the CPU consumption of a process by aggregating the CPU usage of each thread.

Known issues: Some reported metrics have the value “unkown”. These values are marked unknown as they are not values retrieved from the external tester (traffic generator). They were incorrectly derived in a way that made assumptions about packet sizes, as such they have been deprecated from vsperf and marked as unknown. They will be resolved in the next release.

3.3. Rationale for decisions

The tests conducted do not have pass/fail/conditional-pass criteria. The test is simply conducted and the results are reported.

3.4. Conclusions and recommendations

The test results are stable. The vsperf CI jobs that were used to obtain the results can be found at https://artifacts.opnfv.org/logs/vswitchperf/intel-pod12/2018-11-03_00-27-40/vswitchperf_logs_2018-11-03_00-27-40.tar.gz.

3.5. General

3.5.1. Glossary

  • NFV - Network Function Virtualization
  • Mbps - 1,000,000bps

3.5.2. Document change procedures and history

Document ID Author Date Modified
OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT Christopher Nolan 23/01/2015
OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT Christopher Nolan 28/01/2015

4. Performance report for VppDpdkVhost

4.1. Introduction

The objective of the OPNFV project titled “Characterise vSwitch Performance for Telco NFV Use Cases”, is to evaluate a virtual switch to identify its suitability for a Telco Network Function Virtualization (NFV) environment. As well as this, the project aims to identify any gaps or bottlenecks in order to drive architectural changes to improve virtual switch performance and determinism. The purpose of this document is to summarize the results of the tests carried out on the virtual switch in the Network Function Virtualization Infrastructure (NFVI) and, from these results, provide evaluations and recommendations for the virtual switch. Test results will be outlined in details-of-LTR, preceded by the document-identifier and the scope and references).

This document is currently in draft form.

4.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTR. The format for the document id will be: OPNFV_vswitchperf_LTR_rel_STATUS, the status is one of: DRAFT, REVIEWED, CORRECTED or FINAL. The document id for this version of the LTR is: OPNFV_vswitchperf_LTR_Brahmaputra_DRAFT.

4.1.2. Scope

The scope of this report is to detail the results of the tests that have been performed on the virtual switch. This report will also evaluate the results of these tests and, based on these evaluations, provide recommendations on the suitability of the virtual switch for use in a Telco NFV environment.

4.2. Details of the Level Test Report

This section provides a test-results-overview. Also included are the rationale and the conclusions.

4.2.1. Test ID: PHY2PHY_TPUT_VPP

4.2.1.1. Test Environment

Below is the environment that the test was performed in:

  • OS: CentOS Linux 7 Core

  • Kernel Version: 4.4.116-1.el7.elrepo.x86_64

  • NIC(s):
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

  • Board: Intel Corporation S2600WT2R [2 sockets]

  • CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

  • CPU cores: 44

  • Memory: 65917716 kB

  • Virtual Switch Set-up: p2p

  • vswitchperf: GIT tag: 96de8654952a606d28d56c057ba871b5553e4176

  • Traffic Generator: IxNet, Version: 8.01.1029.6, GIT tag: None

  • vSwitch: VppDpdkVhost, Version: v17.07-release, GIT tag: NA

  • DPDK Version: 17.08.0, GIT tag: 02657b4adcb8af773e26ec061b01cd7abdd3f0b6

Below are test details:

  • Test ID: phy2phy_tput_vpp
  • Description: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • Deployment: p2p
  • Traffic type: rfc2544_throughput
  • Bidirectional : True
4.2.1.2. Test results for packet size: 64

A detailed summary of the main results is outlined below.

4.2.1.2.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 20554298.244
tx_rate_mbps Unknown
throughput_rx_mbps 10523.801
tx_rate_percent 69.063
throughput_rx_percent 69.063
frame_loss_percent 0.000
min_latency_ns 9700.000
max_latency_ns 203480.000
avg_latency_ns 40751.000
type rfc2544_throughput
packet_size 64
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:27:49
start_time 2018-11-02 17:41:15
stop_time 2018-11-02 18:09:04
4.2.1.2.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.38
%system 0.56
%guest 0.00
%CPU 600.94
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50600
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.3. Test results for packet size: 128

A detailed summary of the main results is outlined below.

4.2.1.3.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 15454823.111
tx_rate_mbps Unknown
throughput_rx_mbps 15825.739
tx_rate_percent 91.492
throughput_rx_percent 91.492
frame_loss_percent 0.000
min_latency_ns 11400.000
max_latency_ns 70940.000
avg_latency_ns 25527.500
type rfc2544_throughput
packet_size 128
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:27:49
start_time 2018-11-02 17:41:15
stop_time 2018-11-02 18:09:04
4.2.1.3.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.38
%system 0.56
%guest 0.00
%CPU 600.94
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50600
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.4. Test results for packet size: 512

A detailed summary of the main results is outlined below.

4.2.1.4.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 3681565.226
tx_rate_mbps Unknown
throughput_rx_mbps 15079.691
tx_rate_percent 78.344
throughput_rx_percent 78.344
frame_loss_percent 0.000
min_latency_ns 7580.000
max_latency_ns 162540.000
avg_latency_ns 38297.000
type rfc2544_throughput
packet_size 512
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:27:49
start_time 2018-11-02 17:41:15
stop_time 2018-11-02 18:09:04
4.2.1.4.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.38
%system 0.56
%guest 0.00
%CPU 600.94
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50600
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.5. Test results for packet size: 1024

A detailed summary of the main results is outlined below.

4.2.1.5.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 1727879.519
tx_rate_mbps Unknown
throughput_rx_mbps 14154.789
tx_rate_percent 72.156
throughput_rx_percent 72.156
frame_loss_percent 0.000
min_latency_ns 6120.000
max_latency_ns 88160.000
avg_latency_ns 25244.500
type rfc2544_throughput
packet_size 1024
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:27:49
start_time 2018-11-02 17:41:15
stop_time 2018-11-02 18:09:04
4.2.1.5.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.38
%system 0.56
%guest 0.00
%CPU 600.94
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50600
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.6. Test results for packet size: 1518

A detailed summary of the main results is outlined below.

4.2.1.6.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 1135174.397
tx_rate_mbps Unknown
throughput_rx_mbps 13785.558
tx_rate_percent 69.836
throughput_rx_percent 69.836
frame_loss_percent 0.000
min_latency_ns 6520.000
max_latency_ns 205660.000
avg_latency_ns 22990.500
type rfc2544_throughput
packet_size 1518
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:27:49
start_time 2018-11-02 17:41:15
stop_time 2018-11-02 18:09:04
4.2.1.6.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.38
%system 0.56
%guest 0.00
%CPU 600.94
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50600
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.7. Anomalies

No anomalies were detected during the course of this test.

4.2.1.8. Testing Activities/Events

pidstat is used to collect the process statistics, as such some values such as %CPU and %USER maybe > 100% as the values are summed across multiple cores. For more info on pidstat please see: http://linux.die.net/man/1/pidstat. Please note that vsperf recalculates the CPU consumption of a process by aggregating the CPU usage of each thread.

Known issues: Some reported metrics have the value “unkown”. These values are marked unknown as they are not values retrieved from the external tester (traffic generator). They were incorrectly derived in a way that made assumptions about packet sizes, as such they have been deprecated from vsperf and marked as unknown. They will be resolved in the next release.

4.3. Rationale for decisions

The tests conducted do not have pass/fail/conditional-pass criteria. The test is simply conducted and the results are reported.

4.4. Conclusions and recommendations

The test results are stable. The vsperf CI jobs that were used to obtain the results can be found at https://artifacts.opnfv.org/logs/vswitchperf/intel-pod12/2018-11-03_00-27-40/vswitchperf_logs_2018-11-03_00-27-40.tar.gz.

4.5. General

4.5.1. Glossary

  • NFV - Network Function Virtualization
  • Mbps - 1,000,000bps

4.5.2. Document change procedures and history

Document ID Author Date Modified
OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT Christopher Nolan 23/01/2015
OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT Christopher Nolan 28/01/2015