VSPERF Results

1. VSPERF Test Scenarios

Predefined Tests suitable for automated execution with CI:

Test Definition
phy2phy_tput PacketLossRatio for Phy2Phy
back2back BackToBackFrames for Phy2Phy
phy2phy_tput_mod_vlan PacketLossRatioFrameModification for Phy2Phy
phy2phy_cont Phy2Phy blast vswitch at x% TX rate and measure throughput
pvp_cont PVP blast vswitch at x% TX rate and measure throughput
pvvp_cont PVVP blast vswitch at x% TX rate and measure throughput
phy2phy_scalability Scalability0PacketLoss for Phy2Phy
pvp_tput PacketLossRatio for PVP
pvp_back2back BackToBackFrames for PVP
pvvp_tput PacketLossRatio for PVVP
pvvp_back2back BackToBackFrames for PVVP
phy2phy_cpu_load CPU0PacketLoss for Phy2Phy
phy2phy_mem_load Same as CPU0PacketLoss but using a memory intensive app

Deployment topologies:

  • Phy2Phy: Physical port -> vSwitch -> Physical port.
  • PVP: Physical port -> vSwitch -> VNF -> vSwitch -> Physical port.
  • PVVP: Physical port -> vSwitch -> VNF -> vSwitch -> VNF -> vSwitch -> Physical port.

Loopback applications in the Guest:

Supported traffic generators:

  • Spirent Testcenter
  • Ixia: IxOS and IxNet.
  • Xena
  • MoonGen
  • Dummy
  • T-Rex

2. OPNFV Test Results

VSPERF CI jobs are run daily and sample results can be found at https://wiki.opnfv.org/display/vsperf/Vsperf+Results

Testcase names shown in the dashboard are combination of orignal testcase name from VSPERF framework and indication of used vswitch.

Example:

Testcase phy2phy_tput is executed for three vSwitch types: OvsDpdkVhost, OvsVanilla and VppDpdkVhost. In this case, following testcase names will be used in the dashboard: phy2phy_tput_ovsdpdkvhost, phy2phy_tput_ovsvanilla and phy2phy_tput_vppdpdkvhost.

In case of RFC2544 Throughput test, the recorded metric is FPS (frames per second) without packet loss. For RFC2544 Back2Back test, the recorded metric is back-to-back value (number of frames) without packet loss.

The loopback application in the VNF used for PVP and PVVP scenarios was DPDK testpmd.

Guest interface types are vhost-user for OvsDpdkVhost and VppDpdkVhost and virtio-net for OvsVanilla.

3. Performance report for OvsDpdkVhost

3.1. Introduction

The objective of the OPNFV project titled “Characterise vSwitch Performance for Telco NFV Use Cases”, is to evaluate a virtual switch to identify its suitability for a Telco Network Function Virtualization (NFV) environment. As well as this, the project aims to identify any gaps or bottlenecks in order to drive architectural changes to improve virtual switch performance and determinism. The purpose of this document is to summarize the results of the tests carried out on the virtual switch in the Network Function Virtualization Infrastructure (NFVI) and, from these results, provide evaluations and recommendations for the virtual switch. Test results will be outlined in details-of-LTR, preceded by the document-identifier and the scope and references).

This document is currently in draft form.

3.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTR. The format for the document id will be: OPNFV_vswitchperf_LTR_rel_STATUS, the status is one of: DRAFT, REVIEWED, CORRECTED or FINAL. The document id for this version of the LTR is: OPNFV_vswitchperf_LTR_Brahmaputra_DRAFT.

3.1.2. Scope

The scope of this report is to detail the results of the tests that have been performed on the virtual switch. This report will also evaluate the results of these tests and, based on these evaluations, provide recommendations on the suitability of the virtual switch for use in a Telco NFV environment.

3.2. Details of the Level Test Report

This section provides a test-results-overview. Also included are the rationale and the conclusions.

3.2.1. Test ID: PHY2PHY_TPUT

3.2.1.1. Test Environment

Below is the environment that the test was performed in:

  • OS: CentOS Linux 7 Core

  • Kernel Version: 4.4.116-1.el7.elrepo.x86_64

  • NIC(s):
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

  • Board: Intel Corporation S2600WT2R [2 sockets]

  • CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

  • CPU cores: 44

  • Memory: 65917716 kB

  • Virtual Switch Set-up: p2p

  • vswitchperf: GIT tag: 96de8654952a606d28d56c057ba871b5553e4176

  • Traffic Generator: IxNet, Version: 8.01.1029.6, GIT tag: None

  • vSwitch: OvsDpdkVhost, Version: 2.8.1, GIT tag: 5c67e9cfc1eaab19187720c857841be6c8046c40

  • DPDK Version: 17.08.0, GIT tag: 02657b4adcb8af773e26ec061b01cd7abdd3f0b6

Below are test details:

  • Test ID: phy2phy_tput
  • Description: LTD.Throughput.RFC2544.PacketLossRatio
  • Deployment: p2p
  • Traffic type: rfc2544_throughput
  • Bidirectional : True
3.2.1.2. Test results for packet size: 64

A detailed summary of the main results is outlined below.

3.2.1.2.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 23316599.825
tx_rate_mbps Unknown
throughput_rx_mbps 11938.099
tx_rate_percent 78.344
throughput_rx_percent 78.344
frame_loss_percent 0.000
min_latency_ns 4620.000
max_latency_ns 22940.000
avg_latency_ns 11831.000
type rfc2544_throughput
packet_size 64
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:36
start_time 2018-11-06 17:01:10
stop_time 2018-11-06 17:13:46
3.2.1.2.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.05
CPU
minflt/s 0.06
majflt/s 0.00
VSZ 5056812
RSS 42722
%MEM 0.06
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6656
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.3. Test results for packet size: 128

A detailed summary of the main results is outlined below.

3.2.1.3.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 16891686.493
tx_rate_mbps Unknown
throughput_rx_mbps 17297.087
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12260.000
max_latency_ns 49940.000
avg_latency_ns 39102.000
type rfc2544_throughput
packet_size 128
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:36
start_time 2018-11-06 17:01:10
stop_time 2018-11-06 17:13:46
3.2.1.3.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.05
CPU
minflt/s 0.06
majflt/s 0.00
VSZ 5056812
RSS 42722
%MEM 0.06
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6656
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.4. Test results for packet size: 512

A detailed summary of the main results is outlined below.

3.2.1.4.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 4699203.521
tx_rate_mbps Unknown
throughput_rx_mbps 19247.938
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12620.000
max_latency_ns 41180.000
avg_latency_ns 30928.000
type rfc2544_throughput
packet_size 512
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:36
start_time 2018-11-06 17:01:10
stop_time 2018-11-06 17:13:46
3.2.1.4.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.05
CPU
minflt/s 0.06
majflt/s 0.00
VSZ 5056812
RSS 42722
%MEM 0.06
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6656
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.5. Test results for packet size: 1024

A detailed summary of the main results is outlined below.

3.2.1.5.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 2394615.913
tx_rate_mbps Unknown
throughput_rx_mbps 19616.694
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12380.000
max_latency_ns 38980.000
avg_latency_ns 27603.000
type rfc2544_throughput
packet_size 1024
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:36
start_time 2018-11-06 17:01:10
stop_time 2018-11-06 17:13:46
3.2.1.5.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.05
CPU
minflt/s 0.06
majflt/s 0.00
VSZ 5056812
RSS 42722
%MEM 0.06
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6656
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.6. Test results for packet size: 1518

A detailed summary of the main results is outlined below.

3.2.1.6.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 1625474.985
tx_rate_mbps Unknown
throughput_rx_mbps 19739.768
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12880.000
max_latency_ns 36340.000
avg_latency_ns 26034.500
type rfc2544_throughput
packet_size 1518
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:36
start_time 2018-11-06 17:01:10
stop_time 2018-11-06 17:13:46
3.2.1.6.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.05
CPU
minflt/s 0.06
majflt/s 0.00
VSZ 5056812
RSS 42722
%MEM 0.06
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6656
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.7. Anomalies

No anomalies were detected during the course of this test.

3.2.1.8. Testing Activities/Events

pidstat is used to collect the process statistics, as such some values such as %CPU and %USER maybe > 100% as the values are summed across multiple cores. For more info on pidstat please see: http://linux.die.net/man/1/pidstat. Please note that vsperf recalculates the CPU consumption of a process by aggregating the CPU usage of each thread.

Known issues: Some reported metrics have the value “unkown”. These values are marked unknown as they are not values retrieved from the external tester (traffic generator). They were incorrectly derived in a way that made assumptions about packet sizes, as such they have been deprecated from vsperf and marked as unknown. They will be resolved in the next release.

3.3. Rationale for decisions

The tests conducted do not have pass/fail/conditional-pass criteria. The test is simply conducted and the results are reported.

3.4. Conclusions and recommendations

The test results are stable. The vsperf CI jobs that were used to obtain the results can be found at https://artifacts.opnfv.org/logs/vswitchperf/intel-pod12/gambia/2018-11-07_01-01-03/vswitchperf_logs_2018-11-07_01-01-03.tar.gz.

3.5. General

3.5.1. Glossary

  • NFV - Network Function Virtualization
  • Mbps - 1,000,000bps

3.5.2. Document change procedures and history

Document ID Author Date Modified
OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT Christopher Nolan 23/01/2015
OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT Christopher Nolan 28/01/2015

4. Performance report for VppDpdkVhost

4.1. Introduction

The objective of the OPNFV project titled “Characterise vSwitch Performance for Telco NFV Use Cases”, is to evaluate a virtual switch to identify its suitability for a Telco Network Function Virtualization (NFV) environment. As well as this, the project aims to identify any gaps or bottlenecks in order to drive architectural changes to improve virtual switch performance and determinism. The purpose of this document is to summarize the results of the tests carried out on the virtual switch in the Network Function Virtualization Infrastructure (NFVI) and, from these results, provide evaluations and recommendations for the virtual switch. Test results will be outlined in details-of-LTR, preceded by the document-identifier and the scope and references).

This document is currently in draft form.

4.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTR. The format for the document id will be: OPNFV_vswitchperf_LTR_rel_STATUS, the status is one of: DRAFT, REVIEWED, CORRECTED or FINAL. The document id for this version of the LTR is: OPNFV_vswitchperf_LTR_Brahmaputra_DRAFT.

4.1.2. Scope

The scope of this report is to detail the results of the tests that have been performed on the virtual switch. This report will also evaluate the results of these tests and, based on these evaluations, provide recommendations on the suitability of the virtual switch for use in a Telco NFV environment.

4.2. Details of the Level Test Report

This section provides a test-results-overview. Also included are the rationale and the conclusions.

4.2.1. Test ID: PHY2PHY_TPUT_VPP

4.2.1.1. Test Environment

Below is the environment that the test was performed in:

  • OS: CentOS Linux 7 Core

  • Kernel Version: 4.4.116-1.el7.elrepo.x86_64

  • NIC(s):
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

  • Board: Intel Corporation S2600WT2R [2 sockets]

  • CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

  • CPU cores: 44

  • Memory: 65917716 kB

  • Virtual Switch Set-up: p2p

  • vswitchperf: GIT tag: 96de8654952a606d28d56c057ba871b5553e4176

  • Traffic Generator: IxNet, Version: 8.01.1029.6, GIT tag: None

  • vSwitch: VppDpdkVhost, Version: v17.07-release, GIT tag: NA

  • DPDK Version: 17.08.0, GIT tag: 02657b4adcb8af773e26ec061b01cd7abdd3f0b6

Below are test details:

  • Test ID: phy2phy_tput_vpp
  • Description: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • Deployment: p2p
  • Traffic type: rfc2544_throughput
  • Bidirectional : True
4.2.1.2. Test results for packet size: 64

A detailed summary of the main results is outlined below.

4.2.1.2.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 22856314.817
tx_rate_mbps Unknown
throughput_rx_mbps 11702.433
tx_rate_percent 76.797
throughput_rx_percent 76.797
frame_loss_percent 0.000
min_latency_ns 5120.000
max_latency_ns 46540.000
avg_latency_ns 23002.000
type rfc2544_throughput
packet_size 64
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-06 17:13:55
stop_time 2018-11-06 17:30:18
4.2.1.2.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.29
%guest 0.00
%CPU 600.43
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50744
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.3. Test results for packet size: 128

A detailed summary of the main results is outlined below.

4.2.1.3.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 16630608.006
tx_rate_mbps Unknown
throughput_rx_mbps 17029.743
tx_rate_percent 98.453
throughput_rx_percent 98.453
frame_loss_percent 0.000
min_latency_ns 5500.000
max_latency_ns 35580.000
avg_latency_ns 12887.000
type rfc2544_throughput
packet_size 128
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-06 17:13:55
stop_time 2018-11-06 17:30:18
4.2.1.3.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.29
%guest 0.00
%CPU 600.43
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50744
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.4. Test results for packet size: 512

A detailed summary of the main results is outlined below.

4.2.1.4.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 4699205.072
tx_rate_mbps Unknown
throughput_rx_mbps 19247.944
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 5600.000
max_latency_ns 34720.000
avg_latency_ns 22336.000
type rfc2544_throughput
packet_size 512
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-06 17:13:55
stop_time 2018-11-06 17:30:18
4.2.1.4.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.29
%guest 0.00
%CPU 600.43
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50744
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.5. Test results for packet size: 1024

A detailed summary of the main results is outlined below.

4.2.1.5.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 2394614.764
tx_rate_mbps Unknown
throughput_rx_mbps 19616.684
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 6340.000
max_latency_ns 35760.000
avg_latency_ns 22168.500
type rfc2544_throughput
packet_size 1024
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-06 17:13:55
stop_time 2018-11-06 17:30:18
4.2.1.5.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.29
%guest 0.00
%CPU 600.43
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50744
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.6. Test results for packet size: 1518

A detailed summary of the main results is outlined below.

4.2.1.6.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 1625473.592
tx_rate_mbps Unknown
throughput_rx_mbps 19739.751
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 6760.000
max_latency_ns 33980.000
avg_latency_ns 22090.500
type rfc2544_throughput
packet_size 1518
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-06 17:13:55
stop_time 2018-11-06 17:30:18
4.2.1.6.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.29
%guest 0.00
%CPU 600.43
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50744
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.7. Anomalies

No anomalies were detected during the course of this test.

4.2.1.8. Testing Activities/Events

pidstat is used to collect the process statistics, as such some values such as %CPU and %USER maybe > 100% as the values are summed across multiple cores. For more info on pidstat please see: http://linux.die.net/man/1/pidstat. Please note that vsperf recalculates the CPU consumption of a process by aggregating the CPU usage of each thread.

Known issues: Some reported metrics have the value “unkown”. These values are marked unknown as they are not values retrieved from the external tester (traffic generator). They were incorrectly derived in a way that made assumptions about packet sizes, as such they have been deprecated from vsperf and marked as unknown. They will be resolved in the next release.

4.3. Rationale for decisions

The tests conducted do not have pass/fail/conditional-pass criteria. The test is simply conducted and the results are reported.

4.4. Conclusions and recommendations

The test results are stable. The vsperf CI jobs that were used to obtain the results can be found at https://artifacts.opnfv.org/logs/vswitchperf/intel-pod12/gambia/2018-11-07_01-01-03/vswitchperf_logs_2018-11-07_01-01-03.tar.gz.

4.5. General

4.5.1. Glossary

  • NFV - Network Function Virtualization
  • Mbps - 1,000,000bps

4.5.2. Document change procedures and history

Document ID Author Date Modified
OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT Christopher Nolan 23/01/2015
OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT Christopher Nolan 28/01/2015