VSPERF Results

1. VSPERF Test Scenarios

Predefined Tests suitable for automated execution with CI:

Test Definition
phy2phy_tput PacketLossRatio for Phy2Phy
back2back BackToBackFrames for Phy2Phy
phy2phy_tput_mod_vlan PacketLossRatioFrameModification for Phy2Phy
phy2phy_cont Phy2Phy blast vswitch at x% TX rate and measure throughput
pvp_cont PVP blast vswitch at x% TX rate and measure throughput
pvvp_cont PVVP blast vswitch at x% TX rate and measure throughput
phy2phy_scalability Scalability0PacketLoss for Phy2Phy
pvp_tput PacketLossRatio for PVP
pvp_back2back BackToBackFrames for PVP
pvvp_tput PacketLossRatio for PVVP
pvvp_back2back BackToBackFrames for PVVP
phy2phy_cpu_load CPU0PacketLoss for Phy2Phy
phy2phy_mem_load Same as CPU0PacketLoss but using a memory intensive app

Deployment topologies:

  • Phy2Phy: Physical port -> vSwitch -> Physical port.
  • PVP: Physical port -> vSwitch -> VNF -> vSwitch -> Physical port.
  • PVVP: Physical port -> vSwitch -> VNF -> vSwitch -> VNF -> vSwitch -> Physical port.

Loopback applications in the Guest:

Supported traffic generators:

  • Spirent Testcenter
  • Ixia: IxOS and IxNet.
  • Xena
  • MoonGen
  • Dummy
  • T-Rex

2. OPNFV Test Results

VSPERF CI jobs are run daily and sample results can be found at https://wiki.opnfv.org/display/vsperf/Vsperf+Results

Testcase names shown in the dashboard are combination of orignal testcase name from VSPERF framework and indication of used vswitch.

Example:

Testcase phy2phy_tput is executed for three vSwitch types: OvsDpdkVhost, OvsVanilla and VppDpdkVhost. In this case, following testcase names will be used in the dashboard: phy2phy_tput_ovsdpdkvhost, phy2phy_tput_ovsvanilla and phy2phy_tput_vppdpdkvhost.

In case of RFC2544 Throughput test, the recorded metric is FPS (frames per second) without packet loss. For RFC2544 Back2Back test, the recorded metric is back-to-back value (number of frames) without packet loss.

The loopback application in the VNF used for PVP and PVVP scenarios was DPDK testpmd.

Guest interface types are vhost-user for OvsDpdkVhost and VppDpdkVhost and virtio-net for OvsVanilla.

3. Performance report for OvsDpdkVhost

3.1. Introduction

The objective of the OPNFV project titled “Characterise vSwitch Performance for Telco NFV Use Cases”, is to evaluate a virtual switch to identify its suitability for a Telco Network Function Virtualization (NFV) environment. As well as this, the project aims to identify any gaps or bottlenecks in order to drive architectural changes to improve virtual switch performance and determinism. The purpose of this document is to summarize the results of the tests carried out on the virtual switch in the Network Function Virtualization Infrastructure (NFVI) and, from these results, provide evaluations and recommendations for the virtual switch. Test results will be outlined in details-of-LTR, preceded by the document-identifier and the scope and references).

This document is currently in draft form.

3.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTR. The format for the document id will be: OPNFV_vswitchperf_LTR_rel_STATUS, the status is one of: DRAFT, REVIEWED, CORRECTED or FINAL. The document id for this version of the LTR is: OPNFV_vswitchperf_LTR_Brahmaputra_DRAFT.

3.1.2. Scope

The scope of this report is to detail the results of the tests that have been performed on the virtual switch. This report will also evaluate the results of these tests and, based on these evaluations, provide recommendations on the suitability of the virtual switch for use in a Telco NFV environment.

3.2. Details of the Level Test Report

This section provides a test-results-overview. Also included are the rationale and the conclusions.

3.2.1. Test ID: PHY2PHY_TPUT

3.2.1.1. Test Environment

Below is the environment that the test was performed in:

  • OS: CentOS Linux 7 Core

  • Kernel Version: 4.4.116-1.el7.elrepo.x86_64

  • NIC(s):
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

  • Board: Intel Corporation S2600WT2R [2 sockets]

  • CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

  • CPU cores: 44

  • Memory: 65917716 kB

  • Virtual Switch Set-up: p2p

  • vswitchperf: GIT tag: ac6f1684311e6311aa81c0135bf18ee57fbbff6d

  • Traffic Generator: IxNet, Version: 8.01.1029.6, GIT tag: None

  • vSwitch: OvsDpdkVhost, Version: 2.8.1, GIT tag: 5c67e9cfc1eaab19187720c857841be6c8046c40

  • DPDK Version: 17.08.0, GIT tag: 02657b4adcb8af773e26ec061b01cd7abdd3f0b6

Below are test details:

  • Test ID: phy2phy_tput
  • Description: LTD.Throughput.RFC2544.PacketLossRatio
  • Deployment: p2p
  • Traffic type: rfc2544_throughput
  • Bidirectional : True
3.2.1.2. Test results for packet size: 64

A detailed summary of the main results is outlined below.

3.2.1.2.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 23316599.926
tx_rate_mbps Unknown
throughput_rx_mbps 11938.099
tx_rate_percent 78.344
throughput_rx_percent 78.344
frame_loss_percent 0.000
min_latency_ns 4540.000
max_latency_ns 27300.000
avg_latency_ns 11847.000
type rfc2544_throughput
packet_size 64
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:43
start_time 2018-11-10 19:01:25
stop_time 2018-11-10 19:14:08
3.2.1.2.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.11
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43022
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6668
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.3. Test results for packet size: 128

A detailed summary of the main results is outlined below.

3.2.1.3.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 16891677.203
tx_rate_mbps Unknown
throughput_rx_mbps 17297.077
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12620.000
max_latency_ns 57380.000
avg_latency_ns 40584.000
type rfc2544_throughput
packet_size 128
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:43
start_time 2018-11-10 19:01:25
stop_time 2018-11-10 19:14:08
3.2.1.3.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.11
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43022
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6668
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.4. Test results for packet size: 512

A detailed summary of the main results is outlined below.

3.2.1.4.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 4699201.955
tx_rate_mbps Unknown
throughput_rx_mbps 19247.931
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 14460.000
max_latency_ns 46080.000
avg_latency_ns 34079.000
type rfc2544_throughput
packet_size 512
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:43
start_time 2018-11-10 19:01:25
stop_time 2018-11-10 19:14:08
3.2.1.4.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.11
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43022
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6668
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.5. Test results for packet size: 1024

A detailed summary of the main results is outlined below.

3.2.1.5.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 2394615.865
tx_rate_mbps Unknown
throughput_rx_mbps 19616.693
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12460.000
max_latency_ns 38740.000
avg_latency_ns 27653.000
type rfc2544_throughput
packet_size 1024
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:43
start_time 2018-11-10 19:01:25
stop_time 2018-11-10 19:14:08
3.2.1.5.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.11
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43022
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6668
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.6. Test results for packet size: 1518

A detailed summary of the main results is outlined below.

3.2.1.6.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 1625474.942
tx_rate_mbps Unknown
throughput_rx_mbps 19739.768
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 12660.000
max_latency_ns 36180.000
avg_latency_ns 25949.500
type rfc2544_throughput
packet_size 1518
vswitch OvsDpdkVhost
traffic_type udp
test_execution_time 00:12:43
start_time 2018-11-10 19:01:25
stop_time 2018-11-10 19:14:08
3.2.1.6.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: ovs-vswitchd
Statistic Value
%usr 200.02
%system 0.02
%guest 0.00
%CPU 200.11
CPU
minflt/s 0.05
majflt/s 0.00
VSZ 5056816
RSS 43022
%MEM 0.07
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
Process: ovsdb-server
Statistic Value
%usr 0.00
%system 0.00
%guest 0.00
%CPU 0.01
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 50704
RSS 6668
%MEM 0.01
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
3.2.1.7. Anomalies

No anomalies were detected during the course of this test.

3.2.1.8. Testing Activities/Events

pidstat is used to collect the process statistics, as such some values such as %CPU and %USER maybe > 100% as the values are summed across multiple cores. For more info on pidstat please see: http://linux.die.net/man/1/pidstat. Please note that vsperf recalculates the CPU consumption of a process by aggregating the CPU usage of each thread.

Known issues: Some reported metrics have the value “unkown”. These values are marked unknown as they are not values retrieved from the external tester (traffic generator). They were incorrectly derived in a way that made assumptions about packet sizes, as such they have been deprecated from vsperf and marked as unknown. They will be resolved in the next release.

3.3. Rationale for decisions

The tests conducted do not have pass/fail/conditional-pass criteria. The test is simply conducted and the results are reported.

3.4. Conclusions and recommendations

The test results are stable. The vsperf CI jobs that were used to obtain the results can be found at https://artifacts.opnfv.org/logs/vswitchperf/intel-pod12/2018-11-11_03-01-17/vswitchperf_logs_2018-11-11_03-01-17.tar.gz.

3.5. General

3.5.1. Glossary

  • NFV - Network Function Virtualization
  • Mbps - 1,000,000bps

3.5.2. Document change procedures and history

Document ID Author Date Modified
OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT Christopher Nolan 23/01/2015
OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT Christopher Nolan 28/01/2015

4. Performance report for VppDpdkVhost

4.1. Introduction

The objective of the OPNFV project titled “Characterise vSwitch Performance for Telco NFV Use Cases”, is to evaluate a virtual switch to identify its suitability for a Telco Network Function Virtualization (NFV) environment. As well as this, the project aims to identify any gaps or bottlenecks in order to drive architectural changes to improve virtual switch performance and determinism. The purpose of this document is to summarize the results of the tests carried out on the virtual switch in the Network Function Virtualization Infrastructure (NFVI) and, from these results, provide evaluations and recommendations for the virtual switch. Test results will be outlined in details-of-LTR, preceded by the document-identifier and the scope and references).

This document is currently in draft form.

4.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTR. The format for the document id will be: OPNFV_vswitchperf_LTR_rel_STATUS, the status is one of: DRAFT, REVIEWED, CORRECTED or FINAL. The document id for this version of the LTR is: OPNFV_vswitchperf_LTR_Brahmaputra_DRAFT.

4.1.2. Scope

The scope of this report is to detail the results of the tests that have been performed on the virtual switch. This report will also evaluate the results of these tests and, based on these evaluations, provide recommendations on the suitability of the virtual switch for use in a Telco NFV environment.

4.2. Details of the Level Test Report

This section provides a test-results-overview. Also included are the rationale and the conclusions.

4.2.1. Test ID: PHY2PHY_TPUT_VPP

4.2.1.1. Test Environment

Below is the environment that the test was performed in:

  • OS: CentOS Linux 7 Core

  • Kernel Version: 4.4.116-1.el7.elrepo.x86_64

  • NIC(s):
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    • Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

  • Board: Intel Corporation S2600WT2R [2 sockets]

  • CPU: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

  • CPU cores: 44

  • Memory: 65917716 kB

  • Virtual Switch Set-up: p2p

  • vswitchperf: GIT tag: ac6f1684311e6311aa81c0135bf18ee57fbbff6d

  • Traffic Generator: IxNet, Version: 8.01.1029.6, GIT tag: None

  • vSwitch: VppDpdkVhost, Version: v17.07-release, GIT tag: NA

  • DPDK Version: 17.08.0, GIT tag: 02657b4adcb8af773e26ec061b01cd7abdd3f0b6

Below are test details:

  • Test ID: phy2phy_tput_vpp
  • Description: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • Deployment: p2p
  • Traffic type: rfc2544_throughput
  • Bidirectional : True
4.2.1.2. Test results for packet size: 64

A detailed summary of the main results is outlined below.

4.2.1.2.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 22856314.368
tx_rate_mbps Unknown
throughput_rx_mbps 11702.433
tx_rate_percent 76.797
throughput_rx_percent 76.797
frame_loss_percent 0.000
min_latency_ns 5220.000
max_latency_ns 49200.000
avg_latency_ns 24900.000
type rfc2544_throughput
packet_size 64
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-10 19:14:17
stop_time 2018-11-10 19:30:40
4.2.1.2.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.30
%guest 0.00
%CPU 600.44
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50384
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.3. Test results for packet size: 128

A detailed summary of the main results is outlined below.

4.2.1.3.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 16630607.612
tx_rate_mbps Unknown
throughput_rx_mbps 17029.742
tx_rate_percent 98.453
throughput_rx_percent 98.453
frame_loss_percent 0.000
min_latency_ns 5380.000
max_latency_ns 45440.000
avg_latency_ns 13400.500
type rfc2544_throughput
packet_size 128
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-10 19:14:17
stop_time 2018-11-10 19:30:40
4.2.1.3.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.30
%guest 0.00
%CPU 600.44
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50384
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.4. Test results for packet size: 512

A detailed summary of the main results is outlined below.

4.2.1.4.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 4699206.074
tx_rate_mbps Unknown
throughput_rx_mbps 19247.948
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 5720.000
max_latency_ns 32900.000
avg_latency_ns 21392.500
type rfc2544_throughput
packet_size 512
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-10 19:14:17
stop_time 2018-11-10 19:30:40
4.2.1.4.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.30
%guest 0.00
%CPU 600.44
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50384
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.5. Test results for packet size: 1024

A detailed summary of the main results is outlined below.

4.2.1.5.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 2394615.131
tx_rate_mbps Unknown
throughput_rx_mbps 19616.687
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 6420.000
max_latency_ns 33280.000
avg_latency_ns 21695.500
type rfc2544_throughput
packet_size 1024
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-10 19:14:17
stop_time 2018-11-10 19:30:40
4.2.1.5.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.30
%guest 0.00
%CPU 600.44
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50384
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.6. Test results for packet size: 1518

A detailed summary of the main results is outlined below.

4.2.1.6.1. Results/Metrics Collected

The following are the metrics obtained during this test:

Metric Result
tx_rate_fps Unknown
throughput_rx_fps 1625473.376
tx_rate_mbps Unknown
throughput_rx_mbps 19739.749
tx_rate_percent 100
throughput_rx_percent 100
frame_loss_percent 0.000
min_latency_ns 6820.000
max_latency_ns 33860.000
avg_latency_ns 22174.500
type rfc2544_throughput
packet_size 1518
vswitch VppDpdkVhost
traffic_type udp
test_execution_time 00:16:22
start_time 2018-11-10 19:14:17
stop_time 2018-11-10 19:30:40
4.2.1.6.2. Statistics collected

The following system statistics were collected during testcase execution:

Process: vpp_main
Statistic Value
%usr 600.14
%system 0.30
%guest 0.00
%CPU 600.44
CPU
minflt/s 0.00
majflt/s 0.00
VSZ 3937580
RSS 50384
%MEM 0.08
kB_rd/s 0.00
kB_wr/s 0.00
kB_ccwr/s 0.00
4.2.1.7. Anomalies

No anomalies were detected during the course of this test.

4.2.1.8. Testing Activities/Events

pidstat is used to collect the process statistics, as such some values such as %CPU and %USER maybe > 100% as the values are summed across multiple cores. For more info on pidstat please see: http://linux.die.net/man/1/pidstat. Please note that vsperf recalculates the CPU consumption of a process by aggregating the CPU usage of each thread.

Known issues: Some reported metrics have the value “unkown”. These values are marked unknown as they are not values retrieved from the external tester (traffic generator). They were incorrectly derived in a way that made assumptions about packet sizes, as such they have been deprecated from vsperf and marked as unknown. They will be resolved in the next release.

4.3. Rationale for decisions

The tests conducted do not have pass/fail/conditional-pass criteria. The test is simply conducted and the results are reported.

4.4. Conclusions and recommendations

The test results are stable. The vsperf CI jobs that were used to obtain the results can be found at https://artifacts.opnfv.org/logs/vswitchperf/intel-pod12/2018-11-11_03-01-17/vswitchperf_logs_2018-11-11_03-01-17.tar.gz.

4.5. General

4.5.1. Glossary

  • NFV - Network Function Virtualization
  • Mbps - 1,000,000bps

4.5.2. Document change procedures and history

Document ID Author Date Modified
OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT Christopher Nolan 23/01/2015
OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT Christopher Nolan 28/01/2015