Yardstick Release Note

License

OPNFV Euphrates release note for Yardstick Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>.

The Yardstick framework, the Yardstick test cases are open-source software,
licensed under the terms of the Apache License, Version 2.0.

OPNFV Euphrates Release Note for Yardstick

Abstract

This document describes the release note of Yardstick project.

Version History

Date Version Comment
October 20, 2017 5.0.0 Yardstick for Euphrates release

Important Notes

The software delivered in the OPNFV Yardstick Project, comprising the Yardstick framework, the Yardstick test cases and the experimental framework Apex Lake is a realization of the methodology in ETSI-ISG NFV-TST001.

The Yardstick framework is installer, infrastructure and application independent.

OPNFV Euphrates Release

This Euphrates release provides Yardstick as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including:

  • Documentation generated with Sphinx
    • User Guide
    • Developer Guide
    • Release notes (this document)
    • Results
  • Automated Yardstick test suite (daily, weekly)
    • Jenkins Jobs for OPNFV community labs
  • Automated Yardstick test results visualization
    • Dashboard using Grafana (user:opnfv/password: opnfv), influxDB is used as backend
  • Yardstick framework source code
  • Yardstick test cases yaml files
  • Yardstick plug-in configuration yaml files, plug-in install/remove scripts

For Euphrates release, the Yardstick framework is used for the following testing:

  • OPNFV platform testing - generic test cases to measure the categories:
    • Compute
    • Network
    • Storage
  • OPNFV platform network service benchmarking(NSB)
    • NSB
  • Test cases for the following OPNFV Projects:
    • Container4NFV
    • High Availability
    • IPv6
    • KVM
    • Parser
    • StorPerf
    • VSperf
    • virtual Traffic Classifier

The Yardstick framework is developed in the OPNFV community, by the Yardstick team.

Note

The test case description template used for the Yardstick test cases is based on the document ETSI-ISG NFV-TST001; the results report template used for the Yardstick results is based on the IEEE Std 829-2008.

Release Data

Project Yardstick
Repo/tag yardstick/opnfv-5.0.0
Yardstick Docker image tag opnfv-5.0.0
Release designation Euphrates
Release date October 20, 2017
Purpose of the delivery OPNFV Euphrates 5.0.0

Deliverables

Software Deliverables

New Contexts
Context Description
Heat Models orchestration using OpenStack Heat
Node Models Baremetal, Controller, Compute
Standalone Models VM running on Non-Managed NFVi
Kubernetes Models VM running on Non-Managed NFVi
New Runners
Runner Description
Arithmetic Steps every run arithmetically according to specified input value
Duration Runs for a specified period of time
Iteration Runs for a specified number of iterations
Sequence Selects input value to a scenario from an input file and runs all entries sequentially
New Scenarios
Category Delivered
Availability

Attacker:

  • baremetal, process

HA tools:

  • check host, openstack, process, service
  • kill process
  • start/stop service

Monitor:

  • command, process
Compute
  • cpuload
  • cyclictest
  • lmbench
  • lmbench_cache
  • perf
  • unixbench
  • ramspeed
  • cachestat
  • memeoryload
  • computecapacity
  • SpecCPU2006
Networking
  • iperf3
  • netperf
  • netperf_node
  • ping
  • ping6
  • pktgen
  • sfc
  • sfc with tacker
  • vtc instantion validation
  • vtc instantion validation with noisy neighbors
  • vtc throughput
  • vtc throughput in the presence of noisy neighbors
  • networkcapacity
  • netutilization
  • nstat
  • pktgenDPDK
Parser Tosca2Heat
Storage

fio

bonnie++

storagecapacity
StorPerf storperf
NSB vPE thoughput test case

New Test cases

  • Generic NFVI test cases
  • OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006
  • OPNFV_YARDSTICK_TCO79 - Bonnie++
  • Kubernetes Test cases
  • OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER
  • OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM

Version Change

Module Version Changes

This is the fifth tracked release of Yardstick. It is based on following upstream versions:

  • OpenStack Ocata
  • OpenDayLight Nitrogen
  • ONOS Junco

Document Version Changes

This is the fifth tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates:

  • Yardstick User Guide: add “network service benchmarking(NSB)” chapter; add “Yardstick - NSB Testing -Installation” chapter; add “Yardstick API” chapter; add “Yardstick user interface” chapter; Update Yardstick installation chapter;
  • Yardstick Developer Guide
  • Yardstick Release Notes for Yardstick: this document

Feature additions

  • Yardstick RESTful API support
  • Network service benchmarking
  • Stress testing with Bottlenecks team
  • Yardstick framework improvement:
    • yardstick report CLI
    • Node context support OpenStack configuration via Ansible
    • Https support
    • Kubernetes context type
  • Yardstick container local GUI
  • Python 3 support

Scenario Matrix

For Euphrates 5.0.0, Yardstick was tested on the following scenarios:

Scenario Apex Compass Fuel Joid
os-nosdn-nofeature-noha     X X
os-nosdn-nofeature-ha X X X X
os-odl_l2-nofeature-ha   X X X
os-odl_l2-nofeature-noha     X  
os-odl_l3-nofeature-ha X X X  
os-odl_l3-nofeature-noha     X  
os-onos-sfc-ha        
os-onos-nofeature-ha   X   X
os-onos-nofeature-noha        
os-odl_l2-sfc-ha     X  
os-odl_l2-sfc-noha     X  
os-odl_l2-bgpvpn-ha X   X  
os-odl_l2-bgpvpn-noha     X  
os-nosdn-kvm-ha X   X  
os-nosdn-kvm-noha     X  
os-nosdn-ovs-ha     X  
os-nosdn-ovs-noha     X  
os-ocl-nofeature-ha   X    
os-nosdn-lxd-ha       X
os-nosdn-lxd-noha       X
os-nosdn-fdio-ha X      
os-odl_l2-fdio-noha X      
os-odl-gluon-noha X      
os-nosdn-openo-ha   X    
os-nosdn-kvm_ovs_dpdk -noha     X  
os-nosdn-kvm_ovs_dpdk-ha     X  
os-nosdn-kvm_ovs_dpdk _bar-ha     X  
os-nosdn-kvm_ovs_dpdk _bar-noha     X  
opnfv_os-ovn-nofeature- noha_daily X      

Test results

Test results are available in:

The reporting pages can be found at:

apex http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html
compass http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html
fuel@x86 http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html
fuel@aarch64 http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html
joid http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html

Known Issues/Faults

Corrected Faults

Euphrates 5.0.0:

JIRA REFERENCE DESCRIPTION
JIRA: YARDSTICK-599 Could not load EntryPoint.parse when using ‘openstack -h’
JIRA: YARDSTICK-602 Don’t rely on staic ip addresses as they are dynamic

Euphratess 5.0.0 known restrictions/issues

Installer Scenario Issue
any *-bgpvpn Floating ips not supported. Some Test cases related to floating ips are excluded.
any odl_l3-* Some test cases related to using floating IP addresses fail because of a known ODL bug.
compass odl_l2-* In some test cases, VM instance will failed raising network interfaces.

Open JIRA tickets

JIRA REFERENCE DESCRIPTION
JIRA: YARDSTICK-412 IPv6 test case should add support for newton