OPNFV Documentation

Open Platform for NFV (OPNFV) facilitates the development and evolution of NFV components across various open source ecosystems. Through system level integration, deployment and testing, OPNFV creates a reference NFV platform to accelerate the transformation of enterprise and service provider networks. Participation is open to anyone, whether you are an employee of a member company or just passionate about network transformation.

Release

Platform overview

Introduction

Network Functions Virtualization (NFV) is transforming the networking industry via software-defined infrastructures and open source is the proven method for quickly developing software for commercial products and services that can move markets. Open Platform for NFV (OPNFV) facilitates the development and evolution of NFV components across various open source ecosystems. Through system level integration, deployment and testing, OPNFV constructs a reference NFV platform to accelerate the transformation of enterprise and service provider networks. As an open source project, OPNFV is uniquely positioned to bring together the work of standards bodies, open source communities, service providers and commercial suppliers to deliver a de facto NFV platform for the industry.

By integrating components from upstream projects, the community is able to conduct performance and use case-based testing on a variety of solutions to ensure the platform’s suitability for NFV use cases. OPNFV also works upstream with other open source communities to bring contributions and learnings from its work directly to those communities in the form of blueprints, patches, bugs, and new code.

OPNFV focuses on building NFV Infrastructure (NFVI) and Virtualised Infrastructure Management (VIM) by integrating components from upstream projects such as OpenDaylight, ONOS, OpenContrail, OVN, OpenStack, Kubernetes, Ceph Storage, KVM, Open vSwitch, and Linux. More recently, OPNFV has extended its portfolio of forwarding solutions to include DPDK, fd.io and ODP, is able to run on both Intel and ARM commercial and white-box hardware, support VM, Container and BareMetal workloads, and includes Management and Network Orchestration MANO components primarily for application composition and management in the Danube release.

These capabilities, along with application programmable interfaces (APIs) to other NFV elements, form the basic infrastructure required for Virtualized Network Functions (VNF) and MANO components.

Concentrating on these components while also considering proposed projects on additional topics (such as the MANO components and applications themselves), OPNFV aims to enhance NFV services by increasing performance and power efficiency improving reliability, availability and serviceability, and delivering comprehensive platform instrumentation.

OPNFV Platform Architecture

The OPNFV project addresses a number of aspects in the development of a consistent virtualisation platform including common hardware requirements, software architecture, MANO and applications.

OPNFV Platform Overview Diagram

Overview infographic of the opnfv platform and projects.

To address these areas effectively, the OPNFV platform architecture can be decomposed into the following basic building blocks:

  • Hardware: Infrastructure working group, Pharos project and associated activities
  • Software Platform: Platform integration and deployment projects
  • MANO: MANO working group and associated projects
  • Tooling and testing: Testing working group and test projects
  • Applications: All other areas and drive requirements for OPNFV

OPNFV Lab Infrastructure

The infrastructure working group oversees such topics as lab management, workflow, definitions, metrics and tools for OPNFV infrastructure.

Fundamental to the WG is the Pharos Specification which provides a set of defined lab infrastructures over a geographically and technically diverse federated global OPNFV lab.

Labs may instantiate bare-metal and virtual environments that are accessed remotely by the community and used for OPNFV platform and feature development, build, deploy and testing. No two labs are the same and the heterogeneity of the Pharos environment provides the ideal platform for establishing hardware and software abstractions providing well understood performance characteristics.

Community labs are hosted by OPNFV member companies on a voluntary basis. The Linux Foundation also hosts an OPNFV lab that provides centralized CI and other production resources which are linked to community labs. Future lab capabilities will include the ability easily automate deploy and test of any OPNFV install scenario in any lab environment as well as on a nested “lab as a service” virtual infrastructure.

OPNFV Software Platform Architecture

The OPNFV software platform is comprised exclusively of open source implementations of platform component pieces. OPNFV is able to draw from the rich ecosystem of NFV related technologies available in open source communities, and then integrate, test, measure and improve these components in conjunction with our upstream communities.

Virtual Infrastructure Management

OPNFV derives it’s virtual infrastructure management from one of our largest upstream ecosystems OpenStack. OpenStack provides a complete reference cloud management system and associated technologies. While the OpenStack community sustains a broad set of projects, not all technologies are relevant in the NFV domain, the OPNFV community consumes a sub-set of OpenStack projects and the usage and composition may vary depending on the installer and scenario.

For details on the scenarios available in OPNFV and the specific composition of components refer to the OPNFV User Guide & Configuration Guide.

OPNFV now also has initial support for containerized VNFs.

Operating Systems

OPNFV currently uses Linux on all target machines, this can include Ubuntu, Centos or SUSE Linux. The specific version of Linux used for any deployment is documented in the installation guide.

Networking Technologies

SDN Controllers

OPNFV, as an NFV focused project, has a significant investment on networking technologies and provides a broad variety of integrated open source reference solutions. The diversity of controllers able to be used in OPNFV is supported by a similarly diverse set of forwarding technologies.

There are many SDN controllers available today relevant to virtual environments where the OPNFV community supports and contributes to a number of these. The controllers being worked on by the community during this release of OPNFV include:

  • Neutron: an OpenStack project to provide “network connectivity as a service” between interface devices (e.g., vNICs) managed by other OpenStack services (e.g. Nova).
  • OpenDaylight: addresses multivendor, traditional and greenfield networks, establishing the industry’s de facto SDN platform and providing the foundation for networks of the future.
  • OpenContrail: An open source SDN controller designed for cloud and NFV use cases. It has an analytics engine, well defined northbound REST APIs to configure and gather ops/analytics data.
  • OVN: A virtual networking solution developed by the same team that created OVS. OVN stands for Open Virtual Networking and is dissimilar from the above projects in that it focuses only on overlay networks.
Data Plane

OPNFV extends Linux virtual networking capabilities by using virtual switching and routing components. The OPNFV community proactively engages with the following open source communities to address performance, scale and resiliency needs apparent in carrier networks.

  • OVS (Open vSwitch): a production quality, multilayer virtual switch designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols.
  • FD.io (Fast data - Input/Output): a high performance alternative to Open vSwitch, the core engine of FD.io is a vector processing engine (VPP). VPP processes a number of packets in parallel instead of one at a time thus significantly improving packet throughput.
  • DPDK: a set of libraries that bypass the kernel and provide polling mechanisms, instead of interrupt based operations, to speed up packet processing. DPDK works with both OVS and FD.io.
MANO

OPNFV integrates open source MANO projects for NFV orchestration and VNF management. New MANO projects are constantly being added, currently OPNFV integrates: * OpenBaton: Open Baton is a ETSI NFV compliant Management and Orchestration (MANO) Framework.

It enables virtual Network Services deployments on top of heterogeneous NFV Infrastructures. OpenBaton is also used to deploy vIMS (clearwater and openIMS).

Deployment Architecture

A typical OPNFV deployment starts with three controller nodes running in a high availability configuration including control plane components from OpenStack, SDN controllers, etc. and a minimum of two compute nodes for deployment of workloads (VNFs). A detailed description of the hardware requirements required to support the 5 node configuration can be found in pharos specification: Pharos Project

In addition to the deployment on a highly available physical infrastructure, OPNFV can be deployed for development and lab purposes in a virtual environment. In this case each of the hosts is provided by a virtual machine and allows control and workload placement using nested virtualization.

The initial deployment is done using a staging server, referred to as the “jumphost”. This server-either physical or virtual-is first installed with the installation program that then installs OpenStack and other components on the controller nodes and compute nodes. See the OPNFV User Guide & Configuration Guide for more details.

The OPNFV Testing Ecosystem

The OPNFV community has set out to address the needs of virtualization in the carrier network and as such platform validation and measurements are a cornerstone to the iterative releases and objectives.

To simplify the complex task of feature, component and platform validation and characterization the testing community has established a fully automated method for addressing all key areas of platform validation. This required the integration of a variety of testing frameworks in our CI systems, real time and automated analysis of results, storage and publication of key facts for each run as shown in the following diagram.

Overview infographic of the OPNFV testing Ecosystem

Release Verification

The OPNFV community relies on its testing community to establish release criteria for each OPNFV release. With each release cycle the testing criteria become more stringent and better representative of our feature and resiliency requirements. Each release establishes a set of deployment scenarios to validate, the testing infrastructure and test suites need to accommodate these features and capabilities.

The release criteria as established by the testing teams include passing a set of test cases derived from the functional testing project ‘functest,’ a set of test cases derived from our platform system and performance test project ‘yardstick,’ and a selection of test cases for feature capabilities derived from other test projects such as bottlenecks, vsperf, cperf and storperf. The scenario needs to be able to be deployed, pass these tests, and be removed from the infrastructure iteratively in order to fulfill the release criteria.

Functest

Functest provides a functional testing framework incorporating a number of test suites and test cases that test and verify OPNFV platform functionality. The scope of Functest and relevant test cases can be found in the Functest User Guide

Functest provides both feature project and component test suite integration, leveraging OpenStack and SDN controllers testing frameworks to verify the key components of the OPNFV platform are running successfully.

Yardstick

Yardstick is a testing project for verifying the infrastructure compliance when running VNF applications. Yardstick benchmarks a number of characteristics and performance vectors on the infrastructure making it a valuable pre-deployment NFVI testing tools.

Yardstick provides a flexible testing framework for launching other OPNFV testing projects.

There are two types of test cases in Yardstick:

  • Yardstick generic test cases and OPNFV feature test cases; including basic characteristics benchmarking in compute/storage/network area.
  • OPNFV feature test cases include basic telecom feature testing from OPNFV projects; for example nfv-kvm, sfc, ipv6, Parser, Availability and SDN VPN

System Evaluation and compliance testing

The OPNFV community is developing a set of test suites intended to evaluate a set of reference behaviors and capabilities for NFV systems developed externally from the OPNFV ecosystem to evaluate and measure their ability to provide the features and capabilities developed in the OPNFV ecosystem.

The Dovetail project will provide a test framework and methodology able to be used on any NFV platform, including an agreed set of test cases establishing an evaluation criteria for exercising an OPNFV compatible system. The Dovetail project has begun establishing the test framework and will provide a preliminary methodology for the Danube release. Work will continue to develop these test cases to establish a stand alone compliance evaluation solution in future releases.

Additional Testing

Besides the test suites and cases for release verification, additional testing is performed to validate specific features or characteristics of the OPNFV platform. These testing framework and test cases may include some specific needs; such as extended measurements, additional testing stimuli, or tests simulating environmental disturbances or failures.

These additional testing activities provide a more complete evaluation of the OPNFV platform. Some of the projects focused on these testing areas include:

Bottlenecks

Bottlenecks provides a framework to find system limitations and bottlenecks, providing root cause isolation capabilities to facilitate system evaluation.

NFVBench

NFVbench is a lightweight end-to-end dataplane benchmarking framework project. It includes traffic generator(s) and measures a number of packet performance related metrics.

QTIP

QTIP boils down NFVI compute and storage performance into one single metric for easy comparison. QTIP crunches these numbers based on five different categories of compute metrics and relies on Storperf for storage metrics.

Storperf

Storperf measures the performance of external block storage. The goal of this project is to provide a report based on SNIA’s (Storage Networking Industry Association) Performance Test Specification.

VSPERF

VSPERF provides an automated test-framework and comprehensive test suite for measuring data-plane performance of the NFVI including switching technology, physical and virtual network interfaces. The provided test cases with network topologies can be customized while also allowing individual versions of Operating System, vSwitch and hypervisor to be specified.

Installation

Abstract

This an overview document for the installation of the Euphrates release of OPNFV.

The Euphrates release can be installed making use of any of the installer projects in OPNFV: Apex, Compass4Nfv, Daisy4NFV, Fuel or JOID. Each installer provides the ability to install a common OPNFV platform as well as integrating additional features delivered through a variety of scenarios by the OPNFV community.

Introduction

The OPNFV platform is comprised of a variety of upstream components that may be deployed on your infrastructure. A composition of components, tools and configurations is identified in OPNFV as a deployment scenario.

The various OPNFV scenarios provide unique features and capabilities that you may want to leverage, and it is important to understand your required target platform capabilities before installing and configuring your scenarios.

An OPNFV installation requires either a physical infrastructure environment as defined in the Pharos specification, or a virtual one. When configuring a physical infrastructure it is strongly advised to follow the Pharos configuration guidelines.

Scenarios

OPNFV scenarios are designed to host virtualised network functions (VNF’s) in a variety of deployment architectures and locations. Each scenario provides specific capabilities and/or components aimed at solving specific problems for the deployment of VNF’s.

A scenario may, for instance, include components such as OpenStack, OpenDaylight, OVS, KVM etc., where each scenario will include different source components or configurations.

To learn more about the scenarios supported in the Euphrates release refer to the scenario description documents provided:

Installation Procedure

Detailed step by step instructions for working with an installation toolchain and installing the required scenario are provided by the installation projects. The projects providing installation support for the OPNFV Euphrates release are: Apex, Compass4nfv, Daisy4NFV, Fuel and JOID.

The instructions for each toolchain can be found in these links:

OPNFV Test Frameworks

If you have elected to install the OPNFV platform using the deployment toolchain provided by OPNFV, your system will have been validated once the installation is completed. The basic deployment validation only addresses a small part of capabilities in the platform and you may want to execute more exhaustive tests. Some investigation will be required to select the right test suites to run on your platform.

Many of the OPNFV test project provide user-guide documentation and installation instructions in this document

User Guide & Configuration Guide

Abstract

OPNFV is a collaborative project aimed at providing a variety of virtualisation deployments intended to host applications serving the networking and carrier industries. This document provides guidance and instructions for using platform features designed to support these applications that are made available in the OPNFV Euphrates release.

This document is not intended to replace or replicate documentation from other upstream open source projects such as KVM, OpenDaylight, OpenStack, etc., but to highlight the features and capabilities delivered through the OPNFV project.

Introduction

OPNFV provides a suite of scenarios, infrastructure deployment options, which are able to be installed to host virtualised network functions (VNFs). This document intends to help users of the platform leverage the features and capabilities delivered by OPNFV.

OPNFVs’ Continuous Integration builds, deploys and tests combinations of virtual infrastructure components in what are defined as scenarios. A scenario may include components such as KVM, OpenDaylight, OpenStack, OVS, etc., where each scenario will include different source components or configurations. Scenarios are designed to enable specific features and capabilities in the platform that can be leveraged by the OPNFV user community.

Feature Overview

The following links outline the feature deliverables from participating OPNFV projects in the Euphrates release. Each of the participating projects provides detailed descriptions about the delivered features including use cases, implementation, and configuration specifics.

The following Configuration Guides and User Guides assume that the reader already has some knowledge about a given project’s specifics and deliverables. These Guides are intended to be used following the installation with an OPNFV installer to allow users to deploy and implement feature delivered by OPNFV.

If you are unsure about the specifics of a given project, please refer to the OPNFV wiki page at http://wiki.opnfv.org for more details.

Test Frameworks

Test Framework Overview

OPNFV Testing Overview

Introduction

Testing is one of the key activities in OPNFV and includes unit, feature, component, system level testing for development, automated deployment, performance characterization and stress testing.

Test projects are dedicated to provide frameworks, tooling and test-cases categorized as functional, performance or compliance testing. Test projects fulfill different roles such as verifying VIM functionality, benchmarking components and platforms or analysis of measured KPIs for OPNFV release scenarios.

Feature projects also provide their own test suites that either run independently or within a test project.

This document details the OPNFV testing ecosystem, describes common test components used by individual OPNFV projects and provides links to project specific documentation.

The OPNFV Testing Ecosystem

The OPNFV testing projects are represented in the following diagram:

Overview of OPNFV Testing projects

The major testing projects are described in the table below:

Project Description
Bottlenecks This project aims to find system bottlenecks by testing and verifying OPNFV infrastructure in a staging environment before committing it to a production environment. Instead of debugging a deployment in production environment, an automatic method for executing benchmarks which plans to validate the deployment during staging is adopted. This project forms a staging framework to find bottlenecks and to do analysis of the OPNFV infrastructure.
CPerf SDN Controller benchmarks and performance testing, applicable to controllers in general. Collaboration of upstream controller testing experts, external test tool developers and the standards community. Primarily contribute to upstream/external tooling, then add jobs to run those tools on OPNFV’s infrastructure.
Dovetail This project intends to define and provide a set of OPNFV related validation criteria/tests that will provide input for the OPNFV Complaince Verification Program. The Dovetail project is executed with the guidance and oversight of the Complaince and Certification (C&C) committee and work to secure the goals of the C&C committee for each release. The project intends to incrementally define qualification criteria that establish the foundations of how one is able to measure the ability to utilize the OPNFV platform, how the platform itself should behave, and how applications may be deployed on the platform.
Functest This project deals with the functional testing of the VIM and NFVI. It leverages several upstream test suites (OpenStack, ODL, ONOS, etc.) and can be used by feature project to launch feature test suites in CI/CD. The project is used for scenario validation.
NFVbench NFVbench is a compact and self contained data plane performance measurement tool for OpensStack based NFVi platforms. It is agnostic of the NFVi distribution, Neutron networking implementation and hardware. It runs on any Linux server with a DPDK compliant NIC connected to the NFVi platform data plane and bundles a highly efficient software traffic generator. Provides a fully automated measurement of most common packet paths at any level of scale and load using RFC-2544. Available as a Docker container with simple command line and REST interfaces. Easy to use as it takes care of most of the guesswork generally associated to data plane benchmarking. Can run in any lab or in production environments.
QTIP QTIP as the project for “Platform Performance Benchmarking” in OPNFV aims to provide user a simple indicator for performance, supported by comprehensive testing data and transparent calculation formula. It provides a platform with common services for performance benchmarking which helps users to build indicators by themselves with ease.
StorPerf The purpose of this project is to provide a tool to measure block and object storage performance in an NFVI. When complemented with a characterization of typical VF storage performance requirements, it can provide pass/fail thresholds for test, staging, and production NFVI environments.
VSPERF VSPERF is an OPNFV project that provides an automated test-framework and comprehensive test suite based on Industry Test Specifications for measuring NFVI data-plane performance. The data-path includes switching technologies with physical and virtual network interfaces. The VSPERF architecture is switch and traffic generator agnostic and test cases can be easily customized. Software versions and configurations including the vSwitch (OVS or VPP) as well as the network topology are controlled by VSPERF (independent of OpenStack). VSPERF is used as a development tool for optimizing switching technologies, qualification of packet processing components and for pre-deployment evaluation of the NFV platform data-path.
Yardstick The goal of the Project is to verify the infrastructure compliance when running VNF applications. NFV Use Cases described in ETSI GS NFV 001 show a large variety of applications, each defining specific requirements and complex configuration on the underlying infrastructure and test tools.The Yardstick concept decomposes typical VNF work-load performance metrics into a number of characteristics/performance vectors, which each of them can be represented by distinct test-cases.

Testing Working Group Resources

Test Results Collection Framework

Any test project running in the global OPNFV lab infrastructure and is integrated with OPNFV CI can push test results to the community Test Database using a common Test API. This database can be used to track the evolution of testing and analyse test runs to compare results across installers, scenarios and between technically and geographically diverse hardware environments.

Results from the databse are used to generate a dashboard with the current test status for each testing project. Please note that you can also deploy the Test Database and Test API locally in your own environment.

Overall Test Architecture

The management of test results can be summarized as follows:

+-------------+    +-------------+    +-------------+
|             |    |             |    |             |
|   Test      |    |   Test      |    |   Test      |
| Project #1  |    | Project #2  |    | Project #N  |
|             |    |             |    |             |
+-------------+    +-------------+    +-------------+
         |               |               |
         V               V               V
     +---------------------------------------------+
     |                                             |
     |           Test Rest API front end           |
     |    http://testresults.opnfv.org/test        |
     |                                             |
     +---------------------------------------------+
         ^                |                     ^
         |                V                     |
         |     +-------------------------+      |
         |     |                         |      |
         |     |    Test Results DB      |      |
         |     |         Mongo DB        |      |
         |     |                         |      |
         |     +-------------------------+      |
         |                                      |
         |                                      |
   +----------------------+        +----------------------+
   |                      |        |                      |
   | Testing Dashboards   |        |  Test Landing page   |
   |                      |        |                      |
   +----------------------+        +----------------------+
The Test Database

A Mongo DB Database was introduced for the Brahmaputra release. The following collections are declared in this database:

  • pods: the list of pods used for production CI
  • projects: the list of projects providing test cases
  • test cases: the test cases related to a given project
  • results: the results of the test cases
  • scenarios: the OPNFV scenarios tested in CI

This database can be used by any project through the Test API. Please note that projects may also use additional databases. The Test Database is mainly use to collect CI test results and generate scenario trust indicators. The Test Database is also cloned for OPNFV Plugfests in order to provide a private datastore only accessible to Plugfest participants.

Test API description

The Test API is used to declare pods, projects, test cases and test results. Pods correspond to a cluster of machines (3 controller and 2 compute nodes in HA mode) used to run the tests and are defined in the Pharos project. The results pushed in the database are related to pods, projects and test cases. Trying to push results generated from a non-referenced pod will return an error message by the Test API.

The data model is very basic, 5 objects are available:
  • Pods
  • Projects
  • Test cases
  • Results
  • Scenarios

For detailed information, please go to http://artifacts.opnfv.org/releng/docs/testapi.html

The code of the Test API is hosted in the releng repository [TST2]. The static documentation of the Test API can be found at [TST3]. The Test API has been dockerized and may be installed locally in your lab.

The deployment of the Test API has been automated. A jenkins job manages:

  • the unit tests of the Test API
  • the creation of a new docker file
  • the deployment of the new Test API
  • the archive of the old Test API
  • the backup of the Mongo DB
Test API Authorization

PUT/DELETE/POST operations of the TestAPI now require token based authorization. The token needs to be added in the request using a header ‘X-Auth-Token’ for access to the database.

e.g:

headers['X-Auth-Token']

The value of the header i.e the token can be accessed in the jenkins environment variable TestApiToken. The token value is added as a masked password.

headers['X-Auth-Token'] = os.environ.get('TestApiToken')

The above example is in Python. Token based authentication has been added so that only CI pods running Jenkins jobs can access the database. Please note that currently token authorization is implemented but is not yet enabled.

Test Project Reporting

The reporting page for the test projects is http://testresults.opnfv.org/reporting/

Testing group reporting page

This page provides reporting per OPNFV release and per testing project.

Testing group Euphrates reporting page

An evolution of the reporting page is planned to unify test reporting by creating a landing page that shows the scenario status in one glance (this information was previously consolidated manually on a wiki page). The landing page will be displayed per scenario and show:

  • the status of the deployment
  • the score from each test suite. There is no overall score, it is determined

by each test project. * a trust indicator

Test Case Catalog

Until the Colorado release, each testing project managed the list of its test cases. This made it very hard to have a global view of the available test cases from the different test projects. A common view was possible through the API but it was not very user friendly. Test cases per project may be listed by calling:

with project_name: bottlenecks, functest, qtip, storperf, vsperf, yardstick

A test case catalog has now been realized [TST4]. Roll over the project then click to get the list of test cases, and then click on the case to get more details.

Testing group testcase catalog
Test Dashboards

The Test Dashboard is used to provide a consistent view of the results collected in CI. The results shown on the dashboard are post processed from the Database, which only contains raw results. The dashboard can be used in addition to the reporting page (high level view) to allow the creation of specific graphs according to what the test owner wants to show.

In Brahmaputra, a basic dashboard was created in Functest. In Colorado, Yardstick used Grafana (time based graphs) and ELK (complex graphs). Since Danube, the OPNFV testing community decided to adopt the ELK framework and to use Bitergia for creating highly flexible dashboards [TST5].

Testing group testcase catalog
Power Consumption Monitoring Framework
Introduction

Power consumption is a key driver for NFV. As an end user is interested to know which application is good or bad regarding power consumption and explains why he/she has to plug his/her smartphone every day, we would be interested to know which VNF is power consuming.

Power consumption is hard to evaluate empirically. It is however possible to collect information and leverage Pharos federation to try to detect some profiles/footprints. In fact thanks to CI, we know that we are running a known/deterministic list of cases. The idea is to correlate this knowledge with the power consumption to try at the end to find statistical biais.

High Level Architecture

The energy recorder high level architecture may be described as follows:

Energy recorder high level architecture

The energy monitoring system in based on 3 software components:

  • Power info collector: poll server to collect instantaneous power consumption information
  • Energy recording API + influxdb: On one leg receive servers consumption and

on the other, scenarios notfication. It then able to establish te correlation between consumption and scenario and stores it into a time-series database (influxdb) * Python SDK: A Python SDK using decorator to send notification to Energy recording API from testcases scenarios

Power Info Collector

It collects instantaneous power consumption information and send it to Event API in charge of data storing. The collector use different connector to read the power consumption on remote servers:

  • IPMI: this is the basic method and is manufacturer dependent. Depending on manufacturer, refreshing delay may vary (generally for 10 to 30 sec.)
  • RedFish: redfish is an industry RESTFUL API for hardware managment. Unfortunatly it is not yet supported by many suppliers.
  • ILO: HP RESTFULL API: This connector support as well 2.1 as 2.4 version of HP-ILO

IPMI is supported by at least:

  • HP
  • IBM
  • Dell
  • Nokia
  • Advantech
  • Lenovo
  • Huawei

Redfish API has been successfully tested on:

  • HP
  • Dell
  • Huawei (E9000 class servers used in OPNFV Community Labs are IPMI 2.0

compliant and use Redfish login Interface through Browsers supporting JRE1.7/1.8)

Several test campaigns done with physical Wattmeter showed that IPMI results were notvery accurate but RedFish were. So if Redfish is available, it is highly recommended to use it.

Installation

To run the server power consumption collector agent, you need to deploy a docker container locally on your infrastructure.

This container requires:

  • Connectivy on the LAN where server administration services (ILO, eDrac, IPMI,...) are configured and IP access to the POD’s servers
  • Outgoing HTTP access to the Event API (internet)

Build the image by typing:

curl -s https://raw.githubusercontent.com/bherard/energyrecorder/master/docker/server-collector.dockerfile|docker build -t energyrecorder/collector -

Create local folder on your host for logs and config files:

mkdir -p /etc/energyrecorder
mkdir -p /var/log/energyrecorder

In /etc/energyrecorder create a configuration for logging in a file named collector-logging.conf:

curl -s https://raw.githubusercontent.com/bherard/energyrecorder/master/server-collector/conf/collector-logging.conf.sample > /etc/energyrecorder/collector-logging.conf

Check configuration for this file (folders, log levels.....) In /etc/energyrecorder create a configuration for the collector in a file named collector-settings.yaml:

curl -s https://raw.githubusercontent.com/bherard/energyrecorder/master/server-collector/conf/collector-settings.yaml.sample > /etc/energyrecorder/collector-settings.yaml

Define the “PODS” section and their “servers” section according to the environment to monitor. Note: The “environment” key should correspond to the pod name, as defined in the “NODE_NAME” environment variable by CI when running.

IMPORTANT NOTE: To apply a new configuration, you need to kill the running container an start a new one (see below)

Run

To run the container, you have to map folder located on the host to folders in the container (config, logs):

docker run -d --name energy-collector --restart=always -v /etc/energyrecorder:/usr/local/energyrecorder/server-collector/conf -v /var/log/energyrecorder:/var/log/energyrecorder energyrecorder/collector
Energy Recording API

An event API to insert contextual information when monitoring energy (e.g. start Functest, start Tempest, destroy VM, ..) It is associated with an influxDB to store the power consumption measures It is hosted on a shared environment with the folling access points:

Component Connectivity
Energy recording API documentation http://energy.opnfv.fr/resources/doc/
influxDB (data) http://energy.opnfv.fr:8086

In you need, you can also host your own version of the Energy recording API (in such case, the Python SDK may requires a settings update) If you plan to use the default shared API, following steps are not required.

Image creation

First, you need to buid an image:

curl -s https://raw.githubusercontent.com/bherard/energyrecorder/master/docker/recording-api.dockerfile|docker build -t energyrecorder/api -
Setup

Create local folder on your host for logs and config files:

mkdir -p /etc/energyrecorder
mkdir -p /var/log/energyrecorder
mkdir -p /var/lib/influxdb

In /etc/energyrecorder create a configuration for logging in a file named webapp-logging.conf:

curl -s https://raw.githubusercontent.com/bherard/energyrecorder/master/recording-api/conf/webapp-logging.conf.sample > /etc/energyrecorder/webapp-logging.conf

Check configuration for this file (folders, log levels.....)

In /etc/energyrecorder create a configuration for the collector in a file named webapp-settings.yaml:

curl -s https://raw.githubusercontent.com/bherard/energyrecorder/master/recording-api/conf/webapp-settings.yaml.sample > /etc/energyrecorder/webapp-settings.yaml

Normaly included configuration is ready to use except username/passwer for influx (see run-container.sh bellow). Use here the admin user.

IMPORTANT NOTE: To apply a new configuration, you need to kill the running container an start a new one (see bellow)

Run

To run the container, you have to map folder located on the host to folders in the container (config, logs):

docker run -d --name energyrecorder-api -p 8086:8086 -p 8888:8888  -v /etc/energyrecorder:/usr/local/energyrecorder/web.py/conf -v /var/log/energyrecorder/:/var/log/energyrecorder -v /var/lib/influxdb:/var/lib/influxdb energyrecorder/webapp admin-influx-user-name admin-password readonly-influx-user-name user-password

with

Parameter name Description
admin-influx-user-name
admin-password
readonly-influx-user-name
user-password
Influx user with admin grants to create
Influx password to set to admin user
Influx user with readonly grants to create
Influx password to set to readonly user

NOTE: Local folder /var/lib/influxdb is the location web influx data are stored. You may used anything else at your convience. Just remember to define this mapping properly when running the container.

Power consumption Python SDK

a Python SDK - almost not intrusive, based on python decorator to trigger call to the event API.

It is currently hosted in Functest repo but if other projects adopt it, a dedicated project could be created and/or it could be hosted in Releng.

How to use the SDK

import the energy library:

import functest.energy.energy as energy

Notify that you want power recording in your testcase:

@energy.enable_recording
def run(self):
    self.do_some_stuff1()
    self.do_some_stuff2()

If you want to register additional steps during the scenarios you can to it in 2 different ways.

Notify step on method definition:

@energy.set_step("step1")
def do_some_stuff1(self):
...
@energy.set_step("step2")
def do_some_stuff2(self):

Notify directly from code:

@energy.enable_recording
def run(self):
  Energy.set_step("step1")
  self.do_some_stuff1()
  ...
  Energy.set_step("step2")
  self.do_some_stuff2()
SDK Setting

Settings delivered in the project git are ready to use and assume that you will use the sahre energy recording API. If you want to use an other instance, you have to update the key “energy_recorder.api_url” in <FUNCTEST>/functest/ci/config_functest.yaml” by setting the proper hostname/IP

Results

Here is an example of result comming from LF POD2. This sequence represents several CI runs in a raw. (0 power corresponds to hard reboot of the servers)

You may connect http://energy.opnfv.fr:3000 for more results (ask for credentials to infra team).

Energy monitoring of LF POD2
OPNFV Test Group Information

For more information or to participate in the OPNFV test community please see the following:

wiki: https://wiki.opnfv.org/testing

mailing list: test-wg@lists.opnfv.org

IRC channel: #opnfv-testperf

weekly meeting (https://wiki.opnfv.org/display/meetings/TestPerf):
  • Usual time: Every Thursday 15:00-16:00 UTC / 7:00-8:00 PST
  • APAC time: 2nd Wednesday of the month 8:00-9:00 UTC

Testing User Guides

This page provides the links to the installation, configuration and user guides of the different test projects.

Bottlenecks

Bottlenecks User Guide
User Guide

For each testsuite, you can either setup teststory or testcase to run certain test. teststory comprises several testcases as a set in one configuration file. You could call teststory or testcase by using Bottlenecks user interfaces. Details will be shown in the following section.

Brief Introdcution of the Test suites in Project Releases

Brahmaputra:

  • rubbos is introduced, which is an end2end NFVI perforamnce tool.
  • Virtual switch test framework (VSTF) is also introduced, which is an test framework used for vswitch performance test.

Colorado:

  • rubbos is refactored by using puppet, and ease the integration with several load generators(Client) and worker(Tomcat).
  • VSTF is refactored by extracting the test case’s configuration information.

Danube:

  • posca testsuite is introduced to implement stress (factor), feature and tuning test in parametric manner.
  • Two testcases are developed and integrated into community CI pipeline.
  • Rubbos and VSTF are not supported any more.

Euphrates:

  • Introduction of a simple monitoring module, i.e., Prometheus+Collectd+Node+Grafana to monitor the system behavior when executing stress tests.
  • Support VNF scale up/out tests to verify NFVI capability to adapt the resource consuming.
  • Extend Life-cycle test to data-plane to validate the system capability to handle concurrent networks usage.
  • Testing framework is revised to support installer-agnostic testing.

These enhancements and test cases help the end users to gain more comprehensive understanding of the SUT. Graphic reports of the system behavior additional to test cases are provided to indicate the confidence level of SUT. Installer-agnostic testing framework allow end user to do stress testing adaptively over either Open Source or commercial deployments.

Integration Description
Release Integrated Installer Supported Testsuite
Brahmaputra Fuel Rubbos, VSTF
Colorado Compass Rubbos, VSTF
Danube Compass POSCA
Euphrates Any POSCA
Test suite & Test case Description
POSCA posca_factor_ping
posca_factor_system_bandwidth
posca_facotor_througputs
posca_feature_scaleup
posca_feature_scaleout

As for the abandoned test suite in the previous Bottlenecks releases, please refer to http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/deprecated.html.

POSCA Testsuite Guide
POSCA Introduction

The POSCA (Parametric Bottlenecks Testing Catalogue) test suite classifies the bottlenecks test cases and results into 5 categories. Then the results will be analyzed and bottlenecks will be searched among these categories.

The POSCA testsuite aims to locate the bottlenecks in parametric manner and to decouple the bottlenecks regarding the deployment requirements. The POSCA testsuite provides an user friendly way to profile and understand the E2E system behavior and deployment requirements.

Goals of the POSCA testsuite:
  1. Automatically locate the bottlenecks in a iterative manner.
  2. Automatically generate the testing report for bottlenecks in different categories.
  3. Implementing Automated Staging.
Scopes of the POSCA testsuite:
  1. Modeling, Testing and Test Result analysis.
  2. Parameters choosing and Algorithms.
Test stories of POSCA testsuite:
  1. Factor test (Stress test): base test cases that Feature test and Optimization will be dependant on.
  2. Feature test: test cases for features/scenarios.
  3. Optimization test: test to tune the system parameter.

Detailed workflow is illutrated below.

Preinstall Packages
if [ -d usr/local/bin/docker-compose ]; then
    rm -rf usr/local/bin/docker-compose
fi
curl -L https://github.com/docker/compose/releases/download/1.11.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
Run POSCA Locally

The test environment preparation, the installation of the testing tools, the execution of the tests and the reporting/analyisis of POSCA test suite are highly automated. A few steps are needed to run it locally.

In Euphrates, Bottlenecks has modified its framework to support installer-agnostic testing which means that test cases could be executed over different deployments.

Downloading Bottlenecks Software
mkdir /home/opnfv
cd /home/opnfv
git clone https://gerrit.opnfv.org/gerrit/bottlenecks
cd bottlenecks
Preparing Python Virtual Evnironment
. pre_virt_env.sh
Preparing configuration/description files

Put OpenStack RC file (admin_rc.sh), os_carcert and pod.yaml (pod descrition file) in /tmp directory. Edit admin_rc.sh and add the following line

export OS_CACERT=/tmp/os_cacert

If you are using compass, fuel, apex or joid to deploy your openstack environment, you could use the following command to get the required files.

bash /utils/env_prepare/config_prepare.sh -i <installer> [--debug]

Note that if we execute the command above, then admin_rc.sh and pod.yml gets created automatically in /tmp folder along with the line export OS_CACERT=/tmp/os_cacert added in admin_rc.sh file.

Executing Specified Testcase
  1. Bottlenecks provides a CLI interface to run the tests, which is one of the most convenient way since it is more close to our natural languge. An GUI interface with rest API will also be provided in later update.
bottlenecks testcase|teststory run <testname>

For the *testcase* command, testname should be as the same name of the test case configuration file located in testsuites/posca/testcase_cfg.
For stress tests in Danube/Euphrates, *testcase* should be replaced by either *posca_factor_ping* or *posca_factor_system_bandwidth*.
For the *teststory* command, a user can specify the test cases to be executed by defining it in a teststory configuration file located in testsuites/posca/testsuite_story. There is also an example there named *posca_factor_test*.
  1. There are also other 2 ways to run test cases and test stories.

    The first one is to use shell script.

bash run_tests.sh [-h|--help] -s <testsuite>|-c <testcase>


The second is to use python interpreter.
$REPORT=False
opts="--privileged=true -id"
docker_volume="-v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp"
docker run $opts --name bottlenecks-load-master $docker_volume opnfv/bottlenecks:latest /bin/bash
sleep 5
POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca"
docker exec bottlenecks-load-master python ${POSCA_SCRIPT}/../run_posca.py testcase|teststory <testname> ${REPORT}
Showing Report

Bottlenecks uses ELK to illustrate the testing results. Asumming IP of the SUT (System Under Test) is denoted as ipaddr, then the address of Kibana is http://[ipaddr]:5601. One can visit this address to see the illustrations. Address for elasticsearch is http://[ipaddr]:9200. One can use any Rest Tool to visit the testing data stored in elasticsearch.

Cleaning Up Environment
. rm_virt_env.sh

If you want to clean the dockers that established during the test, you can excute the additional commands below.

bash run_tests.sh --cleanup

Note that you can also add cleanup parameter when you run a test case. Then environment will be automatically cleaned up when completing the test.

Run POSCA through Community CI

POSCA test cases are runned by OPNFV CI now. See https://build.opnfv.org for details of the building jobs. Each building job is set up to execute a single test case. The test results/logs will be printed on the web page and reported automatically to community MongoDB. There are two ways to report the results.

  1. Report testing result by shell script
bash run_tests.sh [-h|--help] -s <testsuite>|-c <testcase> --report
  1. Report testing result by python interpreter
REPORT=True
opts="--privileged=true -id"
docker_volume="-v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp"
docker run $opts --name bottlenecks-load-master $docker_volume opnfv/bottlenecks:latest /bin/bash
sleep 5
REPORT="True"
POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca"
docker exec bottlenecks_load-master python ${POSCA_SCRIPT}/../run_posca.py testcase|teststory <testcase> ${REPORT}
Test Result Description
Dashbard guide
Scope

This document provides an overview of the results of test cases developed by the OPNFV Bottlenecks Project, executed on OPNFV community labs.

OPNFV CI(Continous Integration) system provides automated build, deploy and testing for the software developed in OPNFV. Unless stated, the reported tests are automated via Jenkins Jobs.

Test results are visible in the following dashboard:

  • Testing dashboard: uses Mongo DB to store test results and Bitergia for visualization, which includes the rubbos test result, vstf test result.
Bottlenecks - Test Cases
POSCA Stress (Factor) Test of System bandwidth
Test Case
Bottlenecks POSCA Stress Test Traffic
test case name posca_factor_system_bandwith
description Stress test regarding baseline of the system for a single user, i.e., a VM pair while increasing the package size
configuration
config file:
/testsuite/posca/testcase_cfg/
posca_factor_system_bandwith.yaml

stack number: 1

test result PKT loss rate, latency, throupht, cpu usage
Configration
test_config:
  tool: netperf
  protocol: tcp
  test_time: 20
  tx_pkt_sizes: 64, 256, 1024, 4096, 8192, 16384, 32768, 65536
  rx_pkt_sizes: 64, 256, 1024, 4096, 8192, 16384, 32768, 65536
  cpu_load: 0.9
  latency: 100000
runner_config:
  dashboard: "y"
  dashboard_ip:
  stack_create: yardstick
  yardstick_test_ip:
  yardstick_test_dir: "samples"
  yardstick_testcase: "netperf_bottlenecks"
POSCA Stress (Factor) Test of Perfomance Life-Cycle
Test Case
Bottlenecks POSCA Stress Test Ping
test case name posca_posca_ping
description Stress test regarding life-cycle while using ping to validate the VM pairs constructions
configuration
config file:
/testsuite/posca/testcase_cfg/posca_posca_ping.yaml

stack number: 5, 10, 20, 50 ...

test result PKT loss rate, success rate, test time, latency
Configuration
load_manager:
  scenarios:
    tool: ping
    test_times: 100
    package_size:
    num_stack: 5, 10, 20
    package_loss: 10%

  contexts:
    stack_create: yardstick
    flavor:
    yardstick_test_ip:
    yardstick_test_dir: "samples"
    yardstick_testcase: "ping_bottlenecks"

dashboard:
  dashboard: "y"
  dashboard_ip:

Functest

Functest Installation Guide
Introduction

This document describes how to install and configure Functest in OPNFV.

High level architecture

The high level architecture of Functest within OPNFV can be described as follows:

CIMC/Lights+out management               Admin  Mgmt/API  Public  Storage Private
                                          PXE
+                                           +       +        +       +       +
|                                           |       |        |       |       |
|     +----------------------------+        |       |        |       |       |
|     |                            |        |       |        |       |       |
+-----+       Jumphost             |        |       |        |       |       |
|     |                            |        |       |        |       |       |
|     |   +--------------------+   |        |       |        |       |       |
|     |   |                    |   |        |       |        |       |       |
|     |   | Tools              |   +----------------+        |       |       |
|     |   | - Rally            |   |        |       |        |       |       |
|     |   | - Robot            |   |        |       |        |       |       |
|     |   | - RefStack         |   |        |       |        |       |       |
|     |   |                    |   |-------------------------+       |       |
|     |   | Testcases          |   |        |       |        |       |       |
|     |   | - VIM              |   |        |       |        |       |       |
|     |   |                    |   |        |       |        |       |       |
|     |   | - SDN Controller   |   |        |       |        |       |       |
|     |   |                    |   |        |       |        |       |       |
|     |   | - Features         |   |        |       |        |       |       |
|     |   |                    |   |        |       |        |       |       |
|     |   | - VNF              |   |        |       |        |       |       |
|     |   |                    |   |        |       |        |       |       |
|     |   +--------------------+   |        |       |        |       |       |
|     |     Functest Docker        +        |       |        |       |       |
|     |                            |        |       |        |       |       |
|     |                            |        |       |        |       |       |
|     |                            |        |       |        |       |       |
|     +----------------------------+        |       |        |       |       |
|                                           |       |        |       |       |
|    +----------------+                     |       |        |       |       |
|    |             1  |                     |       |        |       |       |
+----+ +--------------+-+                   |       |        |       |       |
|    | |             2  |                   |       |        |       |       |
|    | | +--------------+-+                 |       |        |       |       |
|    | | |             3  |                 |       |        |       |       |
|    | | | +--------------+-+               |       |        |       |       |
|    | | | |             4  |               |       |        |       |       |
|    +-+ | | +--------------+-+             |       |        |       |       |
|      | | | |             5  +-------------+       |        |       |       |
|      +-+ | |  nodes for     |             |       |        |       |       |
|        | | |  deploying     +---------------------+        |       |       |
|        +-+ |  OPNFV         |             |       |        |       |       |
|          | |                +------------------------------+       |       |
|          +-+     SUT        |             |       |        |       |       |
|            |                +--------------------------------------+       |
|            |                |             |       |        |       |       |
|            |                +----------------------------------------------+
|            +----------------+             |       |        |       |       |
|                                           |       |        |       |       |
+                                           +       +        +       +       +
             SUT = System Under Test

Note connectivity to management network is not needed for most of the testcases. But it may be needed for some specific snaps tests.

All the libraries and dependencies needed by all of the Functest tools are pre-installed into the Docker images. This allows running Functest on any platform.

The automated mechanisms inside the Functest Docker containers will:

  • Prepare the environment according to the System Under Test (SUT)
  • Perform the appropriate functional tests
  • Push the test results into the OPNFV test result database (optional)

The OpenStack credentials file must be provided to the container.

These Docker images can be integrated into CI or deployed independently.

Please note that the Functest Docker images have been designed for OPNFV, however, it would be possible to adapt them to any OpenStack based VIM + controller environment, since most of the test cases are integrated from upstream communities.

The functional test cases are described in the Functest User Guide [2]

Prerequisites

The OPNFV deployment is out of the scope of this document but it can be found in http://docs.opnfv.org. The OPNFV platform is considered as the SUT in this document.

Several prerequisites are needed for Functest:

  1. A Jumphost to run Functest on
  2. A Docker daemon shall be installed on the Jumphost
  3. A public/external network created on the SUT
  4. An admin/management network created on the SUT
  5. Connectivity from the Jumphost to the SUT public/external network

Some specific SNAPS tests may require a connectivity from the Jumphost to the SUT admin/management network but most of the test cases do not. This requirement can be changed by overriding the ‘interface’ attribute (OS_INTERFACE) value to ‘public’ in the credentials file. Another means to circumvent this issue would be to change the ‘snaps.use_keystone’ value from True to False.

WARNING: Connectivity from Jumphost is essential and it is of paramount importance to make sure it is working before even considering to install and run Functest. Make also sure you understand how your networking is designed to work.

NOTE: Jumphost refers to any server which meets the previous requirements. Normally it is the same server from where the OPNFV deployment has been triggered previously, but it could be any server with proper connectivity to the SUT.

NOTE: If your Jumphost is operating behind a company http proxy and/or firewall, please consult first the section Proxy Support, towards the end of this document. The section details some tips/tricks which may be of help in a proxified environment.

Docker installation

Docker installation and configuration is only needed to be done once through the life cycle of Jumphost.

If your Jumphost is based on Ubuntu, SUSE, RHEL or CentOS linux, please consult the references below for more detailed instructions. The commands below are offered as a short reference.

Tip: For running docker containers behind the proxy, you need first some extra configuration which is described in section Docker Installation on CentOS behind http proxy. You should follow that section before installing the docker engine.

Docker installation needs to be done as root user. You may use other userid’s to create and run the actual containers later if so desired. Log on to your Jumphost as root user and install the Docker Engine (e.g. for CentOS family):

curl -sSL https://get.docker.com/ | sh
systemctl start docker

*Tip:* If you are working through proxy, please set the https_proxy
environment variable first before executing the curl command.

Add your user to docker group to be able to run commands without sudo:

sudo usermod -aG docker <your_user>
A reconnection is needed. There are 2 ways for this:
  1. Re-login to your account
  2. su - <username>
References - Installing Docker Engine on different Linux Operating Systems:
Public/External network on SUT

Some of the tests against the VIM (Virtual Infrastructure Manager) need connectivity through an existing public/external network in order to succeed. This is needed, for example, to create floating IPs to access VM instances through the public/external network (i.e. from the Docker container).

By default, the five OPNFV installers provide a fresh installation with a public/external network created along with a router. Make sure that the public/external subnet is reachable from the Jumphost and an external router exists.

Hint: For the given OPNFV Installer in use, the IP sub-net address used for the public/external network is usually a planning item and should thus be known. Ensure you can reach each node in the SUT, from the Jumphost using the ‘ping’ command using the respective IP address on the public/external network for each node in the SUT. The details of how to determine the needed IP addresses for each node in the SUT may vary according to the used installer and are therefore ommitted here.

Installation and configuration

Alpine containers have been introduced in Euphrates. Alpine allows Functest testing in several very light containers and thanks to the refactoring on dependency management should allow the creation of light and fully customized docker images.

Functest Dockers

Docker images are available on the dockerhub:

  • opnfv/functest-core
  • opnfv/functest-healthcheck
  • opnfv/functest-smoke
  • opnfv/functest-features
  • opnfv/functest-components
  • opnfv/functest-vnf
  • opnfv/functest-parser
  • opnfv/functest-restapi

The tag “opnfv-5.0.0” is the official release image in Euphrates, but you can also pull “euphrates” tag as it is being maintained by Functest team and might include bugfixes.

The Functest docker container environment can -in principle- be also used with non-OPNFV official installers (e.g. ‘devstack’), with the disclaimer that support for such environments is outside of the scope and responsibility of the OPNFV project.

Preparing your environment

cat env:

INSTALLER_TYPE=XXX
INSTALLER_IP=XXX
EXTERNAL_NETWORK=XXX
DEPLOY_SCENARIO=XXX

See section on environment variables for details.

cat openstack.creds:

export OS_AUTH_URL=XXX
export OS_USER_DOMAIN_NAME=XXX
export OS_PROJECT_DOMAIN_NAME=XXX
export OS_USERNAME=XXX
export OS_TENANT_NAME=XXX
export OS_PROJECT_NAME=XXX
export OS_PASSWORD=XXX
export OS_VOLUME_API_VERSION=XXX
export OS_IDENTITY_API_VERSION=XXX
export OS_IMAGE_API_VERSION=XXX

See section on OpenStack credentials for details.

Create a directory for the different images (attached as a Docker volume):

mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/euphrates | bash -s -- images && ls -1 images/*

images/CentOS-7-aarch64-GenericCloud.qcow2
images/CentOS-7-aarch64-GenericCloud.qcow2.xz
images/CentOS-7-x86_64-GenericCloud.qcow2
images/cirros-0.3.5-x86_64-disk.img
images/cirros-0.3.5-x86_64-lxc.tar.gz
images/cirros-d161201-aarch64-disk.img
images/cirros-d161201-aarch64-initramfs
images/cirros-d161201-aarch64-kernel
images/cloudify-manager-premium-4.0.1.qcow2
images/img
images/trusty-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
images/vyos-1.1.7.img
Testing healthcheck suite

Run healthcheck suite:

sudo docker run --env-file env \
    -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds  \
    -v $(pwd)/images:/home/opnfv/functest/images  \
    opnfv/functest-healthcheck

Results shall be displayed as follows:

+----------------------------+------------------+---------------------+------------------+----------------+
|         TEST CASE          |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+----------------------------+------------------+---------------------+------------------+----------------+
|      connection_check      |     functest     |     healthcheck     |      00:02       |      PASS      |
|         api_check          |     functest     |     healthcheck     |      04:57       |      PASS      |
|     snaps_health_check     |     functest     |     healthcheck     |      00:51       |      PASS      |
+----------------------------+------------------+---------------------+------------------+----------------+
NOTE: the duration is a reference and it might vary depending on your SUT.
Testing smoke suite

Run smoke suite:

sudo docker run --env-file env \
    -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds  \
    -v $(pwd)/images:/home/opnfv/functest/images  \
    opnfv/functest-smoke

Results shall be displayed as follows:

+------------------------------+------------------+---------------+------------------+----------------+
|          TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
+------------------------------+------------------+---------------+------------------+----------------+
|          vping_ssh           |     functest     |     smoke     |      01:19       |      PASS      |
|        vping_userdata        |     functest     |     smoke     |      01:56       |      PASS      |
|     tempest_smoke_serial     |     functest     |     smoke     |      26:30       |      PASS      |
|         rally_sanity         |     functest     |     smoke     |      19:42       |      PASS      |
|       refstack_defcore       |     functest     |     smoke     |      22:00       |      PASS      |
|         snaps_smoke          |     functest     |     smoke     |      41:14       |      PASS      |
|             odl              |     functest     |     smoke     |      00:16       |      PASS      |
|         odl_netvirt          |     functest     |     smoke     |      00:00       |      SKIP      |
|             fds              |     functest     |     smoke     |      00:00       |      SKIP      |
+------------------------------+------------------+---------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
See User guide for details.
Testing features suite

Run features suite:

sudo docker run --env-file env \
    -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds  \
    -v $(pwd)/images:/home/opnfv/functest/images  \
    opnfv/functest-features

Results shall be displayed as follows:

+---------------------------+--------------------------+------------------+------------------+----------------+
|         TEST CASE         |         PROJECT          |       TIER       |     DURATION     |     RESULT     |
+---------------------------+--------------------------+------------------+------------------+----------------+
|          promise          |         promise          |     features     |      00:00       |      SKIP      |
|           bgpvpn          |          sdnvpn          |     features     |      00:00       |      SKIP      |
|       security_scan       |     securityscanning     |     features     |      00:00       |      SKIP      |
|      functest-odl-sfc     |           sfc            |     features     |      00:00       |      SKIP      |
|      domino-multinode     |          domino          |     features     |      00:00       |      SKIP      |
|     barometercollectd     |        barometer         |     features     |      00:00       |      SKIP      |
+---------------------------+--------------------------+------------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
See User guide for details.
Testing components suite

Run components suite:

sudo docker run --env-file env \
    -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds  \
    -v $(pwd)/images:/home/opnfv/functest/images  \
    opnfv/functest-components

Results shall be displayed as follows:

+-------------------------------+------------------+--------------------+------------------+----------------+
|           TEST CASE           |     PROJECT      |        TIER        |     DURATION     |     RESULT     |
+-------------------------------+------------------+--------------------+------------------+----------------+
|     tempest_full_parallel     |     functest     |     components     |      102:48      |      PASS      |
|           rally_full          |     functest     |     components     |      160:58      |      PASS      |
+-------------------------------+------------------+--------------------+------------------+----------------+
Testing vnf suite

Run vnf suite:

sudo docker run –env-file env
-v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds -v $(pwd)/images:/home/opnfv/functest/images opnfv/functest-vnf

Results shall be displayed as follows:

+---------------------------------+------------------+--------------+------------------+----------------+
|            TEST CASE            |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
+---------------------------------+------------------+--------------+------------------+----------------+
|           cloudify_ims          |     functest     |     vnf      |      21:25       |      PASS      |
|        orchestra_openims        |     functest     |     vnf      |      11:02       |      FAIL      |
|     orchestra_clearwaterims     |     functest     |     vnf      |      09:13       |      FAIL      |
+---------------------------------+------------------+--------------+------------------+----------------+
Environment variables
Several environement variables may be specified:
  • INSTALLER_TYPE=(apex|compass|daisy|fuel|joid)
  • INSTALLER_IP=<Specific IP Address>
  • DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode>

INSTALLER IP may be required by some test cases like SFC or Barometer in order to access the installer node and the deployment.

The format for the DEPLOY_SCENARIO env variable can be described as follows:
  • vim: (os|k8s) = OpenStack or Kubernetes
  • controller is one of ( nosdn | odl )
  • nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature )
  • ha_mode (high availability) is one of ( ha | noha )

If several features are pertinent then use the underscore character ‘_’ to separate each feature (e.g. ovs_kvm). ‘nofeature’ indicates that no OPNFV feature is deployed.

The list of supported scenarios per release/installer is indicated in the release note.

NOTE: The scenario name is mainly used to automatically detect if a test suite is runnable or not (e.g. it will prevent ODL test suite to be run on ‘nosdn’ scenarios). If not set, Functest will try to run the default test cases that might not include SDN controller or a specific feature

NOTE: An HA scenario means that 3 OpenStack controller nodes are deployed. It does not necessarily mean that the whole system is HA. See installer release notes for details.

Finally, three additional environment variables can also be passed in to the Functest Docker Container, using the -e “<EnvironmentVariable>=<Value>” mechanism. The first two parameters are only relevant to Jenkins CI invoked testing and should not be used when performing manual test scenarios:

  • NODE_NAME = <Test POD Name>
  • BUILD_TAG = <Jenkins Build Tag>
  • CI_DEBUG = <DebugTraceValue>

where:

  • <Test POD Name> = Symbolic name of the POD where the tests are run.

    Visible in test results files, which are stored to the database. This option is only used when tests are activated under Jenkins CI control. It indicates the POD/hardware where the test has been run. If not specified, then the POD name is defined as “Unknown” by default. DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.

  • <Jenkins Build tag> = Symbolic name of the Jenkins Build Job.

    Visible in test results files, which are stored to the database. This option is only set when tests are activated under Jenkins CI control. It enables the correlation of test results, which are independently pushed to the results database from different Jenkins jobs. DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.

  • <DebugTraceValue> = “true” or “false”

    Default = “false”, if not specified If “true” is specified, then additional debug trace text can be sent to the test results file / log files and also to the standard console output.

Openstack credentials

OpenStack credentials are mandatory and must be provided to Functest. When running the command “functest env prepare”, the framework will automatically look for the Openstack credentials file “/home/opnfv/functest/conf/openstack.creds” and will exit with error if it is not present or is empty.

There are 2 ways to provide that file:

  • by using a Docker volume with -v option when creating the Docker container. This is referred to in docker documentation as “Bind Mounting”. See the usage of this parameter in the following chapter.
  • or creating manually the file ‘/home/opnfv/functest/conf/openstack.creds’ inside the running container and pasting the credentials in it. Consult your installer guide for further details. This is however not instructed in this document.

In proxified environment you may need to change the credentials file. There are some tips in chapter: Proxy support

SSL Support

If you need to connect to a server that is TLS-enabled (the auth URL begins with “https”) and it uses a certificate from a private CA or a self-signed certificate, then you will need to specify the path to an appropriate CA certificate to use, to validate the server certificate with the environment variable OS_CACERT:

echo $OS_CACERT
/etc/ssl/certs/ca.crt

However, this certificate does not exist in the container by default. It has to be copied manually from the OpenStack deployment. This can be done in 2 ways:

  1. Create manually that file and copy the contents from the OpenStack controller.

  2. (Recommended) Add the file using a Docker volume when starting the container:

    -v <path_to_your_cert_file>:/etc/ssl/certs/ca.cert
    

You might need to export OS_CACERT environment variable inside the credentials file:

export OS_CACERT=/etc/ssl/certs/ca.crt

Certificate verification can be turned off using OS_INSECURE=true. For example, Fuel uses self-signed cacerts by default, so an pre step would be:

export OS_INSECURE=true
Functest docker container directory structure

Inside the Functest docker container, the following directory structure should now be in place:

`--
  |- home
  |   |-- opnfv
  |   |     `- functest
  |   |          |-- conf
  |   |          `-- results
  |    `-- repos
  |       `-- vnfs
  |- src
  |   |-- tempest
  |   |-- vims-test
  |   |-- odl_test
  |   `-- fds
  `- usr
      `- lib
         `- python2.7
            `- site-packages
               `- functest
                    |-- ...

Underneath the ‘/home/opnfv/functest’ directory, the Functest docker container includes two main directories:

  • The conf directory stores configuration files (e.g. the OpenStack creds are stored in path ‘/home/opnfv/functest/conf/openstack.creds’),
  • the results directory stores some temporary result log files

src and repos directories are used to host third party code used for the tests.

The structure of functest repo can be described as follows:

|-- INFO
|-- LICENSE
|-- api
|  `-- apidoc
|-- build.sh
|-- commons
|-- docker
|  |-- components
|  |-- core
|  |-- features
|  |-- healthcheck
|  |-- smoke
|  |-- vnf
|  |-- parser
|  |-- restapi
|  |-- thirdparty-requirements.txt
|-- docs
|  |-- com
|  |-- images
|  |-- release
|  |  `-- release-notes
|  |-- results
|  | testing
|  |  |-- developer
|  |    `-- user
|  |      |-- configguide
|  |      `-- userguide
`-- functest
  |-- api
  |  |-- base.py
  |  |-- server.py
  |  |-- urls.py
  |  |-- common
  |  |  |-- api_utils.py
  |  |  |-- thread.py
  |  `-- resources
  |     `-- v1
  |        |-- creds.py
  |        |-- envs.py
  |        |-- testcases.py
  |        |-- tiers.py
  |        |-- tasks.py
  |  `-- database
  |     |-- db.py
  |     `-- v1
  |        |-- handlers.py
  |        |-- models.py
  |  `-- swagger
  |-- ci
  │   |-- check_deployment.py
  │   |-- config_aarch64_patch.yaml
  │   |-- config_functest.yaml
  │   |-- config_patch.yaml
  │   |-- download_images.sh
  │   |-- logging.ini
  │   |-- rally_aarch64_patch.conf
  │   |-- run_tests.py
  │   |-- testcases.yaml
  │   |-- tier_builder.py
  │   |-- tier_handler.py
  |-- cli
  │   |-- cli_base.py
  │   |-- commands
  │   │   |-- cli_env.py
  │   │   |-- cli_os.py
  │   │   |-- cli_testcase.py
  │   │   |-- cli_tier.py
  │   |-- functest-complete.sh
  |-- core
  │   |-- feature.py
  │   |-- robotframework.py
  │   |-- testcase.py
  │   |-- unit.py
  │   |-- vnf.py
  |-- energy
  │   |-- energy.py
  |-- opnfv_tests
  │   `-- openstack
  │       |-- rally
  │       |-- refstack_client
  │       |-- snaps
  │       |-- tempest
  │       |-- vping
  │   `-- sdn
  │   │    `-- odl
  │   `-- vnf
  │       |-- ims
  │       `-- router
  |-- tests
  │   `-- unit
  │       |-- ci
  │       |-- cli
  │       |-- core
  │       |-- energy
  │       |-- features
  │       |-- odl
  │       |-- openstack
  │       |-- opnfv_tests
  │       |-- test_utils.py
  │       |-- utils
  │       `-- vnf
  |-- utils
  |    |-- config.py
  |    |-- constants.py
  |    |-- decorators.py
  |    |-- env.py
  |    |-- functest_utils.py
  |    |-- functest_vacation.py
  |    |-- openstack_clean.py
  |    |-- openstack_snapshot.py
  |    |-- openstack_tacker.py
  |    `-- openstack_utils.py
|-- requirements.txt
|-- setup.cfg
|-- setup.py
|-- test-requirements.txt
|-- tox.ini
|-- upper-constraints.txt

(Note: All *.pyc files removed from above list for brevity...)

We may distinguish several directories, the first level has 5 directories:

  • api: This directory is dedicated to the API (framework) documentations.
  • commons: This directory is dedicated for storage of traffic profile or any other test inputs that could be reused by any test project.
  • docker: This directory includes the needed files and tools to build the Functest Docker images.
  • docs: This directory includes documentation: Release Notes, User Guide, Configuration Guide and Developer Guide.
  • functest: This directory contains all the code needed to run functest internal cases and OPNFV onboarded feature or VNF test cases.

Functest directory has 7 sub-directories, which is located under /usr/lib/python2.7/site-packages/functest:

  • api: This directory is dedicated for the internal Functest API.

  • ci: This directory contains test structure definition files (e.g <filename>.yaml) and bash shell/python scripts used to configure and execute Functional tests. The test execution script can be executed under the control of Jenkins CI jobs.

  • cli: This directory holds the python based Functest CLI utility source code, which is based on the Python ‘click’ framework.

  • core: This directory holds the python based Functest core

    source code. Three abstraction classes have been created to ease the integration of internal, feature or vnf cases.

  • opnfv_tests: This directory includes the scripts required by Functest internal test cases and other feature projects test cases.

  • tests: This directory includes the functest unit tests.

  • utils: this directory holds Python source code for some general purpose helper utilities, which testers can also re-use in their own test code. See for an example the Openstack helper utility: ‘openstack_utils.py’.

Logs

By default all the logs are put un /home/opnfv/functest/results/functest.log. If you want to have more logs in console, you may edit the logging.ini file manually. Connect on the docker then edit the file located in /usr/lib/python2.7/site-packages/functest/ci/logging.ini

Change wconsole to console in the desired module to get more traces.

Configuration

You may also directly modify the python code or the configuration file (e.g. testcases.yaml used to declare test constraints) under /usr/lib/python2.7/site-packages/functest

Tips
Docker

When typing exit in the container prompt, this will cause exiting the container and probably stopping it. When stopping a running Docker container all the changes will be lost, there is a keyboard shortcut to quit the container without stopping it: <CTRL>-P + <CTRL>-Q. To reconnect to the running container DO NOT use the run command again (since it will create a new container), use the exec or attach command instead:

docker ps  # <check the container ID from the output>
docker exec -ti <CONTAINER_ID> /bin/bash

There are other useful Docker commands that might be needed to manage possible issues with the containers.

List the running containers:

docker ps

List all the containers including the stopped ones:

docker ps -a

Start a stopped container named “FunTest”:

docker start FunTest

Attach to a running container named “StrikeTwo”:

docker attach StrikeTwo

It is useful sometimes to remove a container if there are some problems:

docker rm <CONTAINER_ID>

Use the -f option if the container is still running, it will force to destroy it:

docker rm -f <CONTAINER_ID>

Check the Docker documentation [dockerdocs] for more information.

Checking Openstack and credentials

It is recommended and fairly straightforward to check that Openstack and credentials are working as expected.

Once the credentials are there inside the container, they should be sourced before running any Openstack commands:

source /home/opnfv/functest/conf/openstack.creds

After this, try to run any OpenStack command to see if you get any output, for instance:

openstack user list

This will return a list of the actual users in the OpenStack deployment. In any other case, check that the credentials are sourced:

env|grep OS_

This command must show a set of environment variables starting with OS_, for example:

OS_REGION_NAME=RegionOne
OS_USER_DOMAIN_NAME=Default
OS_PROJECT_NAME=admin
OS_AUTH_VERSION=3
OS_IDENTITY_API_VERSION=3
OS_PASSWORD=da54c27ae0d10dfae5297e6f0d6be54ebdb9f58d0f9dfc
OS_AUTH_URL=http://10.1.0.9:5000/v3
OS_USERNAME=admin
OS_TENANT_NAME=admin
OS_ENDPOINT_TYPE=internalURL
OS_INTERFACE=internalURL
OS_NO_CACHE=1
OS_PROJECT_DOMAIN_NAME=Default

If the OpenStack command still does not show anything or complains about connectivity issues, it could be due to an incorrect url given to the OS_AUTH_URL environment variable. Check the deployment settings.

Proxy support

If your Jumphost node is operating behind a http proxy, then there are 2 places where some special actions may be needed to make operations succeed:

  1. Initial installation of docker engine First, try following the official Docker documentation for Proxy settings. Some issues were experienced on CentOS 7 based Jumphost. Some tips are documented in section: Docker Installation on CentOS behind http proxy below.

If that is the case, make sure the resolv.conf and the needed http_proxy and https_proxy environment variables, as well as the ‘no_proxy’ environment variable are set correctly:

# Make double sure that the 'no_proxy=...' line in the
# 'openstack.creds' file is commented out first. Otherwise, the
# values set into the 'no_proxy' environment variable below will
# be ovewrwritten, each time the command
# 'source ~/functest/conf/openstack.creds' is issued.

cd ~/functest/conf/
sed -i 's/export no_proxy/#export no_proxy/' openstack.creds
source ./openstack.creds

# Next calculate some IP addresses for which http_proxy
# usage should be excluded:

publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+")

adminURL_IP=$(openstack catalog show identity | \
grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+")

export http_proxy="<your http proxy settings>"
export https_proxy="<your https proxy settings>"
export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP"

# Ensure that "git" uses the http_proxy
# This may be needed if your firewall forbids SSL based git fetch
git config --global http.sslVerify True
git config --global http.proxy <Your http proxy settings>

For example, try to use the nc command from inside the functest docker container:

nc -v opnfv.org 80
Connection to opnfv.org 80 port [tcp/http] succeeded!

nc -v opnfv.org 443
Connection to opnfv.org 443 port [tcp/https] succeeded!

Note: In a Jumphost node based on the CentOS family OS, the nc commands might not work. You can use the curl command instead.

curl http://www.opnfv.org:80

<HTML><HEAD><meta http-equiv=”content-type” . . </BODY></HTML>

curl https://www.opnfv.org:443

<HTML><HEAD><meta http-equiv=”content-type” . . </BODY></HTML>

(Ignore the content. If command returns a valid HTML page, it proves the connection.)

Docker Installation on CentOS behind http proxy

This section is applicable for CentOS family OS on Jumphost which itself is behind a proxy server. In that case, the instructions below should be followed before installing the docker engine:

1) # Make a directory '/etc/systemd/system/docker.service.d'
   # if it does not exist
   sudo mkdir /etc/systemd/system/docker.service.d

2) # Create a file called 'env.conf' in that directory with
   # the following contents:
   [Service]
   EnvironmentFile=-/etc/sysconfig/docker

3) # Set up a file called 'docker' in directory '/etc/sysconfig'
   # with the following contents:
   HTTP_PROXY="<Your http proxy settings>"
   HTTPS_PROXY="<Your https proxy settings>"
   http_proxy="${HTTP_PROXY}"
   https_proxy="${HTTPS_PROXY}"

4) # Reload the daemon
   systemctl daemon-reload

5) # Sanity check - check the following docker settings:
   systemctl show docker | grep -i env

   Expected result:
   ----------------
   EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
   DropInPaths=/etc/systemd/system/docker.service.d/env.conf

Now follow the instructions in [Install Docker on CentOS] to download and install the docker-engine. The instructions conclude with a “test pull” of a sample “Hello World” docker container. This should now work with the above pre-requisite actions.

Integration in CI

In CI we use the Docker images and execute the appropriate commands within the container from Jenkins.

4 steps have been defined::
  • functest-cleanup: clean existing functest dockers on the jumphost
  • functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, vnf)
  • functest-store-results: push logs to artifacts

See [3] for details.

References

[1] : Keystone and public end point constraint

[2] : Functest User guide

[3] : Functest Jenkins jobs

[4] : Functest Configuration guide

[5] : OPNFV main site

[6] : Functest wiki page

IRC support channel: #opnfv-functest

Functest User Guide
Introduction

The goal of this document is to describe the OPNFV Functest test cases and to provide a procedure to execute them.

IMPORTANT: It is assumed here that Functest has been properly deployed following the installation guide procedure [1].

Overview of the Functest suites

Functest is the OPNFV project primarily targeting functional testing. In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure.

The current list of test suites can be distributed over 4 main domains:
  • VIM (Virtualised Infrastructure Manager)
  • Controllers (i.e. SDN Controllers)
  • Features
  • VNF (Virtual Network Functions)

Functest test suites are also distributed in the OPNFV testing categories: healthcheck, smoke, features, components, performance, VNF, Stress tests.

All the Healthcheck and smoke tests of a given scenario must be succesful to validate the scenario for the release.

Domain Tier Test case Comments
VIM healthcheck connection _check Check OpenStack connectivity through SNAPS framework
api_check Check OpenStack API through SNAPS framework
snaps_health _check basic instance creation, check DHCP
smoke vping_ssh NFV “Hello World” using an SSH connection to a destination VM over a created floating IP address on the SUT Public / External network. Using the SSH connection a test script is then copied to the destination VM and then executed via SSH. The script will ping another VM on a specified IP address over the SUT Private Tenant network
vping_userdata Uses Ping with given userdata to test intra-VM connectivity over the SUT Private Tenant network. The correct operation of the NOVA Metadata service is also verified in this test
tempest_smoke _serial Generate and run a relevant Tempest Test Suite in smoke mode. The generated test set is dependent on the OpenStack deployment environment
rally_sanity Run a subset of the OpenStack Rally Test Suite in smoke mode
snaps_smoke Run the SNAPS-OO integration tests
refstack
_defcore
Reference RefStack suite tempest selection for NFV
components tempest_full _parallel Generate and run a full set of the OpenStack Tempest Test Suite. See the OpenStack reference test suite [2]. The generated test set is dependent on the OpenStack deployment environment
rally_full Run the OpenStack testing tool benchmarking OpenStack modules See the Rally documents [3]
Controllers smoke odl Opendaylight Test suite Limited test suite to check the basic neutron (Layer 2) operations mainly based on upstream testcases. See below for details
odl_netvirt Test Suite for the OpenDaylight SDN Controller when the NetVirt features are installed. It integrates some test suites from upstream using Robot as the test framework
Features features bgpvpn Implementation of the OpenStack bgpvpn API from the SDNVPN feature project. It allows for the creation of BGP VPNs. See SDNVPN User Guide for details
doctor Doctor platform, as of Colorado release, provides the three features: * Immediate Notification * Consistent resource state awareness for compute host down * Valid compute host status given to VM owner See Doctor User Guide for details
odl-sfc SFC testing for odl scenarios See SFC User Guide for details
parser Parser is an integration project which aims to provide placement/deployment templates translation for OPNFV platform, including TOSCA -> HOT, POLICY -> TOSCA and YANG -> TOSCA. it deals with a fake vRNC. See Parser User Guide for details
fds Test Suite for the OpenDaylight SDN Controller when the GBP features are installed. It integrates some test suites from upstream using Robot as the test framework
VNF vnf cloudify_ims Example of a real VNF deployment to show the NFV capabilities of the platform. The IP Multimedia Subsytem is a typical Telco test case, referenced by ETSI. It provides a fully functional VoIP System
orchestra
openims
OpenIMS deployment using Openbaton orchestrator
orchestra
cleawaterims
Cleawater IMS deployment using Openbaton orchestrator
vyos_vrouter vRouter testing
cloudify_ims perf Based on cloudify_ims test case cloudify_ims_perf substitutes the signaling test suite by an automatic deployment of an Ixia loader and generic SIP stress tests. This work has been initiated during the plugfest and allows realistic load tests on top of cloudify_ims. Please note that this test is available but not declared in testcases.yaml as it requires access to proprietary resources (Ixia loader)

As shown in the above table, Functest is structured into different ‘domains’, ‘tiers’ and ‘test cases’. Each ‘test case’ usually represents an actual ‘Test Suite’ comprised -in turn- of several test cases internally.

Test cases also have an implicit execution order. For example, if the early ‘healthcheck’ Tier testcase fails, or if there are any failures in the ‘smoke’ Tier testcases, there is little point to launch a full testcase execution round.

In Danube, we merged smoke and sdn controller tiers in smoke tier.

An overview of the Functest Structural Concept is depicted graphically below:

Functest Concepts Structure

Some of the test cases are developed by Functest team members, whereas others are integrated from upstream communities or other OPNFV projects. For example, Tempest is the OpenStack integration test suite and Functest is in charge of the selection, integration and automation of those tests that fit suitably to OPNFV.

The Tempest test suite is the default OpenStack smoke test suite but no new test cases have been created in OPNFV Functest.

The results produced by the tests run from CI are pushed and collected into a NoSQL database. The goal is to populate the database with results from different sources and scenarios and to show them on a Functest Dashboard. A screenshot of a live Functest Dashboard is shown below:

Functest Dashboard

Basic components (VIM, SDN controllers) are tested through their own suites. Feature projects also provide their own test suites with different ways of running their tests.

The notion of domain has been introduced in the description of the test cases stored in the Database. This parameters as well as possible tags can be used for the Test case catalog.

vIMS test case was integrated to demonstrate the capability to deploy a relatively complex NFV scenario on top of the OPNFV infrastructure.

Functest considers OPNFV as a black box. OPNFV offers a lot of potential combinations (which may change from one version to another):

  • 3 controllers (OpenDaylight, ONOS, OpenContrail)
  • 5 installers (Apex, Compass, Daisy, Fuel, Joid)

Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available deployed features. The system uses the environment variables (INSTALLER_TYPE and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given environment.

A convenience Functest CLI utility is also available to simplify setting up the Functest evironment, management of the OpenStack environment (e.g. resource clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified testcase, all test cases in a specified Tier, or the special case of execution of ALL testcases. The Functest CLI is introduced in more details in next section.

The different test cases are described in the remaining sections of this document.

VIM (Virtualized Infrastructure Manager)
Healthcheck tests

Since Danube, healthcheck tests have been refactored and rely on SNAPS, an OPNFV middleware project.

SNAPS stands for “SDN/NFV Application development Platform and Stack”. SNAPS is an object-oriented OpenStack library packaged with tests that exercise OpenStack. More information on SNAPS can be found in  [13]

Three tests are declared as healthcheck tests and can be used for gating by the installer, they cover functionally the tests previously done by healthcheck test case.

The tests are:

  • connection_check
  • api_check
  • snaps_health_check

Connection_check consists in 9 test cases (test duration < 5s) checking the connectivity with Glance, Keystone, Neutron, Nova and the external network.

Api_check verifies the retrieval of OpenStack clients: Keystone, Glance, Neutron and Nova and may perform some simple queries. When the config value of snaps.use_keystone is True, functest must have access to the cloud’s private network. This suite consists in 49 tests (test duration < 2 minutes).

Snaps_health_check creates a VM with a single port with an IPv4 address that is assigned by DHCP and then validates the expected IP with the actual.

Self-obviously, successful completion of the ‘healthcheck’ testcase is a necessary pre-requisite for the execution of all other test Tiers.

vPing_ssh

Given the script ping.sh:

#!/bin/sh
ping -c 1 $1 2>&1 >/dev/null
RES=$?
if [ "Z$RES" = "Z0" ] ; then
    echo 'vPing OK'
else
    echo 'vPing KO'
fi

The goal of this test is to establish an SSH connection using a floating IP on the Public/External network and verify that 2 instances can talk over a Private Tenant network:

vPing_ssh test case
+-------------+                    +-------------+
|             |                    |             |
|             | Boot VM1 with IP1  |             |
|             +------------------->|             |
|   Tester    |                    |   System    |
|             | Boot VM2           |    Under    |
|             +------------------->|     Test    |
|             |                    |             |
|             | Create floating IP |             |
|             +------------------->|             |
|             |                    |             |
|             | Assign floating IP |             |
|             | to VM2             |             |
|             +------------------->|             |
|             |                    |             |
|             | Establish SSH      |             |
|             | connection to VM2  |             |
|             | through floating IP|             |
|             +------------------->|             |
|             |                    |             |
|             | SCP ping.sh to VM2 |             |
|             +------------------->|             |
|             |                    |             |
|             | VM2 executes       |             |
|             | ping.sh to VM1     |             |
|             +------------------->|             |
|             |                    |             |
|             |    If ping:        |             |
|             |      exit OK       |             |
|             |    else (timeout): |             |
|             |      exit Failed   |             |
|             |                    |             |
+-------------+                    +-------------+

This test can be considered as an “Hello World” example. It is the first basic use case which must work on any deployment.

vPing_userdata

This test case is similar to vPing_ssh but without the use of Floating IPs and the Public/External network to transfer the ping script. Instead, it uses Nova metadata service to pass it to the instance at booting time. As vPing_ssh, it checks that 2 instances can talk to each other on a Private Tenant network:

vPing_userdata test case
+-------------+                    +-------------+
|             |                    |             |
|             | Boot VM1 with IP1  |             |
|             +------------------->|             |
|             |                    |             |
|             | Boot VM2 with      |             |
|             | ping.sh as userdata|             |
|             | with IP1 as $1.    |             |
|             +------------------->|             |
|   Tester    |                    |   System    |
|             | VM2 exeutes ping.sh|    Under    |
|             | (ping IP1)         |     Test    |
|             +------------------->|             |
|             |                    |             |
|             | Monitor nova       |             |
|             |  console-log VM 2  |             |
|             |    If ping:        |             |
|             |      exit OK       |             |
|             |    else (timeout)  |             |
|             |      exit Failed   |             |
|             |                    |             |
+-------------+                    +-------------+

When the second VM boots it will execute the script passed as userdata automatically. The ping will be detected by periodically capturing the output in the console-log of the second VM.

Tempest

Tempest [2] is the reference OpenStack Integration test suite. It is a set of integration tests to be run against a live OpenStack cluster. Tempest has suites of tests for:

  • OpenStack API validation
  • Scenarios
  • Other specific tests useful in validating an OpenStack deployment

Functest uses Rally [3] to run the Tempest suite. Rally generates automatically the Tempest configuration file tempest.conf. Before running the actual test cases, Functest creates the needed resources (user, tenant) and updates the appropriate parameters into the configuration file.

When the Tempest suite is executed, each test duration is measured and the full console output is stored to a log file for further analysis.

The Tempest testcases are distributed across two Tiers:

  • Smoke Tier - Test Case ‘tempest_smoke_serial’
  • Components Tier - Test case ‘tempest_full_parallel’

NOTE: Test case ‘tempest_smoke_serial’ executes a defined set of tempest smoke tests with a single thread (i.e. serial mode). Test case ‘tempest_full_parallel’ executes all defined Tempest tests using several concurrent threads (i.e. parallel mode). The number of threads activated corresponds to the number of available logical CPUs.

The goal of the Tempest test suite is to check the basic functionalities of the different OpenStack components on an OPNFV fresh installation, using the corresponding REST API interfaces.

Rally bench test suites

Rally [3] is a benchmarking tool that answers the question:

How does OpenStack work at scale?

The goal of this test suite is to benchmark all the different OpenStack modules and get significant figures that could help to define Telco Cloud KPIs.

The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:

  • authenticate
  • cinder
  • glance
  • heat
  • keystone
  • neutron
  • nova
  • quotas
  • ceilometer

A basic SLA (stop test on errors) has been implemented.

The Rally testcases are distributed across two Tiers:

  • Smoke Tier - Test Case ‘rally_sanity’
  • Components Tier - Test case ‘rally_full’

NOTE: Test case ‘rally_sanity’ executes a limited number of Rally smoke test cases. Test case ‘rally_full’ executes the full defined set of Rally tests.

Refstack-client to run OpenStack interop testcases

Refstack-client [8] is a command line utility that allows you to execute Tempest test runs based on configurations you specify. It is the official tool to run Openstack Interop (previously known as Defcore) testcases [9], which focus on testing interoperability between OpenStack clouds.

Refstack-client is integrated in Functest, consumed by Dovetail, which intends to define and provide a set of OPNFV related validation criteria that will provide input for the evaluation of the use of OPNFV trademarks. This progress is under the guideline of Compliance Verification Program(CVP).

Running methods

Two running methods are provided after refstack-client integrated into Functest, Functest command line and manually, respectively.

By default, for Defcore test cases run by Functest command line, are run followed with automatically generated configuration file, i.e., refstack_tempest.conf. In some circumstances, the automatic configuration file may not quite satisfied with the SUT, Functest also inherits the refstack-client command line and provides a way for users to set its configuration file according to its own SUT manually.

command line

Inside the Functest container, first to prepare Functest environment:

functest env prepare

then to run default defcore testcases by using refstack-client:

functest testcase run refstack_defcore

In OPNFV Continuous Integration(CI) system, the command line method is used.

manually

Prepare the tempest configuration file and the testcases want to run with the SUT, run the testcases with:

./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>

using help for more information:

./refstack-client --help
./refstack-client test --help
Reference tempest configuration

command line method

When command line method is used, the default tempest configuration file is generated by Rally.

manually

When running manually is used, recommended way to generate tempest configuration file is:

cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client
python tempest_conf.py

a file called tempest.conf is stored in the current path by default, users can do some adjustment according to the SUT:

vim refstack_tempest.conf

a reference article can be used [15].

snaps_smoke

This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config value snaps.use_floating_ips (True|False) to toggle this functionality. Please note that When the configuration value of snaps.use_keystone is True, Functest must have access the cloud’s private network. This suite consists in 38 tests (test duration < 10 minutes)

SDN Controllers
OpenDaylight

The OpenDaylight (ODL) test suite consists of a set of basic tests inherited from the ODL project using the Robot [11] framework. The suite verifies creation and deletion of networks, subnets and ports with OpenDaylight and Neutron.

The list of tests can be described as follows:

  • Basic Restconf test cases * Connect to Restconf URL * Check the HTTP code status
  • Neutron Reachability test cases * Get the complete list of neutron resources (networks, subnets, ports)
  • Neutron Network test cases * Check OpenStack networks * Check OpenDaylight networks * Create a new network via OpenStack and check the HTTP status code returned by Neutron * Check that the network has also been successfully created in OpenDaylight
  • Neutron Subnet test cases * Check OpenStack subnets * Check OpenDaylight subnets * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron * Check that the subnet has also been successfully created in OpenDaylight
  • Neutron Port test cases * Check OpenStack Neutron for known ports * Check OpenDaylight ports * Create a new port via OpenStack and check the HTTP status code returned by Neutron * Check that the new port has also been successfully created in OpenDaylight
  • Delete operations * Delete the port previously created via OpenStack * Check that the port has been also successfully deleted in OpenDaylight * Delete previously subnet created via OpenStack * Check that the subnet has also been successfully deleted in OpenDaylight * Delete the network created via OpenStack * Check that the network has also been successfully deleted in OpenDaylight

Note: the checks in OpenDaylight are based on the returned HTTP status code returned by OpenDaylight.

Features

Functest has been supporting several feature projects since Brahpamutra:

Test Brahma Colorado Danube Euphrates
barometer     X X
bgpvpn   X X X
copper   X    
doctor X X X X
domino   X X X
fds     X X
moon   X    
multisite   X X  
netready     X  
odl_sfc   X X X
opera     X  
orchestra     X X
parser     X X
promise X X X X
security_scan   X X  

Please refer to the dedicated feature user guides for details.

VNF
cloudify_ims

The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an architectural framework for delivering IP multimedia services.

vIMS has been integrated in Functest to demonstrate the capability to deploy a relatively complex NFV scenario on the OPNFV platform. The deployment of a complete functional VNF allows the test of most of the essential functions needed for a NFV platform.

The goal of this test suite consists of:

  • deploy a VNF orchestrator (Cloudify)
  • deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this orchestrator based on a TOSCA blueprint defined in [5]
  • run suite of signaling tests on top of this VNF

The Clearwater architecture is described as follows:

vIMS architecture
cloudify_ims_perf

This testcase extends the cloudify_ims test case. The first part is similar but the testing part is different. The testing part consists in automating a realistic signaling load on the vIMS using an Ixia loader (proprietary tools)

  • You need to have access to an Ixia licence server defined in the configuration

    file and have ixia image locally.

This test case is available but not declared in testcases.yaml. The declaration of the testcase is simple, connect to your functest-vnf docker, add the following section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:

-
    case_name: cloudify_ims_perf
    project_name: functest
    criteria: 80
    blocking: false
    description: >-
        Stress tests based on Cloudify. Ixia loader images and access to Ixia
        server license.
    dependencies:
        installer: ''
        scenario: 'os-nosdn-nofeature-ha'
    run:
        module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf'
        class: 'CloudifyImsPerf'
orchestra_openims

Orchestra test case deals with the deployment of OpenIMS with OpenBaton orchestrator.

orchestra_clearwaterims

Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton orchestrator.

vyos-vrouter

This test case deals with the deployment and the test of vyos vrouter with Cloudify orchestrator. The test case can do testing for interchangeability of BGP Protocol using vyos.

The Workflow is as follows:
  • Deploy

    Deploy VNF Testing topology by Cloudify using blueprint.

  • Configuration

    Setting configuration to Target VNF and reference VNF using ssh

  • Run

    Execution of test command for test item written YAML format file. Check VNF status and behavior.

  • Reporting

    Output of report based on result using JSON format.

The vyos-vrouter architecture is described in [14]

Executing Functest suites

As mentioned in the configuration guide [1], Alpine docker containers have been introduced in Euphrates. Tier containers have been created. Assuming that you pulled the container and your environement is ready, you can simply run the tiers by typing (e.g. with functest-healthcheck):

sudo docker run --env-file env \
    -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds  \
    -v $(pwd)/images:/home/opnfv/functest/images  \
    opnfv/functest-healthcheck

You should get:

+----------------------------+------------------+---------------------+------------------+----------------+
|         TEST CASE          |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+----------------------------+------------------+---------------------+------------------+----------------+
|      connection_check      |     functest     |     healthcheck     |      00:02       |      PASS      |
|         api_check          |     functest     |     healthcheck     |      03:19       |      PASS      |
|     snaps_health_check     |     functest     |     healthcheck     |      00:46       |      PASS      |
+----------------------------+------------------+---------------------+------------------+----------------+

You can run functest-healcheck, functest-smoke, functest-features, functest-components and functest-vnf.

The result tables show the results by test case, it can be:

* PASS
* FAIL
* SKIP: if the scenario/installer does not support the test case
Manual run

If you want to run the test step by step, you may add docker option then run the different commands within the docker.

Considering the healthcheck example, running functest manaully means:

sudo docker run -ti --env-file env \
  -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds  \
  -v $(pwd)/images:/home/opnfv/functest/images  \
  opnfv/functest-healthcheck /bin/bash

The docker prompt shall be returned. Then within the docker run the following commands:

$ source /home/opnfv/functest/conf/openstack.creds
Tier

Each Alpine container provided on the docker hub matches with a tier. The following commands are available:

# functest tier list
  - 0. healthcheck:
  ['connection_check', 'api_check', 'snaps_health_check']
# functest tier show healthcheck
+---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
|        TIERS        |     ORDER     |         CI LOOP          |                   DESCRIPTION                   |             TESTCASES              |
+---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
|     healthcheck     |       0       |     (daily)|(weekly)     |     First tier to be executed to verify the     |     connection_check api_check     |
|                     |               |                          |           basic operations in the VIM.          |         snaps_health_check         |
+---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+

To run all the cases of the tier, type:

# functest tier run healthcheck
Testcase

Testcases can be listed, shown and run though the CLI:

# functest testcase list
 connection_check
 api_check
 snaps_health_check
# functest testcase show api_check
+-------------------+--------------------------------------------------+------------------+---------------------------+
|     TEST CASE     |                   DESCRIPTION                    |     CRITERIA     |         DEPENDENCY        |
+-------------------+--------------------------------------------------+------------------+---------------------------+
|     api_check     |     This test case verifies the retrieval of     |       100        |     ^((?!netvirt).)*$     |
|                   |       OpenStack clients: Keystone, Glance,       |                  |                           |
|                   |      Neutron and Nova and may perform some       |                  |                           |
|                   |     simple queries. When the config value of     |                  |                           |
|                   |       snaps.use_keystone is True, functest       |                  |                           |
|                   |     must have access to the cloud's private      |                  |                           |
|                   |                     network.                     |                  |                           |
+-------------------+--------------------------------------------------+------------------+---------------------------+
# functest testcase run connection_check
...
# functest run all

You can also type run_tests -t all to run all the tests.

Note the list of test cases depend on the installer and the scenario.

Reporting results to the test Database

In OPNFV CI we collect all the results from CI. A test APi shall be available as well as a test database [17].

Functest internal API

An internal API has been introduced in Euphrates. The goal is to trigger Functest operations through an API in addition of the CLI. This could be considered as a first step towards a pseudo micro services approach where the different test projects could expose and consume APIs to the other test projects.

In Euphrates the main method of the APIs are:

  • Show credentials
  • Update openrc file
  • Show environment
  • Update hosts info for domain name
  • Prepare environment
  • List all testcases
  • Show a testcase
  • Run a testcase
  • List all tiers
  • Show a tier
  • List all testcases within given tier
  • Get the result of the specified task
  • Get the log of the specified task

See [16] to get examples on how to use the API.

Test results
Manual testing

In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results.

If you want additional logs, you may configure the logging.ini under /usr/lib/python2.7/site-packages/functest/ci.

Automated testing

In automated mode, test results are displayed in jenkins logs, a summary is provided at the end of the job and can be described as follow:

+-------------------------+----------------------------------------------------------+
|         ENV VAR         |                          VALUE                           |
+-------------------------+----------------------------------------------------------+
|      INSTALLER_TYPE     |                          daisy                           |
|     DEPLOY_SCENARIO     |                  os-nosdn-nofeature-ha                   |
|        BUILD_TAG        |     jenkins-functest-daisy-baremetal-daily-master-67     |
|         CI_LOOP         |                          daily                           |
+-------------------------+----------------------------------------------------------+

+------------------------------+------------------+---------------------+------------------+----------------+
|          TEST CASE           |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+------------------------------+------------------+---------------------+------------------+----------------+
|       connection_check       |     functest     |     healthcheck     |      00:08       |      PASS      |
|          api_check           |     functest     |     healthcheck     |      04:22       |      PASS      |
|      snaps_health_check      |     functest     |     healthcheck     |      00:35       |      PASS      |
|          vping_ssh           |     functest     |        smoke        |      00:54       |      PASS      |
|        vping_userdata        |     functest     |        smoke        |      00:27       |      PASS      |
|     tempest_smoke_serial     |     functest     |        smoke        |      19:39       |      FAIL      |
|         rally_sanity         |     functest     |        smoke        |      15:16       |      PASS      |
|       refstack_defcore       |     functest     |        smoke        |      15:55       |      PASS      |
|         snaps_smoke          |     functest     |        smoke        |      26:45       |      FAIL      |
|           cloudify_ims       |     functest     |         vnf         |      23:56       |      PASS      |
|        orchestra_openims     |     orchestra    |         vnf         |      15:07       |      PASS      |
|     orchestra_clearwaterims  |     orchestra    |         vnf         |      19:10       |      PASS      |
|           vyos_vrouter       |     functest     |         vnf         |      00:00       |      SKIP      |
+------------------------------+------------------+---------------------+------------------+----------------+

Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites.

Based on the results stored in the result database, a Functest reporting portal is also automatically updated. This portal provides information on the overall status per scenario and per installer

Test reporting

An automatic reporting page has been created in order to provide a consistent view of the Functest tests on the different scenarios.

In this page, each scenario is evaluated according to test criteria.

The results are collected from the centralized database every day and, per scenario. A score is calculated based on the results from the last 10 days. This score is the addition of single test scores. Each test case has a success criteria reflected in the criteria field from the results.

As an illustration, let’s consider the scenario os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the scores of all the runnable tests from the categories (tiers, healthcheck, smoke and features) corresponding to this scenario.

Test Apex Compass Fuel Joid
vPing_ssh X X X X
vPing_userdata X X X X
tempest_smoke_serial X X X X
rally_sanity X X X X
odl X X X X
promise     X X
doctor X   X  
security_scan X      
parser     X  
copper X     X

src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version)

All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature scenarios. Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration addons and, as a consequence, specific scenario. There are not considered as runnable on the generic odl_l2 scenario.

If no result is available or if all the results are failed, the test case get 0 point. If it was successful at least once but not anymore during the 4 runs, the case get 1 point (it worked once). If at least 3 of the last 4 runs were successful, the case get 2 points. If the last 4 runs of the test are successful, the test get 3 points.

In the example above, the target score for fuel/os-odl_l2-nofeature-ha is 3 x 8 = 24 points and for compass it is 3 x 5 = 15 points .

The scenario is validated per installer when we got 3 points for all individual test cases (e.g 24/24 for fuel, 15/15 for compass).

Please note that complex or long duration tests are not considered yet for the scoring. In fact the success criteria are not always easy to define and may require specific hardware configuration.

Please also note that all the test cases have the same “weight” for the score calculation whatever the complexity of the test case. Concretely a vping has the same weith than the 200 tempst tests. Moreover some installers support more features than others. The more cases your scenario is dealing with, the most difficult to rich a good scoring.

Therefore the scoring provides 3 types of indicators:

  • the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features
  • the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS
  • the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI)

In any case, the scoring is used to give feedback to the other projects and does not represent an absolute value of the scenario.

See reporting page for details. For the status, click on the version, Functest then the Status menu.

Functest reporting portal Fuel status page
Troubleshooting

This section gives some guidelines about how to troubleshoot the test cases owned by Functest.

IMPORTANT: As in the previous section, the steps defined below must be executed inside the Functest Docker container and after sourcing the OpenStack credentials:

. $creds

or:

source /home/opnfv/functest/conf/openstack.creds
VIM

This section covers the test cases related to the VIM (healthcheck, vping_ssh, vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, rally_full).

vPing common

For both vPing test cases (vPing_ssh, and vPing_userdata), the first steps are similar:

  • Create Glance image
  • Create Network
  • Create Security Group
  • Create Instances

After these actions, the test cases differ and will be explained in their respective section.

These test cases can be run inside the container, using new Functest CLI as follows:

$ functest testcase run vping_ssh
$ functest testcase run vping_userdata

The Functest CLI is designed to route a call to the corresponding internal python scripts, located in paths /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py

Notes:

  1. There is one difference, between the Functest CLI based test case execution compared to the earlier used Bash shell script, which is relevant to point out in troubleshooting scenarios:

    The Functest CLI does not yet support the option to suppress clean-up of the generated OpenStack resources, following the execution of a test case.

    Explanation: After finishing the test execution, the corresponding script will remove, by default, all created resources in OpenStack (image, instances, network and security group). When troubleshooting, it is advisable sometimes to keep those resources in case the test fails and a manual testing is needed.

    It is actually still possible to invoke test execution, with suppression of OpenStack resource cleanup, however this requires invocation of a specific Python script: ‘run_tests’. The OPNFV Functest Developer Guide provides guidance on the use of that Python script in such troubleshooting cases.

Some of the common errors that can appear in this test case are:

vPing_ssh- ERROR - There has been a problem when creating the neutron network....

This means that there has been some problems with Neutron, even before creating the instances. Try to create manually a Neutron network and a Subnet to see if that works. The debug messages will also help to see when it failed (subnet and router creation). Example of Neutron commands (using 10.6.0.0/24 range for example):

neutron net-create net-test
neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \
--gateway 10.6.0.254 net-test 10.6.0.0/24
neutron router-create test_router
neutron router-interface-add <ROUTER_ID> test_subnet
neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME>

Another related error can occur while creating the Security Groups for the instances:

vPing_ssh- ERROR - Failed to create the security group...

In this case, proceed to create it manually. These are some hints:

neutron security-group-create sg-test
neutron security-group-rule-create sg-test --direction ingress --protocol icmp \
--remote-ip-prefix 0.0.0.0/0
neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \
--protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \
--protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0

The next step is to create the instances. The image used is located in /home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img and a Glance image is created with the name functest-vping. If booting the instances fails (i.e. the status is not ACTIVE), you can check why it failed by doing:

nova list
nova show <INSTANCE_ID>

It might show some messages about the booting failure. To try that manually:

nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test

This will spawn a VM using the network created previously manually. In all the OPNFV tested scenarios from CI, it never has been a problem with the previous actions. Further possible problems are explained in the following sections.

vPing_SSH

This test case creates a floating IP on the external network and assigns it to the second instance opnfv-vping-2. The purpose of this is to establish a SSH connection to that instance and SCP a script that will ping the first instance. This script is located in the repository under /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh and takes an IP as a parameter. When the SCP is completed, the test will do a SSH call to that script inside the second instance. Some problems can happen here:

vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting

If this is displayed, stop the test or wait for it to finish, if you have used the special method of test invocation with specific supression of OpenStack resource clean-up, as explained earler. It means that the Container can not reach the Public/External IP assigned to the instance opnfv-vping-2. There are many possible reasons, and they really depend on the chosen scenario. For most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known limitation.

First, make sure that the instance opnfv-vping-2 succeeded to get an IP from the DHCP agent. It can be checked by doing:

nova console-log opnfv-vping-2

If the message Sending discover and No lease, failing is shown, it probably means that the Neutron dhcp-agent failed to assign an IP or even that it was not responding. At this point it does not make sense to try to ping the floating IP.

If the instance got an IP properly, try to ping manually the VM from the container:

nova list
<grab the public IP>
ping <public IP>

If the ping does not return anything, try to ping from the Host where the Docker container is running. If that solves the problem, check the iptable rules because there might be some rules rejecting ICMP or TCP traffic coming/going from/to the container.

At this point, if the ping does not work either, try to reproduce the test manually with the steps described above in the vPing common section with the addition:

neutron floatingip-create <EXT_NET_NAME>
nova floating-ip-associate nova-test <FLOATING_IP>

Further troubleshooting is out of scope of this document, as it might be due to problems with the SDN controller. Contact the installer team members or send an email to the corresponding OPNFV mailing list for more information.

vPing_userdata

This test case does not create any floating IP neither establishes an SSH connection. Instead, it uses nova-metadata service when creating an instance to pass the same script as before (ping.sh) but as 1-line text. This script will be executed automatically when the second instance opnfv-vping-2 is booted.

The only known problem here for this test to fail is mainly the lack of support of cloud-init (nova-metadata service). Check the console of the instance:

nova console-log opnfv-vping-2

If this text or similar is shown:

checking http://169.254.169.254/2009-04-04/instance-id
failed 1/20: up 1.13. request failed
failed 2/20: up 13.18. request failed
failed 3/20: up 25.20. request failed
failed 4/20: up 37.23. request failed
failed 5/20: up 49.25. request failed
failed 6/20: up 61.27. request failed
failed 7/20: up 73.29. request failed
failed 8/20: up 85.32. request failed
failed 9/20: up 97.34. request failed
failed 10/20: up 109.36. request failed
failed 11/20: up 121.38. request failed
failed 12/20: up 133.40. request failed
failed 13/20: up 145.43. request failed
failed 14/20: up 157.45. request failed
failed 15/20: up 169.48. request failed
failed 16/20: up 181.50. request failed
failed 17/20: up 193.52. request failed
failed 18/20: up 205.54. request failed
failed 19/20: up 217.56. request failed
failed 20/20: up 229.58. request failed
failed to read iid from metadata. tried 20

it means that the instance failed to read from the metadata service. Contact the Functest or installer teams for more information.

Tempest

In the upstream OpenStack CI all the Tempest test cases are supposed to pass. If some test cases fail in an OPNFV deployment, the reason is very probably one of the following

Error Details
Resources required for test case execution are missing Such resources could be e.g. an external network and access to the management subnet (adminURL) from the Functest docker container.
OpenStack components or services are missing or not configured properly Check running services in the controller and compute nodes (e.g. with “systemctl” or “service” commands). Configuration parameters can be verified from the related .conf files located under ‘/etc/<component>’ directories.
Some resources required for execution test cases are missing The tempest.conf file, automatically generated by Rally in Functest, does not contain all the needed parameters or some parameters are not set properly. The tempest.conf file is located in directory ‘root/.rally/verification/verifier-<UUID> /for-deployment-<UUID>’ in the Functest Docker container. Use the “rally deployment list” command in order to check the UUID the UUID of the current deployment.

When some Tempest test case fails, captured traceback and possibly also the related REST API requests/responses are output to the console. More detailed debug information can be found from tempest.log file stored into related Rally deployment folder.

Functest offers a possibility to test a customized list of Tempest test cases. To enable that, add a new entry in docker/components/testcases.yaml on the “components” container with the following content:

-
    case_name: tempest_custom
    project_name: functest
    criteria: 100
    blocking: false
    description: >-
        The test case allows running a customized list of tempest
        test cases
    dependencies:
        installer: ''
        scenario: ''
    run:
        module: 'functest.opnfv_tests.openstack.tempest.tempest'
        class: 'TempestCustom'

Also, a list of the Tempest test cases must be provided to the container or modify the existing one in /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt

Example of custom list of tests ‘my-custom-tempest-tests.txt’:

tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke]
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]

This is an example of running a customized list of Tempest tests in Functest:

sudo docker run --env-file env \
    -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \
    -v $(pwd)/images:/home/opnfv/functest/images \
    -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \
    -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \
    opnfv/functest-components run_tests -t tempest_custom
Rally

The same error causes which were mentioned above for Tempest test cases, may also lead to errors in Rally as well.

Possible scenarios are:
  • authenticate
  • glance
  • cinder
  • heat
  • keystone
  • neutron
  • nova
  • ceilometer
  • quotas
  • vm

To know more about what those scenarios are doing, they are defined in directory: /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario For more info about Rally scenario definition please refer to the Rally official documentation. [3]

To check any possible problems with Rally, the logs are stored under /home/opnfv/functest/results/rally/ in the Functest Docker container.

Controllers
Opendaylight

If the Basic Restconf test suite fails, check that the ODL controller is reachable and its Restconf module has been installed.

If the Neutron Reachability test fails, verify that the modules implementing Neutron requirements have been properly installed.

If any of the other test cases fails, check that Neutron and ODL have been correctly configured to work together. Check Neutron configuration files, accounts, IP addresses etc.).

Features

Please refer to the dedicated feature user guides for details.

VNF
cloudify_ims

vIMS deployment may fail for several reasons, the most frequent ones are described in the following table:

Error Comments
Keystone admin API not reachable Impossible to create vIMS user and tenant
Impossible to retrieve admin role id Impossible to create vIMS user and tenant
Error when uploading image from OpenStack to glance impossible to deploy VNF
Cinder quota cannot be updated Default quotas not sufficient, they are adapted in the script
Impossible to create a volume VNF cannot be deployed
SSH connection issue between the Test Docker container and the VM if vPing test fails, vIMS test will fail...
No Internet access from the VM the VMs of the VNF must have an external access to Internet
No access to OpenStack API from the VM Orchestrator can be installed but the vIMS VNF installation fails

Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it is there fore not recommended to run it on a light configuration.

References

[1]: Functest configuration guide

[2]: OpenStack Tempest documentation

[3]: Rally documentation

[4]: Functest in depth (Danube)

[5]: Clearwater vIMS blueprint

[6]: NIST web site

[7]: OpenSCAP web site

[8]: Refstack client

[9]: Defcore

[10]: OpenStack interoperability procedure

[11]: Robot Framework web site

[12]: Functest User guide

[13]: SNAPS wiki

[14]: vRouter

[15]: Testing OpenStack Tempest part 1

[16]: Running Functest through internal REST API

[17]: OPNFV Test API

OPNFV main site: OPNFV official web site

Functest page: Functest wiki page

IRC support chan: #opnfv-functest

NFVbench

NFVbench User Guide

The NFVbench tool provides an automated way to measure the network performance for the most common data plane packet flows on any OpenStack system. It is designed to be easy to install and easy to use by non experts (no need to be an expert in traffic generators and data plane performance testing).

Table of Content
Features
Data Plane Performance Measurement Features

NFVbench supports the following main measurement capabilities:

  • supports 2 measurement modes:
    • fixed rate mode to generate traffic at a fixed rate for a fixed duration
    • NDR (No Drop Rate) and PDR (Partial Drop Rate) measurement mode
  • configurable frame sizes (any list of fixed sizes or ‘IMIX’)

  • built-in packet paths (PVP, PVVP)

  • built-in loopback VNFs based on fast L2 or L3 forwarders running in VMs

  • configurable number of flows and service chains

  • configurable traffic direction (single or bi-directional)

NDR is the highest throughput achieved without dropping packets. PDR is the highest throughput achieved without dropping more than a pre-set limit (called PDR threshold or allowance, expressed in %).

Results of each run include the following data:

  • Aggregated achieved throughput in bps
  • Aggregated achieved packet rate in pps (or fps)
  • Actual drop rate in %
  • Latency in usec (min, max, average in the current version)
Built-in OpenStack support

NFVbench can stage OpenStack resources to build 1 or more service chains using direct OpenStack APIs. Each service chain is composed of:

  • 1 or 2 loopback VM instances per service chain
  • 2 Neutron networks per loopback VM

OpenStack resources are staged before traffic is measured using OpenStack APIs (Nova and Neutron) then disposed after completion of measurements.

The loopback VM flavor to use can be configured in the NFVbench configuration file.

Note that NFVbench does not use OpenStack Heat nor any higher level service (VNFM or NFVO) to create the service chains because its main purpose is to measure the performance of the NFVi infrastructure which is mainly focused on L2 forwarding performance.

External Chains

NFVbench also supports settings that involve externally staged packet paths with or without OpenStack:

  • run benchmarks on existing service chains at the L3 level that are staged externally by any other tool (e.g. any VNF capable of L3 routing)
  • run benchmarks on existing L2 chains that are configured externally (e.g. pure L2 forwarder such as DPDK testpmd)
Traffic Generation

NFVbench currently integrates with the open source TRex traffic generator:

  • TRex (pre-built into the NFVbench container)
Supported Packet Paths

Packet paths describe where packets are flowing in the NFVi platform. The most commonly used paths are identified by 3 or 4 letter abbreviations. A packet path can generally describe the flow of packets associated to one or more service chains, with each service chain composed of 1 or more VNFs.

The following packet paths are currently supported by NFVbench:

  • PVP (Physical interface to VM to Physical interface)
  • PVVP (Physical interface to VM to VM to Physical interface)
  • N*PVP (N concurrent PVP packet paths)
  • N*PVVP (N concurrent PVVP packet paths)

The traffic is made of 1 or more flows of L3 frames (UDP packets) with different payload sizes. Each flow is identified by a unique source and destination MAC/IP tuple.

Loopback VM

NFVbench provides a loopback VM image that runs CentOS with 2 pre-installed forwarders:

  • DPDK testpmd configured to do L2 cross connect between 2 virtual interfaces
  • FD.io VPP configured to perform L3 routing between 2 virtual interfaces

Frames are just forwarded from one interface to the other. In the case of testpmd, the source and destination MAC are rewritten, which corresponds to the mac forwarding mode (–forward-mode=mac). In the case of VPP, VPP will act as a real L3 router, and the packets are routed from one port to the other using static routes.

Which forwarder and what Nova flavor to use can be selected in the NFVbench configuration. Be default the DPDK testpmd forwarder is used with 2 vCPU per VM. The configuration of these forwarders (such as MAC rewrite configuration or static route configuration) is managed by NFVbench.

PVP Packet Path

This packet path represents a single service chain with 1 loopback VNF and 2 Neutron networks:

_images/nfvbench-pvp.png
PVVP Packet Path

This packet path represents a single service chain with 2 loopback VNFs in sequence and 3 Neutron networks. The 2 VNFs can run on the same compute node (PVVP intra-node):

_images/nfvbench-pvvp.png

or on different compute nodes (PVVP inter-node) based on a configuration option:

_images/nfvbench-pvvp2.png
Multi-Chaining (N*PVP or N*PVVP)

Multiple service chains can be setup by NFVbench without any limit on the concurrency (other than limits imposed by available resources on compute nodes). In the case of multiple service chains, NFVbench will instruct the traffic generator to use multiple L3 packet streams (frames directed to each path will have a unique destination MAC address).

Example of multi-chaining with 2 concurrent PVP service chains:

_images/nfvbench-npvp.png

This innovative feature will allow to measure easily the performance of a fully loaded compute node running multiple service chains.

Multi-chaining is currently limited to 1 compute node (PVP or PVVP intra-node) or 2 compute nodes (for PVVP inter-node). The 2 edge interfaces for all service chains will share the same 2 networks.

SR-IOV

By default, service chains will be based on virtual switch interfaces.

NFVbench provides an option to select SR-IOV based virtual interfaces instead (thus bypassing any virtual switch) for those OpenStack system that include and support SR-IOV capable NICs on compute nodes.

The PVP packet path will bypass the virtual switch completely when the SR-IOV option is selected:

_images/nfvbench-sriov-pvp.png

The PVVP packet path will use SR-IOV for the left and right networks and the virtual switch for the middle network by default:

_images/nfvbench-sriov-pvvp.png

Or in the case of inter-node:

_images/nfvbench-sriov-pvvp2.png

This packet path is a good way to approximate VM to VM (V2V) performance (middle network) given the high efficiency of the left and right networks. The V2V throughput will likely be very close to the PVVP throughput while its latency will be very close to the difference between the SR-IOV PVVP latency and the SR-IOV PVP latency.

It is possible to also force the middle network to use SR-IOV (in this version, the middle network is limited to use the same SR-IOV phys net):

_images/nfvbench-all-sriov-pvvp.png

The chain can also span across 2 nodes with the use of 2 SR-IOV ports in each node:

_images/nfvbench-all-sriov-pvvp2.png
Other Misc Packet Paths

P2P (Physical interface to Physical interface - no VM) can be supported using the external chain/L2 forwarding mode.

V2V (VM to VM) is not supported but PVVP provides a more complete (and more realistic) alternative.

Supported Neutron Network Plugins and vswitches
Any Virtual Switch, Any Encapsulation

NFVbench is agnostic of the virtual switch implementation and has been tested with the following virtual switches:

  • ML2/VPP/VLAN (networking-vpp)
  • OVS/VLAN and OVS-DPDK/VLAN
  • ML2/ODL/VPP (OPNFV Fast Data Stack)
Installation and Quick Start Guides
Requirements for running NFVbench
Hardware Requirements

To run NFVbench you need the following hardware: - a Linux server - a DPDK compatible NIC with at least 2 ports (preferably 10Gbps or higher) - 2 ethernet cables between the NIC and the OpenStack pod under test (usually through a top of rack switch)

The DPDK-compliant NIC must be one supported by the TRex traffic generator (such as Intel X710, refer to the Trex Installation Guide for a complete list of supported NIC)

To run the TRex traffic generator (that is bundled with NFVbench) you will need to wire 2 physical interfaces of the NIC to the TOR switch(es):
  • if you have only 1 TOR, wire both interfaces to that same TOR
  • 1 interface to each TOR if you have 2 TORs and want to use bonded links to your compute nodes
_images/nfvbench-trex-setup.png
Switch Configuration

The 2 corresponding ports on the switch(es) facing the Trex ports on the Linux server should be configured in trunk mode (NFVbench will instruct TRex to insert the appropriate vlan tag).

Using a TOR switch is more representative of a real deployment and allows to measure packet flows on any compute node in the rack without rewiring and includes the overhead of the TOR switch.

Although not the primary targeted use case, NFVbench could also support the direct wiring of the traffic generator to a compute node without a switch (although that will limit some of the features that invove multiple compute nodes in the packet path).

Software Requirements

You need Docker to be installed on the Linux server.

TRex uses the DPDK interface to interact with the DPDK compatible NIC for sending and receiving frames. The Linux server will need to be configured properly to enable DPDK.

DPDK requires a uio (User space I/O) or vfio (Virtual Function I/O) kernel module to be installed on the host to work. There are 2 main uio kernel modules implementations (igb_uio and uio_pci_generic) and one vfio kernel module implementation.

To check if a uio or vfio is already loaded on the host:

lsmod | grep -e igb_uio -e uio_pci_generic -e vfio

If missing, it is necessary to install a uio/vfio kernel module on the host server:

  • find a suitable kernel module for your host server (any uio or vfio kernel module built with the same Linux kernel version should work)
  • load it using the modprobe and insmod commands

Example of installation of the igb_uio kernel module:

modprobe uio
insmod ./igb_uio.ko

Finally, the correct iommu options and huge pages to be configured on the Linux server on the boot command line:

  • enable intel_iommu and iommu pass through: “intel_iommu=on iommu=pt”
  • for Trex, pre-allocate 1024 huge pages of 2MB each (for a total of 2GB): “hugepagesz=2M hugepages=1024”

More detailed instructions can be found in the DPDK documentation (https://media.readthedocs.org/pdf/dpdk/latest/dpdk.pdf).

NFVbench Installation and Quick Start Guide

Make sure you satisfy the hardware and software requirements <requirements> before you start .

1. Container installation

To pull the latest NFVbench container image:

docker pull opnfv/nfvbench
2. Docker Container configuration

The NFVbench container requires the following Docker options to operate properly.

Docker options Description
-v /lib/modules/$(uname -r):/lib/modules/$(uname -r) needed by kernel modules in the container
-v /dev:/dev needed by kernel modules in the container
-v $PWD:/tmp/nfvbench optional but recommended to pass files between the host and the docker space (see examples below) Here we map the current directory on the host to the /tmp/nfvbench director in the container but any other similar mapping can work as well
–net=host (optional) needed if you run the NFVbench ok server in the container (or use any appropriate docker network mode other than “host”)
–privileged (optional) required if SELinux is enabled on the host
-e HOST=”127.0.0.1” (optional) required if REST server is enabled
-e PORT=7556 (optional) required if REST server is enabled
-e CONFIG_FILE=”/root/nfvbenchconfig.json (optional) required if REST server is enabled

It can be convenient to write a shell script (or an alias) to automatically insert the necessary options.

The minimal configuration file required must specify the openrc file to use (using in-container path), the PCI addresses of the 2 NIC ports to use for generating traffic and the line rate (in each direction) of each of these 2 interfaces.

Here is an example of mimimal configuration where: the openrc file is located on the host current directory which is mapped under /tmp/nfvbench in the container (this is achieved using -v $PWD:/tmp/nfvbench) the 2 NIC ports to use for generating traffic have the PCI addresses “04:00.0” and “04:00.1”

{
    "openrc_file": "/tmp/nfvbench/openrc",
    "traffic_generator": {
        "generator_profile": [
            {
                "interfaces": [
                    {
                        "pci": "04:00.0",
                        "port": 0,
                    },
                    {
                        "pci": "04:00.1",
                        "port": 1,
                    }
                ],
                "intf_speed": "10Gbps",
                "ip": "127.0.0.1",
                "name": "trex-local",
                "tool": "TRex"
            }
        ]
    }
}

The other options in the minimal configuration must be present and must have the same values as above.

3. Start the Docker container

As for any Docker container, you can execute NFVbench measurement sessions using a temporary container (“docker run” - which exits after each NFVbench run) or you can decide to run the NFVbench container in the background then execute one or more NFVbench measurement sessions on that container (“docker exec”).

The former approach is simpler to manage (since each container is started and terminated after each command) but incurs a small delay at start time (several seconds). The second approach is more responsive as the delay is only incurred once when starting the container.

We will take the second approach and start the NFVbench container in detached mode with the name “nfvbench” (this works with bash, prefix with “sudo” if you do not use the root login)

To run NFVBench without server mode

docker run --detach --net=host --privileged -v $PWD:/tmp/nfvbench -v /dev:/dev -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) --name nfvbench opnfv/nfvbench

To run NFVBench enabling REST server (mount the configuration json and the path for openrc)

docker run --detach --net=host --privileged -e HOST="127.0.0.1" -e PORT=7556 --e CONFIG_FILE="/tmp/nfvbench/nfvbenchconfig.json -v $PWD:/tmp/nfvbench -v /dev:/dev -v /lib/modules/$(uname -r):/lib/modules/$(uname -r) --name nfvbench opnfv/nfvbench start_rest_server

The create an alias to make it easy to execute nfvbench commands directly from the host shell prompt:

alias nfvbench='docker exec -it nfvbench nfvbench'

The next to last “nfvbench” refers to the name of the container while the last “nfvbench” refers to the NFVbench binary that is available to run in the container.

To verify it is working:

nfvbench --version
nfvbench --help
4. NFVbench configuration

Create a new file containing the minimal configuration for NFVbench, we can call it any name, for example “my_nfvbench.cfg” and paste the following yaml template in the file:

openrc_file:
traffic_generator:
    generator_profile:
        - name: trex-local
          tool: TRex
          ip: 127.0.0.1
          cores: 3
          interfaces:
            - port: 0
              switch_port:
              pci:
            - port: 1
              switch_port:
              pci:
          intf_speed: 10Gbps

NFVbench requires an openrc file to connect to OpenStack using the OpenStack API. This file can be downloaded from the OpenStack Horizon dashboard (refer to the OpenStack documentation on how to retrieve the openrc file). The file pathname in the container must be stored in the “openrc_file” property. If it is stored on the host in the current directory, its full pathname must start with /tmp/nfvbench (since the current directory is mapped to /tmp/nfvbench in the container).

The required configuration is the PCI address of the 2 physical interfaces that will be used by the traffic generator. The PCI address can be obtained for example by using the “lspci” Linux command. For example:

[root@sjc04-pod6-build ~]# lspci | grep 710
0a:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0a:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0a:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
0a:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

Example of edited configuration with an OpenStack RC file stored in the current directory with the “openrc” name, and PCI addresses “0a:00.0” and “0a:00.1” (first 2 ports of the quad port NIC):

openrc_file: /tmp/nfvbench/openrc
traffic_generator:
    generator_profile:
        - name: trex-local
          tool: TRex
          ip: 127.0.0.1
          cores: 3
          interfaces:
            - port: 0
              switch_port:
              pci: 0a:00.0
            - port: 1
              switch_port:
              pci: 0a:00.1
          intf_speed: 10Gbps

Alternatively, the full template with comments can be obtained using the –show-default-config option in yaml format:

nfvbench --show-default-config > my_nfvbench.cfg

Edit the nfvbench.cfg file to only keep those properties that need to be modified (preserving the nesting)

5. Run NFVbench

To do a single run at 10,000pps bi-directional (or 5kpps in each direction) using the PVP packet path:

nfvbench -c /tmp/nfvbench/my_nfvbench.cfg --rate 10kpps

NFVbench options used:

  • -c /tmp/nfvbench/my_nfvbench.cfg : specify the config file to use (this must reflect the file path from inside the container)
  • --rate 10kpps : specify rate of packets for test for both directions using the kpps unit (thousands of packets per second)

This should produce a result similar to this (a simple run with the above options should take less than 5 minutes):

[TBP]
7. Terminating the NFVbench container

When no longer needed, the container can be terminated using the usual docker commands:

docker kill nfvbench
docker rm nfvbench
Example of Results

Example run for fixed rate

nfvbench -c /nfvbench/nfvbenchconfig.json --rate 1%
========== NFVBench Summary ==========
Date: 2017-09-21 23:57:44
NFVBench version 1.0.9
Openstack Neutron:
  vSwitch: BASIC
  Encapsulation: BASIC
Benchmarks:
> Networks:
  > Components:
    > TOR:
        Type: None
    > Traffic Generator:
        Profile: trex-local
        Tool: TRex
    > Versions:
      > TOR:
      > Traffic Generator:
          build_date: Aug 30 2017
          version: v2.29
          built_by: hhaim
          build_time: 16:43:55
  > Service chain:
    > PVP:
      > Traffic:
          Profile: traffic_profile_64B
          Bidirectional: True
          Flow count: 10000
          Service chains count: 1
          Compute nodes: []

            Run Summary:

              +-----------------+-------------+----------------------+----------------------+----------------------+
              |   L2 Frame Size |  Drop Rate  |   Avg Latency (usec) |   Min Latency (usec) |   Max Latency (usec) |
              +=================+=============+======================+======================+======================+
              |              64 |   0.0000%   |                   53 |                   20 |                  211 |
              +-----------------+-------------+----------------------+----------------------+----------------------+


            L2 frame size: 64
            Chain analysis duration: 60.076 seconds

            Run Config:

              +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
              |  Direction  |  Requested TX Rate (bps)  |  Actual TX Rate (bps)  |  RX Rate (bps)  |  Requested TX Rate (pps)  |  Actual TX Rate (pps)  |  RX Rate (pps)  |
              +=============+===========================+========================+=================+===========================+========================+=================+
              |   Forward   |       100.0000 Mbps       |      95.4546 Mbps      |  95.4546 Mbps   |        148,809 pps        |      142,045 pps       |   142,045 pps   |
              +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
              |   Reverse   |       100.0000 Mbps       |      95.4546 Mbps      |  95.4546 Mbps   |        148,809 pps        |      142,045 pps       |   142,045 pps   |
              +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
              |    Total    |       200.0000 Mbps       |     190.9091 Mbps      |  190.9091 Mbps  |        297,618 pps        |      284,090 pps       |   284,090 pps   |
              +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+

            Chain Analysis:

              +-------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
              |     Interface     |  Device  |  Packets (fwd)  |   Drops (fwd) |  Drop% (fwd)  |  Packets (rev)  |   Drops (rev) |  Drop% (rev)  |
              +===================+==========+=================+===============+===============+=================+===============+===============+
              | traffic-generator |   trex   |    8,522,729    |               |               |    8,522,729    |             0 |    0.0000%    |
              +-------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+
              | traffic-generator |   trex   |    8,522,729    |             0 |    0.0000%    |    8,522,729    |               |               |
              +-------------------+----------+-----------------+---------------+---------------+-----------------+---------------+---------------+

Example run for NDR/PDR with package size 1518B

nfvbench -c /nfvbench/nfvbenchconfig.json -fs 1518
========== NFVBench Summary ==========
Date: 2017-09-22 00:02:07
NFVBench version 1.0.9
Openstack Neutron:
  vSwitch: BASIC
  Encapsulation: BASIC
Benchmarks:
> Networks:
  > Components:
    > TOR:
        Type: None
    > Traffic Generator:
        Profile: trex-local
        Tool: TRex
    > Versions:
      > TOR:
      > Traffic Generator:
          build_date: Aug 30 2017
          version: v2.29
          built_by: hhaim
          build_time: 16:43:55
  > Measurement Parameters:
      NDR: 0.001
      PDR: 0.1
  > Service chain:
    > PVP:
      > Traffic:
          Profile: custom_traffic_profile
          Bidirectional: True
          Flow count: 10000
          Service chains count: 1
          Compute nodes: []

            Run Summary:

              +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+
              |  -  |   L2 Frame Size |  Rate (fwd+rev)  |  Rate (fwd+rev)  |  Avg Drop Rate  |   Avg Latency (usec) |   Min Latency (usec) |  Max Latency (usec)  |
              +=====+=================+==================+==================+=================+======================+======================+======================+
              | NDR |            1518 |   19.9805 Gbps   |  1,623,900 pps   |     0.0001%     |                  342 |                   30 |         704          |
              +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+
              | PDR |            1518 |   20.0000 Gbps   |  1,625,486 pps   |     0.0022%     |                  469 |                   40 |        1,266         |
              +-----+-----------------+------------------+------------------+-----------------+----------------------+----------------------+----------------------+


            L2 frame size: 1518
            Chain analysis duration: 660.442 seconds
            NDR search duration: 660 seconds
            PDR search duration: 0 seconds
Advanced Usage

This section covers a few examples on how to run NFVbench with multiple different settings. Below are shown the most common and useful use-cases and explained some fields from a default config file.

How to change any NFVbench run configuration (CLI)

NFVbench always starts with a default configuration which can further be refined (overridden) by the user from the CLI or from REST requests.

At first have a look at the default config:

nfvbench --show-default-config

It is sometimes useful derive your own configuration from a copy of the default config:

nfvbench --show-default-config > nfvbench.cfg

At this point you can edit the copy by:

  • removing any parameter that is not to be changed (since NFVbench will always load the default configuration, default values are not needed)
  • edit the parameters that are to be changed changed

A run with the new confguration can then simply be requested using the -c option and by using the actual path of the configuration file as seen from inside the container (in this example, we assume the current directory is mapped to /tmp/nfvbench in the container):

nfvbench -c /tmp/nfvbench/nfvbench.cfg

The same -c option also accepts any valid yaml or json string to override certain parameters without having to create a configuration file.

NFVbench provides many configuration options as optional arguments. For example the number of flows can be specified using the –flow-count option.

The flow count option can be specified in any of 3 ways:

  • by providing a confguration file that has the flow_count value to use (-c myconfig.yaml and myconfig.yaml contains ‘flow_count: 100k’)
  • by passing that yaml paremeter inline (-c “flow_count: 100k”) or (-c “{flow_count: 100k}”)
  • by using the flow count optional argument (–flow-count 100k)
Showing the running configuration

Because configuration parameters can be overriden, it is sometimes useful to show the final configuration (after all oevrrides are done) by using the –show-config option. This final configuration is also called the “running” configuration.

For example, this will only display the running configuration (without actually running anything):

nfvbench -c "{flow_count: 100k, debug: true}" --show-config
Connectivity and Configuration Check

NFVbench allows to test connectivity to devices used with the selected packet path. It runs the whole test, but without actually sending any traffic. It is also a good way to check if everything is configured properly in the configuration file and what versions of components are used.

To verify everything works without sending any traffic, use the –no-traffic option:

nfvbench --no-traffic

Used parameters:

  • --no-traffic or -0 : sending traffic from traffic generator is skipped
Fixed Rate Run

Fixed rate run is the most basic type of NFVbench usage. It can be used to measure the drop rate with a fixed transmission rate of packets.

This example shows how to run the PVP packet path (which is the default packet path) with multiple different settings:

nfvbench -c nfvbench.cfg --no-cleanup --rate 100000pps --duration 30 --interval 15 --json results.json

Used parameters:

  • -c nfvbench.cfg : path to the config file
  • --no-cleanup : resources (networks, VMs, attached ports) are not deleted after test is finished
  • --rate 100000pps : defines rate of packets sent by traffic generator
  • --duration 30 : specifies how long should traffic be running in seconds
  • --interval 15 : stats are checked and shown periodically (in seconds) in this interval when traffic is flowing
  • --json results.json : collected data are stored in this file after run is finished

Note

It is your responsibility to clean up resources if needed when --no-cleanup parameter is used. You can use the nfvbench_cleanup helper script for that purpose.

The --json parameter makes it easy to store NFVbench results. The –show-summary (or -ss) option can be used to display the results in a json results file in a text tabular format:

nfvbench --show-summary results.json

This example shows how to specify a different packet path:

nfvbench -c nfvbench.cfg --rate 1Mbps --inter-node --service-chain PVVP

Used parameters:

  • -c nfvbench.cfg : path to the config file
  • --rate 1Mbps : defines rate of packets sent by traffic generator
  • --inter-node : VMs are created on different compute nodes, works only with PVVP flow
  • --service-chain PVVP or -sc PVVP : specifies the type of service chain (or packet path) to use

Note

When parameter --inter-node is not used or there aren’t enough compute nodes, VMs are on the same compute node.

Rate Units

Parameter --rate accepts different types of values:

  • packets per second (pps, kpps, mpps), e.g. 1000pps or 10kpps
  • load percentage (%), e.g. 50%
  • bits per second (bps, kbps, Mbps, Gbps), e.g. 1Gbps, 1000bps
  • NDR/PDR (ndr, pdr, ndr_pdr), e.g. ndr_pdr

NDR/PDR is the default rate when not specified.

NDR and PDR

The NDR and PDR test is used to determine the maximum throughput performance of the system under test following guidelines defined in RFC-2544:

  • NDR (No Drop Rate): maximum packet rate sent without dropping any packet
  • PDR (Partial Drop Rate): maximum packet rate sent while allowing a given maximum drop rate

The NDR search can also be relaxed to allow some very small amount of drop rate (lower than the PDR maximum drop rate). NFVbench will measure the NDR and PDR values by driving the traffic generator through multiple iterations at different transmission rates using a binary search algorithm.

The configuration file contains section where settings for NDR/PDR can be set.

# NDR/PDR configuration
measurement:
    # Drop rates represent the ratio of dropped packet to the total number of packets sent.
    # Values provided here are percentages. A value of 0.01 means that at most 0.01% of all
    # packets sent are dropped (or 1 packet every 10,000 packets sent)

    # No Drop Rate; Default to 0.001%
    NDR: 0.001
    # Partial Drop Rate; NDR should always be less than PDR
    PDR: 0.1
    # The accuracy of NDR and PDR load percentiles; The actual load percentile that match NDR
    # or PDR should be within `load_epsilon` difference than the one calculated.
    load_epsilon: 0.1

Because NDR/PDR is the default --rate value, it is possible to run NFVbench simply like this:

nfvbench -c nfvbench.cfg

Other possible run options:

nfvbench -c nfvbench.cfg --duration 120 --json results.json

Used parameters:

  • -c nfvbench.cfg : path to the config file
  • --duration 120 : specifies how long should be traffic running in each iteration
  • --json results.json : collected data are stored in this file after run is finished
Multichain

NFVbench allows to run multiple chains at the same time. For example it is possible to stage the PVP service chain N-times, where N can be as much as your compute power can scale. With N = 10, NFVbench will spawn 10 VMs as a part of 10 simultaneous PVP chains.

The number of chains is specified by --service-chain-count or -scc flag with a default value of 1. For example to run NFVbench with 3 PVP chains:

nfvbench -c nfvbench.cfg --rate 10000pps -scc 3

It is not necessary to specify the service chain type (-sc) because PVP is set as default. The PVP service chains will have 3 VMs in 3 chains with this configuration. If -sc PVVP is specified instead, there would be 6 VMs in 3 chains as this service chain has 2 VMs per chain. Both single run or NDR/PDR can be run as multichain. Running multichain is a scenario closer to a real life situation than runs with a single chain.

External Chain

NFVbench can measure the performance of 1 or more L3 service chains that are setup externally. Instead of being setup by NFVbench, the complete environment (VMs and networks) has to be setup prior to running NFVbench.

Each external chain is made of 1 or more VNFs and has exactly 2 end network interfaces (left and right network interfaces) that are connected to 2 neutron networks (left and right networks). The internal composition of a multi-VNF service chain can be arbitrary (usually linear) as far as NFVbench is concerned, the only requirement is that the service chain can route L3 packets properly between the left and right networks.

To run NFVbench on such external service chains:

  • explicitly tell NFVbench to use external service chain by adding -sc EXT or --service-chain EXT to NFVbench CLI options

  • specify the number of external chains using the -scc option (defaults to 1 chain)

  • specify the 2 end point networks of your environment in external_networks inside the config file.
    • The two networks specified there have to exist in Neutron and will be used as the end point networks by NFVbench (‘napa’ and ‘marin’ in the diagram below)
  • specify the router gateway IPs for the external service chains (1.1.0.2 and 2.2.0.2)

  • specify the traffic generator gateway IPs for the external service chains (1.1.0.102 and 2.2.0.102 in diagram below)

  • specify the packet source and destination IPs for the virtual devices that are simulated (10.0.0.0/8 and 20.0.0.0/8)

_images/extchain-config.png

L3 routing must be enabled in the VNF and configured to:

  • reply to ARP requests to its public IP addresses on both left and right networks
  • route packets from each set of remote devices toward the appropriate dest gateway IP in the traffic generator using 2 static routes (as illustrated in the diagram)

Upon start, NFVbench will: - first retrieve the properties of the left and right networks using Neutron APIs, - extract the underlying network ID (typically VLAN segmentation ID), - generate packets with the proper VLAN ID and measure traffic.

Note that in the case of multiple chains, all chains end interfaces must be connected to the same two left and right networks. The traffic will be load balanced across the corresponding gateway IP of these external service chains.

Multiflow

NFVbench always generates L3 packets from the traffic generator but allows the user to specify how many flows to generate. A flow is identified by a unique src/dest MAC IP and port tuple that is sent by the traffic generator. Flows are generated by ranging the IP adresses but using a small fixed number of MAC addresses.

The number of flows will be spread roughly even between chains when more than 1 chain is being tested. For example, for 11 flows and 3 chains, number of flows that will run for each chain will be 3, 4, and 4 flows respectively.

The number of flows is specified by --flow-count or -fc flag, the default value is 2 (1 flow in each direction). To run NFVbench with 3 chains and 100 flows, use the following command:

nfvbench -c nfvbench.cfg --rate 10000pps -scc 3 -fc 100

Note that from a vswitch point of view, the number of flows seen will be higher as it will be at least 4 times the number of flows sent by the traffic generator (add flow to VM and flow from VM).

IP addresses generated can be controlled with the following NFVbench configuration options:

ip_addrs: ['10.0.0.0/8', '20.0.0.0/8']
ip_addrs_step: 0.0.0.1
tg_gateway_ip_addrs: ['1.1.0.100', '2.2.0.100']
tg_gateway_ip_addrs_step: 0.0.0.1
gateway_ip_addrs: ['1.1.0.2', '2.2.0.2']
gateway_ip_addrs_step: 0.0.0.1

ip_addrs are the start of the 2 ip address ranges used by the traffic generators as the packets source and destination packets where each range is associated to virtual devices simulated behind 1 physical interface of the traffic generator. These can also be written in CIDR notation to represent the subnet.

tg_gateway_ip_addrs are the traffic generator gateway (virtual) ip addresses, all traffic to/from the virtual devices go through them.

gateway_ip_addrs are the 2 gateway ip address ranges of the VMs used in the external chains. They are only used with external chains and must correspond to their public IP address.

The corresponding step is used for ranging the IP addresses from the ip_addrs`, tg_gateway_ip_addrs and gateway_ip_addrs base addresses. 0.0.0.1 is the default step for all IP ranges. In ip_addrs, ‘random’ can be configured which tells NFVBench to generate random src/dst IP pairs in the traffic stream.

Traffic Configuration via CLI

While traffic configuration can be modified using the configuration file, it can be inconvenient to have to change the configuration file everytime you need to change a traffic configuration option. Traffic configuration options can be overridden with a few CLI options.

Here is an example of configuring traffic via CLI:

nfvbench --rate 10kpps --service-chain-count 2 -fs 64 -fs IMIX -fs 1518 --unidir

This command will run NFVbench with a unidirectional flow for three packet sizes 64B, IMIX, and 1518B.

Used parameters:

  • --rate 10kpps : defines rate of packets sent by traffic generator (total TX rate)
  • -scc 2 or --service-chain-count 2 : specifies number of parallel chains of given flow to run (default to 1)
  • -fs 64 or --frame-size 64: add the specified frame size to the list of frame sizes to run
  • --unidir : run traffic with unidirectional flow (default to bidirectional flow)
MAC Addresses

NFVbench will dicover the MAC addresses to use for generated frames using: - either OpenStack discovery (find the MAC of an existing VM) in the case of PVP and PVVP service chains - or using dynamic ARP discovery (find MAC from IP) in the case of external chains.

Cleanup Script

The nfvbench_cleanup script will cleanup resources created by NFVbench. You need to pass the OpenStack RC file in order to connect to OpenStack. Example of run:

# nfvbench_cleanup -r /tmp/nfvbench/openrc
Discovering Storage resources...
Discovering Compute resources...
Discovering Network resources...
Discovering Keystone resources...

SELECTED RESOURCES:
+-----------+-------------------+--------------------------------------+
| Type      | Name              | UUID                                 |
|-----------+-------------------+--------------------------------------|
| flavors   | nfvbench.medium   | 362b2215-89d1-4f46-8b89-8e58165ff5bc |
| instances | nfvbench-loop-vm0 | f78dfb74-1b8e-4c5c-8d83-652a7571da95 |
| networks  | nfvbench-net0     | 57d7e6c9-325f-4c13-9b1b-929344cc9c39 |
| networks  | nfvbench-net1     | 2d429bcd-33fa-4aa4-9f2e-299a735177c9 |
+-----------+-------------------+--------------------------------------+

Warning: You didn't specify a resource list file as the input. The script will delete all resources shown above.
Are you sure? (y/n) y
*** STORAGE cleanup
*** COMPUTE cleanup
    . Waiting for 1 instances to be fully deleted...
    . INSTANCE 1 left to be deleted, retries left=5...
    . INSTANCE 1 left to be deleted, retries left=4...
    + INSTANCE nfvbench-loop-vm0 is successfully deleted
    + FLAVOR nfvbench.medium is successfully deleted
*** NETWORK cleanup
    + Network port 075d91f3-fa6a-428c-bd3f-ebd40cd935e1 is successfully deleted
    + Network port 3a7ccd8c-53a6-43d0-a823-4b5ca762d06e is successfully deleted
    + NETWORK nfvbench-net0 is successfully deleted
    + Network port 5b5a75bd-e0b5-4f81-91b9-9e216d194f48 is successfully deleted
    + Network port cc2d8f1b-49fe-491e-9e44-6990fc57e891 is successfully deleted
    + NETWORK nfvbench-net1 is successfully deleted
*** KEYSTONE cleanup
#
NFVbench Fluentd Integration

NFVbench has an optional fluentd integration to save logs and results.

Configuring Fluentd to receive NFVbench logs and results

The following configurations should be added to Fluentd configuration file to enable logs or results.

To receive logs, and forward to a storage server:

In the example below nfvbench is the tag name for logs (which should be matched with logging_tag under NFVbench configuration), and storage backend is elasticsearch which is running at localhost:9200.

<match nfvbench.**>
@type copy
<store>
    @type elasticsearch
    host localhost
    port 9200
    logstash_format true
    logstash_prefix nfvbench
    utc_index false
    flush_interval 15s
</store>
</match>

To receive results, and forward to a storage server:

In the example below resultnfvbench is the tag name for results (which should be matched with result_tag under NFVbench configuration), and storage backend is elasticsearch which is running at localhost:9200.

<match resultnfvbench.**>
@type copy
<store>
    @type elasticsearch
    host localhost
    port 9200
    logstash_format true
    logstash_prefix resultnfvbench
    utc_index false
    flush_interval 15s
</store>
</match>
Configuring NFVbench to connect Fluentd

To configure NFVbench to connect Fluentd, fill following configuration parameters in the configuration file

Configuration Description
logging_tag Tag for NFVbench logs, it should be the same tag defined in Fluentd configuration
result_tag Tag for NFVbench results, it should be the same tag defined in Fluentd configuration
ip ip address of Fluentd server
port port number of Fluentd serverd

An example of configuration for Fluentd working at 127.0.0.1:24224 and tags for logging is nfvbench and result is resultnfvbench

fluentd:
    # by default (logging_tag is empty) nfvbench log messages are not sent to fluentd
    # to enable logging to fluents, specify a valid fluentd tag name to be used for the
    # log records
    logging_tag: nfvbench

    # by default (result_tag is empty) nfvbench results are not sent to fluentd
    # to enable sending nfvbench results to fluentd, specify a valid fluentd tag name
    # to be used for the results records, which is different than logging_tag
    result_tag: resultnfvbench

    # IP address of the server, defaults to loopback
    ip: 127.0.0.1

    # port # to use, by default, use the default fluentd forward port
    port: 24224
Example of logs and results

An example of log obtained from fluentd by elasticsearch:

{
  "_index": "nfvbench-2017.10.17",
  "_type": "fluentd",
  "_id": "AV8rhnCjTgGF_dX8DiKK",
  "_version": 1,
  "_score": 3,
  "_source": {
    "loglevel": "INFO",
    "message": "Service chain 'PVP' run completed.",
    "@timestamp": "2017-10-17T18:09:09.516897+0000",
    "runlogdate": "2017-10-17T18:08:51.851253+0000"
  },
  "fields": {
    "@timestamp": [
      1508263749516
    ]
  }
}

For each packet size and rate a result record is sent. Users can label those results by passing –user-label parameter to NFVbench run

And the results of this command obtained from fluentd by elasticsearch:

{
  "_index": "resultnfvbench-2017.10.17",
  "_type": "fluentd",
  "_id": "AV8rjYlbTgGF_dX8Drl1",
  "_version": 1,
  "_score": null,
  "_source": {
    "compute_nodes": [
      "nova:compute-3"
    ],
    "total_orig_rate_bps": 200000000,
    "@timestamp": "2017-10-17T18:16:43.755240+0000",
    "frame_size": "64",
    "forward_orig_rate_pps": 148809,
    "flow_count": 10000,
    "avg_delay_usec": 6271,
    "total_tx_rate_pps": 283169,
    "total_tx_rate_bps": 190289668,
    "forward_tx_rate_bps": 95143832,
    "reverse_tx_rate_bps": 95145836,
    "forward_tx_rate_pps": 141583,
    "chain_analysis_duration": "60.091",
    "service_chain": "PVP",
    "version": "1.0.10.dev1",
    "runlogdate": "2017-10-17T18:10:12.134260+0000",
    "Encapsulation": "VLAN",
    "user_label": "nfvbench-label",
    "min_delay_usec": 70,
    "profile": "traffic_profile_64B",
    "reverse_rx_rate_pps": 68479,
    "reverse_rx_rate_bps": 46018044,
    "reverse_orig_rate_pps": 148809,
    "total_rx_rate_bps": 92030085,
    "drop_rate_percent": 51.6368455626846,
    "forward_orig_rate_bps": 100000000,
    "bidirectional": true,
    "vSwitch": "OPENVSWITCH",
    "sc_count": 1,
    "total_orig_rate_pps": 297618,
    "type": "single_run",
    "reverse_orig_rate_bps": 100000000,
    "total_rx_rate_pps": 136949,
    "max_delay_usec": 106850,
    "forward_rx_rate_pps": 68470,
    "forward_rx_rate_bps": 46012041,
    "reverse_tx_rate_pps": 141586
  },
  "fields": {
    "@timestamp": [
      1508264203755
    ]
  },
  "sort": [
    1508264203755
  ]
}
Testing SR-IOV

NFVbench supports SR-IOV with the PVP packet flow (PVVP is not supported). SR-IOV support is not applicable for external chains since the networks have to be setup externally (and can themselves be pre-set to use SR-IOV or not).

Pre-requisites

To test SR-IOV you need to have compute nodes configured to support one or more SR-IOV interfaces (also knows as PF or physical function) and you need OpenStack to be configured to support SR-IOV. You will also need to know: - the name of the physical networks associated to your SR-IOV interfaces (this is a configuration in Nova compute) - the VLAN range that can be used on the switch ports that are wired to the SR-IOV ports. Such switch ports are normally configured in trunk mode with a range of VLAN ids enabled on that port

For example, in the case of 2 SR-IOV ports per compute node, 2 physical networks are generally configured in OpenStack with a distinct name. The VLAN range to use is is also allocated and reserved by the network administrator and in coordination with the corresponding top of rack switch port configuration.

Configuration

To enable SR-IOV test, you will need to provide the following configuration options to NFVbench (in the configuration file). This example instructs NFVbench to create the left and right networks of a PVP packet flow to run on 2 SRIOV ports named “phys_sriov0” and “phys_sriov1” using resp. segmentation_id 2000 and 2001:

internal_networks:
   left:
       segmentation_id: 2000
       physical_network: phys_sriov0
   right:
       segmentation_id: 2001
       physical_network: phys_sriov1

The segmentation ID fields must be different. In the case of PVVP, the middle network also needs to be provisioned properly. The same physical network can also be shared by the virtual networks but with different segmentation IDs.

NIC NUMA socket placement and flavors

If the 2 selected ports reside on NICs that are on different NUMA sockets, you will need to explicitly tell Nova to use 2 numa nodes in the flavor used for the VMs in order to satisfy the filters, for example:

flavor:
  # Number of vCPUs for the flavor
  vcpus: 2
  # Memory for the flavor in MB
  ram: 8192
  # Size of local disk in GB
  disk: 0
  extra_specs:
      "hw:cpu_policy": dedicated
      "hw:mem_page_size": large
      "hw:numa_nodes": 2

Failure to do so might cause the VM creation to fail with the Nova error “Instance creation error: Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology.”

NFVbench Server mode and NFVbench client API

NFVbench can run as an HTTP server to:

  • optionally provide access to any arbitrary HTLM files (HTTP server function) - this is optional
  • service fully parameterized aynchronous run requests using the HTTP protocol (REST/json with polling)
  • service fully parameterized run requests with interval stats reporting using the WebSocket/SocketIO protocol.
Start the NFVbench server

To run in server mode, simply use the –server <http_root_path> and optionally the listen address to use (–host <ip>, default is 0.0.0.0) and listening port to use (–port <port>, default is 7555).

If HTTP files are to be serviced, they must be stored right under the http root path. This root path must contain a static folder to hold static files (css, js) and a templates folder with at least an index.html file to hold the template of the index.html file to be used. This mode is convenient when you do not already have a WEB server hosting the UI front end. If HTTP files servicing is not needed (REST only or WebSocket/SocketIO mode), the root path can point to any dummy folder.

Once started, the NFVbench server will be ready to service HTTP or WebSocket/SocketIO requests at the advertised URL.

Example of NFVbench server start in a container:

# get to the container shell (assume the container name is "nfvbench")
docker exec -it nfvbench bash
# from the container shell start the NFVbench server in the background
nfvbench -c /tmp/nfvbench/nfvbench.cfg --server /tmp &
# exit container
exit
HTTP Interface
<http-url>/echo (GET)

This request simply returns whatever content is sent in the body of the request (body should be in json format, only used for testing)

Example request:

curl -XGET '127.0.0.1:7556/echo' -H "Content-Type: application/json" -d '{"nfvbench": "test"}'
Response:
{
  "nfvbench": "test"
}
<http-url>/status (GET)

This request fetches the status of an asynchronous run. It will return in json format:

  • a request pending reply (if the run is still not completed)
  • an error reply if there is no run pending
  • or the complete result of the run

The client can keep polling until the run completes.

Example of return when the run is still pending:

{
  "error_message": "nfvbench run still pending",
  "status": "PENDING"
}

Example of return when the run completes:

{
  "result": {...}
  "status": "OK"
}
<http-url>/start_run (POST)

This request starts an NFVBench run with passed configurations. If no configuration is passed, a run with default configurations will be executed.

Example request: curl -XPOST ‘localhost:7556/start_run’ -H “Content-Type: application/json” -d @nfvbenchconfig.json

See “NFVbench configuration JSON parameter” below for details on how to format this parameter.

The request returns immediately with a json content indicating if there was an error (status=ERROR) or if the request was submitted successfully (status=PENDING). Example of return when the submission is successful:

{
  "error_message": "NFVbench run still pending",
  "request_id": "42cccb7effdc43caa47f722f0ca8ec96",
  "status": "PENDING"
}

If there is already an NFVBench running then it will return:

{
 "error_message": "there is already an NFVbench request running",
 "status": "ERROR"
}
WebSocket/SocketIO events

List of SocketIO events supported:

Client to Server

start_run:

sent by client to start a new run with the configuration passed in argument (JSON). The configuration can be any valid NFVbench configuration passed as a JSON document (see “NFVbench configuration JSON parameter” below)
Server to Client

run_interval_stats:

sent by server to report statistics during a run the message contains the statistics {‘time_ms’: time_ms, ‘tx_pps’: tx_pps, ‘rx_pps’: rx_pps, ‘drop_pct’: drop_pct}

ndr_found:

(during NDR-PDR search) sent by server when the NDR rate is found the message contains the NDR value {‘rate_pps’: ndr_pps}

ndr_found:

(during NDR-PDR search) sent by server when the PDR rate is found the message contains the PDR value {‘rate_pps’: pdr_pps}

run_end:

sent by server to report the end of a run the message contains the complete results in JSON format
NFVbench configuration JSON parameter

The NFVbench configuration describes the parameters of an NFVbench run and can be passed to the NFVbench server as a JSON document.

Default configuration

The simplest JSON document is the empty dictionary “{}” which indicates to use the default NFVbench configuration:

  • PVP
  • NDR-PDR measurement
  • 64 byte packets
  • 1 flow per direction

The entire default configuration can be viewed using the –show-json-config option on the cli:

# nfvbench --show-config
{
    "availability_zone": null,
    "compute_node_user": "root",
    "compute_nodes": null,
    "debug": false,
    "duration_sec": 60,
    "flavor": {
        "disk": 0,
        "extra_specs": {
            "hw:cpu_policy": "dedicated",
            "hw:mem_page_size": 2048
        },
        "ram": 8192,
        "vcpus": 2
    },
    "flavor_type": "nfv.medium",
    "flow_count": 1,
    "generic_poll_sec": 2,
    "generic_retry_count": 100,
    "inter_node": false,
    "internal_networks": {
        "left": {
            "name": "nfvbench-net0",
            "subnet": "nfvbench-subnet0",
            "cidr": "192.168.1.0/24",
        },
        "right": {
            "name": "nfvbench-net1",
            "subnet": "nfvbench-subnet1",
            "cidr": "192.168.2.0/24",
        },
        "middle": {
            "name": "nfvbench-net2",
            "subnet": "nfvbench-subnet2",
            "cidr": "192.168.3.0/24",
        }
    },
    "interval_sec": 10,
    "json": null,
    "loop_vm_name": "nfvbench-loop-vm",
    "measurement": {
        "NDR": 0.001,
        "PDR": 0.1,
        "load_epsilon": 0.1
    },
    "name": "(built-in default config)",
    "no_cleanup": false,
    "no_int_config": false,
    "no_reset": false,
    "no_tor_access": false,
    "no_traffic": false,
    "no_vswitch_access": false,
    "openrc_file": "/tmp/nfvbench/openstack/openrc",
    "openstack_defaults": "/tmp/nfvbench/openstack/defaults.yaml",
    "openstack_setup": "/tmp/nfvbench/openstack/setup_data.yaml",
    "rate": "ndr_pdr",
    "service_chain": "PVP",
    "service_chain_count": 1,
    "sriov": false,
    "std_json": null,
    "tor": {
        "switches": [
            {
                "host": "172.26.233.12",
                "password": "lab",
                "port": 22,
                "username": "admin"
            }
        ],
        "type": "N9K"
    },
    "traffic": {
        "bidirectional": true,
        "profile": "traffic_profile_64B"
    },
    "traffic_generator": {
        "default_profile": "trex-local",
        "gateway_ip_addrs": [
            "1.1.0.2",
            "2.2.0.2"
        ],
        "gateway_ip_addrs_step": "0.0.0.1",
        "generator_profile": [
            {
                "cores": 3,
                "interfaces": [
                    {
                        "pci": "0a:00.0",
                        "port": 0,
                        "switch_port": "Ethernet1/33",
                        "vlan": null
                    },
                    {
                        "pci": "0a:00.1",
                        "port": 1,
                        "switch_port": "Ethernet1/34",
                        "vlan": null
                    }
                ],
                "intf_speed": "10Gbps",
                "ip": "127.0.0.1",
                "name": "trex-local",
                "tool": "TRex"
            }
        ],
        "host_name": "nfvbench_tg",
        "ip_addrs": [
            "10.0.0.0/8",
            "20.0.0.0/8"
        ],
        "ip_addrs_step": "0.0.0.1",
        "mac_addrs": [
            "00:10:94:00:0A:00",
            "00:11:94:00:0A:00"
        ],
        "step_mac": null,
        "tg_gateway_ip_addrs": [
            "1.1.0.100",
            "2.2.0.100"
        ],
        "tg_gateway_ip_addrs_step": "0.0.0.1"
    },
    "traffic_profile": [
        {
            "l2frame_size": [
                "64"
            ],
            "name": "traffic_profile_64B"
        },
        {
            "l2frame_size": [
                "IMIX"
            ],
            "name": "traffic_profile_IMIX"
        },
        {
            "l2frame_size": [
                "1518"
            ],
            "name": "traffic_profile_1518B"
        },
        {
            "l2frame_size": [
                "64",
                "IMIX",
                "1518"
            ],
            "name": "traffic_profile_3sizes"
        }
    ],
    "unidir_reverse_traffic_pps": 1,
    "vlan_tagging": true,
    "vts_ncs": {
        "host": null,
        "password": "secret",
        "port": 22,
        "username": "admin"
    }
}
Common examples of JSON configuration

Use the default configuration but use 10000 flows per direction (instead of 1):

{ "flow_count": 10000 }

Use default confguration but with 10000 flows, “EXT” chain and IMIX packet size:

{
   "flow_count": 10000,
   "service_chain": "EXT",
    "traffic": {
        "profile": "traffic_profile_IMIX"
    },
}

A short run of 5 seconds at a fixed rate of 1Mpps (and everything else same as the default configuration):

{
   "duration": 5,
   "rate": "1Mpps"
}
Example of interaction with the NFVbench server using HTTP and curl

HTTP requests can be sent directly to the NFVbench server from CLI using curl from any host that can connect to the server (here we run it from the local host).

This is a POST request to start a run using the default NFVbench configuration but with traffic generation disabled (“no_traffic” property is set to true):

[root@sjc04-pod3-mgmt ~]# curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"no_traffic":true}' http://127.0.0.1:7555/start_run
{
  "error_message": "nfvbench run still pending",
  "status": "PENDING"
}
[root@sjc04-pod3-mgmt ~]#

This request will return immediately with status set to “PENDING” if the request was started successfully.

The status can be polled until the run completes. Here the poll returns a “PENDING” status, indicating the run is still not completed:

[root@sjc04-pod3-mgmt ~]# curl -G http://127.0.0.1:7555/status
{
  "error_message": "nfvbench run still pending",
  "status": "PENDING"
}
[root@sjc04-pod3-mgmt ~]#

Finally, the status request returns a “OK” status along with the full results (truncated here):

[root@sjc04-pod3-mgmt ~]# curl -G http://127.0.0.1:7555/status
{
  "result": {
    "benchmarks": {
      "network": {
        "service_chain": {
          "PVP": {
            "result": {
              "bidirectional": true,
              "compute_nodes": {
                "nova:sjc04-pod3-compute-4": {
                  "bios_settings": {
                    "Adjacent Cache Line Prefetcher": "Disabled",
                    "All Onboard LOM Ports": "Enabled",
                    "All PCIe Slots OptionROM": "Enabled",
                    "Altitude": "300 M",
...

    "date": "2017-03-31 22:15:41",
    "nfvbench_version": "0.3.5",
    "openstack_spec": {
      "encaps": "VxLAN",
      "vswitch": "VTS"
    }
  },
  "status": "OK"
}
[root@sjc04-pod3-mgmt ~]#
Example of interaction with the NFVbench server using a python CLI app (nfvbench_client)

The module client/client.py contains an example of python class that can be used to control the NFVbench server from a python app using HTTP or WebSocket/SocketIO.

The module client/nfvbench_client.py has a simple main application to control the NFVbench server from CLI. The “nfvbench_client” wrapper script can be used to invoke the client front end (this wrapper is pre-installed in the NFVbench container)

Example of invocation of the nfvbench_client front end, from the host (assume the name of the NFVbench container is “nfvbench”), use the default NFVbench configuration but do not generate traffic (no_traffic property set to true, the full json result is truncated here):

[root@sjc04-pod3-mgmt ~]# docker exec -it nfvbench nfvbench_client -c '{"no_traffic":true}' http://127.0.0.1:7555
{u'status': u'PENDING', u'error_message': u'nfvbench run still pending'}
{u'status': u'PENDING', u'error_message': u'nfvbench run still pending'}
{u'status': u'PENDING', u'error_message': u'nfvbench run still pending'}

{u'status': u'OK', u'result': {u'date': u'2017-03-31 22:04:59', u'nfvbench_version': u'0.3.5',
u'config': {u'compute_nodes': None, u'compute_node_user': u'root', u'vts_ncs': {u'username': u'admin', u'host': None, u'password': u'secret', u'port': 22}, u'traffic_generator': {u'tg_gateway_ip_addrs': [u'1.1.0.100', u'2.2.0.100'], u'ip_addrs_step': u'0.0.0.1', u'step_mac': None, u'generator_profile': [{u'intf_speed': u'10Gbps', u'interfaces': [{u'pci': u'0a:00.0', u'port': 0, u'vlan': 1998, u'switch_port': None},

...

[root@sjc04-pod3-mgmt ~]#

The http interface is used unless –use-socketio is defined.

Example of invocation using Websocket/SocketIO, execute NFVbench using the default configuration but with a duration of 5 seconds and a fixed rate run of 5kpps.

[root@sjc04-pod3-mgmt ~]# docker exec -it nfvbench nfvbench_client -c '{"duration":5,"rate":"5kpps"}' --use-socketio  http://127.0.0.1:7555 >results.json
Frequently Asked Questions
General Questions
Can NFVbench be used with a different traffic generator than TRex?

This is possible but requires developing a new python class to manage the new traffic generator interface.

Can I connect Trex directly to my compute node?

That is possible but you will not be able to run more advanced use cases such as PVVP inter-node which requires 2 compute nodes.

Can I drive NFVbench using a REST interface?

NFVbench can run in server mode and accept HTTP or WebSocket/SocketIO events to run any type of measurement (fixed rate run or NDR_PDR run) with any run configuration.

Can I run NFVbench on a Cisco UCS-B series blade?

Yes provided your UCS-B series server has a Cisco VIC 1340 (with a recent firmware version). TRex will require VIC firmware version 3.1(2) or higher for blade servers (which supports more filtering capabilities). In this setting, the 2 physical interfaces for data plane traffic are simply hooked to the UCS-B fabric interconnect (no need to connect to a switch).

Troubleshooting
TrafficClientException: End-to-end connectivity cannot be ensured

Prior to running a benchmark, NFVbench will make sure that traffic is passing in the service chain by sending a small flow of packets in each direction and verifying that they are received back at the other end. This exception means that NFVbench cannot pass any traffic in the service chain.

The most common issues that prevent traffic from passing are: - incorrect wiring of the NFVbench/TRex interfaces - incorrect vlan_tagging setting in the NFVbench configuration, this needs to match how the NFVbench ports on the switch are configured (trunk or access port)

  • if the switch port is configured as access port, you must disable vlan_tagging in the NFVbench configuration
  • of the switch port is configured as trunk (recommended method), you must enable it

QTIP

QTIP Installation Guide
Configuration

QTIP currently supports by using a Docker image. Detailed steps about setting up QTIP can be found below.

To use QTIP you should have access to an OpenStack environment, with at least Nova, Neutron, Glance, Keystone and Heat installed. Add a brief introduction to configure OPNFV with this specific installer

Installing QTIP using Docker
QTIP docker image

QTIP has a Docker images on the docker hub. Pulling opnfv/qtip docker image from docker hub:

docker pull opnfv/qtip:stable

Verify that opnfv/qtip has been downloaded. It should be listed as an image by running the following command.

docker images
Run and enter the docker instance

1. If you want to run benchmarks:

envs="INSTALLER_TYPE={INSTALLER_TYPE} -e INSTALLER_IP={INSTALLER_IP} -e  NODE_NAME={NODE_NAME}"
docker run -p [HOST_IP:]<HOST_PORT>:5000 --name qtip -id -e $envs opnfv/qtip
docker exec -i -t qtip /bin/bash

INSTALLER_TYPE should be one of OPNFV installer, e.g. apex, compass, daisy, fuel and joid. Currenty, QTIP only supports installer fuel.

INSTALLER_IP is the ip address of the installer that can be accessed by QTIP.

NODE_NAME is the name of opnfv pod, e.g. zte-pod1.

2. If you do not want to run any benchmarks:

docker run --name qtip -id opnfv/qtip
docker exec -i -t qtip /bin/bash

Now you are in the container and QTIP can be found in the /repos/qtip and can be navigated to using the following command.

cd repos/qtip
Install from source code

You may try out the latest version of QTIP by installing from source code. It is recommended to run it under Python virtualenv so it won’t screw system libraries.

Run the following commands:

git clone https://git.opnfv.org/qtip && cd qtip
virtualenv .venv && source .venv/bin/activate
pip install -e .

Use the following command to exit virtualenv:

deactivate

Re-enter the virtualenv with:

cd <qtip-directory>
source .venv/bin/activate
Environment configuration
Hardware configuration

QTIP does not have specific hardware requriements, and it can runs over any OPNFV installer.

Jumphost configuration

Installer Docker on Jumphost, which is used for running QTIP image.

You can refer to these links:

Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntu/

Centos: https://docs.docker.com/engine/installation/linux/centos/

Platform components configuration

Describe the configuration of each component in the installer.

Web Portal installation & configuration

Web Portal for Benchmarking is developed on python Django Framework. Right now the installation is need to be done from source.

Clone QTIP Repo
git clone https://github.com/opnfv/qtip.git
Setup database and Initialize user data
CD into web directory.
cd qtip/qtip/web
Setup migrations
python manage.py makemigrations

In usual case migrations will be already available with source. Console willll notify you of the same.

Run migrations
python manage.py migrate
Create superuser
python manage.py createsuperuser

Console will prompt for adding new web admin. Enter new credentials.

Collecting Static Dependencies
python manage.py importstatic

This will import js and css dependencies for UI in static directory. Now the web application is ready to run.

QTIP User Guide
Overview

QTIP is the project for Platform Performance Benchmarking in OPNFV. It aims to provide user a simple indicator for performance, simple but supported by comprehensive testing data and transparent calculation formula.

QTIP introduces a concept called QPI, a.k.a. QTIP Performance Index, which aims to be a TRUE indicator of performance. TRUE reflects the core value of QPI in four aspects

  • Transparent: being an open source project, user can inspect all details behind QPI, e.g. formulas, metrics, raw data
  • Reliable: the integrity of QPI will be guaranteed by traceability in each step back to raw test result
  • Understandable: QPI is broke down into section scores, and workload scores in report to help user to understand
  • Extensible: users may create their own QPI by composing the existed metrics in QTIP or extend new metrics
Benchmarks

The builtin benchmarks of QTIP are located in <package_root>/benchmarks folder

  • QPI: specifications about how an QPI is calculated and sources of metrics
  • metric: performance metrics referred in QPI, currently it is categorized by performance testing tools
  • plan: executable benchmarking plan which collects metrics and calculate QPI
Getting started with QTIP
Installation

Refer to `installation and configuration guide`_ for details

Create

Create a new project to hold the necessary configurations and test results

qtip create <project_name>

The user would be prompted for OPNFV installer, its hostname etc

**Pod Name [unknown]: zte-pod1**
User's choice to name OPNFV Pod

**OPNFV Installer [manual]: fuel**
QTIP currently supports fuel and apex only

**Installer Hostname [dummy-host]: master**
The hostname for the fuel or apex installer node. The same hostname can be added to **~/.ssh/config** file of current user,
if there are problems resolving the hostname via interactive input.

**OPNFV Scenario [unknown]: os-nosdn-nofeature-ha**
Depends on the OPNFV scenario deployed
Setup

With the project is created, user should now proceed on to setting up testing environment. In this step, ssh connection to hosts in SUT will be configured automatically:

cd <project_name>
$ qtip setup
Run

QTIP uses ssh-agent for authentication of ssh connection to hosts in SUT. It must be started correctly before running the tests:

eval $(ssh-agent)

Then run test with qtip run

Teardown

Clean up the temporary folder on target hosts.

Note

The installed packages for testing won’t be uninstalled.

One more thing

You may use -v for verbose output (-vvv for more, -vvvv to enable connection debugging)

CLI User Manual

QTIP consists of a number of benchmarking tools or metrics, grouped under QPI’s. QPI’s map to the different components of a NFVi ecosystem, such as compute, network and storage. Depending on the type of application, a user may group them under plans.

Bash Command Completion

To enable command completion, an environment variable needs to be enabled. Add the following line to the .bashrc file

eval "$(_QTIP_COMPLETE=source qtip)"
Getting help

QTIP CLI provides interface to all of the above the components. A help page provides a list of all the commands along with a short description.

qtip --help
Usage

QTIP is currently supports two different QPI’s, compute and storage. To list all the supported QPI

qtip qpi list

The details of any QPI can be viewed as follows

qtip qpi show <qpi_name>

In order to benchmark either one of them, their respective templates need to be generated

qtip create --project-template [compute|storage] <workspace_name>

By default, the compute template will be generated. An interactive prompt would gather all parameters specific to OpenStack installation.

Once the template generation is complete, configuration for OpenStack needs to be generated.

cd <workspace_name>
qtip setup

This step generates the inventory, populating it with target nodes.

QTIP can now be run

qtip run

This would start the complete testing suite, which is either compute or storage. Each suite normally takes about half an hour to complete.

Benchmarking report is made for each and every individual section in a QPI, on a particular target node. It consists of the actual test values on that node along with scores calculated by comparison against a baseline.

qtip report show [-n|--node] <node> <section_name>
Debugging options

QTIP uses Ansible as the runner. One can use all of Ansible’s CLI option with QTIP. In order to enable verbose mode

qtip setup -v

One may also be able to achieve the different levels of verbosity

qtip run [-v|-vv|-vvv]
API User Manual

QTIP consists of a number of benchmarking tools or metrics, grouped under QPI’s. QPI’s map to the different components of an NFVI ecosystem, such as compute, network and storage. Depending on the type of application, a user may group them under plans.

QTIP API provides a RESTful interface to all of the above components. User can retrieve list of plans, QPIs and metrics and their individual information.

Running

After installing QTIP. API server can be run using command qtip-api on the local machine.

All the resources and their corresponding operation details can be seen at /v1.0/ui.

The whole API specification in json format can be seen at /v1.0/swagger.json.

The data models are given below:

  • Plan
  • Metric
  • QPI

Plan:

{
  "name": <plan name>,
  "description": <plan profile>,
  "info": <{plan info}>,
  "config": <{plan configuration}>,
  "QPIs": <[list of qpis]>,
},

Metric:

{
  "name": <metric name>,
  "description": <metric description>,
  "links": <[links with metric information]>,
  "workloads": <[cpu workloads(single_cpu, multi_cpu]>,
},

QPI:

{
  "name": <qpi name>,
  "description": <qpi description>,
  "formula": <formula>,
  "sections": <[list of sections with different metrics and formulaes]>,
}

The API can be described as follows

Plans:

Method Path Description
GET /v1.0/plans Get the list of of all plans
GET /v1.0/plans/{name} Get details of the specified plan

Metrics:

Method Path Description
GET /v1.0/metrics Get the list of all metrics
GET /v1.0/metrics/{name} Get details of specified metric

QPIs:

Method Path Description
GET /v1.0/qpis Get the list of all QPIs
GET /v1.0/qpis/{name} Get details of specified QPI
Note:
running API with connexion cli does not require base path (/v1.0/) in url
Web Portal User Manual

QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI’s) such as compute, storage and network. QTIP benchmarking tasks are built upon Ansible playbooks and roles. QTIP web portal is a platform to expose QTIP as a benchmarking service hosted on a central host.

Running

After setting up the web portal as instructed in config guide, cd into the web directory.

and run.

python manage.py runserver 0.0.0.0

You can access the portal by logging onto <host>:8000/bench/login/

If you want to use port 80, you may need sudo permission.

sudo python manage.py runserver 0.0.0.0:80

To Deploy on wsgi, Use the Django deployment tutorial

Features

After logging in You’ll be redirect to QTIP-Web Dashboard. You’ll see following menus on left.

  • Repos
  • Run Benchmarks
  • Tasks
Repo
Repos are links to qtip workspaces. This menu list all the aded repos. Links to new repos can be added here.
Run Benchmarks
To run a benchmark, select the corresponding repo and run. QTIP Benchmarking service will clone the workspace and run the benchmarks. Inventories used are predefined in the workspace repo in the /hosts/ config file.
Tasks
All running or completed benchmark jobs can be seen in Tasks menu with their status.

New users can be added by Admin on the Django Admin app by logging into `/admin/’.

Compute Performance Benchmarking

The compute QPI aims to benchmark the compute components of an OPNFV platform. Such components include, the CPU performance, the memory performance.

The compute QPI consists of both synthetic and application specific benchmarks to test compute components.

All the compute benchmarks could be run in the scenario: On Baremetal Machines provisioned by an OPNFV installer (Host machines)

Note: The Compute benchmank constains relatively old benchmarks such as dhrystone and whetstone. The suite would be updated for better benchmarks such as Linbench for the OPNFV E release.

Getting started

Notice: All descriptions are based on QTIP container.

Inventory File

QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to determine what hosts to work against. QTIP can automatically generate a inventory file via OPNFV installer. Users also can write their own inventory infomation into /home/opnfv/qtip/hosts. This file is just a text file containing a list of host IP addresses. For example:

[hosts]
10.20.0.11
10.20.0.12
QTIP key Pair

QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI, QTIP will generate a key pair named QtipKey under /home/opnfv/qtip/ and pass public key to remote hosts.

If environment variable CI_DEBUG is set to true, users should delete it by manual. If CI_DEBUG is not set or set to false, QTIP will delete the key from remote hosts before the execution ends. Please make sure the key deleted from remote hosts or it can introduce a security flaw.

Commands

In a QTIP container, you can run compute QPI by using QTIP CLI:

mkdir result
qtip plan run <plan_name> -p $PWD/result

QTIP generates results in the $PWD/result directory are listed down under the timestamp name.

you can get more details from userguide/cli.rst.

Metrics

The benchmarks include:

Dhrystone 2.1

Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to evaluate CPU capabilities. Both Single CPU performance is measured along multi-cpu performance.

Dhrystone, however, is a dated benchmark and has some short comings. Written in C, it is a small program that doesn’t test the CPU memory subsystem. Additionally, dhrystone results could be modified by optimizing the compiler and insome cases hardware configuration.

References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf

Whetstone

Whetstone is a synthetic benchmark to measure CPU floating point operation performance. Both Single CPU performance is measured along multi-cpu performance.

Like Dhrystone, Whetstone is a dated benchmark and has short comings.

References:

http://www.netlib.org/benchmark/whetstone.c

OpenSSL Speed

OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two OpenSSL Speed benchmarks are incorporated:

  1. RSA signatunes/sec signed by a machine
  2. AES 128-bit encryption throughput for a machine for cipher block sizes

References:

https://www.openssl.org/docs/manmaster/apps/speed.html

RAMSpeed

RAMSpeed is used to measure a machine’s memory perfomace. The problem(array)size is large enough to ensure Cache Misses so that the main machine memory is used.

INTmem and FLOATmem benchmarks are executed in 4 different scenarios:

  1. Copy: a(i)=b(i)
  2. Add: a(i)=b(i)+c(i)
  3. Scale: a(i)=b(i)*d
  4. Tniad: a(i)=b(i)+c(i)*d

INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points for these benchmarks.

References:

http://alasir.com/software/ramspeed/

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untangling+memory+access+measurements

DPI

nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop. An example application called pcapreader has been developed and is available for use along nDPI.

A sample .pcap file is passed to the pcapreader application. nDPI classifies traffic in the pcap file into different categories based on string matching. The pcapreader application provides a throughput number for the rate at which traffic was classified, indicating a machine’s computational performance. The results are run 10 times and an average is taken for the obtained number.

nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes

References:

http://www.ntop.org/products/deep-packet-inspection/ndpi/

http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf

Storperf

StorPerf User Guide
1. StorPerf Introduction

The purpose of StorPerf is to provide a tool to measure ephemeral and block storage performance of OpenStack.

A key challenge to measuring disk performance is to know when the disk (or, for OpenStack, the virtual disk or volume) is performing at a consistent and repeatable level of performance. Initial writes to a volume can perform poorly due to block allocation, and reads can appear instantaneous when reading empty blocks. How do we know when the data reported is valid? The Storage Network Industry Association (SNIA) has developed methods which enable manufacturers to set, and customers to compare, the performance specifications of Solid State Storage devices. StorPerf applies this methodology to OpenStack Cinder and Glance services to provide a high level of confidence in the performance metrics in the shortest reasonable time.

1.1. How Does StorPerf Work?

Once launched, StorPerf presents you with a ReST interface, along with a Swagger UI that makes it easier to form HTTP ReST requests. Issuing an HTTP POST to the configurations API causes StorPerf to talk to your OpenStack’s heat service to create a new stack with as many agent VMs and attached Cinder volumes as you specify.

After the stack is created, you can issue one or more jobs by issuing a POST to the jobs ReST API. The job is the smallest unit of work that StorPerf can use to measure the disk’s performance.

While the job is running, StorPerf collects the performance metrics from each of the disks under test every minute. Once the trend of metrics match the criteria specified in the SNIA methodology, the job automatically terminates and the valid set of metrics are available for querying.

What is the criteria? Simply put, it specifies that when the metrics measured start to “flat line” and stay within that range for the specified amount of time, then the metrics are considered to be indicative of a repeatable level of performance.

1.2. What Data Can I Get?

StorPerf provides the following metrics:

  • IOPS
  • Bandwidth (number of kilobytes read or written per second)
  • Latency

These metrics are available for every job, and for the specific workloads, I/O loads and I/O types (read, write) associated with the job.

As of this time, StorPerf only provides textual reports of the metrics.

1. StorPerf Installation Guide
1.1. OpenStack Prerequisites

If you do not have an Ubuntu 16.04 image in Glance, you will need to add one. You also need to create the StorPerf flavor, or choose one that closely matches. For Ubuntu 16.04, it must have a minimum of a 4 GB disk. It should also have about 8 GB RAM to support FIO’s memory mapping of written data blocks to ensure 100% coverage of the volume under test.

There are scripts in storperf/ci directory to assist, or you can use the follow code snippets:

# Put an Ubuntu Image in glance
wget -q https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img
openstack image create "Ubuntu 16.04 x86_64" --disk-format qcow2 --public \
    --container-format bare --file ubuntu-16.04-server-cloudimg-amd64-disk1.img

# Create StorPerf flavor
openstack flavor create storperf \
    --id auto \
    --ram 8192 \
    --disk 4 \
    --vcpus 2
1.1.1. OpenStack Credentials

You must have your OpenStack Controller environment variables defined and passed to the StorPerf container. The easiest way to do this is to put the rc file contents into a clean file called admin.rc that looks similar to this for V2 authentication:

cat << 'EOF' > admin.rc
OS_AUTH_URL=http://10.13.182.243:5000/v2.0
OS_TENANT_ID=e8e64985506a4a508957f931d1800aa9
OS_TENANT_NAME=admin
OS_PROJECT_NAME=admin
OS_USERNAME=admin
OS_PASSWORD=admin
OS_REGION_NAME=RegionOne
EOF

For V3 authentication, at a minimum, use the following:

cat << 'EOF' > admin.rc
OS_AUTH_URL=http://10.10.243.14:5000/v3
OS_USERNAME=admin
OS_PASSWORD=admin
OS_PROJECT_DOMAIN_NAME=Default
OS_PROJECT_NAME=admin
OS_USER_DOMAIN_NAME=Default
EOF

Additionally, if you want your results published to the common OPNFV Test Results DB, add the following:

TEST_DB_URL=http://testresults.opnfv.org/testapi
1.2. Planning

StorPerf is delivered as a series of Docker containers managed by docker-compose. There are two possible methods for installation:

  1. Run container on bare metal
  2. Run container in a VM

Requirements:

  • Docker and docker-compose must be installed * (note: sudo may be required if user is not part of docker group)
  • OpenStack Controller credentials are available
  • Host has access to the OpenStack Controller API
  • Host must have internet connectivity for downloading docker image
  • Enough OpenStack floating IPs must be available to match your agent count
  • A local directory for holding the Carbon DB Whisper files

Local disk used for the Carbon DB storage as the default size of the docker container is only 10g. Here is an example of how to create a local storage directory and set its permissions so that StorPerf can write to it:

mkdir -p ./carbon
sudo chown 33:33 ./carbon
1.3. Ports

The following ports are exposed if you use the supplied docker-compose.yaml file:

  • 5000 for StorPerf ReST API and Swagger UI

Note: Port 8000 is no longer exposed and graphite can be accesed via http://storperf:5000/graphite

1.4. Running StorPerf Container

As of Euphrates (development) release (June 2017), StorPerf has changed to use docker-compose in order to start its services.

Docker compose requires a local file to be created in order to define the services that make up the full StorPerf application. This file can be:

  • Manually created
  • Downloaded from the StorPerf git repo, or
  • Create via a helper script from the StorPerf git repo

Manual creation involves taking the sample in the StorPerf git repo and typing in the contents by hand on your target system.

1.5. Downloading From Git Repo
wget https://raw.githubusercontent.com/opnfv/storperf/master/docker-compose/docker-compose.yaml
sha256sum docker-compose.yaml

which should result in:

968c0c2d7c0e24f6777c33b37d9b4fd885575155069fb760405ec8214b2eb672  docker-compose.yaml

To run, you must specify two environment variables:

  • ENV_FILE, which points to your OpenStack admin.rc as noted above.
  • CARBON_DIR, which points to a directory that will be mounted to store the raw metrics.
  • TAG, which specified the Docker tag for the build (ie: latest, danube.3.0, etc).

The following command will start all the StorPerf services:

TAG=latest ENV_FILE=./admin.rc CARBON_DIR=./carbon/ docker-compose pull
TAG=latest ENV_FILE=./admin.rc CARBON_DIR=./carbon/ docker-compose up -d

StorPerf is now available at http://docker-host:5000/

1.6. Downloading Helper Tool

A tool to help you get started with the docker-compose.yaml can be downloaded from:

wget https://raw.githubusercontent.com/opnfv/storperf/master/docker-compose/create-compose.py
sha256sum create-compose.py

which should result in:

00649e02237d27bf0b40d1a66160a68a56c9f5e1ceb78d7858e30715cf4350e3  create-compose.py

Note: The script will run fine on python3. Install python future package to avoid error on python2.

pip install future
1.6.1. Docker Exec

If needed, the container can be entered with docker exec. This is not normally required.

docker exec -it storperf-master bash
1.7. Pulling StorPerf Container
1.7.1. Master (Euphrates)

StorPerf has switched to docker-compose in the latest version. The tag for pulling the latest master Euphrates container is:

docker pull opnfv/storperf-master:latest
docker pull opnfv/storperf-reporting:latest
docker pull opnfv/storperf-httpfrontend:latest
docker pull opnfv/storperf-swaggerui:latest
docker pull opnfv/storperf-graphite:latest

However, by itself, this will no longer provide full functionality. Full instructions are provided in the Running StorPerf Container section of this document.

1.7.2. Danube

The tag for the latest stable Danube is be:

docker pull opnfv/storperf:danube.3.0
1.7.3. Colorado

The tag for the latest stable Colorado release is:

docker pull opnfv/storperf:colorado.0.1
1.7.4. Brahmaputra

The tag for the latest stable Brahmaputra release is:

docker pull opnfv/storperf:brahmaputra.1.2
1.8. StorPerf on ARM Processors

StorPerf now supports docker images on ARM processors as well. However, at the moment there is no upstream image on DockerHub. The user needs to manually build it. Firstly, clone StorPerf repository from GitHub

git clone https://git.opnfv.org/storperf
cd storperf/docker/

Next, build and setup the docker images

TAG=aarch64 ENV_FILE=./admin.rc CARBON_DIR=./carbon docker-compose -f local-docker-compose.yaml build
TAG=aarch64 ENV_FILE=./admin.rc CARBON_DIR=./carbon docker-compose -f local-docker-compose.yaml up -d
2. StorPerf Test Execution Guide
2.1. Prerequisites

This guide requires StorPerf to be running and have its ReST API accessible. If the ReST API is not running on port 5000, adjust the commands provided here as needed.

2.2. Interacting With StorPerf

Once the StorPerf container has been started and the ReST API exposed, you can interact directly with it using the ReST API. StorPerf comes with a Swagger interface that is accessible through the exposed port at:

http://StorPerf:5000/swagger/index.html

The typical test execution follows this pattern:

  1. Configure the environment
  2. Initialize the cinder volumes
  3. Execute one or more performance runs
  4. Delete the environment
2.3. Configure The Environment

The following pieces of information are required to prepare the environment:

  • The number of VMs/Cinder volumes to create
  • The Glance image that holds the VM operating system to use. StorPerf has only been tested with Ubuntu 16.04
  • The OpenStack flavor to use when creating the VMs
  • The name of the public network that agents will use
  • The size, in gigabytes, of the Cinder volumes to create
  • The availability zone (optional) in which the VM is to be launched. Defaults to nova.
  • The username (optional) if we specify a custom image.
  • The password (optional) for the above image.

The ReST API is a POST to http://StorPerf:5000/api/v1.0/configurations and takes a JSON payload as follows.

{
  "agent_count": int,
  "agent_flavor": string
  "agent_image": string,
  "public_network": string,
  "volume_size": int,
  "availability_zone": string,
  "username": string,
  "password": string
}

This call will block until the stack is created, at which point it will return the OpenStack heat stack id.

2.4. Initialize the Cinder Volumes

Before executing a test run for the purpose of measuring performance, it is necessary to fill the Cinder volume with random data. Failure to execute this step can result in meaningless numbers, especially for read performance. Most Cinder drivers are smart enough to know what blocks contain data, and which do not. Uninitialized blocks return “0” immediately without actually reading from the volume.

Initiating the data fill looks the same as a regular performance test, but uses the special workload called “_warm_up”. StorPerf will never push _warm_up data to the OPNFV Test Results DB, nor will it terminate the run on steady state. It is guaranteed to run to completion, which fills 100% of the volume with random data.

The ReST API is a POST to http://StorPerf:5000/api/v1.0/jobs and takes a JSON payload as follows.

{
   "workload": "_warm_up"
}

This will return a job ID as follows.

{
  "job_id": "edafa97e-457e-4d3d-9db4-1d6c0fc03f98"
}

This job ID can be used to query the state to determine when it has completed. See the section on querying jobs for more information.

2.5. Execute a Performance Run

Performance runs can execute either a single workload, or iterate over a matrix of workload types, block sizes and queue depths.

2.5.1. Workload Types
rr
Read, Random. 100% read of random blocks
rs
Read, Sequential. 100% read of sequential blocks of data
rw
Read / Write Mix, Random. 70% random read, 30% random write
wr
Write, Random. 100% write of random blocks
ws
Write, Sequential. 100% write of sequential blocks.
2.5.2. Block Sizes

A comma delimited list of the different block sizes to use when reading and writing data. Note: Some Cinder drivers (such as Ceph) cannot support block sizes larger than 16k (16384).

2.5.3. Queue Depths

A comma delimited list of the different queue depths to use when reading and writing data. The queue depth parameter causes FIO to keep this many I/O requests outstanding at one time. It is used to simulate traffic patterns on the system. For example, a queue depth of 4 would simulate 4 processes constantly creating I/O requests.

2.5.4. Deadline

The deadline is the maximum amount of time in minutes for a workload to run. If steady state has not been reached by the deadline, the workload will terminate and that particular run will be marked as not having reached steady state. Any remaining workloads will continue to execute in order.

{
   "block_sizes": "2048,16384",
   "deadline": 20,
   "queue_depths": "2,4",
   "workload": "wr,rr,rw"
}
2.5.5. Metadata

A job can have metadata associated with it for tagging. The following metadata is required in order to push results to the OPNFV Test Results DB:

"metadata": {
    "disk_type": "HDD or SDD",
    "pod_name": "OPNFV Pod Name",
    "scenario_name": string,
    "storage_node_count": int,
    "version": string,
    "build_tag": string,
    "test_case": "snia_steady_state"
}
2.6. Query Jobs Information

By issuing a GET to the job API http://StorPerf:5000/api/v1.0/jobs?job_id=<ID>, you can fetch information about the job as follows:

  • &type=status: to report on the status of the job.
  • &type=metrics: to report on the collected metrics.
  • &type=metadata: to report back any metadata sent with the job ReST API
2.6.1. Status

The Status field can be: - Running to indicate the job is still in progress, or - Completed to indicate the job is done. This could be either normal completion

or manually terminated via HTTP DELETE call.

Workloads can have a value of: - Pending to indicate the workload has not yet started, - Running to indicate this is the active workload, or - Completed to indicate this workload has completed.

This is an example of a type=status call.

{
  "Status": "Running",
  "TestResultURL": null,
  "Workloads": {
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.1.block-size.16384": "Pending",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.1.block-size.4096": "Pending",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.1.block-size.512": "Pending",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.4.block-size.16384": "Running",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.4.block-size.4096": "Pending",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.4.block-size.512": "Pending",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.8.block-size.16384": "Completed",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.8.block-size.4096": "Pending",
    "eeb2e587-5274-4d2f-ad95-5c85102d055e.ws.queue-depth.8.block-size.512": "Pending"
  }
}

If the job_id is not provided along with type status, then all jobs are returned along with their status. Metrics ~~~~~~~ Metrics can be queried at any time during or after the completion of a run. Note that the metrics show up only after the first interval has passed, and are subject to change until the job completes.

This is a sample of a type=metrics call.

{
  "rw.queue-depth.1.block-size.512.read.bw": 52.8,
  "rw.queue-depth.1.block-size.512.read.iops": 106.76199999999999,
  "rw.queue-depth.1.block-size.512.read.lat_ns.mean": 93.176,
  "rw.queue-depth.1.block-size.512.write.bw": 22.5,
  "rw.queue-depth.1.block-size.512.write.iops": 45.760000000000005,
  "rw.queue-depth.1.block-size.512.write.lat_ns.mean": 21764.184999999998
}
2.7. Abort a Job

Issuing an HTTP DELETE to the job api http://StorPerf:5000/api/v1.0/jobs will force the termination of the whole job, regardless of how many workloads remain to be executed.

curl -X DELETE --header 'Accept: application/json' http://StorPerf:5000/api/v1.0/jobs
2.8. List all Jobs

A list of all Jobs can also be queried. You just need to issue a GET request without any Job ID.

curl -X GET --header 'Accept: application/json' http://StorPerf/api/v1.0/jobs
2.9. Delete the Environment

After you are done testing, you can have StorPerf delete the Heat stack by issuing an HTTP DELETE to the configurations API.

curl -X DELETE --header 'Accept: application/json' http://StorPerf:5000/api/v1.0/configurations

You may also want to delete an environment, and then create a new one with a different number of VMs/Cinder volumes to test the impact of the number of VMs in your environment.

2.10. Viewing StorPerf Logs

Logs are an integral part of any application as they help debugging the application. The user just needs to issue an HTTP request. To view the entire logs

curl -X GET --header 'Accept: application/json' http://StorPerf:5000/api/v1.0/logs?lines=all

Alternatively, one can also view a certain amount of lines by specifying the number in the request. If no lines are specified, then last 35 lines are returned

curl -X GET --header 'Accept: application/json' http://StorPerf:5000/api/v1.0/logs?lines=12
3. Storperf Reporting Module
3.1. About this project
  • This project aims to create a series of graphs to support the SNIA reports.
  • All data for the reports can be fetched either from the OPNFV Test Results DB, or locally from StorPerf’s own database of current run data.
  • The report code may be stored in either the Releng repository (so it can be included in the Test Results Dashboards), or locally in StorPerf’s own git repository.
  • The report (generated by the reporting module) looks like the following example:
_images/demo_example.png
3.2. Usage
  • Enter the URL for the location of the data for which you want to generate the report.
  • Note: You can test the module using the testdata present in the directory storperf-reporting/src/static/testdata. Instead of the URL enter the filename present in the testdata directory, eg. local-data.json
  • After entering the URL, you are taken to the page showing the details of the all the jobs present in the data.
  • Click on the Click here to view details to see the different block sizes for the respective job.
  • Click on the block size and select the parameter for which you want to view the graph.
3.3. Graph Explanation

Example of a graph generated is shown below:-

_images/graph_explanation.png

Steady State Convergence Graph

  • This graph shows the values as reported by StorPerf for the actual and average throughput.
  • Also shown is the average +-10% and the slope.
  • It serves to visually demonstrate the compliance to the steady state definition.
  • The allowed maximum data excursion is 20% of the average (or average x 0.20)
  • The allowed maximum slope excursion is 10% of the average.
  • The measured data excursion is the value from range.
  • The measured slope excursion is the value from range
3.4. Workflow

A Flask server is used to fetch the data and is sent to the client side for formation of the graphs (Using Javascript).

3.4.1. Steps involved
  • Step 1: Data is fetched from the OPNFV Test Results ReST API
  • Step 2: The fields “report_data” and “metrics” are taken from the JSON object retrieved in the above step and sent to the client side.
  • Step 3: The “report_data” is obtained by the client side and a parser written in Javascript along with Plotly.js forms the graphs.
3.5. Directory structure

storperf/docker/storperf-reporting/ contains the code used for this project.

The file structure is as follows:-

storperf-reporting
|+-- Dockerfile                         # Dockerfile for the storperf-reporting container
|+-- requirements.txt                   # pip requirements for the container
+-- src                                 # Contains the code for the flask server
    |+-- app.py                         # Code to run the flask application
    |+-- static                         # Contains the static files (js,css)
    |   |+-- css                        # Contains css files
    |   |   `-- bootstrap.min.css
    |   |+-- images
    |   |+-- js                         # Contains the javascript files
    |   |   |-- bootstrap.min.js
    |   |   |-- Chart.min.js
    |   |   |-- jquery-2.1.3.min.js
    |   |   |-- jquery.bootpag.min.js
    |   |   `-- plotly-latest.min.js    # Used for plotting the graphs
    |   `-- testdata                    # Contains testing data for the module
    `-- templates
        |-- index.html
        |-- plot_jobs.html
        |-- plot_multi_data.html
        `-- plot_tables.html
3.6. Graphing libraries and tools
  • Plotly.js is used as the graphing library for this project (Link: https://plot.ly/javascript/)
  • Bootstrap is used for the UI of the project.

VSPERF

VSPERF Configuration and User Guide
Introduction

VSPERF is an OPNFV testing project.

VSPERF provides an automated test-framework and comprehensive test suite based on Industry Test Specifications for measuring NFVI data-plane performance. The data-path includes switching technologies with physical and virtual network interfaces. The VSPERF architecture is switch and traffic generator agnostic and test cases can be easily customized. VSPERF was designed to be independent of OpenStack therefore OPNFV installer scenarios are not required. VSPERF can source, configure and deploy the device-under-test using specified software versions and network topology. VSPERF is used as a development tool for optimizing switching technologies, qualification of packet processing functions and for evaluation of data-path performance.

The Euphrates release adds new features and improvements that will help advance high performance packet processing on Telco NFV platforms. This includes new test cases, flexibility in customizing test-cases, new results display options, improved tool resiliency, additional traffic generator support and VPP support.

VSPERF provides a framework where the entire NFV Industry can learn about NFVI data-plane performance and try-out new techniques together. A new IETF benchmarking specification (RFC8204) is based on VSPERF work contributed since 2015. VSPERF is also contributing to development of ETSI NFV test specifications through the Test and Open Source Working Group.

VSPERF Install and Configuration
1. Installing vswitchperf
1.1. Downloading vswitchperf

The vswitchperf can be downloaded from its official git repository, which is hosted by OPNFV. It is necessary to install a git at your DUT before downloading vswitchperf. Installation of git is specific to the packaging system used by Linux OS installed at DUT.

Example of installation of GIT package and its dependencies:

  • in case of OS based on RedHat Linux:

    sudo yum install git
    
  • in case of Ubuntu or Debian:

    sudo apt-get install git
    

After the git is successfully installed at DUT, then vswitchperf can be downloaded as follows:

git clone http://git.opnfv.org/vswitchperf

The last command will create a directory vswitchperf with a local copy of vswitchperf repository.

1.2. Supported Operating Systems
  • CentOS 7.3
  • Fedora 24 (kernel 4.8 requires DPDK 16.11 and newer)
  • Fedora 25 (kernel 4.9 requires DPDK 16.11 and newer)
  • openSUSE 42.2
  • openSUSE 42.3
  • RedHat 7.2 Enterprise Linux
  • RedHat 7.3 Enterprise Linux
  • Ubuntu 14.04
  • Ubuntu 16.04
  • Ubuntu 16.10 (kernel 4.8 requires DPDK 16.11 and newer)
1.3. Supported vSwitches

The vSwitch must support Open Flow 1.3 or greater.

  • Open vSwitch
  • Open vSwitch with DPDK support
  • TestPMD application from DPDK (supports p2p and pvp scenarios)
  • Cisco VPP
1.4. Supported Hypervisors
  • Qemu version 2.3 or greater (version 2.5.0 is recommended)
1.5. Supported VNFs

In theory, it is possible to use any VNF image, which is compatible with supported hypervisor. However such VNF must ensure, that appropriate number of network interfaces is configured and that traffic is properly forwarded among them. For new vswitchperf users it is recommended to start with official vloop-vnf image, which is maintained by vswitchperf community.

1.5.1. vloop-vnf

The official VM image is called vloop-vnf and it is available for free download from OPNFV artifactory. This image is based on Linux Ubuntu distribution and it supports following applications for traffic forwarding:

  • DPDK testpmd
  • Linux Bridge
  • Custom l2fwd module

The vloop-vnf can be downloaded to DUT, for example by wget:

wget http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-14.04_20160823.qcow2

NOTE: In case that wget is not installed at your DUT, you could install it at RPM based system by sudo yum install wget or at DEB based system by sudo apt-get install wget.

Changelog of vloop-vnf:

1.6. Installation

The test suite requires Python 3.3 or newer and relies on a number of other system and python packages. These need to be installed for the test suite to function.

Installation of required packages, preparation of Python 3 virtual environment and compilation of OVS, DPDK and QEMU is performed by script systems/build_base_machine.sh. It should be executed under user account, which will be used for vsperf execution.

NOTE: Password-less sudo access must be configured for given user account before script is executed.

$ cd systems
$ ./build_base_machine.sh

NOTE: you don’t need to go into any of the systems subdirectories, simply run the top level build_base_machine.sh, your OS will be detected automatically.

Script build_base_machine.sh will install all the vsperf dependencies in terms of system packages, Python 3.x and required Python modules. In case of CentOS 7 or RHEL it will install Python 3.3 from an additional repository provided by Software Collections (a link). Installation script will also use virtualenv to create a vsperf virtual environment, which is isolated from the default Python environment. This environment will reside in a directory called vsperfenv in $HOME. It will ensure, that system wide Python installation is not modified or broken by VSPERF installation. The complete list of Python packages installed inside virtualenv can be found at file requirements.txt, which is located at vswitchperf repository.

NOTE: For RHEL 7.3 Enterprise and CentOS 7.3 OVS Vanilla is not built from upstream source due to kernel incompatibilities. Please see the instructions in the vswitchperf_design document for details on configuring OVS Vanilla for binary package usage.

1.6.1. VPP installation

VPP installation is now included as part of the VSPerf installation scripts.

In case of an error message about a missing file such as “Couldn’t open file /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7” you can resolve this issue by simply downloading the file.

$ wget https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
1.7. Using vswitchperf

You will need to activate the virtual environment every time you start a new shell session. Its activation is specific to your OS:

  • CentOS 7 and RHEL

    $ scl enable python33 bash
    $ source $HOME/vsperfenv/bin/activate
    
  • Fedora and Ubuntu

    $ source $HOME/vsperfenv/bin/activate
    

After the virtual environment is configued, then VSPERF can be used. For example:

(vsperfenv) $ cd vswitchperf
(vsperfenv) $ ./vsperf --help
1.7.1. Gotcha

In case you will see following error during environment activation:

$ source $HOME/vsperfenv/bin/activate
Badly placed ()'s.

then check what type of shell you are using:

$ echo $SHELL
/bin/tcsh

See what scripts are available in $HOME/vsperfenv/bin

$ ls $HOME/vsperfenv/bin/
activate          activate.csh      activate.fish     activate_this.py

source the appropriate script

$ source bin/activate.csh
1.7.2. Working Behind a Proxy

If you’re behind a proxy, you’ll likely want to configure this before running any of the above. For example:

export http_proxy=proxy.mycompany.com:123
export https_proxy=proxy.mycompany.com:123
1.7.3. Bind Tools DPDK

VSPerf supports the default DPDK bind tool, but also supports driverctl. The driverctl tool is a new tool being used that allows driver binding to be persistent across reboots. The driverctl tool is not provided by VSPerf, but can be downloaded from upstream sources. Once installed set the bind tool to driverctl to allow VSPERF to correctly bind cards for DPDK tests.

PATHS['dpdk']['src']['bind-tool'] = 'driverctl'
1.8. Hugepage Configuration

Systems running vsperf with either dpdk and/or tests with guests must configure hugepage amounts to support running these configurations. It is recommended to configure 1GB hugepages as the pagesize.

The amount of hugepages needed depends on your configuration files in vsperf. Each guest image requires 2048 MB by default according to the default settings in the 04_vnf.conf file.

GUEST_MEMORY = ['2048']

The dpdk startup parameters also require an amount of hugepages depending on your configuration in the 02_vswitch.conf file.

DPDK_SOCKET_MEM = ['1024', '0']

NOTE: Option DPDK_SOCKET_MEM is used by all vSwitches with DPDK support. It means Open vSwitch, VPP and TestPMD.

VSPerf will verify hugepage amounts are free before executing test environments. In case of hugepage amounts not being free, test initialization will fail and testing will stop.

NOTE: In some instances on a test failure dpdk resources may not release hugepages used in dpdk configuration. It is recommended to configure a few extra hugepages to prevent a false detection by VSPerf that not enough free hugepages are available to execute the test environment. Normally dpdk would use previously allocated hugepages upon initialization.

Depending on your OS selection configuration of hugepages may vary. Please refer to your OS documentation to set hugepages correctly. It is recommended to set the required amount of hugepages to be allocated by default on reboots.

Information on hugepage requirements for dpdk can be found at http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html

You can review your hugepage amounts by executing the following command

cat /proc/meminfo | grep Huge

If no hugepages are available vsperf will try to automatically allocate some. Allocation is controlled by HUGEPAGE_RAM_ALLOCATION configuration parameter in 02_vswitch.conf file. Default is 2GB, resulting in either 2 1GB hugepages or 1024 2MB hugepages.

1.9. Tuning Considerations

With the large amount of tuning guides available online on how to properly tune a DUT, it becomes difficult to achieve consistent numbers for DPDK testing. VSPerf recommends a simple approach that has been tested by different companies to achieve proper CPU isolation.

The idea behind CPU isolation when running DPDK based tests is to achieve as few interruptions to a PMD process as possible. There is now a utility available on most Linux Systems to achieve proper CPU isolation with very little effort and customization. The tool is called tuned-adm and is most likely installed by default on the Linux DUT

VSPerf recommends the latest tuned-adm package, which can be downloaded from the following location:

http://www.tuned-project.org/2017/04/27/tuned-2-8-0-released/

Follow the instructions to install the latest tuned-adm onto your system. For current RHEL customers you should already have the most current version. You just need to install the cpu-partitioning profile.

yum install -y tuned-profiles-cpu-partitioning.noarch

Proper CPU isolation starts with knowing what NUMA your NIC is installed onto. You can identify this by checking the output of the following command

cat /sys/class/net/<NIC NAME>/device/numa_node

You can then use utilities such as lscpu or cpu_layout.py which is located in the src dpdk area of VSPerf. These tools will show the CPU layout of which cores/hyperthreads are located on the same NUMA.

Determine which CPUS/Hyperthreads will be used for PMD threads and VCPUs for VNFs. Then modify the /etc/tuned/cpu-partitioning-variables.conf and add the CPUs into the isolated_cores variable in some form of x-y or x,y,z or x-y,z, etc. Then apply the profile.

tuned-adm profile cpu-partitioning

After applying the profile, reboot your system.

After rebooting the DUT, you can verify the profile is active by running

tuned-adm active

Now you should have proper CPU isolation active and can achieve consistent results with DPDK based tests.

The last consideration is when running TestPMD inside of a VNF, it may make sense to enable enough cores to run a PMD thread on separate core/HT. To achieve this, set the number of VCPUs to 3 and enable enough nb-cores in the TestPMD config. You can modify options in the conf files.

GUEST_SMP = ['3']
GUEST_TESTPMD_PARAMS = ['-l 0,1,2 -n 4 --socket-mem 512 -- '
                        '--burst=64 -i --txqflags=0xf00 '
                        '--disable-hw-vlan --nb-cores=2']

Verify you set the VCPU core locations appropriately on the same NUMA as with your PMD mask for OVS-DPDK.

2. Upgrading vswitchperf
2.1. Generic

In case, that VSPERF is cloned from git repository, then it is easy to upgrade it to the newest stable version or to the development version.

You could get a list of stable releases by git command. It is necessary to update local git repository first.

NOTE: Git commands must be executed from directory, where VSPERF repository was cloned, e.g. vswitchperf.

Update of local git repository:

$ git pull

List of stable releases:

$ git tag

brahmaputra.1.0
colorado.1.0
colorado.2.0
colorado.3.0
danube.1.0
euphrates.1.0

You could select which stable release should be used. For example, select danube.1.0:

$ git checkout danube.1.0

Development version of VSPERF can be selected by:

$ git checkout master
2.2. Colorado to Danube upgrade notes
2.2.1. Obsoleted features

Support of vHost Cuse interface has been removed in Danube release. It means, that it is not possible to select QemuDpdkVhostCuse as a VNF anymore. Option QemuDpdkVhostUser should be used instead. Please check you configuration files and definition of your testcases for any occurrence of:

VNF = "QemuDpdkVhostCuse"

or

"VNF" : "QemuDpdkVhostCuse"

In case that QemuDpdkVhostCuse is found, it must be modified to QemuDpdkVhostUser.

NOTE: In case that execution of VSPERF is automated by scripts (e.g. for CI purposes), then these scripts must be checked and updated too. It means, that any occurrence of:

./vsperf --vnf QemuDpdkVhostCuse

must be updated to:

./vsperf --vnf QemuDpdkVhostUser
2.2.2. Configuration

Several configuration changes were introduced during Danube release. The most important changes are discussed below.

2.2.2.1. Paths to DPDK, OVS and QEMU

VSPERF uses external tools for proper testcase execution. Thus it is important to properly configure paths to these tools. In case that tools are installed by installation scripts and are located inside ./src directory inside VSPERF home, then no changes are needed. On the other hand, if path settings was changed by custom configuration file, then it is required to update configuration accordingly. Please check your configuration files for following configuration options:

OVS_DIR
OVS_DIR_VANILLA
OVS_DIR_USER
OVS_DIR_CUSE

RTE_SDK_USER
RTE_SDK_CUSE

QEMU_DIR
QEMU_DIR_USER
QEMU_DIR_CUSE
QEMU_BIN

In case that any of these options is defined, then configuration must be updated. All paths to the tools are now stored inside PATHS dictionary. Please refer to the Configuration of PATHS dictionary and update your configuration where necessary.

2.2.2.2. Configuration change via CLI

In previous releases it was possible to modify selected configuration options (mostly VNF specific) via command line interface, i.e. by --test-params argument. This concept has been generalized in Danube release and it is possible to modify any configuration parameter via CLI or via Parameters section of the testcase definition. Old configuration options were obsoleted and it is required to specify configuration parameter name in the same form as it is defined inside configuration file, i.e. in uppercase. Please refer to the Referencing parameter values for additional details.

NOTE: In case that execution of VSPERF is automated by scripts (e.g. for CI purposes), then these scripts must be checked and updated too. It means, that any occurrence of

guest_loopback
vanilla_tgen_port1_ip
vanilla_tgen_port1_mac
vanilla_tgen_port2_ip
vanilla_tgen_port2_mac
tunnel_type

shall be changed to the uppercase form and data type of entered values must match to data types of original values from configuration files.

In case that guest_nic1_name or guest_nic2_name is changed, then new dictionary GUEST_NICS must be modified accordingly. Please see Configuration of GUEST options and conf/04_vnf.conf for additional details.

2.2.2.3. Traffic configuration via CLI

In previous releases it was possible to modify selected attributes of generated traffic via command line interface. This concept has been enhanced in Danube release and it is now possible to modify all traffic specific options via CLI or by TRAFFIC dictionary in configuration file. Detailed description is available at Configuration of TRAFFIC dictionary section of documentation.

Please check your automated scripts for VSPERF execution for following CLI parameters and update them according to the documentation:

bidir
duration
frame_rate
iload
lossrate
multistream
pkt_sizes
pre-installed_flows
rfc2544_tests
stream_type
traffic_type
3. ‘vsperf’ Traffic Gen Guide
3.1. Overview

VSPERF supports the following traffic generators:

To see the list of traffic gens from the cli:

$ ./vsperf --list-trafficgens

This guide provides the details of how to install and configure the various traffic generators.

3.2. Background Information

The traffic default configuration can be found in conf/03_traffic.conf, and is configured as follows:

TRAFFIC = {
    'traffic_type' : 'rfc2544_throughput',
    'frame_rate' : 100,
    'bidir' : 'True',  # will be passed as string in title format to tgen
    'multistream' : 0,
    'stream_type' : 'L4',
    'pre_installed_flows' : 'No',           # used by vswitch implementation
    'flow_type' : 'port',                   # used by vswitch implementation

    'l2': {
        'framesize': 64,
        'srcmac': '00:00:00:00:00:00',
        'dstmac': '00:00:00:00:00:00',
    },
    'l3': {
        'enabled': True,
        'proto': 'udp',
        'srcip': '1.1.1.1',
        'dstip': '90.90.90.90',
    },
    'l4': {
        'enabled': True,
        'srcport': 3000,
        'dstport': 3001,
    },
    'vlan': {
        'enabled': False,
        'id': 0,
        'priority': 0,
        'cfi': 0,
    },
}

The framesize parameter can be overridden from the configuration files by adding the following to your custom configuration file 10_custom.conf:

TRAFFICGEN_PKT_SIZES = (64, 128,)

OR from the commandline:

$ ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=(x,y)" $TESTNAME

You can also modify the traffic transmission duration and the number of tests run by the traffic generator by extending the example commandline above to:

$ ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=(x,y);TRAFFICGEN_DURATION=10;" \
                         "TRAFFICGEN_RFC2544_TESTS=1" $TESTNAME
3.3. Dummy

The Dummy traffic generator can be used to test VSPERF installation or to demonstrate VSPERF functionality at DUT without connection to a real traffic generator.

You could also use the Dummy generator in case, that your external traffic generator is not supported by VSPERF. In such case you could use VSPERF to setup your test scenario and then transmit the traffic. After the transmission is completed you could specify values for all collected metrics and VSPERF will use them to generate final reports.

3.3.1. Setup

To select the Dummy generator please add the following to your custom configuration file 10_custom.conf.

TRAFFICGEN = 'Dummy'

OR run vsperf with the --trafficgen argument

$ ./vsperf --trafficgen Dummy $TESTNAME

Where $TESTNAME is the name of the vsperf test you would like to run. This will setup the vSwitch and the VNF (if one is part of your test) print the traffic configuration and prompt you to transmit traffic when the setup is complete.

Please send 'continuous' traffic with the following stream config:
30mS, 90mpps, multistream False
and the following flow config:
{
    "flow_type": "port",
    "l3": {
        "enabled": True,
        "srcip": "1.1.1.1",
        "proto": "udp",
        "dstip": "90.90.90.90"
    },
    "traffic_type": "rfc2544_continuous",
    "multistream": 0,
    "bidir": "True",
    "vlan": {
        "cfi": 0,
        "priority": 0,
        "id": 0,
        "enabled": False
    },
    "l4": {
        "enabled": True,
        "srcport": 3000,
        "dstport": 3001,
    },
    "frame_rate": 90,
    "l2": {
        "dstmac": "00:00:00:00:00:00",
        "srcmac": "00:00:00:00:00:00",
        "framesize": 64
    }
}
What was the result for 'frames tx'?

When your traffic generator has completed traffic transmission and provided the results please input these at the VSPERF prompt. VSPERF will try to verify the input:

Is '$input_value' correct?

Please answer with y OR n.

VSPERF will ask you to provide a value for every of collected metrics. The list of metrics can be found at traffic-type-metrics. Finally vsperf will print out the results for your test and generate the appropriate logs and report files.

3.3.2. Metrics collected for supported traffic types

Below you could find a list of metrics collected by VSPERF for each of supported traffic types.

RFC2544 Throughput and Continuous:

  • frames tx
  • frames rx
  • min latency
  • max latency
  • avg latency
  • frameloss

RFC2544 Back2back:

  • b2b frames
  • b2b frame loss %
3.3.3. Dummy result pre-configuration

In case of a Dummy traffic generator it is possible to pre-configure the test results. This is useful for creation of demo testcases, which do not require a real traffic generator. Such testcase can be run by any user and it will still generate all reports and result files.

Result values can be specified within TRAFFICGEN_DUMMY_RESULTS dictionary, where every of collected metrics must be properly defined. Please check the list of traffic-type-metrics.

Dictionary with dummy results can be passed by CLI argument --test-params or specified in Parameters section of testcase definition.

Example of testcase execution with dummy results defined by CLI argument:

$ ./vsperf back2back --trafficgen Dummy --test-params \
  "TRAFFICGEN_DUMMY_RESULTS={'b2b frames':'3000','b2b frame loss %':'0.0'}"

Example of testcase definition with pre-configured dummy results:

{
    "Name": "back2back",
    "Traffic Type": "rfc2544_back2back",
    "Deployment": "p2p",
    "biDirectional": "True",
    "Description": "LTD.Throughput.RFC2544.BackToBackFrames",
    "Parameters" : {
        'TRAFFICGEN_DUMMY_RESULTS' : {'b2b frames':'3000','b2b frame loss %':'0.0'}
    },
},

NOTE: Pre-configured results for the Dummy traffic generator will be used only in case, that the Dummy traffic generator is used. Otherwise the option TRAFFICGEN_DUMMY_RESULTS will be ignored.

3.4. Ixia

VSPERF can use both IxNetwork and IxExplorer TCL servers to control Ixia chassis. However, usage of IxNetwork TCL server is a preferred option. The following sections will describe installation and configuration of IxNetwork components used by VSPERF.

3.4.1. Installation

On the system under the test you need to install IxNetworkTclClient$(VER_NUM)Linux.bin.tgz.

On the IXIA client software system you need to install IxNetwork TCL server. After its installation you should configure it as follows:

  1. Find the IxNetwork TCL server app (start -> All Programs -> IXIA -> IxNetwork -> IxNetwork_$(VER_NUM) -> IxNetwork TCL Server)

  2. Right click on IxNetwork TCL Server, select properties - Under shortcut tab in the Target dialogue box make sure there is the argument “-tclport xxxx” where xxxx is your port number (take note of this port number as you will need it for the 10_custom.conf file).

    _images/TCLServerProperties.png
  3. Hit Ok and start the TCL server application

3.4.2. VSPERF configuration

There are several configuration options specific to the IxNetwork traffic generator from IXIA. It is essential to set them correctly, before the VSPERF is executed for the first time.

Detailed description of options follows:

  • TRAFFICGEN_IXNET_MACHINE - IP address of server, where IxNetwork TCL Server is running
  • TRAFFICGEN_IXNET_PORT - PORT, where IxNetwork TCL Server is accepting connections from TCL clients
  • TRAFFICGEN_IXNET_USER - username, which will be used during communication with IxNetwork TCL Server and IXIA chassis
  • TRAFFICGEN_IXIA_HOST - IP address of IXIA traffic generator chassis
  • TRAFFICGEN_IXIA_CARD - identification of card with dedicated ports at IXIA chassis
  • TRAFFICGEN_IXIA_PORT1 - identification of the first dedicated port at TRAFFICGEN_IXIA_CARD at IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of unidirectional traffic, it is essential to correctly connect 1st IXIA port to the 1st NIC at DUT, i.e. to the first PCI handle from WHITELIST_NICS list. Otherwise traffic may not be able to pass through the vSwitch. NOTE: In case that TRAFFICGEN_IXIA_PORT1 and TRAFFICGEN_IXIA_PORT2 are set to the same value, then VSPERF will assume, that there is only one port connection between IXIA and DUT. In this case it must be ensured, that chosen IXIA port is physically connected to the first NIC from WHITELIST_NICS list.
  • TRAFFICGEN_IXIA_PORT2 - identification of the second dedicated port at TRAFFICGEN_IXIA_CARD at IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of unidirectional traffic, it is essential to correctly connect 2nd IXIA port to the 2nd NIC at DUT, i.e. to the second PCI handle from WHITELIST_NICS list. Otherwise traffic may not be able to pass through the vSwitch. NOTE: In case that TRAFFICGEN_IXIA_PORT1 and TRAFFICGEN_IXIA_PORT2 are set to the same value, then VSPERF will assume, that there is only one port connection between IXIA and DUT. In this case it must be ensured, that chosen IXIA port is physically connected to the first NIC from WHITELIST_NICS list.
  • TRAFFICGEN_IXNET_LIB_PATH - path to the DUT specific installation of IxNetwork TCL API
  • TRAFFICGEN_IXNET_TCL_SCRIPT - name of the TCL script, which VSPERF will use for communication with IXIA TCL server
  • TRAFFICGEN_IXNET_TESTER_RESULT_DIR - folder accessible from IxNetwork TCL server, where test results are stored, e.g. c:/ixia_results; see test-results-share
  • TRAFFICGEN_IXNET_DUT_RESULT_DIR - directory accessible from the DUT, where test results from IxNetwork TCL server are stored, e.g. /mnt/ixia_results; see test-results-share
3.4.3. Test results share

VSPERF is not able to retrieve test results via TCL API directly. Instead, all test results are stored at IxNetwork TCL server. Results are stored at folder defined by TRAFFICGEN_IXNET_TESTER_RESULT_DIR configuration parameter. Content of this folder must be shared (e.g. via samba protocol) between TCL Server and DUT, where VSPERF is executed. VSPERF expects, that test results will be available at directory configured by TRAFFICGEN_IXNET_DUT_RESULT_DIR configuration parameter.

Example of sharing configuration:

  • Create a new folder at IxNetwork TCL server machine, e.g. c:\ixia_results

  • Modify sharing options of ixia_results folder to share it with everybody

  • Create a new directory at DUT, where shared directory with results will be mounted, e.g. /mnt/ixia_results

  • Update your custom VSPERF configuration file as follows:

    TRAFFICGEN_IXNET_TESTER_RESULT_DIR = 'c:/ixia_results'
    TRAFFICGEN_IXNET_DUT_RESULT_DIR = '/mnt/ixia_results'
    

    NOTE: It is essential to use slashes ‘/’ also in path configured by TRAFFICGEN_IXNET_TESTER_RESULT_DIR parameter.

  • Install cifs-utils package.

    e.g. at rpm based Linux distribution:

    yum install cifs-utils
    
  • Mount shared directory, so VSPERF can access test results.

    e.g. by adding new record into /etc/fstab

    mount -t cifs //_TCL_SERVER_IP_OR_FQDN_/ixia_results /mnt/ixia_results
          -o file_mode=0777,dir_mode=0777,nounix
    

It is recommended to verify, that any new file inserted into c:/ixia_results folder is visible at DUT inside /mnt/ixia_results directory.

3.5. Spirent Setup

Spirent installation files and instructions are available on the Spirent support website at:

http://support.spirent.com

Select a version of Spirent TestCenter software to utilize. This example will use Spirent TestCenter v4.57 as an example. Substitute the appropriate version in place of ‘v4.57’ in the examples, below.

3.5.1. On the CentOS 7 System

Download and install the following:

Spirent TestCenter Application, v4.57 for 64-bit Linux Client

3.5.2. Spirent Virtual Deployment Service (VDS)

Spirent VDS is required for both TestCenter hardware and virtual chassis in the vsperf environment. For installation, select the version that matches the Spirent TestCenter Application version. For v4.57, the matching VDS version is 1.0.55. Download either the ova (VMware) or qcow2 (QEMU) image and create a VM with it. Initialize the VM according to Spirent installation instructions.

3.5.3. Using Spirent TestCenter Virtual (STCv)

STCv is available in both ova (VMware) and qcow2 (QEMU) formats. For VMware, download:

Spirent TestCenter Virtual Machine for VMware, v4.57 for Hypervisor - VMware ESX.ESXi

Virtual test port performance is affected by the hypervisor configuration. For best practice results in deploying STCv, the following is suggested:

  • Create a single VM with two test ports rather than two VMs with one port each
  • Set STCv in DPDK mode
  • Give STCv 2*n + 1 cores, where n = the number of ports. For vsperf, cores = 5.
  • Turning off hyperthreading and pinning these cores will improve performance
  • Give STCv 2 GB of RAM

To get the highest performance and accuracy, Spirent TestCenter hardware is recommended. vsperf can run with either stype test ports.

3.5.4. Using STC REST Client

The stcrestclient package provides the stchttp.py ReST API wrapper module. This allows simple function calls, nearly identical to those provided by StcPython.py, to be used to access TestCenter server sessions via the STC ReST API. Basic ReST functionality is provided by the resthttp module, and may be used for writing ReST clients independent of STC.

To use REST interface, follow the instructions in the Project page to install the package. Once installed, the scripts named with ‘rest’ keyword can be used. For example: testcenter-rfc2544-rest.py can be used to run RFC 2544 tests using the REST interface.

3.5.5. Configuration:
  1. The Labserver and license server addresses. These parameters applies to all the tests, and are mandatory for all tests.
TRAFFICGEN_STC_LAB_SERVER_ADDR = " "
TRAFFICGEN_STC_LICENSE_SERVER_ADDR = " "
TRAFFICGEN_STC_PYTHON2_PATH = " "
TRAFFICGEN_STC_TESTCENTER_PATH = " "
TRAFFICGEN_STC_TEST_SESSION_NAME = " "
TRAFFICGEN_STC_CSV_RESULTS_FILE_PREFIX = " "
  1. For RFC2544 tests, the following parameters are mandatory
TRAFFICGEN_STC_EAST_CHASSIS_ADDR = " "
TRAFFICGEN_STC_EAST_SLOT_NUM = " "
TRAFFICGEN_STC_EAST_PORT_NUM = " "
TRAFFICGEN_STC_EAST_INTF_ADDR = " "
TRAFFICGEN_STC_EAST_INTF_GATEWAY_ADDR = " "
TRAFFICGEN_STC_WEST_CHASSIS_ADDR = ""
TRAFFICGEN_STC_WEST_SLOT_NUM = " "
TRAFFICGEN_STC_WEST_PORT_NUM = " "
TRAFFICGEN_STC_WEST_INTF_ADDR = " "
TRAFFICGEN_STC_WEST_INTF_GATEWAY_ADDR = " "
TRAFFICGEN_STC_RFC2544_TPUT_TEST_FILE_NAME
  1. RFC2889 tests: Currently, the forwarding, address-caching, and address-learning-rate tests of RFC2889 are supported. The testcenter-rfc2889-rest.py script implements the rfc2889 tests. The configuration for RFC2889 involves test-case definition, and parameter definition, as described below. New results-constants, as shown below, are added to support these tests.

Example of testcase definition for RFC2889 tests:

{
    "Name": "phy2phy_forwarding",
    "Deployment": "p2p",
    "Description": "LTD.Forwarding.RFC2889.MaxForwardingRate",
    "Parameters" : {
        "TRAFFIC" : {
            "traffic_type" : "rfc2889_forwarding",
        },
    },
}

For RFC2889 tests, specifying the locations for the monitoring ports is mandatory. Necessary parameters are:

TRAFFICGEN_STC_RFC2889_TEST_FILE_NAME
TRAFFICGEN_STC_EAST_CHASSIS_ADDR = " "
TRAFFICGEN_STC_EAST_SLOT_NUM = " "
TRAFFICGEN_STC_EAST_PORT_NUM = " "
TRAFFICGEN_STC_EAST_INTF_ADDR = " "
TRAFFICGEN_STC_EAST_INTF_GATEWAY_ADDR = " "
TRAFFICGEN_STC_WEST_CHASSIS_ADDR = ""
TRAFFICGEN_STC_WEST_SLOT_NUM = " "
TRAFFICGEN_STC_WEST_PORT_NUM = " "
TRAFFICGEN_STC_WEST_INTF_ADDR = " "
TRAFFICGEN_STC_WEST_INTF_GATEWAY_ADDR = " "
TRAFFICGEN_STC_VERBOSE = "True"
TRAFFICGEN_STC_RFC2889_LOCATIONS="//10.1.1.1/1/1,//10.1.1.1/2/2"

Other Configurations are :

TRAFFICGEN_STC_RFC2889_MIN_LR = 1488
TRAFFICGEN_STC_RFC2889_MAX_LR = 14880
TRAFFICGEN_STC_RFC2889_MIN_ADDRS = 1000
TRAFFICGEN_STC_RFC2889_MAX_ADDRS = 65536
TRAFFICGEN_STC_RFC2889_AC_LR = 1000

The first 2 values are for address-learning test where as other 3 values are for the Address caching capacity test. LR: Learning Rate. AC: Address Caching. Maximum value for address is 16777216. Whereas, maximum for LR is 4294967295.

Results for RFC2889 Tests: Forwarding tests outputs following values:

TX_RATE_FPS : "Transmission Rate in Frames/sec"
THROUGHPUT_RX_FPS: "Received Throughput Frames/sec"
TX_RATE_MBPS : " Transmission rate in MBPS"
THROUGHPUT_RX_MBPS: "Received Throughput in MBPS"
TX_RATE_PERCENT: "Transmission Rate in Percentage"
FRAME_LOSS_PERCENT: "Frame loss in Percentage"
FORWARDING_RATE_FPS: " Maximum Forwarding Rate in FPS"

Whereas, the address caching test outputs following values,

CACHING_CAPACITY_ADDRS = 'Number of address it can cache'
ADDR_LEARNED_PERCENT = 'Percentage of address successfully learned'

and address learning test outputs just a single value:

OPTIMAL_LEARNING_RATE_FPS = 'Optimal learning rate in fps'

Note that ‘FORWARDING_RATE_FPS’, ‘CACHING_CAPACITY_ADDRS’, ‘ADDR_LEARNED_PERCENT’ and ‘OPTIMAL_LEARNING_RATE_FPS’ are the new result-constants added to support RFC2889 tests.

3.6. Xena Networks
3.6.1. Installation

Xena Networks traffic generator requires specific files and packages to be installed. It is assumed the user has access to the Xena2544.exe file which must be placed in VSPerf installation location under the tools/pkt_gen/xena folder. Contact Xena Networks for the latest version of this file. The user can also visit www.xenanetworks/downloads to obtain the file with a valid support contract.

Note VSPerf has been fully tested with version v2.43 of Xena2544.exe

To execute the Xena2544.exe file under Linux distributions the mono-complete package must be installed. To install this package follow the instructions below. Further information can be obtained from http://www.mono-project.com/docs/getting-started/install/linux/

rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF"
yum-config-manager --add-repo http://download.mono-project.com/repo/centos/
yum -y install mono-complete

To prevent gpg errors on future yum installation of packages the mono-project repo should be disabled once installed.

yum-config-manager --disable download.mono-project.com_repo_centos_
3.6.2. Configuration

Connection information for your Xena Chassis must be supplied inside the 10_custom.conf or 03_custom.conf file. The following parameters must be set to allow for proper connections to the chassis.

TRAFFICGEN_XENA_IP = ''
TRAFFICGEN_XENA_PORT1 = ''
TRAFFICGEN_XENA_PORT2 = ''
TRAFFICGEN_XENA_USER = ''
TRAFFICGEN_XENA_PASSWORD = ''
TRAFFICGEN_XENA_MODULE1 = ''
TRAFFICGEN_XENA_MODULE2 = ''
3.6.3. RFC2544 Throughput Testing

Xena traffic generator testing for rfc2544 throughput can be modified for different behaviors if needed. The default options for the following are optimized for best results.

TRAFFICGEN_XENA_2544_TPUT_INIT_VALUE = '10.0'
TRAFFICGEN_XENA_2544_TPUT_MIN_VALUE = '0.1'
TRAFFICGEN_XENA_2544_TPUT_MAX_VALUE = '100.0'
TRAFFICGEN_XENA_2544_TPUT_VALUE_RESOLUTION = '0.5'
TRAFFICGEN_XENA_2544_TPUT_USEPASS_THRESHHOLD = 'false'
TRAFFICGEN_XENA_2544_TPUT_PASS_THRESHHOLD = '0.0'

Each value modifies the behavior of rfc 2544 throughput testing. Refer to your Xena documentation to understand the behavior changes in modifying these values.

Xena RFC2544 testing inside VSPerf also includes a final verification option. This option allows for a faster binary search with a longer final verification of the binary search result. This feature can be enabled in the configuration files as well as the length of the final verification in seconds.

..code-block:: python

TRAFFICGEN_XENA_RFC2544_VERIFY = False TRAFFICGEN_XENA_RFC2544_VERIFY_DURATION = 120

If the final verification does not pass the test with the lossrate specified it will continue the binary search from its previous point. If the smart search option is enabled the search will continue by taking the current pass rate minus the minimum and divided by 2. The maximum is set to the last pass rate minus the threshold value set.

For example if the settings are as follows

..code-block:: python

TRAFFICGEN_XENA_RFC2544_BINARY_RESTART_SMART_SEARCH = True TRAFFICGEN_XENA_2544_TPUT_MIN_VALUE = ‘0.5’ TRAFFICGEN_XENA_2544_TPUT_VALUE_RESOLUTION = ‘0.5’

and the verification attempt was 64.5, smart search would take 64.5 - 0.5 / 2. This would continue the search at 32 but still have a maximum possible value of 64.

If smart is not enabled it will just resume at the last pass rate minus the threshold value.

3.6.4. Continuous Traffic Testing

Xena continuous traffic by default does a 3 second learning preemption to allow the DUT to receive learning packets before a continuous test is performed. If a custom test case requires this learning be disabled, you can disable the option or modify the length of the learning by modifying the following settings.

TRAFFICGEN_XENA_CONT_PORT_LEARNING_ENABLED = False
TRAFFICGEN_XENA_CONT_PORT_LEARNING_DURATION = 3
3.7. MoonGen
3.7.1. Installation

MoonGen architecture overview and general installation instructions can be found here:

https://github.com/emmericp/MoonGen

  • Note: Today, MoonGen with VSPERF only supports 10Gbps line speeds.

For VSPERF use, MoonGen should be cloned from here (as opposed to the previously mentioned GitHub):

git clone https://github.com/atheurer/lua-trafficgen

and use the master branch:

git checkout master

VSPERF uses a particular Lua script with the MoonGen project:

trafficgen.lua

Follow MoonGen set up and execution instructions here:

https://github.com/atheurer/lua-trafficgen/blob/master/README.md

Note one will need to set up ssh login to not use passwords between the server running MoonGen and the device under test (running the VSPERF test infrastructure). This is because VSPERF on one server uses ‘ssh’ to configure and run MoonGen upon the other server.

One can set up this ssh access by doing the following on both servers:

ssh-keygen -b 2048 -t rsa
ssh-copy-id <other server>
3.7.2. Configuration

Connection information for MoonGen must be supplied inside the 10_custom.conf or 03_custom.conf file. The following parameters must be set to allow for proper connections to the host with MoonGen.

TRAFFICGEN_MOONGEN_HOST_IP_ADDR = ""
TRAFFICGEN_MOONGEN_USER = ""
TRAFFICGEN_MOONGEN_BASE_DIR = ""
TRAFFICGEN_MOONGEN_PORTS = ""
TRAFFICGEN_MOONGEN_LINE_SPEED_GBPS = ""
3.8. Trex
3.8.1. Installation

Trex architecture overview and general installation instructions can be found here:

https://trex-tgn.cisco.com/trex/doc/trex_stateless.html

You can directly download from GitHub:

git clone https://github.com/cisco-system-traffic-generator/trex-core

and use the master branch:

git checkout master

or Trex latest release you can download from here:

wget --no-cache http://trex-tgn.cisco.com/trex/release/latest

After download, Trex repo has to be built:

cd trex-core/linux_dpdk
./b configure   (run only once)
./b build

Next step is to create a minimum configuration file. It can be created by script dpdk_setup_ports.py. The script with parameter -i will run in interactive mode and it will create file /etc/trex_cfg.yaml.

cd trex-core/scripts
sudo ./dpdk_setup_ports.py -i

Or example of configuration file can be found at location below, but it must be updated manually:

cp trex-core/scripts/cfg/simple_cfg /etc/trex_cfg.yaml

For additional information about configuration file see official documentation (chapter 3.1.2):

https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_creating_minimum_configuration_file

After compilation and configuration it is possible to run trex server in stateless mode. It is neccesary for proper connection between Trex server and VSPERF.

cd trex-core/scripts/
./t-rex-64 -i

For additional information about Trex stateless mode see Trex stateless documentation:

https://trex-tgn.cisco.com/trex/doc/trex_stateless.html

NOTE: One will need to set up ssh login to not use passwords between the server running Trex and the device under test (running the VSPERF test infrastructure). This is because VSPERF on one server uses ‘ssh’ to configure and run Trex upon the other server.

One can set up this ssh access by doing the following on both servers:

ssh-keygen -b 2048 -t rsa
ssh-copy-id <other server>
3.8.2. Configuration

Connection information for Trex must be supplied inside the custom configuration file. The following parameters must be set to allow for proper connections to the host with Trex. Example of this configuration is in conf/03_traffic.conf or conf/10_custom.conf.

TRAFFICGEN_TREX_HOST_IP_ADDR = ''
TRAFFICGEN_TREX_USER = ''
TRAFFICGEN_TREX_BASE_DIR = ''

TRAFFICGEN_TREX_USER has to have sudo permission and password-less access. TRAFFICGEN_TREX_BASE_DIR is the place, where is stored ‘t-rex-64’ file.

It is possible to specify the accuracy of RFC2544 Throughput measurement. Threshold below defines maximal difference between frame rate of successful (i.e. defined frameloss was reached) and unsuccessful (i.e. frameloss was exceeded) iterations.

Default value of this parameter is defined in conf/03_traffic.conf as follows:

TRAFFICGEN_TREX_RFC2544_TPUT_THRESHOLD = ''
3.8.3. SR-IOV and Multistream layer 2

T-Rex by default only accepts packets on the receive side if the destination mac matches the MAC address specified in the /etc/trex-cfg.yaml on the server side. For SR-IOV this creates challenges with modifying the MAC address in the traffic profile to correctly flow packets through specified VFs. To remove this limitation enable promiscuous mode on T-Rex to allow all packets regardless of the destination mac to be accepted.

This also creates problems when doing multistream at layer 2 since the source macs will be modified. Enable Promiscuous mode when doing multistream at layer 2 testing with T-Rex.

TRAFFICGEN_TREX_PROMISCUOUS=True
VSPERF Test Guide
1. vSwitchPerf test suites userguide
1.1. General

VSPERF requires a traffic generators to run tests, automated traffic gen support in VSPERF includes:

  • IXIA traffic generator (IxNetwork hardware) and a machine that runs the IXIA client software.
  • Spirent traffic generator (TestCenter hardware chassis or TestCenter virtual in a VM) and a VM to run the Spirent Virtual Deployment Service image, formerly known as “Spirent LabServer”.
  • Xena Network traffic generator (Xena hardware chassis) that houses the Xena Traffic generator modules.
  • Moongen software traffic generator. Requires a separate machine running moongen to execute packet generation.
  • T-Rex software traffic generator. Requires a separate machine running T-Rex Server to execute packet generation.

If you want to use another traffic generator, please select the Dummy generator.

1.2. VSPERF Installation

To see the supported Operating Systems, vSwitches and system requirements, please follow the installation instructions <vsperf-installation>.

1.3. Traffic Generator Setup

Follow the Traffic generator instructions <trafficgen-installation> to install and configure a suitable traffic generator.

1.4. Cloning and building src dependencies

In order to run VSPERF, you will need to download DPDK and OVS. You can do this manually and build them in a preferred location, OR you could use vswitchperf/src. The vswitchperf/src directory contains makefiles that will allow you to clone and build the libraries that VSPERF depends on, such as DPDK and OVS. To clone and build simply:

$ cd src
$ make

VSPERF can be used with stock OVS (without DPDK support). When build is finished, the libraries are stored in src_vanilla directory.

The ‘make’ builds all options in src:

  • Vanilla OVS
  • OVS with vhost_user as the guest access method (with DPDK support)

The vhost_user build will reside in src/ovs/ The Vanilla OVS build will reside in vswitchperf/src_vanilla

To delete a src subdirectory and its contents to allow you to re-clone simply use:

$ make clobber
1.5. Configure the ./conf/10_custom.conf file

The 10_custom.conf file is the configuration file that overrides default configurations in all the other configuration files in ./conf The supplied 10_custom.conf file MUST be modified, as it contains configuration items for which there are no reasonable default values.

The configuration items that can be added is not limited to the initial contents. Any configuration item mentioned in any .conf file in ./conf directory can be added and that item will be overridden by the custom configuration value.

Further details about configuration files evaluation and special behaviour of options with GUEST_ prefix could be found at design document.

1.6. Using a custom settings file

If your 10_custom.conf doesn’t reside in the ./conf directory of if you want to use an alternative configuration file, the file can be passed to vsperf via the --conf-file argument.

$ ./vsperf --conf-file <path_to_custom_conf> ...

Note that configuration passed in via the environment (--load-env) or via another command line argument will override both the default and your custom configuration files. This “priority hierarchy” can be described like so (1 = max priority):

  1. Testcase definition section Parameters
  2. Command line arguments
  3. Environment variables
  4. Configuration file(s)

Further details about configuration files evaluation and special behaviour of options with GUEST_ prefix could be found at design document.

1.7. Referencing parameter values

It is possible to use a special macro #PARAM() to refer to the value of another configuration parameter. This reference is evaluated during access of the parameter value (by settings.getValue() call), so it can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. It can be used to reflect DUT HW details in the testcase definition.

Example:

{
    ...
    "Name": "testcase",
    "Parameters" : {
        "TRAFFIC" : {
            'l2': {
                # set destination MAC to the MAC of the first
                # interface from WHITELIST_NICS list
                'dstmac' : '#PARAM(NICS[0]["mac"])',
            },
        },
    ...
1.8. Overriding values defined in configuration files

The configuration items can be overridden by command line argument --test-params. In this case, the configuration items and their values should be passed in form of item=value and separated by semicolon.

Example:

$ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \
                         "GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput

The second option is to override configuration items by Parameters section of the test case definition. The configuration items can be added into Parameters dictionary with their new values. These values will override values defined in configuration files or specified by --test-params command line argument.

Example:

"Parameters" : {'TRAFFICGEN_PKT_SIZES' : (128,),
                'TRAFFICGEN_DURATION' : 10,
                'GUEST_LOOPBACK' : ['testpmd','l2fwd'],
               }

NOTE: In both cases, configuration item names and their values must be specified in the same form as they are defined inside configuration files. Parameter names must be specified in uppercase and data types of original and new value must match. Python syntax rules related to data types and structures must be followed. For example, parameter TRAFFICGEN_PKT_SIZES above is defined as a tuple with a single value 128. In this case trailing comma is mandatory, otherwise value can be wrongly interpreted as a number instead of a tuple and vsperf execution would fail. Please check configuration files for default values and their types and use them as a basis for any customized values. In case of any doubt, please check official python documentation related to data structures like tuples, lists and dictionaries.

NOTE: Vsperf execution will terminate with runtime error in case, that unknown parameter name is passed via --test-params CLI argument or defined in Parameters section of test case definition. It is also forbidden to redefine a value of TEST_PARAMS configuration item via CLI or Parameters section.

1.9. vloop_vnf

VSPERF uses a VM image called vloop_vnf for looping traffic in the deployment scenarios involving VMs. The image can be downloaded from http://artifacts.opnfv.org/.

Please see the installation instructions for information on vloop-vnf images.

1.10. l2fwd Kernel Module

A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd

1.11. Executing tests

All examples inside these docs assume, that user is inside the VSPERF directory. VSPERF can be executed from any directory.

Before running any tests make sure you have root permissions by adding the following line to /etc/sudoers:

username ALL=(ALL)       NOPASSWD: ALL

username in the example above should be replaced with a real username.

To list the available tests:

$ ./vsperf --list

To run a single test:

$ ./vsperf $TESTNAME

Where $TESTNAME is the name of the vsperf test you would like to run.

To run a group of tests, for example all tests with a name containing ‘RFC2544’:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests="RFC2544"

To run all tests:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf

Some tests allow for configurable parameters, including test duration (in seconds) as well as packet sizes (in bytes).

$ ./vsperf --conf-file user_settings.py \
    --tests RFC2544Tput \
    --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)"

For all available options, check out the help dialog:

$ ./vsperf --help
1.12. Executing Vanilla OVS tests
  1. If needed, recompile src for all OVS variants

    $ cd src
    $ make distclean
    $ make
    
  2. Update your 10_custom.conf file to use Vanilla OVS:

    VSWITCH = 'OvsVanilla'
    
  3. Run test:

    $ ./vsperf --conf-file=<path_to_custom_conf>
    

    Please note if you don’t want to configure Vanilla OVS through the configuration file, you can pass it as a CLI argument.

    $ ./vsperf --vswitch OvsVanilla
    
1.13. Executing tests with VMs

To run tests using vhost-user as guest access method:

  1. Set VSWITCH and VNF of your settings file to:

    VSWITCH = 'OvsDpdkVhost'
    VNF = 'QemuDpdkVhost'
    
  2. If needed, recompile src for all OVS variants

    $ cd src
    $ make distclean
    $ make
    
  3. Run test:

    $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
    

NOTE: By default vSwitch is acting as a server for dpdk vhost-user sockets. In case, that QEMU should be a server for vhost-user sockets, then parameter VSWITCH_VHOSTUSER_SERVER_MODE should be set to False.

1.14. Executing tests with VMs using Vanilla OVS

To run tests using Vanilla OVS:

  1. Set the following variables:

    VSWITCH = 'OvsVanilla'
    VNF = 'QemuVirtioNet'
    
    VANILLA_TGEN_PORT1_IP = n.n.n.n
    VANILLA_TGEN_PORT1_MAC = nn:nn:nn:nn:nn:nn
    
    VANILLA_TGEN_PORT2_IP = n.n.n.n
    VANILLA_TGEN_PORT2_MAC = nn:nn:nn:nn:nn:nn
    
    VANILLA_BRIDGE_IP = n.n.n.n
    

    or use --test-params option

    $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
               --test-params "VANILLA_TGEN_PORT1_IP=n.n.n.n;" \
                             "VANILLA_TGEN_PORT1_MAC=nn:nn:nn:nn:nn:nn;" \
                             "VANILLA_TGEN_PORT2_IP=n.n.n.n;" \
                             "VANILLA_TGEN_PORT2_MAC=nn:nn:nn:nn:nn:nn"
    
  2. If needed, recompile src for all OVS variants

    $ cd src
    $ make distclean
    $ make
    
  3. Run test:

    $ ./vsperf --conf-file<path_to_custom_conf>/10_custom.conf
    
1.15. Executing VPP tests

Currently it is not possible to use standard scenario deployments for execution of tests with VPP. It means, that deployments p2p, pvp, pvvp and in general any PXP Deployment won’t work with VPP. However it is possible to use VPP in Step driven tests. A basic set of VPP testcases covering phy2phy, pvp and pvvp tests are already prepared.

List of performance tests with VPP support follows:

  • phy2phy_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • phy2phy_cont_vpp: VPP: Phy2Phy Continuous Stream
  • phy2phy_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames
  • pvp_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • pvp_cont_vpp: VPP: PVP Continuous Stream
  • pvp_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames
  • pvvp_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • pvvp_cont_vpp: VPP: PVP Continuous Stream
  • pvvp_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames

In order to execute testcases with VPP it is required to:

After that it is possible to execute VPP testcases listed above.

For example:

$ ./vsperf --conf-file=<path_to_custom_conf> phy2phy_tput_vpp
1.16. Using vfio_pci with DPDK

To use vfio with DPDK instead of igb_uio add into your custom configuration file the following parameter:

PATHS['dpdk']['src']['modules'] = ['uio', 'vfio-pci']

NOTE: In case, that DPDK is installed from binary package, then please set PATHS['dpdk']['bin']['modules'] instead.

NOTE: Please ensure that Intel VT-d is enabled in BIOS.

NOTE: Please ensure your boot/grub parameters include the following:

NOTE: In case of VPP, it is required to explicitly define, that vfio-pci DPDK driver should be used. It means to update dpdk part of VSWITCH_VPP_ARGS dictionary with uio-driver section, e.g. VSWITCH_VPP_ARGS[‘dpdk’] = ‘uio-driver vfio-pci’

iommu=pt intel_iommu=on

To check that IOMMU is enabled on your platform:

$ dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.139882] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.139888] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.139893] IOAPIC id 2 under DRHD base  0xfbffe000 IOMMU 0
[    0.139894] IOAPIC id 0 under DRHD base  0xebffc000 IOMMU 1
[    0.139895] IOAPIC id 1 under DRHD base  0xebffc000 IOMMU 1
[    3.335744] IOMMU: dmar0 using Queued invalidation
[    3.335746] IOMMU: dmar1 using Queued invalidation
....
1.17. Using SRIOV support

To use virtual functions of NIC with SRIOV support, use extended form of NIC PCI slot definition:

WHITELIST_NICS = ['0000:05:00.0|vf0', '0000:05:00.1|vf3']

Where ‘vf’ is an indication of virtual function usage and following number defines a VF to be used. In case that VF usage is detected, then vswitchperf will enable SRIOV support for given card and it will detect PCI slot numbers of selected VFs.

So in example above, one VF will be configured for NIC ‘0000:05:00.0’ and four VFs will be configured for NIC ‘0000:05:00.1’. Vswitchperf will detect PCI addresses of selected VFs and it will use them during test execution.

At the end of vswitchperf execution, SRIOV support will be disabled.

SRIOV support is generic and it can be used in different testing scenarios. For example:

  • vSwitch tests with DPDK or without DPDK support to verify impact of VF usage on vSwitch performance
  • tests without vSwitch, where traffic is forwarded directly between VF interfaces by packet forwarder (e.g. testpmd application)
  • tests without vSwitch, where VM accesses VF interfaces directly by PCI-passthrough to measure raw VM throughput performance.
1.18. Using QEMU with PCI passthrough support

Raw virtual machine throughput performance can be measured by execution of PVP test with direct access to NICs by PCI pass-through. To execute VM with direct access to PCI devices, enable vfio-pci. In order to use virtual functions, SRIOV-support must be enabled.

Execution of test with PCI pass-through with vswitch disabled:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
           --vswitch none --vnf QemuPciPassthrough pvp_tput

Any of supported guest-loopback-application can be used inside VM with PCI pass-through support.

Note: Qemu with PCI pass-through support can be used only with PVP test deployment.

1.19. Selection of loopback application for tests with VMs

To select the loopback applications which will forward packets inside VMs, the following parameter should be configured:

GUEST_LOOPBACK = ['testpmd']

or use --test-params CLI argument:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
      --test-params "GUEST_LOOPBACK=['testpmd']"

Supported loopback applications are:

'testpmd'       - testpmd from dpdk will be built and used
'l2fwd'         - l2fwd module provided by Huawei will be built and used
'linux_bridge'  - linux bridge will be configured
'buildin'       - nothing will be configured by vsperf; VM image must
                  ensure traffic forwarding between its interfaces

Guest loopback application must be configured, otherwise traffic will not be forwarded by VM and testcases with VM related deployments will fail. Guest loopback application is set to ‘testpmd’ by default.

NOTE: In case that only 1 or more than 2 NICs are configured for VM, then ‘testpmd’ should be used. As it is able to forward traffic between multiple VM NIC pairs.

NOTE: In case of linux_bridge, all guest NICs are connected to the same bridge inside the guest.

1.20. Mergable Buffers Options with QEMU

Mergable buffers can be disabled with VSPerf within QEMU. This option can increase performance significantly when not using jumbo frame sized packets. By default VSPerf disables mergable buffers. If you wish to enable it you can modify the setting in the a custom conf file.

GUEST_NIC_MERGE_BUFFERS_DISABLE = [False]

Then execute using the custom conf file.

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf

Alternatively you can just pass the param during execution.

$ ./vsperf --test-params "GUEST_NIC_MERGE_BUFFERS_DISABLE=[False]"
1.21. Selection of dpdk binding driver for tests with VMs

To select dpdk binding driver, which will specify which driver the vm NICs will use for dpdk bind, the following configuration parameter should be configured:

GUEST_DPDK_BIND_DRIVER = ['igb_uio_from_src']

The supported dpdk guest bind drivers are:

'uio_pci_generic'      - Use uio_pci_generic driver
'igb_uio_from_src'     - Build and use the igb_uio driver from the dpdk src
                         files
'vfio_no_iommu'        - Use vfio with no iommu option. This requires custom
                         guest images that support this option. The default
                         vloop image does not support this driver.

Note: uio_pci_generic does not support sr-iov testcases with guests attached. This is because uio_pci_generic only supports legacy interrupts. In case uio_pci_generic is selected with the vnf as QemuPciPassthrough it will be modified to use igb_uio_from_src instead.

Note: vfio_no_iommu requires kernels equal to or greater than 4.5 and dpdk 16.04 or greater. Using this option will also taint the kernel.

Please refer to the dpdk documents at http://dpdk.org/doc/guides for more information on these drivers.

1.22. Guest Core and Thread Binding

VSPERF provides options to achieve better performance by guest core binding and guest vCPU thread binding as well. Core binding is to bind all the qemu threads. Thread binding is to bind the house keeping threads to some CPU and vCPU thread to some other CPU, this helps to reduce the noise from qemu house keeping threads.

GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]

NOTE By default the GUEST_THREAD_BINDING will be none, which means same as the GUEST_CORE_BINDING, i.e. the vcpu threads are sharing the physical CPUs with the house keeping threads. Better performance using vCPU thread binding can be achieved by enabling affinity in the custom configuration file.

For example, if an environment requires 32,33 to be core binded and 29,30&31 for guest thread binding to achieve better performance.

VNF_AFFINITIZATION_ON = True
GUEST_CORE_BINDING = [('32','33')]
GUEST_THREAD_BINDING = [('29', '30', '31')]
1.23. Qemu CPU features

QEMU default to a compatible subset of performance enhancing cpu features. To pass all available host processor features to the guest.

GUEST_CPU_OPTIONS = ['host,migratable=off']

NOTE To enhance the performance, cpu features tsc deadline timer for guest, the guest PMU, the invariant TSC can be provided in the custom configuration file.

1.24. Multi-Queue Configuration

VSPerf currently supports multi-queue with the following limitations:

  1. Requires QEMU 2.5 or greater and any OVS version higher than 2.5. The default upstream package versions installed by VSPerf satisfies this requirement.

  2. Guest image must have ethtool utility installed if using l2fwd or linux bridge inside guest for loopback.

  3. If using OVS versions 2.5.0 or less enable old style multi-queue as shown in the ‘‘02_vswitch.conf’’ file.

    OVS_OLD_STYLE_MQ = True
    

To enable multi-queue for dpdk modify the ‘‘02_vswitch.conf’’ file.

VSWITCH_DPDK_MULTI_QUEUES = 2

NOTE: you should consider using the switch affinity to set a pmd cpu mask that can optimize your performance. Consider the numa of the NIC in use if this applies by checking /sys/class/net/<eth_name>/device/numa_node and setting an appropriate mask to create PMD threads on the same numa node.

When multi-queue is enabled, each dpdk or dpdkvhostuser port that is created on the switch will set the option for multiple queues. If old style multi queue has been enabled a global option for multi queue will be used instead of the port by port option.

To enable multi-queue on the guest modify the ‘‘04_vnf.conf’’ file.

GUEST_NIC_QUEUES = [2]

Enabling multi-queue at the guest will add multiple queues to each NIC port when qemu launches the guest.

In case of Vanilla OVS, multi-queue is enabled on the tuntap ports and nic queues will be enabled inside the guest with ethtool. Simply enabling the multi-queue on the guest is sufficient for Vanilla OVS multi-queue.

Testpmd should be configured to take advantage of multi-queue on the guest if using DPDKVhostUser. This can be done by modifying the ‘‘04_vnf.conf’’ file.

GUEST_TESTPMD_PARAMS = ['-l 0,1,2,3,4  -n 4 --socket-mem 512 -- '
                        '--burst=64 -i --txqflags=0xf00 '
                        '--nb-cores=4 --rxq=2 --txq=2 '
                        '--disable-hw-vlan']

NOTE: The guest SMP cores must be configured to allow for testpmd to use the optimal number of cores to take advantage of the multiple guest queues.

In case of using Vanilla OVS and qemu virtio-net you can increase performance by binding vhost-net threads to cpus. This can be done by enabling the affinity in the ‘‘04_vnf.conf’’ file. This can be done to non multi-queue enabled configurations as well as there will be 2 vhost-net threads.

VSWITCH_VHOST_NET_AFFINITIZATION = True

VSWITCH_VHOST_CPU_MAP = [4,5,8,11]

NOTE: This method of binding would require a custom script in a real environment.

NOTE: For optimal performance guest SMPs and/or vhost-net threads should be on the same numa as the NIC in use if possible/applicable. Testpmd should be assigned at least (nb_cores +1) total cores with the cpu mask.

1.25. Jumbo Frame Testing

VSPERF provides options to support jumbo frame testing with a jumbo frame supported NIC and traffic generator for the following vswitches:

  1. OVSVanilla
  2. OvsDpdkVhostUser
  3. TestPMD loopback with or without a guest

NOTE: There is currently no support for SR-IOV or VPP at this time with jumbo frames.

All packet forwarding applications for pxp testing is supported.

To enable jumbo frame testing simply enable the option in the conf files and set the maximum size that will be used.

VSWITCH_JUMBO_FRAMES_ENABLED = True
VSWITCH_JUMBO_FRAMES_SIZE = 9000

To enable jumbo frame testing with OVSVanilla the NIC in test on the host must have its mtu size changed manually using ifconfig or applicable tools:

ifconfig eth1 mtu 9000 up

NOTE: To make the setting consistent across reboots you should reference the OS documents as it differs from distribution to distribution.

To start a test for jumbo frames modify the conf file packet sizes or pass the option through the VSPERF command line.

TEST_PARAMS = {'TRAFFICGEN_PKT_SIZES':(2000,9000)}
./vsperf --test-params "TRAFFICGEN_PKT_SIZES=2000,9000"

It is recommended to increase the memory size for OvsDpdkVhostUser testing from the default 1024. Your size required may vary depending on the number of guests in your testing. 4096 appears to work well for most typical testing scenarios.

DPDK_SOCKET_MEM = ['4096', '0']

NOTE: For Jumbo frames to work with DpdkVhostUser, mergable buffers will be enabled by default. If testing with mergable buffers in QEMU is desired, disable Jumbo Frames and only test non jumbo frame sizes. Test Jumbo Frames sizes separately to avoid this collision.

1.26. Executing Packet Forwarding tests

To select the applications which will forward packets, the following parameters should be configured:

VSWITCH = 'none'
PKTFWD = 'TestPMD'

or use --vswitch and --fwdapp CLI arguments:

$ ./vsperf phy2phy_cont --conf-file user_settings.py \
           --vswitch none \
           --fwdapp TestPMD

Supported Packet Forwarding applications are:

'testpmd'       - testpmd from dpdk
  1. Update your ‘‘10_custom.conf’’ file to use the appropriate variables for selected Packet Forwarder:

    # testpmd configuration
    TESTPMD_ARGS = []
    # packet forwarding mode supported by testpmd; Please see DPDK documentation
    # for comprehensive list of modes supported by your version.
    # e.g. io|mac|mac_retry|macswap|flowgen|rxonly|txonly|csum|icmpecho|...
    # Note: Option "mac_retry" has been changed to "mac retry" since DPDK v16.07
    TESTPMD_FWD_MODE = 'csum'
    # checksum calculation layer: ip|udp|tcp|sctp|outer-ip
    TESTPMD_CSUM_LAYER = 'ip'
    # checksum calculation place: hw (hardware) | sw (software)
    TESTPMD_CSUM_CALC = 'sw'
    # recognize tunnel headers: on|off
    TESTPMD_CSUM_PARSE_TUNNEL = 'off'
    
  2. Run test:

    $ ./vsperf phy2phy_tput --conf-file <path_to_settings_py>
    
1.27. Executing Packet Forwarding tests with one guest

TestPMD with DPDK 16.11 or greater can be used to forward packets as a switch to a single guest using TestPMD vdev option. To set this configuration the following parameters should be used.

VSWITCH = 'none'
PKTFWD = 'TestPMD'

or use --vswitch and --fwdapp CLI arguments:

$ ./vsperf pvp_tput --conf-file user_settings.py \
           --vswitch none \
           --fwdapp TestPMD

Guest forwarding application only supports TestPMD in this configuration.

GUEST_LOOPBACK = ['testpmd']

For optimal performance one cpu per port +1 should be used for TestPMD. Also set additional params for packet forwarding application to use the correct number of nb-cores.

DPDK_SOCKET_MEM = ['1024', '0']
VSWITCHD_DPDK_ARGS = ['-l', '46,44,42,40,38', '-n', '4']
TESTPMD_ARGS = ['--nb-cores=4', '--txq=1', '--rxq=1']

For guest TestPMD 3 VCpus should be assigned with the following TestPMD params.

GUEST_TESTPMD_PARAMS = ['-l 0,1,2 -n 4 --socket-mem 1024 -- '
                        '--burst=64 -i --txqflags=0xf00 '
                        '--disable-hw-vlan --nb-cores=2 --txq=1 --rxq=1']

Execution of TestPMD can be run with the following command line

./vsperf pvp_tput --vswitch=none --fwdapp=TestPMD --conf-file <path_to_settings_py>

NOTE: To achieve the best 0% loss numbers with rfc2544 throughput testing, other tunings should be applied to host and guest such as tuned profiles and CPU tunings to prevent possible interrupts to worker threads.

1.28. VSPERF modes of operation

VSPERF can be run in different modes. By default it will configure vSwitch, traffic generator and VNF. However it can be used just for configuration and execution of traffic generator. Another option is execution of all components except traffic generator itself.

Mode of operation is driven by configuration parameter -m or –mode

-m MODE, --mode MODE  vsperf mode of operation;
    Values:
        "normal" - execute vSwitch, VNF and traffic generator
        "trafficgen" - execute only traffic generator
        "trafficgen-off" - execute vSwitch and VNF
        "trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission

In case, that VSPERF is executed in “trafficgen” mode, then configuration of traffic generator can be modified through TRAFFIC dictionary passed to the --test-params option. It is not needed to specify all values of TRAFFIC dictionary. It is sufficient to specify only values, which should be changed. Detailed description of TRAFFIC dictionary can be found at Configuration of TRAFFIC dictionary.

Example of execution of VSPERF in “trafficgen” mode:

$ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \
    --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}"
1.29. Code change verification by pylint

Every developer participating in VSPERF project should run pylint before his python code is submitted for review. Project specific configuration for pylint is available at ‘pylint.rc’.

Example of manual pylint invocation:

$ pylint --rcfile ./pylintrc ./vsperf
1.30. GOTCHAs:
1.30.1. Custom image fails to boot

Using custom VM images may not boot within VSPerf pxp testing because of the drive boot and shared type which could be caused by a missing scsi driver inside the image. In case of issues you can try changing the drive boot type to ide.

GUEST_BOOT_DRIVE_TYPE = ['ide']
GUEST_SHARED_DRIVE_TYPE = ['ide']
1.30.2. OVS with DPDK and QEMU

If you encounter the following error: “before (last 100 chars): ‘-path=/dev/hugepages,share=on: unable to map backing store for hugepages: Cannot allocate memoryrnrn” during qemu initialization, check the amount of hugepages on your system:

$ cat /proc/meminfo | grep HugePages

By default the vswitchd is launched with 1Gb of memory, to change this, modify –socket-mem parameter in conf/02_vswitch.conf to allocate an appropriate amount of memory:

DPDK_SOCKET_MEM = ['1024', '0']
VSWITCHD_DPDK_ARGS = ['-c', '0x4', '-n', '4']
VSWITCHD_DPDK_CONFIG = {
    'dpdk-init' : 'true',
    'dpdk-lcore-mask' : '0x4',
    'dpdk-socket-mem' : '1024,0',
}

Note: Option VSWITCHD_DPDK_ARGS is used for vswitchd, which supports --dpdk parameter. In recent vswitchd versions, option VSWITCHD_DPDK_CONFIG will be used to configure vswitchd via ovs-vsctl calls.

1.31. More information

For more information and details refer to the rest of vSwitchPerfuser documentation.

2. Step driven tests

In general, test scenarios are defined by a deployment used in the particular test case definition. The chosen deployment scenario will take care of the vSwitch configuration, deployment of VNFs and it can also affect configuration of a traffic generator. In order to allow a more flexible way of testcase scripting, VSPERF supports a detailed step driven testcase definition. It can be used to configure and program vSwitch, deploy and terminate VNFs, execute a traffic generator, modify a VSPERF configuration, execute external commands, etc.

Execution of step driven tests is done on a step by step work flow starting with step 0 as defined inside the test case. Each step of the test increments the step number by one which is indicated in the log.

(testcases.integration) - Step 0 'vswitch add_vport ['br0']' start

Step driven tests can be used for both performance and integration testing. In case of integration test, each step in the test case is validated. If a step does not pass validation the test will fail and terminate. The test will continue until a failure is detected or all steps pass. A csv report file is generated after a test completes with an OK or FAIL result.

NOTE: It is possible to suppress validation process of given step by prefixing it by ! (exclamation mark). In following example test execution won’t fail if all traffic is dropped:

['!trafficgen', 'send_traffic', {}]

In case of performance test, the validation of steps is not performed and standard output files with results from traffic generator and underlying OS details are generated by vsperf.

Step driven testcases can be used in two different ways:

# description of full testcase - in this case clean deployment is used
to indicate that vsperf should neither configure vSwitch nor deploy any VNF. Test shall perform all required vSwitch configuration and programming and deploy required number of VNFs.
# modification of existing deployment - in this case, any of supported
deployments can be used to perform initial vSwitch configuration and deployment of VNFs. Additional actions defined by TestSteps can be used to alter vSwitch configuration or deploy additional VNFs. After the last step is processed, the test execution will continue with traffic execution.
2.1. Test objects and their functions

Every test step can call a function of one of the supported test objects. The list of supported objects and their most common functions follows:

  • vswitch - provides functions for vSwitch configuration

    List of supported functions:

    • add_switch br_name - creates a new switch (bridge) with given br_name
    • del_switch br_name - deletes switch (bridge) with given br_name
    • add_phy_port br_name - adds a physical port into bridge specified by br_name
    • add_vport br_name - adds a virtual port into bridge specified by br_name
    • del_port br_name port_name - removes physical or virtual port specified by port_name from bridge br_name
    • add_flow br_name flow - adds flow specified by flow dictionary into the bridge br_name; Content of flow dictionary will be passed to the vSwitch. In case of Open vSwitch it will be passed to the ovs-ofctl add-flow command. Please see Open vSwitch documentation for the list of supported flow parameters.
    • del_flow br_name [flow] - deletes flow specified by flow dictionary from bridge br_name; In case that optional parameter flow is not specified or set to an empty dictionary {}, then all flows from bridge br_name will be deleted.
    • dump_flows br_name - dumps all flows from bridge specified by br_name
    • enable_stp br_name - enables Spanning Tree Protocol for bridge br_name
    • disable_stp br_name - disables Spanning Tree Protocol for bridge br_name
    • enable_rstp br_name - enables Rapid Spanning Tree Protocol for bridge br_name
    • disable_rstp br_name - disables Rapid Spanning Tree Protocol for bridge br_name
    • restart - restarts switch, which is useful for failover testcases

    Examples:

    ['vswitch', 'add_switch', 'int_br0']
    
    ['vswitch', 'del_switch', 'int_br0']
    
    ['vswitch', 'add_phy_port', 'int_br0']
    
    ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]']
    
    ['vswitch', 'add_flow', 'int_br0', {'in_port': '1', 'actions': ['output:2'],
     'idle_timeout': '0'}],
    
    ['vswitch', 'enable_rstp', 'int_br0']
    
  • vnf[ID] - provides functions for deployment and termination of VNFs; Optional alfanumerical ID is used for VNF identification in case that testcase deploys multiple VNFs.

    List of supported functions:

    • start - starts a VNF based on VSPERF configuration
    • stop - gracefully terminates given VNF
    • execute command [delay] - executes command cmd inside VNF; Optional delay defines number of seconds to wait before next step is executed. Method returns command output as a string.
    • execute_and_wait command [timeout] [prompt] - executes command cmd inside VNF; Optional timeout defines number of seconds to wait until prompt is detected. Optional prompt defines a string, which is used as detection of successful command execution. In case that prompt is not defined, then content of GUEST_PROMPT_LOGIN parameter will be used. Method returns command output as a string.

    Examples:

    ['vnf1', 'start'],
    ['vnf2', 'start'],
    ['vnf1', 'execute_and_wait', 'ifconfig eth0 5.5.5.1/24 up'],
    ['vnf2', 'execute_and_wait', 'ifconfig eth0 5.5.5.2/24 up', 120, 'root.*#'],
    ['vnf2', 'execute_and_wait', 'ping -c1 5.5.5.1'],
    ['vnf2', 'stop'],
    ['vnf1', 'stop'],
    
  • trafficgen - triggers traffic generation

    List of supported functions:

    • send_traffic traffic - starts a traffic based on the vsperf configuration and given traffic dictionary. More details about traffic dictionary and its possible values are available at Traffic Generator Integration Guide
    • get_results - returns dictionary with results collected from previous execution of send_traffic

    Examples:

    ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_throughput'}]
    
    ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}],
    ['trafficgen', 'get_results'],
    ['tools', 'assert', '#STEP[-1][0]["frame_loss_percent"] < 0.05'],
    
  • settings - reads or modifies VSPERF configuration

    List of supported functions:

    • getValue param - returns value of given param
    • setValue param value - sets value of param to given value

    Examples:

    ['settings', 'getValue', 'TOOLS']
    
    ['settings', 'setValue', 'GUEST_USERNAME', ['root']]
    

    It is possible and more convenient to access any VSPERF configuration option directly via $NAME notation. Option evaluation is done during runtime and vsperf will automatically translate it to the appropriate call of settings.getValue. If the referred parameter does not exist, then vsperf will keep $NAME string untouched and it will continue with testcase execution. The reason is to avoid test execution failure in case that $ sign has been used from different reason than vsperf parameter evaluation.

    NOTE: It is recommended to use ${NAME} notation for any shell parameters used within Exec_Shell call to avoid a clash with configuration parameter evaluation.

    NOTE: It is possible to refer to vsperf parameter value by #PARAM() macro (see Referencing parameter values. However #PARAM() macro is evaluated at the beginning of vsperf execution and it will not reflect any changes made to the vsperf configuration during runtime. On the other hand $NAME notation is evaluated during test execution and thus it contains any modifications to the configuration parameter made by vsperf (e.g. TOOLS and NICS dictionaries) or by testcase definition (e.g. TRAFFIC dictionary).

    Examples:

    ['tools', 'exec_shell', "$TOOLS['ovs-vsctl'] show"]
    
    ['settings', 'setValue', 'TRAFFICGEN_IXIA_PORT2', '$TRAFFICGEN_IXIA_PORT1'],
    
    ['vswitch', 'add_flow', 'int_br0',
     {'in_port': '#STEP[1][1]',
      'dl_type': '0x800',
      'nw_proto': '17',
      'nw_dst': '$TRAFFIC["l3"]["dstip"]/8',
      'actions': ['output:#STEP[2][1]']
     }
    ]
    
  • namespace - creates or modifies network namespaces

    List of supported functions:

    • create_namespace name - creates new namespace with given name
    • delete_namespace name - deletes namespace specified by its name
    • assign_port_to_namespace port name [port_up] - assigns NIC specified by port into given namespace name; If optional parameter port_up is set to True, then port will be brought up.
    • add_ip_to_namespace_eth port name addr cidr - assigns an IP address addr/cidr to the NIC specified by port within namespace name
    • reset_port_to_root port name - returns given port from namespace name back to the root namespace

    Examples:

    ['namespace', 'create_namespace', 'testns']
    
    ['namespace', 'assign_port_to_namespace', 'eth0', 'testns']
    
  • veth - manipulates with eth and veth devices

    List of supported functions:

    • add_veth_port port peer_port - adds a pair of veth ports named port and peer_port
    • del_veth_port port peer_port - deletes a veth port pair specified by port and peer_port
    • bring_up_eth_port eth_port [namespace] - brings up eth_port in (optional) namespace

    Examples:

    ['veth', 'add_veth_port', 'veth', 'veth1']
    
    ['veth', 'bring_up_eth_port', 'eth1']
    
  • tools - provides a set of helper functions

    List of supported functions:

    • Assert condition - evaluates given condition and raises AssertionError in case that condition is not True
    • Eval expression - evaluates given expression as a python code and returns its result
    • Exec_Shell command - executes a shell command
    • Exec_Python code - executes a python code

    Examples:

    ['tools', 'exec_shell', 'numactl -H', 'available: ([0-9]+)']
    ['tools', 'assert', '#STEP[-1][0]>1']
    
  • wait - is used for test case interruption. This object doesn’t have any functions. Once reached, vsperf will pause test execution and waits for press of Enter key. It can be used during testcase design for debugging purposes.

    Examples:

    ['wait']
    
  • sleep - is used to pause testcase execution for defined number of seconds.

    Examples:

    ['sleep', '60']
    
  • log level message - is used to log message of given level into vsperf output. Level is one of info, debug, warning or error.

    Examples:

    ['log', 'error', 'tools $TOOLS']
    
  • pdb - executes python debugger

    Examples:

    ['pdb']
    
2.2. Test Macros

Test profiles can include macros as part of the test step. Each step in the profile may return a value such as a port name. Recall macros use #STEP to indicate the recalled value inside the return structure. If the method the test step calls returns a value it can be later recalled, for example:

{
    "Name": "vswitch_add_del_vport",
    "Deployment": "clean",
    "Description": "vSwitch - add and delete virtual port",
    "TestSteps": [
            ['vswitch', 'add_switch', 'int_br0'],               # STEP 0
            ['vswitch', 'add_vport', 'int_br0'],                # STEP 1
            ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],  # STEP 2
            ['vswitch', 'del_switch', 'int_br0'],               # STEP 3
         ]
}

This test profile uses the vswitch add_vport method which returns a string value of the port added. This is later called by the del_port method using the name from step 1.

It is also possible to use negative indexes in step macros. In that case #STEP[-1] will refer to the result from previous step, #STEP[-2] will refer to result of step called before previous step, etc. It means, that you could change STEP 2 from previous example to achieve the same functionality:

['vswitch', 'del_port', 'int_br0', '#STEP[-1][0]'],  # STEP 2

Another option to refer to previous values, is to define an alias for given step by its first argument with ‘#’ prefix. Alias must be unique and it can’t be a number. Example of step alias usage:

['#port1', 'vswitch', 'add_vport', 'int_br0'],
['vswitch', 'del_port', 'int_br0', '#STEP[port1][0]'],

Also commonly used steps can be created as a separate profile.

STEP_VSWITCH_PVP_INIT = [
    ['vswitch', 'add_switch', 'int_br0'],           # STEP 0
    ['vswitch', 'add_phy_port', 'int_br0'],         # STEP 1
    ['vswitch', 'add_phy_port', 'int_br0'],         # STEP 2
    ['vswitch', 'add_vport', 'int_br0'],            # STEP 3
    ['vswitch', 'add_vport', 'int_br0'],            # STEP 4
]

This profile can then be used inside other testcases

{
    "Name": "vswitch_pvp",
    "Deployment": "clean",
    "Description": "vSwitch - configure switch and one vnf",
    "TestSteps": STEP_VSWITCH_PVP_INIT +
                 [
                    ['vnf', 'start'],
                    ['vnf', 'stop'],
                 ] +
                 STEP_VSWITCH_PVP_FINIT
}

It is possible to refer to vsperf configuration parameters within step macros. Please see step-driven-tests-variable-usage for more details.

In case that step returns a string or list of strings, then it is possible to filter such output by regular expression. This optional filter can be specified as a last step parameter with prefix ‘|’. Output will be split into separate lines and only matching records will be returned. It is also possible to return a specified group of characters from the matching lines, e.g. by regex |ID (\d+).

Examples:

['tools', 'exec_shell', "sudo $TOOLS['ovs-appctl'] dpif-netdev/pmd-rxq-show",
 '|dpdkvhostuser0\s+queue-id: \d'],
['tools', 'assert', 'len(#STEP[-1])==1'],

['vnf', 'execute_and_wait', 'ethtool -L eth0 combined 2'],
['vnf', 'execute_and_wait', 'ethtool -l eth0', '|Combined:\s+2'],
['tools', 'assert', 'len(#STEP[-1])==2']
2.3. HelloWorld and other basic Testcases

The following examples are for demonstration purposes. You can run them by copying and pasting into the conf/integration/01_testcases.conf file. A command-line instruction is shown at the end of each example.

2.3.1. HelloWorld

The first example is a HelloWorld testcase. It simply creates a bridge with 2 physical ports, then sets up a flow to drop incoming packets from the port that was instantiated at the STEP #1. There’s no interaction with the traffic generator. Then the flow, the 2 ports and the bridge are deleted. ‘add_phy_port’ method creates a ‘dpdk’ type interface that will manage the physical port. The string value returned is the port name that will be referred by ‘del_port’ later on.

{
    "Name": "HelloWorld",
    "Description": "My first testcase",
    "Deployment": "clean",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],   # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'actions': ['drop'], 'idle_timeout': '0'}],
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]

},

To run HelloWorld test:

./vsperf --conf-file user_settings.py --integration HelloWorld
2.3.2. Specify a Flow by the IP address

The next example shows how to explicitly set up a flow by specifying a destination IP address. All packets received from the port created at STEP #1 that have a destination IP address = 90.90.90.90 will be forwarded to the port created at the STEP #2.

{
    "Name": "p2p_rule_l3da",
    "Description": "Phy2Phy with rule on L3 Dest Addr",
    "Deployment": "clean",
    "biDirectional": "False",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],   # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_dst': '90.90.90.90', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous'}],
        ['vswitch', 'dump_flows', 'int_br0'],   # STEP 5
        ['vswitch', 'del_flow', 'int_br0'],     # STEP 7 == del-flows
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration p2p_rule_l3da
2.3.3. Multistream feature

The next testcase uses the multistream feature. The traffic generator will send packets with different UDP ports. That is accomplished by using “Stream Type” and “MultiStream” keywords. 4 different flows are set to forward all incoming packets.

{
    "Name": "multistream_l4",
    "Description": "Multistream on UDP ports",
    "Deployment": "clean",
    "Parameters": {
        'TRAFFIC' : {
            "multistream": 4,
            "stream_type": "L4",
        },
    },
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],   # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
        # Setup Flows
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '0', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '1', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '2', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '3', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        # Send mono-dir traffic
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous', \
            'bidir' : 'False'}],
        # Clean up
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
     ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration multistream_l4
2.3.4. PVP with a VM Replacement

This example launches a 1st VM in a PVP topology, then the VM is replaced by another VM. When VNF setup parameter in ./conf/04_vnf.conf is “QemuDpdkVhostUser” ‘add_vport’ method creates a ‘dpdkvhostuser’ type port to connect a VM.

{
    "Name": "ex_replace_vm",
    "Description": "PVP with VM replacement",
    "Deployment": "clean",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],       # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 3    vm1
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 4

        # Setup Flows
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[2][1]', \
            'actions': ['output:#STEP[4][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[3][1]', \
            'actions': ['output:#STEP[1][1]'], 'idle_timeout': '0'}],

        # Start VM 1
        ['vnf1', 'start'],
        # Now we want to replace VM 1 with another VM
        ['vnf1', 'stop'],

        ['vswitch', 'add_vport', 'int_br0'],        # STEP 11    vm2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 12
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'actions': ['output:#STEP[11][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[12][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],

        # Start VM 2
        ['vnf2', 'start'],
        ['vnf2', 'stop'],
        ['vswitch', 'dump_flows', 'int_br0'],

        # Clean up
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'],    # vm1
        ['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[11][0]'],   # vm2
        ['vswitch', 'del_port', 'int_br0', '#STEP[12][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration ex_replace_vm
2.3.5. VM with a Linux bridge

This example setups a PVP topology and routes traffic to the VM based on the destination IP address. A command-line parameter is used to select a Linux bridge as a guest loopback application. It is also possible to select a guest loopback application by a configuration option GUEST_LOOPBACK.

{
    "Name": "ex_pvp_rule_l3da",
    "Description": "PVP with flow on L3 Dest Addr",
    "Deployment": "clean",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],       # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 3    vm1
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 4
        # Setup Flows
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_dst': '90.90.90.90', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        # Each pkt from the VM is forwarded to the 2nd dpdk port
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        # Start VMs
        ['vnf1', 'start'],
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous', \
            'bidir' : 'False'}],
        ['vnf1', 'stop'],
        # Clean up
        ['vswitch', 'dump_flows', 'int_br0'],       # STEP 10
        ['vswitch', 'del_flow', 'int_br0'],         # STEP 11
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'],  # vm1 ports
        ['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --test-params \
        "GUEST_LOOPBACK=['linux_bridge']" --integration ex_pvp_rule_l3da
2.3.6. Forward packets based on UDP port

This examples launches 2 VMs connected in parallel. Incoming packets will be forwarded to one specific VM depending on the destination UDP port.

{
    "Name": "ex_2pvp_rule_l4dp",
    "Description": "2 PVP with flows on L4 Dest Port",
    "Deployment": "clean",
    "Parameters": {
        'TRAFFIC' : {
            "multistream": 2,
            "stream_type": "L4",
        },
    },
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],       # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 3    vm1
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 4
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 5    vm2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 6
        # Setup Flows to reply ICMPv6 and similar packets, so to
        # avoid flooding internal port with their re-transmissions
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:01', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:02', \
            'actions': ['output:#STEP[4][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:03', \
            'actions': ['output:#STEP[5][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:04', \
            'actions': ['output:#STEP[6][1]'], 'idle_timeout': '0'}],
        # Forward UDP packets depending on dest port
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '0', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '1', \
            'actions': ['output:#STEP[5][1]'], 'idle_timeout': '0'}],
        # Send VM output to phy port #2
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[6][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        # Start VMs
        ['vnf1', 'start'],                          # STEP 16
        ['vnf2', 'start'],                          # STEP 17
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous', \
            'bidir' : 'False'}],
        ['vnf1', 'stop'],
        ['vnf2', 'stop'],
        ['vswitch', 'dump_flows', 'int_br0'],
        # Clean up
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'],  # vm1 ports
        ['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[5][0]'],  # vm2 ports
        ['vswitch', 'del_port', 'int_br0', '#STEP[6][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration ex_2pvp_rule_l4dp
2.3.7. Modification of existing PVVP deployment

This is an example of modification of a standard deployment scenario with additional TestSteps. Standard PVVP scenario is used to configure a vSwitch and to deploy two VNFs connected in series. Additional TestSteps will deploy a 3rd VNF and connect it in parallel to already configured VNFs. Traffic generator is instructed (by Multistream feature) to send two separate traffic streams. One stream will be sent to the standalone VNF and second to two chained VNFs.

In case, that test is defined as a performance test, then traffic results will be collected and available in both csv and rst report files.

{
    "Name": "pvvp_pvp_cont",
    "Deployment": "pvvp",
    "Description": "PVVP and PVP in parallel with Continuous Stream",
    "Parameters" : {
        "TRAFFIC" : {
            "traffic_type" : "rfc2544_continuous",
            "multistream": 2,
        },
    },
    "TestSteps": [
                    ['vswitch', 'add_vport', 'br0'],
                    ['vswitch', 'add_vport', 'br0'],
                    # priority must be higher than default 32768, otherwise flows won't match
                    ['vswitch', 'add_flow', 'br0',
                     {'in_port': '1', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800',
                                                  'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}],
                    ['vswitch', 'add_flow', 'br0',
                     {'in_port': '2', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800',
                                                  'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}],
                    ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'],
                                                    'idle_timeout': '0'}],
                    ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'],
                                                    'idle_timeout': '0'}],
                    ['vswitch', 'dump_flows', 'br0'],
                    ['vnf1', 'start'],
                 ]
},

To run the test:

./vsperf --conf-file user_settings.py pvvp_pvp_cont
3. Integration tests

VSPERF includes a set of integration tests defined in conf/integration. These tests can be run by specifying –integration as a parameter to vsperf. Current tests in conf/integration include switch functionality and Overlay tests.

Tests in the conf/integration can be used to test scaling of different switch configurations by adding steps into the test case.

For the overlay tests VSPERF supports VXLAN, GRE and GENEVE tunneling protocols. Testing of these protocols is limited to unidirectional traffic and P2P (Physical to Physical scenarios).

NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only.

3.1. Executing Integration Tests

To execute integration tests VSPERF is run with the integration parameter. To view the current test list simply execute the following command:

./vsperf --integration --list

The standard tests included are defined inside the conf/integration/01_testcases.conf file.

3.2. Executing Tunnel encapsulation tests

The VXLAN OVS DPDK encapsulation tests requires IPs, MAC addresses, bridge names and WHITELIST_NICS for DPDK.

NOTE: Only Ixia traffic generators currently support the execution of the tunnel encapsulation tests. Support for other traffic generators may come in a future release.

Default values are already provided. To customize for your environment, override the following variables in you user_settings.py file:

# Variables defined in conf/integration/02_vswitch.conf
# Tunnel endpoint for Overlay P2P deployment scenario
# used for br0
VTEP_IP1 = '192.168.0.1/24'

# Used as remote_ip in adding OVS tunnel port and
# to set ARP entry in OVS (e.g. tnl/arp/set br-ext 192.168.240.10 02:00:00:00:00:02
VTEP_IP2 = '192.168.240.10'

# Network to use when adding a route for inner frame data
VTEP_IP2_SUBNET = '192.168.240.0/24'

# Bridge names
TUNNEL_INTEGRATION_BRIDGE = 'br0'
TUNNEL_EXTERNAL_BRIDGE = 'br-ext'

# IP of br-ext
TUNNEL_EXTERNAL_BRIDGE_IP = '192.168.240.1/24'

# vxlan|gre|geneve
TUNNEL_TYPE = 'vxlan'

# Variables defined conf/integration/03_traffic.conf
# For OP2P deployment scenario
TRAFFICGEN_PORT1_MAC = '02:00:00:00:00:01'
TRAFFICGEN_PORT2_MAC = '02:00:00:00:00:02'
TRAFFICGEN_PORT1_IP = '1.1.1.1'
TRAFFICGEN_PORT2_IP = '192.168.240.10'

To run VXLAN encapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=vxlan' overlay_p2p_tput

To run GRE encapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=gre' overlay_p2p_tput

To run GENEVE encapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=geneve' overlay_p2p_tput

To run OVS NATIVE tunnel tests (VXLAN/GRE/GENEVE):

  1. Install the OVS kernel modules
cd src/ovs/ovs
sudo -E make modules_install
  1. Set the following variables:
VSWITCH = 'OvsVanilla'
# Specify vport_* kernel module to test.
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_vxlan',
    'vport_gre',
    'vport_geneve',
    'datapath/linux/openvswitch.ko',
]

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run tests:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=vxlan' overlay_p2p_tput
3.3. Executing VXLAN decapsulation tests

To run VXLAN decapsulation tests:

  1. Set the variables used in “Executing Tunnel encapsulation tests”
  2. Run test:
./vsperf --conf-file user_settings.py --integration overlay_p2p_decap_cont

If you want to use different values for your VXLAN frame, you may set:

VXLAN_FRAME_L3 = {'proto': 'udp',
                  'packetsize': 64,
                  'srcip': TRAFFICGEN_PORT1_IP,
                  'dstip': '192.168.240.1',
                 }
VXLAN_FRAME_L4 = {'srcport': 4789,
                  'dstport': 4789,
                  'vni': VXLAN_VNI,
                  'inner_srcmac': '01:02:03:04:05:06',
                  'inner_dstmac': '06:05:04:03:02:01',
                  'inner_srcip': '192.168.0.10',
                  'inner_dstip': '192.168.240.9',
                  'inner_proto': 'udp',
                  'inner_srcport': 3000,
                  'inner_dstport': 3001,
                 }
3.4. Executing GRE decapsulation tests

To run GRE decapsulation tests:

  1. Set the variables used in “Executing Tunnel encapsulation tests”
  2. Run test:
./vsperf --conf-file user_settings.py --test-params 'TUNNEL_TYPE=gre' \
         --integration overlay_p2p_decap_cont

If you want to use different values for your GRE frame, you may set:

GRE_FRAME_L3 = {'proto': 'gre',
                'packetsize': 64,
                'srcip': TRAFFICGEN_PORT1_IP,
                'dstip': '192.168.240.1',
               }

GRE_FRAME_L4 = {'srcport': 0,
                'dstport': 0
                'inner_srcmac': '01:02:03:04:05:06',
                'inner_dstmac': '06:05:04:03:02:01',
                'inner_srcip': '192.168.0.10',
                'inner_dstip': '192.168.240.9',
                'inner_proto': 'udp',
                'inner_srcport': 3000,
                'inner_dstport': 3001,
               }
3.5. Executing GENEVE decapsulation tests

IxNet 7.3X does not have native support of GENEVE protocol. The template, GeneveIxNetTemplate.xml_ClearText.xml, should be imported into IxNET for this testcase to work.

To import the template do:

  1. Run the IxNetwork TCL Server
  2. Click on the Traffic menu
  3. Click on the Traffic actions and click Edit Packet Templates
  4. On the Template editor window, click Import. Select the template located at 3rd_party/ixia/GeneveIxNetTemplate.xml_ClearText.xml and click import.
  5. Restart the TCL Server.

To run GENEVE decapsulation tests:

  1. Set the variables used in “Executing Tunnel encapsulation tests”
  2. Run test:
./vsperf --conf-file user_settings.py --test-params 'tunnel_type=geneve' \
         --integration overlay_p2p_decap_cont

If you want to use different values for your GENEVE frame, you may set:

GENEVE_FRAME_L3 = {'proto': 'udp',
                   'packetsize': 64,
                   'srcip': TRAFFICGEN_PORT1_IP,
                   'dstip': '192.168.240.1',
                  }

GENEVE_FRAME_L4 = {'srcport': 6081,
                   'dstport': 6081,
                   'geneve_vni': 0,
                   'inner_srcmac': '01:02:03:04:05:06',
                   'inner_dstmac': '06:05:04:03:02:01',
                   'inner_srcip': '192.168.0.10',
                   'inner_dstip': '192.168.240.9',
                   'inner_proto': 'udp',
                   'inner_srcport': 3000,
                   'inner_dstport': 3001,
                  }
3.6. Executing Native/Vanilla OVS VXLAN decapsulation tests

To run VXLAN decapsulation tests:

  1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_vxlan',
    'datapath/linux/openvswitch.ko',
]

TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'

VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'

VXLAN_FRAME_L2 = {'srcmac':
                  '01:02:03:04:05:06',
                  'dstmac':
                  '06:05:04:03:02:01',
                 }

VXLAN_FRAME_L3 = {'proto': 'udp',
                  'packetsize': 64,
                  'srcip': TRAFFICGEN_PORT1_IP,
                  'dstip': '172.16.1.1',
                 }

VXLAN_FRAME_L4 = {
                  'srcport': 4789,
                  'dstport': 4789,
                  'protocolpad': 'true',
                  'vni': 99,
                  'inner_srcmac': '01:02:03:04:05:06',
                  'inner_dstmac': '06:05:04:03:02:01',
                  'inner_srcip': '192.168.1.2',
                  'inner_dstip': TRAFFICGEN_PORT2_IP,
                  'inner_proto': 'udp',
                  'inner_srcport': 3000,
                  'inner_dstport': 3001,
                 }

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run test:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'tunnel_type=vxlan' overlay_p2p_decap_cont
3.7. Executing Native/Vanilla OVS GRE decapsulation tests

To run GRE decapsulation tests:

  1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_gre',
    'datapath/linux/openvswitch.ko',
]

TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'

VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'

GRE_FRAME_L2 = {'srcmac':
                '01:02:03:04:05:06',
                'dstmac':
                '06:05:04:03:02:01',
               }

GRE_FRAME_L3 = {'proto': 'udp',
                'packetsize': 64,
                'srcip': TRAFFICGEN_PORT1_IP,
                'dstip': '172.16.1.1',
               }

GRE_FRAME_L4 = {
                'srcport': 4789,
                'dstport': 4789,
                'protocolpad': 'true',
                'inner_srcmac': '01:02:03:04:05:06',
                'inner_dstmac': '06:05:04:03:02:01',
                'inner_srcip': '192.168.1.2',
                'inner_dstip': TRAFFICGEN_PORT2_IP,
                'inner_proto': 'udp',
                'inner_srcport': 3000,
                'inner_dstport': 3001,
               }

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run test:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'tunnel_type=gre' overlay_p2p_decap_cont
3.8. Executing Native/Vanilla OVS GENEVE decapsulation tests

To run GENEVE decapsulation tests:

  1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_geneve',
    'datapath/linux/openvswitch.ko',
]

TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'

VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'

GENEVE_FRAME_L2 = {'srcmac':
                   '01:02:03:04:05:06',
                   'dstmac':
                   '06:05:04:03:02:01',
                  }

GENEVE_FRAME_L3 = {'proto': 'udp',
                   'packetsize': 64,
                   'srcip': TRAFFICGEN_PORT1_IP,
                   'dstip': '172.16.1.1',
                  }

GENEVE_FRAME_L4 = {'srcport': 6081,
                   'dstport': 6081,
                   'protocolpad': 'true',
                   'geneve_vni': 0,
                   'inner_srcmac': '01:02:03:04:05:06',
                   'inner_dstmac': '06:05:04:03:02:01',
                   'inner_srcip': '192.168.1.2',
                   'inner_dstip': TRAFFICGEN_PORT2_IP,
                   'inner_proto': 'udp',
                   'inner_srcport': 3000,
                   'inner_dstport': 3001,
                  }

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run test:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'tunnel_type=geneve' overlay_p2p_decap_cont
3.9. Executing Tunnel encapsulation+decapsulation tests

The OVS DPDK encapsulation/decapsulation tests requires IPs, MAC addresses, bridge names and WHITELIST_NICS for DPDK.

The test cases can test the tunneling encap and decap without using any ingress overlay traffic as compared to above test cases. To achieve this the OVS is configured to perform encap and decap in a series on the same traffic stream as given below.

TRAFFIC-IN –> [ENCAP] –> [MOD-PKT] –> [DECAP] –> TRAFFIC-OUT

Default values are already provided. To customize for your environment, override the following variables in you user_settings.py file:

# Variables defined in conf/integration/02_vswitch.conf

# Bridge names
TUNNEL_EXTERNAL_BRIDGE1 = 'br-phy1'
TUNNEL_EXTERNAL_BRIDGE2 = 'br-phy2'
TUNNEL_MODIFY_BRIDGE1 = 'br-mod1'
TUNNEL_MODIFY_BRIDGE2 = 'br-mod2'

# IP of br-mod1
TUNNEL_MODIFY_BRIDGE_IP1 = '10.0.0.1/24'

# Mac of br-mod1
TUNNEL_MODIFY_BRIDGE_MAC1 = '00:00:10:00:00:01'

# IP of br-mod2
TUNNEL_MODIFY_BRIDGE_IP2 = '20.0.0.1/24'

#Mac of br-mod2
TUNNEL_MODIFY_BRIDGE_MAC2 = '00:00:20:00:00:01'

# vxlan|gre|geneve, Only VXLAN is supported for now.
TUNNEL_TYPE = 'vxlan'

To run VXLAN encapsulation+decapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         overlay_p2p_mod_tput
4. Execution of vswitchperf testcases by Yardstick
4.1. General

Yardstick is a generic framework for a test execution, which is used for validation of installation of OPNFV platform. In the future, Yardstick will support two options of vswitchperf testcase execution:

  • plugin mode, which will execute native vswitchperf testcases; Tests will be executed natively by vsperf, and test results will be processed and reported by yardstick.
  • traffic generator mode, which will run vswitchperf in trafficgen mode only; Yardstick framework will be used to launch VNFs and to configure flows to ensure, that traffic is properly routed. This mode will allow to test OVS performance in real world scenarios.

In Colorado release only the traffic generator mode is supported.

4.2. Yardstick Installation

In order to run Yardstick testcases, you will need to prepare your test environment. Please follow the installation instructions to install the yardstick.

Please note, that yardstick uses OpenStack for execution of testcases. OpenStack must be installed with Heat and Neutron services. Otherwise vswitchperf testcases cannot be executed.

4.3. VM image with vswitchperf

A special VM image is required for execution of vswitchperf specific testcases by yardstick. It is possible to use a sample VM image available at OPNFV artifactory or to build customized image.

4.3.1. Sample VM image with vswitchperf

Sample VM image is available at vswitchperf section of OPNFV artifactory for free download:

$ wget http://artifacts.opnfv.org/vswitchperf/vnf/vsperf-yardstick-image.qcow2

This image can be used for execution of sample testcases with dummy traffic generator.

NOTE: Traffic generators might require an installation of client software. This software is not included in the sample image and must be installed by user.

NOTE: This image will be updated only in case, that new features related to yardstick integration will be added to the vswitchperf.

4.3.2. Preparation of custom VM image

In general, any Linux distribution supported by vswitchperf can be used as a base image for vswitchperf. One of the possibilities is to modify vloop-vnf image, which can be downloaded from http://artifacts.opnfv.org/vswitchperf.html/ (see vloop-vnf).

Please follow the Installing vswitchperf to install vswitchperf inside vloop-vnf image. As vswitchperf will be run in trafficgen mode, it is possible to skip installation and compilation of OVS, QEMU and DPDK to keep image size smaller.

In case, that selected traffic generator requires installation of additional client software, please follow appropriate documentation. For example in case of IXIA, you would need to install IxOS and IxNetowrk TCL API.

4.3.3. VM image usage

Image with vswitchperf must be uploaded into the glance service and vswitchperf specific flavor configured, e.g.:

$ glance --os-username admin --os-image-api-version 1 image-create --name \
  vsperf --is-public true --disk-format qcow2 --container-format bare --file \
  vsperf-yardstick-image.qcow2

$ nova --os-username admin flavor-create vsperf-flavor 100 2048 25 1
4.4. Testcase execution

After installation, yardstick is available as python package within yardstick specific virtual environment. It means, that yardstick environment must be enabled before the test execution, e.g.:

source ~/yardstick_venv/bin/activate

Next step is configuration of OpenStack environment, e.g. in case of devstack:

source /opt/openstack/devstack/openrc
export EXTERNAL_NETWORK=public

Vswitchperf testcases executable by yardstick are located at vswitchperf repository inside yardstick/tests directory. Example of their download and execution follows:

git clone https://gerrit.opnfv.org/gerrit/vswitchperf
cd vswitchperf

yardstick -d task start yardstick/tests/rfc2544_throughput_dummy.yaml

NOTE: Optional argument -d shows debug output.

4.5. Testcase customization

Yardstick testcases are described by YAML files. vswitchperf specific testcases are part of the vswitchperf repository and their yaml files can be found at yardstick/tests directory. For detailed description of yaml file structure, please see yardstick documentation and testcase samples. Only vswitchperf specific parts will be discussed here.

Example of yaml file:

...
scenarios:
-
  type: Vsperf
  options:
    testname: 'p2p_rfc2544_throughput'
    trafficgen_port1: 'eth1'
    trafficgen_port2: 'eth3'
    external_bridge: 'br-ex'
    test_params: 'TRAFFICGEN_DURATION=30;TRAFFIC={'traffic_type':'rfc2544_throughput}'
    conf_file: '~/vsperf-yardstick.conf'

  host: vsperf.demo

  runner:
    type: Sequence
    scenario_option_name: frame_size
    sequence:
    - 64
    - 128
    - 512
    - 1024
    - 1518
  sla:
    metrics: 'throughput_rx_fps'
    throughput_rx_fps: 500000
    action: monitor

context:
...
4.5.1. Section option

Section option defines details of vswitchperf test scenario. Lot of options are identical to the vswitchperf parameters passed through --test-params argument. Following options are supported:

  • frame_size - a packet size for which test should be executed; Multiple packet sizes can be tested by modification of Sequence runner section inside YAML definition. Default: ‘64’
  • conf_file - sets path to the vswitchperf configuration file, which will be uploaded to VM; Default: ‘~/vsperf-yardstick.conf’
  • setup_script - sets path to the setup script, which will be executed during setup and teardown phases
  • trafficgen_port1 - specifies device name of 1st interface connected to the trafficgen
  • trafficgen_port2 - specifies device name of 2nd interface connected to the trafficgen
  • external_bridge - specifies name of external bridge configured in OVS; Default: ‘br-ex’
  • test_params - specifies a string with a list of vsperf configuration parameters, which will be passed to the --test-params CLI argument; Parameters should be stated in the form of param=value and separated by a semicolon. Configuration of traffic generator is driven by TRAFFIC dictionary, which can be also updated by values defined by test_params. Please check VSPERF documentation for details about available configuration parameters and their data types. In case that both test_params and conf_file are specified, then values from test_params will override values defined in the configuration file.

In case that trafficgen_port1 and/or trafficgen_port2 are defined, then these interfaces will be inserted into the external_bridge of OVS. It is expected, that OVS runs at the same node, where the testcase is executed. In case of more complex OpenStack installation or a need of additional OVS configuration, setup_script can be used.

NOTE It is essential to specify a configuration for selected traffic generator. In case, that standalone testcase is created, then traffic generator can be selected and configured directly in YAML file by test_params. On the other hand, if multiple testcases should be executed with the same traffic generator settings, then a customized configuration file should be prepared and its name passed by conf_file option.

4.5.2. Section runner

Yardstick supports several runner types. In case of vswitchperf specific TCs, Sequence runner type can be used to execute the testcase for given list of frame sizes.

4.5.3. Section sla

In case that sla section is not defined, then testcase will be always considered as successful. On the other hand, it is possible to define a set of test metrics and their minimal values to evaluate test success. Any numeric value, reported by vswitchperf inside CSV result file, can be used. Multiple metrics can be defined as a coma separated list of items. Minimal value must be set separately for each metric.

e.g.:

sla:
    metrics: 'throughput_rx_fps,throughput_rx_mbps'
    throughput_rx_fps: 500000
    throughput_rx_mbps: 1000

In case that any of defined metrics will be lower than defined value, then testcase will be marked as failed. Based on action policy, yardstick will either stop test execution (value assert) or it will run next test (value monitor).

NOTE The throughput SLA (or any other SLA) cannot be set to a meaningful value without knowledge of the server and networking environment, possibly including prior testing in that environment to establish a baseline SLA level under well-understood circumstances.

5. List of vswitchperf testcases
5.1. Performance testcases
Testcase Name Description
phy2phy_tput LTD.Throughput.RFC2544.PacketLossRatio
phy2phy_forwarding LTD.Forwarding.RFC2889.MaxForwardingRate
phy2phy_learning LTD.AddrLearning.RFC2889.AddrLearningRate
phy2phy_caching LTD.AddrCaching.RFC2889.AddrCachingCapacity
back2back LTD.Throughput.RFC2544.BackToBackFrames
phy2phy_tput_mod_vlan LTD.Throughput.RFC2544.PacketLossRatioFrameModification
phy2phy_cont Phy2Phy Continuous Stream
pvp_cont PVP Continuous Stream
pvvp_cont PVVP Continuous Stream
pvpv_cont Two VMs in parallel with Continuous Stream
phy2phy_scalability LTD.Scalability.Flows.RFC2544.0PacketLoss
pvp_tput LTD.Throughput.RFC2544.PacketLossRatio
pvp_back2back LTD.Throughput.RFC2544.BackToBackFrames
pvvp_tput LTD.Throughput.RFC2544.PacketLossRatio
pvvp_back2back LTD.Throughput.RFC2544.BackToBackFrames
phy2phy_cpu_load LTD.CPU.RFC2544.0PacketLoss
phy2phy_mem_load LTD.Memory.RFC2544.0PacketLoss
phy2phy_tput_vpp VPP: LTD.Throughput.RFC2544.PacketLossRatio
phy2phy_cont_vpp VPP: Phy2Phy Continuous Stream
phy2phy_back2back_vpp VPP: LTD.Throughput.RFC2544.BackToBackFrames
pvp_tput_vpp VPP: LTD.Throughput.RFC2544.PacketLossRatio
pvp_cont_vpp VPP: PVP Continuous Stream
pvp_back2back_vpp VPP: LTD.Throughput.RFC2544.BackToBackFrames
pvvp_tput_vpp VPP: LTD.Throughput.RFC2544.PacketLossRatio
pvvp_cont_vpp VPP: PVP Continuous Stream
pvvp_back2back_vpp VPP: LTD.Throughput.RFC2544.BackToBackFrames

List of performance testcases above can be obtained by execution of:

$ ./vsperf --list
5.2. Integration testcases
Testcase Name Description
vswitch_vports_add_del_flow vSwitch - configure switch with vports, add and delete flow
vswitch_add_del_flows vSwitch - add and delete flows
vswitch_p2p_tput vSwitch - configure switch and execute RFC2544 throughput test
vswitch_p2p_back2back vSwitch - configure switch and execute RFC2544 back2back test
vswitch_p2p_cont vSwitch - configure switch and execute RFC2544 continuous stream test
vswitch_pvp vSwitch - configure switch and one vnf
vswitch_vports_pvp vSwitch - configure switch with vports and one vnf
vswitch_pvp_tput vSwitch - configure switch, vnf and execute RFC2544 throughput test
vswitch_pvp_back2back vSwitch - configure switch, vnf and execute RFC2544 back2back test
vswitch_pvp_cont vSwitch - configure switch, vnf and execute RFC2544 continuous stream test
vswitch_pvp_all vSwitch - configure switch, vnf and execute all test types
vswitch_pvvp vSwitch - configure switch and two vnfs
vswitch_pvvp_tput vSwitch - configure switch, two chained vnfs and execute RFC2544 throughput test
vswitch_pvvp_back2back vSwitch - configure switch, two chained vnfs and execute RFC2544 back2back test
vswitch_pvvp_cont vSwitch - configure switch, two chained vnfs and execute RFC2544 continuous stream test
vswitch_pvvp_all vSwitch - configure switch, two chained vnfs and execute all test types
vswitch_p4vp Just configure 4 chained vnfs
vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test
vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test
vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test
vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput test
2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port
4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port
6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port
vhost_numa_awareness vSwitch DPDK - verify that PMD threads are served by the same NUMA slot as QEMU instances
ixnet_pvp_tput_1nic PVP Scenario with 1 port towards IXIA
vswitch_vports_add_del_connection_vpp VPP: vSwitch - configure switch with vports, add and delete connection
p2p_l3_multi_IP_ovs OVS: P2P L3 multistream with unique flow for each IP stream
p2p_l3_multi_IP_mask_ovs OVS: P2P L3 multistream with 1 flow for /8 net mask
pvp_l3_multi_IP_mask_ovs OVS: PVP L3 multistream with 1 flow for /8 net mask
pvvp_l3_multi_IP_mask_ovs OVS: PVVP L3 multistream with 1 flow for /8 net mask
p2p_l4_multi_PORT_ovs OVS: P2P L4 multistream with unique flow for each IP stream
p2p_l4_multi_PORT_mask_ovs OVS: P2P L4 multistream with 1 flow for /8 net and port mask
pvp_l4_multi_PORT_mask_ovs OVS: PVP L4 multistream flows for /8 net and port mask
pvvp_l4_multi_PORT_mask_ovs OVS: PVVP L4 multistream with flows for /8 net and port mask
p2p_l3_multi_IP_arp_vpp VPP: P2P L3 multistream with unique ARP entry for each IP stream
p2p_l3_multi_IP_mask_vpp VPP: P2P L3 multistream with 1 route for /8 net mask
p2p_l3_multi_IP_routes_vpp VPP: P2P L3 multistream with unique route for each IP stream
pvp_l3_multi_IP_mask_vpp VPP: PVP L3 multistream with route for /8 netmask
pvvp_l3_multi_IP_mask_vpp VPP: PVVP L3 multistream with route for /8 netmask
p2p_l4_multi_PORT_arp_vpp VPP: P2P L4 multistream with unique ARP entry for each IP stream and port check
p2p_l4_multi_PORT_mask_vpp VPP: P2P L4 multistream with 1 route for /8 net mask and port check
p2p_l4_multi_PORT_routes_vpp VPP: P2P L4 multistream with unique route for each IP stream and port check
pvp_l4_multi_PORT_mask_vpp VPP: PVP L4 multistream with route for /8 net and port mask
pvvp_l4_multi_PORT_mask_vpp VPP: PVVP L4 multistream with route for /8 net and port mask
vxlan_multi_IP_mask_ovs OVS: VxLAN L3 multistream
vxlan_multi_IP_arp_vpp VPP: VxLAN L3 multistream with unique ARP entry for each IP stream
vxlan_multi_IP_mask_vpp VPP: VxLAN L3 multistream with 1 route for /8 netmask

List of integration testcases above can be obtained by execution of:

$ ./vsperf --integration --list
5.3. OVS/DPDK Regression TestCases

These regression tests verify several DPDK features used internally by Open vSwitch. Tests can be used for verification of performance and correct functionality of upcoming DPDK and OVS releases and release candidates.

These tests are part of integration testcases and they must be executed with --integration CLI parameter.

Example of execution of all OVS/DPDK regression tests:

$ ./vsperf --integration --tests ovsdpdk_

Testcases are defined in the file conf/integration/01b_dpdk_regression_tests.conf. This file contains a set of configuration options with prefix OVSDPDK_. These parameters can be used for customization of regression tests and they will override some of standard VSPERF configuration options. It is recommended to check OVSDPDK configuration parameters and modify them in accordance with VSPERF configuration.

At least following parameters should be examined. Their values shall ensure, that DPDK and QEMU threads are pinned to cpu cores of the same NUMA slot, where tested NICs are connected.

_OVSDPDK_1st_PMD_CORE
_OVSDPDK_2nd_PMD_CORE
_OVSDPDK_GUEST_5_CORES
5.3.1. DPDK NIC Support

A set of performance tests to verify support of DPDK accelerated network interface cards. Testcases use standard physical to physical network scenario with several vSwitch and traffic configurations, which includes one and two PMD threads, uni and bidirectional traffic and RFC2544 Continuous or RFC2544 Throughput with 0% packet loss traffic types.

Testcase Name Description
ovsdpdk_nic_p2p_single_pmd_unidir_cont P2P with single PMD in OVS and unidirectional traffic.
ovsdpdk_nic_p2p_single_pmd_bidir_cont P2P with single PMD in OVS and bidirectional traffic.
ovsdpdk_nic_p2p_two_pmd_bidir_cont P2P with two PMDs in OVS and bidirectional traffic.
ovsdpdk_nic_p2p_single_pmd_unidir_tput P2P with single PMD in OVS and unidirectional traffic.
ovsdpdk_nic_p2p_single_pmd_bidir_tput P2P with single PMD in OVS and bidirectional traffic.
ovsdpdk_nic_p2p_two_pmd_bidir_tput P2P with two PMDs in OVS and bidirectional traffic.
5.3.2. DPDK Hotplug Support

A set of functional tests to verify DPDK hotplug support. Tests verify, that it is possible to use port, which was not bound to DPDK driver during vSwitch startup. There is also a test which verifies a possibility to detach port from DPDK driver. However support for manual detachment of a port from DPDK has been removed from recent OVS versions and thus this testcase is expected to fail.

Testcase Name Description
ovsdpdk_hotplug_attach Ensure successful port-add after binding a device to igb_uio after ovs-vswitchd is launched.
ovsdpdk_hotplug_detach Same as ovsdpdk_hotplug_attach, but delete and detach the device after the hotplug. Note Support of netdev-dpdk/detach has been removed from OVS, so testcase will fail with recent OVS/DPDK versions.
5.3.3. RX Checksum Support

A set of functional tests for verification of RX checksum calculation for tunneled traffic. Open vSwitch enables RX checksum offloading by default if NIC supports it. It is to note, that it is not possible to disable or enable RX checksum offloading. In order to verify correct RX checksum calculation in software, user has to execute these testcases at NIC without HW offloading capabilities.

Testcases utilize existing overlay physical to physical (op2p) network deployment implemented in vsperf. This deployment expects, that traffic generator sends unidirectional tunneled traffic (e.g. vxlan) and Open vSwitch performs data decapsulation and sends them back to the traffic generator via second port.

Testcase Name Description
ovsdpdk_checksum_l3 Test verifies RX IP header checksum (offloading) validation for tunneling protocols.
ovsdpdk_checksum_l4 Test verifies RX UDP header checksum (offloading) validation for tunneling protocols.
5.3.4. Flow Control Support

A set of functional testcases for the validation of flow control support in Open vSwitch with DPDK support. If flow control is enabled in both OVS and Traffic Generator, the network endpoint (OVS or TGEN) is not able to process incoming data and thus it detects a RX buffer overflow. It then sends an ethernet pause frame (as defined at 802.3x) to the TX side. This mechanism will ensure, that the TX side will slow down traffic transmission and thus no data is lost at RX side.

Introduced testcases use physical to physical scenario to forward data between traffic generator ports. It is expected that the processing of small frames in OVS is slower than line rate. It means that with flow control disabled, traffic generator will report a frame loss. On the other hand with flow control enabled, there should be 0% frame loss reported by traffic generator.

Testcase Name Description
ovsdpdk_flow_ctrl_rx Test the rx flow control functionality of DPDK PHY ports.
ovsdpdk_flow_ctrl_rx_dynamic Change the rx flow control support at run time and ensure the system honored the changes.
5.3.5. Multiqueue Support

A set of functional testcases for validation of multiqueue support for both physical and vHost User DPDK ports. Testcases utilize P2P and PVP network deployments and native support of multiqueue configuration available in VSPERF.

Testcase Name Description
ovsdpdk_mq_p2p_rxqs Setup rxqs on NIC port.
ovsdpdk_mq_p2p_rxqs_same_core_affinity Affinitize rxqs to the same core.
ovsdpdk_mq_p2p_rxqs_multi_core_affinity Affinitize rxqs to separate cores.
ovsdpdk_mq_pvp_rxqs Setup rxqs on vhost user port.
ovsdpdk_mq_pvp_rxqs_linux_bridge Confirm traffic received over vhost RXQs with Linux virtio device in guest.
ovsdpdk_mq_pvp_rxqs_testpmd Confirm traffic received over vhost RXQs with DPDK device in guest.
5.3.6. Vhost User

A set of functional testcases for validation of vHost User Client and vHost User Server modes in OVS.

NOTE: Vhost User Server mode is deprecated and it will be removed from OVS in the future.

Testcase Name Description
ovsdpdk_vhostuser_client Test vhost-user client mode
ovsdpdk_vhostuser_client_reconnect Test vhost-user client mode reconnect feature
ovsdpdk_vhostuser_server Test vhost-user server mode
ovsdpdk_vhostuser_sock_dir Verify functionality of vhost-sock-dir flag
5.3.7. Virtual Devices Support

A set of functional testcases for verification of correct functionality of virtual device PMD drivers.

Testcase Name Description
ovsdpdk_vdev_add_null_pmd Test addition of port using the null DPDK PMD driver.
ovsdpdk_vdev_del_null_pmd Test deletion of port using the null DPDK PMD driver.
ovsdpdk_vdev_add_af_packet_pmd Test addition of port using the af_packet DPDK PMD driver.
ovsdpdk_vdev_del_af_packet_pmd Test deletion of port using the af_packet DPDK PMD driver.
5.3.8. NUMA Support

A functional testcase for validation of NUMA awareness feature in OVS.

Testcase Name Description
ovsdpdk_numa Test vhost-user NUMA support. Vhostuser PMD threads should migrate to the same numa slot, where QEMU is executed.
5.3.9. Jumbo Frame Support

A set of functional testcases for verification of jumbo frame support in OVS. Testcases utilize P2P and PVP network deployments and native support of jumbo frames available in VSPERF.

Testcase Name Description
ovsdpdk_jumbo_increase_mtu_phy_port_ovsdb Ensure that the increased MTU for a DPDK physical port is updated in OVSDB.
ovsdpdk_jumbo_increase_mtu_vport_ovsdb Ensure that the increased MTU for a DPDK vhost-user port is updated in OVSDB.
ovsdpdk_jumbo_reduce_mtu_phy_port_ovsdb Ensure that the reduced MTU for a DPDK physical port is updated in OVSDB.
ovsdpdk_jumbo_reduce_mtu_vport_ovsdb Ensure that the reduced MTU for a DPDK vhost-user port is updated in OVSDB.
ovsdpdk_jumbo_increase_mtu_phy_port_datapath Ensure that the MTU for a DPDK physical port is updated in the datapath itself when increased to a valid value.
ovsdpdk_jumbo_increase_mtu_vport_datapath Ensure that the MTU for a DPDK vhost-user port is updated in the datapath itself when increased to a valid value.
ovsdpdk_jumbo_reduce_mtu_phy_port_datapath Ensure that the MTU for a DPDK physical port is updated in the datapath itself when decreased to a valid value.
ovsdpdk_jumbo_reduce_mtu_vport_datapath Ensure that the MTU for a DPDK vhost-user port is updated in the datapath itself when decreased to a valid value.
ovsdpdk_jumbo_mtu_upper_bound_phy_port Verify that the upper bound limit is enforced for OvS DPDK Phy ports.
ovsdpdk_jumbo_mtu_upper_bound_vport Verify that the upper bound limit is enforced for OvS DPDK vhost-user ports.
ovsdpdk_jumbo_mtu_lower_bound_phy_port Verify that the lower bound limit is enforced for OvS DPDK Phy ports.
ovsdpdk_jumbo_mtu_lower_bound_vport Verify that the lower bound limit is enforced for OvS DPDK vhost-user ports.
ovsdpdk_jumbo_p2p Ensure that jumbo frames are received, processed and forwarded correctly by DPDK physical ports.
ovsdpdk_jumbo_pvp Ensure that jumbo frames are received, processed and forwarded correctly by DPDK vhost-user ports.
ovsdpdk_jumbo_p2p_upper_bound Ensure that jumbo frames above the configured Rx port’s MTU are not accepted
5.3.10. Rate Limiting

A set of functional testcases for validation of rate limiting support. This feature allows to configure an ingress policing for both physical and vHost User DPDK ports.

NOTE: Desired maximum rate is specified in kilo bits per second and it defines the rate of payload only.

Testcase Name Description
ovsdpdk_rate_create_phy_port Ensure a rate limiting interface can be created on a physical DPDK port.
ovsdpdk_rate_delete_phy_port Ensure a rate limiting interface can be destroyed on a physical DPDK port.
ovsdpdk_rate_create_vport Ensure a rate limiting interface can be created on a vhost-user port.
ovsdpdk_rate_delete_vport Ensure a rate limiting interface can be destroyed on a vhost-user port.
ovsdpdk_rate_no_policing Ensure when a user attempts to create a rate limiting interface but is missing policing rate argument, no rate limitiner is created.
ovsdpdk_rate_no_burst Ensure when a user attempts to create a rate limiting interface but is missing policing burst argument, rate limitiner is created.
ovsdpdk_rate_p2p Ensure when a user creates a rate limiting physical interface that the traffic is limited to the specified policer rate in a p2p setup.
ovsdpdk_rate_pvp Ensure when a user creates a rate limiting vHost User interface that the traffic is limited to the specified policer rate in a pvp setup.
ovsdpdk_rate_p2p_multi_pkt_sizes Ensure that rate limiting works for various frame sizes.
5.3.11. Quality of Service

A set of functional testcases for validation of QoS support. This feature allows to configure an egress policing for both physical and vHost User DPDK ports.

NOTE: Desired maximum rate is specified in bytes per second and it defines the rate of payload only.

Testcase Name Description
ovsdpdk_qos_create_phy_port Ensure a QoS policy can be created on a physical DPDK port
ovsdpdk_qos_delete_phy_port Ensure an existing QoS policy can be destroyed on a physical DPDK port.
ovsdpdk_qos_create_vport Ensure a QoS policy can be created on a virtual vhost user port.
ovsdpdk_qos_delete_vport Ensure an existing QoS policy can be destroyed on a vhost user port.
ovsdpdk_qos_create_no_cir Ensure that a QoS policy cannot be created if the egress policer cir argument is missing.
ovsdpdk_qos_create_no_cbs Ensure that a QoS policy cannot be created if the egress policer cbs argument is missing.
ovsdpdk_qos_p2p In a p2p setup, ensure when a QoS egress policer is created that the traffic is limited to the specified rate.
ovsdpdk_qos_pvp In a pvp setup, ensure when a QoS egress policer is created that the traffic is limited to the specified rate.
VSPERF Test Guide
1. vSwitchPerf test suites userguide
1.1. General

VSPERF requires a traffic generators to run tests, automated traffic gen support in VSPERF includes:

  • IXIA traffic generator (IxNetwork hardware) and a machine that runs the IXIA client software.
  • Spirent traffic generator (TestCenter hardware chassis or TestCenter virtual in a VM) and a VM to run the Spirent Virtual Deployment Service image, formerly known as “Spirent LabServer”.
  • Xena Network traffic generator (Xena hardware chassis) that houses the Xena Traffic generator modules.
  • Moongen software traffic generator. Requires a separate machine running moongen to execute packet generation.
  • T-Rex software traffic generator. Requires a separate machine running T-Rex Server to execute packet generation.

If you want to use another traffic generator, please select the Dummy generator.

1.2. VSPERF Installation

To see the supported Operating Systems, vSwitches and system requirements, please follow the installation instructions <vsperf-installation>.

1.3. Traffic Generator Setup

Follow the Traffic generator instructions <trafficgen-installation> to install and configure a suitable traffic generator.

1.4. Cloning and building src dependencies

In order to run VSPERF, you will need to download DPDK and OVS. You can do this manually and build them in a preferred location, OR you could use vswitchperf/src. The vswitchperf/src directory contains makefiles that will allow you to clone and build the libraries that VSPERF depends on, such as DPDK and OVS. To clone and build simply:

$ cd src
$ make

VSPERF can be used with stock OVS (without DPDK support). When build is finished, the libraries are stored in src_vanilla directory.

The ‘make’ builds all options in src:

  • Vanilla OVS
  • OVS with vhost_user as the guest access method (with DPDK support)

The vhost_user build will reside in src/ovs/ The Vanilla OVS build will reside in vswitchperf/src_vanilla

To delete a src subdirectory and its contents to allow you to re-clone simply use:

$ make clobber
1.5. Configure the ./conf/10_custom.conf file

The 10_custom.conf file is the configuration file that overrides default configurations in all the other configuration files in ./conf The supplied 10_custom.conf file MUST be modified, as it contains configuration items for which there are no reasonable default values.

The configuration items that can be added is not limited to the initial contents. Any configuration item mentioned in any .conf file in ./conf directory can be added and that item will be overridden by the custom configuration value.

Further details about configuration files evaluation and special behaviour of options with GUEST_ prefix could be found at design document.

1.6. Using a custom settings file

If your 10_custom.conf doesn’t reside in the ./conf directory of if you want to use an alternative configuration file, the file can be passed to vsperf via the --conf-file argument.

$ ./vsperf --conf-file <path_to_custom_conf> ...

Note that configuration passed in via the environment (--load-env) or via another command line argument will override both the default and your custom configuration files. This “priority hierarchy” can be described like so (1 = max priority):

  1. Testcase definition section Parameters
  2. Command line arguments
  3. Environment variables
  4. Configuration file(s)

Further details about configuration files evaluation and special behaviour of options with GUEST_ prefix could be found at design document.

1.7. Referencing parameter values

It is possible to use a special macro #PARAM() to refer to the value of another configuration parameter. This reference is evaluated during access of the parameter value (by settings.getValue() call), so it can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. It can be used to reflect DUT HW details in the testcase definition.

Example:

{
    ...
    "Name": "testcase",
    "Parameters" : {
        "TRAFFIC" : {
            'l2': {
                # set destination MAC to the MAC of the first
                # interface from WHITELIST_NICS list
                'dstmac' : '#PARAM(NICS[0]["mac"])',
            },
        },
    ...
1.8. Overriding values defined in configuration files

The configuration items can be overridden by command line argument --test-params. In this case, the configuration items and their values should be passed in form of item=value and separated by semicolon.

Example:

$ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \
                         "GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput

The second option is to override configuration items by Parameters section of the test case definition. The configuration items can be added into Parameters dictionary with their new values. These values will override values defined in configuration files or specified by --test-params command line argument.

Example:

"Parameters" : {'TRAFFICGEN_PKT_SIZES' : (128,),
                'TRAFFICGEN_DURATION' : 10,
                'GUEST_LOOPBACK' : ['testpmd','l2fwd'],
               }

NOTE: In both cases, configuration item names and their values must be specified in the same form as they are defined inside configuration files. Parameter names must be specified in uppercase and data types of original and new value must match. Python syntax rules related to data types and structures must be followed. For example, parameter TRAFFICGEN_PKT_SIZES above is defined as a tuple with a single value 128. In this case trailing comma is mandatory, otherwise value can be wrongly interpreted as a number instead of a tuple and vsperf execution would fail. Please check configuration files for default values and their types and use them as a basis for any customized values. In case of any doubt, please check official python documentation related to data structures like tuples, lists and dictionaries.

NOTE: Vsperf execution will terminate with runtime error in case, that unknown parameter name is passed via --test-params CLI argument or defined in Parameters section of test case definition. It is also forbidden to redefine a value of TEST_PARAMS configuration item via CLI or Parameters section.

1.9. vloop_vnf

VSPERF uses a VM image called vloop_vnf for looping traffic in the deployment scenarios involving VMs. The image can be downloaded from http://artifacts.opnfv.org/.

Please see the installation instructions for information on vloop-vnf images.

1.10. l2fwd Kernel Module

A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd

1.11. Executing tests

All examples inside these docs assume, that user is inside the VSPERF directory. VSPERF can be executed from any directory.

Before running any tests make sure you have root permissions by adding the following line to /etc/sudoers:

username ALL=(ALL)       NOPASSWD: ALL

username in the example above should be replaced with a real username.

To list the available tests:

$ ./vsperf --list

To run a single test:

$ ./vsperf $TESTNAME

Where $TESTNAME is the name of the vsperf test you would like to run.

To run a group of tests, for example all tests with a name containing ‘RFC2544’:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests="RFC2544"

To run all tests:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf

Some tests allow for configurable parameters, including test duration (in seconds) as well as packet sizes (in bytes).

$ ./vsperf --conf-file user_settings.py \
    --tests RFC2544Tput \
    --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)"

For all available options, check out the help dialog:

$ ./vsperf --help
1.12. Executing Vanilla OVS tests
  1. If needed, recompile src for all OVS variants

    $ cd src
    $ make distclean
    $ make
    
  2. Update your 10_custom.conf file to use Vanilla OVS:

    VSWITCH = 'OvsVanilla'
    
  3. Run test:

    $ ./vsperf --conf-file=<path_to_custom_conf>
    

    Please note if you don’t want to configure Vanilla OVS through the configuration file, you can pass it as a CLI argument.

    $ ./vsperf --vswitch OvsVanilla
    
1.13. Executing tests with VMs

To run tests using vhost-user as guest access method:

  1. Set VSWITCH and VNF of your settings file to:

    VSWITCH = 'OvsDpdkVhost'
    VNF = 'QemuDpdkVhost'
    
  2. If needed, recompile src for all OVS variants

    $ cd src
    $ make distclean
    $ make
    
  3. Run test:

    $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
    

NOTE: By default vSwitch is acting as a server for dpdk vhost-user sockets. In case, that QEMU should be a server for vhost-user sockets, then parameter VSWITCH_VHOSTUSER_SERVER_MODE should be set to False.

1.14. Executing tests with VMs using Vanilla OVS

To run tests using Vanilla OVS:

  1. Set the following variables:

    VSWITCH = 'OvsVanilla'
    VNF = 'QemuVirtioNet'
    
    VANILLA_TGEN_PORT1_IP = n.n.n.n
    VANILLA_TGEN_PORT1_MAC = nn:nn:nn:nn:nn:nn
    
    VANILLA_TGEN_PORT2_IP = n.n.n.n
    VANILLA_TGEN_PORT2_MAC = nn:nn:nn:nn:nn:nn
    
    VANILLA_BRIDGE_IP = n.n.n.n
    

    or use --test-params option

    $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
               --test-params "VANILLA_TGEN_PORT1_IP=n.n.n.n;" \
                             "VANILLA_TGEN_PORT1_MAC=nn:nn:nn:nn:nn:nn;" \
                             "VANILLA_TGEN_PORT2_IP=n.n.n.n;" \
                             "VANILLA_TGEN_PORT2_MAC=nn:nn:nn:nn:nn:nn"
    
  2. If needed, recompile src for all OVS variants

    $ cd src
    $ make distclean
    $ make
    
  3. Run test:

    $ ./vsperf --conf-file<path_to_custom_conf>/10_custom.conf
    
1.15. Executing VPP tests

Currently it is not possible to use standard scenario deployments for execution of tests with VPP. It means, that deployments p2p, pvp, pvvp and in general any PXP Deployment won’t work with VPP. However it is possible to use VPP in Step driven tests. A basic set of VPP testcases covering phy2phy, pvp and pvvp tests are already prepared.

List of performance tests with VPP support follows:

  • phy2phy_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • phy2phy_cont_vpp: VPP: Phy2Phy Continuous Stream
  • phy2phy_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames
  • pvp_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • pvp_cont_vpp: VPP: PVP Continuous Stream
  • pvp_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames
  • pvvp_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
  • pvvp_cont_vpp: VPP: PVP Continuous Stream
  • pvvp_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames

In order to execute testcases with VPP it is required to:

After that it is possible to execute VPP testcases listed above.

For example:

$ ./vsperf --conf-file=<path_to_custom_conf> phy2phy_tput_vpp
1.16. Using vfio_pci with DPDK

To use vfio with DPDK instead of igb_uio add into your custom configuration file the following parameter:

PATHS['dpdk']['src']['modules'] = ['uio', 'vfio-pci']

NOTE: In case, that DPDK is installed from binary package, then please set PATHS['dpdk']['bin']['modules'] instead.

NOTE: Please ensure that Intel VT-d is enabled in BIOS.

NOTE: Please ensure your boot/grub parameters include the following:

NOTE: In case of VPP, it is required to explicitly define, that vfio-pci DPDK driver should be used. It means to update dpdk part of VSWITCH_VPP_ARGS dictionary with uio-driver section, e.g. VSWITCH_VPP_ARGS[‘dpdk’] = ‘uio-driver vfio-pci’

iommu=pt intel_iommu=on

To check that IOMMU is enabled on your platform:

$ dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.139882] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.139888] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.139893] IOAPIC id 2 under DRHD base  0xfbffe000 IOMMU 0
[    0.139894] IOAPIC id 0 under DRHD base  0xebffc000 IOMMU 1
[    0.139895] IOAPIC id 1 under DRHD base  0xebffc000 IOMMU 1
[    3.335744] IOMMU: dmar0 using Queued invalidation
[    3.335746] IOMMU: dmar1 using Queued invalidation
....
1.17. Using SRIOV support

To use virtual functions of NIC with SRIOV support, use extended form of NIC PCI slot definition:

WHITELIST_NICS = ['0000:05:00.0|vf0', '0000:05:00.1|vf3']

Where ‘vf’ is an indication of virtual function usage and following number defines a VF to be used. In case that VF usage is detected, then vswitchperf will enable SRIOV support for given card and it will detect PCI slot numbers of selected VFs.

So in example above, one VF will be configured for NIC ‘0000:05:00.0’ and four VFs will be configured for NIC ‘0000:05:00.1’. Vswitchperf will detect PCI addresses of selected VFs and it will use them during test execution.

At the end of vswitchperf execution, SRIOV support will be disabled.

SRIOV support is generic and it can be used in different testing scenarios. For example:

  • vSwitch tests with DPDK or without DPDK support to verify impact of VF usage on vSwitch performance
  • tests without vSwitch, where traffic is forwarded directly between VF interfaces by packet forwarder (e.g. testpmd application)
  • tests without vSwitch, where VM accesses VF interfaces directly by PCI-passthrough to measure raw VM throughput performance.
1.18. Using QEMU with PCI passthrough support

Raw virtual machine throughput performance can be measured by execution of PVP test with direct access to NICs by PCI pass-through. To execute VM with direct access to PCI devices, enable vfio-pci. In order to use virtual functions, SRIOV-support must be enabled.

Execution of test with PCI pass-through with vswitch disabled:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
           --vswitch none --vnf QemuPciPassthrough pvp_tput

Any of supported guest-loopback-application can be used inside VM with PCI pass-through support.

Note: Qemu with PCI pass-through support can be used only with PVP test deployment.

1.19. Selection of loopback application for tests with VMs

To select the loopback applications which will forward packets inside VMs, the following parameter should be configured:

GUEST_LOOPBACK = ['testpmd']

or use --test-params CLI argument:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
      --test-params "GUEST_LOOPBACK=['testpmd']"

Supported loopback applications are:

'testpmd'       - testpmd from dpdk will be built and used
'l2fwd'         - l2fwd module provided by Huawei will be built and used
'linux_bridge'  - linux bridge will be configured
'buildin'       - nothing will be configured by vsperf; VM image must
                  ensure traffic forwarding between its interfaces

Guest loopback application must be configured, otherwise traffic will not be forwarded by VM and testcases with VM related deployments will fail. Guest loopback application is set to ‘testpmd’ by default.

NOTE: In case that only 1 or more than 2 NICs are configured for VM, then ‘testpmd’ should be used. As it is able to forward traffic between multiple VM NIC pairs.

NOTE: In case of linux_bridge, all guest NICs are connected to the same bridge inside the guest.

1.20. Mergable Buffers Options with QEMU

Mergable buffers can be disabled with VSPerf within QEMU. This option can increase performance significantly when not using jumbo frame sized packets. By default VSPerf disables mergable buffers. If you wish to enable it you can modify the setting in the a custom conf file.

GUEST_NIC_MERGE_BUFFERS_DISABLE = [False]

Then execute using the custom conf file.

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf

Alternatively you can just pass the param during execution.

$ ./vsperf --test-params "GUEST_NIC_MERGE_BUFFERS_DISABLE=[False]"
1.21. Selection of dpdk binding driver for tests with VMs

To select dpdk binding driver, which will specify which driver the vm NICs will use for dpdk bind, the following configuration parameter should be configured:

GUEST_DPDK_BIND_DRIVER = ['igb_uio_from_src']

The supported dpdk guest bind drivers are:

'uio_pci_generic'      - Use uio_pci_generic driver
'igb_uio_from_src'     - Build and use the igb_uio driver from the dpdk src
                         files
'vfio_no_iommu'        - Use vfio with no iommu option. This requires custom
                         guest images that support this option. The default
                         vloop image does not support this driver.

Note: uio_pci_generic does not support sr-iov testcases with guests attached. This is because uio_pci_generic only supports legacy interrupts. In case uio_pci_generic is selected with the vnf as QemuPciPassthrough it will be modified to use igb_uio_from_src instead.

Note: vfio_no_iommu requires kernels equal to or greater than 4.5 and dpdk 16.04 or greater. Using this option will also taint the kernel.

Please refer to the dpdk documents at http://dpdk.org/doc/guides for more information on these drivers.

1.22. Guest Core and Thread Binding

VSPERF provides options to achieve better performance by guest core binding and guest vCPU thread binding as well. Core binding is to bind all the qemu threads. Thread binding is to bind the house keeping threads to some CPU and vCPU thread to some other CPU, this helps to reduce the noise from qemu house keeping threads.

GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]

NOTE By default the GUEST_THREAD_BINDING will be none, which means same as the GUEST_CORE_BINDING, i.e. the vcpu threads are sharing the physical CPUs with the house keeping threads. Better performance using vCPU thread binding can be achieved by enabling affinity in the custom configuration file.

For example, if an environment requires 32,33 to be core binded and 29,30&31 for guest thread binding to achieve better performance.

VNF_AFFINITIZATION_ON = True
GUEST_CORE_BINDING = [('32','33')]
GUEST_THREAD_BINDING = [('29', '30', '31')]
1.23. Qemu CPU features

QEMU default to a compatible subset of performance enhancing cpu features. To pass all available host processor features to the guest.

GUEST_CPU_OPTIONS = ['host,migratable=off']

NOTE To enhance the performance, cpu features tsc deadline timer for guest, the guest PMU, the invariant TSC can be provided in the custom configuration file.

1.24. Multi-Queue Configuration

VSPerf currently supports multi-queue with the following limitations:

  1. Requires QEMU 2.5 or greater and any OVS version higher than 2.5. The default upstream package versions installed by VSPerf satisfies this requirement.

  2. Guest image must have ethtool utility installed if using l2fwd or linux bridge inside guest for loopback.

  3. If using OVS versions 2.5.0 or less enable old style multi-queue as shown in the ‘‘02_vswitch.conf’’ file.

    OVS_OLD_STYLE_MQ = True
    

To enable multi-queue for dpdk modify the ‘‘02_vswitch.conf’’ file.

VSWITCH_DPDK_MULTI_QUEUES = 2

NOTE: you should consider using the switch affinity to set a pmd cpu mask that can optimize your performance. Consider the numa of the NIC in use if this applies by checking /sys/class/net/<eth_name>/device/numa_node and setting an appropriate mask to create PMD threads on the same numa node.

When multi-queue is enabled, each dpdk or dpdkvhostuser port that is created on the switch will set the option for multiple queues. If old style multi queue has been enabled a global option for multi queue will be used instead of the port by port option.

To enable multi-queue on the guest modify the ‘‘04_vnf.conf’’ file.

GUEST_NIC_QUEUES = [2]

Enabling multi-queue at the guest will add multiple queues to each NIC port when qemu launches the guest.

In case of Vanilla OVS, multi-queue is enabled on the tuntap ports and nic queues will be enabled inside the guest with ethtool. Simply enabling the multi-queue on the guest is sufficient for Vanilla OVS multi-queue.

Testpmd should be configured to take advantage of multi-queue on the guest if using DPDKVhostUser. This can be done by modifying the ‘‘04_vnf.conf’’ file.

GUEST_TESTPMD_PARAMS = ['-l 0,1,2,3,4  -n 4 --socket-mem 512 -- '
                        '--burst=64 -i --txqflags=0xf00 '
                        '--nb-cores=4 --rxq=2 --txq=2 '
                        '--disable-hw-vlan']

NOTE: The guest SMP cores must be configured to allow for testpmd to use the optimal number of cores to take advantage of the multiple guest queues.

In case of using Vanilla OVS and qemu virtio-net you can increase performance by binding vhost-net threads to cpus. This can be done by enabling the affinity in the ‘‘04_vnf.conf’’ file. This can be done to non multi-queue enabled configurations as well as there will be 2 vhost-net threads.

VSWITCH_VHOST_NET_AFFINITIZATION = True

VSWITCH_VHOST_CPU_MAP = [4,5,8,11]

NOTE: This method of binding would require a custom script in a real environment.

NOTE: For optimal performance guest SMPs and/or vhost-net threads should be on the same numa as the NIC in use if possible/applicable. Testpmd should be assigned at least (nb_cores +1) total cores with the cpu mask.

1.25. Jumbo Frame Testing

VSPERF provides options to support jumbo frame testing with a jumbo frame supported NIC and traffic generator for the following vswitches:

  1. OVSVanilla
  2. OvsDpdkVhostUser
  3. TestPMD loopback with or without a guest

NOTE: There is currently no support for SR-IOV or VPP at this time with jumbo frames.

All packet forwarding applications for pxp testing is supported.

To enable jumbo frame testing simply enable the option in the conf files and set the maximum size that will be used.

VSWITCH_JUMBO_FRAMES_ENABLED = True
VSWITCH_JUMBO_FRAMES_SIZE = 9000

To enable jumbo frame testing with OVSVanilla the NIC in test on the host must have its mtu size changed manually using ifconfig or applicable tools:

ifconfig eth1 mtu 9000 up

NOTE: To make the setting consistent across reboots you should reference the OS documents as it differs from distribution to distribution.

To start a test for jumbo frames modify the conf file packet sizes or pass the option through the VSPERF command line.

TEST_PARAMS = {'TRAFFICGEN_PKT_SIZES':(2000,9000)}
./vsperf --test-params "TRAFFICGEN_PKT_SIZES=2000,9000"

It is recommended to increase the memory size for OvsDpdkVhostUser testing from the default 1024. Your size required may vary depending on the number of guests in your testing. 4096 appears to work well for most typical testing scenarios.

DPDK_SOCKET_MEM = ['4096', '0']

NOTE: For Jumbo frames to work with DpdkVhostUser, mergable buffers will be enabled by default. If testing with mergable buffers in QEMU is desired, disable Jumbo Frames and only test non jumbo frame sizes. Test Jumbo Frames sizes separately to avoid this collision.

1.26. Executing Packet Forwarding tests

To select the applications which will forward packets, the following parameters should be configured:

VSWITCH = 'none'
PKTFWD = 'TestPMD'

or use --vswitch and --fwdapp CLI arguments:

$ ./vsperf phy2phy_cont --conf-file user_settings.py \
           --vswitch none \
           --fwdapp TestPMD

Supported Packet Forwarding applications are:

'testpmd'       - testpmd from dpdk
  1. Update your ‘‘10_custom.conf’’ file to use the appropriate variables for selected Packet Forwarder:

    # testpmd configuration
    TESTPMD_ARGS = []
    # packet forwarding mode supported by testpmd; Please see DPDK documentation
    # for comprehensive list of modes supported by your version.
    # e.g. io|mac|mac_retry|macswap|flowgen|rxonly|txonly|csum|icmpecho|...
    # Note: Option "mac_retry" has been changed to "mac retry" since DPDK v16.07
    TESTPMD_FWD_MODE = 'csum'
    # checksum calculation layer: ip|udp|tcp|sctp|outer-ip
    TESTPMD_CSUM_LAYER = 'ip'
    # checksum calculation place: hw (hardware) | sw (software)
    TESTPMD_CSUM_CALC = 'sw'
    # recognize tunnel headers: on|off
    TESTPMD_CSUM_PARSE_TUNNEL = 'off'
    
  2. Run test:

    $ ./vsperf phy2phy_tput --conf-file <path_to_settings_py>
    
1.27. Executing Packet Forwarding tests with one guest

TestPMD with DPDK 16.11 or greater can be used to forward packets as a switch to a single guest using TestPMD vdev option. To set this configuration the following parameters should be used.

VSWITCH = 'none'
PKTFWD = 'TestPMD'

or use --vswitch and --fwdapp CLI arguments:

$ ./vsperf pvp_tput --conf-file user_settings.py \
           --vswitch none \
           --fwdapp TestPMD

Guest forwarding application only supports TestPMD in this configuration.

GUEST_LOOPBACK = ['testpmd']

For optimal performance one cpu per port +1 should be used for TestPMD. Also set additional params for packet forwarding application to use the correct number of nb-cores.

DPDK_SOCKET_MEM = ['1024', '0']
VSWITCHD_DPDK_ARGS = ['-l', '46,44,42,40,38', '-n', '4']
TESTPMD_ARGS = ['--nb-cores=4', '--txq=1', '--rxq=1']

For guest TestPMD 3 VCpus should be assigned with the following TestPMD params.

GUEST_TESTPMD_PARAMS = ['-l 0,1,2 -n 4 --socket-mem 1024 -- '
                        '--burst=64 -i --txqflags=0xf00 '
                        '--disable-hw-vlan --nb-cores=2 --txq=1 --rxq=1']

Execution of TestPMD can be run with the following command line

./vsperf pvp_tput --vswitch=none --fwdapp=TestPMD --conf-file <path_to_settings_py>

NOTE: To achieve the best 0% loss numbers with rfc2544 throughput testing, other tunings should be applied to host and guest such as tuned profiles and CPU tunings to prevent possible interrupts to worker threads.

1.28. VSPERF modes of operation

VSPERF can be run in different modes. By default it will configure vSwitch, traffic generator and VNF. However it can be used just for configuration and execution of traffic generator. Another option is execution of all components except traffic generator itself.

Mode of operation is driven by configuration parameter -m or –mode

-m MODE, --mode MODE  vsperf mode of operation;
    Values:
        "normal" - execute vSwitch, VNF and traffic generator
        "trafficgen" - execute only traffic generator
        "trafficgen-off" - execute vSwitch and VNF
        "trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission

In case, that VSPERF is executed in “trafficgen” mode, then configuration of traffic generator can be modified through TRAFFIC dictionary passed to the --test-params option. It is not needed to specify all values of TRAFFIC dictionary. It is sufficient to specify only values, which should be changed. Detailed description of TRAFFIC dictionary can be found at Configuration of TRAFFIC dictionary.

Example of execution of VSPERF in “trafficgen” mode:

$ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \
    --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}"
1.29. Code change verification by pylint

Every developer participating in VSPERF project should run pylint before his python code is submitted for review. Project specific configuration for pylint is available at ‘pylint.rc’.

Example of manual pylint invocation:

$ pylint --rcfile ./pylintrc ./vsperf
1.30. GOTCHAs:
1.30.1. Custom image fails to boot

Using custom VM images may not boot within VSPerf pxp testing because of the drive boot and shared type which could be caused by a missing scsi driver inside the image. In case of issues you can try changing the drive boot type to ide.

GUEST_BOOT_DRIVE_TYPE = ['ide']
GUEST_SHARED_DRIVE_TYPE = ['ide']
1.30.2. OVS with DPDK and QEMU

If you encounter the following error: “before (last 100 chars): ‘-path=/dev/hugepages,share=on: unable to map backing store for hugepages: Cannot allocate memoryrnrn” during qemu initialization, check the amount of hugepages on your system:

$ cat /proc/meminfo | grep HugePages

By default the vswitchd is launched with 1Gb of memory, to change this, modify –socket-mem parameter in conf/02_vswitch.conf to allocate an appropriate amount of memory:

DPDK_SOCKET_MEM = ['1024', '0']
VSWITCHD_DPDK_ARGS = ['-c', '0x4', '-n', '4']
VSWITCHD_DPDK_CONFIG = {
    'dpdk-init' : 'true',
    'dpdk-lcore-mask' : '0x4',
    'dpdk-socket-mem' : '1024,0',
}

Note: Option VSWITCHD_DPDK_ARGS is used for vswitchd, which supports --dpdk parameter. In recent vswitchd versions, option VSWITCHD_DPDK_CONFIG will be used to configure vswitchd via ovs-vsctl calls.

1.31. More information

For more information and details refer to the rest of vSwitchPerfuser documentation.

2. Step driven tests

In general, test scenarios are defined by a deployment used in the particular test case definition. The chosen deployment scenario will take care of the vSwitch configuration, deployment of VNFs and it can also affect configuration of a traffic generator. In order to allow a more flexible way of testcase scripting, VSPERF supports a detailed step driven testcase definition. It can be used to configure and program vSwitch, deploy and terminate VNFs, execute a traffic generator, modify a VSPERF configuration, execute external commands, etc.

Execution of step driven tests is done on a step by step work flow starting with step 0 as defined inside the test case. Each step of the test increments the step number by one which is indicated in the log.

(testcases.integration) - Step 0 'vswitch add_vport ['br0']' start

Step driven tests can be used for both performance and integration testing. In case of integration test, each step in the test case is validated. If a step does not pass validation the test will fail and terminate. The test will continue until a failure is detected or all steps pass. A csv report file is generated after a test completes with an OK or FAIL result.

NOTE: It is possible to suppress validation process of given step by prefixing it by ! (exclamation mark). In following example test execution won’t fail if all traffic is dropped:

['!trafficgen', 'send_traffic', {}]

In case of performance test, the validation of steps is not performed and standard output files with results from traffic generator and underlying OS details are generated by vsperf.

Step driven testcases can be used in two different ways:

# description of full testcase - in this case clean deployment is used
to indicate that vsperf should neither configure vSwitch nor deploy any VNF. Test shall perform all required vSwitch configuration and programming and deploy required number of VNFs.
# modification of existing deployment - in this case, any of supported
deployments can be used to perform initial vSwitch configuration and deployment of VNFs. Additional actions defined by TestSteps can be used to alter vSwitch configuration or deploy additional VNFs. After the last step is processed, the test execution will continue with traffic execution.
2.1. Test objects and their functions

Every test step can call a function of one of the supported test objects. The list of supported objects and their most common functions follows:

  • vswitch - provides functions for vSwitch configuration

    List of supported functions:

    • add_switch br_name - creates a new switch (bridge) with given br_name
    • del_switch br_name - deletes switch (bridge) with given br_name
    • add_phy_port br_name - adds a physical port into bridge specified by br_name
    • add_vport br_name - adds a virtual port into bridge specified by br_name
    • del_port br_name port_name - removes physical or virtual port specified by port_name from bridge br_name
    • add_flow br_name flow - adds flow specified by flow dictionary into the bridge br_name; Content of flow dictionary will be passed to the vSwitch. In case of Open vSwitch it will be passed to the ovs-ofctl add-flow command. Please see Open vSwitch documentation for the list of supported flow parameters.
    • del_flow br_name [flow] - deletes flow specified by flow dictionary from bridge br_name; In case that optional parameter flow is not specified or set to an empty dictionary {}, then all flows from bridge br_name will be deleted.
    • dump_flows br_name - dumps all flows from bridge specified by br_name
    • enable_stp br_name - enables Spanning Tree Protocol for bridge br_name
    • disable_stp br_name - disables Spanning Tree Protocol for bridge br_name
    • enable_rstp br_name - enables Rapid Spanning Tree Protocol for bridge br_name
    • disable_rstp br_name - disables Rapid Spanning Tree Protocol for bridge br_name
    • restart - restarts switch, which is useful for failover testcases

    Examples:

    ['vswitch', 'add_switch', 'int_br0']
    
    ['vswitch', 'del_switch', 'int_br0']
    
    ['vswitch', 'add_phy_port', 'int_br0']
    
    ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]']
    
    ['vswitch', 'add_flow', 'int_br0', {'in_port': '1', 'actions': ['output:2'],
     'idle_timeout': '0'}],
    
    ['vswitch', 'enable_rstp', 'int_br0']
    
  • vnf[ID] - provides functions for deployment and termination of VNFs; Optional alfanumerical ID is used for VNF identification in case that testcase deploys multiple VNFs.

    List of supported functions:

    • start - starts a VNF based on VSPERF configuration
    • stop - gracefully terminates given VNF
    • execute command [delay] - executes command cmd inside VNF; Optional delay defines number of seconds to wait before next step is executed. Method returns command output as a string.
    • execute_and_wait command [timeout] [prompt] - executes command cmd inside VNF; Optional timeout defines number of seconds to wait until prompt is detected. Optional prompt defines a string, which is used as detection of successful command execution. In case that prompt is not defined, then content of GUEST_PROMPT_LOGIN parameter will be used. Method returns command output as a string.

    Examples:

    ['vnf1', 'start'],
    ['vnf2', 'start'],
    ['vnf1', 'execute_and_wait', 'ifconfig eth0 5.5.5.1/24 up'],
    ['vnf2', 'execute_and_wait', 'ifconfig eth0 5.5.5.2/24 up', 120, 'root.*#'],
    ['vnf2', 'execute_and_wait', 'ping -c1 5.5.5.1'],
    ['vnf2', 'stop'],
    ['vnf1', 'stop'],
    
  • trafficgen - triggers traffic generation

    List of supported functions:

    • send_traffic traffic - starts a traffic based on the vsperf configuration and given traffic dictionary. More details about traffic dictionary and its possible values are available at Traffic Generator Integration Guide
    • get_results - returns dictionary with results collected from previous execution of send_traffic

    Examples:

    ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_throughput'}]
    
    ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}],
    ['trafficgen', 'get_results'],
    ['tools', 'assert', '#STEP[-1][0]["frame_loss_percent"] < 0.05'],
    
  • settings - reads or modifies VSPERF configuration

    List of supported functions:

    • getValue param - returns value of given param
    • setValue param value - sets value of param to given value

    Examples:

    ['settings', 'getValue', 'TOOLS']
    
    ['settings', 'setValue', 'GUEST_USERNAME', ['root']]
    

    It is possible and more convenient to access any VSPERF configuration option directly via $NAME notation. Option evaluation is done during runtime and vsperf will automatically translate it to the appropriate call of settings.getValue. If the referred parameter does not exist, then vsperf will keep $NAME string untouched and it will continue with testcase execution. The reason is to avoid test execution failure in case that $ sign has been used from different reason than vsperf parameter evaluation.

    NOTE: It is recommended to use ${NAME} notation for any shell parameters used within Exec_Shell call to avoid a clash with configuration parameter evaluation.

    NOTE: It is possible to refer to vsperf parameter value by #PARAM() macro (see Referencing parameter values. However #PARAM() macro is evaluated at the beginning of vsperf execution and it will not reflect any changes made to the vsperf configuration during runtime. On the other hand $NAME notation is evaluated during test execution and thus it contains any modifications to the configuration parameter made by vsperf (e.g. TOOLS and NICS dictionaries) or by testcase definition (e.g. TRAFFIC dictionary).

    Examples:

    ['tools', 'exec_shell', "$TOOLS['ovs-vsctl'] show"]
    
    ['settings', 'setValue', 'TRAFFICGEN_IXIA_PORT2', '$TRAFFICGEN_IXIA_PORT1'],
    
    ['vswitch', 'add_flow', 'int_br0',
     {'in_port': '#STEP[1][1]',
      'dl_type': '0x800',
      'nw_proto': '17',
      'nw_dst': '$TRAFFIC["l3"]["dstip"]/8',
      'actions': ['output:#STEP[2][1]']
     }
    ]
    
  • namespace - creates or modifies network namespaces

    List of supported functions:

    • create_namespace name - creates new namespace with given name
    • delete_namespace name - deletes namespace specified by its name
    • assign_port_to_namespace port name [port_up] - assigns NIC specified by port into given namespace name; If optional parameter port_up is set to True, then port will be brought up.
    • add_ip_to_namespace_eth port name addr cidr - assigns an IP address addr/cidr to the NIC specified by port within namespace name
    • reset_port_to_root port name - returns given port from namespace name back to the root namespace

    Examples:

    ['namespace', 'create_namespace', 'testns']
    
    ['namespace', 'assign_port_to_namespace', 'eth0', 'testns']
    
  • veth - manipulates with eth and veth devices

    List of supported functions:

    • add_veth_port port peer_port - adds a pair of veth ports named port and peer_port
    • del_veth_port port peer_port - deletes a veth port pair specified by port and peer_port
    • bring_up_eth_port eth_port [namespace] - brings up eth_port in (optional) namespace

    Examples:

    ['veth', 'add_veth_port', 'veth', 'veth1']
    
    ['veth', 'bring_up_eth_port', 'eth1']
    
  • tools - provides a set of helper functions

    List of supported functions:

    • Assert condition - evaluates given condition and raises AssertionError in case that condition is not True
    • Eval expression - evaluates given expression as a python code and returns its result
    • Exec_Shell command - executes a shell command
    • Exec_Python code - executes a python code

    Examples:

    ['tools', 'exec_shell', 'numactl -H', 'available: ([0-9]+)']
    ['tools', 'assert', '#STEP[-1][0]>1']
    
  • wait - is used for test case interruption. This object doesn’t have any functions. Once reached, vsperf will pause test execution and waits for press of Enter key. It can be used during testcase design for debugging purposes.

    Examples:

    ['wait']
    
  • sleep - is used to pause testcase execution for defined number of seconds.

    Examples:

    ['sleep', '60']
    
  • log level message - is used to log message of given level into vsperf output. Level is one of info, debug, warning or error.

    Examples:

    ['log', 'error', 'tools $TOOLS']
    
  • pdb - executes python debugger

    Examples:

    ['pdb']
    
2.2. Test Macros

Test profiles can include macros as part of the test step. Each step in the profile may return a value such as a port name. Recall macros use #STEP to indicate the recalled value inside the return structure. If the method the test step calls returns a value it can be later recalled, for example:

{
    "Name": "vswitch_add_del_vport",
    "Deployment": "clean",
    "Description": "vSwitch - add and delete virtual port",
    "TestSteps": [
            ['vswitch', 'add_switch', 'int_br0'],               # STEP 0
            ['vswitch', 'add_vport', 'int_br0'],                # STEP 1
            ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],  # STEP 2
            ['vswitch', 'del_switch', 'int_br0'],               # STEP 3
         ]
}

This test profile uses the vswitch add_vport method which returns a string value of the port added. This is later called by the del_port method using the name from step 1.

It is also possible to use negative indexes in step macros. In that case #STEP[-1] will refer to the result from previous step, #STEP[-2] will refer to result of step called before previous step, etc. It means, that you could change STEP 2 from previous example to achieve the same functionality:

['vswitch', 'del_port', 'int_br0', '#STEP[-1][0]'],  # STEP 2

Another option to refer to previous values, is to define an alias for given step by its first argument with ‘#’ prefix. Alias must be unique and it can’t be a number. Example of step alias usage:

['#port1', 'vswitch', 'add_vport', 'int_br0'],
['vswitch', 'del_port', 'int_br0', '#STEP[port1][0]'],

Also commonly used steps can be created as a separate profile.

STEP_VSWITCH_PVP_INIT = [
    ['vswitch', 'add_switch', 'int_br0'],           # STEP 0
    ['vswitch', 'add_phy_port', 'int_br0'],         # STEP 1
    ['vswitch', 'add_phy_port', 'int_br0'],         # STEP 2
    ['vswitch', 'add_vport', 'int_br0'],            # STEP 3
    ['vswitch', 'add_vport', 'int_br0'],            # STEP 4
]

This profile can then be used inside other testcases

{
    "Name": "vswitch_pvp",
    "Deployment": "clean",
    "Description": "vSwitch - configure switch and one vnf",
    "TestSteps": STEP_VSWITCH_PVP_INIT +
                 [
                    ['vnf', 'start'],
                    ['vnf', 'stop'],
                 ] +
                 STEP_VSWITCH_PVP_FINIT
}

It is possible to refer to vsperf configuration parameters within step macros. Please see step-driven-tests-variable-usage for more details.

In case that step returns a string or list of strings, then it is possible to filter such output by regular expression. This optional filter can be specified as a last step parameter with prefix ‘|’. Output will be split into separate lines and only matching records will be returned. It is also possible to return a specified group of characters from the matching lines, e.g. by regex |ID (\d+).

Examples:

['tools', 'exec_shell', "sudo $TOOLS['ovs-appctl'] dpif-netdev/pmd-rxq-show",
 '|dpdkvhostuser0\s+queue-id: \d'],
['tools', 'assert', 'len(#STEP[-1])==1'],

['vnf', 'execute_and_wait', 'ethtool -L eth0 combined 2'],
['vnf', 'execute_and_wait', 'ethtool -l eth0', '|Combined:\s+2'],
['tools', 'assert', 'len(#STEP[-1])==2']
2.3. HelloWorld and other basic Testcases

The following examples are for demonstration purposes. You can run them by copying and pasting into the conf/integration/01_testcases.conf file. A command-line instruction is shown at the end of each example.

2.3.1. HelloWorld

The first example is a HelloWorld testcase. It simply creates a bridge with 2 physical ports, then sets up a flow to drop incoming packets from the port that was instantiated at the STEP #1. There’s no interaction with the traffic generator. Then the flow, the 2 ports and the bridge are deleted. ‘add_phy_port’ method creates a ‘dpdk’ type interface that will manage the physical port. The string value returned is the port name that will be referred by ‘del_port’ later on.

{
    "Name": "HelloWorld",
    "Description": "My first testcase",
    "Deployment": "clean",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],   # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'actions': ['drop'], 'idle_timeout': '0'}],
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]

},

To run HelloWorld test:

./vsperf --conf-file user_settings.py --integration HelloWorld
2.3.2. Specify a Flow by the IP address

The next example shows how to explicitly set up a flow by specifying a destination IP address. All packets received from the port created at STEP #1 that have a destination IP address = 90.90.90.90 will be forwarded to the port created at the STEP #2.

{
    "Name": "p2p_rule_l3da",
    "Description": "Phy2Phy with rule on L3 Dest Addr",
    "Deployment": "clean",
    "biDirectional": "False",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],   # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_dst': '90.90.90.90', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous'}],
        ['vswitch', 'dump_flows', 'int_br0'],   # STEP 5
        ['vswitch', 'del_flow', 'int_br0'],     # STEP 7 == del-flows
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration p2p_rule_l3da
2.3.3. Multistream feature

The next testcase uses the multistream feature. The traffic generator will send packets with different UDP ports. That is accomplished by using “Stream Type” and “MultiStream” keywords. 4 different flows are set to forward all incoming packets.

{
    "Name": "multistream_l4",
    "Description": "Multistream on UDP ports",
    "Deployment": "clean",
    "Parameters": {
        'TRAFFIC' : {
            "multistream": 4,
            "stream_type": "L4",
        },
    },
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],   # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'], # STEP 2
        # Setup Flows
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '0', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '1', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '2', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '3', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        # Send mono-dir traffic
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous', \
            'bidir' : 'False'}],
        # Clean up
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
     ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration multistream_l4
2.3.4. PVP with a VM Replacement

This example launches a 1st VM in a PVP topology, then the VM is replaced by another VM. When VNF setup parameter in ./conf/04_vnf.conf is “QemuDpdkVhostUser” ‘add_vport’ method creates a ‘dpdkvhostuser’ type port to connect a VM.

{
    "Name": "ex_replace_vm",
    "Description": "PVP with VM replacement",
    "Deployment": "clean",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],       # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 3    vm1
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 4

        # Setup Flows
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[2][1]', \
            'actions': ['output:#STEP[4][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[3][1]', \
            'actions': ['output:#STEP[1][1]'], 'idle_timeout': '0'}],

        # Start VM 1
        ['vnf1', 'start'],
        # Now we want to replace VM 1 with another VM
        ['vnf1', 'stop'],

        ['vswitch', 'add_vport', 'int_br0'],        # STEP 11    vm2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 12
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'actions': ['output:#STEP[11][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[12][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],

        # Start VM 2
        ['vnf2', 'start'],
        ['vnf2', 'stop'],
        ['vswitch', 'dump_flows', 'int_br0'],

        # Clean up
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'],    # vm1
        ['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[11][0]'],   # vm2
        ['vswitch', 'del_port', 'int_br0', '#STEP[12][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration ex_replace_vm
2.3.5. VM with a Linux bridge

This example setups a PVP topology and routes traffic to the VM based on the destination IP address. A command-line parameter is used to select a Linux bridge as a guest loopback application. It is also possible to select a guest loopback application by a configuration option GUEST_LOOPBACK.

{
    "Name": "ex_pvp_rule_l3da",
    "Description": "PVP with flow on L3 Dest Addr",
    "Deployment": "clean",
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],       # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 3    vm1
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 4
        # Setup Flows
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_dst': '90.90.90.90', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        # Each pkt from the VM is forwarded to the 2nd dpdk port
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        # Start VMs
        ['vnf1', 'start'],
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous', \
            'bidir' : 'False'}],
        ['vnf1', 'stop'],
        # Clean up
        ['vswitch', 'dump_flows', 'int_br0'],       # STEP 10
        ['vswitch', 'del_flow', 'int_br0'],         # STEP 11
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'],  # vm1 ports
        ['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --test-params \
        "GUEST_LOOPBACK=['linux_bridge']" --integration ex_pvp_rule_l3da
2.3.6. Forward packets based on UDP port

This examples launches 2 VMs connected in parallel. Incoming packets will be forwarded to one specific VM depending on the destination UDP port.

{
    "Name": "ex_2pvp_rule_l4dp",
    "Description": "2 PVP with flows on L4 Dest Port",
    "Deployment": "clean",
    "Parameters": {
        'TRAFFIC' : {
            "multistream": 2,
            "stream_type": "L4",
        },
    },
    "TestSteps": [
        ['vswitch', 'add_switch', 'int_br0'],       # STEP 0
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 1
        ['vswitch', 'add_phy_port', 'int_br0'],     # STEP 2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 3    vm1
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 4
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 5    vm2
        ['vswitch', 'add_vport', 'int_br0'],        # STEP 6
        # Setup Flows to reply ICMPv6 and similar packets, so to
        # avoid flooding internal port with their re-transmissions
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:01', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:02', \
            'actions': ['output:#STEP[4][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:03', \
            'actions': ['output:#STEP[5][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', \
            {'priority': '1', 'dl_src': '00:00:00:00:00:04', \
            'actions': ['output:#STEP[6][1]'], 'idle_timeout': '0'}],
        # Forward UDP packets depending on dest port
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '0', \
            'actions': ['output:#STEP[3][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[1][1]', \
            'dl_type': '0x0800', 'nw_proto': '17', 'udp_dst': '1', \
            'actions': ['output:#STEP[5][1]'], 'idle_timeout': '0'}],
        # Send VM output to phy port #2
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[4][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        ['vswitch', 'add_flow', 'int_br0', {'in_port': '#STEP[6][1]', \
            'actions': ['output:#STEP[2][1]'], 'idle_timeout': '0'}],
        # Start VMs
        ['vnf1', 'start'],                          # STEP 16
        ['vnf2', 'start'],                          # STEP 17
        ['trafficgen', 'send_traffic', \
            {'traffic_type' : 'rfc2544_continuous', \
            'bidir' : 'False'}],
        ['vnf1', 'stop'],
        ['vnf2', 'stop'],
        ['vswitch', 'dump_flows', 'int_br0'],
        # Clean up
        ['vswitch', 'del_flow', 'int_br0'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[1][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[2][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[3][0]'],  # vm1 ports
        ['vswitch', 'del_port', 'int_br0', '#STEP[4][0]'],
        ['vswitch', 'del_port', 'int_br0', '#STEP[5][0]'],  # vm2 ports
        ['vswitch', 'del_port', 'int_br0', '#STEP[6][0]'],
        ['vswitch', 'del_switch', 'int_br0'],
    ]
},

To run the test:

./vsperf --conf-file user_settings.py --integration ex_2pvp_rule_l4dp
2.3.7. Modification of existing PVVP deployment

This is an example of modification of a standard deployment scenario with additional TestSteps. Standard PVVP scenario is used to configure a vSwitch and to deploy two VNFs connected in series. Additional TestSteps will deploy a 3rd VNF and connect it in parallel to already configured VNFs. Traffic generator is instructed (by Multistream feature) to send two separate traffic streams. One stream will be sent to the standalone VNF and second to two chained VNFs.

In case, that test is defined as a performance test, then traffic results will be collected and available in both csv and rst report files.

{
    "Name": "pvvp_pvp_cont",
    "Deployment": "pvvp",
    "Description": "PVVP and PVP in parallel with Continuous Stream",
    "Parameters" : {
        "TRAFFIC" : {
            "traffic_type" : "rfc2544_continuous",
            "multistream": 2,
        },
    },
    "TestSteps": [
                    ['vswitch', 'add_vport', 'br0'],
                    ['vswitch', 'add_vport', 'br0'],
                    # priority must be higher than default 32768, otherwise flows won't match
                    ['vswitch', 'add_flow', 'br0',
                     {'in_port': '1', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800',
                                                  'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}],
                    ['vswitch', 'add_flow', 'br0',
                     {'in_port': '2', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800',
                                                  'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}],
                    ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'],
                                                    'idle_timeout': '0'}],
                    ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'],
                                                    'idle_timeout': '0'}],
                    ['vswitch', 'dump_flows', 'br0'],
                    ['vnf1', 'start'],
                 ]
},

To run the test:

./vsperf --conf-file user_settings.py pvvp_pvp_cont
3. Integration tests

VSPERF includes a set of integration tests defined in conf/integration. These tests can be run by specifying –integration as a parameter to vsperf. Current tests in conf/integration include switch functionality and Overlay tests.

Tests in the conf/integration can be used to test scaling of different switch configurations by adding steps into the test case.

For the overlay tests VSPERF supports VXLAN, GRE and GENEVE tunneling protocols. Testing of these protocols is limited to unidirectional traffic and P2P (Physical to Physical scenarios).

NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only.

3.1. Executing Integration Tests

To execute integration tests VSPERF is run with the integration parameter. To view the current test list simply execute the following command:

./vsperf --integration --list

The standard tests included are defined inside the conf/integration/01_testcases.conf file.

3.2. Executing Tunnel encapsulation tests

The VXLAN OVS DPDK encapsulation tests requires IPs, MAC addresses, bridge names and WHITELIST_NICS for DPDK.

NOTE: Only Ixia traffic generators currently support the execution of the tunnel encapsulation tests. Support for other traffic generators may come in a future release.

Default values are already provided. To customize for your environment, override the following variables in you user_settings.py file:

# Variables defined in conf/integration/02_vswitch.conf
# Tunnel endpoint for Overlay P2P deployment scenario
# used for br0
VTEP_IP1 = '192.168.0.1/24'

# Used as remote_ip in adding OVS tunnel port and
# to set ARP entry in OVS (e.g. tnl/arp/set br-ext 192.168.240.10 02:00:00:00:00:02
VTEP_IP2 = '192.168.240.10'

# Network to use when adding a route for inner frame data
VTEP_IP2_SUBNET = '192.168.240.0/24'

# Bridge names
TUNNEL_INTEGRATION_BRIDGE = 'br0'
TUNNEL_EXTERNAL_BRIDGE = 'br-ext'

# IP of br-ext
TUNNEL_EXTERNAL_BRIDGE_IP = '192.168.240.1/24'

# vxlan|gre|geneve
TUNNEL_TYPE = 'vxlan'

# Variables defined conf/integration/03_traffic.conf
# For OP2P deployment scenario
TRAFFICGEN_PORT1_MAC = '02:00:00:00:00:01'
TRAFFICGEN_PORT2_MAC = '02:00:00:00:00:02'
TRAFFICGEN_PORT1_IP = '1.1.1.1'
TRAFFICGEN_PORT2_IP = '192.168.240.10'

To run VXLAN encapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=vxlan' overlay_p2p_tput

To run GRE encapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=gre' overlay_p2p_tput

To run GENEVE encapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=geneve' overlay_p2p_tput

To run OVS NATIVE tunnel tests (VXLAN/GRE/GENEVE):

  1. Install the OVS kernel modules
cd src/ovs/ovs
sudo -E make modules_install
  1. Set the following variables:
VSWITCH = 'OvsVanilla'
# Specify vport_* kernel module to test.
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_vxlan',
    'vport_gre',
    'vport_geneve',
    'datapath/linux/openvswitch.ko',
]

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run tests:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'TUNNEL_TYPE=vxlan' overlay_p2p_tput
3.3. Executing VXLAN decapsulation tests

To run VXLAN decapsulation tests:

  1. Set the variables used in “Executing Tunnel encapsulation tests”
  2. Run test:
./vsperf --conf-file user_settings.py --integration overlay_p2p_decap_cont

If you want to use different values for your VXLAN frame, you may set:

VXLAN_FRAME_L3 = {'proto': 'udp',
                  'packetsize': 64,
                  'srcip': TRAFFICGEN_PORT1_IP,
                  'dstip': '192.168.240.1',
                 }
VXLAN_FRAME_L4 = {'srcport': 4789,
                  'dstport': 4789,
                  'vni': VXLAN_VNI,
                  'inner_srcmac': '01:02:03:04:05:06',
                  'inner_dstmac': '06:05:04:03:02:01',
                  'inner_srcip': '192.168.0.10',
                  'inner_dstip': '192.168.240.9',
                  'inner_proto': 'udp',
                  'inner_srcport': 3000,
                  'inner_dstport': 3001,
                 }
3.4. Executing GRE decapsulation tests

To run GRE decapsulation tests:

  1. Set the variables used in “Executing Tunnel encapsulation tests”
  2. Run test:
./vsperf --conf-file user_settings.py --test-params 'TUNNEL_TYPE=gre' \
         --integration overlay_p2p_decap_cont

If you want to use different values for your GRE frame, you may set:

GRE_FRAME_L3 = {'proto': 'gre',
                'packetsize': 64,
                'srcip': TRAFFICGEN_PORT1_IP,
                'dstip': '192.168.240.1',
               }

GRE_FRAME_L4 = {'srcport': 0,
                'dstport': 0
                'inner_srcmac': '01:02:03:04:05:06',
                'inner_dstmac': '06:05:04:03:02:01',
                'inner_srcip': '192.168.0.10',
                'inner_dstip': '192.168.240.9',
                'inner_proto': 'udp',
                'inner_srcport': 3000,
                'inner_dstport': 3001,
               }
3.5. Executing GENEVE decapsulation tests

IxNet 7.3X does not have native support of GENEVE protocol. The template, GeneveIxNetTemplate.xml_ClearText.xml, should be imported into IxNET for this testcase to work.

To import the template do:

  1. Run the IxNetwork TCL Server
  2. Click on the Traffic menu
  3. Click on the Traffic actions and click Edit Packet Templates
  4. On the Template editor window, click Import. Select the template located at 3rd_party/ixia/GeneveIxNetTemplate.xml_ClearText.xml and click import.
  5. Restart the TCL Server.

To run GENEVE decapsulation tests:

  1. Set the variables used in “Executing Tunnel encapsulation tests”
  2. Run test:
./vsperf --conf-file user_settings.py --test-params 'tunnel_type=geneve' \
         --integration overlay_p2p_decap_cont

If you want to use different values for your GENEVE frame, you may set:

GENEVE_FRAME_L3 = {'proto': 'udp',
                   'packetsize': 64,
                   'srcip': TRAFFICGEN_PORT1_IP,
                   'dstip': '192.168.240.1',
                  }

GENEVE_FRAME_L4 = {'srcport': 6081,
                   'dstport': 6081,
                   'geneve_vni': 0,
                   'inner_srcmac': '01:02:03:04:05:06',
                   'inner_dstmac': '06:05:04:03:02:01',
                   'inner_srcip': '192.168.0.10',
                   'inner_dstip': '192.168.240.9',
                   'inner_proto': 'udp',
                   'inner_srcport': 3000,
                   'inner_dstport': 3001,
                  }
3.6. Executing Native/Vanilla OVS VXLAN decapsulation tests

To run VXLAN decapsulation tests:

  1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_vxlan',
    'datapath/linux/openvswitch.ko',
]

TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'

VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'

VXLAN_FRAME_L2 = {'srcmac':
                  '01:02:03:04:05:06',
                  'dstmac':
                  '06:05:04:03:02:01',
                 }

VXLAN_FRAME_L3 = {'proto': 'udp',
                  'packetsize': 64,
                  'srcip': TRAFFICGEN_PORT1_IP,
                  'dstip': '172.16.1.1',
                 }

VXLAN_FRAME_L4 = {
                  'srcport': 4789,
                  'dstport': 4789,
                  'protocolpad': 'true',
                  'vni': 99,
                  'inner_srcmac': '01:02:03:04:05:06',
                  'inner_dstmac': '06:05:04:03:02:01',
                  'inner_srcip': '192.168.1.2',
                  'inner_dstip': TRAFFICGEN_PORT2_IP,
                  'inner_proto': 'udp',
                  'inner_srcport': 3000,
                  'inner_dstport': 3001,
                 }

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run test:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'tunnel_type=vxlan' overlay_p2p_decap_cont
3.7. Executing Native/Vanilla OVS GRE decapsulation tests

To run GRE decapsulation tests:

  1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_gre',
    'datapath/linux/openvswitch.ko',
]

TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'

VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'

GRE_FRAME_L2 = {'srcmac':
                '01:02:03:04:05:06',
                'dstmac':
                '06:05:04:03:02:01',
               }

GRE_FRAME_L3 = {'proto': 'udp',
                'packetsize': 64,
                'srcip': TRAFFICGEN_PORT1_IP,
                'dstip': '172.16.1.1',
               }

GRE_FRAME_L4 = {
                'srcport': 4789,
                'dstport': 4789,
                'protocolpad': 'true',
                'inner_srcmac': '01:02:03:04:05:06',
                'inner_dstmac': '06:05:04:03:02:01',
                'inner_srcip': '192.168.1.2',
                'inner_dstip': TRAFFICGEN_PORT2_IP,
                'inner_proto': 'udp',
                'inner_srcport': 3000,
                'inner_dstport': 3001,
               }

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run test:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'tunnel_type=gre' overlay_p2p_decap_cont
3.8. Executing Native/Vanilla OVS GENEVE decapsulation tests

To run GENEVE decapsulation tests:

  1. Set the following variables in your user_settings.py file:
PATHS['vswitch']['OvsVanilla']['src']['modules'] = [
    'vport_geneve',
    'datapath/linux/openvswitch.ko',
]

TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'

VTEP_IP1 = '172.16.1.2/24'
VTEP_IP2 = '192.168.1.1'
VTEP_IP2_SUBNET = '192.168.1.0/24'
TUNNEL_EXTERNAL_BRIDGE_IP = '172.16.1.1/24'
TUNNEL_INT_BRIDGE_IP = '192.168.1.1'

GENEVE_FRAME_L2 = {'srcmac':
                   '01:02:03:04:05:06',
                   'dstmac':
                   '06:05:04:03:02:01',
                  }

GENEVE_FRAME_L3 = {'proto': 'udp',
                   'packetsize': 64,
                   'srcip': TRAFFICGEN_PORT1_IP,
                   'dstip': '172.16.1.1',
                  }

GENEVE_FRAME_L4 = {'srcport': 6081,
                   'dstport': 6081,
                   'protocolpad': 'true',
                   'geneve_vni': 0,
                   'inner_srcmac': '01:02:03:04:05:06',
                   'inner_dstmac': '06:05:04:03:02:01',
                   'inner_srcip': '192.168.1.2',
                   'inner_dstip': TRAFFICGEN_PORT2_IP,
                   'inner_proto': 'udp',
                   'inner_srcport': 3000,
                   'inner_dstport': 3001,
                  }

NOTE: In case, that Vanilla OVS is installed from binary package, then please set PATHS['vswitch']['OvsVanilla']['bin']['modules'] instead.

  1. Run test:
./vsperf --conf-file user_settings.py --integration \
         --test-params 'tunnel_type=geneve' overlay_p2p_decap_cont
3.9. Executing Tunnel encapsulation+decapsulation tests

The OVS DPDK encapsulation/decapsulation tests requires IPs, MAC addresses, bridge names and WHITELIST_NICS for DPDK.

The test cases can test the tunneling encap and decap without using any ingress overlay traffic as compared to above test cases. To achieve this the OVS is configured to perform encap and decap in a series on the same traffic stream as given below.

TRAFFIC-IN –> [ENCAP] –> [MOD-PKT] –> [DECAP] –> TRAFFIC-OUT

Default values are already provided. To customize for your environment, override the following variables in you user_settings.py file:

# Variables defined in conf/integration/02_vswitch.conf

# Bridge names
TUNNEL_EXTERNAL_BRIDGE1 = 'br-phy1'
TUNNEL_EXTERNAL_BRIDGE2 = 'br-phy2'
TUNNEL_MODIFY_BRIDGE1 = 'br-mod1'
TUNNEL_MODIFY_BRIDGE2 = 'br-mod2'

# IP of br-mod1
TUNNEL_MODIFY_BRIDGE_IP1 = '10.0.0.1/24'

# Mac of br-mod1
TUNNEL_MODIFY_BRIDGE_MAC1 = '00:00:10:00:00:01'

# IP of br-mod2
TUNNEL_MODIFY_BRIDGE_IP2 = '20.0.0.1/24'

#Mac of br-mod2
TUNNEL_MODIFY_BRIDGE_MAC2 = '00:00:20:00:00:01'

# vxlan|gre|geneve, Only VXLAN is supported for now.
TUNNEL_TYPE = 'vxlan'

To run VXLAN encapsulation+decapsulation tests:

./vsperf --conf-file user_settings.py --integration \
         overlay_p2p_mod_tput
4. Execution of vswitchperf testcases by Yardstick
4.1. General

Yardstick is a generic framework for a test execution, which is used for validation of installation of OPNFV platform. In the future, Yardstick will support two options of vswitchperf testcase execution:

  • plugin mode, which will execute native vswitchperf testcases; Tests will be executed natively by vsperf, and test results will be processed and reported by yardstick.
  • traffic generator mode, which will run vswitchperf in trafficgen mode only; Yardstick framework will be used to launch VNFs and to configure flows to ensure, that traffic is properly routed. This mode will allow to test OVS performance in real world scenarios.

In Colorado release only the traffic generator mode is supported.

4.2. Yardstick Installation

In order to run Yardstick testcases, you will need to prepare your test environment. Please follow the installation instructions to install the yardstick.

Please note, that yardstick uses OpenStack for execution of testcases. OpenStack must be installed with Heat and Neutron services. Otherwise vswitchperf testcases cannot be executed.

4.3. VM image with vswitchperf

A special VM image is required for execution of vswitchperf specific testcases by yardstick. It is possible to use a sample VM image available at OPNFV artifactory or to build customized image.

4.3.1. Sample VM image with vswitchperf

Sample VM image is available at vswitchperf section of OPNFV artifactory for free download:

$ wget http://artifacts.opnfv.org/vswitchperf/vnf/vsperf-yardstick-image.qcow2

This image can be used for execution of sample testcases with dummy traffic generator.

NOTE: Traffic generators might require an installation of client software. This software is not included in the sample image and must be installed by user.

NOTE: This image will be updated only in case, that new features related to yardstick integration will be added to the vswitchperf.

4.3.2. Preparation of custom VM image

In general, any Linux distribution supported by vswitchperf can be used as a base image for vswitchperf. One of the possibilities is to modify vloop-vnf image, which can be downloaded from http://artifacts.opnfv.org/vswitchperf.html/ (see vloop-vnf).

Please follow the Installing vswitchperf to install vswitchperf inside vloop-vnf image. As vswitchperf will be run in trafficgen mode, it is possible to skip installation and compilation of OVS, QEMU and DPDK to keep image size smaller.

In case, that selected traffic generator requires installation of additional client software, please follow appropriate documentation. For example in case of IXIA, you would need to install IxOS and IxNetowrk TCL API.

4.3.3. VM image usage

Image with vswitchperf must be uploaded into the glance service and vswitchperf specific flavor configured, e.g.:

$ glance --os-username admin --os-image-api-version 1 image-create --name \
  vsperf --is-public true --disk-format qcow2 --container-format bare --file \
  vsperf-yardstick-image.qcow2

$ nova --os-username admin flavor-create vsperf-flavor 100 2048 25 1
4.4. Testcase execution

After installation, yardstick is available as python package within yardstick specific virtual environment. It means, that yardstick environment must be enabled before the test execution, e.g.:

source ~/yardstick_venv/bin/activate

Next step is configuration of OpenStack environment, e.g. in case of devstack:

source /opt/openstack/devstack/openrc
export EXTERNAL_NETWORK=public

Vswitchperf testcases executable by yardstick are located at vswitchperf repository inside yardstick/tests directory. Example of their download and execution follows:

git clone https://gerrit.opnfv.org/gerrit/vswitchperf
cd vswitchperf

yardstick -d task start yardstick/tests/rfc2544_throughput_dummy.yaml

NOTE: Optional argument -d shows debug output.

4.5. Testcase customization

Yardstick testcases are described by YAML files. vswitchperf specific testcases are part of the vswitchperf repository and their yaml files can be found at yardstick/tests directory. For detailed description of yaml file structure, please see yardstick documentation and testcase samples. Only vswitchperf specific parts will be discussed here.

Example of yaml file:

...
scenarios:
-
  type: Vsperf
  options:
    testname: 'p2p_rfc2544_throughput'
    trafficgen_port1: 'eth1'
    trafficgen_port2: 'eth3'
    external_bridge: 'br-ex'
    test_params: 'TRAFFICGEN_DURATION=30;TRAFFIC={'traffic_type':'rfc2544_throughput}'
    conf_file: '~/vsperf-yardstick.conf'

  host: vsperf.demo

  runner:
    type: Sequence
    scenario_option_name: frame_size
    sequence:
    - 64
    - 128
    - 512
    - 1024
    - 1518
  sla:
    metrics: 'throughput_rx_fps'
    throughput_rx_fps: 500000
    action: monitor

context:
...
4.5.1. Section option

Section option defines details of vswitchperf test scenario. Lot of options are identical to the vswitchperf parameters passed through --test-params argument. Following options are supported:

  • frame_size - a packet size for which test should be executed; Multiple packet sizes can be tested by modification of Sequence runner section inside YAML definition. Default: ‘64’
  • conf_file - sets path to the vswitchperf configuration file, which will be uploaded to VM; Default: ‘~/vsperf-yardstick.conf’
  • setup_script - sets path to the setup script, which will be executed during setup and teardown phases
  • trafficgen_port1 - specifies device name of 1st interface connected to the trafficgen
  • trafficgen_port2 - specifies device name of 2nd interface connected to the trafficgen
  • external_bridge - specifies name of external bridge configured in OVS; Default: ‘br-ex’
  • test_params - specifies a string with a list of vsperf configuration parameters, which will be passed to the --test-params CLI argument; Parameters should be stated in the form of param=value and separated by a semicolon. Configuration of traffic generator is driven by TRAFFIC dictionary, which can be also updated by values defined by test_params. Please check VSPERF documentation for details about available configuration parameters and their data types. In case that both test_params and conf_file are specified, then values from test_params will override values defined in the configuration file.

In case that trafficgen_port1 and/or trafficgen_port2 are defined, then these interfaces will be inserted into the external_bridge of OVS. It is expected, that OVS runs at the same node, where the testcase is executed. In case of more complex OpenStack installation or a need of additional OVS configuration, setup_script can be used.

NOTE It is essential to specify a configuration for selected traffic generator. In case, that standalone testcase is created, then traffic generator can be selected and configured directly in YAML file by test_params. On the other hand, if multiple testcases should be executed with the same traffic generator settings, then a customized configuration file should be prepared and its name passed by conf_file option.

4.5.2. Section runner

Yardstick supports several runner types. In case of vswitchperf specific TCs, Sequence runner type can be used to execute the testcase for given list of frame sizes.

4.5.3. Section sla

In case that sla section is not defined, then testcase will be always considered as successful. On the other hand, it is possible to define a set of test metrics and their minimal values to evaluate test success. Any numeric value, reported by vswitchperf inside CSV result file, can be used. Multiple metrics can be defined as a coma separated list of items. Minimal value must be set separately for each metric.

e.g.:

sla:
    metrics: 'throughput_rx_fps,throughput_rx_mbps'
    throughput_rx_fps: 500000
    throughput_rx_mbps: 1000

In case that any of defined metrics will be lower than defined value, then testcase will be marked as failed. Based on action policy, yardstick will either stop test execution (value assert) or it will run next test (value monitor).

NOTE The throughput SLA (or any other SLA) cannot be set to a meaningful value without knowledge of the server and networking environment, possibly including prior testing in that environment to establish a baseline SLA level under well-understood circumstances.

5. List of vswitchperf testcases
5.1. Performance testcases
Testcase Name Description
phy2phy_tput LTD.Throughput.RFC2544.PacketLossRatio
phy2phy_forwarding LTD.Forwarding.RFC2889.MaxForwardingRate
phy2phy_learning LTD.AddrLearning.RFC2889.AddrLearningRate
phy2phy_caching LTD.AddrCaching.RFC2889.AddrCachingCapacity
back2back LTD.Throughput.RFC2544.BackToBackFrames
phy2phy_tput_mod_vlan LTD.Throughput.RFC2544.PacketLossRatioFrameModification
phy2phy_cont Phy2Phy Continuous Stream
pvp_cont PVP Continuous Stream
pvvp_cont PVVP Continuous Stream
pvpv_cont Two VMs in parallel with Continuous Stream
phy2phy_scalability LTD.Scalability.Flows.RFC2544.0PacketLoss
pvp_tput LTD.Throughput.RFC2544.PacketLossRatio
pvp_back2back LTD.Throughput.RFC2544.BackToBackFrames
pvvp_tput LTD.Throughput.RFC2544.PacketLossRatio
pvvp_back2back LTD.Throughput.RFC2544.BackToBackFrames
phy2phy_cpu_load LTD.CPU.RFC2544.0PacketLoss
phy2phy_mem_load LTD.Memory.RFC2544.0PacketLoss
phy2phy_tput_vpp VPP: LTD.Throughput.RFC2544.PacketLossRatio
phy2phy_cont_vpp VPP: Phy2Phy Continuous Stream
phy2phy_back2back_vpp VPP: LTD.Throughput.RFC2544.BackToBackFrames
pvp_tput_vpp VPP: LTD.Throughput.RFC2544.PacketLossRatio
pvp_cont_vpp VPP: PVP Continuous Stream
pvp_back2back_vpp VPP: LTD.Throughput.RFC2544.BackToBackFrames
pvvp_tput_vpp VPP: LTD.Throughput.RFC2544.PacketLossRatio
pvvp_cont_vpp VPP: PVP Continuous Stream
pvvp_back2back_vpp VPP: LTD.Throughput.RFC2544.BackToBackFrames

List of performance testcases above can be obtained by execution of:

$ ./vsperf --list
5.2. Integration testcases
Testcase Name Description
vswitch_vports_add_del_flow vSwitch - configure switch with vports, add and delete flow
vswitch_add_del_flows vSwitch - add and delete flows
vswitch_p2p_tput vSwitch - configure switch and execute RFC2544 throughput test
vswitch_p2p_back2back vSwitch - configure switch and execute RFC2544 back2back test
vswitch_p2p_cont vSwitch - configure switch and execute RFC2544 continuous stream test
vswitch_pvp vSwitch - configure switch and one vnf
vswitch_vports_pvp vSwitch - configure switch with vports and one vnf
vswitch_pvp_tput vSwitch - configure switch, vnf and execute RFC2544 throughput test
vswitch_pvp_back2back vSwitch - configure switch, vnf and execute RFC2544 back2back test
vswitch_pvp_cont vSwitch - configure switch, vnf and execute RFC2544 continuous stream test
vswitch_pvp_all vSwitch - configure switch, vnf and execute all test types
vswitch_pvvp vSwitch - configure switch and two vnfs
vswitch_pvvp_tput vSwitch - configure switch, two chained vnfs and execute RFC2544 throughput test
vswitch_pvvp_back2back vSwitch - configure switch, two chained vnfs and execute RFC2544 back2back test
vswitch_pvvp_cont vSwitch - configure switch, two chained vnfs and execute RFC2544 continuous stream test
vswitch_pvvp_all vSwitch - configure switch, two chained vnfs and execute all test types
vswitch_p4vp Just configure 4 chained vnfs
vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test
vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test
vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test
vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput test
2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port
4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port
6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port
vhost_numa_awareness vSwitch DPDK - verify that PMD threads are served by the same NUMA slot as QEMU instances
ixnet_pvp_tput_1nic PVP Scenario with 1 port towards IXIA
vswitch_vports_add_del_connection_vpp VPP: vSwitch - configure switch with vports, add and delete connection
p2p_l3_multi_IP_ovs OVS: P2P L3 multistream with unique flow for each IP stream
p2p_l3_multi_IP_mask_ovs OVS: P2P L3 multistream with 1 flow for /8 net mask
pvp_l3_multi_IP_mask_ovs OVS: PVP L3 multistream with 1 flow for /8 net mask
pvvp_l3_multi_IP_mask_ovs OVS: PVVP L3 multistream with 1 flow for /8 net mask
p2p_l4_multi_PORT_ovs OVS: P2P L4 multistream with unique flow for each IP stream
p2p_l4_multi_PORT_mask_ovs OVS: P2P L4 multistream with 1 flow for /8 net and port mask
pvp_l4_multi_PORT_mask_ovs OVS: PVP L4 multistream flows for /8 net and port mask
pvvp_l4_multi_PORT_mask_ovs OVS: PVVP L4 multistream with flows for /8 net and port mask
p2p_l3_multi_IP_arp_vpp VPP: P2P L3 multistream with unique ARP entry for each IP stream
p2p_l3_multi_IP_mask_vpp VPP: P2P L3 multistream with 1 route for /8 net mask
p2p_l3_multi_IP_routes_vpp VPP: P2P L3 multistream with unique route for each IP stream
pvp_l3_multi_IP_mask_vpp VPP: PVP L3 multistream with route for /8 netmask
pvvp_l3_multi_IP_mask_vpp VPP: PVVP L3 multistream with route for /8 netmask
p2p_l4_multi_PORT_arp_vpp VPP: P2P L4 multistream with unique ARP entry for each IP stream and port check
p2p_l4_multi_PORT_mask_vpp VPP: P2P L4 multistream with 1 route for /8 net mask and port check
p2p_l4_multi_PORT_routes_vpp VPP: P2P L4 multistream with unique route for each IP stream and port check
pvp_l4_multi_PORT_mask_vpp VPP: PVP L4 multistream with route for /8 net and port mask
pvvp_l4_multi_PORT_mask_vpp VPP: PVVP L4 multistream with route for /8 net and port mask
vxlan_multi_IP_mask_ovs OVS: VxLAN L3 multistream
vxlan_multi_IP_arp_vpp VPP: VxLAN L3 multistream with unique ARP entry for each IP stream
vxlan_multi_IP_mask_vpp VPP: VxLAN L3 multistream with 1 route for /8 netmask

List of integration testcases above can be obtained by execution of:

$ ./vsperf --integration --list
5.3. OVS/DPDK Regression TestCases

These regression tests verify several DPDK features used internally by Open vSwitch. Tests can be used for verification of performance and correct functionality of upcoming DPDK and OVS releases and release candidates.

These tests are part of integration testcases and they must be executed with --integration CLI parameter.

Example of execution of all OVS/DPDK regression tests:

$ ./vsperf --integration --tests ovsdpdk_

Testcases are defined in the file conf/integration/01b_dpdk_regression_tests.conf. This file contains a set of configuration options with prefix OVSDPDK_. These parameters can be used for customization of regression tests and they will override some of standard VSPERF configuration options. It is recommended to check OVSDPDK configuration parameters and modify them in accordance with VSPERF configuration.

At least following parameters should be examined. Their values shall ensure, that DPDK and QEMU threads are pinned to cpu cores of the same NUMA slot, where tested NICs are connected.

_OVSDPDK_1st_PMD_CORE
_OVSDPDK_2nd_PMD_CORE
_OVSDPDK_GUEST_5_CORES
5.3.1. DPDK NIC Support

A set of performance tests to verify support of DPDK accelerated network interface cards. Testcases use standard physical to physical network scenario with several vSwitch and traffic configurations, which includes one and two PMD threads, uni and bidirectional traffic and RFC2544 Continuous or RFC2544 Throughput with 0% packet loss traffic types.

Testcase Name Description
ovsdpdk_nic_p2p_single_pmd_unidir_cont P2P with single PMD in OVS and unidirectional traffic.
ovsdpdk_nic_p2p_single_pmd_bidir_cont P2P with single PMD in OVS and bidirectional traffic.
ovsdpdk_nic_p2p_two_pmd_bidir_cont P2P with two PMDs in OVS and bidirectional traffic.
ovsdpdk_nic_p2p_single_pmd_unidir_tput P2P with single PMD in OVS and unidirectional traffic.
ovsdpdk_nic_p2p_single_pmd_bidir_tput P2P with single PMD in OVS and bidirectional traffic.
ovsdpdk_nic_p2p_two_pmd_bidir_tput P2P with two PMDs in OVS and bidirectional traffic.
5.3.2. DPDK Hotplug Support

A set of functional tests to verify DPDK hotplug support. Tests verify, that it is possible to use port, which was not bound to DPDK driver during vSwitch startup. There is also a test which verifies a possibility to detach port from DPDK driver. However support for manual detachment of a port from DPDK has been removed from recent OVS versions and thus this testcase is expected to fail.

Testcase Name Description
ovsdpdk_hotplug_attach Ensure successful port-add after binding a device to igb_uio after ovs-vswitchd is launched.
ovsdpdk_hotplug_detach Same as ovsdpdk_hotplug_attach, but delete and detach the device after the hotplug. Note Support of netdev-dpdk/detach has been removed from OVS, so testcase will fail with recent OVS/DPDK versions.
5.3.3. RX Checksum Support

A set of functional tests for verification of RX checksum calculation for tunneled traffic. Open vSwitch enables RX checksum offloading by default if NIC supports it. It is to note, that it is not possible to disable or enable RX checksum offloading. In order to verify correct RX checksum calculation in software, user has to execute these testcases at NIC without HW offloading capabilities.

Testcases utilize existing overlay physical to physical (op2p) network deployment implemented in vsperf. This deployment expects, that traffic generator sends unidirectional tunneled traffic (e.g. vxlan) and Open vSwitch performs data decapsulation and sends them back to the traffic generator via second port.

Testcase Name Description
ovsdpdk_checksum_l3 Test verifies RX IP header checksum (offloading) validation for tunneling protocols.
ovsdpdk_checksum_l4 Test verifies RX UDP header checksum (offloading) validation for tunneling protocols.
5.3.4. Flow Control Support

A set of functional testcases for the validation of flow control support in Open vSwitch with DPDK support. If flow control is enabled in both OVS and Traffic Generator, the network endpoint (OVS or TGEN) is not able to process incoming data and thus it detects a RX buffer overflow. It then sends an ethernet pause frame (as defined at 802.3x) to the TX side. This mechanism will ensure, that the TX side will slow down traffic transmission and thus no data is lost at RX side.

Introduced testcases use physical to physical scenario to forward data between traffic generator ports. It is expected that the processing of small frames in OVS is slower than line rate. It means that with flow control disabled, traffic generator will report a frame loss. On the other hand with flow control enabled, there should be 0% frame loss reported by traffic generator.

Testcase Name Description
ovsdpdk_flow_ctrl_rx Test the rx flow control functionality of DPDK PHY ports.
ovsdpdk_flow_ctrl_rx_dynamic Change the rx flow control support at run time and ensure the system honored the changes.
5.3.5. Multiqueue Support

A set of functional testcases for validation of multiqueue support for both physical and vHost User DPDK ports. Testcases utilize P2P and PVP network deployments and native support of multiqueue configuration available in VSPERF.

Testcase Name Description
ovsdpdk_mq_p2p_rxqs Setup rxqs on NIC port.
ovsdpdk_mq_p2p_rxqs_same_core_affinity Affinitize rxqs to the same core.
ovsdpdk_mq_p2p_rxqs_multi_core_affinity Affinitize rxqs to separate cores.
ovsdpdk_mq_pvp_rxqs Setup rxqs on vhost user port.
ovsdpdk_mq_pvp_rxqs_linux_bridge Confirm traffic received over vhost RXQs with Linux virtio device in guest.
ovsdpdk_mq_pvp_rxqs_testpmd Confirm traffic received over vhost RXQs with DPDK device in guest.
5.3.6. Vhost User

A set of functional testcases for validation of vHost User Client and vHost User Server modes in OVS.

NOTE: Vhost User Server mode is deprecated and it will be removed from OVS in the future.

Testcase Name Description
ovsdpdk_vhostuser_client Test vhost-user client mode
ovsdpdk_vhostuser_client_reconnect Test vhost-user client mode reconnect feature
ovsdpdk_vhostuser_server Test vhost-user server mode
ovsdpdk_vhostuser_sock_dir Verify functionality of vhost-sock-dir flag
5.3.7. Virtual Devices Support

A set of functional testcases for verification of correct functionality of virtual device PMD drivers.

Testcase Name Description
ovsdpdk_vdev_add_null_pmd Test addition of port using the null DPDK PMD driver.
ovsdpdk_vdev_del_null_pmd Test deletion of port using the null DPDK PMD driver.
ovsdpdk_vdev_add_af_packet_pmd Test addition of port using the af_packet DPDK PMD driver.
ovsdpdk_vdev_del_af_packet_pmd Test deletion of port using the af_packet DPDK PMD driver.
5.3.8. NUMA Support

A functional testcase for validation of NUMA awareness feature in OVS.

Testcase Name Description
ovsdpdk_numa Test vhost-user NUMA support. Vhostuser PMD threads should migrate to the same numa slot, where QEMU is executed.
5.3.9. Jumbo Frame Support

A set of functional testcases for verification of jumbo frame support in OVS. Testcases utilize P2P and PVP network deployments and native support of jumbo frames available in VSPERF.

Testcase Name Description
ovsdpdk_jumbo_increase_mtu_phy_port_ovsdb Ensure that the increased MTU for a DPDK physical port is updated in OVSDB.
ovsdpdk_jumbo_increase_mtu_vport_ovsdb Ensure that the increased MTU for a DPDK vhost-user port is updated in OVSDB.
ovsdpdk_jumbo_reduce_mtu_phy_port_ovsdb Ensure that the reduced MTU for a DPDK physical port is updated in OVSDB.
ovsdpdk_jumbo_reduce_mtu_vport_ovsdb Ensure that the reduced MTU for a DPDK vhost-user port is updated in OVSDB.
ovsdpdk_jumbo_increase_mtu_phy_port_datapath Ensure that the MTU for a DPDK physical port is updated in the datapath itself when increased to a valid value.
ovsdpdk_jumbo_increase_mtu_vport_datapath Ensure that the MTU for a DPDK vhost-user port is updated in the datapath itself when increased to a valid value.
ovsdpdk_jumbo_reduce_mtu_phy_port_datapath Ensure that the MTU for a DPDK physical port is updated in the datapath itself when decreased to a valid value.
ovsdpdk_jumbo_reduce_mtu_vport_datapath Ensure that the MTU for a DPDK vhost-user port is updated in the datapath itself when decreased to a valid value.
ovsdpdk_jumbo_mtu_upper_bound_phy_port Verify that the upper bound limit is enforced for OvS DPDK Phy ports.
ovsdpdk_jumbo_mtu_upper_bound_vport Verify that the upper bound limit is enforced for OvS DPDK vhost-user ports.
ovsdpdk_jumbo_mtu_lower_bound_phy_port Verify that the lower bound limit is enforced for OvS DPDK Phy ports.
ovsdpdk_jumbo_mtu_lower_bound_vport Verify that the lower bound limit is enforced for OvS DPDK vhost-user ports.
ovsdpdk_jumbo_p2p Ensure that jumbo frames are received, processed and forwarded correctly by DPDK physical ports.
ovsdpdk_jumbo_pvp Ensure that jumbo frames are received, processed and forwarded correctly by DPDK vhost-user ports.
ovsdpdk_jumbo_p2p_upper_bound Ensure that jumbo frames above the configured Rx port’s MTU are not accepted
5.3.10. Rate Limiting

A set of functional testcases for validation of rate limiting support. This feature allows to configure an ingress policing for both physical and vHost User DPDK ports.

NOTE: Desired maximum rate is specified in kilo bits per second and it defines the rate of payload only.

Testcase Name Description
ovsdpdk_rate_create_phy_port Ensure a rate limiting interface can be created on a physical DPDK port.
ovsdpdk_rate_delete_phy_port Ensure a rate limiting interface can be destroyed on a physical DPDK port.
ovsdpdk_rate_create_vport Ensure a rate limiting interface can be created on a vhost-user port.
ovsdpdk_rate_delete_vport Ensure a rate limiting interface can be destroyed on a vhost-user port.
ovsdpdk_rate_no_policing Ensure when a user attempts to create a rate limiting interface but is missing policing rate argument, no rate limitiner is created.
ovsdpdk_rate_no_burst Ensure when a user attempts to create a rate limiting interface but is missing policing burst argument, rate limitiner is created.
ovsdpdk_rate_p2p Ensure when a user creates a rate limiting physical interface that the traffic is limited to the specified policer rate in a p2p setup.
ovsdpdk_rate_pvp Ensure when a user creates a rate limiting vHost User interface that the traffic is limited to the specified policer rate in a pvp setup.
ovsdpdk_rate_p2p_multi_pkt_sizes Ensure that rate limiting works for various frame sizes.
5.3.11. Quality of Service

A set of functional testcases for validation of QoS support. This feature allows to configure an egress policing for both physical and vHost User DPDK ports.

NOTE: Desired maximum rate is specified in bytes per second and it defines the rate of payload only.

Testcase Name Description
ovsdpdk_qos_create_phy_port Ensure a QoS policy can be created on a physical DPDK port
ovsdpdk_qos_delete_phy_port Ensure an existing QoS policy can be destroyed on a physical DPDK port.
ovsdpdk_qos_create_vport Ensure a QoS policy can be created on a virtual vhost user port.
ovsdpdk_qos_delete_vport Ensure an existing QoS policy can be destroyed on a vhost user port.
ovsdpdk_qos_create_no_cir Ensure that a QoS policy cannot be created if the egress policer cir argument is missing.
ovsdpdk_qos_create_no_cbs Ensure that a QoS policy cannot be created if the egress policer cbs argument is missing.
ovsdpdk_qos_p2p In a p2p setup, ensure when a QoS egress policer is created that the traffic is limited to the specified rate.
ovsdpdk_qos_pvp In a pvp setup, ensure when a QoS egress policer is created that the traffic is limited to the specified rate.

Yardstick

Yardstick User Guide
1. Introduction

Welcome to Yardstick’s documentation !

Yardstick is an OPNFV Project.

The project’s goal is to verify infrastructure compliance, from the perspective of a Virtual Network Function (VNF).

The Project’s scope is the development of a test framework, Yardstick, test cases and test stimuli to enable Network Function Virtualization Infrastructure (NFVI) verification.

Yardstick is used in OPNFV for verifying the OPNFV infrastructure and some of the OPNFV features. The Yardstick framework is deployed in several OPNFV community labs. It is installer, infrastructure and application independent.

See also

Pharos for information on OPNFV community labs and this Presentation for an overview of Yardstick

1.1. About This Document

This document consists of the following chapters:

  • Chapter Introduction provides a brief introduction to Yardstick project’s background and describes the structure of this document.
  • Chapter Methodology describes the methodology implemented by the Yardstick Project for NFVI verification.
  • Chapter Architecture provides information on the software architecture of Yardstick.
  • Chapter Yardstick Installation provides instructions to install Yardstick.
  • Chapter Installing a plug-in into Yardstick provides information on how to integrate other OPNFV testing projects into Yardstick.
  • Chapter Store Other Project’s Test Results in InfluxDB provides inforamtion on how to run plug-in test cases and store test results into community’s InfluxDB.
  • Chapter Grafana dashboard provides inforamtion on Yardstick grafana dashboard and how to add a dashboard into Yardstick grafana dashboard.
  • Chapter Yardstick Restful API provides inforamtion on Yardstick ReST API and how to use Yardstick API.
  • Chapter Yardstick User Interface provides inforamtion on how to use yardstick report CLI to view the test result in table format and also values pinned on to a graph
  • Chapter Virtual Traffic Classifier provides information on the VTC.
  • Chapter 13-nsb-overview describes the methodology implemented by the Yardstick - Network service benchmarking to test real world usecase for a given VNF.
  • Chapter 14-nsb_installation provides instructions to install Yardstick - Network service benchmarking testing.
  • Chapter Yardstick Test Cases includes a list of available Yardstick test cases.
1.2. Contact Yardstick

Feedback? Contact us

2. Methodology
2.1. Abstract

This chapter describes the methodology implemented by the Yardstick project for verifying the NFVI from the perspective of a VNF.

2.2. ETSI-NFV

The document ETSI GS NFV-TST001, “Pre-deployment Testing; Report on Validation of NFV Environments and Services”, recommends methods for pre-deployment testing of the functional components of an NFV environment.

The Yardstick project implements the methodology described in chapter 6, “Pre- deployment validation of NFV infrastructure”.

The methodology consists in decomposing the typical VNF work-load performance metrics into a number of characteristics/performance vectors, which each can be represented by distinct test-cases.

The methodology includes five steps:

  • Step1: Define Infrastruture - the Hardware, Software and corresponding

    configuration target for validation; the OPNFV infrastructure, in OPNFV community labs.

  • Step2: Identify VNF type - the application for which the

    infrastructure is to be validated, and its requirements on the underlying infrastructure.

  • Step3: Select test cases - depending on the workload that represents the

    application for which the infrastruture is to be validated, the relevant test cases amongst the list of available Yardstick test cases.

  • Step4: Execute tests - define the duration and number of iterations for the

    selected test cases, tests runs are automated via OPNFV Jenkins Jobs.

  • Step5: Collect results - using the common API for result collection.

See also

Yardsticktst for material on alignment ETSI TST001 and Yardstick.

2.3. Metrics

The metrics, as defined by ETSI GS NFV-TST001, are shown in Table1, Table2 and Table3.

In OPNFV Colorado release, generic test cases covering aspects of the listed metrics are available; further OPNFV releases will provide extended testing of these metrics. The view of available Yardstick test cases cross ETSI definitions in Table1, Table2 and Table3 is shown in Table4. It shall be noticed that the Yardstick test cases are examples, the test duration and number of iterations are configurable, as are the System Under Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario options).

Table 1 - Performance/Speed Metrics

Category Performance/Speed
Compute
  • Latency for random memory access
  • Latency for cache read/write operations
  • Processing speed (instructions per second)
  • Throughput for random memory access (bytes per second)
Network
  • Throughput per NFVI node (frames/byte per second)
  • Throughput provided to a VM (frames/byte per second)
  • Latency per traffic flow
  • Latency between VMs
  • Latency between NFVI nodes
  • Packet delay variation (jitter) between VMs
  • Packet delay variation (jitter) between NFVI nodes
Storage
  • Sequential read/write IOPS
  • Random read/write IOPS
  • Latency for storage read/write operations
  • Throughput for storage read/write operations

Table 2 - Capacity/Scale Metrics

Category Capacity/Scale
Compute
  • Number of cores and threads- Available memory size
  • Cache size
  • Processor utilization (max, average, standard deviation)
  • Memory utilization (max, average, standard deviation)
  • Cache utilization (max, average, standard deviation)
Network
  • Number of connections
  • Number of frames sent/received
  • Maximum throughput between VMs (frames/byte per second)
  • Maximum throughput between NFVI nodes (frames/byte per second)
  • Network utilization (max, average, standard deviation)
  • Number of traffic flows
Storage
  • Storage/Disk size
  • Capacity allocation (block-based, object-based)
  • Block size
  • Maximum sequential read/write IOPS
  • Maximum random read/write IOPS
  • Disk utilization (max, average, standard deviation)

Table 3 - Availability/Reliability Metrics

Category Availability/Reliability
Compute
  • Processor availability (Error free processing time)
  • Memory availability (Error free memory time)
  • Processor mean-time-to-failure
  • Memory mean-time-to-failure
  • Number of processing faults per second
Network
  • NIC availability (Error free connection time)
  • Link availability (Error free transmission time)
  • NIC mean-time-to-failure
  • Network timeout duration due to link failure
  • Frame loss rate
Storage
  • Disk availability (Error free disk access time)
  • Disk mean-time-to-failure
  • Number of failed storage read/write operations per second

Table 4 - Yardstick Generic Test Cases

Category Performance/Speed Capacity/Scale Availability/Reliability
Compute TC003 [1] TC004 TC010 TC012 TC014 TC069 TC003 [1] TC004 TC024 TC055 TC013 [1] TC015 [1]
Network TC001 TC002 TC009 TC011 TC042 TC043 TC044 TC073 TC075 TC016 [1] TC018 [1]
Storage TC005 TC063 TC017 [1]

Note

The description in this OPNFV document is intended as a reference for users to understand the scope of the Yardstick Project and the deliverables of the Yardstick framework. For complete description of the methodology, please refer to the ETSI document.

Footnotes

[1](1, 2, 3, 4, 5, 6, 7) To be included in future deliveries.
3. Architecture
3.1. Abstract

This chapter describes the yardstick framework software architecture. we will introduce it from Use-Case View, Logical View, Process View and Deployment View. More technical details will be introduced in this chapter.

3.2. Overview
3.2.1. Architecture overview

Yardstick is mainly written in Python, and test configurations are made in YAML. Documentation is written in reStructuredText format, i.e. .rst files. Yardstick is inspired by Rally. Yardstick is intended to run on a computer with access and credentials to a cloud. The test case is described in a configuration file given as an argument.

How it works: the benchmark task configuration file is parsed and converted into an internal model. The context part of the model is converted into a Heat template and deployed into a stack. Each scenario is run using a runner, either serially or in parallel. Each runner runs in its own subprocess executing commands in a VM using SSH. The output of each scenario is written as json records to a file or influxdb or http server, we use influxdb as the backend, the test result will be shown with grafana.

3.2.2. Concept

Benchmark - assess the relative performance of something

Benchmark configuration file - describes a single test case in yaml format

Context - The set of Cloud resources used by a scenario, such as user names, image names, affinity rules and network configurations. A context is converted into a simplified Heat template, which is used to deploy onto the Openstack environment.

Data - Output produced by running a benchmark, written to a file in json format

Runner - Logic that determines how a test scenario is run and reported, for example the number of test iterations, input value stepping and test duration. Predefined runner types exist for re-usage, see Runner types.

Scenario - Type/class of measurement for example Ping, Pktgen, (Iperf, LmBench, ...)

SLA - Relates to what result boundary a test case must meet to pass. For example a latency limit, amount or ratio of lost packets and so on. Action based on SLA can be configured, either just to log (monitor) or to stop further testing (assert). The SLA criteria is set in the benchmark configuration file and evaluated by the runner.

3.2.3. Runner types

There exists several predefined runner types to choose between when designing a test scenario:

Arithmetic: Every test run arithmetically steps the specified input value(s) in the test scenario, adding a value to the previous input value. It is also possible to combine several input values for the same test case in different combinations.

Snippet of an Arithmetic runner configuration:

runner:
    type: Arithmetic
    iterators:
    -
      name: stride
      start: 64
      stop: 128
      step: 64

Duration: The test runs for a specific period of time before completed.

Snippet of a Duration runner configuration:

runner:
  type: Duration
  duration: 30

Sequence: The test changes a specified input value to the scenario. The input values to the sequence are specified in a list in the benchmark configuration file.

Snippet of a Sequence runner configuration:

runner:
  type: Sequence
  scenario_option_name: packetsize
  sequence:
  - 100
  - 200
  - 250

Iteration: Tests are run a specified number of times before completed.

Snippet of an Iteration runner configuration:

runner:
  type: Iteration
  iterations: 2
3.3. Use-Case View

Yardstick Use-Case View shows two kinds of users. One is the Tester who will do testing in cloud, the other is the User who is more concerned with test result and result analyses.

For testers, they will run a single test case or test case suite to verify infrastructure compliance or bencnmark their own infrastructure performance. Test result will be stored by dispatcher module, three kinds of store method (file, influxdb and http) can be configured. The detail information of scenarios and runners can be queried with CLI by testers.

For users, they would check test result with four ways.

If dispatcher module is configured as file(default), there are two ways to check test result. One is to get result from yardstick.out ( default path: /tmp/yardstick.out), the other is to get plot of test result, it will be shown if users execute command “yardstick-plot”.

If dispatcher module is configured as influxdb, users will check test result on Grafana which is most commonly used for visualizing time series data.

If dispatcher module is configured as http, users will check test result on OPNFV testing dashboard which use MongoDB as backend.

Yardstick Use-Case View
3.4. Logical View

Yardstick Logical View describes the most important classes, their organization, and the most important use-case realizations.

Main classes:

TaskCommands - “yardstick task” subcommand handler.

HeatContext - Do test yaml file context section model convert to HOT, deploy and undeploy Openstack heat stack.

Runner - Logic that determines how a test scenario is run and reported.

TestScenario - Type/class of measurement for example Ping, Pktgen, (Iperf, LmBench, ...)

Dispatcher - Choose user defined way to store test results.

TaskCommands is the “yardstick task” subcommand’s main entry. It takes yaml file (e.g. test.yaml) as input, and uses HeatContext to convert the yaml file’s context section to HOT. After Openstack heat stack is deployed by HeatContext with the converted HOT, TaskCommands use Runner to run specified TestScenario. During first runner initialization, it will create output process. The output process use Dispatcher to push test results. The Runner will also create a process to execute TestScenario. And there is a multiprocessing queue between each runner process and output process, so the runner process can push the real-time test results to the storage media. TestScenario is commonly connected with VMs by using ssh. It sets up VMs and run test measurement scripts through the ssh tunnel. After all TestScenaio is finished, TaskCommands will undeploy the heat stack. Then the whole test is finished.

Yardstick framework architecture in Danube
3.5. Process View (Test execution flow)

Yardstick process view shows how yardstick runs a test case. Below is the sequence graph about the test execution flow using heat context, and each object represents one module in yardstick:

Yardstick Process View

A user wants to do a test with yardstick. He can use the CLI to input the command to start a task. “TaskCommands” will receive the command and ask “HeatContext” to parse the context. “HeatContext” will then ask “Model” to convert the model. After the model is generated, “HeatContext” will inform “Openstack” to deploy the heat stack by heat template. After “Openstack” deploys the stack, “HeatContext” will inform “Runner” to run the specific test case.

Firstly, “Runner” would ask “TestScenario” to process the specific scenario. Then “TestScenario” will start to log on the openstack by ssh protocal and execute the test case on the specified VMs. After the script execution finishes, “TestScenario” will send a message to inform “Runner”. When the testing job is done, “Runner” will inform “Dispatcher” to output the test result via file, influxdb or http. After the result is output, “HeatContext” will call “Openstack” to undeploy the heat stack. Once the stack is undepoyed, the whole test ends.

3.6. Deployment View

Yardstick deployment view shows how the yardstick tool can be deployed into the underlying platform. Generally, yardstick tool is installed on JumpServer(see 07-installation for detail installation steps), and JumpServer is connected with other control/compute servers by networking. Based on this deployment, yardstick can run the test cases on these hosts, and get the test result for better showing.

Yardstick Deployment View
3.7. Yardstick Directory structure

yardstick/ - Yardstick main directory.

tests/ci/ - Used for continuous integration of Yardstick at different PODs and
with support for different installers.
docs/ - All documentation is stored here, such as configuration guides,
user guides and Yardstick descriptions.

etc/ - Used for test cases requiring specific POD configurations.

samples/ - test case samples are stored here, most of all scenario and
feature’s samples are shown in this directory.
tests/ - Here both Yardstick internal tests (functional/ and unit/) as
well as the test cases run to verify the NFVI (opnfv/) are stored. Also configurations of what to run daily and weekly at the different PODs is located here.
tools/ - Currently contains tools to build image for VMs which are deployed
by Heat. Currently contains how to build the yardstick-trusty-server image with the different tools that are needed from within the image.

plugin/ - Plug-in configuration files are stored here.

vTC/ - Contains the files for running the virtual Traffic Classifier tests.

yardstick/ - Contains the internals of Yardstick: Runners, Scenario, Contexts,
CLI parsing, keys, plotting tools, dispatcher, plugin install/remove scripts and so on.
4. Yardstick Installation
4.1. Abstract

Yardstick supports installation by Docker or directly in Ubuntu. The installation procedure for Docker and direct installation are detailed in the sections below.

To use Yardstick you should have access to an OpenStack environment, with at least Nova, Neutron, Glance, Keystone and Heat installed.

The steps needed to run Yardstick are:

  1. Install Yardstick.
  2. Load OpenStack environment variables.
  3. Create Yardstick flavor.
  4. Build a guest image and load it into the OpenStack environment.
  5. Create the test configuration .yaml file and run the test case/suite.
4.2. Prerequisites

The OPNFV deployment is out of the scope of this document and can be found here. The OPNFV platform is considered as the System Under Test (SUT) in this document.

Several prerequisites are needed for Yardstick:

  1. A Jumphost to run Yardstick on
  2. A Docker daemon or a virtual environment installed on the Jumphost
  3. A public/external network created on the SUT
  4. Connectivity from the Jumphost to the SUT public/external network

NOTE: Jumphost refers to any server which meets the previous requirements. Normally it is the same server from where the OPNFV deployment has been triggered.

WARNING: Connectivity from Jumphost is essential and it is of paramount importance to make sure it is working before even considering to install and run Yardstick. Make also sure you understand how your networking is designed to work.

NOTE: If your Jumphost is operating behind a company http proxy and/or Firewall, please consult first the section `Proxy Support (**Todo**)`_, towards the end of this document. That section details some tips/tricks which may be of help in a proxified environment.

4.4. Install Yardstick directly in Ubuntu

Alternatively you can install Yardstick framework directly in Ubuntu or in an Ubuntu Docker image. No matter which way you choose to install Yardstick, the following installation steps are identical.

If you choose to use the Ubuntu Docker image, you can pull the Ubuntu Docker image from Docker hub:

docker pull ubuntu:16.04
4.4.1. Install Yardstick

Prerequisite preparation:

apt-get update && apt-get install -y git python-setuptools python-pip
easy_install -U setuptools==30.0.0
pip install appdirs==1.4.0
pip install virtualenv

Create a virtual environment:

virtualenv ~/yardstick_venv
export YARDSTICK_VENV=~/yardstick_venv
source ~/yardstick_venv/bin/activate

Download the source code and install Yardstick from it:

git clone https://gerrit.opnfv.org/gerrit/yardstick
export YARDSTICK_REPO_DIR=~/yardstick
cd yardstick
./install.sh
4.4.2. Configure the Yardstick environment (Todo)

For installing Yardstick directly in Ubuntu, the yardstick env command is not available. You need to prepare OpenStack environment variables and create Yardstick flavor and guest images manually.

4.4.3. Uninstall Yardstick

For unistalling Yardstick, just delete the virtual environment:

rm -rf ~/yardstick_venv
4.5. Verify the installation

It is recommended to verify that Yardstick was installed successfully by executing some simple commands and test samples. Before executing Yardstick test cases make sure yardstick-flavor and yardstick-image can be found in OpenStack and the openrc file is sourced. Below is an example invocation of Yardstick help command and ping.py test sample:

yardstick -h
yardstick task start samples/ping.yaml

NOTE: The above commands could be run in both the Yardstick container and the Ubuntu directly.

Each testing tool supported by Yardstick has a sample configuration file. These configuration files can be found in the samples directory.

Default location for the output is /tmp/yardstick.out.

4.6. Deploy InfluxDB and Grafana using Docker

Without InfluxDB, Yardstick stores results for runnning test case in the file /tmp/yardstick.out. However, it’s unconvenient to retrieve and display test results. So we will show how to use InfluxDB to store data and use Grafana to display data in the following sections.

4.6.2. Manually deploy InfluxDB and Grafana containers

You could also deploy influxDB and Grafana containers manually on the Jumphost. The following sections show how to do.

4.6.2.1. Pull docker images
docker pull tutum/influxdb
docker pull grafana/grafana
4.6.2.2. Run and configure influxDB

Run influxDB:

docker run -d --name influxdb \
-p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \
tutum/influxdb
docker exec -it influxdb bash

Configure influxDB:

influx
>CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
>CREATE DATABASE yardstick;
>use yardstick;
>show MEASUREMENTS;
4.6.2.3. Run and configure Grafana

Run Grafana:

docker run -d --name grafana -p 3000:3000 grafana/grafana

Log on http://{YOUR_IP_HERE}:3000 using admin/admin and configure database resource to be {YOUR_IP_HERE}:8086.

Grafana data source configration
4.6.2.4. Configure yardstick.conf
docker exec -it yardstick /bin/bash
cp etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
vi /etc/yardstick/yardstick.conf

Modify yardstick.conf:

[DEFAULT]
debug = True
dispatcher = influxdb

[dispatcher_influxdb]
timeout = 5
target = http://{YOUR_IP_HERE}:8086
db_name = yardstick
username = root
password = root

Now you can run Yardstick test cases and store the results in influxDB.

4.7. Deploy InfluxDB and Grafana directly in Ubuntu (Todo)
4.8. Yardstick common CLI

yardstick testcase list

This command line would list all test cases in yardstick. It would show like below:

+---------------------------------------------------------------------------------------
| Testcase Name         | Description
+---------------------------------------------------------------------------------------
| opnfv_yardstick_tc001 | Measure network throughput using pktgen
| opnfv_yardstick_tc002 | measure network latency using ping
| opnfv_yardstick_tc005 | Measure Storage IOPS, throughput and latency using fio.
| opnfv_yardstick_tc006 | Measure volume storage IOPS, throughput and latency using fio.
| opnfv_yardstick_tc008 | Measure network throughput and packet loss using Pktgen
| opnfv_yardstick_tc009 | Measure network throughput and packet loss using pktgen
| opnfv_yardstick_tc010 | measure memory read latency using lmbench.
| opnfv_yardstick_tc011 | Measure packet delay variation (jitter) using iperf3.
| opnfv_yardstick_tc012 | Measure memory read and write bandwidth using lmbench.
| opnfv_yardstick_tc014 | Measure Processing speed using unixbench.
| opnfv_yardstick_tc019 | Sample test case for the HA of controller node service.
...
+---------------------------------------------------------------------------------------

Take opnfv_yardstick_tc002 for an example. This test case measure network latency. You just need to type in yardstick testcase show opnfv_yardstick_tc002, and the console would show the config yaml of this test case:

##############################################################################
# Copyright (c) 2017 kristian.hunt@gmail.com and others.
#
# All rights reserved. This program and the accompanying materials
# are made available under the terms of the Apache License, Version 2.0
# which accompanies this distribution, and is available at
# http://www.apache.org/licenses/LICENSE-2.0
##############################################################################
---

schema: "yardstick:task:0.1"
description: >
    Yardstick TC002 config file;
    measure network latency using ping;

{% set image = image or "cirros-0.3.5" %}

{% set provider = provider or none %}
{% set physical_network = physical_network or 'physnet1' %}
{% set segmentation_id = segmentation_id or none %}
{% set packetsize = packetsize or 100 %}

scenarios:
{% for i in range(2) %}
-
  type: Ping
  options:
    packetsize: {{packetsize}}
  host: athena.demo
  target: ares.demo

  runner:
    type: Duration
    duration: 60
    interval: 10

  sla:
    max_rtt: 10
    action: monitor
{% endfor %}

context:
  name: demo
  image: {{image}}
  flavor: yardstick-flavor
  user: cirros

  placement_groups:
    pgrp1:
      policy: "availability"

  servers:
    athena:
      floating_ip: true
      placement: "pgrp1"
    ares:
      placement: "pgrp1"

  networks:
    test:
      cidr: '10.0.1.0/24'
      {% if provider == "vlan" %}
      provider: {{provider}}
      physical_network: {{physical_network}}å
        {% if segmentation_id %}
      segmentation_id: {{segmentation_id}}
        {% endif %}
      {% endif %}

If you want run a test case, then you need to use yardstick task start <test_case_path> this command support some parameters as below:

Parameters Detail
-d show debug log of yardstick running
–task-args If you want to customize test case parameters, use “–task-args” to pass the value. The format is a json string with parameter key-value pair.
–task-args-file If you want to use yardstick env prepare command(or related API) to load the
–parse-only  
–output-file OUTPUT_FILE_PATH Specify where to output the log. if not pass, the default value is “/tmp/yardstick/yardstick.log”
–suite TEST_SUITE_PATH run a test suite, TEST_SUITE_PATH speciy where the test suite locates
4.9. Run Yardstick in a local environment

We also have a guide about how to run Yardstick in a local environment. This work is contributed by Tapio Tallgren. You can find this guide at here.

4.10. Create a test suite for Yardstick

A test suite in yardstick is a yaml file which include one or more test cases. Yardstick is able to support running test suite task, so you can customize your own test suite and run it in one task.

tests/opnfv/test_suites is the folder where Yardstick puts CI test suite. A typical test suite is like below (the fuel_test_suite.yaml example):

---
# Fuel integration test task suite

schema: "yardstick:suite:0.1"

name: "fuel_test_suite"
test_cases_dir: "samples/"
test_cases:
-
  file_name: ping.yaml
-
  file_name: iperf3.yaml

As you can see, there are two test cases in the fuel_test_suite.yaml. The schema and the name must be specified. The test cases should be listed via the tag test_cases and their relative path is also marked via the tag test_cases_dir.

Yardstick test suite also supports constraints and task args for each test case. Here is another sample (the os-nosdn-nofeature-ha.yaml example) to show this, which is digested from one big test suite:

---

schema: "yardstick:suite:0.1"

name: "os-nosdn-nofeature-ha"
test_cases_dir: "tests/opnfv/test_cases/"
test_cases:
-
    file_name: opnfv_yardstick_tc002.yaml
-
    file_name: opnfv_yardstick_tc005.yaml
-
    file_name: opnfv_yardstick_tc043.yaml
       constraint:
          installer: compass
          pod: huawei-pod1
       task_args:
          huawei-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
          "host": "node4.LF","target": "node5.LF"}'

As you can see in test case opnfv_yardstick_tc043.yaml, there are two tags, constraint and task_args. constraint is to specify which installer or pod it can be run in the CI environment. task_args is to specify the task arguments for each pod.

All in all, to create a test suite in Yardstick, you just need to create a yaml file and add test cases, constraint or task arguments if necessary.

4.11. Proxy Support (Todo)
5. Installing a plug-in into Yardstick
5.1. Abstract

Yardstick provides a plugin CLI command to support integration with other OPNFV testing projects. Below is an example invocation of Yardstick plugin command and Storperf plug-in sample.

5.2. Installing Storperf into Yardstick

Storperf is delivered as a Docker container from https://hub.docker.com/r/opnfv/storperf/tags/.

There are two possible methods for installation in your environment:

  • Run container on Jump Host
  • Run container in a VM

In this introduction we will install Storperf on Jump Host.

5.2.1. Step 0: Environment preparation

Running Storperf on Jump Host Requirements:

  • Docker must be installed
  • Jump Host must have access to the OpenStack Controller API
  • Jump Host must have internet connectivity for downloading docker image
  • Enough floating IPs must be available to match your agent count

Before installing Storperf into yardstick you need to check your openstack environment and other dependencies:

  1. Make sure docker is installed.
  2. Make sure Keystone, Nova, Neutron, Glance, Heat are installed correctly.
  3. Make sure Jump Host have access to the OpenStack Controller API.
  4. Make sure Jump Host must have internet connectivity for downloading docker image.
  5. You need to know where to get basic openstack Keystone authorization info, such as OS_PASSWORD, OS_TENANT_NAME, OS_AUTH_URL, OS_USERNAME.
  6. To run a Storperf container, you need to have OpenStack Controller environment variables defined and passed to Storperf container. The best way to do this is to put environment variables in a “storperf_admin-rc” file. The storperf_admin-rc should include credential environment variables at least:
  • OS_AUTH_URL
  • OS_USERNAME
  • OS_PASSWORD
  • OS_TENANT_ID
  • OS_TENANT_NAME
  • OS_PROJECT_NAME
  • OS_PROJECT_ID
  • OS_USER_DOMAIN_ID

Yardstick has a “prepare_storperf_admin-rc.sh” script which can be used to generate the “storperf_admin-rc” file, this script is located at test/ci/prepare_storperf_admin-rc.sh

#!/bin/bash
# Prepare storperf_admin-rc for StorPerf.
AUTH_URL=${OS_AUTH_URL}
USERNAME=${OS_USERNAME:-admin}
PASSWORD=${OS_PASSWORD:-console}

TENANT_NAME=${OS_TENANT_NAME:-admin}
TENANT_ID=`openstack project show admin|grep '\bid\b' |awk -F '|' '{print $3}'|sed -e 's/^[[:space:]]*//'`
PROJECT_NAME=${OS_PROJECT_NAME:-$TENANT_NAME}
PROJECT_ID=`openstack project show admin|grep '\bid\b' |awk -F '|' '{print $3}'|sed -e 's/^[[:space:]]*//'`
USER_DOMAIN_ID=${OS_USER_DOMAIN_ID:-default}

rm -f ~/storperf_admin-rc
touch ~/storperf_admin-rc

echo "OS_AUTH_URL="$AUTH_URL >> ~/storperf_admin-rc
echo "OS_USERNAME="$USERNAME >> ~/storperf_admin-rc
echo "OS_PASSWORD="$PASSWORD >> ~/storperf_admin-rc
echo "OS_PROJECT_NAME="$PROJECT_NAME >> ~/storperf_admin-rc
echo "OS_PROJECT_ID="$PROJECT_ID >> ~/storperf_admin-rc
echo "OS_TENANT_NAME="$TENANT_NAME >> ~/storperf_admin-rc
echo "OS_TENANT_ID="$TENANT_ID >> ~/storperf_admin-rc
echo "OS_USER_DOMAIN_ID="$USER_DOMAIN_ID >> ~/storperf_admin-rc

The generated “storperf_admin-rc” file will be stored in the root directory. If you installed Yardstick using Docker, this file will be located in the container. You may need to copy it to the root directory of the Storperf deployed host.

5.2.2. Step 1: Plug-in configuration file preparation

To install a plug-in, first you need to prepare a plug-in configuration file in YAML format and store it in the “plugin” directory. The plugin configration file work as the input of yardstick “plugin” command. Below is the Storperf plug-in configuration file sample:

---
# StorPerf plugin configuration file
# Used for integration StorPerf into Yardstick as a plugin
schema: "yardstick:plugin:0.1"
plugins:
  name: storperf
deployment:
  ip: 192.168.23.2
  user: root
  password: root

In the plug-in configuration file, you need to specify the plug-in name and the plug-in deployment info, including node ip, node login username and password. Here the Storperf will be installed on IP 192.168.23.2 which is the Jump Host in my local environment.

5.2.3. Step 2: Plug-in install/remove scripts preparation

In “yardstick/resource/scripts” directory, there are two folders: a “install” folder and a “remove” folder. You need to store the plug-in install/remove scripts in these two folders respectively.

The detailed installation or remove operation should de defined in these two scripts. The name of both install and remove scripts should match the plugin-in name that you specified in the plug-in configuration file.

For example, the install and remove scripts for Storperf are both named to “storperf.bash”.

5.2.4. Step 3: Install and remove Storperf

To install Storperf, simply execute the following command:

# Install Storperf
yardstick plugin install plugin/storperf.yaml
5.2.4.1. removing Storperf from yardstick

To remove Storperf, simply execute the following command:

# Remove Storperf
yardstick plugin remove plugin/storperf.yaml

What yardstick plugin command does is using the username and password to log into the deployment target and then execute the corresponding install or remove script.

6. Store Other Project’s Test Results in InfluxDB
6.1. Abstract

This chapter illustrates how to run plug-in test cases and store test results into community’s InfluxDB. The framework is shown in Framework.

Store Other Project's Test Results in InfluxDB
6.2. Store Storperf Test Results into Community’s InfluxDB

As shown in Framework, there are two ways to store Storperf test results into community’s InfluxDB:

  1. Yardstick executes Storperf test case (TC074), posting test job to Storperf container via ReST API. After the test job is completed, Yardstick reads test results via ReST API from Storperf and posts test data to the influxDB.
  2. Additionally, Storperf can run tests by itself and post the test result directly to the InfluxDB. The method for posting data directly to influxDB will be supported in the future.

Our plan is to support rest-api in D release so that other testing projects can call the rest-api to use yardstick dispatcher service to push data to yardstick’s influxdb database.

For now, influxdb only support line protocol, and the json protocol is deprecated.

Take ping test case for example, the raw_result is json format like this:

  "benchmark": {
      "timestamp": 1470315409.868095,
      "errors": "",
      "data": {
        "rtt": {
        "ares": 1.125
        }
      },
    "sequence": 1
    },
  "runner_id": 2625
}

With the help of “influxdb_line_protocol”, the json is transform to like below as a line string:

'ping,deploy_scenario=unknown,host=athena.demo,installer=unknown,pod_name=unknown,
  runner_id=2625,scenarios=Ping,target=ares.demo,task_id=77755f38-1f6a-4667-a7f3-
    301c99963656,version=unknown rtt.ares=1.125 1470315409868094976'

So, for data output of json format, you just need to transform json into line format and call influxdb api to post the data into the database. All this function has been implemented in Influxdb. If you need support on this, please contact Mingjiang.

curl -i -XPOST 'http://104.197.68.199:8086/write?db=yardstick' --
  data-binary 'ping,deploy_scenario=unknown,host=athena.demo,installer=unknown, ...'

Grafana will be used for visualizing the collected test data, which is shown in Visual. Grafana can be accessed by Login.

results visualization
7. Grafana dashboard
7.1. Abstract

This chapter describes the Yardstick grafana dashboard. The Yardstick grafana dashboard can be found here: http://testresults.opnfv.org/grafana/

Yardstick grafana dashboard
7.2. Public access

Yardstick provids a public account for accessing to the dashboard. The username and password are both set to ‘opnfv’.

7.3. Testcase dashboard

For each test case, there is a dedicated dashboard. Shown here is the dashboard of TC002.

For each test case dashboard. On the top left, we have a dashboard selection, you can switch to different test cases using this pull-down menu.

Underneath, we have a pod and scenario selection. All the pods and scenarios that have ever published test data to the InfluxDB will be shown here.

You can check multiple pods or scenarios.

For each test case, we have a short description and a link to detailed test case information in Yardstick user guide.

Underneath, it is the result presentation section. You can use the time period selection on the top right corner to zoom in or zoom out the chart.

7.4. Administration access

For a user with administration rights it is easy to update and save any dashboard configuration. Saved updates immediately take effect and become live. This may cause issues like:

  • Changes and updates made to the live configuration in Grafana can compromise existing Grafana content in an unwanted, unpredicted or incompatible way. Grafana as such is not version controlled, there exists one single Grafana configuration per dashboard.
  • There is a risk several people can disturb each other when doing updates to the same Grafana dashboard at the same time.

Any change made by administrator should be careful.

7.5. Add a dashboard into yardstick grafana

Due to security concern, users that using the public opnfv account are not able to edit the yardstick grafana directly.It takes a few more steps for a non-yardstick user to add a custom dashboard into yardstick grafana.

There are 6 steps to go.

Add a dashboard into yardstick grafana
  1. You need to build a local influxdb and grafana, so you can do the work locally. You can refer to How to deploy InfluxDB and Grafana locally wiki page about how to do this.
  2. Once step one is done, you can fetch the existing grafana dashboard configuration file from the yardstick repository and import it to your local grafana. After import is done, you grafana dashboard will be ready to use just like the community’s dashboard.
  3. The third step is running some test cases to generate test results and publishing it to your local influxdb.
  4. Now you have some data to visualize in your dashboard. In the fourth step, it is time to create your own dashboard. You can either modify an existing dashboard or try to create a new one from scratch. If you choose to modify an existing dashboard then in the curtain menu of the existing dashboard do a “Save As...” into a new dashboard copy instance, and then continue doing all updates and saves within the dashboard copy.
  5. When finished with all Grafana configuration changes in this temporary dashboard then chose “export” of the updated dashboard copy into a JSON file and put it up for review in Gerrit, in file /yardstick/dashboard/Yardstick-TCxxx-yyyyyyyyyyyyy. For instance a typical default name of the file would be “Yardstick-TC001 Copy-1234567891234”.
  6. Once you finish your dashboard, the next step is exporting the configuration file and propose a patch into Yardstick. Yardstick team will review and merge it into Yardstick repository. After approved review Yardstick team will do an “import” of the JSON file and also a “save dashboard” as soon as possible to replace the old live dashboard configuration.
8. Yardstick Restful API
8.1. Abstract

Yardstick support restful API since Danube.

8.2. Available API
8.2.1. /yardstick/env/action

Description: This API is used to prepare Yardstick test environment. For Euphrates, it supports:

  1. Prepare yardstick test environment, including set external network environment variable, load Yardstick VM images and create flavors;
  2. Start an InfluxDB Docker container and config Yardstick output to InfluxDB;
  3. Start a Grafana Docker container and config it with the InfluxDB.

Which API to call will depend on the parameters.

Method: POST

Prepare Yardstick test environment Example:

{
    'action': 'prepareYardstickEnv'
}

This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.

Start and config an InfluxDB docker container Example:

{
    'action': 'createInfluxDBContainer'
}

This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.

Start and config a Grafana docker container Example:

{
    'action': 'createGrafanaContainer'
}

This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result.

8.2.2. /yardstick/asynctask

Description: This API is used to get the status of asynchronous tasks

Method: GET

Get the status of asynchronous tasks Example:

http://localhost:8888/yardstick/asynctask?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c

The returned status will be 0(running), 1(finished) and 2(failed).

8.2.3. /yardstick/testcases

Description: This API is used to list all released Yardstick test cases.

Method: GET

Get a list of released test cases Example:

http://localhost:8888/yardstick/testcases
8.2.4. /yardstick/testcases/release/action

Description: This API is used to run a Yardstick released test case.

Method: POST

Run a released test case Example:

{
    'action': 'runTestCase',
    'args': {
        'opts': {},
        'testcase': 'tc002'
    }
}

This is an asynchronous API. You need to call /yardstick/results to get the result.

8.2.5. /yardstick/testcases/samples/action

Description: This API is used to run a Yardstick sample test case.

Method: POST

Run a sample test case Example:

{
    'action': 'runTestCase',
    'args': {
        'opts': {},
        'testcase': 'ping'
    }
}

This is an asynchronous API. You need to call /yardstick/results to get the result.

8.2.6. /yardstick/testcases/<testcase_name>/docs

Description: This API is used to the documentation of a certain released test case.

Method: GET

Get the documentation of a certain test case Example:

http://localhost:8888/yardstick/taskcases/opnfv_yardstick_tc002/docs
8.2.7. /yardstick/testsuites/action

Description: This API is used to run a Yardstick test suite.

Method: POST

Run a test suite Example:

{
    'action': 'runTestSuite',
    'args': {
        'opts': {},
        'testcase': 'smoke'
    }
}

This is an asynchronous API. You need to call /yardstick/results to get the result.

/yardstick/tasks/<task_id>/log

Description: This API is used to get the real time log of test case execution.

Method: GET

Get real time of test case execution Example:

http://localhost:8888/yardstick/tasks/14795be8-f144-4f54-81ce-43f4e3eab33f/log?index=0
8.2.8. /yardstick/results

Description: This API is used to get the test results of tasks. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use the returned task id to get the results by using this API.

Method: GET

Get test results of one task Example:

http://localhost:8888/yardstick/results?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c

This API will return a list of test case result

/api/v2/yardstick/openrcs/action

Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports:

  1. Upload an openrc file for an OpenStack environment;
  2. Update an openrc file;
  3. Get openrc file information;
  4. Delete an openrc file.

Which API to call will depend on the parameters.

METHOD: POST

Upload an openrc file for an OpenStack environment Example:

{
    'action': 'upload_openrc',
    'args': {
        'file': file,
        'environment_id': environment_id
    }
}

METHOD: POST

Update an openrc file Example:

{
    'action': 'update_openrc',
    'args': {
        'openrc': {
            "EXTERNAL_NETWORK": "ext-net",
            "OS_AUTH_URL": "http://192.168.23.51:5000/v3",
            "OS_IDENTITY_API_VERSION": "3",
            "OS_IMAGE_API_VERSION": "2",
            "OS_PASSWORD": "console",
            "OS_PROJECT_DOMAIN_NAME": "default",
            "OS_PROJECT_NAME": "admin",
            "OS_TENANT_NAME": "admin",
            "OS_USERNAME": "admin",
            "OS_USER_DOMAIN_NAME": "default"
        },
        'environment_id': environment_id
    }
}

METHOD: GET

Get openrc file information Example:

http://localhost:8888/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete openrc file Example:

http://localhost:8888/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/pods/action

Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports:

  1. Upload a pod file;
  2. Get pod file information;
  3. Delete an openrc file.

Which API to call will depend on the parameters.

METHOD: POST

Upload a pod.yaml file Example:

{
    'action': 'upload_pod_file',
    'args': {
        'file': file,
        'environment_id': environment_id
    }
}

METHOD: GET

Get pod file information Example:

http://localhost:8888/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete openrc file Example:

http://localhost:8888/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/images/action

Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports:

  1. Load Yardstick VM images;
  2. Get image’s information;
  3. Delete images.

Which API to call will depend on the parameters.

METHOD: POST

Load VM images Example:

{
    'action': 'load_images'
}

METHOD: GET

Get image information Example:

http://localhost:8888/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete images Example:

http://localhost:8888/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/tasks/action

Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports:

  1. Create a Yardstick task;
  2. run a Yardstick task;
  3. Add a test case to a task;
  4. Add a test suite to a task;
  5. Get a tasks’ information;
  6. Delete a task.

Which API to call will depend on the parameters.

METHOD: POST

Create a Yardstick task Example:

{
    'action': 'create_task',
        'args': {
            'name': 'task1',
            'project_id': project_id
        }
}

METHOD: PUT

Run a task Example:

{
    'action': 'run'
}

METHOD: PUT

Add a test case to a task Example:

{
    'action': 'add_case',
    'args': {
        'case_name': 'opnfv_yardstick_tc002',
        'case_content': case_content
    }
}

METHOD: PUT

Add a test suite to a task Example:

{
    'action': 'add_suite',
    'args': {
        'suite_name': 'opnfv_smoke',
        'suite_content': suite_content
    }
}

METHOD: GET

Get a task’s information Example:

http://localhost:8888/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c

METHOD: DELETE

Delete a task Example:

http://localhost:8888/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c

/api/v2/yardstick/testcases/action

Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports:

  1. Upload a test case;
  2. Get all released test cases’ information;
  3. Get a certain released test case’s information;
  4. Delete a test case.

Which API to call will depend on the parameters.

METHOD: POST

Upload a test case Example:

{
    'action': 'upload_case',
    'args': {
        'file': file
    }
}

METHOD: GET

Get all released test cases’ information Example:

http://localhost:8888/api/v2/yardstick/testcases

METHOD: GET

Get a certain released test case’s information Example:

http://localhost:8888/api/v2/yardstick/testcases/opnfv_yardstick_tc002

METHOD: DELETE

Delete a certain test case Example:

http://localhost:8888/api/v2/yardstick/testcases/opnfv_yardstick_tc002

/api/v2/yardstick/testsuites/action

Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports:

  1. Create a test suite;
  2. Get a certain test suite’s information;
  3. Get all test suites;
  4. Delete a test case.

Which API to call will depend on the parameters.

METHOD: POST

Create a test suite Example:

{
    'action': 'create_sutie',
    'args': {
        'name': <suite_name>,
        'testcases': [
            'opnfv_yardstick_tc002'
        ]
    }
}

METHOD: GET

Get a certain test suite’s information Example:

http://localhost:8888/api/v2/yardstick/testsuites/<suite_name>

METHOD: GET

Get all test suite Example:

http://localhost:8888/api/v2/yardstick/testsuites

METHOD: DELETE

Delete a certain test suite Example:

http://localhost:8888/api/v2/yardstick/testsuites/<suite_name>

/api/v2/yardstick/projects/action

Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports:

  1. Create a Yardstick project;
  2. Get a certain project’s information;
  3. Get all projects;
  4. Delete a project.

Which API to call will depend on the parameters.

METHOD: POST

Create a Yardstick project Example:

{
    'action': 'create_project',
    'args': {
        'name': 'project1'
    }
}

METHOD: GET

Get a certain project’s information Example:

http://localhost:8888/api/v2/yardstick/projects/<project_id>

METHOD: GET

Get all projects’ information Example:

http://localhost:8888/api/v2/yardstick/projects

METHOD: DELETE

Delete a certain project Example:

http://localhost:8888/api/v2/yardstick/projects/<project_id>

/api/v2/yardstick/containers/action

Description: This API is used to do some work related to Docker containers. For Euphrates, it supports:

  1. Create a Grafana Docker container;
  2. Create an InfluxDB Docker container;
  3. Get a certain container’s information;
  4. Delete a container.

Which API to call will depend on the parameters.

METHOD: POST

Create a Grafana Docker container Example:

{
    'action': 'create_grafana',
    'args': {
        'environment_id': <environment_id>
    }
}

METHOD: POST

Create an InfluxDB Docker container Example:

{
    'action': 'create_influxdb',
    'args': {
        'environment_id': <environment_id>
    }
}

METHOD: GET

Get a certain container’s information Example:

http://localhost:8888/api/v2/yardstick/containers/<container_id>

METHOD: DELETE

Delete a certain container Example:

http://localhost:8888/api/v2/yardstick/containers/<container_id>
9. Yardstick User Interface

This interface provides a user to view the test result in table format and also values pinned on to a graph.

9.1. Command
yardstick report generate <task-ID> <testcase-filename>
9.2. Description

1. When the command is triggered using the task-id and the testcase name provided the respective values are retrieved from the database (influxdb in this particular case).

2. The values are then formatted and then provided to the html template framed with complete html body using Django Framework.

  1. Then the whole template is written into a html file.

The graph is framed with Timestamp on x-axis and output values (differ from testcase to testcase) on y-axis with the help of “Highcharts”.

10. Virtual Traffic Classifier
10.1. Abstract

This chapter provides an overview of the virtual Traffic Classifier, a contribution to OPNFV Yardstick from the EU Project TNOVA. Additional documentation is available in TNOVAresults.

10.2. Overview

The virtual Traffic Classifier (VTC) VNF, comprises of a Virtual Network Function Component (VNFC). The VNFC contains both the Traffic Inspection module, and the Traffic forwarding module, needed to run the VNF. The exploitation of Deep Packet Inspection (DPI) methods for traffic classification is built around two basic assumptions:

  • third parties unaffiliated with either source or recipient are able to

inspect each IP packet’s payload

  • the classifier knows the relevant syntax of each application’s packet

payloads (protocol signatures, data patterns, etc.).

The proposed DPI based approach will only use an indicative, small number of the initial packets from each flow in order to identify the content and not inspect each packet.

In this respect it follows the Packet Based per Flow State (term:PBFS). This method uses a table to track each session based on the 5-tuples (src address, dest address, src port,dest port, transport protocol) that is maintained for each flow.

10.3. Concepts
  • Traffic Inspection: The process of packet analysis and application

identification of network traffic that passes through the VTC.

  • Traffic Forwarding: The process of packet forwarding from an incoming

network interface to a pre-defined outgoing network interface.

  • Traffic Rule Application: The process of packet tagging, based on a

predefined set of rules. Packet tagging may include e.g. Type of Service (ToS) field modification.

10.4. Architecture

The Traffic Inspection module is the most computationally intensive component of the VNF. It implements filtering and packet matching algorithms in order to support the enhanced traffic forwarding capability of the VNF. The component supports a flow table (exploiting hashing algorithms for fast indexing of flows) and an inspection engine for traffic classification.

The implementation used for these experiments exploits the nDPI library. The packet capturing mechanism is implemented using libpcap. When the DPI engine identifies a new flow, the flow register is updated with the appropriate information and transmitted across the Traffic Forwarding module, which then applies any required policy updates.

The Traffic Forwarding moudle is responsible for routing and packet forwarding. It accepts incoming network traffic, consults the flow table for classification information for each incoming flow and then applies pre-defined policies marking e.g. ToS/Differentiated Services Code Point (DSCP) multimedia traffic for Quality of Service (QoS) enablement on the forwarded traffic. It is assumed that the traffic is forwarded using the default policy until it is identified and new policies are enforced.

The expected response delay is considered to be negligible, as only a small number of packets are required to identify each flow.

10.5. Graphical Overview
+----------------------------+
|                            |
| Virtual Traffic Classifier |
|                            |
|     Analysing/Forwarding   |
|        ------------>       |
|     ethA          ethB     |
|                            |
+----------------------------+
     |              ^
     |              |
     v              |
+----------------------------+
|                            |
|     Virtual Switch         |
|                            |
+----------------------------+
10.6. Install

run the vTC/build.sh with root privileges

10.7. Run
sudo ./pfbridge -a eth1 -b eth2

Note

Virtual Traffic Classifier is not support in OPNFV Danube release.

10.8. Development Environment

Ubuntu 14.04 Ubuntu 16.04

11. Network Services Benchmarking (NSB)
11.1. Abstract

This chapter provides an overview of the NSB, a contribution to OPNFV Yardstick from Intel.

11.2. Overview

The goal of NSB is to Extend Yardstick to perform real world VNFs and NFVi Characterization and benchmarking with repeatable and deterministic methods.

The Network Service Benchmarking (NSB) extends the yardstick framework to do VNF characterization and benchmarking in three different execution environments - bare metal i.e. native Linux environment, standalone virtual environment and managed virtualized environment (e.g. Open stack etc.). It also brings in the capability to interact with external traffic generators both hardware & software based for triggering and validating the traffic according to user defined profiles.

NSB extension includes:

  • Generic data models of Network Services, based on ETSI spec ETSI GS NFV-TST 001

  • New Standalone context for VNF testing like SRIOV, OVS, OVS-DPDK etc

  • Generic VNF configuration models and metrics implemented with Python classes

  • Traffic generator features and traffic profiles

    • L1-L3 state-less traffic profiles
    • L4-L7 state-full traffic profiles
    • Tunneling protocol / network overlay support
  • Test case samples

    • Ping
    • Trex
    • vPE,vCGNAT, vFirewall etc - ipv4 throughput, latency etc
  • Traffic generators like Trex, ab/nginx, ixia, iperf etc

  • KPIs for a given use case:

    • System agent support for collecting NFVi KPI. This includes:

      • CPU statistic
      • Memory BW
      • OVS-DPDK Stats
    • Network KPIs, e.g., inpackets, outpackets, thoughput, latency etc

    • VNF KPIs, e.g., packet_in, packet_drop, packet_fwd etc

11.3. Architecture

The Network Service (NS) defines a set of Virtual Network Functions (VNF) connected together using NFV infrastructure.

The Yardstick NSB extension can support multiple VNFs created by different vendors including traffic generators. Every VNF being tested has its own data model. The Network service defines a VNF modelling on base of performed network functionality. The part of the data model is a set of the configuration parameters, number of connection points used and flavor including core and memory amount.

The ETSI defines a Network Service as a set of configurable VNFs working in some NFV Infrastructure connecting each other using Virtual Links available through Connection Points. The ETSI MANO specification defines a set of management entities called Network Service Descriptors (NSD) and VNF Descriptors (VNFD) that define real Network Service. The picture below makes an example how the real Network Operator use-case can map into ETSI Network service definition

Network Service framework performs the necessary test steps. It may involve

  • Interacting with traffic generator and providing the inputs on traffic type / packet structure to generate the required traffic as per the test case. Traffic profiles will be used for this.
  • Executing the commands required for the test procedure and analyses the command output for confirming whether the command got executed correctly or not. E.g. As per the test case, run the traffic for the given time period / wait for the necessary time delay
  • Verify the test result.
  • Validate the traffic flow from SUT
  • Fetch the table / data from SUT and verify the value as per the test case
  • Upload the logs from SUT onto the Test Harness server
  • Read the KPI’s provided by particular VNF
11.3.1. Components of Network Service
  • Models for Network Service benchmarking: The Network Service benchmarking requires the proper modelling approach. The NSB provides models using Python files and defining of NSDs and VNFDs.

The benchmark control application being a part of OPNFV yardstick can call that python models to instantiate and configure the VNFs. Depending on infrastructure type (bare-metal or fully virtualized) that calls could be made directly or using MANO system.

  • Traffic generators in NSB: Any benchmark application requires a set of traffic generator and traffic profiles defining the method in which traffic is generated.

The Network Service benchmarking model extends the Network Service definition with a set of Traffic Generators (TG) that are treated same way as other VNFs being a part of benchmarked network service. Same as other VNFs the traffic generator are instantiated and terminated.

Every traffic generator has own configuration defined as a traffic profile and a set of KPIs supported. The python models for TG is extended by specific calls to listen and generate traffic.

  • The stateless TREX traffic generator: The main traffic generator used as Network Service stimulus is open source TREX tool.

The TREX tool can generate any kind of stateless traffic.

+--------+      +-------+      +--------+
|        |      |       |      |        |
|  Trex  | ---> |  VNF  | ---> |  Trex  |
|        |      |       |      |        |
+--------+      +-------+      +--------+

Supported testcases scenarios:

  • Correlated UDP traffic using TREX traffic generator and replay VNF.

    • using different IMIX configuration like pure voice, pure video traffic etc
    • using different number IP flows like 1 flow, 1K, 16K, 64K, 256K, 1M flows
    • Using different number of rules configured like 1 rule, 1K, 10K rules

For UDP correlated traffic following Key Performance Indicators are collected for every combination of test case parameters:

  • RFC2544 throughput for various loss rate defined (1% is a default)
11.4. Graphical Overview

NSB Testing with yardstick framework facilitate performance testing of various VNFs provided.

+-----------+
|           |                                                     +-----------+
|   vPE     |                                                   ->|TGen Port 0|
| TestCase  |                                                   | +-----------+
|           |                                                   |
+-----------+     +------------------+            +-------+     |
                  |                  | -- API --> |  VNF  | <--->
+-----------+     |     Yardstick    |            +-------+     |
| Test Case | --> |    NSB Testing   |                          |
+-----------+     |                  |                          |
      |           |                  |                          |
      |           +------------------+                          |
+-----------+                                                   | +-----------+
|   Traffic |                                                   ->|TGen Port 1|
|  patterns |                                                     +-----------+
+-----------+

            Figure 1: Network Service - 2 server configuration
11.4.1. VNFs supported for chracterization:
  1. CGNAPT - Carrier Grade Network Address and port Translation
  2. vFW - Virtual Firewall

3. vACL - Access Control List 5. Prox - Packet pROcessing eXecution engine:

  • VNF can act as Drop, Basic Forwarding (no touch), L2 Forwarding (change MAC), GRE encap/decap, Load balance based on packet fields, Symmetric load balancing,
  • QinQ encap/decap IPv4/IPv6, ARP, QoS, Routing, Unmpls, Policing, ACL
  1. UDP_Replay
12. Yardstick - NSB Testing -Installation
12.1. Abstract

The Network Service Benchmarking (NSB) extends the yardstick framework to do VNF characterization and benchmarking in three different execution environments viz., bare metal i.e. native Linux environment, standalone virtual environment and managed virtualized environment (e.g. Open stack etc.). It also brings in the capability to interact with external traffic generators both hardware & software based for triggering and validating the traffic according to user defined profiles.

The steps needed to run Yardstick with NSB testing are:

  • Install Yardstick (NSB Testing).
  • Setup/Reference pod.yaml describing Test topology
  • Create/Reference the test configuration yaml file.
  • Run the test case.
12.2. Prerequisites

Refer chapter Yardstick Installation for more information on yardstick prerequisites

Several prerequisites are needed for Yardstick(VNF testing):

  • Python Modules: pyzmq, pika.
  • flex
  • bison
  • build-essential
  • automake
  • libtool
  • librabbitmq-dev
  • rabbitmq-server
  • collectd
  • intel-cmt-cat
12.2.1. Hardware & Software Ingredients

SUT requirements:

Item Description
Memory Min 20GB
NICs 2 x 10G
OS Ubuntu 16.04.3 LTS
kernel 4.4.0-34-generic
DPDK 17.02

Boot and BIOS settings:

Boot settings default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 isolcpus=1-11,22-33 nohz_full=1-11,22-33 rcu_nocbs=1-11,22-33 iommu=on iommu=pt intel_iommu=on Note: nohz_full and rcu_nocbs is to disable Linux kernel interrupts
BIOS CPU Power and Performance Policy <Performance> CPU C-state Disabled CPU P-state Disabled Enhanced Intel® Speedstep® Tech Disabled Hyper-Threading Technology (If supported) Enabled Virtualization Techology Enabled Intel(R) VT for Direct I/O Enabled Coherency Enabled Turbo Boost Disabled
12.3. Install Yardstick (NSB Testing)

Download the source code and install Yardstick from it

git clone https://gerrit.opnfv.org/gerrit/yardstick

cd yardstick

# Switch to latest stable branch
# git checkout <tag or stable branch>
git checkout stable/euphrates

# For Bare-Metal or Standalone Virtualization
./nsb_setup.sh

# For OpenStack
./nsb_setup.sh <path to admin-openrc.sh>

Above command setup docker with latest yardstick code. To execute

docker exec -it yardstick bash

It will also automatically download all the packages needed for NSB Testing setup. Refer chapter Yardstick Installation for more on docker Install Yardstick using Docker (recommended)

12.4. System Topology:
+----------+              +----------+
|          |              |          |
|          | (0)----->(0) |          |
|    TG1   |              |    DUT   |
|          |              |          |
|          | (1)<-----(1) |          |
+----------+              +----------+
trafficgen_1                   vnf
12.5. Environment parameters and credentials
12.5.1. Config yardstick conf

If user did not run ‘yardstick env influxdb’ inside the container, which will generate correct yardstick.conf, then create the config file manually (run inside the container):

cp ./etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf vi /etc/yardstick/yardstick.conf

Add trex_path, trex_client_lib and bin_path in ‘nsb’ section.

[DEFAULT]
debug = True
dispatcher = file, influxdb

[dispatcher_influxdb]
timeout = 5
target = http://{YOUR_IP_HERE}:8086
db_name = yardstick
username = root
password = root

[nsb]
trex_path=/opt/nsb_bin/trex/scripts
bin_path=/opt/nsb_bin
trex_client_lib=/opt/nsb_bin/trex_client/stl
12.6. Run Yardstick - Network Service Testcases
12.6.1. NS testing - using yardstick CLI
docker exec -it yardstick /bin/bash
source /etc/yardstick/openstack.creds (only for heat TC if nsb_setup.sh was NOT used)
export EXTERNAL_NETWORK="<openstack public network>" (only for heat TC)
yardstick --debug task start yardstick/samples/vnf_samples/nsut/<vnf>/<test case>
12.7. Network Service Benchmarking - Bare-Metal
12.7.1. Bare-Metal Config pod.yaml describing Topology
12.7.1.1. Bare-Metal 2-Node setup:
+----------+              +----------+
|          |              |          |
|          | (0)----->(0) |          |
|    TG1   |              |    DUT   |
|          |              |          |
|          | (n)<-----(n) |          |
+----------+              +----------+
trafficgen_1                   vnf
12.7.1.2. Bare-Metal 3-Node setup - Correlated Traffic:
+----------+              +----------+            +------------+
|          |              |          |            |            |
|          |              |          |            |            |
|          | (0)----->(0) |          |            |    UDP     |
|    TG1   |              |    DUT   |            |   Replay   |
|          |              |          |            |            |
|          |              |          |(1)<---->(0)|            |
+----------+              +----------+            +------------+
trafficgen_1                   vnf                 trafficgen_2
12.7.2. Bare-Metal Config pod.yaml

Before executing Yardstick test cases, make sure that pod.yaml reflects the topology and update all the required fields.:

cp /etc/yardstick/nodes/pod.yaml.nsb.sample /etc/yardstick/nodes/pod.yaml
nodes:
-
    name: trafficgen_1
    role: TrafficGen
    ip: 1.1.1.1
    user: root
    password: r00t
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.0"
            driver:    i40e # default kernel driver
            dpdk_port_num: 0
            local_ip: "152.16.100.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:01"
        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.1"
            driver:    i40e # default kernel driver
            dpdk_port_num: 1
            local_ip: "152.16.40.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00.00:00:00:02"

-
    name: vnf
    role: vnf
    ip: 1.1.1.2
    user: root
    password: r00t
    host: 1.1.1.2 #BM - host == ip, virtualized env - Host - compute node
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.0"
            driver:    i40e # default kernel driver
            dpdk_port_num: 0
            local_ip: "152.16.100.19"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:03"

        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.1"
            driver:    i40e # default kernel driver
            dpdk_port_num: 1
            local_ip: "152.16.40.19"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:04"
    routing_table:
    - network: "152.16.100.20"
      netmask: "255.255.255.0"
      gateway: "152.16.100.20"
      if: "xe0"
    - network: "152.16.40.20"
      netmask: "255.255.255.0"
      gateway: "152.16.40.20"
      if: "xe1"
    nd_route_tbl:
    - network: "0064:ff9b:0:0:0:0:9810:6414"
      netmask: "112"
      gateway: "0064:ff9b:0:0:0:0:9810:6414"
      if: "xe0"
    - network: "0064:ff9b:0:0:0:0:9810:2814"
      netmask: "112"
      gateway: "0064:ff9b:0:0:0:0:9810:2814"
      if: "xe1"
12.8. Network Service Benchmarking - Standalone Virtualization
12.8.1. SR-IOV:
12.8.1.1. SR-IOV Pre-requisites
On Host:
  1. Create a bridge for VM to connect to external network
brctl addbr br-int
brctl addif br-int <interface_name>    #This interface is connected to internet
  1. Build guest image for VNF to run. Most of the sample test cases in Yardstick are using a guest image called yardstick-image which deviates from an Ubuntu Cloud Server image Yardstick has a tool for building this custom image with samplevnf. It is necessary to have sudo rights to use this tool.

    Also you may need to install several additional packages to use this tool, by following the commands below:

    sudo apt-get update && sudo apt-get install -y qemu-utils kpartx
    

    This image can be built using the following command in the directory where Yardstick is installed

    export YARD_IMG_ARCH='amd64'
    sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers
    

    Please use ansible script to generate a cloud image refer to Yardstick Installation

    for more details refer to chapter Yardstick Installation

    Note

    VM should be build with static IP and should be accessible from yardstick host.

12.8.1.2. SR-IOV Config pod.yaml describing Topology
12.8.1.3. SR-IOV 2-Node setup:
                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | VF NIC |  | VF NIC |
                             +--------+  +--------+
                                   ^          ^
                                   |          |
                                   |          |
+----------+               +-------------------------+
|          |               |       ^          ^      |
|          |               |       |          |      |
|          | (0)<----->(0) | ------           |      |
|    TG1   |               |           SUT    |      |
|          |               |                  |      |
|          | (n)<----->(n) |------------------       |
+----------+               +-------------------------+
trafficgen_1                          host
12.8.1.4. SR-IOV 3-Node setup - Correlated Traffic
                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | VF NIC |  | VF NIC |
                             +--------+  +--------+
                                   ^          ^
                                   |          |
                                   |          |
+----------+               +-------------------------+            +--------------+
|          |               |       ^          ^      |            |              |
|          |               |       |          |      |            |              |
|          | (0)<----->(0) | ------           |      |            |     TG2      |
|    TG1   |               |           SUT    |      |            | (UDP Replay) |
|          |               |                  |      |            |              |
|          | (n)<----->(n) |                  ------ | (n)<-->(n) |              |
+----------+               +-------------------------+            +--------------+
trafficgen_1                          host                       trafficgen_2

Before executing Yardstick test cases, make sure that pod.yaml reflects the topology and update all the required fields.

cp <yardstick>/etc/yardstick/nodes/standalone/trex_bm.yaml.sample /etc/yardstick/nodes/standalone/pod_trex.yaml
cp <yardstick>/etc/yardstick/nodes/standalone/host_sriov.yaml /etc/yardstick/nodes/standalone/host_sriov.yaml

Note

Update all the required fields like ip, user, password, pcis, etc...

12.8.1.5. SR-IOV Config pod_trex.yaml
nodes:
-
    name: trafficgen_1
    role: TrafficGen
    ip: 1.1.1.1
    user: root
    password: r00t
    key_filename: /root/.ssh/id_rsa
    interfaces:
        xe0:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.0"
            driver:    i40e # default kernel driver
            dpdk_port_num: 0
            local_ip: "152.16.100.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00:00:00:00:01"
        xe1:  # logical name from topology.yaml and vnfd.yaml
            vpci:      "0000:07:00.1"
            driver:    i40e # default kernel driver
            dpdk_port_num: 1
            local_ip: "152.16.40.20"
            netmask:   "255.255.255.0"
            local_mac: "00:00.00:00:00:02"
12.8.1.6. SR-IOV Config host_sriov.yaml
nodes:
-
   name: sriov
   role: Sriov
   ip: 192.168.100.101
   user: ""
   password: ""

SR-IOV testcase update: <yardstick>/samples/vnf_samples/nsut/vfw/tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml

12.8.1.6.1. Update “contexts” section
contexts:
 - name: yardstick
   type: Node
   file: /etc/yardstick/nodes/standalone/pod_trex.yaml
 - type: StandaloneSriov
   file: /etc/yardstick/nodes/standalone/host_sriov.yaml
   name: yardstick
   vm_deploy: True
   flavor:
     images: "/var/lib/libvirt/images/ubuntu.qcow2"
     ram: 4096
     extra_specs:
       hw:cpu_sockets: 1
       hw:cpu_cores: 6
       hw:cpu_threads: 2
     user: "" # update VM username
     password: "" # update password
   servers:
     vnf:
       network_ports:
         mgmt:
           cidr: '1.1.1.61/24'  # Update VM IP address, if static, <ip>/<mask> or if dynamic, <start of ip>/<mask>
         xe0:
           - uplink_0
         xe1:
           - downlink_0
   networks:
     uplink_0:
       phy_port: "0000:05:00.0"
       vpci: "0000:00:07.0"
       cidr: '152.16.100.10/24'
       gateway_ip: '152.16.100.20'
     downlink_0:
       phy_port: "0000:05:00.1"
       vpci: "0000:00:08.0"
       cidr: '152.16.40.10/24'
       gateway_ip: '152.16.100.20'
12.8.2. OVS-DPDK:
12.8.2.1. OVS-DPDK Pre-requisites
On Host:
  1. Create a bridge for VM to connect to external network
brctl addbr br-int
brctl addif br-int <interface_name>    #This interface is connected to internet
  1. Build guest image for VNF to run. Most of the sample test cases in Yardstick are using a guest image called yardstick-image which deviates from an Ubuntu Cloud Server image Yardstick has a tool for building this custom image with samplevnf. It is necessary to have sudo rights to use this tool.

    Also you may need to install several additional packages to use this tool, by following the commands below:

    sudo apt-get update && sudo apt-get install -y qemu-utils kpartx
    

    This image can be built using the following command in the directory where Yardstick is installed:

    export YARD_IMG_ARCH='amd64'
    sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers
    sudo tools/yardstick-img-dpdk-modify tools/ubuntu-server-cloudimg-samplevnf-modify.sh
    

    for more details refer to chapter Yardstick Installation

    Note

    VM should be build with static IP and should be accessible from yardstick host.

  2. OVS & DPDK version.
    • OVS 2.7 and DPDK 16.11.1 above version is supported
  3. Setup OVS/DPDK on host.

    Please refer to below link on how to setup OVS-DPDK

12.8.2.2. OVS-DPDK Config pod.yaml describing Topology
12.8.2.3. OVS-DPDK 2-Node setup:
                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | virtio |  | virtio |
                             +--------+  +--------+
                                  ^          ^
                                  |          |
                                  |          |
                             +--------+  +--------+
                             | vHOST0 |  | vHOST1 |
+----------+               +-------------------------+
|          |               |       ^          ^      |
|          |               |       |          |      |
|          | (0)<----->(0) | ------           |      |
|    TG1   |               |          SUT     |      |
|          |               |       (ovs-dpdk) |      |
|          | (n)<----->(n) |------------------       |
+----------+               +-------------------------+
trafficgen_1                          host
12.8.2.4. OVS-DPDK 3-Node setup - Correlated Traffic
                             +--------------------+
                             |                    |
                             |                    |
                             |        DUT         |
                             |       (VNF)        |
                             |                    |
                             +--------------------+
                             | virtio |  | virtio |
                             +--------+  +--------+
                                  ^          ^
                                  |          |
                                  |          |
                             +--------+  +--------+
                             | vHOST0 |  | vHOST1 |
+----------+               +-------------------------+          +------------+
|          |               |       ^          ^      |          |            |
|          |               |       |          |      |          |            |
|          | (0)<----->(0) | ------           |      |          |    TG2     |
|    TG1   |               |          SUT     |      |          |(UDP Replay)|
|          |               |      (ovs-dpdk)  |      |          |            |
|          | (n)<----->(n) |                  ------ |(n)<-->(n)|            |
+----------+               +-------------------------+          +------------+
trafficgen_1                          host                       trafficgen_2

Before executing Yardstick test cases, make sure that pod.yaml reflects the topology and update all the required fields.

cp <yardstick>/etc/yardstick/nodes/standalone/trex_bm.yaml.sample /etc/yardstick/nodes/standalone/pod_trex.yaml
cp <yardstick>/etc/yardstick/nodes/standalone/host_ovs.yaml /etc/yardstick/nodes/standalone/host_ovs.yaml

Note

Update all the required fields like ip, user, password, pcis, etc...

12.8.2.5. OVS-DPDK Config pod_trex.yaml
nodes:
-
  name: trafficgen_1
  role: TrafficGen
  ip: 1.1.1.1
  user: root
  password: r00t
  interfaces:
      xe0:  # logical name from topology.yaml and vnfd.yaml
          vpci:      "0000:07:00.0"
          driver:    i40e # default kernel driver
          dpdk_port_num: 0
          local_ip: "152.16.100.20"
          netmask:   "255.255.255.0"
          local_mac: "00:00:00:00:00:01"
      xe1:  # logical name from topology.yaml and vnfd.yaml
          vpci:      "0000:07:00.1"
          driver:    i40e # default kernel driver
          dpdk_port_num: 1
          local_ip: "152.16.40.20"
          netmask:   "255.255.255.0"
          local_mac: "00:00.00:00:00:02"
12.8.2.6. OVS-DPDK Config host_ovs.yaml
nodes:
-
   name: ovs_dpdk
   role: OvsDpdk
   ip: 192.168.100.101
   user: ""
   password: ""

ovs_dpdk testcase update: <yardstick>/samples/vnf_samples/nsut/vfw/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml

12.8.2.6.1. Update “contexts” section
contexts:
 - name: yardstick
   type: Node
   file: /etc/yardstick/nodes/standalone/pod_trex.yaml
 - type: StandaloneOvsDpdk
   name: yardstick
   file: /etc/yardstick/nodes/standalone/pod_ovs.yaml
   vm_deploy: True
   ovs_properties:
     version:
       ovs: 2.7.0
       dpdk: 16.11.1
     pmd_threads: 2
     ram:
       socket_0: 2048
       socket_1: 2048
     queues: 4
     vpath: "/usr/local"

   flavor:
     images: "/var/lib/libvirt/images/ubuntu.qcow2"
     ram: 4096
     extra_specs:
       hw:cpu_sockets: 1
       hw:cpu_cores: 6
       hw:cpu_threads: 2
     user: "" # update VM username
     password: "" # update password
   servers:
     vnf:
       network_ports:
         mgmt:
           cidr: '1.1.1.61/24'  # Update VM IP address, if static, <ip>/<mask> or if dynamic, <start of ip>/<mask>
         xe0:
           - uplink_0
         xe1:
           - downlink_0
   networks:
     uplink_0:
       phy_port: "0000:05:00.0"
       vpci: "0000:00:07.0"
       cidr: '152.16.100.10/24'
       gateway_ip: '152.16.100.20'
     downlink_0:
       phy_port: "0000:05:00.1"
       vpci: "0000:00:08.0"
       cidr: '152.16.40.10/24'
       gateway_ip: '152.16.100.20'
12.9. Enabling other Traffic generator
12.9.1. IxLoad:
  1. Software needed: IxLoadAPI <IxLoadTclApi verson>Linux64.bin.tgz and <IxOS version>Linux64.bin.tar.gz (Download from ixia support site)

    Install - <IxLoadTclApi verson>Linux64.bin.tgz & <IxOS version>Linux64.bin.tar.gz

    If the installation was not done inside the container, after installing the IXIA client, check /opt/ixia/ixload/<ver>/bin/ixloadpython and make sure you can run this cmd inside the yardstick container. Usually user is required to copy or link /opt/ixia/python/<ver>/bin/ixiapython to /usr/bin/ixiapython<ver> inside the container.

  2. Update pod_ixia.yaml file with ixia details.

cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml

Config pod_ixia.yaml

nodes:
    -
      name: trafficgen_1
      role: IxNet
      ip: 1.2.1.1 #ixia machine ip
      user: user
      password: r00t
      key_filename: /root/.ssh/id_rsa
      tg_config:
          ixchassis: "1.2.1.7" #ixia chassis ip
          tcl_port: "8009" # tcl server port
          lib_path: "/opt/ixia/ixos-api/8.01.0.2/lib/ixTcl1.0"
          root_dir: "/opt/ixia/ixos-api/8.01.0.2/"
          py_bin_path: "/opt/ixia/ixload/8.01.106.3/bin/"
          py_lib_path: "/opt/ixia/ixnetwork/8.01.1029.14/lib/PythonApi"
          dut_result_dir: "/mnt/ixia"
          version: 8.1
      interfaces:
          xe0:  # logical name from topology.yaml and vnfd.yaml
              vpci: "2:5" # Card:port
              driver:    "none"
              dpdk_port_num: 0
              local_ip: "152.16.100.20"
              netmask:   "255.255.0.0"
              local_mac: "00:98:10:64:14:00"
          xe1:  # logical name from topology.yaml and vnfd.yaml
              vpci: "2:6" # [(Card, port)]
              driver:    "none"
              dpdk_port_num: 1
              local_ip: "152.40.40.20"
              netmask:   "255.255.0.0"
              local_mac: "00:98:28:28:14:00"

for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration

  1. Start IxOS TCL Server (Install ‘Ixia IxExplorer IxOS <version>’) You will also need to configure the IxLoad machine to start the IXIA IxosTclServer. This can be started like so:

    • Connect to the IxLoad machine using RDP
    • Go to:
    Start->Programs->Ixia->IxOS->IxOS 8.01-GA-Patch1->Ixia Tcl Server IxOS 8.01-GA-Patch1

    or

    "C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe"

  2. Create a folder “Results” in c:and share the folder on the network.

  3. execute testcase in samplevnf folder. eg <repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_http_ixload_1b_Requests-65000_Concurrency.yaml

12.9.2. IxNetwork:
  1. Software needed: IxNetworkAPI<ixnetwork verson>Linux64.bin.tgz (Download from ixia support site)

    Install - IxNetworkAPI<ixnetwork verson>Linux64.bin.tgz

  2. Update pod_ixia.yaml file with ixia details.

cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml

Config pod_ixia.yaml

nodes:
    -
      name: trafficgen_1
      role: IxNet
      ip: 1.2.1.1 #ixia machine ip
      user: user
      password: r00t
      key_filename: /root/.ssh/id_rsa
      tg_config:
          ixchassis: "1.2.1.7" #ixia chassis ip
          tcl_port: "8009" # tcl server port
          lib_path: "/opt/ixia/ixos-api/8.01.0.2/lib/ixTcl1.0"
          root_dir: "/opt/ixia/ixos-api/8.01.0.2/"
          py_bin_path: "/opt/ixia/ixload/8.01.106.3/bin/"
          py_lib_path: "/opt/ixia/ixnetwork/8.01.1029.14/lib/PythonApi"
          dut_result_dir: "/mnt/ixia"
          version: 8.1
      interfaces:
          xe0:  # logical name from topology.yaml and vnfd.yaml
              vpci: "2:5" # Card:port
              driver:    "none"
              dpdk_port_num: 0
              local_ip: "152.16.100.20"
              netmask:   "255.255.0.0"
              local_mac: "00:98:10:64:14:00"
          xe1:  # logical name from topology.yaml and vnfd.yaml
              vpci: "2:6" # [(Card, port)]
              driver:    "none"
              dpdk_port_num: 1
              local_ip: "152.40.40.20"
              netmask:   "255.255.0.0"
              local_mac: "00:98:28:28:14:00"

for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration

  1. Start IxNetwork TCL Server You will also need to configure the IxNetwork machine to start the IXIA IxNetworkTclServer. This can be started like so:

    • Connect to the IxNetwork machine using RDP
    • Go to: Start->Programs->Ixia->IxNetwork->IxNetwork 7.21.893.14 GA->IxNetworkTclServer (or IxNetworkApiServer)
  2. execute testcase in samplevnf folder. eg <repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml

13. Yardstick - NSB Testing - Operation
13.1. Abstract

NSB test configuration and OpenStack setup requirements

13.2. OpenStack Network Configuration

NSB requires certain OpenStack deployment configurations. For optimal VNF characterization using external traffic generators NSB requires provider/external networks.

13.2.1. Provider networks

The VNFs require a clear L2 connect to the external network in order to generate realistic traffic from multiple address ranges and port

In order to prevent Neutron from filtering traffic we have to disable Neutron Port Security. We also disable DHCP on the data ports because we are binding the ports to DPDK and do not need DHCP addresses. We also disable gateways because multiple default gateways can prevent SSH access to the VNF from the floating IP. We only want a gateway on the mgmt network

uplink_0:
  cidr: '10.1.0.0/24'
  gateway_ip: 'null'
  port_security_enabled: False
  enable_dhcp: 'false'
13.2.2. Heat Topologies

By default Heat will attach every node to every Neutron network that is created. For scale-out tests we do not want to attach every node to every network.

For each node you can specify which ports are on which network using the network_ports dictionary.

In this example we have TRex xe0 <-> xe0 VNF xe1 <-> xe0 UDP_Replay

vnf_0:
  floating_ip: true
  placement: "pgrp1"
  network_ports:
    mgmt:
      - mgmt
    uplink_0:
      - xe0
    downlink_0:
      - xe1
tg_0:
  floating_ip: true
  placement: "pgrp1"
  network_ports:
    mgmt:
      - mgmt
    uplink_0:
      - xe0
    # Trex always needs two ports
    uplink_1:
      - xe1
tg_1:
  floating_ip: true
  placement: "pgrp1"
  network_ports:
    mgmt:
     - mgmt
    downlink_0:
     - xe0
13.3. Collectd KPIs

NSB can collect KPIs from collected. We have support for various plugins enabled by the Barometer project.

The default yardstick-samplevnf has collectd installed. This allows for collecting KPIs from the VNF.

Collecting KPIs from the NFVi is more complicated and requires manual setup. We assume that collectd is not installed on the compute nodes.

To collectd KPIs from the NFVi compute nodes:

  • install_collectd on the compute nodes
  • create pod.yaml for the compute nodes
  • enable specific plugins depending on the vswitch and DPDK

example pod.yaml section for Compute node running collectd.

-
  name: "compute-1"
  role: Compute
  ip: "10.1.2.3"
  user: "root"
  ssh_port: "22"
  password: ""
  collectd:
    interval: 5
    plugins:
      # for libvirtd stats
      virt: {}
      intel_pmu: {}
      ovs_stats:
        # path to OVS socket
        ovs_socket_path: /var/run/openvswitch/db.sock
      intel_rdt: {}
13.4. Scale-Up

VNFs performance data with scale-up

  • Helps to figure out optimal number of cores specification in the Virtual Machine template creation or VNF
  • Helps in comparison between different VNF vendor offerings
  • Better the scale-up index, indicates the performance scalability of a particular solution
13.4.1. Heat

For VNF scale-up tests we increase the number for VNF worker threads. In the case of VNFs we also need to increase the number of VCPUs and memory allocated to the VNF.

An example scale-up Heat testcase is:

<repo>/samples/vnf_samples/nsut/acl/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml

This testcase template requires specifying the number of VCPUs and Memory. We set the VCPUs and memory using the –task-args options

yardstick --debug task start --task-args='{"mem": 20480, "vcpus": 10}'   samples/vnf_samples/nsut/acl/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml
13.4.2. Baremetal
  1. Follow above traffic generator section to setup.
  2. edit num of threads in <repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml

e.g, 6 Threads for given VNF

schema: yardstick:task:0.1
scenarios:
{% for worker_thread in [1, 2 ,3 , 4, 5, 6] %}
- type: NSPerf
  traffic_profile: ../../traffic_profiles/ipv4_throughput.yaml
  topology: vfw-tg-topology.yaml
  nodes:
    tg__0: trafficgen_1.yardstick
    vnf__0: vnf.yardstick
  options:
    framesize:
      uplink: {64B: 100}
      downlink: {64B: 100}
    flow:
      src_ip: [{'tg__0': 'xe0'}]
      dst_ip: [{'tg__0': 'xe1'}]
      count: 1
    traffic_type: 4
    rfc2544:
      allowed_drop_rate: 0.0001 - 0.0001
    vnf__0:
      rules: acl_1rule.yaml
      vnf_config: {lb_config: 'HW', lb_count: 1, worker_config: '1C/1T', worker_threads: {{worker_thread}}}
      nfvi_enable: True
  runner:
    type: Iteration
    iterations: 10
    interval: 35
{% endfor %}
context:
  type: Node
  name: yardstick
  nfvi_type: baremetal
  file: /etc/yardstick/nodes/pod.yaml
13.5. Scale-Out

VNFs performance data with scale-out

  • Helps in capacity planning to meet the given network node requirements
  • Helps in comparison between different VNF vendor offerings
  • Better the scale-out index, provides the flexibility in meeting future capacity requirements
13.5.1. Standalone

Scale-out not supported on Baremetal.

  1. Follow above traffic generator section to setup.
  2. Generate testcase for standalone virtualization using ansible scripts
cd <repo>/ansible
trex: standalone_ovs_scale_out_trex_test.yaml or standalone_sriov_scale_out_trex_test.yaml
ixia: standalone_ovs_scale_out_ixia_test.yaml or standalone_sriov_scale_out_ixia_test.yaml
ixia_correlated: standalone_ovs_scale_out_ixia_correlated_test.yaml or standalone_sriov_scale_out_ixia_correlated_test.yaml

update the ovs_dpdk or sriov above Ansible scripts reflect the setup

  1. run the test
<repo>/samples/vnf_samples/nsut/tc_sriov_vfw_udp_ixia_correlated_scale_out-1.yaml
<repo>/samples/vnf_samples/nsut/tc_sriov_vfw_udp_ixia_correlated_scale_out-2.yaml
13.5.2. Heat

There are sample scale-out all-VM Heat tests. These tests only use VMs and don’t use external traffic.

The tests use UDP_Replay and correlated traffic.

<repo>/samples/vnf_samples/nsut/cgnapt/tc_heat_rfc2544_ipv4_1flow_64B_trex_correlated_scale_4.yaml

To run the test you need to increase OpenStack CPU, Memory and Port quotas.

13.6. Traffic Generator tuning

The TRex traffic generator can be setup to use multiple threads per core, this is for multiqueue testing.

TRex does not automatically enable multiple threads because we currently cannot detect the number of queues on a device.

To enable multiple queue set the queues_per_port value in the TG VNF options section.

scenarios:
  - type: NSPerf
    nodes:
      tg__0: tg_0.yardstick

    options:
      tg_0:
        queues_per_port: 2
14. Yardstick Test Cases
14.1. Abstract

This chapter lists available Yardstick test cases. Yardstick test cases are divided in two main categories:

  • Generic NFVI Test Cases - Test Cases developed to realize the methodology

described in Methodology

  • OPNFV Feature Test Cases - Test Cases developed to verify one or more

aspect of a feature delivered by an OPNFV Project, including the test cases developed for the VTC.

14.2. Generic NFVI Test Case Descriptions
14.2.1. Yardstick Test Case Description TC001
Network Performance
test case id OPNFV_YARDSTICK_TC001_NETWORK PERFORMANCE
metric Number of flows and throughput
test purpose

The purpose of TC001 is to evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of flows matter for the throughput between hosts on different compute blades. Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

pktgen

Linux packet generator is a tool to generate packets at very high speed in the kernel. pktgen is mainly used to drive and LAN equipment test network. pktgen supports multi threading. To generate random MAC address, IP address, port number UDP packets, pktgen uses multiple CPU processors in the different PCI bus (PCI, PCIe bus) with Gigabit Ethernet tested (pktgen performance depends on the CPU processing speed, memory delay, PCI bus speed hardware parameters), Transmit data rate can be even larger than 10GBit/s. Visible can satisfy most card test requirements.

(Pktgen is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Docker image. As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

test description This test case uses Pktgen to generate packet flow between two hosts for simulating network workloads on the SUT.
traffic profile An IP table is setup on server to monitor for received packets.
configuration

file: opnfv_yardstick_tc001.yaml

Packet size is set to 60 bytes. Number of ports: 10, 50, 100, 500 and 1000, where each runs for 20 seconds. The whole sequence is run twice The client and server are distributed on different hardware.

For SLA max_ppm is set to 1000. The amount of configured ports map to between 110 up to 1001000 flows, respectively.

applicability

Test can be configured with different:

  • packet sizes;
  • amount of flows;
  • test duration.

Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to loose, not received.

usability This test case is used for generating high network throughput to simulate certain workloads on the SUT. Hence it should work with other test cases.
references

pktgen

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with pktgen included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 Two host VMs are booted, as server and client.
step 2 Yardstick is connected with the server VM by using ssh. ‘pktgen_benchmark’ bash script is copyied from Jump Host to the server VM via the ssh tunnel.
step 3 An IP table is setup on server to monitor for received packets.
step 4

pktgen is invoked to generate packet flow between two server and client for simulating network workloads on the SUT. Results are processed and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 5 Two host VMs are deleted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.2. Yardstick Test Case Description TC002
Network Latency
test case id OPNFV_YARDSTICK_TC002_NETWORK LATENCY
metric RTT (Round Trip Time)
test purpose

The purpose of TC002 is to do a basic verification that network latency is within acceptable boundaries when packets travel between hosts located on same or different compute blades.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

ping

Ping is a computer network administration software utility used to test the reachability of a host on an Internet Protocol (IP) network. It measures the round-trip time for packet sent from the originating host to a destination computer that are echoed back to the source.

Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Docker image. (For example also a Cirros image can be downloaded from cirros-image, it includes ping)

test topology

Ping packets (ICMP protocol’s mandatory ECHO_REQUEST datagram) are sent from host VM to target VM(s) to elicit ICMP ECHO_RESPONSE.

For one host VM there can be multiple target VMs. Host VM and target VM(s) can be on same or different compute blades.

configuration

file: opnfv_yardstick_tc002.yaml

Packet size 100 bytes. Test duration 60 seconds. One ping each 10 seconds. Test is iterated two times. SLA RTT is set to maximum 10 ms.

applicability

This test case can be configured with different:

  • packet sizes;
  • burst sizes;
  • ping intervals;
  • test durations;
  • test iterations.

Default values exist.

SLA is optional. The SLA in this test case serves as an example. Considerably lower RTT is expected, and also normal to achieve in balanced L2 environments. However, to cover most configurations, both bare metal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many real time applications start to suffer badly if the RTT time is higher than this. Some may suffer bad also close to this RTT, while others may not suffer at all. It is a compromise that may have to be tuned for different configuration purposes.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

Ping

ETSI-NFV-TST001

pre-test conditions

The test case image (cirros-image) needs to be installed into Glance with ping included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 Two host VMs are booted, as server and client.
step 2 Yardstick is connected with the server VM by using ssh. ‘ping_benchmark’ bash script is copied from Jump Host to the server VM via the ssh tunnel.
step 3

Ping is invoked. Ping packets are sent from server VM to client VM. RTT results are calculated and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 Two host VMs are deleted.
test verdict Test should not PASS if any RTT is above the optional SLA value, or if there is a test case execution problem.
14.2.3. Yardstick Test Case Description TC004
Cache Utilization
test case id OPNFV_YARDSTICK_TC004_CACHE Utilization
metric cache hit, cache miss, hit/miss ratio, buffer size and page cache size
test purpose

The purpose of TC004 is to evaluate the IaaS compute capability with regards to cache utilization.This test case should be run in parallel with other Yardstick test cases and not run as a stand-alone test case.

This test case measures cache usage statistics, including cache hit, cache miss, hit ratio, buffer cache size and page cache size, with some wokloads runing on the infrastructure. Both average and maximun values are collected.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

cachestat

cachestat is a tool using Linux ftrace capabilities for showing Linux page cache hit/miss statistics.

(cachestat is not always part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with cachestat included.)

test description cachestat test is invoked in a host VM on a compute blade, cachestat test requires some other test cases running in the host to stimulate workload.
configuration

File: cachestat.yaml (in the ‘samples’ directory)

Interval is set 1. Test repeat, pausing every 1 seconds in-between. Test durarion is set to 60 seconds.

SLA is not available in this test case.

applicability

Test can be configured with different:

  • interval;
  • runner Duration.

Default values exist.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

cachestat

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with cachestat included in the image.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 A host VM with cachestat installed is booted.
step 2 Yardstick is connected with the host VM by using ssh. ‘cache_stat’ bash script is copyied from Jump Host to the server VM via the ssh tunnel.
step 3

‘cache_stat’ script is invoked. Raw cache usage statistics are collected and filtrated. Average and maximum values are calculated and recorded. Logs are produced and stored.

Result: Logs are stored.

step 4 The host VM is deleted.
test verdict None. Cache utilization results are collected and stored.
14.2.4. Yardstick Test Case Description TC005
Storage Performance
test case id OPNFV_YARDSTICK_TC005_STORAGE PERFORMANCE
metric IOPS (Average IOs performed per second), Throughput (Average disk read/write bandwidth rate), Latency (Average disk read/write latency)
test purpose

The purpose of TC005 is to evaluate the IaaS storage performance with regards to IOPS, throughput and latency.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

fio

fio is an I/O tool meant to be used both for benchmark and stress/hardware verification. It has support for 19 different types of I/O engines (sync, mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio, and more), I/O priorities (for newer Linux kernels), rate I/O, forked or threaded jobs, and much more.

(fio is not always part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with fio included.)

test description fio test is invoked in a host VM on a compute blade, a job file as well as parameters are passed to fio and fio will start doing what the job file tells it to do.
configuration

file: opnfv_yardstick_tc005.yaml

IO types is set to read, write, randwrite, randread, rw. IO block size is set to 4KB, 64KB, 1024KB. fio is run for each IO type and IO block size scheme, each iteration runs for 30 seconds (10 for ramp time, 20 for runtime).

For SLA, minimum read/write iops is set to 100, minimum read/write throughput is set to 400 KB/s, and maximum read/write latency is set to 20000 usec.

applicability

This test case can be configured with different:

  • IO types;
  • IO block size;
  • IO depth;
  • ramp time;
  • test duration.

Default values exist.

SLA is optional. The SLA in this test case serves as an example. Considerably higher throughput and lower latency are expected. However, to cover most configurations, both baremetal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many heavy IO applications start to suffer badly if the read/write bandwidths are lower than this.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

fio

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with fio included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 A host VM with fio installed is booted.
step 2 Yardstick is connected with the host VM by using ssh. ‘fio_benchmark’ bash script is copyied from Jump Host to the host VM via the ssh tunnel.
step 3

‘fio_benchmark’ script is invoked. Simulated IO operations are started. IOPS, disk read/write bandwidth and latency are recorded and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 The host VM is deleted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.5. Yardstick Test Case Description TC008
Packet Loss Extended Test
test case id OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test
metric Number of flows, packet size and throughput
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of packet sizes and flows matter for the throughput between VMs on different compute blades. Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used. The purpose is also to be able to spot trends. Test results, graphs ans similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc008.yaml

Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes.

Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of configured ports map from 2 up to 1001000 flows, respectively. Each packet_size/port_amount combination is run ten times, for 20 seconds each. Then the next packet_size/port_amount combination is run, and so on.

The client and server are distributed on different HW.

For SLA max_ppm is set to 1000.

test tool

pktgen

(Pktgen is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Docker image. As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

references

pktgen

ETSI-NFV-TST001

applicability

Test can be configured with different packet sizes, amount of flows and test duration. Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to loose, not received.

pre-test conditions

The test case image needs to be installed into Glance with pktgen included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The hosts are installed, as server and client. pktgen is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.6. Yardstick Test Case Description TC009
Packet Loss
test case id OPNFV_YARDSTICK_TC009_NW PERF, Packet loss
metric Number of flows, packets lost and throughput
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of flows matter for the throughput between VMs on different compute blades. Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used. The purpose is also to be able to spot trends. Test results, graphs ans similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc009.yaml

Packet size: 64 bytes

Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of configured ports map from 2 up to 1001000 flows, respectively. Each port amount is run ten times, for 20 seconds each. Then the next port_amount is run, and so on.

The client and server are distributed on different HW.

For SLA max_ppm is set to 1000.

test tool

pktgen

(Pktgen is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Docker image. As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

references

pktgen

ETSI-NFV-TST001

applicability

Test can be configured with different packet sizes, amount of flows and test duration. Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to loose, not received.

pre-test conditions

The test case image needs to be installed into Glance with pktgen included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The hosts are installed, as server and client. pktgen is invoked and logs are produced and stored.

Result: logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.7. Yardstick Test Case Description TC010
Memory Latency
test case id OPNFV_YARDSTICK_TC010_MEMORY LATENCY
metric Memory read latency (nanoseconds)
test purpose

The purpose of TC010 is to evaluate the IaaS compute performance with regards to memory read latency. It measures the memory read latency for varying memory sizes and strides. Whole memory hierarchy is measured.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

Lmbench

Lmbench is a suite of operating system microbenchmarks. This test uses lat_mem_rd tool from that suite including:

  • Context switching
  • Networking: connection establishment, pipe, TCP, UDP, and RPC hot potato
  • File system creates and deletes
  • Process creation
  • Signal handling
  • System call overhead
  • Memory read latency

(LMbench is not always part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with LMbench included.)

test description

LMbench lat_mem_rd benchmark measures memory read latency for varying memory sizes and strides.

The benchmark runs as two nested loops. The outer loop is the stride size. The inner loop is the array size. For each array size, the benchmark creates a ring of pointers that point backward one stride.Traversing the array is done by:

p = (char **)*p;

in a for loop (the over head of the for loop is not significant; the loop is an unrolled loop 100 loads long). The size of the array varies from 512 bytes to (typically) eight megabytes. For the small sizes, the cache will have an effect, and the loads will be much faster. This becomes much more apparent when the data is plotted.

Only data accesses are measured; the instruction cache is not measured.

The results are reported in nanoseconds per load and have been verified accurate to within a few nanoseconds on an SGI Indy.

configuration

File: opnfv_yardstick_tc010.yaml

  • SLA (max_latency): 30 nanoseconds
  • Stride - 128 bytes
  • Stop size - 64 megabytes
  • Iterations: 10 - test is run 10 times iteratively.
  • Interval: 1 - there is 1 second delay between each iteration.

SLA is optional. The SLA in this test case serves as an example. Considerably lower read latency is expected. However, to cover most configurations, both baremetal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many heavy IO applications start to suffer badly if the read latency is higher than this.

applicability

Test can be configured with different:

  • strides;
  • stop_size;
  • iterations and intervals.

Default values exist.

SLA (optional) : max_latency: The maximum memory latency that is accepted.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

LMbench lat_mem_rd

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with Lmbench included in the image.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The host is installed as client. LMbench’s lat_mem_rd tool is invoked and logs are produced and stored.

Result: logs are stored.

step 1 A host VM with LMbench installed is booted.
step 2 Yardstick is connected with the host VM by using ssh. ‘lmbench_latency_benchmark’ bash script is copyied from Jump Host to the host VM via the ssh tunnel.
step 3

‘lmbench_latency_benchmark’ script is invoked. LMbench’s lat_mem_rd benchmark starts to measures memory read latency for varying memory sizes and strides. Memory read latency are recorded and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 The host VM is deleted.
test verdict Test fails if the measured memory latency is above the SLA value or if there is a test case execution problem.
14.2.8. Yardstick Test Case Description TC011
Packet delay variation between VMs
test case id OPNFV_YARDSTICK_TC011_PACKET DELAY VARIATION BETWEEN VMs
metric jitter: packet delay variation (ms)
test purpose

The purpose of TC011 is to evaluate the IaaS network performance with regards to network jitter (packet delay variation). It measures the packet delay variation sending the packets from one VM to the other.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

iperf3

iPerf3 is a tool for active measurements of the maximum achievable bandwidth on IP networks. It supports tuning of various parameters related to timing, buffers and protocols. The UDP protocols can be used to measure jitter delay.

(iperf3 is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Docker image. As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

test description

iperf3 test is invoked between a host VM and a target VM.

Jitter calculations are continuously computed by the server, as specified by RTP in RFC 1889. The client records a 64 bit second/microsecond timestamp in the packet. The server computes the relative transit time as (server’s receive time - client’s send time). The client’s and server’s clocks do not need to be synchronized; any difference is subtracted outin the jitter calculation. Jitter is the smoothed mean of differences between consecutive transit times.

configuration

File: opnfv_yardstick_tc011.yaml

  • options: protocol: udp # The protocol used by iperf3 tools bandwidth: 20m # It will send the given number of packets

    without pausing

  • runner: duration: 30 # Total test duration 30 seconds.

  • SLA (optional): jitter: 10 (ms) # The maximum amount of jitter that is

    accepted.

applicability

Test can be configured with different:

  • bandwidth: Test case can be configured with different

    bandwidth.

  • duration: The test duration can be configured.

  • jitter: SLA is optional. The SLA in this test case

    serves as an example.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

iperf3

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with iperf3 included in the image.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 Two host VMs with iperf3 installed are booted, as server and client.
step 2 Yardstick is connected with the host VM by using ssh. A iperf3 server is started on the server VM via the ssh tunnel.
step 3

iperf3 benchmark is invoked. Jitter is calculated and check against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 The host VMs are deleted.
test verdict Test should not PASS if any jitter is above the optional SLA value, or if there is a test case execution problem.
14.2.9. Yardstick Test Case Description TC012
Memory Bandwidth
test case id OPNFV_YARDSTICK_TC012_MEMORY BANDWIDTH
metric Memory read/write bandwidth (MBps)
test purpose

The purpose of TC012 is to evaluate the IaaS compute performance with regards to memory throughput. It measures the rate at which data can be read from and written to the memory (this includes all levels of memory).

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

LMbench

LMbench is a suite of operating system microbenchmarks. This test uses bw_mem tool from that suite including:

  • Cached file read
  • Memory copy (bcopy)
  • Memory read
  • Memory write
  • Pipe
  • TCP

(LMbench is not always part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with LMbench included.)

test description LMbench bw_mem benchmark allocates twice the specified amount of memory, zeros it, and then times the copying of the first half to the second half. The benchmark is invoked in a host VM on a compute blade. Results are reported in megabytes moved per second.
configuration

File: opnfv_yardstick_tc012.yaml

  • SLA (optional): 15000 (MBps) min_bw: The minimum amount of memory bandwidth that is accepted.
  • Size: 10 240 kB - test allocates twice that size (20 480kB) zeros it and then measures the time it takes to copy from one side to another.
  • Benchmark: rdwr - measures the time to read data into memory and then write data to the same location.
  • Warmup: 0 - the number of iterations to perform before taking actual measurements.
  • Iterations: 10 - test is run 10 times iteratively.
  • Interval: 1 - there is 1 second delay between each iteration.

SLA is optional. The SLA in this test case serves as an example. Considerably higher bandwidth is expected. However, to cover most configurations, both baremetal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many heavy IO applications start to suffer badly if the read/write bandwidths are lower than this.

applicability

Test can be configured with different:

  • memory sizes;
  • memory operations (such as rd, wr, rdwr, cp, frd, fwr, fcp, bzero, bcopy);
  • number of warmup iterations;
  • iterations and intervals.

Default values exist.

SLA (optional) : min_bandwidth: The minimun memory bandwidth that is accepted.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

LMbench bw_mem

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with Lmbench included in the image.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 A host VM with LMbench installed is booted.
step 2 Yardstick is connected with the host VM by using ssh. “lmbench_bandwidth_benchmark” bash script is copied from Jump Host to the host VM via ssh tunnel.
step 3

‘lmbench_bandwidth_benchmark’ script is invoked. LMbench’s bw_mem benchmark starts to measures memory read/write bandwidth. Memory read/write bandwidth results are recorded and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 The host VM is deleted.
test verdict Test fails if the measured memory bandwidth is below the SLA value or if there is a test case execution problem.
14.2.10. Yardstick Test Case Description TC014
Processing speed
test case id OPNFV_YARDSTICK_TC014_PROCESSING SPEED
metric score of single cpu running, score of parallel running
test purpose

The purpose of TC014 is to evaluate the IaaS compute performance with regards to CPU processing speed. It measures score of single cpu running and parallel running.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

UnixBench

Unixbench is the most used CPU benchmarking software tool. It can measure the performance of bash scripts, CPUs in multithreading and single threading. It can also measure the performance for parallel taks. Also, specific disk IO for small and large files are performed. You can use it to measure either linux dedicated servers and linux vps servers, running CentOS, Debian, Ubuntu, Fedora and other distros.

(UnixBench is not always part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with UnixBench included.)

test description

The UnixBench runs system benchmarks in a host VM on a compute blade, getting information on the CPUs in the system. If the system has more than one CPU, the tests will be run twice – once with a single copy of each test running at once, and once with N copies, where N is the number of CPUs.

UnixBench will processs a set of results from a single test by averaging the individal pass results into a single final value.

configuration

file: opnfv_yardstick_tc014.yaml

run_mode: Run unixbench in quiet mode or verbose mode test_type: dhry2reg, whetstone and so on

For SLA with single_score and parallel_score, both can be set by user, default is NA.

applicability

Test can be configured with different:

  • test types;
  • dhry2reg;
  • whetstone.

Default values exist.

SLA (optional) : min_score: The minimun UnixBench score that is accepted.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

unixbench

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with unixbench included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 A host VM with UnixBench installed is booted.
step 2 Yardstick is connected with the host VM by using ssh. “unixbench_benchmark” bash script is copied from Jump Host to the host VM via ssh tunnel.
step 3

UnixBench is invoked. All the tests are executed using the “Run” script in the top-level of UnixBench directory. The “Run” script will run a standard “index” test, and save the report in the “results” directory. Then the report is processed by “unixbench_benchmark” and checked againsted the SLA.

Result: Logs are stored.

step 4 The host VM is deleted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.11. Yardstick Test Case Description TC024
CPU Load
test case id OPNFV_YARDSTICK_TC024_CPU Load
metric CPU load
test purpose To evaluate the CPU load performance of the IaaS. This test case should be run in parallel to other Yardstick test cases and not run as a stand-alone test case. Average, minimum and maximun values are obtained. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: cpuload.yaml (in the ‘samples’ directory)

  • interval: 1 - repeat, pausing every 1 seconds in-between.
  • count: 10 - display statistics 10 times, then exit.
test tool

mpstat

(mpstat is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Glance image. However, if mpstat is not present the TC instead uses /proc/stats as source to produce “mpstat” output.

references man-pages
applicability

Test can be configured with different:

  • interval;
  • count;
  • runner Iteration and intervals.

There are default values for each above-mentioned option. Run in background with other test cases.

pre-test conditions

The test case image needs to be installed into Glance with mpstat included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The host is installed. The related TC, or TCs, is invoked and mpstat logs are produced and stored.

Result: Stored logs

test verdict None. CPU load results are fetched and stored.
14.2.12. Yardstick Test Case Description TC037
Latency, CPU Load, Throughput, Packet Loss
test case id OPNFV_YARDSTICK_TC037_LATENCY,CPU LOAD,THROUGHPUT, PACKET LOSS
metric Number of flows, latency, throughput, packet loss CPU utilization percentage, CPU interrupt per second
test purpose

The purpose of TC037 is to evaluate the IaaS compute capacity and network performance with regards to CPU utilization, packet flows and network throughput, such as if and how different amounts of flows matter for the throughput between hosts on different compute blades, and the CPU load variation.

Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

Ping, Pktgen, mpstat

Ping is a computer network administration software utility used to test the reachability of a host on an Internet Protocol (IP) network. It measures the round-trip time for packet sent from the originating host to a destination computer that are echoed back to the source.

Linux packet generator is a tool to generate packets at very high speed in the kernel. pktgen is mainly used to drive and LAN equipment test network. pktgen supports multi threading. To generate random MAC address, IP address, port number UDP packets, pktgen uses multiple CPU processors in the different PCI bus (PCI, PCIe bus) with Gigabit Ethernet tested (pktgen performance depends on the CPU processing speed, memory delay, PCI bus speed hardware parameters), Transmit data rate can be even larger than 10GBit/s. Visible can satisfy most card test requirements.

The mpstat command writes to standard output activities for each available processor, processor 0 being the first one. Global average activities among all processors are also reported. The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed.

(Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Docker image. For example also a Cirros image can be downloaded from cirros-image, it includes ping.

Pktgen and mpstat are not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Docker image. As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen and mpstat included.)

test description This test case uses Pktgen to generate packet flow between two hosts for simulating network workloads on the SUT. Ping packets (ICMP protocol’s mandatory ECHO_REQUEST datagram) are sent from a host VM to the target VM(s) to elicit ICMP ECHO_RESPONSE, meanwhile CPU activities are monitored by mpstat.
configuration

file: opnfv_yardstick_tc037.yaml

Packet size is set to 64 bytes. Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. The amount configured ports map from 2 up to 1001000 flows, respectively. Each port amount is run two times, for 20 seconds each. Then the next port_amount is run, and so on. During the test CPU load on both client and server, and the network latency between the client and server are measured. The client and server are distributed on different hardware. mpstat monitoring interval is set to 1 second. ping packet size is set to 100 bytes. For SLA max_ppm is set to 1000.

applicability

Test can be configured with different:

  • pktgen packet sizes;
  • amount of flows;
  • test duration;
  • ping packet size;
  • mpstat monitor interval.

Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to loose, not received.

references

Ping

mpstat

pktgen

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with pktgen, mpstat included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 Two host VMs are booted, as server and client.
step 2 Yardstick is connected with the server VM by using ssh. ‘pktgen_benchmark’, “ping_benchmark” bash script are copyied from Jump Host to the server VM via the ssh tunnel.
step 3 An IP table is setup on server to monitor for received packets.
step 4

pktgen is invoked to generate packet flow between two server and client for simulating network workloads on the SUT. Ping is invoked. Ping packets are sent from server VM to client VM. mpstat is invoked, recording activities for each available processor. Results are processed and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 5 Two host VMs are deleted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.13. Yardstick Test Case Description TC038
Latency, CPU Load, Throughput, Packet Loss (Extended measurements)
test case id OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Loss
metric Number of flows, latency, throughput, CPU load, packet loss
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of flows matter for the throughput between hosts on different compute blades. Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used. The purpose is also to be able to spot trends. Test results, graphs ans similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc038.yaml

Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. The amount configured ports map from 2 up to 1001000 flows, respectively. Each port amount is run ten times, for 20 seconds each. Then the next port_amount is run, and so on. During the test CPU load on both client and server, and the network latency between the client and server are measured. The client and server are distributed on different HW. For SLA max_ppm is set to 1000.

test tool

pktgen

(Pktgen is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Glance image. As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

ping

Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Glance image. (For example also a cirros image can be downloaded, it includes ping)

mpstat

(Mpstat is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Glance image.

references

Ping and Mpstat man pages

pktgen

ETSI-NFV-TST001

applicability

Test can be configured with different packet sizes, amount of flows and test duration. Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to loose, not received.

pre-test conditions

The test case image needs to be installed into Glance with pktgen included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The hosts are installed, as server and client. pktgen is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.14. Yardstick Test Case Description TC0042
Network Performance
test case id OPNFV_YARDSTICK_TC042_DPDK pktgen latency measurements
metric L2 Network Latency
test purpose Measure L2 network latency when DPDK is enabled between hosts on different compute blades.
configuration

file: opnfv_yardstick_tc042.yaml

  • Packet size: 64 bytes
  • SLA(max_latency): 100usec
test tool

DPDK Pktgen-dpdk

(DPDK and Pktgen-dpdk are not part of a Linux distribution, hence they needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with DPDK and pktgen-dpdk included.)

references

DPDK

Pktgen-dpdk

ETSI-NFV-TST001

applicability Test can be configured with different packet sizes. Default values exist.
pre-test conditions

The test case image needs to be installed into Glance with DPDK and pktgen-dpdk included in it.

The NICs of compute nodes must support DPDK on POD.

And at least compute nodes setup hugepage.

If you want to achievement a hight performance result, it is recommend to use NUAM, CPU pin, OVS and so on.

test sequence description and expected result
step 1 The hosts are installed on different blades, as server and client. Both server and client have three interfaces. The first one is management such as ssh. The other two are used by DPDK.
step 2 Testpmd is invoked with configurations to forward packets from one DPDK port to the other on server.
step 3

Pktgen-dpdk is invoked with configurations as a traffic generator and logs are produced and stored on client.

Result: Logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.15. Yardstick Test Case Description TC043
Network Latency Between NFVI Nodes
test case id OPNFV_YARDSTICK_TC043_LATENCY_BETWEEN_NFVI_NODES
metric RTT (Round Trip Time)
test purpose

The purpose of TC043 is to do a basic verification that network latency is within acceptable boundaries when packets travel between different NFVI nodes.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

ping

Ping is a computer network administration software utility used to test the reachability of a host on an Internet Protocol (IP) network. It measures the round-trip time for packet sent from the originating host to a destination computer that are echoed back to the source.

test topology Ping packets (ICMP protocol’s mandatory ECHO_REQUEST datagram) are sent from host node to target node to elicit ICMP ECHO_RESPONSE.
configuration

file: opnfv_yardstick_tc043.yaml

Packet size 100 bytes. Total test duration 600 seconds. One ping each 10 seconds. SLA RTT is set to maximum 10 ms.

applicability

This test case can be configured with different:

  • packet sizes;
  • burst sizes;
  • ping intervals;
  • test durations;
  • test iterations.

Default values exist.

SLA is optional. The SLA in this test case serves as an example. Considerably lower RTT is expected, and also normal to achieve in balanced L2 environments. However, to cover most configurations, both bare metal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many real time applications start to suffer badly if the RTT time is higher than this. Some may suffer bad also close to this RTT, while others may not suffer at all. It is a compromise that may have to be tuned for different configuration purposes.

references

Ping

ETSI-NFV-TST001

pre_test conditions Each pod node must have ping included in it.
test sequence description and expected result
step 1 Yardstick is connected with the NFVI node by using ssh. ‘ping_benchmark’ bash script is copyied from Jump Host to the NFVI node via the ssh tunnel.
step 2

Ping is invoked. Ping packets are sent from server node to client node. RTT results are calculated and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

test verdict Test should not PASS if any RTT is above the optional SLA value, or if there is a test case execution problem.
14.2.16. Yardstick Test Case Description TC044
Memory Utilization
test case id OPNFV_YARDSTICK_TC044_Memory Utilization
metric Memory utilization
test purpose To evaluate the IaaS compute capability with regards to memory utilization.This test case should be run in parallel to other Yardstick test cases and not run as a stand-alone test case. Measure the memory usage statistics including used memory, free memory, buffer, cache and shared memory. Both average and maximun values are obtained. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

File: memload.yaml (in the ‘samples’ directory)

  • interval: 1 - repeat, pausing every 1 seconds in-between.
  • count: 10 - display statistics 10 times, then exit.
test tool

free

free provides information about unused and used memory and swap space on any computer running Linux or another Unix-like operating system. free is normally part of a Linux distribution, hence it doesn’t needs to be installed.

references

man-pages

ETSI-NFV-TST001

applicability

Test can be configured with different:

  • interval;
  • count;
  • runner Iteration and intervals.

There are default values for each above-mentioned option. Run in background with other test cases.

pre-test conditions

The test case image needs to be installed into Glance with free included in the image.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The host is installed as client. The related TC, or TCs, is invoked and free logs are produced and stored.

Result: logs are stored.

test verdict None. Memory utilization results are fetched and stored.
14.2.17. Yardstick Test Case Description TC055
Compute Capacity
test case id OPNFV_YARDSTICK_TC055_Compute Capacity
metric Number of cpus, number of cores, number of threads, available memory size and total cache size.
test purpose To evaluate the IaaS compute capacity with regards to hardware specification, including number of cpus, number of cores, number of threads, available memory size and total cache size. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc055.yaml

There is are no additional configurations to be set for this TC.

test tool

/proc/cpuinfo

this TC uses /proc/cpuinfo as source to produce compute capacity output.

references

/proc/cpuinfo_

ETSI-NFV-TST001

applicability None.
pre-test conditions No POD specific requirements have been identified.
test sequence description and expected result
step 1

The hosts are installed, TC is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict None. Hardware specification are fetched and stored.
14.2.18. Yardstick Test Case Description TC061
Network Utilization
test case id OPNFV_YARDSTICK_TC061_Network Utilization
metric Network utilization
test purpose To evaluate the IaaS network capability with regards to network utilization, including Total number of packets received per second, Total number of packets transmitted per second, Total number of kilobytes received per second, Total number of kilobytes transmitted per second, Number of compressed packets received per second (for cslip etc.), Number of compressed packets transmitted per second, Number of multicast packets received per second, Utilization percentage of the network interface. This test case should be run in parallel to other Yardstick test cases and not run as a stand-alone test case. Measure the network usage statistics from the network devices Average, minimum and maximun values are obtained. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

File: netutilization.yaml (in the ‘samples’ directory)

  • interval: 1 - repeat, pausing every 1 seconds in-between.
  • count: 1 - display statistics 1 times, then exit.
test tool

sar

The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. sar is normally part of a Linux distribution, hence it doesn’t needs to be installed.

references

man-pages

ETSI-NFV-TST001

applicability

Test can be configured with different:

  • interval;
  • count;
  • runner Iteration and intervals.

There are default values for each above-mentioned option. Run in background with other test cases.

pre-test conditions

The test case image needs to be installed into Glance with sar included in the image.

No POD specific requirements have been identified.

test sequence description and expected result.
step 1

The host is installed as client. The related TC, or TCs, is invoked and sar logs are produced and stored.

Result: logs are stored.

test verdict None. Network utilization results are fetched and stored.
14.2.19. Yardstick Test Case Description TC063
Storage Capacity
test case id OPNFV_YARDSTICK_TC063_Storage Capacity
metric Storage/disk size, block size Disk Utilization
test purpose This test case will check the parameters which could decide several models and each model has its specified task to measure. The test purposes are to measure disk size, block size and disk utilization. With the test results, we could evaluate the storage capacity of the host.
configuration
file: opnfv_yardstick_tc063.yaml
  • test_type: “disk_size”

  • runner:

    type: Iteration iterations: 1 - test is run 1 time iteratively.

test tool

fdisk A command-line utility that provides disk partitioning functions

iostat This is a computer system monitor tool used to collect and show operating system storage input and output statistics.

references

iostat fdisk

ETSI-NFV-TST001

applicability

Test can be configured with different:

  • test_type: “disk size”, “block size”, “disk utilization”

  • interval: 1 - how ofter to stat disk utilization

    type: int unit: seconds

  • count: 15 - how many times to stat disk utilization

    type: int unit: na

There are default values for each above-mentioned option. Run in background with other test cases.

pre-test conditions

The test case image needs to be installed into Glance

No POD specific requirements have been identified.

test sequence Output the specific storage capacity of disk information as the sequence into file.
step 1

The pod is available and the hosts are installed. Node5 is used and logs are produced and stored.

Result: Logs are stored.

test verdict None.
14.2.20. Yardstick Test Case Description TC069
Memory Bandwidth
test case id OPNFV_YARDSTICK_TC069_Memory Bandwidth
metric Megabyte per second (MBps)
test purpose To evaluate the IaaS compute performance with regards to memory bandwidth. Measure the maximum possible cache and memory performance while reading and writing certain blocks of data (starting from 1Kb and further in power of 2) continuously through ALU and FPU respectively. Measure different aspects of memory performance via synthetic simulations. Each simulation consists of four performances (Copy, Scale, Add, Triad). Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

File: opnfv_yardstick_tc069.yaml

  • SLA (optional): 7000 (MBps) min_bandwidth: The minimum amount of memory bandwidth that is accepted.

  • type_id: 1 - runs a specified benchmark (by an ID number):

    1 – INTmark [writing] 4 – FLOATmark [writing] 2 – INTmark [reading] 5 – FLOATmark [reading] 3 – INTmem 6 – FLOATmem

  • block_size: 64 Megabytes - the maximum block

    size per array.

  • load: 32 Gigabytes - the amount of data load per pass.

  • iterations: 5 - test is run 5 times iteratively.

  • interval: 1 - there is 1 second delay between each iteration.

test tool

RAMspeed

RAMspeed is a free open source command line utility to measure cache and memory performance of computer systems. RAMspeed is not always part of a Linux distribution, hence it needs to be installed in the test image.

references

RAMspeed

ETSI-NFV-TST001

applicability

Test can be configured with different:

  • benchmark operations (such as INTmark [writing], INTmark [reading], FLOATmark [writing], FLOATmark [reading], INTmem, FLOATmem);
  • block size per array;
  • load per pass;
  • number of batch run iterations;
  • iterations and intervals.

There are default values for each above-mentioned option.

pre-test conditions

The test case image needs to be installed into Glance with RAmspeed included in the image.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The host is installed as client. RAMspeed is invoked and logs are produced and stored.

Result: logs are stored.

test verdict Test fails if the measured memory bandwidth is below the SLA value or if there is a test case execution problem.
14.2.21. Yardstick Test Case Description TC070
Latency, Memory Utilization, Throughput, Packet Loss
test case id OPNFV_YARDSTICK_TC070_Latency, Memory Utilization, Throughput,Packet Loss
metric Number of flows, latency, throughput, Memory Utilization, packet loss
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of flows matter for the throughput between hosts on different compute blades. Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc070.yaml

Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. The amount configured ports map from 2 up to 1001000 flows, respectively. Each port amount is run two times, for 20 seconds each. Then the next port_amount is run, and so on. During the test Memory Utilization on both client and server, and the network latency between the client and server are measured. The client and server are distributed on different HW. For SLA max_ppm is set to 1000.

test tool

pktgen

Pktgen is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Glance image. (As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

ping

Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Glance image. (For example also a cirros image can be downloaded, it includes ping)

free

free provides information about unused and used memory and swap space on any computer running Linux or another Unix-like operating system. free is normally part of a Linux distribution, hence it doesn’t needs to be installed.

references

Ping and free man pages

pktgen

ETSI-NFV-TST001

applicability

Test can be configured with different packet sizes, amount of flows and test duration. Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to lose, not received.

pre-test conditions

The test case image needs to be installed into Glance with pktgen included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The hosts are installed, as server and client. pktgen is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.22. Yardstick Test Case Description TC071
Latency, Cache Utilization, Throughput, Packet Loss
test case id OPNFV_YARDSTICK_TC071_Latency, Cache Utilization, Throughput,Packet Loss
metric Number of flows, latency, throughput, Cache Utilization, packet loss
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of flows matter for the throughput between hosts on different compute blades. Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc071.yaml

Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. The amount configured ports map from 2 up to 1001000 flows, respectively. Each port amount is run two times, for 20 seconds each. Then the next port_amount is run, and so on. During the test Cache Utilization on both client and server, and the network latency between the client and server are measured. The client and server are distributed on different HW. For SLA max_ppm is set to 1000.

test tool

pktgen

Pktgen is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Glance image. (As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

ping

Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Glance image. (For example also a cirros image can be downloaded, it includes ping)

cachestat

cachestat is not always part of a Linux distribution, hence it needs to be installed.

references

Ping man pages

pktgen

cachestat

ETSI-NFV-TST001

applicability

Test can be configured with different packet sizes, amount of flows and test duration. Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to lose, not received.

pre-test conditions

The test case image needs to be installed into Glance with pktgen included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The hosts are installed, as server and client. pktgen is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.23. Yardstick Test Case Description TC072
Latency, Network Utilization, Throughput, Packet Loss
test case id OPNFV_YARDSTICK_TC072_Latency, Network Utilization, Throughput,Packet Loss
metric Number of flows, latency, throughput, Network Utilization, packet loss
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of flows matter for the throughput between hosts on different compute blades. Typically e.g. the performance of a vSwitch depends on the number of flows running through it. Also performance of other equipment or entities can depend on the number of flows or the packet sizes used. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc072.yaml

Packet size: 64 bytes Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. The amount configured ports map from 2 up to 1001000 flows, respectively. Each port amount is run two times, for 20 seconds each. Then the next port_amount is run, and so on. During the test Network Utilization on both client and server, and the network latency between the client and server are measured. The client and server are distributed on different HW. For SLA max_ppm is set to 1000.

test tool

pktgen

Pktgen is not always part of a Linux distribution, hence it needs to be installed. It is part of the Yardstick Glance image. (As an example see the /yardstick/tools/ directory for how to generate a Linux image with pktgen included.)

ping

Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Glance image. (For example also a cirros image can be downloaded, it includes ping)

sar

The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. sar is normally part of a Linux distribution, hence it doesn’t needs to be installed.

references

Ping and sar man pages

pktgen

ETSI-NFV-TST001

applicability

Test can be configured with different packet sizes, amount of flows and test duration. Default values exist.

SLA (optional): max_ppm: The number of packets per million packets sent that are acceptable to lose, not received.

pre-test conditions

The test case image needs to be installed into Glance with pktgen included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The hosts are installed, as server and client. pktgen is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.24. Yardstick Test Case Description TC073
Throughput per NFVI node test
test case id OPNFV_YARDSTICK_TC073_Network latency and throughput between nodes
metric Network latency and throughput
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of packet sizes and flows matter for the throughput between nodes in one pod.
configuration

file: opnfv_yardstick_tc073.yaml

Packet size: default 1024 bytes.

Test length: default 20 seconds.

The client and server are distributed on different nodes.

For SLA max_mean_latency is set to 100.

test tool netperf Netperf is a software application that provides network bandwidth testing between two hosts on a network. It supports Unix domain sockets, TCP, SCTP, DLPI and UDP via BSD Sockets. Netperf provides a number of predefined tests e.g. to measure bulk (unidirectional) data transfer or request response performance. (netperf is not always part of a Linux distribution, hence it needs to be installed.)
references netperf Man pages ETSI-NFV-TST001
applicability

Test can be configured with different packet sizes and test duration. Default values exist.

SLA (optional): max_mean_latency

pre-test conditions The POD can be reached by external ip and logged on via ssh
test sequence description and expected result
step 1 Install netperf tool on each specified node, one is as the server, and the other as the client.
step 2 Log on to the client node and use the netperf command to execute the network performance test
step 3 The throughput results stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.2.25. Yardstick Test Case Description TC074
Storperf
test case id OPNFV_YARDSTICK_TC074_Storperf
metric Storage performance
test purpose

Storperf integration with yardstick. The purpose of StorPerf is to provide a tool to measure block and object storage performance in an NFVI. When complemented with a characterization of typical VF storage performance requirements, it can provide pass/fail thresholds for test, staging, and production NFVI environments.

The benchmarks developed for block and object storage will be sufficiently varied to provide a good preview of expected storage performance behavior for any type of VNF workload.

configuration

file: opnfv_yardstick_tc074.yaml

  • agent_count: 1 - the number of VMs to be created
  • agent_image: “Ubuntu-14.04” - image used for creating VMs
  • public_network: “ext-net” - name of public network
  • volume_size: 2 - cinder volume size
  • block_sizes: “4096” - data block size
  • queue_depths: “4”
  • StorPerf_ip: “192.168.200.2”
  • query_interval: 10 - state query interval
  • timeout: 600 - maximum allowed job time
test tool

Storperf

StorPerf is a tool to measure block and object storage performance in an NFVI.

StorPerf is delivered as a Docker container from https://hub.docker.com/r/opnfv/storperf/tags/.

references

Storperf

ETSI-NFV-TST001

applicability

Test can be configured with different:

  • agent_count

  • volume_size

  • block_sizes

  • queue_depths

  • query_interval

  • timeout

  • target=[device or path] The path to either an attached storage device (/dev/vdb, etc) or a directory path (/opt/storperf) that will be used to execute the performance test. In the case of a device, the entire device will be used. If not specified, the current directory will be used.

  • workload=[workload module] If not specified, the default is to run all workloads. The workload types are:

    • rs: 100% Read, sequential data
    • ws: 100% Write, sequential data
    • rr: 100% Read, random access
    • wr: 100% Write, random access
    • rw: 70% Read / 30% write, random access
  • nossd: Do not perform SSD style preconditioning.

  • nowarm: Do not perform a warmup prior to measurements.

  • report= [job_id] Query the status of the supplied job_id and report on metrics. If a workload is supplied, will report on only that subset.

    There are default values for each above-mentioned option.

pre-test conditions

If you do not have an Ubuntu 14.04 image in Glance, you will need to add one. A key pair for launching agents is also required.

Storperf is required to be installed in the environment. There are two possible methods for Storperf installation:

Run container on Jump Host Run container in a VM

Running StorPerf on Jump Host Requirements:

  • Docker must be installed
  • Jump Host must have access to the OpenStack Controller API
  • Jump Host must have internet connectivity for downloading docker image
  • Enough floating IPs must be available to match your agent count

Running StorPerf in a VM Requirements:

  • VM has docker installed
  • VM has OpenStack Controller credentials and can communicate with the Controller API
  • VM has internet connectivity for downloading the docker image
  • Enough floating IPs must be available to match your agent count

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The Storperf is installed and Ubuntu 14.04 image is stored in glance. TC is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict None. Storage performance results are fetched and stored.
14.2.26. Yardstick Test Case Description TC075
Network Capacity and Scale Testing
test case id OPNFV_YARDSTICK_TC075_Network_Capacity_and_Scale_testing
metric Number of connections, Number of frames sent/received
test purpose To evaluate the network capacity and scale with regards to connections and frmaes.
configuration

file: opnfv_yardstick_tc075.yaml

There is no additional configuration to be set for this TC.

test tool

netstar

Netstat is normally part of any Linux distribution, hence it doesn’t need to be installed.

references

Netstat man page

ETSI-NFV-TST001

applicability This test case is mainly for evaluating network performance.
pre_test conditions Each pod node must have netstat included in it.
test sequence description and expected result
step 1

The pod is available. Netstat is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict None. Number of connections and frames are fetched and stored.
14.2.27. Yardstick Test Case Description TC076
Monitor Network Metrics
test case id OPNFV_YARDSTICK_TC076_Monitor_Network_Metrics
metric IP datagram error rate, ICMP message error rate, TCP segment error rate and UDP datagram error rate
test purpose

The purpose of TC076 is to evaluate the IaaS network reliability with regards to IP datagram error rate, ICMP message error rate, TCP segment error rate and UDP datagram error rate.

TC076 monitors network metrics provided by the Linux kernel in a host and calculates IP datagram error rate, ICMP message error rate, TCP segment error rate and UDP datagram error rate.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

nstat

nstat is a simple tool to monitor kernel snmp counters and network interface statistics.

(nstat is not always part of a Linux distribution, hence it needs to be installed. nstat is provided by the iproute2 collection, which is usually also the name of the package in many Linux distributions.As an example see the /yardstick/tools/ directory for how to generate a Linux image with iproute2 included.)

test description

Ping packets (ICMP protocol’s mandatory ECHO_REQUEST datagram) are sent from host VM to target VM(s) to elicit ICMP ECHO_RESPONSE.

nstat is invoked on the target vm to monitors network metrics provided by the Linux kernel.

configuration

file: opnfv_yardstick_tc076.yaml

There is no additional configuration to be set for this TC.

references

nstat man page

ETSI-NFV-TST001

applicability This test case is mainly for monitoring network metrics.
pre_test conditions

The test case image needs to be installed into Glance with fio included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 Two host VMs are booted, as server and client.
step 2 Yardstick is connected with the server VM by using ssh. ‘ping_benchmark’ bash script is copyied from Jump Host to the server VM via the ssh tunnel.
step 3

Ping is invoked. Ping packets are sent from server VM to client VM. RTT results are calculated and checked against the SLA. nstat is invoked on the client vm to monitors network metrics provided by the Linux kernel. IP datagram error rate, ICMP message error rate, TCP segment error rate and UDP datagram error rate are calculated. Logs are produced and stored.

Result: Logs are stored.

step 4 Two host VMs are deleted.
test verdict None.
14.2.28. Yardstick Test Case Description TC078
Compute Performance
test case id OPNFV_YARDSTICK_TC078_SPEC CPU 2006
metric compute-intensive performance
test purpose The purpose of TC078 is to evaluate the IaaS compute performance by using SPEC CPU 2006 benchmark. The SPEC CPU 2006 benchmark has several different ways to measure computer performance. One way is to measure how fast the computer completes a single task; this is called a speed measurement. Another way is to measure how many tasks computer can accomplish in a certain amount of time; this is called a throughput, capacity or rate measurement.
test tool

SPEC CPU 2006

The SPEC CPU 2006 benchmark is SPEC’s industry-standardized, CPU-intensive benchmark suite, stressing a system’s processor, memory subsystem and compiler. This benchmark suite includes the SPECint benchmarks and the SPECfp benchmarks. The SPECint 2006 benchmark contains 12 different enchmark tests and the SPECfp 2006 benchmark contains 19 different benchmark tests.

SPEC CPU 2006 is not always part of a Linux distribution. SPEC requires that users purchase a license and agree with their terms and conditions. For this test case, users must manually download cpu2006-1.2.iso from the SPEC website and save it under the yardstick/resources folder (e.g. /home/ opnfv/repos/yardstick/yardstick/resources/cpu2006-1.2.iso) SPEC CPU® 2006 benchmark is available for purchase via the SPEC order form (https://www.spec.org/order.html).

test description This test case uses SPEC CPU 2006 benchmark to measure compute-intensive performance of hosts.
configuration

file: spec_cpu.yaml (in the ‘samples’ directory)

benchmark_subset is set to int.

SLA is not available in this test case.

applicability

Test can be configured with different:

  • benchmark_subset - a subset of SPEC CPU2006 benchmarks to run;
  • SPECint_benchmark - a SPECint benchmark to run;
  • SPECint_benchmark - a SPECfp benchmark to run;
  • output_format - desired report format;
  • runspec_config - SPEC CPU2006 config file provided to the runspec binary;
  • runspec_iterations - the number of benchmark iterations to execute. For a reportable run, must be 3;
  • runspec_tune - tuning to use (base, peak, or all). For a reportable run, must be either base or all. Reportable runs do base first, then (optionally) peak;
  • runspec_size - size of input data to run (test, train, or ref). Reportable runs ensure that your binaries can produce correct results with the test and train workloads
usability This test case is used for executing SPEC CPU 2006 benchmark physical servers. The SPECint 2006 benchmark takes approximately 5 hours.
references

spec_cpu2006

ETSI-NFV-TST001

pre-test conditions
To run and install SPEC CPU2006, the following are required:
  • For SPECint2006: Both C99 and C++98 compilers;
  • For SPECfp2006: All three of C99, C++98 and Fortran-95 compilers;
  • At least 8GB of disk space availabile on the system.
test sequence description and expected result
step 1 cpu2006-1.2.iso has been saved under the yardstick/resources folder (e.g. /home/opnfv/repos/yardstick/yardstick/resources /cpu2006-1.2.iso). Additional, to use your custom runspec config file you can save it under the yardstick/resources/ files folder and specify the config file name in the runspec_config parameter.
step 2 Upload SPEC CPU2006 ISO to the target server and install SPEC CPU2006 via ansible.
step 3 Yardstick is connected with the target server by using ssh. If custom runspec config file is used, this file is copyied from yardstick to the target server via the ssh tunnel.
step 4 SPEC CPU2006 benchmark is invoked and SPEC CPU 2006 metrics are generated.
step 5 Text, HTML, CSV, PDF, and Configuration file outputs for the SPEC CPU 2006 metrics are fetch from the server and stored under /tmp/result folder.
step 6 uninstall SPEC CPU2006 and remove cpu2006-1.2.iso from the target server .
test verdict None. SPEC CPU2006 results are collected and stored.
14.2.29. Yardstick Test Case Description TC079
Storage Performance
test case id OPNFV_YARDSTICK_TC079_Bonnie++
metric Sequential Input/Output and Sequential/Random Create speed and CPU useage.
test purpose The purpose of TC078 is to evaluate the IaaS storage performance with regards to Sequential Input/Output and Sequential/Random Create speed and CPU useage statistics.
test tool

Bonnie++

Bonnie++ is a disk and file system benchmarking tool for measuring I/O performance. With Bonnie++ you can quickly and easily produce a meaningful value to represent your current file system performance.

Bonnie++ is not always part of a Linux distribution, hence it needs to be installed in the test image.

test description
This test case uses Bonnie++ to perform the tests below:
  • Create files in sequential order
  • Stat files in sequential order
  • Delete files in sequential order
  • Create files in random order
  • Stat files in random order
  • Delete files in random order
configuration

file: bonnie++.yaml (in the ‘samples’ directory)

file_size is set to 1024; ram_size is set to 512; test_dir is set to ‘/tmp’; concurrency is set to 1.

SLA is not available in this test case.

applicability

Test can be configured with different:

  • file_size - size fo the test file in MB. File size should be double RAM for good results;
  • ram_size - specify RAM size in MB to use, this is used to reduce testing time;
  • test_dir - this directory is where bonnie++ will create the benchmark operations;
  • test_user - the user who should perform the test. This is not required if you are not running as root;
  • concurrency - number of thread to perform test;
usability This test case is used for executing Bonnie++ benchmark in VMs.
references

bonnie++_

ETSI-NFV-TST001

pre-test conditions The Bonnie++ distribution includes a ‘bon_csv2html’ Perl script, which takes the comma-separated values reported by Bonnie++ and generates an HTML page displaying them. To use this feature, bonnie++ is required to be install with yardstick (e.g. in yardstick docker).
test sequence description and expected result
step 1 A host VM with fio installed is booted.
step 2 Yardstick is connected with the host VM by using ssh.
step 3

Bonnie++ benchmark is invoked. Simulated IO operations are started. Logs are produced and stored.

Result: Logs are stored.

step 4 An HTML report is generated using bonnie++ benchmark results and stored under /tmp/bonnie.html.
step 5 The host VM is deleted.
test verdict None. Bonnie++ html report is generated.
14.2.30. Yardstick Test Case Description TC080
Network Latency
test case id OPNFV_YARDSTICK_TC080_NETWORK_LATENCY_BETWEEN_CONTAINER
metric RTT (Round Trip Time)
test purpose

The purpose of TC080 is to do a basic verification that network latency is within acceptable boundaries when packets travel between containers located in two different Kubernetes pods.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

ping

Ping is a computer network administration software utility used to test the reachability of a host on an Internet Protocol (IP) network. It measures the round-trip time for packet sent from the originating host to a destination computer that are echoed back to the source.

Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Docker image.

test topology Ping packets (ICMP protocol’s mandatory ECHO_REQUEST datagram) are sent from host container to target container to elicit ICMP ECHO_RESPONSE.
configuration

file: opnfv_yardstick_tc080.yaml

Packet size 200 bytes. Test duration 60 seconds. SLA RTT is set to maximum 10 ms.

applicability

This test case can be configured with different:

  • packet sizes;
  • burst sizes;
  • ping intervals;
  • test durations;
  • test iterations.

Default values exist.

SLA is optional. The SLA in this test case serves as an example. Considerably lower RTT is expected, and also normal to achieve in balanced L2 environments. However, to cover most configurations, both bare metal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many real time applications start to suffer badly if the RTT time is higher than this. Some may suffer bad also close to this RTT, while others may not suffer at all. It is a compromise that may have to be tuned for different configuration purposes.

usability This test case should be run in Kunernetes environment.
references

Ping

ETSI-NFV-TST001

pre-test conditions

The test case Docker image (openretriever/yardstick) needs to be pulled into Kubernetes environment.

No further requirements have been identified.

test sequence description and expected result
step 1 Two containers are booted, as server and client.
step 2 Yardstick is connected with the server container by using ssh. ‘ping_benchmark’ bash script is copied from Jump Host to the server container via the ssh tunnel.
step 3

Ping is invoked. Ping packets are sent from server container to client container. RTT results are calculated and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 Two containers are deleted.
test verdict Test should not PASS if any RTT is above the optional SLA value, or if there is a test case execution problem.
14.2.31. Yardstick Test Case Description TC080
Network Latency
test case id OPNFV_YARDSTICK_TC081_NETWORK_LATENCY_BETWEEN_CONTAINER_AND_ VM
metric RTT (Round Trip Time)
test purpose

The purpose of TC080 is to do a basic verification that network latency is within acceptable boundaries when packets travel between a containers and a VM.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

ping

Ping is a computer network administration software utility used to test the reachability of a host on an Internet Protocol (IP) network. It measures the round-trip time for packet sent from the originating host to a destination computer that are echoed back to the source.

Ping is normally part of any Linux distribution, hence it doesn’t need to be installed. It is also part of the Yardstick Docker image. (For example also a Cirros image can be downloaded from cirros-image, it includes ping)

test topology Ping packets (ICMP protocol’s mandatory ECHO_REQUEST datagram) are sent from host container to target vm to elicit ICMP ECHO_RESPONSE.
configuration

file: opnfv_yardstick_tc081.yaml

Packet size 200 bytes. Test duration 60 seconds. SLA RTT is set to maximum 10 ms.

applicability

This test case can be configured with different:

  • packet sizes;
  • burst sizes;
  • ping intervals;
  • test durations;
  • test iterations.

Default values exist.

SLA is optional. The SLA in this test case serves as an example. Considerably lower RTT is expected, and also normal to achieve in balanced L2 environments. However, to cover most configurations, both bare metal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many real time applications start to suffer badly if the RTT time is higher than this. Some may suffer bad also close to this RTT, while others may not suffer at all. It is a compromise that may have to be tuned for different configuration purposes.

usability This test case should be run in Kunernetes environment.
references

Ping

ETSI-NFV-TST001

pre-test conditions

The test case Docker image (openretriever/yardstick) needs to be pulled into Kubernetes environment. The VM image (cirros-image) needs to be installed into Glance with ping included in it.

No further requirements have been identified.

test sequence description and expected result
step 1 A containers is booted, as server and a VM is booted as client.
step 2 Yardstick is connected with the server container by using ssh. ‘ping_benchmark’ bash script is copied from Jump Host to the server container via the ssh tunnel.
step 3

Ping is invoked. Ping packets are sent from server container to client VM. RTT results are calculated and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 The container and VM are deleted.
test verdict Test should not PASS if any RTT is above the optional SLA value, or if there is a test case execution problem.
14.2.32. Yardstick Test Case Description TC083
Throughput per VM test
test case id OPNFV_YARDSTICK_TC083_Network latency and throughput between VMs
metric Network latency and throughput
test purpose To evaluate the IaaS network performance with regards to flows and throughput, such as if and how different amounts of packet sizes and flows matter for the throughput between 2 VMs in one pod.
configuration

file: opnfv_yardstick_tc083.yaml

Packet size: default 1024 bytes.

Test length: default 20 seconds.

The client and server are distributed on different nodes.

For SLA max_mean_latency is set to 100.

test tool netperf Netperf is a software application that provides network bandwidth testing between two hosts on a network. It supports Unix domain sockets, TCP, SCTP, DLPI and UDP via BSD Sockets. Netperf provides a number of predefined tests e.g. to measure bulk (unidirectional) data transfer or request response performance. (netperf is not always part of a Linux distribution, hence it needs to be installed.)
references netperf Man pages ETSI-NFV-TST001
applicability

Test can be configured with different packet sizes and test duration. Default values exist.

SLA (optional): max_mean_latency

pre-test conditions The POD can be reached by external ip and logged on via ssh
test sequence description and expected result
step 1 Install netperf tool on each specified node, one is as the server, and the other as the client.
step 2 Log on to the client node and use the netperf command to execute the network performance test
step 3 The throughput results stored.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3. OPNFV Feature Test Cases
14.3.1. H A
14.3.1.1. Yardstick Test Case Description TC019
Control Node Openstack Service High Availability
test case id OPNFV_YARDSTICK_TC019_HA: Control node Openstack service down
test purpose This test case will verify the high availability of the service provided by OpenStack (like nova-api, neutro-server) on control node.
test method This test case kills the processes of a specific Openstack service on a selected control node, then checks whether the request of the related Openstack command is OK and the killed processes are recovered.
attackers

In this test case, an attacker called “kill-process” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “kill-process” in this test case. 2) process_name: which is the process name of the specified OpenStack service. If there are multiple processes use the same name on the host, all of them are killed by this attacker. 3) host: which is the name of a control node being attacked.

e.g. -fault_type: “kill-process” -process_name: “nova-api” -host: node1

monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific

Openstack command, which needs two parameters:

1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request

  1. the “process” monitor check whether a process is running on a specific node, which needs three parameters:

1) monitor_type: which used for finding the monitor class and related scritps. It should be always set to “process” for this monitor. 2) process_name: which is the process name for monitor 3) host: which is the name of the node runing the process

e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “openstack server list” monitor2: -monitor_type: “process” -process_name: “nova-api” -host: node1

metrics In this test case, there are two metrics: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request. 2)process_recover_time: which indicates the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc019.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the kill process script with param value specified by “process_name”

Result: Process will be killed.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It will check the status of the specified process on the host, and restart the process if it is not running for next test cases
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.2. Yardstick Test Case Description TC025
OpenStack Controller Node abnormally shutdown High Availability
test case id OPNFV_YARDSTICK_TC025_HA: OpenStack Controller Node abnormally shutdown
test purpose This test case will verify the high availability of controller node. When one of the controller node abnormally shutdown, the service provided by it should be OK.
test method This test case shutdowns a specified controller node with some fault injection tools, then checks whether all services provided by the controller node are OK with some monitor tools.
attackers

In this test case, an attacker called “host-shutdown” is needed. This attacker includes two parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “host-shutdown” in this test case. 2) host: the name of a controller node being attacked.

e.g. -fault_type: “host-shutdown” -host: node1

monitors

In this test case, one kind of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific

Openstack command, which needs two parameters

1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request

There are four instance of the “openstack-cmd” monitor: monitor1: -monitor_type: “openstack-cmd” -api_name: “nova image-list” monitor2: -monitor_type: “openstack-cmd” -api_name: “neutron router-list” monitor3: -monitor_type: “openstack-cmd” -api_name: “heat stack-list” monitor4: -monitor_type: “openstack-cmd” -api_name: “cinder list”

metrics In this test case, there is one metric: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc019.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute shutdown script on the host

Result: The host will be shutdown.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: All monitor result will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It restarts the specified controller node if it is not restarted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.3. Yardstick Test Case Description TC045
Control Node Openstack Service High Availability - Neutron Server
test case id OPNFV_YARDSTICK_TC045: Control node Openstack service down - neutron server
test purpose This test case will verify the high availability of the network service provided by OpenStack (neutro-server) on control node.
test method This test case kills the processes of neutron-server service on a selected control node, then checks whether the request of the related Openstack command is OK and the killed processes are recovered.
attackers

In this test case, an attacker called “kill-process” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “kill-process” in this test case. 2) process_name: which is the process name of the specified OpenStack service. If there are multiple processes use the same name on the host, all of them are killed by this attacker. In this case. This parameter should always set to “neutron- server”. 3) host: which is the name of a control node being attacked.

e.g. -fault_type: “kill-process” -process_name: “neutron-server” -host: node1

monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request. In this case, the command name should be neutron related commands.

2. the “process” monitor check whether a process is running on a specific node, which needs three parameters: 1) monitor_type: which used for finding the monitor class and related scritps. It should be always set to “process” for this monitor. 2) process_name: which is the process name for monitor 3) host: which is the name of the node runing the process

e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “neutron agent-list” monitor2: -monitor_type: “process” -process_name: “neutron-server” -host: node1

metrics In this test case, there are two metrics: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request. 2)process_recover_time: which indicates the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc045.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the kill process script with param value specified by “process_name”

Result: Process will be killed.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It will check the status of the specified process on the host, and restart the process if it is not running for next test cases
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.4. Yardstick Test Case Description TC046
Control Node Openstack Service High Availability - Keystone
test case id OPNFV_YARDSTICK_TC046: Control node Openstack service down - keystone
test purpose This test case will verify the high availability of the user service provided by OpenStack (keystone) on control node.
test method This test case kills the processes of keystone service on a selected control node, then checks whether the request of the related Openstack command is OK and the killed processes are recovered.
attackers

In this test case, an attacker called “kill-process” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “kill-process” in this test case. 2) process_name: which is the process name of the specified OpenStack service. If there are multiple processes use the same name on the host, all of them are killed by this attacker. In this case. This parameter should always set to “keystone” 3) host: which is the name of a control node being attacked.

e.g. -fault_type: “kill-process” -process_name: “keystone” -host: node1

monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request. In this case, the command name should be keystone related commands.

2. the “process” monitor check whether a process is running on a specific node, which needs three parameters: 1) monitor_type: which used for finding the monitor class and related scritps. It should be always set to “process” for this monitor. 2) process_name: which is the process name for monitor 3) host: which is the name of the node runing the process

e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “keystone user-list” monitor2: -monitor_type: “process” -process_name: “keystone” -host: node1

metrics In this test case, there are two metrics: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request. 2)process_recover_time: which indicates the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc046.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the kill process script with param value specified by “process_name”

Result: Process will be killed.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It will check the status of the specified process on the host, and restart the process if it is not running for next test cases
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.5. Yardstick Test Case Description TC047
Control Node Openstack Service High Availability - Glance Api
test case id OPNFV_YARDSTICK_TC047: Control node Openstack service down - glance api
test purpose This test case will verify the high availability of the image service provided by OpenStack (glance-api) on control node.
test method This test case kills the processes of glance-api service on a selected control node, then checks whether the request of the related Openstack command is OK and the killed processes are recovered.
attackers

In this test case, an attacker called “kill-process” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “kill-process” in this test case. 2) process_name: which is the process name of the specified OpenStack service. If there are multiple processes use the same name on the host, all of them are killed by this attacker. In this case. This parameter should always set to “glance- api”. 3) host: which is the name of a control node being attacked.

e.g. -fault_type: “kill-process” -process_name: “glance-api” -host: node1

monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request. In this case, the command name should be glance related commands.

2. the “process” monitor check whether a process is running on a specific node, which needs three parameters: 1) monitor_type: which used for finding the monitor class and related scritps. It should be always set to “process” for this monitor. 2) process_name: which is the process name for monitor 3) host: which is the name of the node runing the process

e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “glance image-list” monitor2: -monitor_type: “process” -process_name: “glance-api” -host: node1

metrics In this test case, there are two metrics: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request. 2)process_recover_time: which indicates the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc047.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the kill process script with param value specified by “process_name”

Result: Process will be killed.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It will check the status of the specified process on the host, and restart the process if it is not running for next test cases
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.6. Yardstick Test Case Description TC048
Control Node Openstack Service High Availability - Cinder Api
test case id OPNFV_YARDSTICK_TC048: Control node Openstack service down - cinder api
test purpose This test case will verify the high availability of the volume service provided by OpenStack (cinder-api) on control node.
test method This test case kills the processes of cinder-api service on a selected control node, then checks whether the request of the related Openstack command is OK and the killed processes are recovered.
attackers

In this test case, an attacker called “kill-process” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “kill-process” in this test case. 2) process_name: which is the process name of the specified OpenStack service. If there are multiple processes use the same name on the host, all of them are killed by this attacker. In this case. This parameter should always set to “cinder- api”. 3) host: which is the name of a control node being attacked.

e.g. -fault_type: “kill-process” -process_name: “cinder-api” -host: node1

monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request. In this case, the command name should be cinder related commands.

2. the “process” monitor check whether a process is running on a specific node, which needs three parameters: 1) monitor_type: which used for finding the monitor class and related scritps. It should be always set to “process” for this monitor. 2) process_name: which is the process name for monitor 3) host: which is the name of the node runing the process

e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “cinder list” monitor2: -monitor_type: “process” -process_name: “cinder-api” -host: node1

metrics In this test case, there are two metrics: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request. 2)process_recover_time: which indicates the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc048.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the kill process script with param value specified by “process_name”

Result: Process will be killed.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It will check the status of the specified process on the host, and restart the process if it is not running for next test cases
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.7. Yardstick Test Case Description TC049
Control Node Openstack Service High Availability - Swift Proxy
test case id OPNFV_YARDSTICK_TC049: Control node Openstack service down - swift proxy
test purpose This test case will verify the high availability of the storage service provided by OpenStack (swift-proxy) on control node.
test method This test case kills the processes of swift-proxy service on a selected control node, then checks whether the request of the related Openstack command is OK and the killed processes are recovered.
attackers

In this test case, an attacker called “kill-process” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “kill-process” in this test case. 2) process_name: which is the process name of the specified OpenStack service. If there are multiple processes use the same name on the host, all of them are killed by this attacker. In this case. This parameter should always set to “swift- proxy”. 3) host: which is the name of a control node being attacked.

e.g. -fault_type: “kill-process” -process_name: “swift-proxy” -host: node1

monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request. In this case, the command name should be swift related commands.

2. the “process” monitor check whether a process is running on a specific node, which needs three parameters: 1) monitor_type: which used for finding the monitor class and related scritps. It should be always set to “process” for this monitor. 2) process_name: which is the process name for monitor 3) host: which is the name of the node runing the process

e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “swift stat” monitor2: -monitor_type: “process” -process_name: “swift-proxy” -host: node1

metrics In this test case, there are two metrics: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request. 2)process_recover_time: which indicates the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc049.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the kill process script with param value specified by “process_name”

Result: Process will be killed.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It will check the status of the specified process on the host, and restart the process if it is not running for next test cases
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.8. Yardstick Test Case Description TC050
OpenStack Controller Node Network High Availability
test case id OPNFV_YARDSTICK_TC050: OpenStack Controller Node Network High Availability
test purpose This test case will verify the high availability of control node. When one of the controller failed to connect the network, which breaks down the Openstack services on this node. These Openstack service should able to be accessed by other controller nodes, and the services on failed controller node should be isolated.
test method This test case turns off the network interfaces of a specified control node, then checks whether all services provided by the control node are OK with some monitor tools.
attackers

In this test case, an attacker called “close-interface” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “close-interface” in this test case. 2) host: which is the name of a control node being attacked. 3) interface: the network interface to be turned off.

There are four instance of the “close-interface” monitor: attacker1(for public netork): -fault_type: “close-interface” -host: node1 -interface: “br-ex” attacker2(for management netork): -fault_type: “close-interface” -host: node1 -interface: “br-mgmt” attacker3(for storage netork): -fault_type: “close-interface” -host: node1 -interface: “br-storage” attacker4(for private netork): -fault_type: “close-interface” -host: node1 -interface: “br-mesh”

monitors

In this test case, the monitor named “openstack-cmd” is needed. The monitor needs needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request

There are four instance of the “openstack-cmd” monitor: monitor1: -monitor_type: “openstack-cmd” -command_name: “nova image-list” monitor2: -monitor_type: “openstack-cmd” -command_name: “neutron router-list” monitor3: -monitor_type: “openstack-cmd” -command_name: “heat stack-list” monitor4: -monitor_type: “openstack-cmd” -command_name: “cinder list”

metrics In this test case, there is one metric: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc050.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the turnoff network interface script with param value specified by “interface”.

Result: Network interfaces will be turned down.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It turns up the network interface of the control node if it is not turned up.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.9. Yardstick Test Case Description TC051
OpenStack Controller Node CPU Overload High Availability
test case id OPNFV_YARDSTICK_TC051: OpenStack Controller Node CPU Overload High Availability
test purpose This test case will verify the high availability of control node. When the CPU usage of a specified controller node is stressed to 100%, which breaks down the Openstack services on this node. These Openstack service should able to be accessed by other controller nodes, and the services on failed controller node should be isolated.
test method This test case stresses the CPU uasge of a specified control node to 100%, then checks whether all services provided by the environment are OK with some monitor tools.
attackers In this test case, an attacker called “stress-cpu” is needed. This attacker includes two parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “stress-cpu” in this test case. 2) host: which is the name of a control node being attacked. e.g. -fault_type: “stress-cpu” -host: node1
monitors

In this test case, the monitor named “openstack-cmd” is needed. The monitor needs needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request

There are four instance of the “openstack-cmd” monitor: monitor1: -monitor_type: “openstack-cmd” -command_name: “nova image-list” monitor2: -monitor_type: “openstack-cmd” -command_name: “neutron router-list” monitor3: -monitor_type: “openstack-cmd” -command_name: “heat stack-list” monitor4: -monitor_type: “openstack-cmd” -command_name: “cinder list”

metrics In this test case, there is one metric: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc051.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the stress cpu script on the host.

Result: The CPU usage of the host will be stressed to 100%.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It kills the process that stresses the CPU usage.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.10. Yardstick Test Case Description TC052
OpenStack Controller Node Disk I/O Block High Availability
test case id OPNFV_YARDSTICK_TC052: OpenStack Controller Node Disk I/O Block High Availability
test purpose This test case will verify the high availability of control node. When the disk I/O of a specified disk is blocked, which breaks down the Openstack services on this node. Read and write services should still be accessed by other controller nodes, and the services on failed controller node should be isolated.
test method This test case blocks the disk I/O of a specified control node, then checks whether the services that need to read or wirte the disk of the control node are OK with some monitor tools.
attackers In this test case, an attacker called “disk-block” is needed. This attacker includes two parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “disk-block” in this test case. 2) host: which is the name of a control node being attacked. e.g. -fault_type: “disk-block” -host: node1
monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scripts. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request.

e.g. -monitor_type: “openstack-cmd” -command_name: “nova flavor-list”

2. the second monitor verifies the read and write function by a “operation” and a “result checker”. the “operation” have two parameters: 1) operation_type: which is used for finding the operation class and related scripts. 2) action_parameter: parameters for the operation. the “result checker” have three parameters: 1) checker_type: which is used for finding the reuslt checker class and realted scripts. 2) expectedValue: the expected value for the output of the checker script. 3) condition: whether the expected value is in the output of checker script or is totally same with the output.

In this case, the “operation” adds a flavor and the “result checker” checks whether ths flavor is created. Their parameters show as follows: operation: -operation_type: “nova-create-flavor” -action_parameter:

flavorconfig: “test-001 test-001 100 1 1”

result checker: -checker_type: “check-flavor” -expectedValue: “test-001” -condition: “in”

metrics In this test case, there is one metric: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc052.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

do attacker: connect the host through SSH, and then execute the block disk I/O script on the host.

Result: The disk I/O of the host will be blocked

step 2

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 3 do operation: add a flavor
step 4 do result checker: check whether the falvor is created
step 5

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 6

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It excutes the release disk I/O script to release the blocked I/O.
test verdict Fails if monnitor SLA is not passed or the result checker is not passed, or if there is a test case execution problem.
14.3.1.11. Yardstick Test Case Description TC053
OpenStack Controller Load Balance Service High Availability
test case id OPNFV_YARDSTICK_TC053: OpenStack Controller Load Balance Service High Availability
test purpose This test case will verify the high availability of the load balance service(current is HAProxy) that supports OpenStack on controller node. When the load balance service of a specified controller node is killed, whether other load balancers on other controller nodes will work, and whether the controller node will restart the load balancer are checked.
test method This test case kills the processes of load balance service on a selected control node, then checks whether the request of the related Openstack command is OK and the killed processes are recovered.
attackers

In this test case, an attacker called “kill-process” is needed. This attacker includes three parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “kill-process” in this test case. 2) process_name: which is the process name of the specified OpenStack service. If there are multiple processes use the same name on the host, all of them are killed by this attacker. In this case. This parameter should always set to “swift- proxy”. 3) host: which is the name of a control node being attacked.

e.g. -fault_type: “kill-process” -process_name: “haproxy” -host: node1

monitors

In this test case, two kinds of monitor are needed: 1. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scritps. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request.

2. the “process” monitor check whether a process is running on a specific node, which needs three parameters: 1) monitor_type: which used for finding the monitor class and related scripts. It should be always set to “process” for this monitor. 2) process_name: which is the process name for monitor 3) host: which is the name of the node runing the process In this case, the command_name of monitor1 should be services that is supported by load balancer and the process- name of monitor2 should be “haproxy”, for example:

e.g. monitor1: -monitor_type: “openstack-cmd” -command_name: “nova image-list” monitor2: -monitor_type: “process” -process_name: “haproxy” -host: node1

metrics In this test case, there are two metrics: 1)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request. 2)process_recover_time: which indicates the maximun time (seconds) from the process being killed to recovered
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc053.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the kill process script with param value specified by “process_name”

Result: Process will be killed.

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It will check the status of the specified process on the host, and restart the process if it is not running for next test cases.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.1.12. Yardstick Test Case Description TC054
OpenStack Virtual IP High Availability
test case id OPNFV_YARDSTICK_TC054: OpenStack Virtual IP High Availability
test purpose This test case will verify the high availability for virtual ip in the environment. When master node of virtual ip is abnormally shutdown, connection to virtual ip and the services binded to the virtual IP it should be OK.
test method This test case shutdowns the virtual IP master node with some fault injection tools, then checks whether virtual ips can be pinged and services binded to virtual ip are OK with some monitor tools.
attackers

In this test case, an attacker called “control-shutdown” is needed. This attacker includes two parameters: 1) fault_type: which is used for finding the attacker’s scripts. It should be always set to “control-shutdown” in this test case. 2) host: which is the name of a control node being attacked.

In this case the host should be the virtual ip master node, that means the host ip is the virtual ip, for exapmle: -fault_type: “control-shutdown” -host: node1(the VIP Master node)

monitors

In this test case, two kinds of monitor are needed: 1. the “ip_status” monitor that pings a specific ip to check the connectivity of this ip, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scripts. It should be always set to “ip_status” for this monitor. 2) ip_address: The ip to be pinged. In this case, ip_address should be the virtual IP.

2. the “openstack-cmd” monitor constantly request a specific Openstack command, which needs two parameters: 1) monitor_type: which is used for finding the monitor class and related scripts. It should be always set to “openstack-cmd” for this monitor. 2) command_name: which is the command name used for request.

e.g. monitor1: -monitor_type: “ip_status” -host: 192.168.0.2 monitor2: -monitor_type: “openstack-cmd” -command_name: “nova image-list”

metrics In this test case, there are two metrics: 1) ping_outage_time: which-indicates the maximum outage time to ping the specified host. 2)service_outage_time: which indicates the maximum outage time (seconds) of the specified Openstack command request.
test tool Developed by the project. Please see folder: “yardstick/benchmark/scenarios/availability/ha_tools”
references ETSI NFV REL001
configuration

This test case needs two configuration files: 1) test case file: opnfv_yardstick_tc054.yaml -Attackers: see above “attackers” discription -waiting_time: which is the time (seconds) from the process being killed to stoping monitors the monitors -Monitors: see above “monitors” discription -SLA: see above “metrics” discription

2)POD file: pod.yaml The POD configuration should record on pod.yaml first. the “host” item in this test case will use the node name in the pod.yaml.

test sequence description and expected result
step 1

start monitors: each monitor will run with independently process

Result: The monitor info will be collected.

step 2

do attacker: connect the host through SSH, and then execute the shutdown script on the VIP master node.

Result: VIP master node will be shutdown

step 3

stop monitors after a period of time specified by “waiting_time”

Result: The monitor info will be aggregated.

step 4

verify the SLA

Result: The test case is passed or not.

post-action It is the action when the test cases exist. It restarts the original VIP master node if it is not restarted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.2. IPv6
14.3.2.1. Yardstick Test Case Description TC027
IPv6 connectivity between nodes on the tenant network
test case id OPNFV_YARDSTICK_TC027_IPv6 connectivity
metric RTT, Round Trip Time
test purpose To do a basic verification that IPv6 connectivity is within acceptable boundaries when ipv6 packets travel between hosts located on same or different compute blades. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration

file: opnfv_yardstick_tc027.yaml

Packet size 56 bytes. SLA RTT is set to maximum 30 ms. ipv6 test case can be configured as three independent modules (setup, run, teardown). if you only want to setup ipv6 testing environment, do some tests as you want, “run_step” of task yaml file should be configured as “setup”. if you want to setup and run ping6 testing automatically, “run_step” should be configured as “setup, run”. and if you have had a environment which has been setup, you only wan to verify the connectivity of ipv6 network, “run_step” should be “run”. Of course, default is that three modules run sequentially.

test tool

ping6

Ping6 is normally part of Linux distribution, hence it doesn’t need to be installed.

references

ipv6

ETSI-NFV-TST001

applicability Test case can be configured with different run step you can run setup, run benchmark, teardown independently SLA is optional. The SLA in this test case serves as an example. Considerably lower RTT is expected.
pre-test conditions

The test case image needs to be installed into Glance with ping6 included in it.

For Brahmaputra, a compass_os_nosdn_ha deploy scenario is need. more installer and more sdn deploy scenario will be supported soon

test sequence description and expected result
step 1 To setup IPV6 testing environment: 1. disable security group 2. create (ipv6, ipv4) router, network and subnet 3. create VRouter, VM1, VM2
step 2 To run ping6 to verify IPV6 connectivity : 1. ssh to VM1 2. Ping6 to ipv6 router from VM1 3. Get the result(RTT) and logs are stored
step 3 To teardown IPV6 testing environment 1. delete VRouter, VM1, VM2 2. delete (ipv6, ipv4) router, network and subnet 3. enable security group
test verdict Test should not PASS if any RTT is above the optional SLA value, or if there is a test case execution problem.
14.3.3. KVM
14.3.3.1. Yardstick Test Case Description TC028
KVM Latency measurements
test case id OPNFV_YARDSTICK_TC028_KVM Latency measurements
metric min, avg and max latency
test purpose To evaluate the IaaS KVM virtualization capability with regards to min, avg and max latency. The purpose is also to be able to spot trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.
configuration file: samples/cyclictest-node-context.yaml
test tool

Cyclictest

(Cyclictest is not always part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with cyclictest included.)

references Cyclictest
applicability This test case is mainly for kvm4nfv project CI verify. Upgrade host linux kernel, boot a gust vm update it’s linux kernel, and then run the cyclictest to test the new kernel is work well.
pre-test conditions

The test kernel rpm, test sequence scripts and test guest image need put the right folders as specified in the test case yaml file. The test guest image needs with cyclictest included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The host and guest os kernel is upgraded. Cyclictest is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.3.4. Parser
14.3.4.1. Yardstick Test Case Description TC040
Verify Parser Yang-to-Tosca
test case id OPNFV_YARDSTICK_TC040 Verify Parser Yang-to-Tosca
metric
  1. tosca file which is converted from yang file by Parser
  2. result whether the output is same with expected outcome
test purpose To verify the function of Yang-to-Tosca in Parser.
configuration

file: opnfv_yardstick_tc040.yaml

yangfile: the path of the yangfile which you want to convert toscafile: the path of the toscafile which is your expected outcome.

test tool

Parser

(Parser is not part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/benchmark/scenarios/parser/parser_setup.sh for how to install it manual. Of course, it will be installed and uninstalled automatically when you run this test case by yardstick)

references Parser
applicability Test can be configured with different path of yangfile and toscafile to fit your real environment to verify Parser
pre-test conditions No POD specific requirements have been identified. it can be run without VM
test sequence description and expected result
step 1

parser is installed without VM, running Yang-to-Tosca module to convert yang file to tosca file, validating output against expected outcome.

Result: Logs are stored.

test verdict Fails only if output is different with expected outcome or if there is a test case execution problem.

14.2.25. Yardstick Test Case Description TC074
Storperf
test case id OPNFV_YARDSTICK_TC074_Storperf
metric Storage performance
test purpose

Storperf integration with yardstick. The purpose of StorPerf is to provide a tool to measure block and object storage performance in an NFVI. When complemented with a characterization of typical VF storage performance requirements, it can provide pass/fail thresholds for test, staging, and production NFVI environments.

The benchmarks developed for block and object storage will be sufficiently varied to provide a good preview of expected storage performance behavior for any type of VNF workload.

configuration

file: opnfv_yardstick_tc074.yaml

  • agent_count: 1 - the number of VMs to be created
  • agent_image: “Ubuntu-14.04” - image used for creating VMs
  • public_network: “ext-net” - name of public network
  • volume_size: 2 - cinder volume size
  • block_sizes: “4096” - data block size
  • queue_depths: “4”
  • StorPerf_ip: “192.168.200.2”
  • query_interval: 10 - state query interval
  • timeout: 600 - maximum allowed job time
test tool

Storperf

StorPerf is a tool to measure block and object storage performance in an NFVI.

StorPerf is delivered as a Docker container from https://hub.docker.com/r/opnfv/storperf/tags/.

references

Storperf

ETSI-NFV-TST001

applicability

Test can be configured with different:

  • agent_count

  • volume_size

  • block_sizes

  • queue_depths

  • query_interval

  • timeout

  • target=[device or path] The path to either an attached storage device (/dev/vdb, etc) or a directory path (/opt/storperf) that will be used to execute the performance test. In the case of a device, the entire device will be used. If not specified, the current directory will be used.

  • workload=[workload module] If not specified, the default is to run all workloads. The workload types are:

    • rs: 100% Read, sequential data
    • ws: 100% Write, sequential data
    • rr: 100% Read, random access
    • wr: 100% Write, random access
    • rw: 70% Read / 30% write, random access
  • nossd: Do not perform SSD style preconditioning.

  • nowarm: Do not perform a warmup prior to measurements.

  • report= [job_id] Query the status of the supplied job_id and report on metrics. If a workload is supplied, will report on only that subset.

    There are default values for each above-mentioned option.

pre-test conditions

If you do not have an Ubuntu 14.04 image in Glance, you will need to add one. A key pair for launching agents is also required.

Storperf is required to be installed in the environment. There are two possible methods for Storperf installation:

Run container on Jump Host Run container in a VM

Running StorPerf on Jump Host Requirements:

  • Docker must be installed
  • Jump Host must have access to the OpenStack Controller API
  • Jump Host must have internet connectivity for downloading docker image
  • Enough floating IPs must be available to match your agent count

Running StorPerf in a VM Requirements:

  • VM has docker installed
  • VM has OpenStack Controller credentials and can communicate with the Controller API
  • VM has internet connectivity for downloading the docker image
  • Enough floating IPs must be available to match your agent count

No POD specific requirements have been identified.

test sequence description and expected result
step 1

The Storperf is installed and Ubuntu 14.04 image is stored in glance. TC is invoked and logs are produced and stored.

Result: Logs are stored.

test verdict None. Storage performance results are fetched and stored.
14.3.5. virtual Traffic Classifier
14.3.5.1. Yardstick Test Case Description TC006
Volume storage Performance
test case id OPNFV_YARDSTICK_TC006_VOLUME STORAGE PERFORMANCE
metric IOPS (Average IOs performed per second), Throughput (Average disk read/write bandwidth rate), Latency (Average disk read/write latency)
test purpose

The purpose of TC006 is to evaluate the IaaS volume storage performance with regards to IOPS, throughput and latency.

The purpose is also to be able to spot the trends. Test results, graphs and similar shall be stored for comparison reasons and product evolution understanding between different OPNFV versions and/or configurations.

test tool

fio

fio is an I/O tool meant to be used both for benchmark and stress/hardware verification. It has support for 19 different types of I/O engines (sync, mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio, and more), I/O priorities (for newer Linux kernels), rate I/O, forked or threaded jobs, and much more.

(fio is not always part of a Linux distribution, hence it needs to be installed. As an example see the /yardstick/tools/ directory for how to generate a Linux image with fio included.)

test description fio test is invoked in a host VM with a volume attached on a compute blade, a job file as well as parameters are passed to fio and fio will start doing what the job file tells it to do.
configuration

file: opnfv_yardstick_tc006.yaml

Fio job file is provided to define the benchmark process Target volume is mounted at /FIO_Test directory

For SLA, minimum read/write iops is set to 100, minimum read/write throughput is set to 400 KB/s, and maximum read/write latency is set to 20000 usec.

applicability

This test case can be configured with different:

  • Job file;
  • Volume mount directory.

SLA is optional. The SLA in this test case serves as an example. Considerably higher throughput and lower latency are expected. However, to cover most configurations, both baremetal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many heavy IO applications start to suffer badly if the read/write bandwidths are lower than this.

usability This test case is one of Yardstick’s generic test. Thus it is runnable on most of the scenarios.
references

fio

ETSI-NFV-TST001

pre-test conditions

The test case image needs to be installed into Glance with fio included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 A host VM with fio installed is booted. A 200G volume is attached to the host VM
step 2 Yardstick is connected with the host VM by using ssh. ‘job_file.ini’ is copyied from Jump Host to the host VM via the ssh tunnel. The attached volume is formated and mounted.
step 3

Fio benchmark is invoked. Simulated IO operations are started. IOPS, disk read/write bandwidth and latency are recorded and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 The host VM is deleted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.
14.4. Templates
14.4.1. Yardstick Test Case Description TCXXX
test case slogan e.g. Network Latency
test case id e.g. OPNFV_YARDSTICK_TC001_NW Latency
metric what will be measured, e.g. latency
test purpose describe what is the purpose of the test case
configuration what .yaml file to use, state SLA if applicable, state test duration, list and describe the scenario options used in this TC and also list the options using default values.
test tool e.g. ping
references e.g. RFCxxx, ETSI-NFVyyy
applicability describe variations of the test case which can be performend, e.g. run the test for different packet sizes
pre-test conditions describe configuration in the tool(s) used to perform the measurements (e.g. fio, pktgen), POD-specific configuration required to enable running the test
test sequence description and expected result
step 1

use this to describe tests that require sveveral steps e.g collect logs.

Result: what happens in this step e.g. logs collected

step 2

remove interface

Result: interface down.

step N

what is done in step N

Result: what happens

test verdict expected behavior, or SLA, pass/fail criteria
14.4.2. Task Template Syntax
14.4.2.1. Basic template syntax

A nice feature of the input task format used in Yardstick is that it supports the template syntax based on Jinja2. This turns out to be extremely useful when, say, you have a fixed structure of your task but you want to parameterize this task in some way. For example, imagine your input task file (task.yaml) runs a set of Ping scenarios:

# Sample benchmark task config file
# measure network latency using ping
schema: "yardstick:task:0.1"

scenarios:
-
  type: Ping
  options:
    packetsize: 200
  host: athena.demo
  target: ares.demo

  runner:
    type: Duration
    duration: 60
    interval: 1

  sla:
    max_rtt: 10
    action: monitor

context:
    ...

Let’s say you want to run the same set of scenarios with the same runner/ context/sla, but you want to try another packetsize to compare the performance. The most elegant solution is then to turn the packetsize name into a template variable:

# Sample benchmark task config file
# measure network latency using ping

schema: "yardstick:task:0.1"
scenarios:
-
  type: Ping
  options:
    packetsize: {{packetsize}}
  host: athena.demo
  target: ares.demo

  runner:
    type: Duration
    duration: 60
    interval: 1

  sla:
    max_rtt: 10
    action: monitor

context:
    ...

and then pass the argument value for {{packetsize}} when starting a task with this configuration file. Yardstick provides you with different ways to do that:

1.Pass the argument values directly in the command-line interface (with either a JSON or YAML dictionary):

yardstick task start samples/ping-template.yaml
--task-args'{"packetsize":"200"}'

2.Refer to a file that specifies the argument values (JSON/YAML):

yardstick task start samples/ping-template.yaml --task-args-file args.yaml
14.4.2.2. Using the default values

Note that the Jinja2 template syntax allows you to set the default values for your parameters. With default values set, your task file will work even if you don’t parameterize it explicitly while starting a task. The default values should be set using the {% set ... %} clause (task.yaml). For example:

# Sample benchmark task config file
# measure network latency using ping
schema: "yardstick:task:0.1"
{% set packetsize = packetsize or "100" %}
scenarios:
-
  type: Ping
  options:
  packetsize: {{packetsize}}
  host: athena.demo
  target: ares.demo

  runner:
    type: Duration
    duration: 60
    interval: 1
  ...

If you don’t pass the value for {{packetsize}} while starting a task, the default one will be used.

14.4.2.3. Advanced templates

Yardstick makes it possible to use all the power of Jinja2 template syntax, including the mechanism of built-in functions. As an example, let us make up a task file that will do a block storage performance test. The input task file (fio-template.yaml) below uses the Jinja2 for-endfor construct to accomplish that:

#Test block sizes of 4KB, 8KB, 64KB, 1MB
#Test 5 workloads: read, write, randwrite, randread, rw
schema: "yardstick:task:0.1"

 scenarios:
{% for bs in ['4k', '8k', '64k', '1024k' ] %}
  {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %}
-
  type: Fio
  options:
    filename: /home/ubuntu/data.raw
    bs: {{bs}}
    rw: {{rw}}
    ramp_time: 10
  host: fio.demo
  runner:
    type: Duration
    duration: 60
    interval: 60

  {% endfor %}
{% endfor %}
context
    ...
15. NSB Sample Test Cases
15.1. Abstract

This chapter lists available NSB test cases.

15.2. NSB PROX Test Case Descriptions
15.2.1. Yardstick Test Case Description: NSB PROX ACL
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_acl-{port_num}

  • context = baremetal or heat_context;
  • port_num = 2 or 4;
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

This test allows to measure how well the SUT can exploit structures in the list of ACL rules. The ACL rules are matched against a 7-tuple of the input packet: the regular 5-tuple and two VLAN tags. The rules in the rule set allow the packet to be forwarded and the rule set contains a default “match all” rule.

The KPI is measured with the rule set that has a moderate number of rules with moderate similarity between the rules & the fraction of rules that were used.

The ACL test cases are implemented to run in baremetal and heat context for 2 port and 4 port configuration.

configuration

The ACL test cases are listed below:

  • tc_prox_baremetal_acl-2.yaml
  • tc_prox_baremetal_acl-4.yaml
  • tc_prox_heat_context_acl-2.yaml
  • tc_prox_heat_context_acl-4.yaml

Test duration is set as 300sec for each test. Packet size set as 64 bytes in traffic profile. These can be configured

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability

This PROX ACL test cases can be configured with different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it. The test need multi-queue enabled in Glance image.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(ACL workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packets to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI is the number of packets per second for 64 bytes packet size with an accepted minimal packet loss for the default configuration.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
15.2.2. Yardstick Test Case Description: NSB PROX BNG
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_bng-{port_num}

  • context = baremetal or heat_context;
  • port_num = 4;
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

The BNG workload converts packets from QinQ to GRE tunnels, handles routing and adds/removes MPLS tags. This use case simulates a realistic and complex application. The number of users is 32K per port and the number of routes is 8K.

The BNG test cases are implemented to run in baremetal and heat context an require 4 port topology to run the default configuration.

configuration

The BNG test cases are listed below:

  • tc_prox_baremetal_bng-2.yaml
  • tc_prox_baremetal_bng-4.yaml
  • tc_prox_heat_context_bng-2.yaml
  • tc_prox_heat_context_bng-4.yaml

Test duration is set as 300sec for each test. The minimum packet size for BNG test is 78 bytes. This is set in the BNG traffic profile and can be configured to use a higher packet size for the test.

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability

The PROX BNG test cases can be configured with different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it. The test need multi-queue enabled in Glance image.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(BNG workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packets to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI is the number of packets per second for 78 bytes packet size with an accepted minimal packet loss for the default configuration.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
15.2.3. Yardstick Test Case Description: NSB PROX BNG_QoS
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_bng_qos-{port_num}

  • context = baremetal or heat_context;
  • port_num = 4;
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

The BNG+QoS workload converts packets from QinQ to GRE tunnels, handles routing and adds/removes MPLS tags and performs a QoS. This use case simulates a realistic and complex application. The number of users is 32K per port and the number of routes is 8K.

The BNG_QoS test cases are implemented to run in baremetal and heat context an require 4 port topology to run the default configuration.

configuration

The BNG_QoS test cases are listed below:

  • tc_prox_baremetal_bng_qos-2.yaml
  • tc_prox_baremetal_bng_qos-4.yaml
  • tc_prox_heat_context_bng_qos-2.yaml
  • tc_prox_heat_context_bng_qos-4.yaml

Test duration is set as 300sec for each test. The minumum packet size for BNG_QoS test is 78 bytes. This is set in the bng_qos traffic profile and can be configured to use a higher packet size for the test.

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability

This PROX BNG_QoS test cases can be configured with different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it. The test need multi-queue enabled in Glance image.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(BNG_QoS workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packets to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI is the number of packets per second for 78 bytes packet size with an accepted minimal packet loss for the default configuration.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
15.2.4. Yardstick Test Case Description: NSB PROX L2FWD
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_l2fwd-{port_num}

  • context = baremetal or heat_context;
  • port_num = 2 or 4;
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

The PROX L2FWD test has 3 types of test cases: L2FWD: The application will take packets in from one port and forward them unmodified to another port L2FWD_Packet_Touch: The application will take packets in from one port, update src and dst MACs and forward them to another port. L2FWD_Multi_Flow: The application will take packets in from one port, update src and dst MACs and forward them to another port. This test case exercises the softswitch with 200k flows.

The above test cases are implemented for baremetal and heat context for 2 port and 4 port configuration.

configuration

The L2FWD test cases are listed below:

  • tc_prox_baremetal_l2fwd-2.yaml
  • tc_prox_baremetal_l2fwd-4.yaml
  • tc_prox_baremetal_l2fwd_pktTouch-2.yaml
  • tc_prox_baremetal_l2fwd_pktTouch-4.yaml
  • tc_prox_baremetal_l2fwd_multiflow-2.yaml
  • tc_prox_baremetal_l2fwd_multiflow-4.yaml
  • tc_prox_heat_context_l2fwd-2.yaml
  • tc_prox_heat_context_l2fwd-4.yaml
  • tc_prox_heat_context_l2fwd_pktTouch-2.yaml
  • tc_prox_heat_context_l2fwd_pktTouch-4.yaml
  • tc_prox_heat_context_l2fwd_multiflow-2.yaml
  • tc_prox_heat_context_l2fwd_multiflow-4.yaml

Test duration is set as 300sec for each test. Packet size set as 64 bytes in traffic profile These can be configured

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability

The PROX L2FWD test cases can be configured with different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(L2FWD workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packets to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI is the number of packets per second for 64 bytes packet size with an accepted minimal packet loss for the default configuration.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
15.2.5. Yardstick Test Case Description: NSB PROX L3FWD
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_l3fwd-{port_num}

  • context = baremetal or heat_context;
  • port_num = 2 or 4;
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

The PROX L3FWD application performs basic routing of packets with LPM based look-up method.

The L3FWD test cases are implemented for baremetal and heat context for 2 port and 4 port configuration.

configuration

The L3FWD test cases are listed below:

  • tc_prox_baremetal_l3fwd-2.yaml
  • tc_prox_baremetal_l3fwd-4.yaml
  • tc_prox_heat_context_l3fwd-2.yaml
  • tc_prox_heat_context_l3fwd-4.yaml

Test duration is set as 300sec for each test. The minimum packet size for L3FWD test is 64 bytes. This is set in the traffic profile and can be configured to use a higher packet size for the test.

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability

This PROX L3FWD test cases can be configured with different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it. The test need multi-queue enabled in Glance image.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(L3FWD workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packet to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI is the number of packets per second for 64 byte packets with an accepted minimal packet loss for the default configuration.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
15.2.6. Yardstick Test Case Description: NSB PROX MPLS Tagging
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_mpls_tagging-{port_num}

  • context = baremetal or heat_context;
  • port_num = 2 or 4;
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

The PROX MPLS Tagging test will take packets in from one port add an MPLS tag and forward them to another port. While forwarding packets in other direction MPLS tags will be removed.

The MPLS test cases are implemented to run in baremetal and heat context an require 4 port topology to run the default configuration.

configuration

The MPLS Tagging test cases are listed below:

  • tc_prox_baremetal_mpls_tagging-2.yaml
  • tc_prox_baremetal_mpls_tagging-4.yaml
  • tc_prox_heat_context_mpls_tagging-2.yaml
  • tc_prox_heat_context_mpls_tagging-4.yaml

Test duration is set as 300sec for each test. The minimum packet size for MPLS test is 68 bytes. This is set in the traffic profile and can be configured to use higher packet sizes.

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability

The PROX MPLS Tagging test cases can be configured with different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(MPLS workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packets to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI is the number of packets per second for 68 bytes packet size with an accepted minimal packet loss for the default configuration.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
15.2.7. Yardstick Test Case Description: NSB PROX Packet Buffering
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_buffering-{port_num}

  • context = baremetal or heat_context
  • port_num = 1
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

This test measures the impact of the condition when packets get buffered, thus they stay in memory for the extended period of time, 125ms in this case.

The Packet Buffering test cases are implemented to run in baremetal and heat context.

The test runs only on the first port of the SUT.

configuration

The Packet Buffering test cases are listed below:

  • tc_prox_baremetal_buffering-1.yaml
  • tc_prox_heat_context_buffering-1.yaml

Test duration is set as 300sec for each test. The minimum packet size for Buffering test is 64 bytes. This is set in the traffic profile and can be configured to use a higher packet size for the test.

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability
The PROX Packet Buffering test cases can be configured with

different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it. The test need multi-queue enabled in Glance image.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(Packet Buffering workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packets to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI in this test is the maximum number of packets that can be forwarded given the requirement that the latency of each packet is at least 125 millisecond.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
15.2.8. Yardstick Test Case Description: NSB PROX Load Balancer
NSB PROX test for NFVI characterization
test case id

tc_prox_{context}_lb-{port_num}

  • context = baremetal or heat_context
  • port_num = 4
metric
  • Network Throughput;
  • TG Packets Out;
  • TG Packets In;
  • VNF Packets Out;
  • VNF Packets In;
  • Dropped packets;
test purpose

The applciation transmits packets on one port and revieves them on 4 ports. The conventional 5-tuple is used in this test as it requires some extraction steps and allows defining enough distinct values to find the performance limits.

The load is increased (adding more ports if needed) while packets are load balanced using a hash table of 8M entries

The number of packets per second that can be forwarded determines the KPI. The default packet size is 64 bytes.

configuration

The Load Balancer test cases are listed below:

  • tc_prox_baremetal_lb-4.yaml
  • tc_prox_heat_context_lb-4.yaml

Test duration is set as 300sec for each test. Packet size set as 64 bytes in traffic profile. These can be configured

test tool PROX PROX is a DPDK application that can simulate VNF workloads and can generate traffic and used for NFVI characterization
applicability
The PROX Load Balancer test cases can be configured with

different:

  • packet sizes;
  • test durations;
  • tolerated loss;

Default values exist.

pre-test conditions

For Openstack test case image (yardstick-samplevnfs) needs to be installed into Glance with Prox and Dpdk included in it. The test need multi-queue enabled in Glance image.

For Baremetal tests cases Prox and Dpdk must be installed in the hosts where the test is executed. The pod.yaml file must have the necessary system and NIC information

test sequence description and expected result
step 1

For Baremetal test: The TG and VNF are started on the hosts based on the pod file.

For Heat test: Two host VMs are booted, as Traffic generator and VNF(Load Balancer workload) based on the test flavor.

step 2 Yardstick is connected with the TG and VNF by using ssh. The test will resolve the topology and instantiate the VNF and TG and collect the KPI’s/metrics.
step 3

The TG will send packets to the VNF. If the number of dropped packets is more than the tolerated loss the line rate or throughput is halved. This is done until the dropped packets are within an acceptable tolerated loss.

The KPI is the number of packets per second for 78 bytes packet size with an accepted minimal packet loss for the default configuration.

step 4

In Baremetal test: The test quits the application and unbind the dpdk ports.

In Heat test: Two host VMs are deleted on test completion.

test verdict The test case will achieve a Throughput with an accepted minimal tolerated packet loss.
16. Glossary
API
Application Programming Interface
DPDK
Data Plane Development Kit
DPI
Deep Packet Inspection
DSCP
Differentiated Services Code Point
IGMP
Internet Group Management Protocol
IOPS
Input/Output Operations Per Second
NFVI
Network Function Virtualization Infrastructure
NIC
Network Interface Controller
PBFS
Packet Based per Flow State
QoS
Quality of Service
SR-IOV
Single Root IO Virtualization
SUT
System Under Test
ToS
Type of Service
VLAN
Virtual LAN
VM
Virtual Machine
VNF
Virtual Network Function
VNFC
Virtual Network Function Component
VTC
Virtual Traffic Classifier
17. References

Testing Developer Guides

Testing group

Test Framework Overview
Testing developer guide
Introduction

The OPNFV testing ecosystem is wide.

The goal of this guide consists in providing some guidelines for new developers involved in test areas.

For the description of the ecosystem, see [DEV1].

Developer journey

There are several ways to join test projects as a developer. In fact you may:

  • Develop new test cases
  • Develop frameworks
  • Develop tooling (reporting, dashboards, graphs, middleware,...)
  • Troubleshoot results
  • Post-process results

These different tasks may be done within a specific project or as a shared resource accross the different projects.

If you develop new test cases, the best practice is to contribute upstream as much as possible. You may contact the testing group to know which project - in OPNFV or upstream - would be the best place to host the test cases. Such contributions are usually directly connected to a specific project, more details can be found in the user guides of the testing projects.

Each OPNFV testing project provides test cases and the framework to manage them. As a developer, you can obviously contribute to them. The developer guide of the testing projects shall indicate the procedure to follow.

Tooling may be specific to a project or generic to all the projects. For specific tooling, please report to the test project user guide. The tooling used by several test projects will be detailed in this document.

The best event to meet the testing community is probably the plugfest. Such an event is organized after each release. Most of the test projects are present.

The summit is also a good opportunity to meet most of the actors [DEV4].

Be involved in the testing group

The testing group is a self organized working group. The OPNFV projects dealing with testing are invited to participate in order to elaborate and consolidate a consistant test strategy (test case definition, scope of projects, resources for long duration, documentation, ...) and align tooling or best practices.

A weekly meeting is organized, the agenda may be amended by any participant. 2 slots have been defined (US/Europe and APAC). Agendas and minutes are public. See [DEV3] for details. The testing group IRC channel is #opnfv-testperf

Best practices

All the test projects do not have the same maturity and/or number of contributors. The nature of the test projects may be also different. The following best practices may not be acurate for all the projects and are only indicative. Contact the testing group for further details.

Repository structure

Most of the projects have a similar structure, which can be defined as follows:

`-- home
  |-- requirements.txt
  |-- setup.py
  |-- tox.ini
  |
  |-- <project>
  |       |-- <api>
  |       |-- <framework>
  |       `-- <test cases>
  |
  |-- docker
  |     |-- Dockerfile
  |     `-- Dockerfile.aarch64.patch
  |-- <unit tests>
  `- docs
     |-- release
     |   |-- release-notes
     |   `-- results
     `-- testing
         |-- developer
         |     `-- devguide
         |-- user
               `-- userguide
API

Test projects are installing tools and triggering tests. When it is possible it is recommended to implement an API in order to perform the different actions.

Each test project should be able to expose and consume APIs from other test projects. This pseudo micro service approach should allow a flexible use of the different projects and reduce the risk of overlapping. In fact if project A provides an API to deploy a traffic generator, it is better to reuse it rather than implementing a new way to deploy it. This approach has not been implemented yet but the prerequisites consiting in exposing and API has already been done by several test projects.

CLI

Most of the test projects provide a docker as deliverable. Once connected, it is possible to prepare the environement and run tests through a CLI.

Dockerization

Dockerization has been introduced in Brahmaputra and adopted by most of the test projects. Docker containers are pulled on the jumphost of OPNFV POD. <TODO Jose/Mark/Alec>

Code quality

It is recommended to control the quality of the code of the testing projects, and more precisely to implement some verifications before any merge:

  • pep8
  • pylint
  • unit tests (python 2.7)
  • unit tests (python 3.5)

The code of the test project must be covered by unit tests. The coverage shall be reasonable and not decrease when adding new features to the framework. The use of tox is recommended. It is possible to implement strict rules (no decrease of pylint score, unit test coverages) on critical python classes.

Third party tooling

Several test projects integrate third party tooling for code quality check and/or traffic generation. Some of the tools can be listed as follows:

Project Tool Comments
Bottlenecks TODO  
Functest Tempest Rally Refstack RobotFramework OpenStack test tooling OpenStack test tooling OpenStack test tooling Used for ODL tests
QTIP Unixbench RAMSpeed nDPI openSSL inxi  
Storperf TODO  
VSPERF TODO  
Yardstick Moongen Trex Pktgen IxLoad, IxNet SPEC Unixbench RAMSpeed LMBench Iperf3 Netperf Pktgen-DPDK Testpmd L2fwd Fio Bonnie++ Traffic generator Traffic generator Traffic generator Traffic generator Compute Compute Compute Compute Network Network Network Network Network Storage Storage
Testing group configuration parameters
Testing categories

The testing group defined several categories also known as tiers. These categories can be used to group test suites.

Category Description
Healthcheck Simple and quick healthcheck tests case
Smoke Set of smoke test cases/suites to validate the release
Features Test cases that validate a specific feature on top of OPNFV. Those come from Feature projects and need a bit of support for integration
Components Tests on a specific component (e.g. OpenStack, OVS, DPDK,..) It may extend smoke tests
Performance Performance qualification
VNF Test cases related to deploy an open source VNF including an orchestrator
Stress Stress and robustness tests
In Service In service testing
Testing domains

The domains deal with the technical scope of the tests. It shall correspond to domains defined for the certification program:

  • compute
  • network
  • storage
  • hypervisor
  • container
  • vim
  • mano
  • vnf
  • ...
Testing coverage

One of the goals of the testing working group is to identify the poorly covered areas and avoid testing overlap. Ideally based on the declaration of the test cases, through the tags, domains and tier fields, it shall be possible to create heuristic maps.

Reliability, Stress and Long Duration Testing

Resiliency of NFV refers to the ability of the NFV framework to limit disruption and return to normal or at a minimum acceptable service delivery level in the face of a fault, failure, or an event that disrupts the normal operation [DEV5].

Reliability testing evaluates the ability of SUT to recover in face of fault, failure or disrupts in normal operation or simply the ability of SUT absorbing “disruptions”.

Reliability tests use different forms of faults as stimulus, and the test must measure the reaction in terms of the outage time or impairments to transmission.

Stress testing involves producing excess load as stimulus, and the test must measure the reaction in terms of unexpected outages or (more likely) impairments to transmission.

These kinds of “load” will cause “disruption” which could be easily found in system logs. It is the purpose to raise such “load” to evaluate the SUT if it could provide an acceptable level of service or level of confidence during such circumstances. In Danube and Euphrates, we only considered the stress test with excess load over OPNFV Platform.

In Danube, Bottlenecks and Yardstick project jointly implemented 2 stress tests (concurrently create/destroy VM pairs and do ping, system throughput limit) while Bottlenecks acts as the load manager calling yardstick to execute each test iteration. These tests are designed to test for breaking points and provide level of confidence of the system to users. Summary of the test cases are listed in the following addresses:

Stress test cases for OPNFV Euphrates (OS Ocata) release can be seen as extension/enhancement of those in D release. These tests are located in Bottlenecks/Yardstick repo (Bottlenecks as load manager while Yardstick execute each test iteration):

network usage from different VM pairs): https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Baseline+Stress+Test+Case+for+Bottlenecks+E+Release

In OPNFV E release, we also plan to do long duration testing over OS Ocata. A separate CI pipe testing OPNFV XCI (OSA) is proposed to accomplish the job. We have applied specific pod for the testing. Proposals and details are listed below:

The long duration testing is supposed to be started when OPNFV E release is published. A simple monitoring module for these tests is also planned to be added: https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Monitoring+Stress+Testing+for+Bottlenecks+E+Release

How TOs
Where can I find information on the different test projects?

On http://docs.opnfv.org! A section is dedicated to the testing projects. You will find the overview of the ecosystem and the links to the project documents.

Another source is the testing wiki on https://wiki.opnfv.org/display/testing

You may also contact the testing group on the IRC channel #opnfv-testperf or by mail at test-wg AT lists.opnfv.org (testing group) or opnfv-tech-discuss AT lists.opnfv.org (generic technical discussions).

How can I contribute to a test project?

As any project, the best solution is to contact the project. The project members with their email address can be found under https://git.opnfv.org/<project>/tree/INFO

You may also send a mail to the testing mailing list or use the IRC channel #opnfv-testperf

Where can I find hardware resources?

You should discuss this topic with the project you are working with. If you need access to an OPNFV community POD, it is possible to contact the infrastructure group. Depending on your needs (scenario/installer/tooling), it should be possible to find free time slots on one OPNFV community POD from the Pharos federation. Create a JIRA ticket to describe your needs on https://jira.opnfv.org/projects/INFRA. You must already be an OPNFV contributor. See https://wiki.opnfv.org/display/DEV/Developer+Getting+Started.

Please note that lots of projects have their own “how to contribute” or “get started” page on the OPNFV wiki.

How do I integrate my tests in CI?

It shall be discussed directly with the project you are working with. It is done through jenkins jobs calling testing project files but the way to onboard cases differ from one project to another.

How to declare my tests in the test Database?

If you have access to the test API swagger (access granted to contributors), you may use the swagger interface of the test API to declare your project. The URL is http://testresults.opnfv.org/test/swagger/spec.html.

Testing Group Test API swagger

Click on Spec, the list of available methods must be displayed.

Testing Group Test API swagger

For the declaration of a new project use the POST /api/v1/projects method. For the declaration of new test cases in an existing project, use the POST

/api/v1/projects/{project_name}/cases method

Testing group declare new test case
How to push your results into the Test Database?

The test database is used to collect test results. By default it is enabled only for CI tests from Production CI pods.

Please note that it is possible to create your own local database.

A dedicated database is for instance created for each plugfest.

The architecture and associated API is described in previous chapter. If you want to push your results from CI, you just have to call the API at the end of your script.

You can also reuse a python function defined in functest_utils.py [DEV2]

Where can I find the documentation on the test API?

The Test API is now documented in this document (see sections above). You may also find autogenerated documentation in http://artifacts.opnfv.org/releng/docs/testapi.html A web protal is also under construction for certification at http://testresults.opnfv.org/test/#/

I have tests, to which category should I declare them?

See table above.

The main ambiguity could be between features and VNF. In fact sometimes you have to spawn VMs to demonstrate the capabilities of the feature you introduced. We recommend to declare your test in the feature category.

VNF category is really dedicated to test including:

  • creation of resources
  • deployement of an orchestrator/VNFM
  • deployment of the VNF
  • test of the VNFM
  • free resources

The goal is not to study a particular feature on the infrastructure but to have a whole end to end test of a VNF automatically deployed in CI. Moreover VNF are run in weekly jobs (one a week), feature tests are in daily jobs and use to get a scenario score.

Where are the logs of CI runs?

Logs and configuration files can be pushed to artifact server from the CI under http://artifacts.opnfv.org/<project name>

References

[DEV1]: OPNFV Testing Ecosystem

[DEV2]: Python code sample to push results into the Database

[DEV3]: Testing group wiki page

[DEV4]: Conversation with the testing community, OPNFV Beijing Summit

[DEV5]: GS NFV 003

IRC support chan: #opnfv-testperf

Bottlenecks

Bottlenecks Developer Guide
Bottlenecks - A Developer Quick Start
Introduction

This document will provide general view of the project for developers as a quick guide.

Bottlenecks - Framework Guide
Introduction

This document will provide a comprehensive guilde on Bottlenecks testing framework development.

Bottlenecks - Package Guilde
Introduction

This document will provide a comprehensive guilde on packages library for developers.

Functest

Functest Developer Guide
Introduction

Functest is a project dealing with functional testing. The project produces its own internal test cases but can also be considered as a framework to support feature and VNF onboarding project testing.

Therefore there are many ways to contribute to Functest. You can:

  • Develop new internal test cases
  • Integrate the tests from your feature project
  • Develop the framework to ease the integration of external test cases

Additional tasks involving Functest but addressing all the test projects may also be mentioned:

  • The API / Test collection framework
  • The dashboards
  • The automatic reporting portals
  • The testcase catalog

This document describes how, as a developer, you may interact with the Functest project. The first section details the main working areas of the project. The Second part is a list of “How to” to help you to join the Functest family whatever your field of interest is.

Functest developer areas
Functest High level architecture

Functest is a project delivering test containers dedicated to OPNFV. It includes the tools, the scripts and the test scenarios. In Euphrates Alpine containers have been introduced in order to lighten the container and manage testing slicing. The new containers are created according to the different tiers:

Standalone functest dockers are maintained for Euphrates but Alpine containers are recommended.

Functest can be described as follow:

+----------------------+
|                      |
|   +--------------+   |                  +-------------------+
|   |              |   |    Public        |                   |
|   | Tools        |   +------------------+      OPNFV        |
|   | Scripts      |   |                  | System Under Test |
|   | Scenarios    |   |                  |                   |
|   |              |   |                  |                   |
|   +--------------+   |                  +-------------------+
|                      |
|    Functest Docker   |
|                      |
+----------------------+
Functest internal test cases

The internal test cases in Euphrates are:

  • api_check
  • connection_check
  • snaps_health_check
  • vping_ssh
  • vping_userdata
  • odl
  • odl_netvirt
  • rally_full
  • rally_sanity
  • tempest_smoke_serial
  • tempest_full_parallel
  • cloudify_ims

By internal, we mean that this particular test cases have been developed and/or integrated by functest contributors and the associated code is hosted in the Functest repository. An internal case can be fully developed or a simple integration of upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are just integrated in Functest).

The structure of this repository is detailed in [1]. The main internal test cases are in the opnfv_tests subfolder of the repository, the internal test cases can be grouped by domain:

  • sdn: odl, odl_netvirt, odl_fds
  • openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*
  • vnf: cloudify_ims

If you want to create a new test case you will have to create a new folder under the testcases directory (See next section for details).

Functest external test cases

The external test cases are inherited from other OPNFV projects, especially the feature projects.

The external test cases are:

  • barometer
  • bgpvpn
  • doctor
  • domino
  • fds
  • parser
  • promise
  • refstack_defcore
  • snaps_smoke
  • functest-odl-sfc
  • orchestra_clearwaterims
  • orchestra_openims
  • vyos_vrouter
  • juju_vepc

External test cases integrated in previous versions but not released in Euphrates:

  • copper
  • moon
  • netready
  • security_scan

The code to run these test cases is hosted in the repository of the project. Please note that orchestra test cases are hosted in Functest repository and not in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in functest as there are no dedicated projects.

Functest framework

Functest is a framework.

Historically Functest is released as a docker file, including tools, scripts and a CLI to prepare the environment and run tests. It simplifies the integration of external test suites in CI pipeline and provide commodity tools to collect and display results.

Since Colorado, test categories also known as tiers have been created to group similar tests, provide consistent sub-lists and at the end optimize test duration for CI (see How To section).

The definition of the tiers has been agreed by the testing working group.

The tiers are:
  • healthcheck
  • smoke
  • features
  • components
  • vnf
Functest abstraction classes

In order to harmonize test integration, abstraction classes have been introduced:

  • testcase: base for any test case
  • unit: run unit tests as test case
  • feature: abstraction for feature project
  • vnf: abstraction for vnf onboarding

The goal is to unify the way to run tests in Functest.

Feature, unit and vnf_base inherit from testcase:

            +----------------------------------------------------------------+
            |                                                                |
            |                   TestCase                                     |
            |                                                                |
            |                   - init()                                     |
            |                   - run()                                      |
            |                   - push_to_db()                               |
            |                   - is_successful()                            |
            |                                                                |
            +----------------------------------------------------------------+
               |             |                 |                           |
               V             V                 V                           V
+--------------------+   +---------+   +------------------------+   +-----------------+
|                    |   |         |   |                        |   |                 |
|    feature         |   |  unit   |   |    vnf                 |   | robotframework  |
|                    |   |         |   |                        |   |                 |
|                    |   |         |   |- prepare()             |   |                 |
|  - execute()       |   |         |   |- deploy_orchestrator() |   |                 |
| BashFeature class  |   |         |   |- deploy_vnf()          |   |                 |
|                    |   |         |   |- test_vnf()            |   |                 |
|                    |   |         |   |- clean()               |   |                 |
+--------------------+   +---------+   +------------------------+   +-----------------+
Testcase functest.core.testcase module — OPNFV Functest master documentation

functest.core.testcase module

Define the parent class of all Functest TestCases.

class functest.core.testcase.OSGCTestCase(**kwargs)

Bases: functest.core.testcase.TestCase

Model for single test case which requires an OpenStack Garbage Collector.

clean()

Clean the OpenStack resources.

create_snapshot()

Create a snapshot listing the OpenStack resources.

Returns:
TestCase.EX_OK if os_snapshot.main() returns 0. TestCase.EX_RUN_ERROR otherwise.
class functest.core.testcase.TestCase(**kwargs)

Bases: object

Base model for single test case.

EX_OK = 0

everything is OK

EX_PUSH_TO_DB_ERROR = 69

push_to_db() failed

EX_RUN_ERROR = 70

run() failed

EX_TESTCASE_FAILED = 68

results are false

clean()

Clean the resources.

It can be overriden if resources must be deleted after running the test case.

create_snapshot()

Save the testing environment before running test.

It can be overriden if resources must be listed running the test case.

Returns:
TestCase.EX_OK
get_duration()

Return the duration of the test case.

Returns:
duration if start_time and stop_time are set “XX:XX” otherwise.
is_successful()

Interpret the result of the test case.

It allows getting the result of TestCase. It completes run() which only returns the execution status.

It can be overriden if checking result is not suitable.

Returns:
TestCase.EX_OK if result is ‘PASS’. TestCase.EX_TESTCASE_FAILED otherwise.
push_to_db()

Push the results of the test case to the DB.

It allows publishing the results and to check the status.

It could be overriden if the common implementation is not suitable. The following attributes must be set before pushing the results to DB:

  • project_name,
  • case_name,
  • result,
  • start_time,
  • stop_time.
Returns:
TestCase.EX_OK if results were pushed to DB. TestCase.EX_PUSH_TO_DB_ERROR otherwise.
run(**kwargs)

Run the test case.

It allows running TestCase and getting its execution status.

The subclasses must override the default implementation which is false on purpose.

The new implementation must set the following attributes to push the results to DB:

  • result,
  • start_time,
  • stop_time.
Args:
kwargs: Arbitrary keyword arguments.
Returns:
TestCase.EX_RUN_ERROR.
Feature functest.core.feature module — OPNFV Functest master documentation

functest.core.feature module

Define the parent classes of all Functest Features.

Feature is considered as TestCase offered by Third-party. It offers helpers to run any python method or any bash command.

class functest.core.feature.BashFeature(**kwargs)

Bases: functest.core.feature.Feature

Class designed to run any bash command.

execute(**kwargs)

Execute the cmd passed as arg

Args:
kwargs: Arbitrary keyword arguments.
Returns:
0 if cmd returns 0, -1 otherwise.
class functest.core.feature.Feature(**kwargs)

Bases: functest.core.testcase.TestCase

Base model for single feature.

execute(**kwargs)

Execute the Python method.

The subclasses must override the default implementation which is false on purpose.

The new implementation must return 0 if success or anything else if failure.

Args:
kwargs: Arbitrary keyword arguments.
Returns:
-1.
run(**kwargs)

Run the feature.

It allows executing any Python method by calling execute().

It sets the following attributes required to push the results to DB:

  • result,
  • start_time,
  • stop_time.

It doesn’t fulfill details when pushing the results to the DB.

Args:
kwargs: Arbitrary keyword arguments.
Returns:
TestCase.EX_OK if execute() returns 0, TestCase.EX_RUN_ERROR otherwise.
Unit functest.core.unit module — OPNFV Functest master documentation

functest.core.unit module

Define the parent class to run unittest.TestSuite as TestCase.

class functest.core.unit.Suite(**kwargs)

Bases: functest.core.testcase.TestCase

Base model for running unittest.TestSuite.

run(**kwargs)

Run the test suite.

It allows running any unittest.TestSuite and getting its execution status.

By default, it runs the suite defined as instance attribute. It can be overriden by passing name as arg. It must conform with TestLoader.loadTestsFromName().

It sets the following attributes required to push the results to DB:

  • result,
  • start_time,
  • stop_time,
  • details.
Args:
kwargs: Arbitrary keyword arguments.
Returns:
TestCase.EX_OK if any TestSuite has been run, TestCase.EX_RUN_ERROR otherwise.
VNF functest.core.vnf module — OPNFV Functest master documentation

functest.core.vnf module

Define the parent class of all VNF TestCases.

exception functest.core.vnf.OrchestratorDeploymentException

Bases: exceptions.Exception

Raise when orchestrator cannot be deployed.

exception functest.core.vnf.VnfDeploymentException

Bases: exceptions.Exception

Raise when VNF cannot be deployed.

class functest.core.vnf.VnfOnBoarding(**kwargs)

Bases: functest.core.testcase.TestCase

Base model for VNF test cases.

clean()

Clean VNF test case.

It is up to the test providers to delete resources used for the tests. By default we clean:

  • the user,
  • the tenant
deploy_orchestrator()

Deploy an orchestrator (optional).

If this method is overriden then raise orchestratorDeploymentException if error during orchestrator deployment

deploy_vnf()

Deploy the VNF

This function MUST be implemented by vnf test cases. The details section MAY be updated in the vnf test cases.

The deployment can be executed via a specific orchestrator or using build-in orchestrators such as heat, OpenBaton, cloudify, juju, onap, …

Returns:
True if the VNF is properly deployed False if the VNF is not deployed

Raise VnfDeploymentException if error during VNF deployment

prepare()

Prepare the environment for VNF testing:

  • Creation of a user,
  • Creation of a tenant,
  • Allocation admin role to the user on this tenant

Returns base.TestCase.EX_OK if preparation is successfull

Raise VnfPreparationException in case of problem

run(**kwargs)

Run of the VNF test case:

  • Deploy an orchestrator if needed (e.g. heat, cloudify, ONAP,…),
  • Deploy the VNF,
  • Perform tests on the VNF

A VNF test case is successfull when the 3 steps are PASS If one of the step is FAIL, the test case is FAIL

Returns:
TestCase.EX_OK if result is ‘PASS’. TestCase.EX_TESTCASE_FAILED otherwise.
test_vnf()

Test the VNF

This function MUST be implemented by vnf test cases. The details section MAY be updated in the vnf test cases.

Once a VNF is deployed, it is assumed that specific test suite can be run to validate the VNF. Please note that the same test suite can be used on several test case (e.g. clearwater test suite can be used whatever the orchestrator used for the deployment)

Returns:
True if VNF tests are PASS False if test suite is FAIL

Raise VnfTestException if error during VNF test

exception functest.core.vnf.VnfPreparationException

Bases: exceptions.Exception

Raise when VNF preparation cannot be executed.

exception functest.core.vnf.VnfTestException

Bases: exceptions.Exception

Raise when VNF cannot be tested.

Robotframework functest.core.robotframework module — OPNFV Functest master documentation

functest.core.robotframework module

Define classes required to run any Robot suites.

class functest.core.robotframework.ResultVisitor

Bases: robot.result.visitor.ResultVisitor

Visitor to get result details.

get_data()

Get the details of the result.

visit_test(test)
class functest.core.robotframework.RobotFramework(**kwargs)

Bases: functest.core.testcase.TestCase

RobotFramework runner.

parse_results()

Parse output.xml and get the details in it.

run(**kwargs)

Run the RobotFramework suites

Here are the steps:
  • create the output directories if required,
  • get the results in output.xml,
  • delete temporary files.
Args:
kwargs: Arbitrary keyword arguments.
Returns:
EX_OK if all suites ran well. EX_RUN_ERROR otherwise.

see [5] to get code samples

Functest util classes

In order to simplify the creation of test cases, Functest develops also some functions that are used by internal test cases. Several features are supported such as logger, configuration management and Openstack capabilities (snapshot, clean, tacker,..). These functions can be found under <repo>/functest/utils and can be described as follows:

functest/utils/
|-- config.py
|-- constants.py
|-- decorators.py
|-- env.py
|-- functest_utils.py
|-- openstack_clean.py
|-- openstack_snapshot.py
|-- openstack_tacker.py
`-- openstack_utils.py

It is recommended to use the SNAPS-OO library for deploying OpenStack instances. SNAPS [4] is an OPNFV project providing OpenStack utils.

TestAPI

Functest is using the Test collection framework and the TestAPI developed by the OPNFV community. See [6] for details.

Reporting

A web page is automatically generated every day to display the status based on jinja2 templates [3].

Dashboard

Additional dashboarding is managed at the testing group level, see [7] for details.

How TOs

See How to section on Functest wiki [8]

QTIP

QTIP Developer Guide
Overview

QTIP uses Python as primary programming language and build the framework from the following packages

Module Package
api Connexion - API first applications with OpenAPI/Swagger and Flask
cli Click - the “Command Line Interface Creation Kit”
template Jinja2 - a full featured template engine for Python
docs sphinx - a tool that makes it easy to create intelligent and beautiful documentation
testing pytest - a mature full-featured Python testing tool that helps you write better programs
Source Code

The structure of repository is based on the recommended sample in The Hitchhiker’s Guide to Python

Path Content
./benchmarks/ builtin benchmark assets including plan, QPI and metrics
./contrib/ independent project/plugin/code contributed to QTIP
./docker/ configuration for building Docker image for QTIP deployment
./docs/ release notes, user and developer documentation, design proposals
./legacy/ legacy obsoleted code that is unmaintained but kept for reference
./opt/ optional component, e.g. scripts to setup infrastructure services for QTIP
./qtip/ the actual package
./tests/ package functional and unit tests
./third-party/ third part included in QTIP project
Coding Style

QTIP follows OpenStack Style Guidelines for source code and commit message.

Specially, it is recommended to link each patch set with a JIRA issue. Put:

JIRA: QTIP-n

in commit message to create an automatic link.

Testing

All testing related code are stored in ./tests/

Path Content
./tests/data/ data fixtures for testing
./tests/unit/ unit test for each module, follow the same layout as ./qtip/
./conftest.py pytest configuration in project scope

tox is used to automate the testing tasks

cd <project_root>
pip install tox
tox

The test cases are written in pytest. You may run it selectively with

pytest tests/unit/reporter
Branching

Stable branches are created when features are frozen for next release. According to OPNFV release milestone description, stable branch window is open on MS6 and closed on MS7.

  1. Contact gerrit admin <opnfv-helpdesk@rt.linuxfoundation.org> to create branch for project.
  2. Setup qtip jobs and docker jobs for stable branch in releng
  3. Follow instructions for stable branch.

NOTE: we do NOT create branches for feature development as in the popular GitHub Flow

Releasing

Tag Deliverable and write release note

Git repository

Follow the example in Git Tagging Instructions for Danube to tag the source code:

git fetch gerrit
git checkout stable/<release-name>
git tag -am "<release-version>" <release-version>
git push gerrit <release-version>
Docker image
  1. Login OPNFV Jenkins
  2. Go to the `qtip-docker-build-push-<release>`_ and click “Build With Parameters”
  3. Fill in RELEASE_VERSION with version number not including release name, e.g. 1.0
  4. Trigger a manual build
Python Package

QTIP is also available as a Python Package. It is hosted on the Python Package Index(PyPI).

  1. Install twine with pip install twine
  2. Build the distributions python setup.py  sdist bdist_wheel
  3. Upload the distributions built with twine upload dist/*

NOTE: only package maintainers are permitted to upload the package versions.

Release note

Create release note under qtip/docs/release/release-notes and update index.rst

Run with Ansible

QTIP benchmarking tasks are built upon Ansible playbooks and roles. If you are familiar with Ansible, it is possible to run it with ansible-playbook command. And it is useful during development of ansible modules or testing roles.

Create workspace

There is a playbook in resources/ansible_roles/qtip-workspace used for creating a new workspace:

cd resources/ansible_roles/qtip-workspace
ansible-playbook create.yml

NOTE: if this playbook is moved to other directory, configuration in ansible.cfg needs to be updated accordingly. The ansible roles from QTIP, i.e. <path_of_qtip>/resources/ansible_roles must be added to roles_path in Ansible configuration file. For example:

roles_path = ~/qtip/resources/ansible_roles
Executing benchmark

Before executing the setup playbook, make sure ~/.ssh/config has been configured properly so that you can login the master node “directly”. Skip next section, if you can login with ssh <master-host> from localhost,

SSH access to master node

It is common that the master node is behind some jump host. In this case, ssh option ProxyCommand and ssh-agent shall be required.

Assume that you need to login to deploy server, then login to the master node from there. An example configuration is as following:

Host fuel-deploy
  HostName 172.50.0.250
  User root

Host fuel-master
  HostName 192.168.122.63
  User root
  ProxyCommand ssh -o 'ForwardAgent yes' apex-deploy 'ssh-add && nc %h %p'

If several jumps are required to reach the master node, we may chain the jump hosts like below:

Host jumphost
  HostName 10.62.105.31
  User zte
  Port 22

Host fuel-deploy
  HostName 172.50.0.250
  User root
  ProxyJump jumphost

Host fuel-master
  HostName 192.168.122.63
  User root
  ProxyCommand ssh -o 'ForwardAgent yes' apex-deploy 'ssh-add && nc %h %p'

NOTE: ProxyJump is equivalent to the long ProxyCommand option, but it is only available since OpenSSH 7.3

Automatic setup
  1. Modify <workspace>/group_vars/all.yml to set installer information correctly
  2. Modify <workspace>/hosts file to set installer master host correctly

#. Run the setup playbook to generate ansible inventory of system under test by querying the slave nodes from the installer master:

cd workspace
ansible-playbook setup.yml

It will update the hosts and ssh.cfg

Currently, QTIP supports automatic discovery from apex and fuel.

Manual setup

If your installer is not supported or you are testing hosts not managed by installer, you may add them manually in [compute] group in <workspace>/hosts:

[compute:vars]
ansible_ssh_common_args=-F ./ssh.cfg

[compute]
node-2
node-4
node-6
node-7

And ssh.cfg for ssh connection configuration:

Host node-5
  HostName 10.20.5.12
  User root
Run the tests

Run the benchmarks with the following command:

ansible-playbook run.yml

CAVEAT: QTIP will install required packages in system under test.

Inspect the results

The test results and calculated output are stored in results:

current/
    node-2/
        arithmetic/
            metric.json
            report
            unixbench.log
        dpi/
        ...
    node-4/
    ...
    qtip-pod-qpi.json
qtip-pod-20170425-1710/
qtip-pod-20170425-1914/
...

The folders are named as <pod_name>-<start_time>/ and the results are organized by hosts under test. Inside each host, the test data are organized by metrics as defined in QPI specification.

For each metrics, it usually includes the following content

  • log file generated by the performance testing tool
  • metrics collected from the log files
  • reported rendered with the metrics collected
Teardown the test environment

QTIP will create temporary files for testing in system under test. Execute the teardown playbook to clean it up:

ansible-playbook teardown.yml
Architecture

In Danube, QTIP releases its standalone mode, which is also know as solo:

QTIP standalone mode

The runner could be launched from CLI (command line interpreter) or API (application programming interface) and drives the testing jobs. The generated data including raw performance data and testing environment are fed to collector. Performance metrics will be parsed from the raw data and used for QPI calculation. Then the benchmark report is rendered with the benchmarking results.

The execution can be detailed in the diagram below:

QTIP execution sequence
Framework

QTIP is built upon Ansible by extending modules, playbook roles and plugins.

Modules

QTIP creates dedicated modules to gather slave node list and information from installer master. See embedded document in qtip/ansible_library/modules for details

Plugins

Stored in qtip/ansible_library/plugins

Action plugins

Several action plugins have been created for test data post processing

  • collect - parse and collect metrics from raw test results like log files
  • calculate - calculate score according to specification
  • aggregate - aggregate calculated results from all hosts under test
Playbook roles
QTIP roles
  • qtip - main qtip tasks
  • qtip-common - common tasks required in QTIP
  • qtip-workspace - generate a workspace for running benchmarks

qtip roles should be included with a specified action and output directory, e.g.:

- { role: inxi, output: "{{ qtip_results }}/sysinfo", tags: [run, inxi, sysinfo] }
testing roles

Testing roles are organized by testing tools

  • inxi - system information tool
  • nDPI
  • openssl
  • ramspeed
  • unixbench

supporting roles

  • opnfv-testapi - report result to testapi
Tags

Tags are used to categorize the test tasks from different aspects.

  • stages like run, collect, calculate, aggregate, report
  • test tools like inxi, ndpi and etc
  • information or metrics like sysinfo, dpi, ssl

Use

  • ansible-playbook run.yml --list-tags to list all tags
  • ansible-playbook run.yml --list-tasks to list all tasks

During development of post processing, you may skip run stage to save time, e.g. ansible-playbook run.yml --tags collect,calculate,aggregate

CLI - Command Line Interface

QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI’s) such as compute, storage and network. A plan consists of one or more QPI’s, depending upon how the end user would want to measure performance. CLI is designed to help the user, execute benchmarks and view respective scores.

Framework

QTIP CLI has been created using the Python package Click, Command Line Interface Creation Kit. It has been chosen for number of reasons. It presents the user with a very simple yet powerful API to build complex applications. One of the most striking features is command nesting.

As explained, QTIP consists of metrics, QPI’s and plans. CLI is designed to provide interface to all these components. It is responsible for execution, as well as provide listing and details of each individual element making up these components.

Design

CLI’s entry point extends Click’s built in MultiCommand class object. It provides two methods, which are overridden to provide custom configurations.

class QtipCli(click.MultiCommand):

    def list_commands(self, ctx):
        rv = []
        for filename in os.listdir(cmd_folder):
            if filename.endswith('.py') and \
                filename.startswith('cmd_'):
                rv.append(filename[4:-3])
        rv.sort()
        return rv

    def get_command(self, ctx, name):
        try:
            if sys.version_info[0] == 2:
                name = name.encode('ascii', 'replace')
            mod = __import__('qtip.cli.commands.cmd_' + name,
                             None, None, ['cli'])
        except ImportError:
            return
        return mod.cli

Commands and subcommands will then be loaded by the get_command method above.

Extending the Framework

Framework can be easily extended, as per the users requirements. One such example can be to override the builtin configurations with user defined ones. These can be written in a file, loaded via a Click Context and passed through to all the commands.

class Context:

    def __init__():

        self.config = ConfigParser.ConfigParser()
        self.config.read('path/to/configuration_file')

    def get_paths():

        paths = self.config.get('section', 'path')
        return paths

The above example loads configuration from user defined paths, which then need to be provided to the actual command definitions.

from qtip.cli.entry import Context

pass_context = click.make_pass_decorator(Context, ensure=False)

@cli.command('list', help='List the Plans')
@pass_context
def list(ctx):
    plans = Plan.list_all(ctx.paths())
    table = utils.table('Plans', plans)
    click.echo(table)
API - Application Programming Interface

QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI’s) such as compute, storage and network. A plan consists of one or more QPI’s, depending upon how the end-user would want to measure performance. API is designed to expose a RESTful interface to the user for executing benchmarks and viewing respective scores.

Framework

QTIP API has been created using the Python package Connexion. It has been chosen for a number of reasons. It follows API First approach to create micro-services. Hence, firstly the API specifications are defined from the client side perspective, followed by the implementation of the micro-service. It decouples the business logic from routing and resource mapping making design and implementation cleaner.

It has two major components:

API Specifications

The API specification is defined in a yaml or json file. Connexion follows Open API specification to determine the design and maps the endpoints to methods in python.
Micro-service Implementation
Connexion maps the operationId corresponding to every operation in API Specification to methods in python which handles request and responses.

As explained, QTIP consists of metrics, QPI’s and plans. The API is designed to provide a RESTful interface to all these components. It is responsible to provide listing and details of each individual element making up these components.

Design
Specification

API’s entry point (main) runs connexion App class object after adding API Specification using App.add_api method. It loads specification from swagger.yaml file by specifying specification_dir.

Connexion reads API’s endpoints(paths), operations, their request and response parameter details and response definitions from the API specification i.e. swagger.yaml in this case.

Following example demonstrates specification for the resource plans.

paths:
  /plans/{name}:
    get:
      summary: Get a plan by plan name
      operationId: qtip.api.controllers.plan.get_plan
      tags:
        - Plan
        - Standalone
      parameters:
        - name: name
          in: path
          description: Plan name
          required: true
          type: string
      responses:
        200:
          description: Plan information
          schema:
            $ref: '#/definitions/Plan'
        404:
          description: Plan not found
          schema:
            $ref: '#/definitions/Error'
        501:
          description: Resource not implemented
          schema:
            $ref: '#/definitions/Error'
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'
definitions:
  Plan:
    type: object
    required:
      - name
    properties:
      name:
        type: string
      description:
        type: string
      info:
        type: object
      config:
        type: object

Every operationId in above operations corresponds to a method in controllers. QTIP has three controller modules each for plan, QPI and metric. Connexion will read these mappings and automatically route endpoints to business logic.

Swagger Editor can be explored to play with more such examples and to validate the specification.

Controllers

The request is handled through these methods and response is sent back to the client. Connexion takes care of data validation.

@common.check_endpoint_for_error(resource='Plan')
def get_plan(name):
    plan_spec = plan.Plan(name)
    return plan_spec.content

In above code get_plan takes a plan name and return its content.

The decorator check_endpoint_for_error defined in common is used to handle error and return a suitable error response.

During Development the server can be run by passing specification file(swagger.yaml in this case) to connexion cli -

connexion run <path_to_specification_file> -v
Extending the Framework
Modifying Existing API:

API can be modified by adding entries in swagger.yaml and adding the corresponding controller mapped from operationID.

Adding endpoints:

New endpoints can be defined in paths section in swagger.yaml. To add a new resource dummy -

paths:
  /dummies:
    get:
      summary: Get all dummies
      operationId: qtip.api.controllers.dummy.get_dummies
      tags:
        - dummy
      responses:
        200:
          description: Foo information
          schema:
            $ref: '#/definitions/Dummy
        default:
          description: Unexpected error
          schema:
            $ref: '#/definitions/Error'

And then model of the resource can be defined in the definitions section.

definitions:
  Dummy:
    type: object
    required:
      - name
    properties:
      name:
        type: string
      description:
        type: string
      id:
        type: string
Adding controller methods:

Methods for handling requests and responses for every operation for the endpoint added can be implemented in controller.

In controllers.dummy

def get_dummies():
    all_dummies = [<code to get all dummies>]
    return all_dummies, httplib.OK
Adding error responses

Decorators for handling errors are defined in common.py in api.

from qtip.api import common

@common.check_endpoint_for_error(resource='dummy',operation='get')
def get_dummies()
    all_dummies = [<code to get all dummies>]
    return all_dummies
Adding new API:

API can easily be extended by adding more APIs to Connexion.App class object using add_api class method.

In __main__

def get_app():
app = connexion.App(__name__, specification_dir=swagger_dir)
app.add_api('swagger.yaml', base_path='/v1.0', strict_validation=True)
return app

Extending it to add new APIs. The new API should have all endpoints mapped using operationId.

from qtip.api import __main__
my_app = __main__.get_app()
my_app.add_api('new_api.yaml',base_path'api2',strict_validation=True)
my_app.run(host="0.0.0.0", port=5000)
Web Portal for Benchmarking Services

QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI’s) such as compute, storage and network. QTIP benchmarking tasks are built upon Ansible playbooks and roles. QTIP web portal is a platform to expose QTIP as a benchmarking service hosted on a central host.

Framework

The web travel has been developed on Python Django framework. Dig into the documentation to learn about Django.

Design

Django is a MTV (Model Template View) framework. Database objects are mapped to models in models.py. Views handle the requests from client side and interact with database using Django ORM. Templates are responsible for UI rendering based on response context from Views.

Models
Repo

Model for workspace repos

Repo:
    name
    git_link
Task

Tasks keep track of every benchmark run through QTIP-Web Services. Whenever you run a benchmark, a new task is created which keep track of time stats and log task progress and ansible output for the respective playbook.

Task
    start_time
    end_time
    status
    run_time
    repo
    log
Views
Dashboard
  • Base class - TemplateVIew

Class based view serving as home page for the application.

ReposView
  • Base class - LoginRequiredMixin, CreateView

Class based view for listing and add new repos

RepoUpdate
  • Base class - LoginRequiredMixin, UpdateView

Class based View for listing and updating an existing repo details.

Both ReposView and RepoUpdate View use same template ``repo_form.html``. The context has an extra variable ``template_role`` which is used to distinguish if repo form is for create or edit operation.

Run
  • Base class - LoginRequiredMixin, View
  • template name - run.html

Class based View for adding new task and run benchmark based on task details. The logs are saved in logs/run_<log_id> directory.

Compute QPI

The compute QPI gives user an overall score for system compute performace.

Summary

The compute QPI are calibrated a ZTE E9000 server as a baseline with score of 2500 points. Higher scores are better, with double the score indicating double the performance. The compute QPI provides three different kinds of scores:

  • Workload Scores
  • Section Scores
  • Compute QPI Scores
Baseline

ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory.

Workload Scores

Each time a workload is executed QTIP calculates a score based on the computer’s performance compared to the baseline performance.

Section Scores

QTIP uses a number of different tests, or workloads, to measure performance. The workloads are divided into five different sections:

Section Detail Indication
Arithmetic Arithmetic workloads measure integer operations floating point operations and mathematical functions with whetstone and dhrystone instructions. Software with heavy calculation tasks.
Memory Memory workloads measure memory transfer performance with RamSpeed test. Software working with large scale data operation.
DPI DPI workloads measure deep-packet inspection speed by performing nDPI test. Software working with network packet analysis relies on DPI performance.
SSL SSL Performance workloads measure cipher speeds by using the OpenSSL tool. Software working with cipher large amounts data relies on SSL Performance.

A section score is the geometric mean of all the workload scores for workloads that are part of the section. These scores are useful for determining the performance of the computer in a particular area.

Compute QPI Scores

The compute QPI score is the weighted arithmetic mean of the five section scores. The compute QPI score provides a way to quickly compare performance across different computers and different platforms without getting bogged down in details.

Storage QPI

The storage QPI gives user an overall score for storage performance.

The measurement is done by StorPerf.

System Information

System Information are environmental parameters and factors may affect storage performance:

System Factors Detail Extraction Method
Ceph Node List List of nodes which has ceph-osd roles. For example [node-2, node-3, node-4]. Getting from return result of installer node list CLI command.
Ceph Client RDB Cache Mode Values: “None”, “write-through”, “write-back”. Getting from value of “rbd cache” and “rbd cache max dirty” keys in client section of ceph configuration; To enable write-through mode, set rbd cache max dirty to 0.
Ceph Client RDB Cache Size The RBD cache size in bytes. Default is 32 MiB. Getting from value of “rdb cache size” key in client section of ceph configuration.
Ceph OSD Tier Cache Mode Values: “None”, “Write-back”, “Readonly”. Getting from ceph CLI “ceph report” output info.
Use SSD Backed OSD Cache Values: “Yes”, “No”. Getting from POD description and CEPH CLI “ceph-disk list” output info.
Use SSD For Journal Values: “Yes”, “No”. Getting from POD description and CEPH CLI “ceph-disk list” output info.
Ceph Cluster Network Bandwidth Values: “1G”, “10G”, “40G”. Getting from physical interface information in POD description, “ifconfig” output info on ceph osd node, and value of “cluster network” key in global section of ceph configuration.
Test Condition
Test Condition Detail Extraction Method
Number of Testing VMs Number of VMs which are created, during running Storperf test case. It equals the number of Cinder nodes of the SUT.
Distribution of Testing VMS Number of VMs on each computer node, for example [(node-2: 1), (node-3: 2))]. Recording the distribution when runing Storperf test case.
Baseline

Baseline is established by testing with a set of work loads:

  • Queue depth (1, 2, 8)
  • Block size (2KB, 8KB, 16KB)
  • Read write - sequential read - sequential write - random read - random write - random mixed read write 70/30
Metrics
  • Throughput: data transfer rate
  • IOPS: I/O operations per second
  • Latency: response time
Workload Scores

For each test run, if an equivalent work load in baseline is available, a score will be calculated by comparing the result to baseline.

Section Scores
Section Detail Indication
IOPS Read write I/O Operation per second under steady state Workloads : random read/write Important for frequent storage access such as event sinks
Throughput Read write data transfer rate under steady state Workloads: sequential read/write, block size 16KB Important for high throughput services such as video server
Latency Average response latency under steady state Workloads: all Important for real time applications

Section score is the geometric mean of all workload score.

Storage QPI

Storage QPI is the weighted arithmetic mean of all section scores.

VSPERF

OPNFV VSPERF Developer Guide
Introduction

VSPERF is an OPNFV testing project.

VSPERF provides an automated test-framework and comprehensive test suite based on Industry Test Specifications for measuring NFVI data-plane performance. The data-path includes switching technologies with physical and virtual network interfaces. The VSPERF architecture is switch and traffic generator agnostic and test cases can be easily customized. VSPERF was designed to be independent of OpenStack therefore OPNFV installer scenarios are not required. VSPERF can source, configure and deploy the device-under-test using specified software versions and network topology. VSPERF is used as a development tool for optimizing switching technologies, qualification of packet processing functions and for evaluation of data-path performance.

The Euphrates release adds new features and improvements that will help advance high performance packet processing on Telco NFV platforms. This includes new test cases, flexibility in customizing test-cases, new results display options, improved tool resiliency, additional traffic generator support and VPP support.

VSPERF provides a framework where the entire NFV Industry can learn about NFVI data-plane performance and try-out new techniques together. A new IETF benchmarking specification (RFC8204) is based on VSPERF work contributed since 2015. VSPERF is also contributing to development of ETSI NFV test specifications through the Test and Open Source Working Group.

Design Guides
1. Traffic Generator Integration Guide
1.1. Intended Audience

This document is intended to aid those who want to integrate new traffic generator into the vsperf code. It is expected, that reader has already read generic part of VSPERF Design Document.

Let us create a sample traffic generator called sample_tg, step by step.

1.2. Step 1 - create a directory

Implementation of trafficgens is located at tools/pkt_gen/ directory, where every implementation has its dedicated sub-directory. It is required to create a new directory for new traffic generator implementations.

E.g.

$ mkdir tools/pkt_gen/sample_tg
1.3. Step 2 - create a trafficgen module

Every trafficgen class must inherit from generic ITrafficGenerator interface class. VSPERF during its initialization scans content of pkt_gen directory for all python modules, that inherit from ITrafficGenerator. These modules are automatically added into the list of supported traffic generators.

Example:

Let us create a draft of tools/pkt_gen/sample_tg/sample_tg.py module.

from tools.pkt_gen import trafficgen

class SampleTG(trafficgen.ITrafficGenerator):
    """
    A sample traffic generator implementation
    """
    pass

VSPERF is immediately aware of the new class:

$ ./vsperf --list-trafficgen

Output should look like:

Classes derived from: ITrafficGenerator
======

* Dummy:            A dummy traffic generator whose data is generated by the user.

* IxNet:            A wrapper around IXIA IxNetwork applications.

* Ixia:             A wrapper around the IXIA traffic generator.

* Moongen:          Moongen Traffic generator wrapper.

* TestCenter:       Spirent TestCenter

* Trex:             Trex Traffic generator wrapper.

* Xena:             Xena Traffic generator wrapper class
1.4. Step 3 - configuration

All configuration values, required for correct traffic generator function, are passed from VSPERF to the traffic generator in a dictionary. Default values shared among all traffic generators are defined in conf/03_traffic.conf within TRAFFIC dictionary. Default values are loaded by ITrafficGenerator interface class automatically, so it is not needed to load them explicitly. In case that there are any traffic generator specific default values, then they should be set within class specific __init__ function.

VSPERF passes test specific configuration within traffic dictionary to every start and send function. So implementation of these functions must ensure, that default values are updated with the testcase specific values. Proper merge of values is assured by call of merge_spec function from conf module.

Example of merge_spec usage in tools/pkt_gen/sample_tg/sample_tg.py module:

from conf import merge_spec

def start_rfc2544_throughput(self, traffic=None, duration=30):
    self._params = {}
    self._params['traffic'] = self.traffic_defaults.copy()
    if traffic:
        self._params['traffic'] = merge_spec(
            self._params['traffic'], traffic)
1.5. Step 4 - generic functions

There are some generic functions, which every traffic generator should provide. Although these functions are mainly optional, at least empty implementation must be provided. This is required, so that developer is explicitly aware of these functions.

The connect function is called from the traffic generator controller from its __enter__ method. This function should assure proper connection initialization between DUT and traffic generator. In case, that such implementation is not needed, empty implementation is required.

The disconnect function should perform clean up of any connection specific actions called from the connect function.

Example in tools/pkt_gen/sample_tg/sample_tg.py module:

def connect(self):
    pass

def disconnect(self):
    pass
1.6. Step 5 - supported traffic types

Currently VSPERF supports three different types of tests for traffic generators, these are identified in vsperf through the traffic type, which include:

  • RFC2544 throughput - Send fixed size packets at different rates, using

    traffic configuration, until minimum rate at which no packet loss is detected is found. Methods with its implementation have suffix _rfc2544_throughput.

  • RFC2544 back2back - Send fixed size packets at a fixed rate, using traffic

    configuration, for specified time interval. Methods with its implementation have suffix _rfc2544_back2back.

  • continuous flow - Send fixed size packets at given framerate, using traffic

    configuration, for specified time interval. Methods with its implementation have suffix _cont_traffic.

In general, both synchronous and asynchronous interfaces must be implemented for each traffic type. Synchronous functions start with prefix send_. Asynchronous with prefixes start_ and wait_ in case of throughput and back2back and start_ and stop_ in case of continuous traffic type.

Example of synchronous interfaces:

def send_rfc2544_throughput(self, traffic=None, tests=1, duration=20,
                            lossrate=0.0):
def send_rfc2544_back2back(self, traffic=None, tests=1, duration=20,
                           lossrate=0.0):
def send_cont_traffic(self, traffic=None, duration=20):

Example of asynchronous interfaces:

def start_rfc2544_throughput(self, traffic=None, tests=1, duration=20,
                             lossrate=0.0):
def wait_rfc2544_throughput(self):

def start_rfc2544_back2back(self, traffic=None, tests=1, duration=20,
                            lossrate=0.0):
def wait_rfc2544_back2back(self):

def start_cont_traffic(self, traffic=None, duration=20):
def stop_cont_traffic(self):

Description of parameters used by send, start, wait and stop functions:

  • param traffic: A dictionary with detailed definition of traffic pattern. It contains following parameters to be implemented by traffic generator.

    Note: Traffic dictionary has also virtual switch related parameters, which are not listed below.

    Note: There are parameters specific to testing of tunnelling protocols, which are discussed in detail at Integration tests userguide.

    • param traffic_type: One of the supported traffic types, e.g. rfc2544_throughput, rfc2544_continuous or rfc2544_back2back.
    • param frame_rate: Defines desired percentage of frame rate used during continuous stream tests.
    • param bidir: Specifies if generated traffic will be full-duplex (true) or half-duplex (false).
    • param multistream: Defines number of flows simulated by traffic generator. Value 0 disables MultiStream feature.
    • param stream_type: Stream Type defines ISO OSI network layer used for simulation of multiple streams. Supported values:
      • L2 - iteration of destination MAC address
      • L3 - iteration of destination IP address
      • L4 - iteration of destination port of selected transport protocol
    • param l2: A dictionary with data link layer details, e.g. srcmac, dstmac and framesize.
    • param l3: A dictionary with network layer details, e.g. srcip, dstip, proto and l3 on/off switch enabled.
    • param l4: A dictionary with transport layer details, e.g. srcport, dstport and l4 on/off switch enabled.
    • param vlan: A dictionary with vlan specific parameters, e.g. priority, cfi, id and vlan on/off switch enabled.
  • param tests: Number of times the test is executed.

  • param duration: Duration of continuous test or per iteration duration in case of RFC2544 throughput or back2back traffic types.

  • param lossrate: Acceptable lossrate percentage.

1.7. Step 6 - passing back results

It is expected that methods send, wait and stop will return values measured by traffic generator within a dictionary. Dictionary keys are defined in ResultsConstants implemented in core/results/results_constants.py. Please check sections for RFC2544 Throughput & Continuous and for Back2Back. The same key names should be used by all traffic generator implementations.

2. VSPERF Design Document
2.1. Intended Audience

This document is intended to aid those who want to modify the vsperf code. Or to extend it - for example to add support for new traffic generators, deployment scenarios and so on.

2.2. Usage
2.2.1. Example Connectivity to DUT

Establish connectivity to the VSPERF DUT Linux host. If this is in an OPNFV lab following the steps provided by Pharos to access the POD

The followign steps establish the VSPERF environment.

2.2.2. Example Command Lines

List all the cli options:

$ ./vsperf -h

Run all tests that have tput in their name - phy2phy_tput, pvp_tput etc.:

$ ./vsperf --tests 'tput'

As above but override default configuration with settings in ‘10_custom.conf’. This is useful as modifying configuration directly in the configuration files in conf/NN_*.py shows up as changes under git source control:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests 'tput'

Override specific test parameters. Useful for shortening the duration of tests for development purposes:

$ ./vsperf --test-params 'TRAFFICGEN_DURATION=10;TRAFFICGEN_RFC2544_TESTS=1;' \
                         'TRAFFICGEN_PKT_SIZES=(64,)' pvp_tput
2.3. Typical Test Sequence

This is a typical flow of control for a test.

_images/vsperf.png
2.4. Configuration

The conf package contains the configuration files (*.conf) for all system components, it also provides a settings object that exposes all of these settings.

Settings are not passed from component to component. Rather they are available globally to all components once they import the conf package.

from conf import settings
...
log_file = settings.getValue('LOG_FILE_DEFAULT')

Settings files (*.conf) are valid python code so can be set to complex types such as lists and dictionaries as well as scalar types:

first_packet_size = settings.getValue('PACKET_SIZE_LIST')[0]
2.4.1. Configuration Procedure and Precedence

Configuration files follow a strict naming convention that allows them to be processed in a specific order. All the .conf files are named NNx_name.conf, where NN is a decimal number and x is an optional alphabetical suffix. The files are processed in order from 00_name.conf to 99_name.conf (and from 00a_name to 00z_name), so that if the name setting is given in both a lower and higher numbered conf file then the higher numbered file is the effective setting as it is processed after the setting in the lower numbered file.

The values in the file specified by --conf-file takes precedence over all the other configuration files and does not have to follow the naming convention.

2.4.2. Configuration of PATHS dictionary

VSPERF uses external tools like Open vSwitch and Qemu for execution of testcases. These tools may be downloaded and built automatically (see Installation) or installed manually by user from binary packages. It is also possible to use a combination of both approaches, but it is essential to correctly set paths to all required tools. These paths are stored within a PATHS dictionary, which is evaluated before execution of each testcase, in order to setup testcase specific environment. Values selected for testcase execution are internally stored inside TOOLS dictionary, which is used by VSPERF to execute external tools, load kernel modules, etc.

The default configuration of PATHS dictionary is spread among three different configuration files to follow logical grouping of configuration options. Basic description of PATHS dictionary is placed inside conf/00_common.conf. The configuration specific to DPDK and vswitches is located at conf/02_vswitch.conf. The last part related to the Qemu is defined inside conf/04_vnf.conf. Default configuration values can be used in case, that all required tools were downloaded and built automatically by vsperf itself. In case, that some of tools were installed manually from binary packages, then it will be necessary to modify the content of PATHS dictionary accordingly.

Dictionary has a specific section of configuration options for every tool type, it means:

  • PATHS['vswitch'] - contains a separate dictionary for each of vswitches supported by VSPEF

    Example:

    PATHS['vswitch'] = {
       'OvsDpdkVhost': { ... },
       'OvsVanilla' : { ... },
       ...
    }
    
  • PATHS['dpdk'] - contains paths to the dpdk sources, kernel modules and tools (e.g. testpmd)

    Example:

    PATHS['dpdk'] = {
       'type' : 'src',
       'src': {
           'path': os.path.join(ROOT_DIR, 'src/dpdk/dpdk/'),
           'modules' : ['uio', os.path.join(RTE_TARGET, 'kmod/igb_uio.ko')],
           'bind-tool': 'tools/dpdk*bind.py',
           'testpmd': os.path.join(RTE_TARGET, 'app', 'testpmd'),
       },
       ...
    }
    
  • PATHS['qemu'] - contains paths to the qemu sources and executable file

    Example:

    PATHS['qemu'] = {
        'type' : 'bin',
        'bin': {
            'qemu-system': 'qemu-system-x86_64'
        },
        ...
    }
    

Every section specific to the particular vswitch, dpdk or qemu may contain following types of configuration options:

  • option type - is a string, which defines the type of configured paths (‘src’ or ‘bin’) to be selected for a given section:

    • value src means, that VSPERF will use vswitch, DPDK or QEMU built from sources e.g. by execution of systems/build_base_machine.sh script during VSPERF installation
    • value bin means, that VSPERF will use vswitch, DPDK or QEMU binaries installed directly in the operating system, e.g. via OS specific packaging system
  • option path - is a string with a valid system path; Its content is checked for existence, prefixed with section name and stored into TOOLS for later use e.g. TOOLS['dpdk_src'] or TOOLS['vswitch_src']

  • option modules - is list of strings with names of kernel modules; Every module name from given list is checked for a ‘.ko’ suffix. In case that it matches and if it is not an absolute path to the module, then module name is prefixed with value of path option defined for the same section

    Example:

    """
    snippet of PATHS definition from the configuration file:
    """
    PATHS['vswitch'] = {
        'OvsVanilla' = {
            'type' : 'src',
            'src': {
                'path': '/tmp/vsperf/src_vanilla/ovs/ovs/',
                'modules' : ['datapath/linux/openvswitch.ko'],
                ...
            },
            ...
        }
        ...
    }
    
    """
    Final content of TOOLS dictionary used during runtime:
    """
    TOOLS['vswitch_modules'] = ['/tmp/vsperf/src_vanilla/ovs/ovs/datapath/linux/openvswitch.ko']
    
  • all other options are strings with names and paths to specific tools; If a given string contains a relative path and option path is defined for a given section, then string content will be prefixed with content of the path. Otherwise the name of the tool will be searched within standard system directories. In case that filename contains OS specific wildcards, then they will be expanded to the real path. At the end of the processing, every absolute path will be checked for its existence. In case that temporary path (i.e. path with a _tmp suffix) does not exist, then log will be written and vsperf will continue. If any other path will not exist, then vsperf execution will be terminated with a runtime error.

    Example:

    """
    snippet of PATHS definition from the configuration file:
    """
    PATHS['vswitch'] = {
        'OvsDpdkVhost': {
            'type' : 'src',
            'src': {
                'path': '/tmp/vsperf/src_vanilla/ovs/ovs/',
                'ovs-vswitchd': 'vswitchd/ovs-vswitchd',
                'ovsdb-server': 'ovsdb/ovsdb-server',
                ...
            }
            ...
        }
        ...
    }
    
    """
    Final content of TOOLS dictionary used during runtime:
    """
    TOOLS['ovs-vswitchd'] = '/tmp/vsperf/src_vanilla/ovs/ovs/vswitchd/ovs-vswitchd'
    TOOLS['ovsdb-server'] = '/tmp/vsperf/src_vanilla/ovs/ovs/ovsdb/ovsdb-server'
    

Note: In case that bin type is set for DPDK, then TOOLS['dpdk_src'] will be set to the value of PATHS['dpdk']['src']['path']. The reason is, that VSPERF uses downloaded DPDK sources to copy DPDK and testpmd into the GUEST, where testpmd is built. In case, that DPDK sources are not available, then vsperf will continue with test execution, but testpmd can’t be used as a guest loopback. This is useful in case, that other guest loopback applications (e.g. buildin or l2fwd) are used.

Note: In case of RHEL 7.3 OS usage, binary package configuration is required for Vanilla OVS tests. With the installation of a supported rpm for OVS there is a section in the conf\10_custom.conf file that can be used.

2.4.3. Configuration of TRAFFIC dictionary

TRAFFIC dictionary is used for configuration of traffic generator. Default values can be found in configuration file conf/03_traffic.conf. These default values can be modified by (first option has the highest priorty):

  1. Parameters section of testcase definition
  2. command line options specified by --test-params argument
  3. custom configuration file

It is to note, that in case of option 1 and 2, it is possible to specify only values, which should be changed. In case of custom configuration file, it is required to specify whole TRAFFIC dictionary with its all values or explicitly call and update() method of TRAFFIC dictionary.

Detailed description of TRAFFIC dictionary items follows:

'traffic_type'  - One of the supported traffic types.
                  E.g. rfc2544_throughput, rfc2544_back2back
                  or rfc2544_continuous
                  Data type: str
                  Default value: "rfc2544_throughput".
'bidir'         - Specifies if generated traffic will be full-duplex (True)
                  or half-duplex (False)
                  Data type: str
                  Supported values: "True", "False"
                  Default value: "False".
'frame_rate'    - Defines desired percentage of frame rate used during
                  continuous stream tests.
                  Data type: int
                  Default value: 100.
'multistream'   - Defines number of flows simulated by traffic generator.
                  Value 0 disables multistream feature
                  Data type: int
                  Supported values: 0-65536 for 'L4' stream type
                                    unlimited for 'L2' and 'L3' stream types
                  Default value: 0.
'stream_type'   - Stream type is an extension of the "multistream" feature.
                  If multistream is disabled, then stream type will be
                  ignored. Stream type defines ISO OSI network layer used
                  for simulation of multiple streams.
                  Data type: str
                  Supported values:
                     "L2" - iteration of destination MAC address
                     "L3" - iteration of destination IP address
                     "L4" - iteration of destination port
                            of selected transport protocol
                  Default value: "L4".
'pre_installed_flows'
               -  Pre-installed flows is an extension of the "multistream"
                  feature. If enabled, it will implicitly insert a flow
                  for each stream. If multistream is disabled, then
                  pre-installed flows will be ignored.
                  Note: It is supported only for p2p deployment scenario.
                  Data type: str
                  Supported values:
                     "Yes" - flows will be inserted into OVS
                     "No"  - flows won't be inserted into OVS
                  Default value: "No".
'flow_type'     - Defines flows complexity.
                  Data type: str
                  Supported values:
                     "port" - flow is defined by ingress ports
                     "IP"   - flow is defined by ingress ports
                              and src and dst IP addresses
                  Default value: "port"
'flow_control'  - Controls flow control support by traffic generator.
                  Supported values:
                     False  - flow control is disabled
                     True   - flow control is enabled
                  Default value: False
                  Note: Currently it is supported by IxNet only
'learning_frames' - Controls learning frames support by traffic generator.
                  Supported values:
                     False  - learning frames are disabled
                     True   - learning frames are enabled
                  Default value: True
                  Note: Currently it is supported by IxNet only
'l2'            - A dictionary with l2 network layer details. Supported
                  values are:
    'srcmac'    - Specifies source MAC address filled by traffic generator.
                  NOTE: It can be modified by vsperf in some scenarios.
                  Data type: str
                  Default value: "00:00:00:00:00:00".
    'dstmac'    - Specifies destination MAC address filled by traffic generator.
                  NOTE: It can be modified by vsperf in some scenarios.
                  Data type: str
                  Default value: "00:00:00:00:00:00".
    'framesize' - Specifies default frame size. This value should not be
                  changed directly. It will be overridden during testcase
                  execution by values specified by list TRAFFICGEN_PKT_SIZES.
                  Data type: int
                  Default value: 64
'l3'            - A dictionary with l3 network layer details. Supported
                  values are:
    'enabled'   - Specifies if l3 layer should be enabled or disabled.
                  Data type: bool
                  Default value: True
                  NOTE: Supported only by IxNet trafficgen class
    'srcip'     - Specifies source MAC address filled by traffic generator.
                  NOTE: It can be modified by vsperf in some scenarios.
                  Data type: str
                  Default value: "1.1.1.1".
    'dstip'     - Specifies destination MAC address filled by traffic generator.
                  NOTE: It can be modified by vsperf in some scenarios.
                  Data type: str
                  Default value: "90.90.90.90".
    'proto'     - Specifies deflaut protocol type.
                  Please check particular traffic generator implementation
                  for supported protocol types.
                  Data type: str
                  Default value: "udp".
'l4'            - A dictionary with l4 network layer details. Supported
                  values are:
    'enabled'   - Specifies if l4 layer should be enabled or disabled.
                  Data type: bool
                  Default value: True
                  NOTE: Supported only by IxNet trafficgen class
    'srcport'   - Specifies source port of selected transport protocol.
                  NOTE: It can be modified by vsperf in some scenarios.
                  Data type: int
                  Default value: 3000
    'dstport'   - Specifies destination port of selected transport protocol.
                  NOTE: It can be modified by vsperf in some scenarios.
                  Data type: int
                  Default value: 3001
'vlan'          - A dictionary with vlan encapsulation details. Supported
                  values are:
    'enabled'   - Specifies if vlan encapsulation should be enabled or
                  disabled.
                  Data type: bool
                  Default value: False
    'id'        - Specifies vlan id.
                  Data type: int (NOTE: must fit to 12 bits)
                  Default value: 0
    'priority'  - Specifies a vlan priority (PCP header field).
                  Data type: int (NOTE: must fit to 3 bits)
                  Default value: 0
    'cfi'       - Specifies if frames can or cannot be dropped during
                  congestion (DEI header field).
                  Data type: int (NOTE: must fit to 1 bit)
                  Default value: 0
2.4.4. Configuration of GUEST options

VSPERF is able to setup scenarios involving a number of VMs in series or in parallel. All configuration options related to a particular VM instance are defined as lists and prefixed with GUEST_ label. It is essential, that there is enough items in all GUEST_ options to cover all VM instances involved in the test. In case there is not enough items, then VSPERF will use the first item of particular GUEST_ option to expand the list to required length.

Example of option expansion for 4 VMs:

"""
Original values:
"""
GUEST_SMP = ['2']
GUEST_MEMORY = ['2048', '4096']

"""
Values after automatic expansion:
"""
GUEST_SMP = ['2', '2', '2', '2']
GUEST_MEMORY = ['2048', '4096', '2048', '2048']

First option can contain macros starting with # to generate VM specific values. These macros can be used only for options of list or str types with GUEST_ prefix.

Example of macros and their expansion for 2 VMs:

"""
Original values:
"""
GUEST_SHARE_DIR = ['/tmp/qemu#VMINDEX_share']
GUEST_BRIDGE_IP = ['#IP(1.1.1.5)/16']

"""
Values after automatic expansion:
"""
GUEST_SHARE_DIR = ['/tmp/qemu0_share', '/tmp/qemu1_share']
GUEST_BRIDGE_IP = ['1.1.1.5/16', '1.1.1.6/16']

Additional examples are available at 04_vnf.conf.

Note: In case, that macro is detected in the first item of the list, then all other items are ignored and list content is created automatically.

Multiple macros can be used inside one configuration option definition, but macros cannot be used inside other macros. The only exception is macro #VMINDEX, which is expanded first and thus it can be used inside other macros.

Following macros are supported:

  • #VMINDEX - it is replaced by index of VM being executed; This macro is expanded first, so it can be used inside other macros.

    Example:

    GUEST_SHARE_DIR = ['/tmp/qemu#VMINDEX_share']
    
  • #MAC(mac_address[, step]) - it will iterate given mac_address with optional step. In case that step is not defined, then it is set to 1. It means, that first VM will use the value of mac_address, second VM value of mac_address increased by step, etc.

    Example:

    GUEST_NICS = [[{'mac' : '#MAC(00:00:00:00:00:01,2)'}]]
    
  • #IP(ip_address[, step]) - it will iterate given ip_address with optional step. In case that step is not defined, then it is set to 1. It means, that first VM will use the value of ip_address, second VM value of ip_address increased by step, etc.

    Example:

    GUEST_BRIDGE_IP = ['#IP(1.1.1.5)/16']
    
  • #EVAL(expression) - it will evaluate given expression as python code; Only simple expressions should be used. Call of the functions is not supported.

    Example:

    GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]
    
2.4.5. Other Configuration

conf.settings also loads configuration from the command line and from the environment.

2.5. PXP Deployment

Every testcase uses one of the supported deployment scenarios to setup test environment. The controller responsible for a given scenario configures flows in the vswitch to route traffic among physical interfaces connected to the traffic generator and virtual machines. VSPERF supports several deployments including PXP deployment, which can setup various scenarios with multiple VMs.

These scenarios are realized by VswitchControllerPXP class, which can configure and execute given number of VMs in serial or parallel configurations. Every VM can be configured with just one or an even number of interfaces. In case that VM has more than 2 interfaces, then traffic is properly routed among pairs of interfaces.

Example of traffic routing for VM with 4 NICs in serial configuration:

         +------------------------------------------+
         |  VM with 4 NICs                          |
         |  +---------------+    +---------------+  |
         |  |  Application  |    |  Application  |  |
         |  +---------------+    +---------------+  |
         |      ^       |            ^       |      |
         |      |       v            |       v      |
         |  +---------------+    +---------------+  |
         |  | logical ports |    | logical ports |  |
         |  |   0       1   |    |   2       3   |  |
         +--+---------------+----+---------------+--+
                ^       :            ^       :
                |       |            |       |
                :       v            :       v
+-----------+---------------+----+---------------+----------+
| vSwitch   |   0       1   |    |   2       3   |          |
|           | logical ports |    | logical ports |          |
| previous  +---------------+    +---------------+   next   |
| VM or PHY     ^       |            ^       |     VM or PHY|
|   port   -----+       +------------+       +--->   port   |
+-----------------------------------------------------------+

It is also possible to define different number of interfaces for each VM to better simulate real scenarios.

Example of traffic routing for 2 VMs in serial configuration, where 1st VM has 4 NICs and 2nd VM 2 NICs:

         +------------------------------------------+  +---------------------+
         |  1st VM with 4 NICs                      |  |  2nd VM with 2 NICs |
         |  +---------------+    +---------------+  |  |  +---------------+  |
         |  |  Application  |    |  Application  |  |  |  |  Application  |  |
         |  +---------------+    +---------------+  |  |  +---------------+  |
         |      ^       |            ^       |      |  |      ^       |      |
         |      |       v            |       v      |  |      |       v      |
         |  +---------------+    +---------------+  |  |  +---------------+  |
         |  | logical ports |    | logical ports |  |  |  | logical ports |  |
         |  |   0       1   |    |   2       3   |  |  |  |   0       1   |  |
         +--+---------------+----+---------------+--+  +--+---------------+--+
                ^       :            ^       :               ^       :
                |       |            |       |               |       |
                :       v            :       v               :       v
+-----------+---------------+----+---------------+-------+---------------+----------+
| vSwitch   |   0       1   |    |   2       3   |       |   4       5   |          |
|           | logical ports |    | logical ports |       | logical ports |          |
| previous  +---------------+    +---------------+       +---------------+   next   |
| VM or PHY     ^       |            ^       |               ^       |     VM or PHY|
|   port   -----+       +------------+       +---------------+       +---->  port   |
+-----------------------------------------------------------------------------------+

The number of VMs involved in the test and the type of their connection is defined by deployment name as follows:

  • pvvp[number] - configures scenario with VMs connected in series with optional number of VMs. In case that number is not specified, then 2 VMs will be used.

    Example of 2 VMs in a serial configuration:

    +----------------------+  +----------------------+
    |   1st VM             |  |   2nd VM             |
    |   +---------------+  |  |   +---------------+  |
    |   |  Application  |  |  |   |  Application  |  |
    |   +---------------+  |  |   +---------------+  |
    |       ^       |      |  |       ^       |      |
    |       |       v      |  |       |       v      |
    |   +---------------+  |  |   +---------------+  |
    |   | logical ports |  |  |   | logical ports |  |
    |   |   0       1   |  |  |   |   0       1   |  |
    +---+---------------+--+  +---+---------------+--+
            ^       :                 ^       :
            |       |                 |       |
            :       v                 :       v
    +---+---------------+---------+---------------+--+
    |   |   0       1   |         |   3       4   |  |
    |   | logical ports | vSwitch | logical ports |  |
    |   +---------------+         +---------------+  |
    |       ^       |                 ^       |      |
    |       |       +-----------------+       v      |
    |   +----------------------------------------+   |
    |   |              physical ports            |   |
    |   |      0                         1       |   |
    +---+----------------------------------------+---+
               ^                         :
               |                         |
               :                         v
    +------------------------------------------------+
    |                                                |
    |                traffic generator               |
    |                                                |
    +------------------------------------------------+
    
  • pvpv[number] - configures scenario with VMs connected in parallel with optional number of VMs. In case that number is not specified, then 2 VMs will be used. Multistream feature is used to route traffic to particular VMs (or NIC pairs of every VM). It means, that VSPERF will enable multistream feature and sets the number of streams to the number of VMs and their NIC pairs. Traffic will be dispatched based on Stream Type, i.e. by UDP port, IP address or MAC address.

    Example of 2 VMs in a parallel configuration, where traffic is dispatched

    based on the UDP port.

    +----------------------+  +----------------------+
    |   1st VM             |  |   2nd VM             |
    |   +---------------+  |  |   +---------------+  |
    |   |  Application  |  |  |   |  Application  |  |
    |   +---------------+  |  |   +---------------+  |
    |       ^       |      |  |       ^       |      |
    |       |       v      |  |       |       v      |
    |   +---------------+  |  |   +---------------+  |
    |   | logical ports |  |  |   | logical ports |  |
    |   |   0       1   |  |  |   |   0       1   |  |
    +---+---------------+--+  +---+---------------+--+
            ^       :                 ^       :
            |       |                 |       |
            :       v                 :       v
    +---+---------------+---------+---------------+--+
    |   |   0       1   |         |   3       4   |  |
    |   | logical ports | vSwitch | logical ports |  |
    |   +---------------+         +---------------+  |
    |      ^         |                 ^       :     |
    |      |     ......................:       :     |
    |  UDP | UDP :   |                         :     |
    |  port| port:   +--------------------+    :     |
    |   0  |  1  :                        |    :     |
    |      |     :                        v    v     |
    |   +----------------------------------------+   |
    |   |              physical ports            |   |
    |   |    0                               1   |   |
    +---+----------------------------------------+---+
             ^                               :
             |                               |
             :                               v
    +------------------------------------------------+
    |                                                |
    |                traffic generator               |
    |                                                |
    +------------------------------------------------+
    

PXP deployment is backward compatible with PVP deployment, where pvp is an alias for pvvp1 and it executes just one VM.

The number of interfaces used by VMs is defined by configuration option GUEST_NICS_NR. In case that more than one pair of interfaces is defined for VM, then:

  • for pvvp (serial) scenario every NIC pair is connected in serial before connection to next VM is created
  • for pvpv (parallel) scenario every NIC pair is directly connected to the physical ports and unique traffic stream is assigned to it

Examples:

  • Deployment pvvp10 will start 10 VMs and connects them in series
  • Deployment pvpv4 will start 4 VMs and connects them in parallel
  • Deployment pvpv1 and GUEST_NICS_NR = [4] will start 1 VM with 4 interfaces and every NIC pair is directly connected to the physical ports
  • Deployment pvvp and GUEST_NICS_NR = [2, 4] will start 2 VMs; 1st VM will have 2 interfaces and 2nd VM 4 interfaces. These interfaces will be connected in serial, i.e. traffic will flow as follows: PHY1 -> VM1_1 -> VM1_2 -> VM2_1 -> VM2_2 -> VM2_3 -> VM2_4 -> PHY2

Note: In case that only 1 or more than 2 NICs are configured for VM, then testpmd should be used as forwarding application inside the VM. As it is able to forward traffic between multiple VM NIC pairs.

Note: In case of linux_bridge, all NICs are connected to the same bridge inside the VM.

2.6. VM, vSwitch, Traffic Generator Independence

VSPERF supports different VSwitches, Traffic Generators, VNFs and Forwarding Applications by using standard object-oriented polymorphism:

  • Support for vSwitches is implemented by a class inheriting from IVSwitch.
  • Support for Traffic Generators is implemented by a class inheriting from ITrafficGenerator.
  • Support for VNF is implemented by a class inheriting from IVNF.
  • Support for Forwarding Applications is implemented by a class inheriting from IPktFwd.

By dealing only with the abstract interfaces the core framework can support many implementations of different vSwitches, Traffic Generators, VNFs and Forwarding Applications.

2.6.1. IVSwitch
class IVSwitch:
  start(self)
  stop(self)
  add_switch(switch_name)
  del_switch(switch_name)
  add_phy_port(switch_name)
  add_vport(switch_name)
  get_ports(switch_name)
  del_port(switch_name, port_name)
  add_flow(switch_name, flow)
  del_flow(switch_name, flow=None)
2.6.2. ITrafficGenerator
class ITrafficGenerator:
  connect()
  disconnect()

  send_burst_traffic(traffic, numpkts, time, framerate)

  send_cont_traffic(traffic, time, framerate)
  start_cont_traffic(traffic, time, framerate)
  stop_cont_traffic(self):

  send_rfc2544_throughput(traffic, tests, duration, lossrate)
  start_rfc2544_throughput(traffic, tests, duration, lossrate)
  wait_rfc2544_throughput(self)

  send_rfc2544_back2back(traffic, tests, duration, lossrate)
  start_rfc2544_back2back(traffic, , tests, duration, lossrate)
  wait_rfc2544_back2back()

Note send_xxx() blocks whereas start_xxx() does not and must be followed by a subsequent call to wait_xxx().

2.6.3. IVnf
class IVnf:
  start(memory, cpus,
        monitor_path, shared_path_host,
        shared_path_guest, guest_prompt)
  stop()
  execute(command)
  wait(guest_prompt)
  execute_and_wait (command)
2.6.4. IPktFwd
class IPktFwd:
    start()
    stop()
2.6.5. Controllers

Controllers are used in conjunction with abstract interfaces as way of decoupling the control of vSwtiches, VNFs, TrafficGenerators and Forwarding Applications from other components.

The controlled classes provide basic primitive operations. The Controllers sequence and co-ordinate these primitive operation in to useful actions. For instance the vswitch_controller_p2p can be used to bring any vSwitch (that implements the primitives defined in IVSwitch) into the configuration required by the Phy-to-Phy Deployment Scenario.

In order to support a new vSwitch only a new implementation of IVSwitch needs be created for the new vSwitch to be capable of fulfilling all the Deployment Scenarios provided for by existing or future vSwitch Controllers.

Similarly if a new Deployment Scenario is required it only needs to be written once as a new vSwitch Controller and it will immediately be capable of controlling all existing and future vSwitches in to that Deployment Scenario.

Similarly the Traffic Controllers can be used to co-ordinate basic operations provided by implementers of ITrafficGenerator to provide useful tests. Though traffic generators generally already implement full test cases i.e. they both generate suitable traffic and analyse returned traffic in order to implement a test which has typically been predefined in an RFC document. However the Traffic Controller class allows for the possibility of further enhancement - such as iterating over tests for various packet sizes or creating new tests.

2.6.6. Traffic Controller’s Role _images/traffic_controller.png
2.6.7. Loader & Component Factory

The working of the Loader package (which is responsible for finding arbitrary classes based on configuration data) and the Component Factory which is responsible for choosing the correct class for a particular situation - e.g. Deployment Scenario can be seen in this diagram.

_images/factory_and_loader.png
2.7. Routing Tables

Vsperf uses a standard set of routing tables in order to allow tests to easily mix and match Deployment Scenarios (PVP, P2P topology), Tuple Matching and Frame Modification requirements.

+--------------+
|              |
| Table 0      |  table#0 - Match table. Flows designed to force 5 & 10
|              |  tuple matches go here.
|              |
+--------------+
       |
       |
       v
+--------------+  table#1 - Routing table. Flow entries to forward
|              |  packets between ports goes here.
| Table 1      |  The chosen port is communicated to subsequent tables by
|              |  setting the metadata value to the egress port number.
|              |  Generally this table is set-up by by the
+--------------+  vSwitchController.
       |
       |
       v
+--------------+  table#2 - Frame modification table. Frame modification
|              |  flow rules are isolated in this table so that they can
| Table 2      |  be turned on or off without affecting the routing or
|              |  tuple-matching flow rules. This allows the frame
|              |  modification and tuple matching required by the tests
|              |  in the VSWITCH PERFORMANCE FOR TELCO NFV test
+--------------+  specification to be independent of the Deployment
       |          Scenario set up by the vSwitchController.
       |
       v
+--------------+
|              |
| Table 3      |  table#3 - Egress table. Egress packets on the ports
|              |  setup in Table 1.
+--------------+
3. VSPERF LEVEL TEST DESIGN (LTD)
3.1. Introduction

The intention of this Level Test Design (LTD) document is to specify the set of tests to carry out in order to objectively measure the current characteristics of a virtual switch in the Network Function Virtualization Infrastructure (NFVI) as well as the test pass criteria. The detailed test cases will be defined in details-of-LTD, preceded by the doc-id-of-LTD and the scope-of-LTD.

This document is currently in draft form.

3.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTD. The format for the document id will be: OPNFV_vswitchperf_LTD_REL_STATUS, where by the status is one of: draft, reviewed, corrected or final. The document id for this version of the LTD is: OPNFV_vswitchperf_LTD_Brahmaputra_REVIEWED.

3.1.2. Scope

The main purpose of this project is to specify a suite of performance tests in order to objectively measure the current packet transfer characteristics of a virtual switch in the NFVI. The intent of the project is to facilitate testing of any virtual switch. Thus, a generic suite of tests shall be developed, with no hard dependencies to a single implementation. In addition, the test case suite shall be architecture independent.

The test cases developed in this project shall not form part of a separate test framework, all of these tests may be inserted into the Continuous Integration Test Framework and/or the Platform Functionality Test Framework - if a vSwitch becomes a standard component of an OPNFV release.

3.2. Details of the Level Test Design

This section describes the features to be tested (FeaturesToBeTested-of-LTD), and identifies the sets of test cases or scenarios (TestIdentification-of-LTD).

3.2.1. Features to be tested

Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) includes measuring the following performance metrics:

  • Throughput
  • Packet delay
  • Packet delay variation
  • Packet loss
  • Burst behaviour
  • Packet re-ordering
  • Packet correctness
  • Availability and capacity of the DUT
3.2.2. Test identification
3.2.2.1. Throughput tests

The following tests aim to determine the maximum forwarding rate that can be achieved with a virtual switch. The list is not exhaustive but should indicate the type of tests that should be required. It is expected that more will be added.

3.2.2.1.1. Test ID: LTD.Throughput.RFC2544.PacketLossRatio

Title: RFC 2544 X% packet loss ratio Throughput and Latency Test

Prerequisite Test: N/A

Priority:

Description:

This test determines the DUT’s maximum forwarding rate with X% traffic loss for a constant load (fixed length frames at a fixed interval time). The default loss percentages to be tested are: - X = 0% - X = 10^-7%

Note: Other values can be tested if required by the user.

The selected frame sizes are those previously defined under Default Test Parameters. The test can also be used to determine the average latency of the traffic.

Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to obtain the final result.

Expected Result: At the end of each trial, the presence or absence of loss determines the modification of offered load for the next trial, converging on a maximum rate, or RFC2544 Throughput with X% loss. The Throughput load is re-used in related RFC2544 tests and other tests.

Metrics Collected:

The following are the metrics collected for this test:

  • The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss.
  • The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
3.2.2.1.2. Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification

Title: RFC 2544 X% packet loss Throughput and Latency Test with packet modification

Prerequisite Test: N/A

Priority:

Description:

This test determines the DUT’s maximum forwarding rate with X% traffic loss for a constant load (fixed length frames at a fixed interval time). The default loss percentages to be tested are: - X = 0% - X = 10^-7%

Note: Other values can be tested if required by the user.

The selected frame sizes are those previously defined under Default Test Parameters. The test can also be used to determine the average latency of the traffic.

Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to obtain the final result.

During this test, the DUT must perform the following operations on the traffic flow:

  • Perform packet parsing on the DUT’s ingress port.
  • Perform any relevant address look-ups on the DUT’s ingress ports.
  • Modify the packet header before forwarding the packet to the DUT’s egress port. Packet modifications include:
    • Modifying the Ethernet source or destination MAC address.
    • Modifying/adding a VLAN tag. (Recommended).
    • Modifying/adding a MPLS tag.
    • Modifying the source or destination ip address.
    • Modifying the TOS/DSCP field.
    • Modifying the source or destination ports for UDP/TCP/SCTP.
    • Modifying the TTL.

Expected Result: The Packet parsing/modifications require some additional degree of processing resource, therefore the RFC2544 Throughput is expected to be somewhat lower than the Throughput level measured without additional steps. The reduction is expected to be greatest on tests with the smallest packet sizes (greatest header processing rates).

Metrics Collected:

The following are the metrics collected for this test:

  • The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss and packet modification operations being performed by the DUT.
  • The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
  • The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
3.2.2.1.3. Test ID: LTD.Throughput.RFC2544.Profile

Title: RFC 2544 Throughput and Latency Profile

Prerequisite Test: N/A

Priority:

Description:

This test reveals how throughput and latency degrades as the offered rate varies in the region of the DUT’s maximum forwarding rate as determined by LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). For example it can be used to determine if the degradation of throughput and latency as the offered rate increases is slow and graceful or sudden and severe.

The selected frame sizes are those previously defined under Default Test Parameters.

The offered traffic rate is described as a percentage delta with respect to the DUT’s RFC 2544 Throughput as determined by LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta of 0% is equivalent to an offered traffic rate equal to the RFC 2544 Maximum Throughput; A delta of +50% indicates an offered rate half-way between the Maximum RFC2544 Throughput and line-rate, whereas a delta of -50% indicates an offered rate of half the RFC 2544 Maximum Throughput. Therefore the range of the delta figure is natuarlly bounded at -100% (zero offered traffic) and +100% (traffic offered at line rate).

The following deltas to the maximum forwarding rate should be applied:

  • -50%, -10%, 0%, +10% & +50%

Expected Result: For each packet size a profile should be produced of how throughput and latency vary with offered rate.

Metrics Collected:

The following are the metrics collected for this test:

  • The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each delta to the maximum forwarding rate and for each frame size.
  • The average latency for each delta to the maximum forwarding rate and for each frame size.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
  • Any failures experienced (for example if the vSwitch crashes, stops processing packets, restarts or becomes unresponsive to commands) when the offered load is above Maximum Throughput MUST be recorded and reported with the results.
3.2.2.1.4. Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime

Title: RFC 2544 System Recovery Time Test

Prerequisite Test LTD.Throughput.RFC2544.PacketLossRatio

Priority:

Description:

The aim of this test is to determine the length of time it takes the DUT to recover from an overload condition for a constant load (fixed length frames at a fixed interval time). The selected frame sizes are those previously defined under Default Test Parameters, traffic should be sent to the DUT under normal conditions. During the duration of the test and while the traffic flows are passing though the DUT, at least one situation leading to an overload condition for the DUT should occur. The time from the end of the overload condition to when the DUT returns to normal operations should be measured to determine recovery time. Prior to overloading the DUT, one should record the average latency for 10,000 packets forwarded through the DUT.

The overload condition SHOULD be to transmit traffic at a very high frame rate to the DUT (150% of the maximum 0% packet loss rate as determined by LTD.Throughput.RFC2544.PacketLossRatio or line-rate whichever is lower), for at least 60 seconds, then reduce the frame rate to 75% of the maximum 0% packet loss rate. A number of time-stamps should be recorded: - Record the time-stamp at which the frame rate was reduced and record a second time-stamp at the time of the last frame lost. The recovery time is the difference between the two timestamps. - Record the average latency for 10,000 frames after the last frame loss and continue to record average latency measurements for every 10,000 frames, when latency returns to within 10% of pre-overload levels record the time-stamp.

Expected Result:

Metrics collected

The following are the metrics collected for this test:

  • The length of time it takes the DUT to recover from an overload condition.
  • The length of time it takes the DUT to recover the average latency to pre-overload conditions.

Deployment scenario:

  • Physical → virtual switch → physical.
3.2.2.1.5. Test ID: LTD.Throughput.RFC2544.BackToBackFrames

Title: RFC2544 Back To Back Frames Test

Prerequisite Test: N

Priority:

Description:

The aim of this test is to characterize the ability of the DUT to process back-to-back frames. For each frame size previously defined under Default Test Parameters, a burst of traffic is sent to the DUT with the minimum inter-frame gap between each frame. If the number of received frames equals the number of frames that were transmitted, the burst size should be increased and traffic is sent to the DUT again. The value measured is the back-to-back value, that is the maximum burst size the DUT can handle without any frame loss. Please note a trial must run for a minimum of 2 seconds and should be repeated 50 times (at a minimum).

Expected Result:

Tests of back-to-back frames with physical devices have produced unstable results in some cases. All tests should be repeated in multiple test sessions and results stability should be examined.

Metrics collected

The following are the metrics collected for this test:

  • The average back-to-back value across the trials, which is the number of frames in the longest burst that the DUT will handle without the loss of any frames.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.

Deployment scenario:

  • Physical → virtual switch → physical.
3.2.2.1.6. Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak

Title: RFC 2889 X% packet loss Max Forwarding Rate Soak Test

Prerequisite Test LTD.Throughput.RFC2544.PacketLossRatio

Priority:

Description:

The aim of this test is to understand the Max Forwarding Rate stability over an extended test duration in order to uncover any outliers. To allow for an extended test duration, the test should ideally run for 24 hours or, if this is not possible, for at least 6 hours. For this test, each frame size must be sent at the highest Throughput rate with X% packet loss, as determined in the prerequisite test. The default loss percentages to be tested are: - X = 0% - X = 10^-7%

Note: Other values can be tested if required by the user.

Expected Result:

Metrics Collected:

The following are the metrics collected for this test:

  • Max Forwarding Rate stability of the DUT.
    • This means reporting the number of packets lost per time interval and reporting any time intervals with packet loss. The RFC2889 Forwarding Rate shall be measured in each interval. An interval of 60s is suggested.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
  • The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile.
3.2.2.1.7. Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification

Title: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification

Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss)

Priority:

Description:

The aim of this test is to understand the Max Forwarding Rate stability over an extended test duration in order to uncover any outliers. To allow for an extended test duration, the test should ideally run for 24 hours or, if this is not possible, for at least 6 hour. For this test, each frame size must be sent at the highest Throughput rate with 0% packet loss, as determined in the prerequisite test.

During this test, the DUT must perform the following operations on the traffic flow:

  • Perform packet parsing on the DUT’s ingress port.
  • Perform any relevant address look-ups on the DUT’s ingress ports.
  • Modify the packet header before forwarding the packet to the DUT’s egress port. Packet modifications include:
    • Modifying the Ethernet source or destination MAC address.
    • Modifying/adding a VLAN tag (Recommended).
    • Modifying/adding a MPLS tag.
    • Modifying the source or destination ip address.
    • Modifying the TOS/DSCP field.
    • Modifying the source or destination ports for UDP/TCP/SCTP.
    • Modifying the TTL.

Expected Result:

Metrics Collected:

The following are the metrics collected for this test:

  • Max Forwarding Rate stability of the DUT.
    • This means reporting the number of packets lost per time interval and reporting any time intervals with packet loss. The RFC2889 Forwarding Rate shall be measured in each interval. An interval of 60s is suggested.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
  • The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile.
3.2.2.1.8. Test ID: LTD.Throughput.RFC6201.ResetTime

Title: RFC 6201 Reset Time Test

Prerequisite Test: N/A

Priority:

Description:

The aim of this test is to determine the length of time it takes the DUT to recover from a reset.

Two reset methods are defined - planned and unplanned. A planned reset requires stopping and restarting the virtual switch by the usual ‘graceful’ method defined by it’s documentation. An unplanned reset requires simulating a fatal internal fault in the virtual switch - for example by using kill -SIGKILL on a Linux environment.

Both reset methods SHOULD be exercised.

For each frame size previously defined under Default Test Parameters, traffic should be sent to the DUT under normal conditions. During the duration of the test and while the traffic flows are passing through the DUT, the DUT should be reset and the Reset time measured. The Reset time is the total time that a device is determined to be out of operation and includes the time to perform the reset and the time to recover from it (cf. RFC6201).

RFC6201 defines two methods to measure the Reset time:

  • Frame-Loss Method: which requires the monitoring of the number of lost frames and calculates the Reset time based on the number of frames lost and the offered rate according to the following formula:

                       Frames_lost (packets)
    Reset_time = -------------------------------------
                   Offered_rate (packets per second)
    
  • Timestamp Method: which measures the time from which the last frame is forwarded from the DUT to the time the first frame is forwarded after the reset. This involves time-stamping all transmitted frames and recording the timestamp of the last frame that was received prior to the reset and also measuring the timestamp of the first frame that is received after the reset. The Reset time is the difference between these two timestamps.

According to RFC6201 the choice of method depends on the test tool’s capability; the Frame-Loss method SHOULD be used if the test tool supports:

  • Counting the number of lost frames per stream.
  • Transmitting test frame despite the physical link status.

whereas the Timestamp method SHOULD be used if the test tool supports:

  • Timestamping each frame.
  • Monitoring received frame’s timestamp.
  • Transmitting frames only if the physical link status is up.

Expected Result:

Metrics collected

The following are the metrics collected for this test:

  • Average Reset Time over the number of trials performed.

Results of this test should include the following information:

  • The reset method used.
  • Throughput in Fps and Mbps.
  • Average Frame Loss over the number of trials performed.
  • Average Reset Time in milliseconds over the number of trials performed.
  • Number of trials performed.
  • Protocol: IPv4, IPv6, MPLS, etc.
  • Frame Size in Octets
  • Port Media: Ethernet, Gigabit Ethernet (GbE), etc.
  • Port Speed: 10 Gbps, 40 Gbps etc.
  • Interface Encapsulation: Ethernet, Ethernet VLAN, etc.

Deployment scenario:

  • Physical → virtual switch → physical.
3.2.2.1.9. Test ID: LTD.Throughput.RFC2889.MaxForwardingRate

Title: RFC2889 Forwarding Rate Test

Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatio

Priority:

Description:

This test measures the DUT’s Max Forwarding Rate when the Offered Load is varied between the throughput and the Maximum Offered Load for fixed length frames at a fixed time interval. The selected frame sizes are those previously defined under Default Test Parameters. The throughput is the maximum offered load with 0% frame loss (measured by the prerequisite test), and the Maximum Offered Load (as defined by RFC2285) is “the highest number of frames per second that an external source can transmit to a DUT/SUT for forwarding to a specified output interface or interfaces”.

Traffic should be sent to the DUT at a particular rate (TX rate) starting with TX rate equal to the throughput rate. The rate of successfully received frames at the destination counted (in FPS). If the RX rate is equal to the TX rate, the TX rate should be increased by a fixed step size and the RX rate measured again until the Max Forwarding Rate is found.

The trial duration for each iteration should last for the period of time needed for the system to reach steady state for the frame size being tested. Under RFC2889 (Sec. 5.6.3.1) test methodology, the test duration should run for a minimum period of 30 seconds, regardless whether the system reaches steady state before the minimum duration ends.

Expected Result: According to RFC2889 The Max Forwarding Rate is the highest forwarding rate of a DUT taken from an iterative set of forwarding rate measurements. The iterative set of forwarding rate measurements are made by setting the intended load transmitted from an external source and measuring the offered load (i.e what the DUT is capable of forwarding). If the Throughput == the Maximum Offered Load, it follows that Max Forwarding Rate is equal to the Maximum Offered Load.

Metrics Collected:

The following are the metrics collected for this test:

  • The Max Forwarding Rate for the DUT for each packet size.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.

Deployment scenario:

  • Physical → virtual switch → physical. Note: Full mesh tests with multiple ingress and egress ports are a key aspect of RFC 2889 benchmarks, and scenarios with both 2 and 4 ports should be tested. In any case, the number of ports used must be reported.
3.2.2.1.10. Test ID: LTD.Throughput.RFC2889.ForwardPressure

Title: RFC2889 Forward Pressure Test

Prerequisite Test: LTD.Throughput.RFC2889.MaxForwardingRate

Priority:

Description:

The aim of this test is to determine if the DUT transmits frames with an inter-frame gap that is less than 12 bytes. This test overloads the DUT and measures the output for forward pressure. Traffic should be transmitted to the DUT with an inter-frame gap of 11 bytes, this will overload the DUT by 1 byte per frame. The forwarding rate of the DUT should be measured.

Expected Result: The forwarding rate should not exceed the maximum forwarding rate of the DUT collected by LTD.Throughput.RFC2889.MaxForwardingRate.

Metrics collected

The following are the metrics collected for this test:

  • Forwarding rate of the DUT in FPS or Mbps.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.

Deployment scenario:

  • Physical → virtual switch → physical.
3.2.2.1.11. Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering

Title: RFC2889 Error Frames Filtering Test

Prerequisite Test: N/A

Priority:

Description:

The aim of this test is to determine whether the DUT will propagate any erroneous frames it receives or whether it is capable of filtering out the erroneous frames. Traffic should be sent with erroneous frames included within the flow at random intervals. Illegal frames that must be tested include: - Oversize Frames. - Undersize Frames. - CRC Errored Frames. - Dribble Bit Errored Frames - Alignment Errored Frames

The traffic flow exiting the DUT should be recorded and checked to determine if the erroneous frames where passed through the DUT.

Expected Result: Broken frames are not passed!

Metrics collected

No Metrics are collected in this test, instead it determines:

  • Whether the DUT will propagate erroneous frames.
  • Or whether the DUT will correctly filter out any erroneous frames from traffic flow with out removing correct frames.

Deployment scenario:

  • Physical → virtual switch → physical.
3.2.2.1.12. Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding

Title: RFC2889 Broadcast Frame Forwarding Test

Prerequisite Test: N

Priority:

Description:

The aim of this test is to determine the maximum forwarding rate of the DUT when forwarding broadcast traffic. For each frame previously defined under Default Test Parameters, the traffic should be set up as broadcast traffic. The traffic throughput of the DUT should be measured.

The test should be conducted with at least 4 physical ports on the DUT. The number of ports used MUST be recorded.

As broadcast involves forwarding a single incoming packet to several destinations, the latency of a single packet is defined as the average of the latencies for each of the broadcast destinations.

The incoming packet is transmitted on each of the other physical ports, it is not transmitted on the port on which it was received. The test MAY be conducted using different broadcasting ports to uncover any performance differences.

Expected Result:

Metrics collected:

The following are the metrics collected for this test:

  • The forwarding rate of the DUT when forwarding broadcast traffic.
  • The minimum, average & maximum packets latencies observed.

Deployment scenario:

  • Physical → virtual switch 3x physical. In the Broadcast rate testing, four test ports are required. One of the ports is connected to the test device, so it can send broadcast frames and listen for miss-routed frames.
3.2.2.1.13. Test ID: LTD.Throughput.RFC2544.WorstN-BestN

Title: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test

Prerequisite Test: N/A

Priority:

Description:

This test determines the DUT’s maximum forwarding rate with X% traffic loss for a constant load (fixed length frames at a fixed interval time). The default loss percentages to be tested are: X = 0%, X = 10^-7%

Modified RFC 2544 throughput benchmarking methodology aims to quantify the throughput measurement variations observed during standard RFC 2544 benchmarking measurements of virtual switches and VNFs. The RFC2544 binary search algorithm is modified to use more samples per test trial to drive the binary search and yield statistically more meaningful results. This keeps the heart of the RFC2544 methodology, still relying on the binary search of throughput at specified loss tolerance, while providing more useful information about the range of results seen in testing. Instead of using a single traffic trial per iteration step, each traffic trial is repeated N times and the success/failure of the iteration step is based on these N traffic trials. Two types of revised tests are defined - Worst-of-N and Best-of-N.

Worst-of-N

Worst-of-N indicates the lowest expected maximum throughput for ( packet size, loss tolerance) when repeating the test.

  1. Repeat the same test run N times at a set packet rate, record each result.
  2. Take the WORST result (highest packet loss) out of N result samples, called the Worst-of-N sample.
  3. If Worst-of-N sample has loss less than the set loss tolerance, then the step is successful - increase the test traffic rate.
  4. If Worst-of-N sample has loss greater than the set loss tolerance then the step failed - decrease the test traffic rate.
  5. Go to step 1.

Best-of-N

Best-of-N indicates the highest expected maximum throughput for ( packet size, loss tolerance) when repeating the test.

  1. Repeat the same traffic run N times at a set packet rate, record each result.
  2. Take the BEST result (least packet loss) out of N result samples, called the Best-of-N sample.
  3. If Best-of-N sample has loss less than the set loss tolerance, then the step is successful - increase the test traffic rate.
  4. If Best-of-N sample has loss greater than the set loss tolerance, then the step failed - decrease the test traffic rate.
  5. Go to step 1.

Performing both Worst-of-N and Best-of-N benchmark tests yields lower and upper bounds of expected maximum throughput under the operating conditions, giving a very good indication to the user of the deterministic performance range for the tested setup.

Expected Result: At the end of each trial series, the presence or absence of loss determines the modification of offered load for the next trial series, converging on a maximum rate, or RFC2544 Throughput with X% loss. The Throughput load is re-used in related RFC2544 tests and other tests.

Metrics Collected:

The following are the metrics collected for this test:

  • The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss.
  • The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
  • Following may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system:
  • CPU core utilization.
  • CPU cache utilization.
  • Memory footprint.
  • System bus (QPI, PCI, ...) utilization.
  • CPU cycles consumed per packet.
3.2.2.1.14. Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio

Title: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test

NOTE: Throughout this test, four interchangeable overlay technologies are covered by the same test description. They are: VXLAN, GRE, NVGRE and GENEVE.

Prerequisite Test: N/A

Priority:

Description: This test evaluates standard switch performance benchmarks for the scenario where an Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered (replacing <tech> in the test name) include:

  • VXLAN
  • GRE
  • NVGRE
  • GENEVE

Performance will be assessed for each of the following overlay network functions:

  • Encapsulation only
  • De-encapsulation only
  • Both Encapsulation and De-encapsulation

For each native packet, the DUT must perform the following operations:

  • Examine the packet and classify its correct overlay net (tunnel) assignment
  • Encapsulate the packet
  • Switch the packet to the correct port

For each encapsulated packet, the DUT must perform the following operations:

  • Examine the packet and classify its correct native network assignment
  • De-encapsulate the packet, if required
  • Switch the packet to the correct port

The selected frame sizes are those previously defined under Default Test Parameters.

Thus, each test comprises an overlay technology, a network function, and a packet size with overlay network overhead included (but see also the discussion at https://etherpad.opnfv.org/p/vSwitchTestsDrafts ).

The test can also be used to determine the average latency of the traffic.

Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to obtain the final result for Throughput.

Expected Result: At the end of each trial, the presence or absence of loss determines the modification of offered load for the next trial, converging on a maximum rate, or RFC2544 Throughput with X% loss (where the value of X is typically equal to zero). The Throughput load is re-used in related RFC2544 tests and other tests.

Metrics Collected: The following are the metrics collected for this test:

  • The maximum Throughput in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss.
  • The average latency of the traffic flow when passing through the DUT and VNFs (if testing for latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
3.2.2.1.15. Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio

Title: RFC 2544 X% packet loss ratio match action Throughput and Latency Test

Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatio

Priority:

Description:

The aim of this test is to determine the cost of carrying out match action(s) on the DUT’s RFC2544 Throughput with X% traffic loss for a constant load (fixed length frames at a fixed interval time).

Each test case requires:

  • selection of a specific match action(s),
  • specifying a percentage of total traffic that is elligible for the match action,
  • determination of the specific test configuration (number of flows, number of test ports, presence of an external controller, etc.), and
  • measurement of the RFC 2544 Throughput level with X% packet loss: Traffic shall be bi-directional and symmetric.

Note: It would be ideal to verify that all match action-elligible traffic was forwarded to the correct port, and if forwarded to an unintended port it should be considered lost.

A match action is an action that is typically carried on a frame or packet that matches a set of flow classification parameters (typically frame/packet header fields). A match action may or may not modify a packet/frame. Match actions include [1]:

  • output : outputs a packet to a particular port.
  • normal: Subjects the packet to traditional L2/L3 processing (MAC learning).
  • flood: Outputs the packet on all switch physical ports other than the port on which it was received and any ports on which flooding is disabled.
  • all: Outputs the packet on all switch physical ports other than the port on which it was received.
  • local: Outputs the packet on the local port, which corresponds to the network device that has the same name as the bridge.
  • in_port: Outputs the packet on the port from which it was received.
  • Controller: Sends the packet and its metadata to the OpenFlow controller as a packet in message.
  • enqueue: Enqueues the packet on the specified queue within port.
  • drop: discard the packet.

Modifications include [1]:

  • mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification
  • mod_dl_src: Sets the source Ethernet address.
  • mod_dl_dst: Sets the destination Ethernet address.
  • mod_nw_src: Sets the IPv4 source address.
  • mod_nw_dst: Sets the IPv4 destination address.
  • mod_tp_src: Sets the TCP or UDP or SCTP source port.
  • mod_tp_dst: Sets the TCP or UDP or SCTP destination port.
  • mod_nw_tos: Sets the DSCP bits in the IPv4 ToS/DSCP or IPv6 traffic class field.
  • mod_nw_ecn: Sets the ECN bits in the appropriate IPv4 or IPv6 field.
  • mod_nw_ttl: Sets the IPv4 TTL or IPv6 hop limit field.

Note: This comprehensive list requires extensive traffic generator capabilities.

The match action(s) that were applied as part of the test should be reported in the final test report.

During this test, the DUT must perform the following operations on the traffic flow:

  • Perform packet parsing on the DUT’s ingress port.
  • Perform any relevant address look-ups on the DUT’s ingress ports.
  • Carry out one or more of the match actions specified above.

The default loss percentages to be tested are: - X = 0% - X = 10^-7% Other values can be tested if required by the user. The selected frame sizes are those previously defined under Default Test Parameters.

The test can also be used to determine the average latency of the traffic when a match action is applied to packets in a flow. Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to obtain the final result.

Expected Result:

At the end of each trial, the presence or absence of loss determines the modification of offered load for the next trial, converging on a maximum rate, or RFC2544Throughput with X% loss. The Throughput load is re-used in related RFC2544 tests and other tests.

Metrics Collected:

The following are the metrics collected for this test:

  • The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss.
  • The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 ofRFC2544).
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.

The metrics collected can be compared to that of the prerequisite test to determine the cost of the match action(s) in the pipeline.

Deployment scenario:

  • Physical → virtual switch → physical (and others are possible)
[1] ovs-ofctl - administer OpenFlow switches
[http://openvswitch.org/support/dist-docs/ovs-ofctl.8.txt ]
3.2.2.2. Packet Latency tests

These tests will measure the store and forward latency as well as the packet delay variation for various packet types through the virtual switch. The following list is not exhaustive but should indicate the type of tests that should be required. It is expected that more will be added.

3.2.2.2.1. Test ID: LTD.PacketLatency.InitialPacketProcessingLatency

Title: Initial Packet Processing Latency

Prerequisite Test: N/A

Priority:

Description:

In some virtual switch architectures, the first packets of a flow will take the system longer to process than subsequent packets in the flow. This test determines the latency for these packets. The test will measure the latency of the packets as they are processed by the flow-setup-path of the DUT. There are two methods for this test, a recommended method and a nalternative method that can be used if it is possible to disable the fastpath of the virtual switch.

Recommended method: This test will send 64,000 packets to the DUT, each belonging to a different flow. Average packet latency will be determined over the 64,000 packets.

Alternative method: This test will send a single packet to the DUT after a fixed interval of time. The time interval will be equivalent to the amount of time it takes for a flow to time out in the virtual switch plus 10%. Average packet latency will be determined over 1,000,000 packets.

This test is intended only for non-learning virtual switches; For learning virtual switches use RFC2889.

For this test, only unidirectional traffic is required.

Expected Result: The average latency for the initial packet of all flows should be greater than the latency of subsequent traffic.

Metrics Collected:

The following are the metrics collected for this test:

  • Average latency of the initial packets of all flows that are processed by the DUT.

Deployment scenario:

  • Physical → Virtual Switch → Physical.
3.2.2.2.2. Test ID: LTD.PacketDelayVariation.RFC3393.Soak

Title: Packet Delay Variation Soak Test

Prerequisite Tests: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)

Priority:

Description:

The aim of this test is to understand the distribution of packet delay variation for different frame sizes over an extended test duration and to determine if there are any outliers. To allow for an extended test duration, the test should ideally run for 24 hours or, if this is not possible, for at least 6 hour. For this test, each frame size must be sent at the highest possible throughput with 0% packet loss, as determined in the prerequisite test.

Expected Result:

Metrics Collected:

The following are the metrics collected for this test:

  • The packet delay variation value for traffic passing through the DUT.
  • The RFC5481 PDV form of delay variation on the traffic flow, using the 99th percentile, for each 60s interval during the test.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
3.2.2.3. Scalability tests

The general aim of these tests is to understand the impact of large flow table size and flow lookups on throughput. The following list is not exhaustive but should indicate the type of tests that should be required. It is expected that more will be added.

3.2.2.3.1. Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss

Title: RFC 2544 0% loss Flow Scalability throughput test

Prerequisite Test: LTD.Throughput.RFC2544.PacketLossRatio, IF the delta Throughput between the single-flow RFC2544 test and this test with a variable number of flows is desired.

Priority:

Description:

The aim of this test is to measure how throughput changes as the number of flows in the DUT increases. The test will measure the throughput through the fastpath, as such the flows need to be installed on the DUT before passing traffic.

For each frame size previously defined under Default Test Parameters and for each of the following number of flows:

  • 1,000
  • 2,000
  • 4,000
  • 8,000
  • 16,000
  • 32,000
  • 64,000
  • Max supported number of flows.

This test will be conducted under two conditions following the establishment of all flows as required by RFC 2544, regarding the flow expiration time-out:

  1. The time-out never expires during each trial.

2) The time-out expires for all flows periodically. This would require a short time-out compared with flow re-appearance for a small number of flows, and may not be possible for all flow conditions.

The maximum 0% packet loss Throughput should be determined in a manner identical to LTD.Throughput.RFC2544.PacketLossRatio.

Expected Result:

Metrics Collected:

The following are the metrics collected for this test:

  • The maximum number of frames per second that can be forwarded at the specified number of flows and the specified frame size, with zero packet loss.
3.2.2.3.2. Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability

Title: RFC 2544 0% loss Memory Bandwidth Scalability test

Prerequisite Tests: LTD.Throughput.RFC2544.PacketLossRatio, IF the delta Throughput between an undisturbed RFC2544 test and this test with the Throughput affected by cache and memory bandwidth contention is desired.

Priority:

Description:

The aim of this test is to understand how the DUT’s performance is affected by cache sharing and memory bandwidth between processes.

During the test all cores not used by the vSwitch should be running a memory intensive application. This application should read and write random data to random addresses in unused physical memory. The random nature of the data and addresses is intended to consume cache, exercise main memory access (as opposed to cache) and exercise all memory buses equally. Furthermore:

  • the ratio of reads to writes should be recorded. A ratio of 1:1 SHOULD be used.
  • the reads and writes MUST be of cache-line size and be cache-line aligned.
  • in NUMA architectures memory access SHOULD be local to the core’s node. Whether only local memory or a mix of local and remote memory is used MUST be recorded.
  • the memory bandwidth (reads plus writes) used per-core MUST be recorded; the test MUST be run with a per-core memory bandwidth equal to half the maximum system memory bandwidth divided by the number of cores. The test MAY be run with other values for the per-core memory bandwidth.
  • the test MAY also be run with the memory intensive application running on all cores.

Under these conditions the DUT’s 0% packet loss throughput is determined as per LTD.Throughput.RFC2544.PacketLossRatio.

Expected Result:

Metrics Collected:

The following are the metrics collected for this test:

  • The DUT’s 0% packet loss throughput in the presence of cache sharing and memory bandwidth between processes.
3.2.2.3.3. Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
Title: VNF Scalability RFC 2544 X% packet loss ratio Throughput and
Latency Test

Prerequisite Test: N/A

Priority:

Description:

This test determines the DUT’s throughput rate with X% traffic loss for a constant load (fixed length frames at a fixed interval time) when the number of VNFs on the DUT increases. The default loss percentages to be tested are: - X = 0% - X = 10^-7% . The minimum number of VNFs to be tested are 3.

Flow classification should be conducted with L2, L3 and L4 matching to understand the matching and scaling capability of the vSwitch. The matching fields which were used as part of the test should be reported as part of the benchmark report.

The vSwitch is responsible for forwarding frames between the VNFs

The SUT (vSwitch and VNF daisy chain) operation should be validated before running the test. This may be completed by running a burst or continuous stream of traffic through the SUT to ensure proper operation before a test.

Note: The traffic rate used to validate SUT operation should be low enough not to stress the SUT.

Note: Other values can be tested if required by the user.

Note: The same VNF should be used in the “daisy chain” formation. Each addition of a VNF should be conducted in a new test setup (The DUT is brought down, then the DUT is brought up again). An atlernative approach would be to continue to add VNFs without bringing down the DUT. The approach used needs to be documented as part of the test report.

The selected frame sizes are those previously defined under Default Test Parameters. The test can also be used to determine the average latency of the traffic.

Under the RFC2544 test methodology, the test duration will include a number of trials; each trial should run for a minimum period of 60 seconds. A binary search methodology must be applied for each trial to obtain the final result for Throughput.

Expected Result: At the end of each trial, the presence or absence of loss determines the modification of offered load for the next trial, converging on a maximum rate, or RFC2544 Throughput with X% loss. The Throughput load is re-used in related RFC2544 tests and other tests.

If the test VNFs are rather light-weight in terms of processing, the test provides a view of multiple passes through the vswitch on logical interfaces. In other words, the test produces an optimistic count of daisy-chained VNFs, but the cumulative effect of traffic on the vSwitch is “real” (assuming that the vSwitch has some dedicated resources, and the effects on shared resources is understood).

Metrics Collected: The following are the metrics collected for this test:

  • The maximum Throughput in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss.
  • The average latency of the traffic flow when passing through the DUT and VNFs (if testing for latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
3.2.2.3.4. Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile

Title: VNF Scalability RFC 2544 Throughput and Latency Profile

Prerequisite Test: N/A

Priority:

Description:

This test reveals how throughput and latency degrades as the number of VNFs increases and offered rate varies in the region of the DUT’s maximum forwarding rate as determined by LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). For example it can be used to determine if the degradation of throughput and latency as the number of VNFs and offered rate increases is slow and graceful, or sudden and severe. The minimum number of VNFs to be tested is 3.

The selected frame sizes are those previously defined under Default Test Parameters.

The offered traffic rate is described as a percentage delta with respect to the DUT’s RFC 2544 Throughput as determined by LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta of 0% is equivalent to an offered traffic rate equal to the RFC 2544 Throughput; A delta of +50% indicates an offered rate half-way between the Throughput and line-rate, whereas a delta of -50% indicates an offered rate of half the maximum rate. Therefore the range of the delta figure is natuarlly bounded at -100% (zero offered traffic) and +100% (traffic offered at line rate).

The following deltas to the maximum forwarding rate should be applied:

  • -50%, -10%, 0%, +10% & +50%

Note: Other values can be tested if required by the user.

Note: The same VNF should be used in the “daisy chain” formation. Each addition of a VNF should be conducted in a new test setup (The DUT is brought down, then the DUT is brought up again). An atlernative approach would be to continue to add VNFs without bringing down the DUT. The approach used needs to be documented as part of the test report.

Flow classification should be conducted with L2, L3 and L4 matching to understand the matching and scaling capability of the vSwitch. The matching fields which were used as part of the test should be reported as part of the benchmark report.

The SUT (vSwitch and VNF daisy chain) operation should be validated before running the test. This may be completed by running a burst or continuous stream of traffic through the SUT to ensure proper operation before a test.

Note: the traffic rate used to validate SUT operation should be low enough not to stress the SUT

Expected Result: For each packet size a profile should be produced of how throughput and latency vary with offered rate.

Metrics Collected:

The following are the metrics collected for this test:

  • The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each delta to the maximum forwarding rate and for each frame size.
  • The average latency for each delta to the maximum forwarding rate and for each frame size.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.
  • Any failures experienced (for example if the vSwitch crashes, stops processing packets, restarts or becomes unresponsive to commands) when the offered load is above Maximum Throughput MUST be recorded and reported with the results.
3.2.2.4. Activation tests

The general aim of these tests is to understand the capacity of the and speed with which the vswitch can accommodate new flows.

3.2.2.4.1. Test ID: LTD.Activation.RFC2889.AddressCachingCapacity

Title: RFC2889 Address Caching Capacity Test

Prerequisite Test: N/A

Priority:

Description:

Please note this test is only applicable to virtual switches that are capable of MAC learning. The aim of this test is to determine the address caching capacity of the DUT for a constant load (fixed length frames at a fixed interval time). The selected frame sizes are those previously defined under Default Test Parameters.

In order to run this test the aging time, that is the maximum time the DUT will keep a learned address in its flow table, and a set of initial addresses, whose value should be >= 1 and <= the max number supported by the implementation must be known. Please note that if the aging time is configurable it must be longer than the time necessary to produce frames from the external source at the specified rate. If the aging time is fixed the frame rate must be brought down to a value that the external source can produce in a time that is less than the aging time.

Learning Frames should be sent from an external source to the DUT to install a number of flows. The Learning Frames must have a fixed destination address and must vary the source address of the frames. The DUT should install flows in its flow table based on the varying source addresses. Frames should then be transmitted from an external source at a suitable frame rate to see if the DUT has properly learned all of the addresses. If there is no frame loss and no flooding, the number of addresses sent to the DUT should be increased and the test is repeated until the max number of cached addresses supported by the DUT determined.

Expected Result:

Metrics collected:

The following are the metrics collected for this test:

  • Number of cached addresses supported by the DUT.
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.

Deployment scenario:

  • Physical → virtual switch → 2 x physical (one receiving, one listening).
3.2.2.4.2. Test ID: LTD.Activation.RFC2889.AddressLearningRate

Title: RFC2889 Address Learning Rate Test

Prerequisite Test: LTD.Memory.RFC2889.AddressCachingCapacity

Priority:

Description:

Please note this test is only applicable to virtual switches that are capable of MAC learning. The aim of this test is to determine the rate of address learning of the DUT for a constant load (fixed length frames at a fixed interval time). The selected frame sizes are those previously defined under Default Test Parameters, traffic should be sent with each IPv4/IPv6 address incremented by one. The rate at which the DUT learns a new address should be measured. The maximum caching capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken into consideration as the maximum number of addresses for which the learning rate can be obtained.

Expected Result: It may be worthwhile to report the behaviour when operating beyond address capacity - some DUTs may be more friendly to new addresses than others.

Metrics collected:

The following are the metrics collected for this test:

  • The address learning rate of the DUT.

Deployment scenario:

  • Physical → virtual switch → 2 x physical (one receiving, one listening).
3.2.2.5. Coupling between control path and datapath Tests

The following tests aim to determine how tightly coupled the datapath and the control path are within a virtual switch. The following list is not exhaustive but should indicate the type of tests that should be required. It is expected that more will be added.

3.2.2.5.1. Test ID: LTD.CPDPCouplingFlowAddition

Title: Control Path and Datapath Coupling

Prerequisite Test:

Priority:

Description:

The aim of this test is to understand how exercising the DUT’s control path affects datapath performance.

Initially a certain number of flow table entries are installed in the vSwitch. Then over the duration of an RFC2544 throughput test flow-entries are added and removed at the rates specified below. No traffic is ‘hitting’ these flow-entries, they are simply added and removed.

The test MUST be repeated with the following initial number of flow-entries installed: - < 10 - 1000 - 100,000 - 10,000,000 (or the maximum supported number of flow-entries)

The test MUST be repeated with the following rates of flow-entry addition and deletion per second: - 0 - 1 (i.e. 1 addition plus 1 deletion) - 100 - 10,000

Expected Result:

Metrics Collected:

The following are the metrics collected for this test:

  • The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT.
  • The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 of RFC2544).
  • CPU and memory utilization may also be collected as part of this test, to determine the vSwitch’s performance footprint on the system.

Deployment scenario:

  • Physical → virtual switch → physical.
3.2.2.6. CPU and memory consumption

The following tests will profile a virtual switch’s CPU and memory utilization under various loads and circumstances. The following list is not exhaustive but should indicate the type of tests that should be required. It is expected that more will be added.

3.2.2.6.1. Test ID: LTD.Stress.RFC2544.0PacketLoss

Title: RFC 2544 0% Loss CPU OR Memory Stress Test

Prerequisite Test:

Priority:

Description:

The aim of this test is to understand the overall performance of the system when a CPU or Memory intensive application is run on the same DUT as the Virtual Switch. For each frame size, an LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be performed. Throughout the entire test a CPU or Memory intensive application should be run on all cores on the system not in use by the Virtual Switch. For NUMA system only cores on the same NUMA node are loaded.

It is recommended that stress-ng be used for loading the non-Virtual Switch cores but any stress tool MAY be used.

Expected Result:

Metrics Collected:

The following are the metrics collected for this test:

  • Memory and CPU utilization of the cores running the Virtual Switch.
  • The number of identity of the cores allocated to the Virtual Switch.
  • The configuration of the stress tool (for example the command line parameters used to start it.)
Note: Stress in the test ID can be replaced with the name of the
component being stressed, when reporting the results: LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss
3.2.2.7. Summary List of Tests
  1. Throughput tests
  • Test ID: LTD.Throughput.RFC2544.PacketLossRatio
  • Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
  • Test ID: LTD.Throughput.RFC2544.Profile
  • Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
  • Test ID: LTD.Throughput.RFC2544.BackToBackFrames
  • Test ID: LTD.Throughput.RFC2889.Soak
  • Test ID: LTD.Throughput.RFC2889.SoakFrameModification
  • Test ID: LTD.Throughput.RFC6201.ResetTime
  • Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
  • Test ID: LTD.Throughput.RFC2889.ForwardPressure
  • Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
  • Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
  • Test ID: LTD.Throughput.RFC2544.WorstN-BestN
  • Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
  1. Packet Latency tests
  • Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
  • Test ID: LTD.PacketDelayVariation.RFC3393.Soak
  1. Scalability tests
  • Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
  • Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
  • LTD.Scalability.VNF.RFC2544.PacketLossProfile
  • LTD.Scalability.VNF.RFC2544.PacketLossRatio
  1. Activation tests
  • Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
  • Test ID: LTD.Activation.RFC2889.AddressLearningRate
  1. Coupling between control path and datapath Tests
  • Test ID: LTD.CPDPCouplingFlowAddition
  1. CPU and memory consumption
  • Test ID: LTD.Stress.RFC2544.0PacketLoss
4. VSPERF LEVEL TEST PLAN (LTP)
4.1. Introduction

The objective of the OPNFV project titled Characterize vSwitch Performance for Telco NFV Use Cases, is to evaluate the performance of virtual switches to identify its suitability for a Telco Network Function Virtualization (NFV) environment. The intention of this Level Test Plan (LTP) document is to specify the scope, approach, resources, and schedule of the virtual switch performance benchmarking activities in OPNFV. The test cases will be identified in a separate document called the Level Test Design (LTD) document.

This document is currently in draft form.

4.1.1. Document identifier

The document id will be used to uniquely identify versions of the LTP. The format for the document id will be: OPNFV_vswitchperf_LTP_REL_STATUS, where by the status is one of: draft, reviewed, corrected or final. The document id for this version of the LTP is: OPNFV_vswitchperf_LTP_Colorado_REVIEWED.

4.1.2. Scope

The main purpose of this project is to specify a suite of performance tests in order to objectively measure the current packet transfer characteristics of a virtual switch in the NFVI. The intent of the project is to facilitate the performance testing of any virtual switch. Thus, a generic suite of tests shall be developed, with no hard dependencies to a single implementation. In addition, the test case suite shall be architecture independent.

The test cases developed in this project shall not form part of a separate test framework, all of these tests may be inserted into the Continuous Integration Test Framework and/or the Platform Functionality Test Framework - if a vSwitch becomes a standard component of an OPNFV release.

4.1.4. Level in the overall sequence

The level of testing conducted by vswitchperf in the overall testing sequence (among all the testing projects in OPNFV) is the performance benchmarking of a specific component (the vswitch) in the OPNFV platfrom. It’s expected that this testing will follow on from the functional and integration testing conducted by other testing projects in OPNFV, namely Functest and Yardstick.

4.1.5. Test classes and overall test conditions

A benchmark is defined by the IETF as: A standardized test that serves as a basis for performance evaluation and comparison. It’s important to note that benchmarks are not Functional tests. They do not provide PASS/FAIL criteria, and most importantly ARE NOT performed on live networks, or performed with live network traffic.

In order to determine the packet transfer characteristics of a virtual switch, the benchmarking tests will be broken down into the following categories:

  • Throughput Tests to measure the maximum forwarding rate (in frames per second or fps) and bit rate (in Mbps) for a constant load (as defined by RFC1242) without traffic loss.
  • Packet and Frame Delay Tests to measure average, min and max packet and frame delay for constant loads.
  • Stream Performance Tests (TCP, UDP) to measure bulk data transfer performance, i.e. how fast systems can send and receive data through the virtual switch.
  • Request/Response Performance Tests (TCP, UDP) the measure the transaction rate through the virtual switch.
  • Packet Delay Tests to understand latency distribution for different packet sizes and over an extended test run to uncover outliers.
  • Scalability Tests to understand how the virtual switch performs as the number of flows, active ports, complexity of the forwarding logic’s configuration... it has to deal with increases.
  • Control Path and Datapath Coupling Tests, to understand how closely coupled the datapath and the control path are as well as the effect of this coupling on the performance of the DUT.
  • CPU and Memory Consumption Tests to understand the virtual switch’s footprint on the system, this includes:
    • CPU core utilization.
    • CPU cache utilization.
    • Memory footprint.
    • System bus (QPI, PCI, ..) utilization.
    • Memory lanes utilization.
    • CPU cycles consumed per packet.
    • Time To Establish Flows Tests.
  • Noisy Neighbour Tests, to understand the effects of resource sharing on the performance of a virtual switch.

Note: some of the tests above can be conducted simultaneously where the combined results would be insightful, for example Packet/Frame Delay and Scalability.

4.2. Details of the Level Test Plan

This section describes the following items: * Test items and their identifiers (TestItems) * Test Traceability Matrix (TestMatrix) * Features to be tested (FeaturesToBeTested) * Features not to be tested (FeaturesNotToBeTested) * Approach (Approach) * Item pass/fail criteria (PassFailCriteria) * Suspension criteria and resumption requirements (SuspensionResumptionReqs)

4.2.1. Test items and their identifiers

The test item/application vsperf is trying to test are virtual switches and in particular their performance in an nfv environment. vsperf will first try to measure the maximum achievable performance by a virtual switch and then it will focus in on usecases that are as close to real life deployment scenarios as possible.

4.2.2. Test Traceability Matrix

vswitchperf leverages the “3x3” matrix (introduced in https://tools.ietf.org/html/draft-ietf-bmwg-virtual-net-02) to achieve test traceability. The matrix was expanded to 3x4 to accommodate scale metrics when displaying the coverage of many metrics/benchmarks). Test case covreage in the LTD is tracked using the following catagories:

  SPEED ACCURACY RELIABILITY SCALE
Activation X X X X
Operation X X X X
De-activation        

X = denotes a test catagory that has 1 or more test cases defined.

4.2.3. Features to be tested

Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) includes measuring the following performance metrics:

  • Throughput as defined by RFC1242: The maximum rate at which none of the offered frames are dropped by the DUT. The maximum frame rate and bit rate that can be transmitted by the DUT without any error should be recorded. Note there is an equivalent bit rate and a specific layer at which the payloads contribute to the bits. Errors and improperly formed frames or packets are dropped.
  • Packet delay introduced by the DUT and its cumulative effect on E2E networks. Frame delay can be measured equivalently.
  • Packet delay variation: measured from the perspective of the VNF/application. Packet delay variation is sometimes called “jitter”. However, we will avoid the term “jitter” as the term holds different meaning to different groups of people. In this document we will simply use the term packet delay variation. The preferred form for this metric is the PDV form of delay variation defined in RFC5481. The most relevant measurement of PDV considers the delay variation of a single user flow, as this will be relevant to the size of end-system buffers to compensate for delay variation. The measurement system’s ability to store the delays of individual packets in the flow of interest is a key factor that determines the specific measurement method. At the outset, it is ideal to view the complete PDV distribution. Systems that can capture and store packets and their delays have the freedom to calculate the reference minimum delay and to determine various quantiles of the PDV distribution accurately (in post-measurement processing routines). Systems without storage must apply algorithms to calculate delay and statistical measurements on the fly. For example, a system may store temporary estimates of the mimimum delay and the set of (100) packets with the longest delays during measurement (to calculate a high quantile, and update these sets with new values periodically. In some cases, a limited number of delay histogram bins will be available, and the bin limits will need to be set using results from repeated experiments. See section 8 of RFC5481.
  • Packet loss (within a configured waiting time at the receiver): All packets sent to the DUT should be accounted for.
  • Burst behaviour: measures the ability of the DUT to buffer packets.
  • Packet re-ordering: measures the ability of the device under test to maintain sending order throughout transfer to the destination.
  • Packet correctness: packets or Frames must be well-formed, in that they include all required fields, conform to length requirements, pass integrity checks, etc.
  • Availability and capacity of the DUT i.e. when the DUT is fully “up” and connected, following measurements should be captured for DUT without any network packet load:
    • Includes average power consumption of the CPUs (in various power states) and system over specified period of time. Time period should not be less than 60 seconds.
    • Includes average per core CPU utilization over specified period of time. Time period should not be less than 60 seconds.
    • Includes the number of NIC interfaces supported.
    • Includes headroom of VM workload processing cores (i.e. available for applications).
4.2.4. Features not to be tested

vsperf doesn’t intend to define or perform any functional tests. The aim is to focus on performance.

4.2.5. Approach

The testing approach adoped by the vswitchperf project is black box testing, meaning the test inputs can be generated and the outputs captured and completely evaluated from the outside of the System Under Test. Some metrics can be collected on the SUT, such as cpu or memory utilization if the collection has no/minimal impact on benchmark. This section will look at the deployment scenarios and the general methodology used by vswitchperf. In addition, this section will also specify the details of the Test Report that must be collected for each of the test cases.

4.2.5.1. Deployment Scenarios

The following represents possible deployment test scenarios which can help to determine the performance of both the virtual switch and the datapaths to physical ports (to NICs) and to logical ports (to VNFs):

4.2.5.1.1. Physical port → vSwitch → physical port
                                                     _
+--------------------------------------------------+  |
|              +--------------------+              |  |
|              |                    |              |  |
|              |                    v              |  |  Host
|   +--------------+            +--------------+   |  |
|   |   phy port   |  vSwitch   |   phy port   |   |  |
+---+--------------+------------+--------------+---+ _|
           ^                           :
           |                           |
           :                           v
+--------------------------------------------------+
|                                                  |
|                traffic generator                 |
|                                                  |
+--------------------------------------------------+
4.2.5.1.2. Physical port → vSwitch → VNF → vSwitch → physical port
                                                      _
+---------------------------------------------------+  |
|                                                   |  |
|   +-------------------------------------------+   |  |
|   |                 Application               |   |  |
|   +-------------------------------------------+   |  |
|       ^                                  :        |  |
|       |                                  |        |  |  Guest
|       :                                  v        |  |
|   +---------------+           +---------------+   |  |
|   | logical port 0|           | logical port 1|   |  |
+---+---------------+-----------+---------------+---+ _|
        ^                                  :
        |                                  |
        :                                  v         _
+---+---------------+----------+---------------+---+  |
|   | logical port 0|          | logical port 1|   |  |
|   +---------------+          +---------------+   |  |
|       ^                                  :       |  |
|       |                                  |       |  |  Host
|       :                                  v       |  |
|   +--------------+            +--------------+   |  |
|   |   phy port   |  vSwitch   |   phy port   |   |  |
+---+--------------+------------+--------------+---+ _|
           ^                           :
           |                           |
           :                           v
+--------------------------------------------------+
|                                                  |
|                traffic generator                 |
|                                                  |
+--------------------------------------------------+
4.2.5.1.3. Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
                                                   _
+----------------------+  +----------------------+  |
|   Guest 1            |  |   Guest 2            |  |
|   +---------------+  |  |   +---------------+  |  |
|   |  Application  |  |  |   |  Application  |  |  |
|   +---------------+  |  |   +---------------+  |  |
|       ^       |      |  |       ^       |      |  |
|       |       v      |  |       |       v      |  |  Guests
|   +---------------+  |  |   +---------------+  |  |
|   | logical ports |  |  |   | logical ports |  |  |
|   |   0       1   |  |  |   |   0       1   |  |  |
+---+---------------+--+  +---+---------------+--+ _|
        ^       :                 ^       :
        |       |                 |       |
        :       v                 :       v        _
+---+---------------+---------+---------------+--+  |
|   |   0       1   |         |   3       4   |  |  |
|   | logical ports |         | logical ports |  |  |
|   +---------------+         +---------------+  |  |
|       ^       |                 ^       |      |  |  Host
|       |       L-----------------+       v      |  |
|   +--------------+          +--------------+   |  |
|   |   phy ports  | vSwitch  |   phy ports  |   |  |
+---+--------------+----------+--------------+---+ _|
        ^       ^                 :       :
        |       |                 |       |
        :       :                 v       v
+--------------------------------------------------+
|                                                  |
|                traffic generator                 |
|                                                  |
+--------------------------------------------------+
4.2.5.1.4. Physical port → VNF → vSwitch → VNF → physical port
                                                    _
+----------------------+  +----------------------+   |
|   Guest 1            |  |   Guest 2            |   |
|+-------------------+ |  | +-------------------+|   |
||     Application   | |  | |     Application   ||   |
|+-------------------+ |  | +-------------------+|   |
|       ^       |      |  |       ^       |      |   |  Guests
|       |       v      |  |       |       v      |   |
|+-------------------+ |  | +-------------------+|   |
||   logical ports   | |  | |   logical ports   ||   |
||  0              1 | |  | | 0              1  ||   |
++--------------------++  ++--------------------++  _|
    ^              :          ^              :
(PCI passthrough)  |          |     (PCI passthrough)
    |              v          :              |      _
+--------++------------+-+------------++---------+   |
|   |    ||        0   | |    1       ||     |   |   |
|   |    ||logical port| |logical port||     |   |   |
|   |    |+------------+ +------------+|     |   |   |
|   |    |     |                 ^     |     |   |   |
|   |    |     L-----------------+     |     |   |   |
|   |    |                             |     |   |   |  Host
|   |    |           vSwitch           |     |   |   |
|   |    +-----------------------------+     |   |   |
|   |                                        |   |   |
|   |                                        v   |   |
| +--------------+              +--------------+ |   |
| | phy port/VF  |              | phy port/VF  | |   |
+-+--------------+--------------+--------------+-+  _|
    ^                                        :
    |                                        |
    :                                        v
+--------------------------------------------------+
|                                                  |
|                traffic generator                 |
|                                                  |
+--------------------------------------------------+
4.2.5.1.5. Physical port → vSwitch → VNF
                                                      _
+---------------------------------------------------+  |
|                                                   |  |
|   +-------------------------------------------+   |  |
|   |                 Application               |   |  |
|   +-------------------------------------------+   |  |
|       ^                                           |  |
|       |                                           |  |  Guest
|       :                                           |  |
|   +---------------+                               |  |
|   | logical port 0|                               |  |
+---+---------------+-------------------------------+ _|
        ^
        |
        :                                            _
+---+---------------+------------------------------+  |
|   | logical port 0|                              |  |
|   +---------------+                              |  |
|       ^                                          |  |
|       |                                          |  |  Host
|       :                                          |  |
|   +--------------+                               |  |
|   |   phy port   |  vSwitch                      |  |
+---+--------------+------------ -------------- ---+ _|
           ^
           |
           :
+--------------------------------------------------+
|                                                  |
|                traffic generator                 |
|                                                  |
+--------------------------------------------------+
4.2.5.1.6. VNF → vSwitch → physical port
                                                      _
+---------------------------------------------------+  |
|                                                   |  |
|   +-------------------------------------------+   |  |
|   |                 Application               |   |  |
|   +-------------------------------------------+   |  |
|                                          :        |  |
|                                          |        |  |  Guest
|                                          v        |  |
|                               +---------------+   |  |
|                               | logical port  |   |  |
+-------------------------------+---------------+---+ _|
                                           :
                                           |
                                           v         _
+------------------------------+---------------+---+  |
|                              | logical port  |   |  |
|                              +---------------+   |  |
|                                          :       |  |
|                                          |       |  |  Host
|                                          v       |  |
|                               +--------------+   |  |
|                     vSwitch   |   phy port   |   |  |
+-------------------------------+--------------+---+ _|
                                       :
                                       |
                                       v
+--------------------------------------------------+
|                                                  |
|                traffic generator                 |
|                                                  |
+--------------------------------------------------+
4.2.5.1.7. VNF → vSwitch → VNF → vSwitch
                                                         _
+-------------------------+  +-------------------------+  |
|   Guest 1               |  |   Guest 2               |  |
|   +-----------------+   |  |   +-----------------+   |  |
|   |   Application   |   |  |   |   Application   |   |  |
|   +-----------------+   |  |   +-----------------+   |  |
|                :        |  |       ^                 |  |
|                |        |  |       |                 |  |  Guest
|                v        |  |       :                 |  |
|     +---------------+   |  |   +---------------+     |  |
|     | logical port 0|   |  |   | logical port 0|     |  |
+-----+---------------+---+  +---+---------------+-----+ _|
                :                    ^
                |                    |
                v                    :                    _
+----+---------------+------------+---------------+-----+  |
|    |     port 0    |            |     port 1    |     |  |
|    +---------------+            +---------------+     |  |
|              :                    ^                   |  |
|              |                    |                   |  |  Host
|              +--------------------+                   |  |
|                                                       |  |
|                     vswitch                           |  |
+-------------------------------------------------------+ _|

HOST 1(Physical port → virtual switch → VNF → virtual switch → Physical port) → HOST 2(Physical port → virtual switch → VNF → virtual switch → Physical port)

4.2.5.1.8. HOST 1 (PVP) → HOST 2 (PVP)
                                                   _
+----------------------+  +----------------------+  |
|   Guest 1            |  |   Guest 2            |  |
|   +---------------+  |  |   +---------------+  |  |
|   |  Application  |  |  |   |  Application  |  |  |
|   +---------------+  |  |   +---------------+  |  |
|       ^       |      |  |       ^       |      |  |
|       |       v      |  |       |       v      |  |  Guests
|   +---------------+  |  |   +---------------+  |  |
|   | logical ports |  |  |   | logical ports |  |  |
|   |   0       1   |  |  |   |   0       1   |  |  |
+---+---------------+--+  +---+---------------+--+ _|
        ^       :                 ^       :
        |       |                 |       |
        :       v                 :       v        _
+---+---------------+--+  +---+---------------+--+  |
|   |   0       1   |  |  |   |   3       4   |  |  |
|   | logical ports |  |  |   | logical ports |  |  |
|   +---------------+  |  |   +---------------+  |  |
|       ^       |      |  |       ^       |      |  |  Hosts
|       |       v      |  |       |       v      |  |
|   +--------------+   |  |   +--------------+   |  |
|   |   phy ports  |   |  |   |   phy ports  |   |  |
+---+--------------+---+  +---+--------------+---+ _|
        ^       :                 :       :
        |       +-----------------+       |
        :                                 v
+--------------------------------------------------+
|                                                  |
|                traffic generator                 |
|                                                  |
+--------------------------------------------------+

Note: For tests where the traffic generator and/or measurement receiver are implemented on VM and connected to the virtual switch through vNIC, the issues of shared resources and interactions between the measurement devices and the device under test must be considered.

Note: Some RFC 2889 tests require a full-mesh sending and receiving pattern involving more than two ports. This possibility is illustrated in the Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port diagram above (with 2 sending and 2 receiving ports, though all ports could be used bi-directionally).

Note: When Deployment Scenarios are used in RFC 2889 address learning or cache capacity testing, an additional port from the vSwitch must be connected to the test device. This port is used to listen for flooded frames.

4.2.5.2. General Methodology:

To establish the baseline performance of the virtual switch, tests would initially be run with a simple workload in the VNF (the recommended simple workload VNF would be DPDK‘s testpmd application forwarding packets in a VM or vloop_vnf a simple kernel module that forwards traffic between two network interfaces inside the virtualized environment while bypassing the networking stack). Subsequently, the tests would also be executed with a real Telco workload running in the VNF, which would exercise the virtual switch in the context of higher level Telco NFV use cases, and prove that its underlying characteristics and behaviour can be measured and validated. Suitable real Telco workload VNFs are yet to be identified.

4.2.5.2.1. Default Test Parameters

The following list identifies the default parameters for suite of tests:

  • Reference application: Simple forwarding or Open Source VNF.
  • Frame size (bytes): 64, 128, 256, 512, 1024, 1280, 1518, 2K, 4k OR Packet size based on use-case (e.g. RTP 64B, 256B) OR Mix of packet sizes as maintained by the Functest project <https://wiki.opnfv.org/traffic_profile_management>.
  • Reordering check: Tests should confirm that packets within a flow are not reordered.
  • Duplex: Unidirectional / Bidirectional. Default: Full duplex with traffic transmitting in both directions, as network traffic generally does not flow in a single direction. By default the data rate of transmitted traffic should be the same in both directions, please note that asymmetric traffic (e.g. downlink-heavy) tests will be mentioned explicitly for the relevant test cases.
  • Number of Flows: Default for non scalability tests is a single flow. For scalability tests the goal is to test with maximum supported flows but where possible will test up to 10 Million flows. Start with a single flow and scale up. By default flows should be added sequentially, tests that add flows simultaneously will explicitly call out their flow addition behaviour. Packets are generated across the flows uniformly with no burstiness. For multi-core tests should consider the number of packet flows based on vSwitch/VNF multi-thread implementation and behavior.
  • Traffic Types: UDP, SCTP, RTP, GTP and UDP traffic.
  • Deployment scenarios are:
  • Physical → virtual switch → physical.
  • Physical → virtual switch → VNF → virtual switch → physical.
  • Physical → virtual switch → VNF → virtual switch → VNF → virtual switch → physical.
  • Physical → VNF → virtual switch → VNF → physical.
  • Physical → virtual switch → VNF.
  • VNF → virtual switch → Physical.
  • VNF → virtual switch → VNF.

Tests MUST have these parameters unless otherwise stated. Test cases with non default parameters will be stated explicitly.

Note: For throughput tests unless stated otherwise, test configurations should ensure that traffic traverses the installed flows through the virtual switch, i.e. flows are installed and have an appropriate time out that doesn’t expire before packet transmission starts.

4.2.5.2.2. Flow Classification

Virtual switches classify packets into flows by processing and matching particular header fields in the packet/frame and/or the input port where the packets/frames arrived. The vSwitch then carries out an action on the group of packets that match the classification parameters. Thus a flow is considered to be a sequence of packets that have a shared set of header field values or have arrived on the same port and have the same action applied to them. Performance results can vary based on the parameters the vSwitch uses to match for a flow. The recommended flow classification parameters for L3 vSwitch performance tests are: the input port, the source IP address, the destination IP address and the Ethernet protocol type field. It is essential to increase the flow time-out time on a vSwitch before conducting any performance tests that do not measure the flow set-up time. Normally the first packet of a particular flow will install the flow in the vSwitch which adds an additional latency, subsequent packets of the same flow are not subject to this latency if the flow is already installed on the vSwitch.

4.2.5.2.3. Test Priority

Tests will be assigned a priority in order to determine which tests should be implemented immediately and which tests implementations can be deferred.

Priority can be of following types: - Urgent: Must be implemented immediately. - High: Must be implemented in the next release. - Medium: May be implemented after the release. - Low: May or may not be implemented at all.

4.2.5.2.4. SUT Setup

The SUT should be configured to its “default” state. The SUT’s configuration or set-up must not change between tests in any way other than what is required to do the test. All supported protocols must be configured and enabled for each test set up.

4.2.5.2.5. Port Configuration

The DUT should be configured with n ports where n is a multiple of 2. Half of the ports on the DUT should be used as ingress ports and the other half of the ports on the DUT should be used as egress ports. Where a DUT has more than 2 ports, the ingress data streams should be set-up so that they transmit packets to the egress ports in sequence so that there is an even distribution of traffic across ports. For example, if a DUT has 4 ports 0(ingress), 1(ingress), 2(egress) and 3(egress), the traffic stream directed at port 0 should output a packet to port 2 followed by a packet to port 3. The traffic stream directed at port 1 should also output a packet to port 2 followed by a packet to port 3.

4.2.5.2.6. Frame Formats

Frame formats Layer 2 (data link layer) protocols

  • Ethernet II
+---------------------------+-----------+
| Ethernet Header | Payload | Check Sum |
+-----------------+---------+-----------+
|_________________|_________|___________|
      14 Bytes     46 - 1500   4 Bytes
                     Bytes

Layer 3 (network layer) protocols

  • IPv4
+-----------------+-----------+---------+-----------+
| Ethernet Header | IP Header | Payload | Checksum  |
+-----------------+-----------+---------+-----------+
|_________________|___________|_________|___________|
      14 Bytes       20 bytes  26 - 1480   4 Bytes
                                 Bytes
  • IPv6
+-----------------+-----------+---------+-----------+
| Ethernet Header | IP Header | Payload | Checksum  |
+-----------------+-----------+---------+-----------+
|_________________|___________|_________|___________|
      14 Bytes       40 bytes  26 - 1460   4 Bytes
                                 Bytes

Layer 4 (transport layer) protocols

  • TCP
  • UDP
  • SCTP
+-----------------+-----------+-----------------+---------+-----------+
| Ethernet Header | IP Header | Layer 4 Header  | Payload | Checksum  |
+-----------------+-----------+-----------------+---------+-----------+
|_________________|___________|_________________|_________|___________|
      14 Bytes      40 bytes      20 Bytes       6 - 1460   4 Bytes
                                                  Bytes

Layer 5 (application layer) protocols

  • RTP
  • GTP
+-----------------+-----------+-----------------+---------+-----------+
| Ethernet Header | IP Header | Layer 4 Header  | Payload | Checksum  |
+-----------------+-----------+-----------------+---------+-----------+
|_________________|___________|_________________|_________|___________|
      14 Bytes      20 bytes     20 Bytes        >= 6 Bytes   4 Bytes
4.2.5.2.7. Packet Throughput

There is a difference between an Ethernet frame, an IP packet, and a UDP datagram. In the seven-layer OSI model of computer networking, packet refers to a data unit at layer 3 (network layer). The correct term for a data unit at layer 2 (data link layer) is a frame, and at layer 4 (transport layer) is a segment or datagram.

Important concepts related to 10GbE performance are frame rate and throughput. The MAC bit rate of 10GbE, defined in the IEEE standard 802 .3ae, is 10 billion bits per second. Frame rate is based on the bit rate and frame format definitions. Throughput, defined in IETF RFC 1242, is the highest rate at which the system under test can forward the offered load, without loss.

The frame rate for 10GbE is determined by a formula that divides the 10 billion bits per second by the preamble + frame length + inter-frame gap.

The maximum frame rate is calculated using the minimum values of the following parameters, as described in the IEEE 802 .3ae standard:

  • Preamble: 8 bytes * 8 = 64 bits
  • Frame Length: 64 bytes (minimum) * 8 = 512 bits
  • Inter-frame Gap: 12 bytes (minimum) * 8 = 96 bits

Therefore, Maximum Frame Rate (64B Frames) = MAC Transmit Bit Rate / (Preamble + Frame Length + Inter-frame Gap) = 10,000,000,000 / (64 + 512 + 96) = 10,000,000,000 / 672 = 14,880,952.38 frame per second (fps)

4.2.5.3. RFCs for testing virtual switch performance

The starting point for defining the suite of tests for benchmarking the performance of a virtual switch is to take existing RFCs and standards that were designed to test their physical counterparts and adapting them for testing virtual switches. The rationale behind this is to establish a fair comparison between the performance of virtual and physical switches. This section outlines the RFCs that are used by this specification.

4.2.5.3.1. RFC 1242 Benchmarking Terminology for Network Interconnection

Devices RFC 1242 defines the terminology that is used in describing performance benchmarking tests and their results. Definitions and discussions covered include: Back-to-back, bridge, bridge/router, constant load, data link frame size, frame loss rate, inter frame gap, latency, and many more.

4.2.5.3.2. RFC 2544 Benchmarking Methodology for Network Interconnect Devices

RFC 2544 outlines a benchmarking methodology for network Interconnect Devices. The methodology results in performance metrics such as latency, frame loss percentage, and maximum data throughput.

In this document network “throughput” (measured in millions of frames per second) is based on RFC 2544, unless otherwise noted. Frame size refers to Ethernet frames ranging from smallest frames of 64 bytes to largest frames of 9K bytes.

Types of tests are:

  1. Throughput test defines the maximum number of frames per second that can be transmitted without any error, or 0% loss ratio. In some Throughput tests (and those tests with long duration), evaluation of an additional frame loss ratio is suggested. The current ratio (10^-7 %) is based on understanding the typical user-to-user packet loss ratio needed for good application performance and recognizing that a single transfer through a vswitch must contribute a tiny fraction of user-to-user loss. Further, the ratio 10^-7 % also recognizes practical limitations when measuring loss ratio.
  2. Latency test measures the time required for a frame to travel from the originating device through the network to the destination device. Please note that RFC2544 Latency measurement will be superseded with a measurement of average latency over all successfully transferred packets or frames.
  3. Frame loss test measures the network’s response in overload conditions - a critical indicator of the network’s ability to support real-time applications in which a large amount of frame loss will rapidly degrade service quality.
  4. Burst test assesses the buffering capability of a virtual switch. It measures the maximum number of frames received at full line rate before a frame is lost. In carrier Ethernet networks, this measurement validates the excess information rate (EIR) as defined in many SLAs.
  5. System recovery to characterize speed of recovery from an overload condition.
  6. Reset to characterize speed of recovery from device or software reset. This type of test has been updated by RFC6201 as such, the methodology defined by this specification will be that of RFC 6201.

Although not included in the defined RFC 2544 standard, another crucial measurement in Ethernet networking is packet delay variation. The definition set out by this specification comes from RFC5481.

4.2.5.3.3. RFC 2285 Benchmarking Terminology for LAN Switching Devices

RFC 2285 defines the terminology that is used to describe the terminology for benchmarking a LAN switching device. It extends RFC 1242 and defines: DUTs, SUTs, Traffic orientation and distribution, bursts, loads, forwarding rates, etc.

4.2.5.3.4. RFC 2889 Benchmarking Methodology for LAN Switching

RFC 2889 outlines a benchmarking methodology for LAN switching, it extends RFC 2544. The outlined methodology gathers performance metrics for forwarding, congestion control, latency, address handling and finally filtering.

4.2.5.3.5. RFC 3918 Methodology for IP Multicast Benchmarking

RFC 3918 outlines a methodology for IP Multicast benchmarking.

4.2.5.3.6. RFC 4737 Packet Reordering Metrics

RFC 4737 describes metrics for identifying and counting re-ordered packets within a stream, and metrics to measure the extent each packet has been re-ordered.

4.2.5.3.7. RFC 5481 Packet Delay Variation Applicability Statement

RFC 5481 defined two common, but different forms of delay variation metrics, and compares the metrics over a range of networking circumstances and tasks. The most suitable form for vSwitch benchmarking is the “PDV” form.

4.2.5.3.8. RFC 6201 Device Reset Characterization

RFC 6201 extends the methodology for characterizing the speed of recovery of the DUT from device or software reset described in RFC 2544.

4.2.6. Item pass/fail criteria

vswitchperf does not specify Pass/Fail criteria for the tests in terms of a threshold, as benchmarks do not (and should not do this). The results/metrics for a test are simply reported. If it had to be defined, a test is considered to have passed if it succesfully completed and a relavent metric was recorded/reported for the SUT.

4.2.7. Suspension criteria and resumption requirements

In the case of a throughput test, a test should be suspended if a virtual switch is failing to forward any traffic. A test should be restarted from a clean state if the intention is to carry out the test again.

4.2.8. Test deliverables

Each test should produce a test report that details SUT information as well as the test results. There are a number of parameters related to the system, DUT and tests that can affect the repeatability of a test results and should be recorded. In order to minimise the variation in the results of a test, it is recommended that the test report includes the following information:

  • Hardware details including:
    • Platform details.
    • Processor details.
    • Memory information (see below)
    • Number of enabled cores.
    • Number of cores used for the test.
    • Number of physical NICs, as well as their details (manufacturer, versions, type and the PCI slot they are plugged into).
    • NIC interrupt configuration.
    • BIOS version, release date and any configurations that were modified.
  • Software details including:
    • OS version (for host and VNF)
    • Kernel version (for host and VNF)
    • GRUB boot parameters (for host and VNF).
    • Hypervisor details (Type and version).
    • Selected vSwitch, version number or commit id used.
    • vSwitch launch command line if it has been parameterised.
    • Memory allocation to the vSwitch – which NUMA node it is using, and how many memory channels.
    • Where the vswitch is built from source: compiler details including versions and the flags that were used to compile the vSwitch.
    • DPDK or any other SW dependency version number or commit id used.
    • Memory allocation to a VM - if it’s from Hugpages/elsewhere.
    • VM storage type: snapshot/independent persistent/independent non-persistent.
    • Number of VMs.
    • Number of Virtual NICs (vNICs), versions, type and driver.
    • Number of virtual CPUs and their core affinity on the host.
    • Number vNIC interrupt configuration.
    • Thread affinitization for the applications (including the vSwitch itself) on the host.
    • Details of Resource isolation, such as CPUs designated for Host/Kernel (isolcpu) and CPUs designated for specific processes (taskset).
  • Memory Details
    • Total memory
    • Type of memory
    • Used memory
    • Active memory
    • Inactive memory
    • Free memory
    • Buffer memory
    • Swap cache
    • Total swap
    • Used swap
    • Free swap
  • Test duration.
  • Number of flows.
  • Traffic Information:
    • Traffic type - UDP, TCP, IMIX / Other.
    • Packet Sizes.
  • Deployment Scenario.

Note: Tests that require additional parameters to be recorded will explicitly specify this.

4.2.9. Test management

This section will detail the test activities that will be conducted by vsperf as well as the infrastructure that will be used to complete the tests in OPNFV.

4.2.10. Planned activities and tasks; test progression

A key consideration when conducting any sort of benchmark is trying to ensure the consistency and repeatability of test results between runs. When benchmarking the performance of a virtual switch there are many factors that can affect the consistency of results. This section describes these factors and the measures that can be taken to limit their effects. In addition, this section will outline some system tests to validate the platform and the VNF before conducting any vSwitch benchmarking tests.

System Isolation:

When conducting a benchmarking test on any SUT, it is essential to limit (and if reasonable, eliminate) any noise that may interfere with the accuracy of the metrics collected by the test. This noise may be introduced by other hardware or software (OS, other applications), and can result in significantly varying performance metrics being collected between consecutive runs of the same test. In the case of characterizing the performance of a virtual switch, there are a number of configuration parameters that can help increase the repeatability and stability of test results, including:

  • OS/GRUB configuration:
    • maxcpus = n where n >= 0; limits the kernel to using ‘n’ processors. Only use exactly what you need.
    • isolcpus: Isolate CPUs from the general scheduler. Isolate all CPUs bar one which will be used by the OS.
    • use taskset to affinitize the forwarding application and the VNFs onto isolated cores. VNFs and the vSwitch should be allocated their own cores, i.e. must not share the same cores. vCPUs for the VNF should be affinitized to individual cores also.
    • Limit the amount of background applications that are running and set OS to boot to runlevel 3. Make sure to kill any unnecessary system processes/daemons.
    • Only enable hardware that you need to use for your test – to ensure there are no other interrupts on the system.
    • Configure NIC interrupts to only use the cores that are not allocated to any other process (VNF/vSwitch).
  • NUMA configuration: Any unused sockets in a multi-socket system should be disabled.
  • CPU pinning: The vSwitch and the VNF should each be affinitized to separate logical cores using a combination of maxcpus, isolcpus and taskset.
  • BIOS configuration: BIOS should be configured for performance where an explicit option exists, sleep states should be disabled, any virtualization optimization technologies should be enabled, and hyperthreading should also be enabled, turbo boost and overclocking should be disabled.

System Validation:

System validation is broken down into two sub-categories: Platform validation and VNF validation. The validation test itself involves verifying the forwarding capability and stability for the sub-system under test. The rationale behind system validation is two fold. Firstly to give a tester confidence in the stability of the platform or VNF that is being tested; and secondly to provide base performance comparison points to understand the overhead introduced by the virtual switch.

  • Benchmark platform forwarding capability: This is an OPTIONAL test used to verify the platform and measure the base performance (maximum forwarding rate in fps and latency) that can be achieved by the platform without a vSwitch or a VNF. The following diagram outlines the set-up for benchmarking Platform forwarding capability:

                                                         __
    +--------------------------------------------------+   |
    |   +------------------------------------------+   |   |
    |   |                                          |   |   |
    |   |          l2fw or DPDK L2FWD app          |   |  Host
    |   |                                          |   |   |
    |   +------------------------------------------+   |   |
    |   |                 NIC                      |   |   |
    +---+------------------------------------------+---+ __|
               ^                           :
               |                           |
               :                           v
    +--------------------------------------------------+
    |                                                  |
    |                traffic generator                 |
    |                                                  |
    +--------------------------------------------------+
    
  • Benchmark VNF forwarding capability: This test is used to verify the VNF and measure the base performance (maximum forwarding rate in fps and latency) that can be achieved by the VNF without a vSwitch. The performance metrics collected by this test will serve as a key comparison point for NIC passthrough technologies and vSwitches. VNF in this context refers to the hypervisor and the VM. The following diagram outlines the set-up for benchmarking VNF forwarding capability:

                                                         __
    +--------------------------------------------------+   |
    |   +------------------------------------------+   |   |
    |   |                                          |   |   |
    |   |                 VNF                      |   |   |
    |   |                                          |   |   |
    |   +------------------------------------------+   |   |
    |   |          Passthrough/SR-IOV              |   |  Host
    |   +------------------------------------------+   |   |
    |   |                 NIC                      |   |   |
    +---+------------------------------------------+---+ __|
               ^                           :
               |                           |
               :                           v
    +--------------------------------------------------+
    |                                                  |
    |                traffic generator                 |
    |                                                  |
    +--------------------------------------------------+
    

Methodology to benchmark Platform/VNF forwarding capability

The recommended methodology for the platform/VNF validation and benchmark is: - Run RFC2889 Maximum Forwarding Rate test, this test will produce maximum forwarding rate and latency results that will serve as the expected values. These expected values can be used in subsequent steps or compared with in subsequent validation tests. - Transmit bidirectional traffic at line rate/max forwarding rate (whichever is higher) for at least 72 hours, measure throughput (fps) and latency. - Note: Traffic should be bidirectional. - Establish a baseline forwarding rate for what the platform can achieve. - Additional validation: After the test has completed for 72 hours run bidirectional traffic at the maximum forwarding rate once more to see if the system is still functional and measure throughput (fps) and latency. Compare the measure the new obtained values with the expected values.

NOTE 1: How the Platform is configured for its forwarding capability test (BIOS settings, GRUB configuration, runlevel...) is how the platform should be configured for every test after this

NOTE 2: How the VNF is configured for its forwarding capability test (# of vCPUs, vNICs, Memory, affinitization…) is how it should be configured for every test that uses a VNF after this.

Methodology to benchmark the VNF to vSwitch to VNF deployment scenario

vsperf has identified the following concerns when benchmarking the VNF to vSwitch to VNF deployment scenario:

  • The accuracy of the timing synchronization between VNFs/VMs.
  • The clock accuracy of a VNF/VM if they were to be used as traffic generators.
  • VNF traffic generator/receiver may be using resources of the system under test, causing at least three forms of workload to increase as the traffic load increases (generation, switching, receiving).

The recommendation from vsperf is that tests for this sceanario must include an external HW traffic generator to act as the tester/traffic transmitter and receiver. The perscribed methodology to benchmark this deployment scanrio with an external tester involves the following three steps:

#. Determine the forwarding capability and latency through the virtual interface connected to the VNF/VM.

_images/vm2vm_virtual_interface_benchmark.png

Virtual interfaces performance benchmark

  1. Determine the forwarding capability and latency through the VNF/hypervisor.
_images/vm2vm_hypervisor_benchmark.png

Hypervisor performance benchmark

  1. Determine the forwarding capability and latency for the VNF to vSwitch to VNF taking the information from the previous two steps into account.
_images/vm2vm_benchmark.png

VNF to vSwitch to VNF performance benchmark

vsperf also identified an alternative configuration for the final step:

_images/vm2vm_alternative_benchmark.png

VNF to vSwitch to VNF alternative performance benchmark

4.2.11. Environment/infrastructure

VSPERF CI jobs are run using the OPNFV lab infrastructure as described by the ‘Pharos Project <https://www.opnfv.org/community/projects/pharos>`_ . A VSPERF POD is described here https://wiki.opnfv.org/display/pharos/VSPERF+in+Intel+Pharos+Lab+-+Pod+12

4.2.11.1. vsperf CI

vsperf CI jobs are broken down into:

  • Daily job:
    • Runs everyday takes about 10 hours to complete.
    • TESTCASES_DAILY=’phy2phy_tput back2back phy2phy_tput_mod_vlan phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back’.
    • TESTPARAM_DAILY=’–test-params TRAFFICGEN_PKT_SIZES=(64,128,512,1024,1518)’.
  • Merge job:
    • Runs whenever patches are merged to master.
    • Runs a basic Sanity test.
  • Verify job:
    • Runs every time a patch is pushed to gerrit.
    • Builds documentation.
4.2.11.2. Scripts:

There are 2 scripts that are part of VSPERFs CI:

  • build-vsperf.sh: Lives in the VSPERF repository in the ci/ directory and is used to run vsperf with the appropriate cli parameters.
  • vswitchperf.yml: YAML description of our jenkins job. lives in the RELENG repository.

More info on vsperf CI can be found here: https://wiki.opnfv.org/display/vsperf/VSPERF+CI

4.2.12. Responsibilities and authority

The group responsible for managing, designing, preparing and executing the tests listed in the LTD are the vsperf committers and contributors. The vsperf committers and contributors should work with the relavent OPNFV projects to ensure that the infrastructure is in place for testing vswitches, and that the results are published to common end point (a results database).

IETF RFC 8204

The IETF Benchmarking Methodology Working Group (BMWG) was re-chartered in 2014 to include benchmarking for Virtualized Network Functions (VNFs) and their infrastructure. A version of the VSPERF test specification was summarized in an Internet Draft ... Benchmarking Virtual Switches in OPNFV and contributed to the BMWG. In June 2017 the Internet Engineering Steering Group of the IETF approved the most recent version of the draft for publication as a new test specification (RFC 8204).

VSPERF CI Test Cases

CI Test cases run daily on the VSPERF Pharos POD for master and stable branches.

./results/scenario.rst ./results/results.rst

Yardstick

Yardstick Developer Guide
1. Introduction

Yardstick is a project dealing with performance testing. Yardstick produces its own test cases but can also be considered as a framework to support feature project testing.

Yardstick developed a test API that can be used by any OPNFV project. Therefore there are many ways to contribute to Yardstick.

You can:

  • Develop new test cases
  • Review codes
  • Develop Yardstick API / framework
  • Develop Yardstick grafana dashboards and Yardstick reporting page
  • Write Yardstick documentation

This developer guide describes how to interact with the Yardstick project. The first section details the main working areas of the project. The Second part is a list of “How to” to help you to join the Yardstick family whatever your field of interest is.

1.1. Where can I find some help to start?

This guide is made for you. You can have a look at the user guide. There are also references on documentation, video tutorials, tips in the project wiki page. You can also directly contact us by mail with [Yardstick] prefix in the title at opnfv-tech-discuss@lists.opnfv.org or on the IRC chan #opnfv-yardstick.

2. Yardstick developer areas
2.1. Yardstick framework

Yardstick can be considered as a framework. Yardstick is release as a docker file, including tools, scripts and a CLI to prepare the environement and run tests. It simplifies the integration of external test suites in CI pipeline and provide commodity tools to collect and display results.

Since Danube, test categories also known as tiers have been created to group similar tests, provide consistant sub-lists and at the end optimize test duration for CI (see How To section).

The definition of the tiers has been agreed by the testing working group.

The tiers are:

  • smoke
  • features
  • components
  • performance
  • vnf
3. How Todos?
3.1. How Yardstick works?

The installation and configuration of the Yardstick is described in the user guide.

3.2. How to work with test cases?

Sample Test cases

Yardstick provides many sample test cases which are located at “samples” directory of repo.

Sample test cases are designed as following goals:

  1. Helping user better understand yardstick features(including new feature and new test capacity).
  2. Helping developer to debug his new feature and test case before it is offical released.
  3. Helping other developers understand and verify the new patch before the patch merged.

So developers should upload your sample test case as well when they are trying to upload a new patch which is about the yardstick new test case or new feature.

OPNFV Release Test cases

OPNFV Release test cases which are located at “tests/opnfv/test_cases” of repo. those test cases are runing by OPNFV CI jobs, It means those test cases should be more mature than sample test cases. OPNFV scenario owners can select related test cases and add them into the test suites which is represent the scenario.

Test case Description File

This section will introduce the meaning of the Test case description file. we will use ping.yaml as a example to show you how to understand the test case description file. In this Yaml file, you can easily find it consists of two sections. One is “Scenarios”, the other is “Context”.:

---
  # Sample benchmark task config file
  # measure network latency using ping

  schema: "yardstick:task:0.1"

  {% set provider = provider or none %}
  {% set physical_network = physical_network or 'physnet1' %}
  {% set segmentation_id = segmentation_id or none %}
  scenarios:
  -
    type: Ping
    options:
      packetsize: 200
    host: athena.demo
    target: ares.demo

    runner:
      type: Duration
      duration: 60
      interval: 1

    sla:
      max_rtt: 10
      action: monitor

  context:
    name: demo
    image: cirros-0.3.5
    flavor: yardstick-flavor
    user: cirros

    placement_groups:
      pgrp1:
        policy: "availability"

    servers:
      athena:
        floating_ip: true
        placement: "pgrp1"
      ares:
        placement: "pgrp1"

    networks:
      test:
        cidr: '10.0.1.0/24'
        {% if provider == "vlan" %}
        provider: {{provider}}
        physical_network: {{physical_network}}
          {% if segmentation_id %}
        segmentation_id: {{segmentation_id}}
          {% endif %}
       {% endif %}

“Contexts” section is the description of pre-condition of testing. As ping.yaml shown, you can configure the image, flavor , name ,affinity and network of Test VM(servers), with this section, you will get a pre-condition env for Testing. Yardstick will automatic setup the stack which are described in this section. In fact, yardstick use convert this section to heat template and setup the VMs by heat-client (Meanwhile, yardstick can support to convert this section to Kubernetes template to setup containers).

Two Test VMs(athena and ares) are configured by keyword “servers”. “flavor” will determine how many vCPU, how much memory for test VMs. As “yardstick-flavor” is a basic flavor which will be automatically created when you run command “yardstick env prepare”. “yardstick-flavor” is “1 vCPU 1G RAM,3G Disk”. “image” is the image name of test VMs. if you use cirros.3.5.0, you need fill the username of this image into “user”. the “policy” of placement of Test VMs have two values (affinity and availability). “availability” means anti-affinity. In “network” section, you can configure which provide network and physical_network you want Test VMs use. you may need to configure segmentation_id when your network is vlan.

Moreover, you can configure your specific flavor as below, yardstick will setup the stack for you.

flavor:
  name: yardstick-new-flavor
  vcpus: 12
  ram: 1024
  disk: 2

Besides default heat stack, yardstick also allow you to setup other two types stack. they are “Node” and “Kubernetes”.

context:
  type: Kubernetes
  name: k8s

and

context:
  type: Node
  name: LF

“Scenarios” section is the description of testing step, you can orchestrate the complex testing step through orchestrate scenarios.

Each scenario will do one testing step, In one scenario, you can configure the type of scenario(operation), runner type and SLA of the scenario.

For TC002, We only have one step , that is Ping from host VM to target VM. In this step, we also have some detail operation implement ( such as ssh to VM, ping from VM1 to VM2. Get the latency, verify the SLA, report the result).

If you want to get this detail implement , you can check with the scenario.py file. For Ping scenario, you can find it in yardstick repo ( yardstick / yardstick / benchmark / scenarios / networking / ping.py)

after you select the type of scenario( such as Ping), you will select one type of runner, there are 4 types of runner. Usually, we use the “Iteration” and “Duration”. and Default is “Iteration”. For Iteration, you can specify the iteration number and interval of iteration.

runner:
  type: Iteration
  iterations: 10
  interval: 1

That means yardstick will iterate the 10 times of Ping test and the interval of each iteration is one second.

For Duration, you can specify the duration of this scenario and the interval of each ping test.

runner:
  type: Duration
  duration: 60
  interval: 10

That means yardstick will run the ping test as loop until the total time of this scenario reach the 60s and the interval of each loop is ten seconds.

SLA is the criterion of this scenario. that depends on the scenario. different scenario can have different SLA metric.

How to write a new test case

Yardstick already provide a library of testing step. that means yardstick provide lots of type scenario.

Basiclly, What you need to do is to orchestrate the scenario from the library.

Here, We will show two cases. One is how to write a simple test case, the other is how to write a quite complex test case.

Write a new simple test case

First, you can image a basic test case description as below.

Storage Performance
metric IOPS (Average IOs performed per second), Throughput (Average disk read/write bandwidth rate), Latency (Average disk read/write latency)
test purpose The purpose of TC005 is to evaluate the IaaS storage performance with regards to IOPS, throughput and latency.
test description fio test is invoked in a host VM on a compute blade, a job file as well as parameters are passed to fio and fio will start doing what the job file tells it to do.
configuration

file: opnfv_yardstick_tc005.yaml

IO types is set to read, write, randwrite, randread, rw. IO block size is set to 4KB, 64KB, 1024KB. fio is run for each IO type and IO block size scheme, each iteration runs for 30 seconds (10 for ramp time, 20 for runtime).

For SLA, minimum read/write iops is set to 100, minimum read/write throughput is set to 400 KB/s, and maximum read/write latency is set to 20000 usec.

applicability

This test case can be configured with different:

  • IO types;
  • IO block size;
  • IO depth;
  • ramp time;
  • test duration.

Default values exist.

SLA is optional. The SLA in this test case serves as an example. Considerably higher throughput and lower latency are expected. However, to cover most configurations, both baremetal and fully virtualized ones, this value should be possible to achieve and acceptable for black box testing. Many heavy IO applications start to suffer badly if the read/write bandwidths are lower than this.

pre-test conditions

The test case image needs to be installed into Glance with fio included in it.

No POD specific requirements have been identified.

test sequence description and expected result
step 1 A host VM with fio installed is booted.
step 2 Yardstick is connected with the host VM by using ssh. ‘fio_benchmark’ bash script is copyied from Jump Host to the host VM via the ssh tunnel.
step 3

‘fio_benchmark’ script is invoked. Simulated IO operations are started. IOPS, disk read/write bandwidth and latency are recorded and checked against the SLA. Logs are produced and stored.

Result: Logs are stored.

step 4 The host VM is deleted.
test verdict Fails only if SLA is not passed, or if there is a test case execution problem.

TODO

3.3. How can I contribute to Yardstick?

If you are already a contributor of any OPNFV project, you can contribute to Yardstick. If you are totally new to OPNFV, you must first create your Linux Foundation account, then contact us in order to declare you in the repository database.

We distinguish 2 levels of contributors:

  • the standard contributor can push patch and vote +1/0/-1 on any Yardstick patch
  • The commitor can vote -2/-1/0/+1/+2 and merge

Yardstick commitors are promoted by the Yardstick contributors.

3.3.1. Gerrit & JIRA introduction

OPNFV uses Gerrit for web based code review and repository management for the Git Version Control System. You can access OPNFV Gerrit. Please note that you need to have Linux Foundation ID in order to use OPNFV Gerrit. You can get one from this link.

OPNFV uses JIRA for issue management. An important principle of change management is to have two-way trace-ability between issue management (i.e. JIRA) and the code repository (via Gerrit). In this way, individual commits can be traced to JIRA issues and we also know which commits were used to resolve a JIRA issue.

If you want to contribute to Yardstick, you can pick a issue from Yardstick’s JIRA dashboard or you can create you own issue and submit it to JIRA.

3.3.2. Install Git and Git-reviews

Installing and configuring Git and Git-Review is necessary in order to submit code to Gerrit. The Getting to the code page will provide you with some help for that.

3.3.3. Verify your patch locally before submitting

Once you finish a patch, you can submit it to Gerrit for code review. A developer sends a new patch to Gerrit will trigger patch verify job on Jenkins CI. The yardstick patch verify job includes python flake8 check, unit test and code coverage test. Before you submit your patch, it is recommended to run the patch verification in your local environment first.

Open a terminal window and set the project’s directory to the working directory using the cd command. Assume that YARDSTICK_REPO_DIR is the path to the Yardstick project folder on your computer:

cd $YARDSTICK_REPO_DIR

Verify your patch:

tox

It is used in CI but also by the CLI.

3.3.4. Submit the code with Git

Tell Git which files you would like to take into account for the next commit. This is called ‘staging’ the files, by placing them into the staging area, using the git add command (or the synonym git stage command):

git add $YARDSTICK_REPO_DIR/samples/sample.yaml

Alternatively, you can choose to stage all files that have been modified (that is the files you have worked on) since the last time you generated a commit, by using the -a argument:

git add -a

Git won’t let you push (upload) any code to Gerrit if you haven’t pulled the latest changes first. So the next step is to pull (download) the latest changes made to the project by other collaborators using the pull command:

git pull

Now that you have the latest version of the project and you have staged the files you wish to push, it is time to actually commit your work to your local Git repository:

git commit --signoff -m "Title of change"

Test of change that describes in high level what was done. There is a lot of
documentation in code so you do not need to repeat it here.

JIRA: YARDSTICK-XXX

The message that is required for the commit should follow a specific set of rules. This practice allows to standardize the description messages attached to the commits, and eventually navigate among the latter more easily.

This document happened to be very clear and useful to get started with that.

3.3.5. Push the code to Gerrit for review

Now that the code has been comitted into your local Git repository the following step is to push it online to Gerrit for it to be reviewed. The command we will use is git review:

git review

This will automatically push your local commit into Gerrit. You can add Yardstick committers and contributors to review your codes.

Gerrit for code review

You can find Yardstick people info here.

3.3.6. Modify the code under review in Gerrit

At the same time the code is being reviewed in Gerrit, you may need to edit it to make some changes and then send it back for review. The following steps go through the procedure.

Once you have modified/edited your code files under your IDE, you will have to stage them. The ‘status’ command is very helpful at this point as it provides an overview of Git’s current state:

git status

The output of the command provides us with the files that have been modified after the latest commit.

You can now stage the files that have been modified as part of the Gerrit code review edition/modification/improvement using git add command. It is now time to commit the newly modified files, but the objective here is not to create a new commit, we simply want to inject the new changes into the previous commit. You can achieve that with the ‘–amend’ option on the git commit command:

git commit --amend

If the commit was successful, the git status command should not return the updated files as about to be commited.

The final step consists in pushing the newly modified commit to Gerrit:

git review
4. Plugins

For information about Yardstick plugins, refer to the chapter Installing a plug-in into Yardstick in the user guide.

Developer

Documentation Guide

Documentation Guide

This page intends to cover the documentation handling for OPNFV. OPNFV projects are expected to create a variety of document types, according to the nature of the project. Some of these are common to projects that develop/integrate features into the OPNFV platform, e.g. Installation Instructions and User/Configurations Guides. Other document types may be project-specific.

Getting Started with Documentation for Your Project

OPNFV documentation is automated and integrated into our git & gerrit toolchains.

We use RST document templates in our repositories and automatically render to HTML and PDF versions of the documents in our artifact store, our WiKi is also able to integrate these rendered documents directly allowing projects to use the revision controlled documentation process for project information, content and deliverables. Read this page which elaborates on how documentation is to be included within opnfvdocs.

Licencing your documentation

All contributions to the OPNFV project are done in accordance with the OPNFV licensing requirements. Documentation in OPNFV is contributed in accordance with the Creative Commons 4.0 and the `SPDX https://spdx.org/>`_ licence. All documentation files need to be licensed using the text below. The license may be applied in the first lines of all contributed RST files:

.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
.. (c) <optionally add copywriters name>

These lines will not be rendered in the html and pdf files.
How and where to store the document content files in your repository

All documentation for your project should be structured and stored in the <repo>/docs/ directory. The documentation toolchain will look in these directories and be triggered on events in these directories when generating documents.

Document structure and contribution

A general structure is proposed for storing and handling documents that are common across many projects but also for documents that may be project specific. The documentation is divided into three areas Release, Development and Testing. Templates for these areas can be found under opnfvdocs/docs/templates/.

Project teams are encouraged to use templates provided by the opnfvdocs project to ensure that there is consistency across the community. Following representation shows the expected structure:

docs/
├── development
│   ├── design
│   ├── overview
│   └── requirements
├── release
│   ├── configguide
│   ├── installation
│   ├── release-notes
│   ├── scenarios
│   │   └── scenario.name
│   └── userguide
├── testing
│   ├── developer
│   └── user
└── infrastructure
    ├── hardware-infrastructure
    ├── software-infrastructure
    ├── continuous-integration
    └── cross-community-continuous-integration
Release documentation

Release documentation is the set of documents that are published for each OPNFV release. These documents are created and developed following the OPNFV release process and milestones and should reflect the content of the OPNFV release. These documents have a master index.rst file in the <opnfvdocs> repository and extract content from other repositories. To provide content into these documents place your <content>.rst files in a directory in your repository that matches the master document and add a reference to that file in the correct place in the corresponding index.rst file in opnfvdocs/docs/release/.

Platform Overview: opnfvdocs/docs/release/overview

  • Note this document is not a contribution driven document
  • Content for this is prepared by the Marketing team together with the opnfvdocs team

Installation Instruction: <repo>/docs/release/installation

  • Folder for documents describing how to deploy each installer and scenario descriptions
  • Release notes will be included here <To Confirm>
  • Security related documents will be included here
  • Note that this document will be compiled into ‘OPNFV Installation Instruction’

User Guide: <repo>/docs/release/userguide

  • Folder for manuals to use specific features
  • Folder for documents describing how to install/configure project specific components and features
  • Can be the directory where API reference for project specific features are stored
  • Note this document will be compiled into ‘OPNFV userguide’

Configuration Guide: <repo>/docs/release/configguide

  • Brief introduction to configure OPNFV with its dependencies.

Release Notes: <repo>/docs/release/release-notes

  • Changes brought about in the release cycle.
  • Include version details.
Testing documentation

Documentation created by test projects can be stored under two different sub directories /user or /developemnt. Release notes will be stored under <repo>/docs/release/release-notes

User documentation: <repo>/testing/user/ Will collect the documentation of the test projects allowing the end user to perform testing towards a OPNFV SUT e.g. Functest/Yardstick/Vsperf/Storperf/Bottlenecks/Qtip installation/config & user guides.

Development documentation: <repo>/testing/developent/ Will collect documentation to explain how to create your own test case and leverage existing testing frameworks e.g. developer guides.

Development Documentation

Project specific documents such as design documentation, project overview or requirement documentation can be stored under /docs/development. Links to generated documents will be dislayed under Development Documentaiton section on docs.opnfv.org. You are encouraged to establish the following basic structure for your project as needed:

Requirement Documentation: <repo>/docs/development/requirements/

  • Folder for your requirement documentation
  • For details on requirements projects’ structures see the Requirements Projects page.

Design Documentation: <repo>/docs/development/design

  • Folder for your upstream design documents (blueprints, development proposals, etc..)

Project overview: <repo>/docs/development/overview

  • Folder for any project specific documentation.
Infrastructure Documentation

Infrastructure documentation can be stored under <repo>/docs/ folder of corresponding infrastructure project.

Including your Documentation

In your project repository

Add your documentation to your repository in the folder structure and according to the templates listed above. The documentation templates you will require are available in opnfvdocs/docs/templates/ repository, you should copy the relevant templates to your <repo>/docs/ directory in your repository. For instance if you want to document userguide, then your steps shall be as follows:

git clone ssh://<your_id>@gerrit.opnfv.org:29418/opnfvdocs.git
cp -p opnfvdocs/docs/userguide/* <my_repo>/docs/userguide/

You should then add the relevant information to the template that will explain the documentation. When you are done writing, you can commit the documentation to the project repository.

git add .
git commit --signoff --all
git review
In OPNFVDocs Composite Documentation
In toctree
To import project documents from project repositories, we use submodules.
Each project is stored in opnfvdocs/docs/submodule/ as follows:
_images/Submodules.jpg

To include your project specific documentation in the composite documentation, first identify where your project documentation should be included. Say your project userguide should figure in the ‘OPNFV Userguide’, then:

vim opnfvdocs/docs/release/userguide.introduction.rst

This opens the text editor. Identify where you want to add the userguide. If the userguide is to be added to the toctree, simply include the path to it, example:

.. toctree::
    :maxdepth: 1

 submodules/functest/docs/userguide/index
 submodules/bottlenecks/docs/userguide/index
 submodules/yardstick/docs/userguide/index
 <submodules/path-to-your-file>
‘doc8’ Validation

It is recommended that all rst content is validated by doc8 standards. To validate your rst files using doc8, install doc8.

sudo pip install doc8

doc8 can now be used to check the rst files. Execute as,

doc8 --ignore D000,D001 <file>
Testing: Build Documentation Locally
Composite OPNFVDOCS documentation

To build whole documentation under opnfvdocs/, follow these steps:

Install virtual environment.

sudo pip install virtualenv
cd /local/repo/path/to/project

Download the OPNFVDOCS repository.

git clone https://gerrit.opnfv.org/gerrit/opnfvdocs

Change directory to opnfvdocs & install requirements.

cd opnfvdocs
sudo pip install -r etc/requirements.txt

Update submodules, build documentation using tox & then open using any browser.

cd opnfvdocs
git submodule update --init
tox -edocs
firefox docs/_build/html/index.html

Note

Make sure to run tox -edocs and not just tox.

Individual project documentation

To test how the documentation renders in HTML, follow these steps:

Install virtual environment.

sudo pip install virtualenv
cd /local/repo/path/to/project

Download the opnfvdocs repository.

git clone https://gerrit.opnfv.org/gerrit/opnfvdocs

Change directory to opnfvdocs & install requirements.

cd opnfvdocs
sudo pip install -r etc/requirements.txt

Move the conf.py file to your project folder where RST files have been kept:

mv opnfvdocs/docs/conf.py <path-to-your-folder>/

Move the static files to your project folder:

mv opnfvdocs/_static/ <path-to-your-folder>/

Build the documentation from within your project folder:

sphinx-build -b html <path-to-your-folder> <path-to-output-folder>

Your documentation shall be built as HTML inside the specified output folder directory.

Note

Be sure to remove the conf.py, the static/ files and the output folder from the <project>/docs/. This is for testing only. Only commit the rst files and related content.

Adding your project repository as a submodule

Clone the opnfvdocs repository and your submodule to .gitmodules following the convention of the file

cd docs/submodules/
git submodule add https://gerrit.opnfv.org/gerrit/$reponame
git submodule init $reponame/
git submodule update $reponame/
git add .
git commit -sv
git review
Removing a project repository as a submodule
git rm docs/submodules/$reponame rm -rf .git/modules/$reponame git config -f .git/config –remove-section submodule.$reponame 2> /dev/null git add . git commit -sv git review

Addendum

Index File

The index file must relatively refence your other rst files in that directory.

Here is an example index.rst :

*******************
Documentation Title
*******************

.. toctree::
   :numbered:
   :maxdepth: 2

   documentation-example
Source Files

Document source files have to be written in reStructuredText format (rst). Each file would be build as an html page.

Here is an example source rst file :

=============
Chapter Title
=============

Section Title
=============

Subsection Title
----------------

Hello!
Writing RST Markdown

See http://sphinx-doc.org/rest.html .

Hint: You can add dedicated contents by using ‘only’ directive with build type (‘html’ and ‘singlehtml’) for OPNFV document. But, this is not encouraged to use since this may make different views.

.. only:: html
    This line will be shown only in html version.
Verify Job

The verify job name is docs-verify-rtd-{branch}.

When you send document changes to gerrit, jenkins will create your documents in HTML formats (normal and single-page) to verify that new document can be built successfully. Please check the jenkins log and artifact carefully. You can improve your document even though if the build job succeeded.

Merge Job

The merge job name is docs-merge-rtd-{branch}.

Once the patch is merged, jenkins will automatically trigger building of the new documentation. This might take about 15 minutes while readthedocs builds the documentatation. The newly built documentation shall show up as appropriate placed in docs.opnfv.org/{branch}/path-to-file.

OPNFV Projects

Apex

OPNFV Installation instructions (Apex)

Contents:

1. Abstract

This document describes how to install the Euphrates release of OPNFV when using Apex as a deployment tool covering it’s limitations, dependencies and required system resources.

2. License

Euphrates release of OPNFV when using Apex as a deployment tool Docs (c) by Tim Rozet (Red Hat) and Dan Radez (Red Hat)

Euphrates release of OPNFV when using Apex as a deployment tool Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>.

3. Introduction

This document describes the steps to install an OPNFV Euphrates reference platform, as defined by the Genesis Project using the Apex installer.

The audience is assumed to have a good background in networking and Linux administration.

4. Preface

Apex uses Triple-O from the RDO Project OpenStack distribution as a provisioning tool. The Triple-O image based life cycle installation tool provisions an OPNFV Target System (3 controllers, 2 or more compute nodes) with OPNFV specific configuration provided by the Apex deployment tool chain.

The Apex deployment artifacts contain the necessary tools to deploy and configure an OPNFV target system using the Apex deployment toolchain. These artifacts offer the choice of using the Apex bootable ISO (opnfv-apex-euphrates.iso) to both install CentOS 7 and the necessary materials to deploy or the Apex RPMs (opnfv-apex*.rpm), and their associated dependencies, which expects installation to a CentOS 7 libvirt enabled host. The RPM contains a collection of configuration files, prebuilt disk images, and the automatic deployment script (opnfv-deploy).

An OPNFV install requires a “Jump Host” in order to operate. The bootable ISO will allow you to install a customized CentOS 7 release to the Jump Host, which includes the required packages needed to run opnfv-deploy. If you already have a Jump Host with CentOS 7 installed, you may choose to skip the ISO step and simply install the (opnfv-apex*.rpm) RPMs. The RPMs are the same RPMs included in the ISO and include all the necessary disk images and configuration files to execute an OPNFV deployment. Either method will prepare a host to the same ready state for OPNFV deployment.

opnfv-deploy instantiates a Triple-O Undercloud VM server using libvirt as its provider. This VM is then configured and used to provision the OPNFV target deployment (3 controllers, n compute nodes). These nodes can be either virtual or bare metal. This guide contains instructions for installing either method.

5. Triple-O Deployment Architecture

Apex is based on the OpenStack Triple-O project as distributed by the RDO Project. It is important to understand the basics of a Triple-O deployment to help make decisions that will assist in successfully deploying OPNFV.

Triple-O stands for OpenStack On OpenStack. This means that OpenStack will be used to install OpenStack. The target OPNFV deployment is an OpenStack cloud with NFV features built-in that will be deployed by a smaller all-in-one deployment of OpenStack. In this deployment methodology there are two OpenStack installations. They are referred to as the undercloud and the overcloud. The undercloud is used to deploy the overcloud.

The undercloud is the all-in-one installation of OpenStack that includes baremetal provisioning capability. The undercloud will be deployed as a virtual machine on a Jump Host. This VM is pre-built and distributed as part of the Apex RPM.

The overcloud is OPNFV. Configuration will be passed into undercloud and the undercloud will use OpenStack’s orchestration component, named Heat, to execute a deployment that will provision the target OPNFV nodes.

6. Apex High Availability Architecture
6.1. Undercloud

The undercloud is not Highly Available. End users do not depend on the undercloud. It is only for management purposes.

6.2. Overcloud

Apex will deploy three control nodes in an HA deployment. Each of these nodes will run the following services:

  • Stateless OpenStack services
  • MariaDB / Galera
  • RabbitMQ
  • OpenDaylight
  • HA Proxy
  • Pacemaker & VIPs
  • Ceph Monitors and OSDs
Stateless OpenStack services
All running stateless OpenStack services are load balanced by HA Proxy. Pacemaker monitors the services and ensures that they are running.
Stateful OpenStack services
All running stateful OpenStack services are load balanced by HA Proxy. They are monitored by pacemaker in an active/passive failover configuration.
MariaDB / Galera
The MariaDB database is replicated across the control nodes using Galera. Pacemaker is responsible for a proper start up of the Galera cluster. HA Proxy provides and active/passive failover methodology to connections to the database.
RabbitMQ
The message bus is managed by Pacemaker to ensure proper start up and establishment of clustering across cluster members.
OpenDaylight
OpenDaylight is currently installed on all three control nodes and started as an HA cluster unless otherwise noted for that scenario. OpenDaylight’s database, known as MD-SAL, breaks up pieces of the database into “shards”. Each shard will have its own election take place, which will determine which OpenDaylight node is the leader for that shard. The other OpenDaylight nodes in the cluster will be in standby. Every Open vSwitch node connects to every OpenDaylight to enable HA.
HA Proxy
HA Proxy is monitored by Pacemaker to ensure it is running across all nodes and available to balance connections.
Pacemaker & VIPs
Pacemaker has relationships and restraints setup to ensure proper service start up order and Virtual IPs associated with specific services are running on the proper host.
Ceph Monitors & OSDs
The Ceph monitors run on each of the control nodes. Each control node also has a Ceph OSD running on it. By default the OSDs use an autogenerated virtual disk as their target device. A non-autogenerated device can be specified in the deploy file.

VM Migration is configured and VMs can be evacuated as needed or as invoked by tools such as heat as part of a monitored stack deployment in the overcloud.

7. OPNFV Scenario Architecture

OPNFV distinguishes different types of SDN controllers, deployment options, and features into “scenarios”. These scenarios are universal across all OPNFV installers, although some may or may not be supported by each installer.

The standard naming convention for a scenario is: <VIM platform>-<SDN type>-<feature>-<ha/noha>

The only supported VIM type is “OS” (OpenStack), while SDN types can be any supported SDN controller. “feature” includes things like ovs_dpdk, sfc, etc. “ha” or “noha” determines if the deployment will be highly available. If “ha” is used at least 3 control nodes are required.

8. OPNFV Scenarios in Apex

Apex provides pre-built scenario files in /etc/opnfv-apex which a user can select from to deploy the desired scenario. Simply pass the desired file to the installer as a (-d) deploy setting. Read further in the Apex documentation to learn more about invoking the deploy command. Below is quick reference matrix for OPNFV scenarios supported in Apex. Please refer to the respective OPNFV Docs documentation for each scenario in order to see a full scenario description. Also, please refer to release notes for information about known issues per scenario. The following scenarios correspond to a supported <Scenario>.yaml deploy settings file:

Scenario Owner Supported
os-nosdn-nofeature-ha Apex Yes
os-nosdn-nofeature-noha Apex Yes
os-nosdn-bar-ha Barometer Yes
os-nosdn-bar-noha Barometer Yes
os-nosdn-calipso-noha Calipso Yes
os-nosdn-ovs_dpdk-ha Apex Yes
os-nosdn-ovs_dpdk-noha Apex Yes
os-nosdn-fdio-ha FDS No
os-nosdn-fdio-noha FDS No
os-nosdn-kvm_ovs_dpdk-ha KVM for NFV Yes
os-nosdn-kvm_ovs_dpdk -noha KVM for NFV Yes
os-nosdn-performance-ha Apex Yes
os-odl-nofeature-ha Apex Yes
os-odl-nofeature-noha Apex Yes
os-odl-ovs_dpdk-ha Apex No
os-odl-ovs_dpdk-noha Apex No
os-odl-bgpvpn-ha SDNVPN Yes
os-odl-bgpvpn-noha SDNVPN Yes
os-odl-sfc-ha SFC No
os-odl-sfc-noha SFC Yes
os-odl-gluon-noha Gluon No
os-odl-csit-noha Apex No
os-odl-fdio-ha FDS Yes
os-odl-fdio-noha FDS Yes
os-odl-fdio_dvr-ha FDS No
os-odl-fdio_dvr-noha FDS No
os-onos-nofeature-ha ONOSFW No
os-onos-sfc-ha ONOSFW No
os-ovn-nofeature-noha Apex Yes
9. Setup Requirements
9.1. Jump Host Requirements

The Jump Host requirements are outlined below:

  1. CentOS 7 (from ISO or self-installed).
  2. Root access.
  3. libvirt virtualization support.
  4. minimum 1 networks and maximum 5 networks, multiple NIC and/or VLAN combinations are supported. This is virtualized for a VM deployment.
  5. The Euphrates Apex RPMs and their dependencies.
  6. 16 GB of RAM for a bare metal deployment, 64 GB of RAM for a Virtual Deployment.
9.2. Network Requirements

Network requirements include:

  1. No DHCP or TFTP server running on networks used by OPNFV.
  2. 1-5 separate networks with connectivity between Jump Host and nodes.
    • Control Plane (Provisioning)
    • Private Tenant-Networking Network*
    • External Network*
    • Storage Network*
    • Internal API Network* (required for IPv6 **)
  3. Lights out OOB network access from Jump Host with IPMI node enabled (bare metal deployment only).
  4. External network is a routable network from outside the cloud, deployment. The External network is where public internet access would reside if available.

*These networks can be combined with each other or all combined on the Control Plane network.

**Internal API network, by default, is collapsed with provisioning in IPv4 deployments, this is not possible with the current lack of PXE boot support and therefore the API network is required to be its own network in an IPv6 deployment.

9.3. Bare Metal Node Requirements

Bare metal nodes require:

  1. IPMI enabled on OOB interface for power control.
  2. BIOS boot priority should be PXE first then local hard disk.
  3. BIOS PXE interface should include Control Plane network mentioned above.
9.4. Execution Requirements (Bare Metal Only)

In order to execute a deployment, one must gather the following information:

  1. IPMI IP addresses for the nodes.
  2. IPMI login information for the nodes (user/pass).
  3. MAC address of Control Plane / Provisioning interfaces of the overcloud nodes.
10. Installation High-Level Overview - Bare Metal Deployment

The setup presumes that you have 6 or more bare metal servers already setup with network connectivity on at least 1 or more network interfaces for all servers via a TOR switch or other network implementation.

The physical TOR switches are not automatically configured from the OPNFV reference platform. All the networks involved in the OPNFV infrastructure as well as the provider networks and the private tenant VLANs needs to be manually configured.

The Jump Host can be installed using the bootable ISO or by using the (opnfv-apex*.rpm) RPMs and their dependencies. The Jump Host should then be configured with an IP gateway on its admin or public interface and configured with a working DNS server. The Jump Host should also have routable access to the lights out network for the overcloud nodes.

opnfv-deploy is then executed in order to deploy the undercloud VM and to provision the overcloud nodes. opnfv-deploy uses three configuration files in order to know how to install and provision the OPNFV target system. The information gathered under section Execution Requirements (Bare Metal Only) is put into the YAML file /etc/opnfv-apex/inventory.yaml configuration file. Deployment options are put into the YAML file /etc/opnfv-apex/deploy_settings.yaml. Alternatively there are pre-baked deploy_settings files available in /etc/opnfv-apex/. These files are named with the naming convention os-sdn_controller-enabled_feature-[no]ha.yaml. These files can be used in place of the /etc/opnfv-apex/deploy_settings.yaml file if one suites your deployment needs. Networking definitions gathered under section Network Requirements are put into the YAML file /etc/opnfv-apex/network_settings.yaml. opnfv-deploy will boot the undercloud VM and load the target deployment configuration into the provisioning toolchain. This information includes MAC address, IPMI, Networking Environment and OPNFV deployment options.

Once configuration is loaded and the undercloud is configured it will then reboot the overcloud nodes via IPMI. The nodes should already be set to PXE boot first off the admin interface. The nodes will first PXE off of the undercloud PXE server and go through a discovery/introspection process.

Introspection boots off of custom introspection PXE images. These images are designed to look at the properties of the hardware that is being booted and report the properties of it back to the undercloud node.

After introspection the undercloud will execute a Heat Stack Deployment to continue node provisioning and configuration. The nodes will reboot and PXE from the undercloud PXE server again to provision each node using Glance disk images provided by the undercloud. These disk images include all the necessary packages and configuration for an OPNFV deployment to execute. Once the disk images have been written to node’s disks the nodes will boot locally and execute cloud-init which will execute the final node configuration. This configuration is largely completed by executing a puppet apply on each node.

11. Installation Guide - Bare Metal Deployment

This section goes step-by-step on how to correctly install and provision the OPNFV target system to bare metal nodes.

11.1. Install Bare Metal Jump Host
1a. If your Jump Host does not have CentOS 7 already on it, or you would like
to do a fresh install, then download the Apex bootable ISO from the OPNFV artifacts site <http://artifacts.opnfv.org/apex.html>. There have been isolated reports of problems with the ISO having trouble completing installation successfully. In the unexpected event the ISO does not work please workaround this by downloading the CentOS 7 DVD and performing a “Virtualization Host” install. If you perform a “Minimal Install” or install type other than “Virtualization Host” simply run sudo yum -y groupinstall "Virtualization Host" chkconfig libvirtd on && reboot to install virtualization support and enable libvirt on boot. If you use the CentOS 7 DVD proceed to step 1b once the CentOS 7 with “Virtualization Host” support is completed.
1b. Boot the ISO off of a USB or other installation media and walk through

installing OPNFV CentOS 7. The ISO comes prepared to be written directly to a USB drive with dd as such:

dd if=opnfv-apex.iso of=/dev/sdX bs=4M

Replace /dev/sdX with the device assigned to your usb drive. Then select the USB device as the boot media on your Jump Host

2a. When not using the OPNFV Apex ISO, install these repos:

sudo yum install https://repos.fedorapeople.org/repos/openstack/openstack-pike/rdo-release-pike-1.noarch.rpm sudo yum install epel-release sudo curl -o /etc/yum/repos.d/opnfv-apex.repo http://artifacts.opnfv.org/apex/euphrates/opnfv-apex.repo

The RDO Project release repository is needed to install OpenVSwitch, which is a dependency of opnfv-apex. If you do not have external connectivity to use this repository you need to download the OpenVSwitch RPM from the RDO Project repositories and install it with the opnfv-apex RPM. The opnfv-apex repo hosts all of the Apex dependencies which will automatically be installed when installing RPMs, but will be pre-installed with the ISO.

2b. If you chose not to use the Apex ISO, then you must download and install

the Apex RPMs to the Jump Host. Download the first 3 Apex RPMs from the OPNFV downloads page, under the TripleO RPMs https://www.opnfv.org/software/downloads. The following RPMs are available for installation:

  • opnfv-apex - OpenDaylight, OVN, and nosdn support
  • opnfv-apex-undercloud - (reqed) Undercloud Image
  • python34-opnfv-apex - (reqed) OPNFV Apex Python package
  • python34-markupsafe - (reqed) Dependency of python34-opnfv-apex **
  • python34-jinja2 - (reqed) Dependency of python34-opnfv-apex **
  • python3-ipmi - (reqed) Dependency of python34-opnfv-apex **
  • python34-pbr - (reqed) Dependency of python34-opnfv-apex **
  • python34-virtualbmc - (reqed) Dependency of python34-opnfv-apex **
  • python34-iptables - (reqed) Dependency of python34-opnfv-apex **
  • python34-cryptography - (reqed) Dependency of python34-opnfv-apex **
  • python34-libvirt - (reqed) Dependency of python34-opnfv-apex **

** These RPMs are not yet distributed by CentOS or EPEL. Apex has built these for distribution with Apex while CentOS and EPEL do not distribute them. Once they are carried in an upstream channel Apex will no longer carry them and they will not need special handling for installation. You do not need to explicitly install these as they will be automatically installed by installing python34-opnfv-apex when the opnfv-apex.repo has been previously downloaded to /etc/yum.repos.d/.

Install the three required RPMs (replace <rpm> with the actual downloaded artifact): yum -y install <opnfv-apex.rpm> <opnfv-apex-undercloud> <python34-opnfv-apex>

  1. After the operating system and the opnfv-apex RPMs are installed, login to your Jump Host as root.
  2. Configure IP addresses on the interfaces that you have selected as your networks.
  3. Configure the IP gateway to the Internet either, preferably on the public interface.
  4. Configure your /etc/resolv.conf to point to a DNS server (8.8.8.8 is provided by Google).
11.2. Creating a Node Inventory File

IPMI configuration information gathered in section Execution Requirements (Bare Metal Only) needs to be added to the inventory.yaml file.

  1. Copy /usr/share/doc/opnfv/inventory.yaml.example as your inventory file template to /etc/opnfv-apex/inventory.yaml.

  2. The nodes dictionary contains a definition block for each baremetal host that will be deployed. 1 or more compute nodes and 3 controller nodes are required. (The example file contains blocks for each of these already). It is optional at this point to add more compute nodes into the node list.

  3. Edit the following values for each node:

    • mac_address: MAC of the interface that will PXE boot from undercloud

    • ipmi_ip: IPMI IP Address

    • ipmi_user: IPMI username

    • ipmi_password: IPMI password

    • pm_type: Power Management driver to use for the node

      values: pxe_ipmitool (tested) or pxe_wol (untested) or pxe_amt (untested)

    • cpus: (Introspected*) CPU cores available

    • memory: (Introspected*) Memory available in Mib

    • disk: (Introspected*) Disk space available in Gb

    • disk_device: (Opt***) Root disk device to use for installation

    • arch: (Introspected*) System architecture

    • capabilities: (Opt**) Node’s role in deployment

      values: profile:control or profile:compute

    * Introspection looks up the overcloud node’s resources and overrides these value. You can leave default values and Apex will get the correct values when it runs introspection on the nodes.

    ** If capabilities profile is not specified then Apex will select node’s roles in the OPNFV cluster in a non-deterministic fashion.

    *** disk_device declares which hard disk to use as the root device for installation. The format is a comma delimited list of devices, such as “sda,sdb,sdc”. The disk chosen will be the first device in the list which is found by introspection to exist on the system. Currently, only a single definition is allowed for all nodes. Therefore if multiple disk_device definitions occur within the inventory, only the last definition on a node will be used for all nodes.

11.3. Creating the Settings Files

Edit the 2 settings files in /etc/opnfv-apex/. These files have comments to help you customize them.

  1. deploy_settings.yaml This file includes basic configuration options deployment, and also documents all available options. Alternatively, there are pre-built deploy_settings files available in (/etc/opnfv-apex/). These files are named with the naming convention os-sdn_controller-enabled_feature-[no]ha.yaml. These files can be used in place of the (/etc/opnfv-apex/deploy_settings.yaml) file if one suites your deployment needs. If a pre-built deploy_settings file is chosen there is no need to customize (/etc/opnfv-apex/deploy_settings.yaml). The pre-built file can be used in place of the (/etc/opnfv-apex/deploy_settings.yaml) file.
  2. network_settings.yaml This file provides Apex with the networking information that satisfies the prerequisite Network Requirements. These are specific to your environment.
11.4. Running opnfv-deploy

You are now ready to deploy OPNFV using Apex! opnfv-deploy will use the inventory and settings files to deploy OPNFV.

Follow the steps below to execute:

  1. Execute opnfv-deploy sudo opnfv-deploy -n network_settings.yaml -i inventory.yaml -d deploy_settings.yaml If you need more information about the options that can be passed to opnfv-deploy use opnfv-deploy --help. -n network_settings.yaml allows you to customize your networking topology. Note it can also be useful to run the command with the --debug argument which will enable a root login on the overcloud nodes with password: ‘opnfv-apex’. It is also useful in some cases to surround the deploy command with nohup. For example: nohup <deploy command> &, will allow a deployment to continue even if ssh access to the Jump Host is lost during deployment.
  2. Wait while deployment is executed. If something goes wrong during this part of the process, start by reviewing your network or the information in your configuration files. It’s not uncommon for something small to be overlooked or mis-typed. You will also notice outputs in your shell as the deployment progresses.
  3. When the deployment is complete the undercloud IP and overcloud dashboard url will be printed. OPNFV has now been deployed using Apex.
12. Installation High-Level Overview - Virtual Deployment

Deploying virtually is an alternative deployment method to bare metal, where only a single bare metal Jump Host server is required to execute deployment. In this deployment type, the Jump Host server will host the undercloud VM along with any number of OPNFV overcloud control/compute nodes. This deployment type is useful when physical resources are constrained, or there is a desire to deploy a temporary sandbox environment.

The virtual deployment operates almost the same way as the bare metal deployment with a few differences mainly related to power management. opnfv-deploy still deploys an undercloud VM. In addition to the undercloud VM a collection of VMs (3 control nodes + 2 compute for an HA deployment or 1 control node and 1 or more compute nodes for a Non-HA Deployment) will be defined for the target OPNFV deployment. All overcloud VMs are registered with a Virtual BMC emulator which will service power management (IPMI) commands. The overcloud VMs are still provisioned with the same disk images and configuration that baremetal would use.

To Triple-O these nodes look like they have just built and registered the same way as bare metal nodes, the main difference is the use of a libvirt driver for the power management. Finally, the default network settings file will deploy without modification. Customizations are welcome but not needed if a generic set of network settings are acceptable.

13. Installation Guide - Virtual Deployment

This section goes step-by-step on how to correctly install and provision the OPNFV target system to VM nodes.

13.1. Special Requirements for Virtual Deployments

In scenarios where advanced performance options or features are used, such as using huge pages with nova instances, DPDK, or iommu; it is required to enabled nested KVM support. This allows hardware extensions to be passed to the overcloud VMs, which will allow the overcloud compute nodes to bring up KVM guest nova instances, rather than QEMU. This also provides a great performance increase even in non-required scenarios and is recommended to be enabled.

During deployment the Apex installer will detect if nested KVM is enabled, and if not, it will attempt to enable it; while printing a warning message if it cannot. Check to make sure before deployment that Nested Virtualization is enabled in BIOS, and that the output of cat /sys/module/kvm_intel/parameters/nested returns “Y”. Also verify using lsmod that the kvm_intel module is loaded for x86_64 machines, and kvm_amd is loaded for AMD64 machines.

13.2. Install Jump Host

Follow the instructions in the Install Bare Metal Jump Host section.

13.3. Running opnfv-deploy

You are now ready to deploy OPNFV! opnfv-deploy has virtual deployment capability that includes all of the configuration necessary to deploy OPNFV with no modifications.

If no modifications are made to the included configurations the target environment will deploy with the following architecture:

  • 1 undercloud VM
  • The option of 3 control and 2 or more compute VMs (HA Deploy / default) or 1 control and 1 or more compute VM (Non-HA deploy / pass -n)
  • 1-5 networks: provisioning, private tenant networking, external, storage and internal API. The API, storage and tenant networking networks can be collapsed onto the provisioning network.

Follow the steps below to execute:

  1. sudo opnfv-deploy -v [ --virtual-computes n ] [ --virtual-cpus n ] [ --virtual-ram n ] -n network_settings.yaml -d deploy_settings.yaml Note it can also be useful to run the command with the --debug argument which will enable a root login on the overcloud nodes with password: ‘opnfv-apex’. It is also useful in some cases to surround the deploy command with nohup. For example: nohup <deploy command> &, will allow a deployment to continue even if ssh access to the Jump Host is lost during deployment.
  2. It will take approximately 45 minutes to an hour to stand up undercloud, define the target virtual machines, configure the deployment and execute the deployment. You will notice different outputs in your shell.
  3. When the deployment is complete the IP for the undercloud and a url for the OpenStack dashboard will be displayed
13.4. Verifying the Setup - VMs

To verify the set you can follow the instructions in the Verifying the Setup section.

14. Verifying the Setup

Once the deployment has finished, the OPNFV deployment can be accessed via the undercloud node. From the Jump Host ssh to the undercloud host and become the stack user. Alternatively ssh keys have been setup such that the root user on the Jump Host can ssh to undercloud directly as the stack user. For convenience a utility script has been provided to look up the undercloud’s ip address and ssh to the undercloud all in one command. An optional user name can be passed to indicate whether to connect as the stack or root user. The stack user is default if a username is not specified.

opnfv-util undercloud root
su - stack

Once connected to undercloud as the stack user look for two keystone files that can be used to interact with the undercloud and the overcloud. Source the appropriate RC file to interact with the respective OpenStack deployment.

source stackrc (undercloud)
source overcloudrc (overcloud / OPNFV)

The contents of these files include the credentials for the administrative user for undercloud and OPNFV respectively. At this point both undercloud and OPNFV can be interacted with just as any OpenStack installation can be. Start by listing the nodes in the undercloud that were used to deploy the overcloud.

source stackrc
openstack server list

The control and compute nodes will be listed in the output of this server list command. The IP addresses that are listed are the control plane addresses that were used to provision the nodes. Use these IP addresses to connect to these nodes. Initial authentication requires using the user heat-admin.

ssh heat-admin@192.0.2.7

To begin creating users, images, networks, servers, etc in OPNFV source the overcloudrc file or retrieve the admin user’s credentials from the overcloudrc file and connect to the web Dashboard.

You are now able to follow the OpenStack Verification section.

15. OpenStack Verification

Once connected to the OPNFV Dashboard make sure the OPNFV target system is working correctly:

  1. In the left pane, click Compute -> Images, click Create Image.
  2. Insert a name “cirros”, Insert an Image Location http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img.
  3. Select format “QCOW2”, select Public, then click Create Image.
  4. Now click Project -> Network -> Networks, click Create Network.
  5. Enter a name “internal”, click Next.
  6. Enter a subnet name “internal_subnet”, and enter Network Address 172.16.1.0/24, click Next.
  7. Now go to Project -> Compute -> Instances, click Launch Instance.
  8. Enter Instance Name “first_instance”, select Instance Boot Source “Boot from image”, and then select Image Name “cirros”.
  9. Click Launch, status will cycle though a couple states before becoming “Active”.
  10. Steps 7 though 9 can be repeated to launch more instances.
  11. Once an instance becomes “Active” their IP addresses will display on the Instances page.
  12. Click the name of an instance, then the “Console” tab and login as “cirros”/”cubswin:)”
  13. To verify storage is working, click Project -> Compute -> Volumes, Create Volume
  14. Give the volume a name and a size of 1 GB
  15. Once the volume becomes “Available” click the dropdown arrow and attach it to an instance.

Congratulations you have successfully installed OPNFV!

16. Developer Guide and Troubleshooting

This section aims to explain in more detail the steps that Apex follows to make a deployment. It also tries to explain possible issues you might find in the process of building or deploying an environment.

After installing the Apex RPMs in the Jump Host, some files will be located around the system.

  1. /etc/opnfv-apex: this directory contains a bunch of scenarios to be deployed with different characteristics such HA (High Availability), SDN controller integration (OpenDaylight/ONOS), BGPVPN, FDIO, etc. Having a look at any of these files will give you an idea of how to make a customized scenario setting up different flags.
  2. /usr/bin/: it contains the binaries for the commands opnfv-deploy, opnfv-clean and opnfv-util.
  3. /usr/share/opnfv/: contains Ansible playbooks and other non-python based configuration and libraries.
  4. /var/opt/opnfv/: contains disk images for Undercloud and Overcloud
16.1. Utilization of Images

As mentioned earlier in this guide, the Undercloud VM will be in charge of deploying OPNFV (Overcloud VMs). Since the Undercloud is an all-in-one OpenStack deployment, it will use Glance to manage the images that will be deployed as the Overcloud.

So whatever customization that is done to the images located in the jumpserver (/var/opt/opnfv/images) will be uploaded to the undercloud and consequently, to the overcloud.

Make sure, the customization is performed on the right image. For example, if I virt-customize the following image overcloud-full-opendaylight.qcow2, but then I deploy OPNFV with the following command:

sudo opnfv-deploy -n network_settings.yaml -d /etc/opnfv-apex/os-onos-nofeature-ha.yaml

It will not have any effect over the deployment, since the customized image is the opendaylight one, and the scenario indicates that the image to be deployed is the overcloud-full-onos.qcow2.

16.2. Post-deployment Configuration

Post-deployment scripts will perform some configuration tasks such ssh-key injection, network configuration, NATing, OpenVswitch creation. It will take care of some OpenStack tasks such creation of endpoints, external networks, users, projects, etc.

If any of these steps fail, the execution will be interrupted. In some cases, the interruption occurs at very early stages, so a new deployment must be executed. However, some other cases it could be worth it to try to debug it.

  1. There is not external connectivity from the overcloud nodes:

    Post-deployment scripts will configure the routing, nameservers and a bunch of other things between the overcloud and the undercloud. If local connectivity, like pinging between the different nodes, is working fine, script must have failed when configuring the NAT via iptables. The main rules to enable external connectivity would look like these:

    iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE iptables -t nat -A POSTROUTING -s ${external_cidr} -o eth0 -j MASQUERADE iptables -A FORWARD -i eth2 -j ACCEPT iptables -A FORWARD -s ${external_cidr} -m state --state ESTABLISHED,RELATED -j ACCEPT service iptables save

    These rules must be executed as root (or sudo) in the undercloud machine.

16.3. OpenDaylight Integration

When a user deploys a scenario that starts with os-odl*:

OpenDaylight (ODL) SDN controller will be deployed and integrated with OpenStack. ODL will run as a systemd service, and can be managed as as a regular service:

systemctl start/restart/stop opendaylight.service

This command must be executed as root in the controller node of the overcloud, where OpenDaylight is running. ODL files are located in /opt/opendaylight. ODL uses karaf as a Java container management system that allows the users to install new features, check logs and configure a lot of things. In order to connect to Karaf’s console, use the following command:

opnfv-util opendaylight

This command is very easy to use, but in case it is not connecting to Karaf, this is the command that is executing underneath:

ssh -p 8101 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no karaf@localhost

Of course, localhost when the command is executed in the overcloud controller, but you use its public IP to connect from elsewhere.

16.4. Debugging Failures

This section will try to gather different type of failures, the root cause and some possible solutions or workarounds to get the process continued.

  1. I can see in the output log a post-deployment error messages:

    Heat resources will apply puppet manifests during this phase. If one of these processes fail, you could try to see the error and after that, re-run puppet to apply that manifest. Log into the controller (see verification section for that) and check as root /var/log/messages. Search for the error you have encountered and see if you can fix it. In order to re-run the puppet manifest, search for “puppet apply” in that same log. You will have to run the last “puppet apply” before the error. And It should look like this:

    FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/5b4c7a01-0d63-4a71-81e9-d5ee6f0a1f2f"  FACTER_fqdn="overcloud-controller-0.localdomain.com" \ FACTER_deploy_config_name="ControllerOvercloudServicesDeployment_Step4"  puppet apply --detailed-exitcodes -l syslog -l console \ /var/lib/heat-config/heat-config-puppet/5b4c7a01-0d63-4a71-81e9-d5ee6f0a1f2f.pp

    As a comment, Heat will trigger the puppet run via os-apply-config and it will pass a different value for step each time. There is a total of five steps. Some of these steps will not be executed depending on the type of scenario that is being deployed.

17. Frequently Asked Questions
18. License

All Apex and “common” entities are protected by the Apache 2.0 License.

19. References
19.1. OPNFV

OPNFV Home Page

OPNFV Apex project page

OPNFV Apex Release Notes

19.3. OpenDaylight

Upstream OpenDaylight provides a number of packaging and deployment options meant for consumption by downstream projects like OPNFV.

Currently, OPNFV Apex uses OpenDaylight’s Puppet module, which in turn depends on OpenDaylight’s RPM.

19.4. RDO Project

RDO Project website

Authors:Tim Rozet (trozet@redhat.com)
Authors:Dan Radez (dradez@redhat.com)
Version:5.0
Indices and tables

Armband

Installation instruction for Fuel@OPNFV
1. Abstract

This document describes how to install the Euphrates release of OPNFV when using Fuel as a deployment tool, covering its usage, limitations, dependencies and required system resources. This is an unified documentation for both x86_64 and aarch64 architectures. All information is common for both architectures except when explicitly stated.

2. Introduction

This document provides guidelines on how to install and configure the Euphrates release of OPNFV when using Fuel as a deployment tool, including required software and hardware configurations.

Although the available installation options provide a high degree of freedom in how the system is set up, including architecture, services and features, etc., said permutations may not provide an OPNFV compliant reference architecture. This document provides a step-by-step guide that results in an OPNFV Euphrates compliant deployment.

The audience of this document is assumed to have good knowledge of networking and Unix/Linux administration.

3. Preface

Before starting the installation of the Euphrates release of OPNFV, using Fuel as a deployment tool, some planning must be done.

3.1. Preparations

Prior to installation, a number of deployment specific parameters must be collected, those are:

  1. Provider sub-net and gateway information
  2. Provider VLAN information
  3. Provider DNS addresses
  4. Provider NTP addresses
  5. Network overlay you plan to deploy (VLAN, VXLAN, FLAT)
  6. How many nodes and what roles you want to deploy (Controllers, Storage, Computes)
  7. Monitoring options you want to deploy (Ceilometer, Syslog, etc.).
  8. Other options not covered in the document are available in the links above

This information will be needed for the configuration procedures provided in this document.

4. Hardware Requirements for Virtual Deploys

The following minimum hardware requirements must be met for the virtual installation of Euphrates using Fuel:

HW Aspect Requirement
1 Jumpserver A physical node (also called Foundation Node) that will host a Salt Master VM and each of the VM nodes in the virtual deploy
CPU Minimum 1 socket with Virtualization support
RAM Minimum 32GB/server (Depending on VNF work load)
Disk Minimum 100GB (SSD or SCSI (15krpm) highly recommended
5. Hardware Requirements for Baremetal Deploys

The following minimum hardware requirements must be met for the baremetal installation of Euphrates using Fuel:

HW Aspect Requirement
# of nodes

Minimum 5

  • 3 KVM servers which will run all the controller services
  • 2 Compute nodes
CPU Minimum 1 socket with Virtualization support
RAM Minimum 16GB/server (Depending on VNF work load)
Disk Minimum 256GB 10kRPM spinning disks
Networks

4 VLANs (PUBLIC, MGMT, STORAGE, PRIVATE) - can be a mix of tagged/native

1 Un-Tagged VLAN for PXE Boot - ADMIN Network

Note: These can be allocated to a single NIC - or spread out over multiple NICs

1 Jumpserver A physical node (also called Foundation Node) that hosts the Salt Master and MaaS VMs
Power management All targets need to have power management tools that allow rebooting the hardware and setting the boot order (e.g. IPMI)

NOTE: All nodes including the Jumpserver must have the same architecture (either x86_64 or aarch64).

NOTE: For aarch64 deployments an UEFI compatible firmware with PXE support is needed (e.g. EDK2).

6. Help with Hardware Requirements

Calculate hardware requirements:

For information on compatible hardware types available for use, please see Fuel OpenStack Hardware Compatibility List.

When choosing the hardware on which you will deploy your OpenStack environment, you should think about:

  • CPU – Consider the number of virtual machines that you plan to deploy in your cloud environment and the CPUs per virtual machine.
  • Memory – Depends on the amount of RAM assigned per virtual machine and the controller node.
  • Storage – Depends on the local drive space per virtual machine, remote volumes that can be attached to a virtual machine, and object storage.
  • Networking – Depends on the Choose Network Topology, the network bandwidth per virtual machine, and network storage.
7. Top of the Rack (TOR) Configuration Requirements

The switching infrastructure provides connectivity for the OPNFV infrastructure operations, tenant networks (East/West) and provider connectivity (North/South); it also provides needed connectivity for the Storage Area Network (SAN). To avoid traffic congestion, it is strongly suggested that three physically separated networks are used, that is: 1 physical network for administration and control, one physical network for tenant private and public networks, and one physical network for SAN. The switching connectivity can (but does not need to) be fully redundant, in such case it comprises a redundant 10GE switch pair for each of the three physically separated networks.

The physical TOR switches are not automatically configured from the Fuel OPNFV reference platform. All the networks involved in the OPNFV infrastructure as well as the provider networks and the private tenant VLANs needs to be manually configured.

Manual configuration of the Euphrates hardware platform should be carried out according to the OPNFV Pharos Specification.

8. OPNFV Software Prerequisites

The Jumpserver node should be pre-provisioned with an operating system, according to the Pharos specification. Relevant network bridges should also be pre-configured (e.g. admin, management, public).

Fuel@OPNFV has been validated by CI using the following distributions installed on the Jumpserver:

  • CentOS 7 (recommended by Pharos specification);
  • Ubuntu Xenial;

NOTE: The install script expects ‘libvirt’ to be installed and running on the Jumpserver. In case the packages are missing, the script will install them; but depending on the OS distribution, the user might have to start the ‘libvirtd’ service manually.

9. OPNFV Software Installation and Deployment

This section describes the process of installing all the components needed to deploy the full OPNFV reference platform stack across a server cluster.

The installation is done with Mirantis Cloud Platform (MCP), which is based on a reclass model. This model provides the formula inputs to Salt, to make the deploy automatic based on deployment scenario. The reclass model covers:

  • Infrastucture node definition: Salt Master node (cfg01) and MaaS node (mas01)
  • Openstack node defition: Controler nodes (ctl01, ctl02, ctl03) and Compute nodes (cmp001, cmp002)
  • Infrastructure components to install (software packages, services etc.)
  • Openstack components and services (rabbitmq, galera etc.), as well as all configuration for them
9.1. Automatic Installation of a Virtual POD

For virtual deploys all the targets are VMs on the Jumpserver. The deploy script will:

  • Create a Salt Master VM on the Jumpserver which will drive the installation

  • Create the bridges for networking with virsh (only if a real bridge does not already exists for a given network)

  • Install Openstack on the targets
    • Leverage Salt to install & configure Openstack services
Fuel@OPNFV Virtual POD Network Layout Examples

Fuel@OPNFV Virtual POD Network Layout Examples

cfg01 Salt Master VM
ctl01 Controller VM
cmp01/cmp02 Compute VMs
gtw01 Gateway VM with neutron services (dhcp agent, L3 agent, metadata, etc)
odl01 VM on which ODL runs (for scenarios deployed with ODL)
In this figure there are examples of two virtual deploys:
  • Jumphost 1 has only virsh bridges, created by the deploy script
  • Jumphost 2 has a mix of linux and virsh briges; when linux bridge exist for a specified network, the deploy script will skip creating a virsh bridge for it

Note: A virtual network “mcpcontrol” is always created. For virtual deploys, “mcpcontrol” is also used for Admin, leaving the PXE/Admin bridge unused.

9.2. Automatic Installation of a Baremetal POD

The baremetal installation process can be done by editing the information about hardware and enviroment in the reclass files, or by using a Pod Descriptor File (PDF). This file contains all the information about the hardware and network of the deployment the will be fed to the reclass model during deployment.

The installation is done automatically with the deploy script, which will:

  • Create a Salt Master VM on the Jumpserver which will drive the installation

  • Create a MaaS Node VM on the Jumpserver which will provision the targets

  • Install Openstack on the targets
    • Leverage MaaS to provision baremetal nodes with the operating system
    • Leverage Salt to configure the operatign system on the baremetal nodes
    • Leverage Salt to install & configure Openstack services
Fuel@OPNFV Baremetal POD Network Layout Example

Fuel@OPNFV Baremetal POD Network Layout Example

cfg01 Salt Master VM
mas01 MaaS Node VM
kvm01..03 Baremetals which hold the VMs with controller functions
cmp001/cmp002 Baremetal compute nodes
prx01/prx02 Proxy VMs for Nginx
msg01..03 RabbitMQ Service VMs
dbs01..03 MySQL service VMs
mdb01..03 Telemetry VMs
odl01 VM on which ODL runs (for scenarios deployed with ODL)
Tenant VM VM running in the cloud

In the baremetal deploy all bridges but “mcpcontrol” are linux bridges. For the Jumpserver, if they are already created they will be used; otherwise they will be created. For the targets, the bridges are created by the deploy script.

Note: A virtual network “mcpcontrol” is always created. For baremetal deploys, PXE bridge is used for baremetal node provisioning, while “mcpcontrol” is used to provision the infrastructure VMs only.

9.3. Steps to Start the Automatic Deploy

These steps are common both for virtual and baremetal deploys.

  1. Clone the Fuel code from gerrit

    For x86_64

    $ git clone https://git.opnfv.org/fuel
    $ cd fuel
    

    For aarch64

    $ git clone https://git.opnfv.org/armband
    $ cd armband
    
  2. Checkout the Euphrates release

    $ git checkout opnfv-5.0.2
    
  3. Start the deploy script

    $ ci/deploy.sh -l <lab_name> \
                   -p <pod_name> \
                   -b <URI to configuration repo containing the PDF file> \
                   -s <scenario> \
                   -B <list of admin, management, private and public bridges>
    
9.3.1. Examples
  1. Virtual deploy

    To start a virtual deployment, it is required to have the virtual keyword while specifying the pod name to the installer script.

    It will create the required bridges and networks, configure Salt Master and install OpenStack.

    $ ci/deploy.sh -b file:///home/jenkins/tmpdir/securedlab \
                   -l ericsson \
                   -p virtual_kvm \
                   -s os-nosdn-nofeature-noha
    

    Once the deployment is complete, the OpenStack Dashboard, Horizon is available at http://<controller VIP>:8078, e.g. http://10.16.0.101:8078. The administrator credentials are admin / opnfv_secret.

  2. Baremetal deploy

    A x86 deploy on pod2 from Linux Foundation lab

    $ ci/deploy.sh -b file:///home/jenkins/tmpdir/securedlab \
                   -l lf \
                   -p pod2 \
                   -s os-nosdn-nofeature-ha \
                   -B pxebr,br-ctl
    
    Fuel@OPNFV LF POD2 Network Layout

    Fuel@OPNFV LF POD2 Network Layout

    Once the deployment is complete, the SaltStack Deployment Documentation is available at http://<Proxy VIP>:8090, e.g. http://172.30.10.103:8090.

    An aarch64 deploy on pod5 from Arm lab

    $ ci/deploy.sh -b file:///home/jenkins/tmpdir/securedlab \
                   -l arm \
                   -p pod5 \
                   -s os-nosdn-nofeature-ha \
                   -B admin7_br0,mgmt7_br0,,public7_br0
    
    Fuel@OPNFV ARM POD5 Network Layout

    Fuel@OPNFV ARM POD5 Network Layout

    Once the deployment is complete, the SaltStack Deployment Documentation is available at http://<Proxy VIP>:8090, e.g. http://10.0.8.103:8090.

9.4. Pod Descriptor Files

Descriptor files provide the installer with an abstraction of the target pod with all its hardware characteristics and required parameters. This information is split into two different files: Pod Descriptor File (PDF) and Installer Descriptor File (IDF).

The Pod Descriptor File is a hardware and network description of the pod infrastructure. The information is modeled under a yaml structure. A reference file with the expected yaml structure is available at mcp/config/labs/local/pod1.yaml

A common network section describes all the internal and provider networks assigned to the pod. Each network is expected to have a vlan tag, IP subnet and attached interface on the boards. Untagged vlans shall be defined as “native”.

The hardware description is arranged into a main “jumphost” node and a “nodes” set for all target boards. For each node the following characteristics are defined:

  • Node parameters including CPU features and total memory.
  • A list of available disks.
  • Remote management parameters.
  • Network interfaces list including mac address, speed and advanced features.
  • IP list of fixed IPs for the node

Note: the fixed IPs are ignored by the MCP installer script and it will instead assign based on the network ranges defined under the pod network configuration.

The Installer Descriptor File extends the PDF with pod related parameters required by the installer. This information may differ per each installer type and it is not considered part of the pod infrastructure. Fuel installer relies on the IDF model to map the networks to the bridges on the foundation node and to setup all node NICs by defining the expected OS device name and bus address.

The file follows a yaml structure and a “fuel” section is expected. Contents and references must be aligned with the PDF file. The IDF file must be named after the PDF with the prefix “idf-”. A reference file with the expected structure is available at mcp/config/labs/local/idf-pod1.yaml

10. Release Notes

Please refer to the Release Notes article.

User guide for Fuel@OPNFV
1. Abstract

This document contains details about how to use OPNFV Fuel - Euphrates release - after it was deployed. For details on how to deploy check the installation instructions in the References section.

This is an unified documentation for both x86_64 and aarch64 architectures. All information is common for both architectures except when explicitly stated.

2. Network Overview

Fuel uses several networks to deploy and administer the cloud:

Network name Deploy Type Description
PXE/ADMIN baremetal only Used for booting the nodes via PXE
MCPCONTROL baremetal & virtual Used to provision the infrastructure VMs (Salt & MaaS). On virtual deploys, it is used for Admin too (on target VMs) leaving the PXE/Admin bridge unused
Mgmt baremetal & virtual Used for internal communication between OpenStack components
Internal baremetal & virtual Used for VM data communication within the cloud deployment
Public baremetal & virtual Used to provide Virtual IPs for public endpoints that are used to connect to OpenStack services APIs. Used by Virtual machines to access the Internet

These networks - except mcpcontrol - can be linux bridges configured before the deploy on the Jumpserver. If they don’t exists at deploy time, they will be created by the scripts as virsh networks.

Mcpcontrol exists only on the Jumpserver and needs to be virtual because a DHCP server runs on this network and associates static host entry IPs for Salt and Maas VMs.

3. Accessing the Cloud

Access to any component of the deployed cloud is done from Jumpserver to user ubuntu with ssh key /var/lib/opnfv/mcp.rsa. The example below is a connection to Salt master.

$ ssh -o StrictHostKeyChecking=no -i  /var/lib/opnfv/mcp.rsa  -l ubuntu 10.20.0.2

Note: The Salt master IP is not hard set, it is configurable via INSTALLER_IP during deployment

The Fuel baremetal deploy has a Virtualized Control Plane (VCP) which means that the controller services are installed in VMs on the baremetal targets (kvm servers). These VMs can also be accessed with virsh console: user opnfv, password opnfv_secret. This method does not apply to infrastructure VMs (Salt master and MaaS).

The example below is a connection to a controller VM. The connection is made from the baremetal server kvm01.

$ ssh -o StrictHostKeyChecking=no -i  /var/lib/opnfv/mcp.rsa  -l ubuntu x.y.z.141
ubuntu@kvm01:~$ virsh console ctl01

User ubuntu has sudo rights. User opnfv has sudo rights only on aarch64 deploys.

4. Exploring the Cloud with Salt

To gather information about the cloud, the salt commands can be used. It is based around a master-minion idea where the salt-master pushes config to the minions to execute actions.

For example tell salt to execute a ping to 8.8.8.8 on all the nodes.

_images/saltstack.png

Complex filters can be done to the target like compound queries or node roles. For more information about Salt see the References section.

Some examples are listed below. Note that these commands are issued from Salt master with root user.

  1. View the IPs of all the components

    root@cfg01:~$ salt "*" network.ip_addrs
    cfg01.baremetal-mcp-ocata-odl-ha.local:
        - 10.20.0.2
        - 172.16.10.100
    mas01.baremetal-mcp-ocata-odl-ha.local:
        - 10.20.0.3
        - 172.16.10.3
        - 192.168.11.3
    .........................
    
  2. View the interfaces of all the components and put the output in a file with yaml format

    root@cfg01:~$ salt "*" network.interfaces --out yaml --output-file interfaces.yaml
    root@cfg01:~# cat interfaces.yaml
    cfg01.baremetal-mcp-ocata-odl-ha.local:
      enp1s0:
        hwaddr: 52:54:00:72:77:12
        inet:
        - address: 10.20.0.2
          broadcast: 10.20.0.255
          label: enp1s0
          netmask: 255.255.255.0
        inet6:
        - address: fe80::5054:ff:fe72:7712
          prefixlen: '64'
          scope: link
        up: true
    .........................
    
  3. View installed packages in MaaS node

    root@cfg01:~# salt "mas*" pkg.list_pkgs
    mas01.baremetal-mcp-ocata-odl-ha.local:
        ----------
        accountsservice:
            0.6.40-2ubuntu11.3
        acl:
            2.2.52-3
        acpid:
            1:2.0.26-1ubuntu2
        adduser:
            3.113+nmu3ubuntu4
        anerd:
            1
    .........................
    
  4. Execute any linux command on all nodes (list the content of /var/log in this example)

    root@cfg01:~# salt "*" cmd.run 'ls /var/log'
    cfg01.baremetal-mcp-ocata-odl-ha.local:
       alternatives.log
       apt
       auth.log
       boot.log
       btmp
       cloud-init-output.log
       cloud-init.log
    .........................
    
  5. Execute any linux command on nodes using compound queries filter

    root@cfg01:~# salt -C '* and cfg01*' cmd.run 'ls /var/log'
    cfg01.baremetal-mcp-ocata-odl-ha.local:
       alternatives.log
       apt
       auth.log
       boot.log
       btmp
       cloud-init-output.log
       cloud-init.log
    .........................
    
  6. Execute any linux command on nodes using role filter

    root@cfg01:~# salt -I 'nova:compute' cmd.run 'ls /var/log'
    cmp001.baremetal-mcp-ocata-odl-ha.local:
       alternatives.log
       apache2
       apt
       auth.log
       btmp
       ceilometer
       cinder
       cloud-init-output.log
       cloud-init.log
    .........................
    
5. Accessing Openstack

Once the deployment is complete, Openstack CLI is accessible from controller VMs (ctl01..03). Openstack credentials are at /root/keystonercv3.

root@ctl01:~# source keystonercv3
root@ctl01:~# openstack image list
+--------------------------------------+-----------------------------------------------+--------+
| ID                                   | Name                                          | Status |
+======================================+===============================================+========+
| 152930bf-5fd5-49c2-b3a1-cae14973f35f | CirrosImage                                   | active |
| 7b99a779-78e4-45f3-9905-64ae453e3dcb | Ubuntu16.04                                   | active |
+--------------------------------------+-----------------------------------------------+--------+

The OpenStack Dashboard, Horizon is available at http://<controller VIP>:8078, e.g. http://10.16.0.101:8078. The administrator credentials are admin/opnfv_secret.

_images/horizon_login.png

A full list of IPs/services is available at <proxy public VIP>:8090 for baremetal deploys.

_images/salt_services_ip.png

For Virtual deploys, the most commonly used IPs are in the table below.

Component IP Default value
gtw01 x.y.z.110 172.16.10.110
ctl01 x.y.z.100 172.16.10.100
cmp001 x.y.z.105 172.16.10.105
cmp002 x.y.z.106 172.16.10.106

Availability

Reach-thru Guest Monitoring and Services for High Availability
1   Overview
abstract:This document describes a set of new optional capabilities where the OpenStack Cloud messages into the Guest VMs in order to provide improved Availability of the hosted VMs. The initial set of new capabilities include: enabling the detection of and recovery from internal VM faults and providing a simple out-of-band messaging service to prevent scenarios such as split brain.
2   Introduction

This document provides an overview and rationale for a set of new capabilities where the OpenStack Cloud messages into the Guest VMs in order to provide improved Availability of the hosted VMs.

The initial set of new capabilities specifically include:

  • VM Heartbeating and Health Checking
  • VM Peer State Notification and Messaging

All of these capabilities leverage Host-to-Guest Messaging Interfaces / APIs which are built on a messaging service between the OpenStack Host and the Guest VM that uses a simple low-bandwidth datagram messaging capability in the hypervisor and therefore has no requirements on OpenStack Networking, and is available very early after spawning the VM.

For each capability, the document outlines the interaction with the Guest VM, any key technologies involved, the integration into the larger OpenStack and OPNFV Architectures (e.g. interactions with VNFM), specific OPNFV HA Team deliverables, and the use cases for how availability of the hosted VM is improved.

3   Messaging Layer

The Host-to-Guest messaging APIs used by the services discussed in this document use a JSON-formatted application messaging layer on top of a ‘virtio serial device??between QEMU on the OpenStack Host and the Guest VM. JSON formatting provides a simple, humanly readable messaging format which can be easily parsed and formatted using any high level programming language being used in the Guest VM (e.g. C/C++, Python, Java, etc.). Use of the ‘virtio serial device??provides a simple, direct communication channel between host and guest which is independent of the Guest’s L2/L3 networking.

The upper layer JSON messaging format is actually structured as a hierarchical JSON format containing a Base JSON Message Layer and an Application JSON Message Layer:

  • the Base Layer provides the ability to multiplex different groups of message types on top of a single ‘virtio serial device?? e.g.

    • heartbeating and healthchecks,
    • server group messaging,

    and

  • the Application Layer provides the specific message types and fields of a particular group of message types.

4   VM Heartbeating and Health Checking

Normally OpenStack monitoring of the health of a Guest VM is limited to a black-box approach of simply monitoring the presence of the QEMU/KVM PID containing the VM, and/or by enabling libvirt’s emulated hardware watchdog.

VM Heartbeating and Health Checking provides a heartbeat service to enhance the monitoring of the health of guest application(s) within a VM running under the OpenStack Cloud. Loss of heartbeat or a failed health check status will result in a fault event being reported to OPNFV’s DOCTOR infrastructure for alarm identification, impact analysis and reporting. This would then enable VNF Managers (VNFMs) listening to OPNFV’s DOCTOR External Alarm Reporting through Telemetry’s AODH, to initiate any required fault recovery actions.

_images/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png

Or, in the context of the OPNFV DOCTOR’s Fault Management Architecture:

_images/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png

The VM Heartbeating and Health Checking functionality is enabled on a VM through a new flavor extraspec indicating that the VM supports and wants to enable Guest Heartbeating. An extension to Nova Compute uses this extraspec to setup the required ‘virtio serial device’ for Host-to-Guest messaging, on the QEMU/KVM instance created for the VM.

A daemon within the Guest VM will register with the OpenStack Guest Heartbeat Service on the compute node to initiate the heartbeating on itself (i.e. the Guest VM). The OpenStack Compute Node will start heartbeating the Guest VM, and if the heartbeat fails, the OpenStack Compute Node will report the VM Fault thru DOCTOR and ultimately VNFM will see this thru NOVA VM State Change Notifications thru AODH. I.e. VNFM wouild see the VM Heartbeat Failure events in the same way it sees all other VM Faults, thru DOCTOR initiated VM state changes.

Part of the Guest VM’s registration process is the specification of the heartbeat interval in msecs. I.e. the registering Guest VM specifies the heartbeating interval.

Guest heartbeat works on a challenge response model. The OpenStack Guest Heartbeat Service on the compute node will challenge the registered Guest VM daemon with a message each interval. The registered Guest VM daemon must respond prior to the next interval with a message indicating good health. If the OpenStack Host does not receive a valid response, or if the response specifies that the VM is in ill health, then a fault event for the Guest VM is reported to the OpenStack Guest Heartbeat Service on the controller node which will report the event to OPNFV’s DOCTOR (i.e. thru the Doctor SouthBound (SB) APIs).

In summary, the Guest Heartbeating Messaging Specification is quite simple, including the following PDUs: Init, Init-Ack, Challenge-Request, Challenge-Response, Exit. The Challenge-Response returning a healthy / not-healthy boolean.

The registered Guest VM daemon’s response to the challenge can be as simple as just immediately responding with OK. This alone allows for detection of a failed or hung QEMU/KVM instance, or a failure of the OS within the VM to schedule the registered Guest VM’s daemon or failure to route basic IO within the Guest VM.

However the registered Guest VM daemon’s response to the challenge can be more complex, running anything from a quick simple sanity check of the health of applications running in the Guest VM, to a more thorough audit of the application state and data. In either case returning the status of the health check enables the OpenStack host to detect and report the event in order to initiate recovery from application level errors or failures within the Guest VM.

In summary, the deliverables of this activity would be:

  • Host Deliverables: (OpenStack and OPNFV blueprints and implementation)
  • an OpenStack Nova or libvirt extension to interpret the new flavor extraspec and if present setup the required ‘virtio serial device’ for Host-to-Guest heartbeat / health-check messaging, on the QEMU/KVM instance created for the VM,
  • an OPNFV Base Host-to-Guest Msging Layer Agent for multiplexing of Application Layer messaging over the ‘virtio serial device’ to the VM,
  • an OPNFV Heartbeat / Health-Check Compute Agent for local heartbeating of VM and reporting of failures to the OpenStack Controller,
  • an OPNFV Heartbeat / Health-check Server on the OpenStack Controller for receiving VM failure notifications and reporting these to Vitrage thru Vitrage’s Data Source API,
  • Guest Deliverables:
  • a Heartbeat / Health-Check Message Specification covering

    • Heartbeat / Health-Check Application Layer JSON Protocol,
    • Base Host-to-Guest JSON Protocol,
    • Details on the use of the underlying ‘virtio serial device’,
  • a Reference Implementation of the Guest-side support of Heartbeat / Health-check containing the peer protocol layers within the Guest.

    • will provide code and compile instructions,
    • Guest will compile based on its specific OS.

NOTE that the described VM Heartbeating and Healthchecking functionality provides enhanced monitoring over and above libvirt’s emulated hardware watchdog. VM Heartbeating and Healthchecking can detect a wider range of issues than simply lack of cpu time scheduling for a lower priority process feeding the hardware watchdog. VM Heartbeating and Healthchecking can ensure that specific key processes within the application are not blocked, kernel resources for basic IO within the Guest VM are available, and/or ensure the application-specific health of the VM is good.

This proposal has been reviewed with both the OPNFV’s Doctor and Management and Orchestration teams, and general agreement was that the proposal integrated / inter-worked correctly with the OPNFV DOCTOR’s Vitrage, Congress and the overall OPNFV fault reporting architecture.

5   VM Peer State Notification and Messaging

Server Group State Notification and Messaging is a service to provide simple low-bandwidth datagram messaging and notifications for servers that are part of the same server group. This messaging channel is available regardless of whether IP networking is functional within the server, and it requires no knowledge within the server about the other members of the group.

NOTE: A Server Group here is the OpenStack Nova Server Group concept where VMs are grouped together for purposes of scheduling. E.g. A specific Server Group instance can specify whether the VMs within the group should be scheduled to run on the same compute host or different compute hosts. A ‘peer’ VM in the context of this section refers to a VM within the same Nova Server Group.

This Server Group Messaging service provides three types of messaging:

  • Broadcast: this allows a server to send a datagram (size of up to 3050 bytes) to all other servers within the server group.
  • Notification: this provides servers with information about changes to the (Nova) state of other servers within the server group.
  • Status: this allows a server to query the current (Nova) state of all servers within the server group (including itself).

A Server Group Messaging entity on both the controller node and the compute nodes manage the routing of of VM-to-VM messages through the platform, leveraging Nova to determine Server Group membership and compute node locations of VMs. The Server Group Messaging entity on the controller also listens to Nova VM state change notifications and querys VM state data from Nova, in order to provide the VM query and notification functionality of this service.

_images/OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png

This service is not intended for high bandwidth or low-latency operations. It is best-effort, not reliable. Applications should do end-to-end acks and retries if they care about reliability.

This service provides building block type capabilities for the Guest VMs that contribute to higher availability of the VMs in the Guest VM Server Group. Notifications of VM Status changes potentially provide a faster and more accurate notification of failed peer VMs than traditional peer VM monitoring over Tenant Networks. While the Broadcast Messaging mechanism provides an out-of-band messaging mechanism to monitor and control a peer VM under fault conditions; e.g. providing the ability to avoid potential split brain scenarios between 1:1 VMs when faults in Tenant Networking occur.

In summary, the deliverables for Server Group Messaging would be:

  • Host Deliverables:
  • a Nova or libvirt extension to interpret the new flavor extraspec and if present setup the required ‘virtio serial device’ for Host-to-Guest Server Group Messaging, on the QEMU/KVM instance created for the VM,
  • [ leveraging the Base Host-to-Guest Msging Layer Agent from previous section ],
  • a Server Group Messaging Compute Agent for implementing the Application Layer Server Group Messaging JSON Protocol with the VM, and forwarding the messages to/from the Server Group Messaging Server on the Controller,
  • a Server Group Messaging Server on the Controller for routing broadcast messages to the proper Computes and VMs, as well as listening for Nova VM State Change Notifications and forwarding these to applicable Computes and VMs,
  • Guest Deliverables:
  • a Server Group Messaging Message Specification covering

    • Server Group Messaging Application Layer JSON Protocol,
    • [ leveraging Base Host-to-Guest JSON Protocol from previous section ],
    • [ leveraging Details on the use of the underlying ‘virtio serial device’ from previous section ],
  • a Reference Implementation of the Guest-side support of Server Group Messaging containing the peer protocol layers and Guest Application hooks within the Guest.

This proposal has been reviewed with both the OPNFV’s Doctor and Management and Orchestration teams, and general agreement was that the proposal did not conflict with the OPNFV Doctor Architecture, and provided, at the very least, an alternative messaging and state-change-notification mechanism for hosted VMs in various HA use cases.

6   Conclusion

The Reach-thru Guest Monitoring and Services described in this document leverage Host-to-Guest messaging to provide a number of extended capabilities that improve the Availability of the hosted VMs. These new capabilities enable detection of and recovery from internal VM faults and provides a simple out-of-band messaging service to prevent scenarios such as split brain.

The next steps in progressing this proposal will be to submit blueprints to the appropriate OpenStack working groups; Vitrage for VM Heartbeating and Healthchecking and Nova for VM Server Group Messaging.

Barometer

OPNFV Barometer Requirements
Problem Statement

Providing carrier grade Service Assurance is critical in the network transformation to a software defined and virtualized network (NFV). Medium-/large-scale cloud environments account for between hundreds and hundreds of thousands of infrastructure systems. It is vital to monitor systems for malfunctions that could lead to users application service disruption and promptly react to these fault events to facilitate improving overall system performance. As the size of infrastructure and virtual resources grow, so does the effort of monitoring back-ends. SFQM aims to expose as much useful information as possible off the platform so that faults and errors in the NFVI can be detected promptly and reported to the appropriate fault management entity.

The OPNFV platform (NFVI) requires functionality to:

  • Create a low latency, high performance packet processing path (fast path) through the NFVI that VNFs can take advantage of;
  • Measure Telco Traffic and Performance KPIs through that fast path;
  • Detect and report violations that can be consumed by VNFs and higher level EMS/OSS systems

Examples of local measurable QoS factors for Traffic Monitoring which impact both Quality of Experience and five 9’s availability would be (using Metro Ethernet Forum Guidelines as reference):

  • Packet loss
  • Packet Delay Variation
  • Uni-directional frame delay

Other KPIs such as Call drops, Call Setup Success Rate, Call Setup time etc. are measured by the VNF.

In addition to Traffic Monitoring, the NFVI must also support Performance Monitoring of the physical interfaces themselves (e.g. NICs), i.e. an ability to monitor and trace errors on the physical interfaces and report them.

All these traffic statistics for Traffic and Performance Monitoring must be measured in-service and must be capable of being reported by standard Telco mechanisms (e.g. SNMP traps), for potential enforcement actions.

Barometer updated scope

The scope of the project is to provide interfaces to support monitoring of the NFVI. The project will develop plugins for telemetry frameworks to enable the collection of platform stats and events and relay gathered information to fault management applications or the VIM. The scope is limited to collecting/gathering the events and stats and relaying them to a relevant endpoint. The project will not enforce or take any actions based on the gathered information.

Scope of SFQM

NOTE: The SFQM project has been replaced by Barometer. The output of the project will provide interfaces and functions to support monitoring of Packet Latency and Network Interfaces while the VNF is in service.

The DPDK interface/API will be updated to support:

  • Exposure of NIC MAC/PHY Level Counters
  • Interface for Time stamp on RX
  • Interface for Time stamp on TX
  • Exposure of DPDK events

collectd will be updated to support the exposure of DPDK metrics and events.

Specific testing and integration will be carried out to cover:

  • Unit/Integration Test plans: A sample application provided to demonstrate packet latency monitoring and interface monitoring

The following list of features and functionality will be developed:

  • DPDK APIs and functions for latency and interface monitoring
  • A sample application to demonstrate usage
  • collectd plugins

The scope of the project involves developing the relavant DPDK APIs, OVS APIs, sample applications, as well as the utilities in collectd to export all the relavent information to a telemetry and events consumer.

VNF specific processing, Traffic Monitoring, Performance Monitoring and Management Agent are out of scope.

The Proposed Interface counters include:

  • Packet RX
  • Packet TX
  • Packet loss
  • Interface errors + other stats

The Proposed Packet Latency Monitor include:

  • Cycle accurate stamping on ingress
  • Supports latency measurements on egress

Support for failover of DPDK enabled cores is also out of scope of the current proposal. However, this is an important requirement and must-have functionality for any DPDK enabled framework in the NFVI. To that end, a second phase of this project will be to implement DPDK Keep Alive functionality that would address this and would report to a VNF-level Failover and High Availability mechanism that would then determine what actions, including failover, may be triggered.

Consumption Models

In reality many VNFs will have an existing performance or traffic monitoring utility used to monitor VNF behavior and report statistics, counters, etc.

The consumption of performance and traffic related information/events provided by this project should be a logical extension of any existing VNF/NFVI monitoring framework. It should not require a new framework to be developed. We do not see the Barometer gathered metrics and evetns as major additional effort for monitoring frameworks to consume; this project would be sympathetic to existing monitoring frameworks. The intention is that this project represents an interface for NFVI monitoring to be used by higher level fault management entities (see below).

Allowing the Barometer metrics and events to be handled within existing telemetry frameoworks makes it simpler for overall interfacing with higher level management components in the VIM, MANO and OSS/BSS. The Barometer proposal would be complementary to the Doctor project, which addresses NFVI Fault Management support in the VIM, and the VES project, which addresses the integration of VNF telemetry-related data into automated VNF management systems. To that end, the project committers and contributors for the Barometer project wish to collaborate with the Doctor and VES projects to facilitate this.

collectd

collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanisms to publish the collected metrics. It supports more than 90 different input and output plugins. Input plugins retrieve metrics and publish them to the collectd deamon, while output plugins publish the data they receive to an end point. collectd also has infrastructure to support thresholding and notification.

collectd statistics and Notifications

Within collectd notifications and performance data are dispatched in the same way. There are producer plugins (plugins that create notifications/metrics), and consumer plugins (plugins that receive notifications/metrics and do something with them).

Statistics in collectd consist of a value list. A value list includes:

  • Values, can be one of:
    • Derive: used for values where a change in the value since it’s last been read is of interest. Can be used to calculate and store a rate.
    • Counter: similar to derive values, but take the possibility of a counter wrap around into consideration.
    • Gauge: used for values that are stored as is.
    • Absolute: used for counters that are reset after reading.
  • Value length: the number of values in the data set.
  • Time: timestamp at which the value was collected.
  • Interval: interval at which to expect a new value.
  • Host: used to identify the host.
  • Plugin: used to identify the plugin.
  • Plugin instance (optional): used to group a set of values together. For e.g. values belonging to a DPDK interface.
  • Type: unit used to measure a value. In other words used to refer to a data set.
  • Type instance (optional): used to distinguish between values that have an identical type.
  • meta data: an opaque data structure that enables the passing of additional information about a value list. “Meta data in the global cache can be used to store arbitrary information about an identifier” [7].

Host, plugin, plugin instance, type and type instance uniquely identify a collectd value.

Values lists are often accompanied by data sets that describe the values in more detail. Data sets consist of:

  • A type: a name which uniquely identifies a data set.
  • One or more data sources (entries in a data set) which include:
    • The name of the data source. If there is only a single data source this is set to “value”.
    • The type of the data source, one of: counter, gauge, absolute or derive.
    • A min and a max value.

Types in collectd are defined in types.db. Examples of types in types.db:

bitrate    value:GAUGE:0:4294967295
counter    value:COUNTER:U:U
if_octets  rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295

In the example above if_octets has two data sources: tx and rx.

Notifications in collectd are generic messages containing:

  • An associated severity, which can be one of OKAY, WARNING, and FAILURE.
  • A time.
  • A Message
  • A host.
  • A plugin.
  • A plugin instance (optional).
  • A type.
  • A types instance (optional).
  • Meta-data.
DPDK Enhancements

This section will discuss the Barometer features that were integrated with DPDK.

Measuring Telco Traffic and Performance KPIs

This section will discuss the Barometer features that enable Measuring Telco Traffic and Performance KPIs.

_images/stats_and_timestamps.png

Measuring Telco Traffic and Performance KPIs

  • The very first thing Barometer enabled was a call-back API in DPDK and an associated application that used the API to demonstrate how to timestamp packets and measure packet latency in DPDK (the sample app is called rxtx_callbacks). This was upstreamed to DPDK 2.0 and is represented by the interfaces 1 and 2 in Figure 1.2.
  • The second thing Barometer implemented in DPDK is the extended NIC statistics API, which exposes NIC stats including error stats to the DPDK user by reading the registers on the NIC. This is represented by interface 3 in Figure 1.2.
    • For DPDK 2.1 this API was only implemented for the ixgbe (10Gb) NIC driver, in association with a sample application that runs as a DPDK secondary process and retrieves the extended NIC stats.
    • For DPDK 2.2 the API was implemented for igb, i40e and all the Virtual Functions (VFs) for all drivers.
    • For DPDK 16.07 the API migrated from using string value pairs to using id value pairs, improving the overall performance of the API.
Monitoring DPDK interfaces

With the features Barometer enabled in DPDK to enable measuring Telco traffic and performance KPIs, we can now retrieve NIC statistics including error stats and relay them to a DPDK user. The next step is to enable monitoring of the DPDK interfaces based on the stats that we are retrieving from the NICs, by relaying the information to a higher level Fault Management entity. To enable this Barometer has been enabling a number of plugins for collectd.

DPDK Keep Alive description

SFQM aims to enable fault detection within DPDK, the very first feature to meet this goal is the DPDK Keep Alive Sample app that is part of DPDK 2.2.

DPDK Keep Alive or KA is a sample application that acts as a heartbeat/watchdog for DPDK packet processing cores, to detect application thread failure. The application supports the detection of ‘failed’ DPDK cores and notification to a HA/SA middleware. The purpose is to detect Packet Processing Core fails (e.g. infinite loop) and ensure the failure of the core does not result in a fault that is not detectable by a management entity.

_images/dpdk_ka.png

DPDK Keep Alive Sample Application

Essentially the app demonstrates how to detect ‘silent outages’ on DPDK packet processing cores. The application can be decomposed into two specific parts: detection and notification.

  • The detection period is programmable/configurable but defaults to 5ms if no timeout is specified.
  • The Notification support is enabled by simply having a hook function that where this can be ‘call back support’ for a fault management application with a compliant heartbeat mechanism.
DPDK Keep Alive Sample App Internals

This section provides some explanation of the The Keep-Alive/’Liveliness’ conceptual scheme as well as the DPDK Keep Alive App. The initialization and run-time paths are very similar to those of the L2 forwarding application (see L2 Forwarding Sample Application (in Real and Virtualized Environments) for more information).

There are two types of cores: a Keep Alive Monitor Agent Core (master DPDK core) and Worker cores (Tx/Rx/Forwarding cores). The Keep Alive Monitor Agent Core will supervise worker cores and report any failure (2 successive missed pings). The Keep-Alive/’Liveliness’ conceptual scheme is:

  • DPDK worker cores mark their liveliness as they forward traffic.
  • A Keep Alive Monitor Agent Core runs a function every N Milliseconds to inspect worker core liveliness.
  • If keep-alive agent detects time-outs, it notifies the fault management entity through a call-back function.

Note: Only the worker cores state is monitored. There is no mechanism or agent to monitor the Keep Alive Monitor Agent Core.

DPDK Keep Alive Sample App Code Internals

The following section provides some explanation of the code aspects that are specific to the Keep Alive sample application.

The heartbeat functionality is initialized with a struct rte_heartbeat and the callback function to invoke in the case of a timeout.

rte_global_keepalive_info = rte_keepalive_create(&dead_core, NULL);
if (rte_global_hbeat_info == NULL)
    rte_exit(EXIT_FAILURE, "keepalive_create() failed");

The function that issues the pings hbeat_dispatch_pings() is configured to run every check_period milliseconds.

if (rte_timer_reset(&hb_timer,
        (check_period * rte_get_timer_hz()) / 1000,
        PERIODICAL,
        rte_lcore_id(),
        &hbeat_dispatch_pings, rte_global_keepalive_info
        ) != 0 )
    rte_exit(EXIT_FAILURE, "Keepalive setup failure.\n");

The rest of the initialization and run-time path follows the same paths as the the L2 forwarding application. The only addition to the main processing loop is the mark alive functionality and the example random failures.

rte_keepalive_mark_alive(&rte_global_hbeat_info);
cur_tsc = rte_rdtsc();

/* Die randomly within 7 secs for demo purposes.. */
if (cur_tsc - tsc_initial > tsc_lifetime)
break;

The rte_keepalive_mark_alive() function simply sets the core state to alive.

static inline void
rte_keepalive_mark_alive(struct rte_heartbeat *keepcfg)
{
    keepcfg->state_flags[rte_lcore_id()] = 1;
}

Keep Alive Monitor Agent Core Monitoring Options The application can run on either a host or a guest. As such there are a number of options for monitoring the Keep Alive Monitor Agent Core through a Local Agent on the compute node:

Application Location DPDK KA LOCAL AGENT
HOST X HOST/GUEST
GUEST X HOST/GUEST

For the first implementation of a Local Agent SFQM will enable:

Application Location DPDK KA LOCAL AGENT
HOST X HOST

Through extending the dpdkstat plugin for collectd with KA functionality, and integrating the extended plugin with Monasca for high performing, resilient, and scalable fault detection.

OPNFV Barometer configuration Guide
Barometer Configuration Guide

This document provides guidelines on how to install and configure Barometer with Apex. The deployment script installs and enables a series of collectd plugins on the compute node(s), which collect and dispatch specific metrics and events from the platform.

Pre-configuration activities

Deploying the Barometer components in Apex is done through the deploy-opnfv command by selecting a scenario-file which contains the barometer: true option. These files are located on the Jump Host in the /etc/opnfv-apex/ folder. Two scenarios are pre-defined to include Barometer, and they are: os-nosdn-bar-ha.yaml and os-nosdn-bar-noha.yaml.

$ cd /etc/opnfv-apex
$ opnfv-deploy -d os-nosdn-bar-ha.yaml -n network_settings.yaml -i inventory.yaml –- debug
Hardware configuration

There’s no specific Hardware configuration required. However, the intel_rdt plugin works only on platforms with Intel CPUs.

Feature configuration

All Barometer plugins are automatically deployed on all compute nodes. There is no option to selectively install only a subset of plugins. Any custom disabling or configuration must be done directly on the compute node(s) after the deployment is completed.

Upgrading the plugins

The Barometer components are built-in in the Apex ISO image, and respectively the Apex RPMs. There is no simple way to update only the Barometer plugins in an existing deployment.

Barometer post installation procedures

This document describes briefly the methods of validating the Barometer installation.

Automated post installation activities

The Barometer test-suite in Functest is called barometercollectd and is part of the Features tier. Running these tests is done automatically by the OPNFV deployment pipeline on the supported scenarios. The testing consists of basic verifications that each plugin is functional per their default configurations. Inside the Functest container, the detailed results can be found in the /home/opnfv/functest/results/barometercollectd.log.

Barometer post configuration procedures

The functionality for each plugin (such as enabling/disabling and configuring its capabilities) is controlled as described in the User Guide through their individual .conf file located in the /etc/collectd/collectd.conf.d/ folder on the compute node(s). In order for any changes to take effect, the collectd service must be stopped and then started again.

Platform components validation

The following steps describe how to perform a simple “manual” testing of the Barometer components:

  1. Connect to any compute node and ensure that the collectd service is running. The log file collectd.log should contain no errors and should indicate that each plugin was successfully loaded. For example, from the Jump Host:

    $ opnfv-util overcloud compute0
    $ ls /etc/collectd/collectd.conf.d/
    $ systemctl status collectd
    $ vi /opt/stack/collectd.log
    

    The following plugings should be found loaded: aodh, gnocchi, hugepages, intel_rdt, mcelog, ovs_events, ovs_stats, snmp, virt

  2. On the compute node, induce an event monitored by the plugins; e.g. a corrected memory error:

    $ git clone https://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
    $ cd mce-inject
    $ make
    $ modprobe mce-inject
    

    Modify the test/corrected script to include the following:

    CPU 0 BANK 0
    STATUS 0xcc00008000010090
    ADDR 0x0010FFFFFFF
    

    Inject the error:

    $ ./mce-inject < test/corrected
    
  3. Connect to the controller and query the monitoring services. Make sure the overcloudrc.v3 file has been copied to the controller (from the undercloud VM or from the Jump Host) in order to be able to authenticate for OpenStack services.

    $ opnfv-util overcloud controller0
    $ su
    $ source overcloudrc.v3
    $ gnocchi metric list
    $ aodh alarm list
    

    The output for the gnocchi and aodh queries should be similar to the excerpts below:

    +--------------------------------------+---------------------+------------------------------------------------------------------------------------------------------------+-----------+-------------+
    | id                                   | archive_policy/name | name                                                                                                       | unit      | resource_id |
    +--------------------------------------+---------------------+------------------------------------------------------------------------------------------------------------+-----------+-------------+
      [...]
    | 0550d7c1-384f-4129-83bc-03321b6ba157 | high                | overcloud-novacompute-0.jf.intel.com-hugepages-mm-2048Kb@vmpage_number.free                                | Pages     | None        |
    | 0cf9f871-0473-4059-9497-1fea96e5d83a | high                | overcloud-novacompute-0.jf.intel.com-hugepages-node0-2048Kb@vmpage_number.free                             | Pages     | None        |
    | 0d56472e-99d2-4a64-8652-81b990cd177a | high                | overcloud-novacompute-0.jf.intel.com-hugepages-node1-1048576Kb@vmpage_number.used                          | Pages     | None        |
    | 0ed71a49-6913-4e57-a475-d30ca2e8c3d2 | high                | overcloud-novacompute-0.jf.intel.com-hugepages-mm-1048576Kb@vmpage_number.used                             | Pages     | None        |
    | 11c7be53-b2c1-4c0e-bad7-3152d82c6503 | high                | overcloud-novacompute-0.jf.intel.com-mcelog-                                                               | None      | None        |
    |                                      |                     | SOCKET_0_CHANNEL_any_DIMM_any@errors.uncorrected_memory_errors_in_24h                                      |           |             |
    | 120752d4-385e-4153-aed8-458598a2a0e0 | high                | overcloud-novacompute-0.jf.intel.com-cpu-24@cpu.interrupt                                                  | jiffies   | None        |
    | 1213161e-472e-4e1b-9e56-5c6ad1647c69 | high                | overcloud-novacompute-0.jf.intel.com-cpu-6@cpu.softirq                                                     | jiffies   | None        |
      [...]
    
    +--------------------------------------+-------+------------------------------------------------------------------+-------+----------+---------+
    | alarm_id                             | type  | name                                                             | state | severity | enabled |
    +--------------------------------------+-------+------------------------------------------------------------------+-------+----------+---------+
    | fbd06539-45dd-42c5-a991-5c5dbf679730 | event | gauge.memory_erros(overcloud-novacompute-0.jf.intel.com-mcelog)  | ok    | moderate | True    |
    | d73251a5-1c4e-4f16-bd3d-377dd1e8cdbe | event | gauge.mcelog_status(overcloud-novacompute-0.jf.intel.com-mcelog) | ok    | moderate | True    |
      [...]
    
OPNFV Barometer User Guide
OPNFV Barometer User Guide
Barometer collectd plugins description

Collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanisms to publish the collected metrics. It supports more than 90 different input and output plugins. Input plugins retrieve metrics and publish them to the collectd deamon, while output plugins publish the data they receive to an end point. collectd also has infrastructure to support thresholding and notification.

Barometer has enabled the following collectd plugins:

  • dpdkstat plugin: A read plugin that retrieves stats from the DPDK extended NIC stats API.
  • dpdkevents plugin: A read plugin that retrieves DPDK link status and DPDK forwarding cores liveliness status (DPDK Keep Alive).
  • gnocchi plugin: A write plugin that pushes the retrieved stats to Gnocchi. It’s capable of pushing any stats read through collectd to Gnocchi, not just the DPDK stats.
  • aodh plugin: A notification plugin that pushes events to Aodh, and creates/updates alarms appropriately.
  • hugepages plugin: A read plugin that retrieves the number of available and free hugepages on a platform as well as what is available in terms of hugepages per socket.
  • Open vSwitch events Plugin: A read plugin that retrieves events from OVS.
  • Open vSwitch stats Plugin: A read plugin that retrieves flow and interface stats from OVS.
  • mcelog plugin: A read plugin that uses mcelog client protocol to check for memory Machine Check Exceptions and sends the stats for reported exceptions.
  • PMU plugin: A read plugin that provides performance counters data on Intel CPUs using Linux perf interface.
  • RDT plugin: A read plugin that provides the last level cache utilization and memory bandwidth utilization.
  • virt: A read plugin that uses virtualization API libvirt to gather statistics about virtualized guests on a system directly from the hypervisor, without a need to install collectd instance on the guest.
  • SNMP Agent: A write plugin that will act as a AgentX subagent that receives and handles queries from SNMP master agent and returns the data collected by read plugins. The SNMP Agent plugin handles requests only for OIDs specified in configuration file. To handle SNMP queries the plugin gets data from collectd and translates requested values from collectd’s internal format to SNMP format. Supports SNMP: get, getnext and walk requests.

All the plugins above are available on the collectd master, except for the Gnocchi and Aodh plugins as they are Python-based plugins and only C plugins are accepted by the collectd community. The Gnocchi and Aodh plugins live in the OpenStack repositories.

Other plugins existing as a pull request into collectd master:

  • Legacy/IPMI: A read plugin that reports platform thermals, voltages, fanspeed, current, flow, power etc. Also, the plugin monitors Intelligent Platform Management Interface (IPMI) System Event Log (SEL) and sends the appropriate notifications based on monitored SEL events.
  • PCIe AER: A read plugin that monitors PCIe standard and advanced errors and sends notifications about those errors.

Third party application in Barometer repository:

  • Open vSwitch PMD stats: An aplication that retrieves PMD stats from OVS. It is run through exec plugin.

Plugins and application included in the Euphrates release:

Write Plugins: aodh plugin, SNMP agent plugin, gnocchi plugin.

Read Plugins/application: Intel RDT plugin, virt plugin, Open vSwitch stats plugin, Open vSwitch PMD stats application.

Collectd capabilities and usage

Note

Plugins included in the OPNFV E release will be built-in for Apex integration and can be configured as shown in the examples below.

The collectd plugins in OPNFV are configured with reasonable defaults, but can be overridden.

Building all Barometer upstreamed plugins from scratch

The plugins that have been merged to the collectd master branch can all be built and configured through the barometer repository.

Note

  • sudo permissions are required to install collectd.
  • These instructions are for Centos 7.

To build all the upstream plugins, clone the barometer repo:

$ git clone https://gerrit.opnfv.org/gerrit/barometer

To install collectd as a service and install all it’s dependencies:

$ cd barometer/systems && ./build_base_machine.sh

This will install collectd as a service and the base install directory will be /opt/collectd.

Sample configuration files can be found in ‘/opt/collectd/etc/collectd.conf.d’

Note

If you don’t want to use one of the Barometer plugins, simply remove the sample config file from ‘/opt/collectd/etc/collectd.conf.d’

Note

If you plan on using the Exec plugin (for OVS_PMD_STATS or for executing scripts on notification generation), the plugin requires a non-root user to execute scripts. By default, collectd_exec user is used in the exec.conf provided in the sample configurations directory under src/collectd in the Barometer repo. These scripts DO NOT create this user. You need to create this user or modify the configuration in the sample configurations directory under src/collectd to use another existing non root user before running build_base_machine.sh.

Note

If you are using any Open vSwitch plugins you need to run:

$ sudo ovs-vsctl set-manager ptcp:6640

After this, you should be able to start collectd as a service

$ sudo systemctl status collectd

If you want to use granfana to display the metrics you collect, please see: grafana guide

For more information on configuring and installing OpenStack plugins for collectd, check out the collectd-ceilometer-plugin GSG.

Below is the per plugin installation and configuration guide, if you only want to install some/particular plugins.

DPDK plugins

Repo: https://github.com/collectd/collectd

Branch: master

Dependencies: DPDK (http://dpdk.org/)

Note

DPDK statistics plugin requires DPDK version 16.04 or later.

To build and install DPDK to /usr please see: https://github.com/collectd/collectd/blob/master/docs/BUILD.dpdkstat.md

Building and installing collectd:

$ git clone https://github.com/collectd/collectd.git
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --enable-debug
$ make
$ sudo make install

Note

If DPDK was installed in a non standard location you will need to specify paths to the header files and libraries using LIBDPDK_CPPFLAGS and LIBDPDK_LDFLAGS. You will also need to add the DPDK library symbols to the shared library path using ldconfig. Note that this update to the shared library path is not persistant (i.e. it will not survive a reboot).

Example of specifying custom paths to DPDK headers and libraries:

$ ./configure LIBDPDK_CPPFLAGS="path to DPDK header files" LIBDPDK_LDFLAGS="path to DPDK libraries"

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc. To configure the dpdkstats plugin you need to modify the configuration file to include:

LoadPlugin dpdkstat
<Plugin dpdkstat>
   Coremask "0xf"
   ProcessType "secondary"
   FilePrefix "rte"
   EnabledPortMask 0xffff
   PortName "interface1"
   PortName "interface2"
</Plugin>

To configure the dpdkevents plugin you need to modify the configuration file to include:

<LoadPlugin dpdkevents>
  Interval 1
</LoadPlugin>

<Plugin "dpdkevents">
  <EAL>
    Coremask "0x1"
    MemoryChannels "4"
    FilePrefix "rte"
  </EAL>
  <Event "link_status">
    SendEventsOnUpdate false
    EnabledPortMask 0xffff
    SendNotification true
  </Event>
  <Event "keep_alive">
    SendEventsOnUpdate false
    LCoreMask "0xf"
    KeepAliveShmName "/dpdk_keepalive_shm_name"
    SendNotification true
  </Event>
</Plugin>

Note

Currently, the DPDK library doesn’t support API to de-initialize the DPDK resources allocated on the initialization. It means, the collectd plugin will not be able to release the allocated DPDK resources (locks/memory/pci bindings etc.) correctly on collectd shutdown or reinitialize the DPDK library if primary DPDK process is restarted. The only way to release those resources is to terminate the process itself. For this reason, the plugin forks off a separate collectd process. This child process becomes a secondary DPDK process which can be run on specific CPU cores configured by user through collectd configuration file (“Coremask” EAL configuration option, the hexadecimal bitmask of the cores to run on).

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

Note

dpdkstat plugin initialization time depends on read interval. It requires 5 read cycles to set up internal buffers and states, during that time no statistics are submitted. Also, if plugin is running and the number of DPDK ports is increased, internal buffers are resized. That requires 3 read cycles and no port statistics are submitted during that time.

The Address-Space Layout Randomization (ASLR) security feature in Linux should be disabled, in order for the same hugepage memory mappings to be present in all DPDK multi-process applications.

To disable ASLR:

$ sudo echo 0 > /proc/sys/kernel/randomize_va_space

To fully enable ASLR:

$ sudo echo 2 > /proc/sys/kernel/randomize_va_space

Warning

Disabling Address-Space Layout Randomization (ASLR) may have security implications. It is recommended to be disabled only when absolutely necessary, and only when all implications of this change have been understood.

For more information on multi-process support, please see: http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html

DPDK stats plugin limitations:

  1. The DPDK primary process application should use the same version of DPDK that collectd DPDK plugin is using;
  2. L2 statistics are only supported;
  3. The plugin has been tested on Intel NIC’s only.

DPDK stats known issues:

  • DPDK port visibility

    When network port controlled by Linux is bound to DPDK driver, the port will not be available in the OS. It affects the SNMP write plugin as those ports will not be present in standard IF-MIB. Thus, additional work is required to be done to support DPDK ports and statistics.

Hugepages Plugin

Repo: https://github.com/collectd/collectd

Branch: master

Dependencies: None, but assumes hugepages are configured.

To configure some hugepages:

$ sudo mkdir -p /mnt/huge
$ sudo mount -t hugetlbfs nodev /mnt/huge
$ sudo bash -c "echo 14336 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages"

Building and installing collectd:

$ git clone https://github.com/collectd/collectd.git
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --enable-hugepages --enable-debug
$ make
$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc. To configure the hugepages plugin you need to modify the configuration file to include:

LoadPlugin hugepages
<Plugin hugepages>
    ReportPerNodeHP  true
    ReportRootHP     true
    ValuesPages      true
    ValuesBytes      false
    ValuesPercentage false
</Plugin>

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

Intel PMU Plugin

Repo: https://github.com/collectd/collectd

Branch: master

Dependencies:

To be suitable for use in collectd plugin shared library libjevents should be compiled as position-independent code. To do this add the following line to pmu-tools/jevents/Makefile:

CFLAGS += -fPIC

Building and installing jevents library:

$ git clone https://github.com/andikleen/pmu-tools.git
$ cd pmu-tools/jevents/
$ make
$ sudo make install

Building and installing collectd:

$ git clone https://github.com/collectd/collectd.git
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --with-libjevents=/usr/local --enable-debug
$ make
$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc. To configure the PMU plugin you need to modify the configuration file to include:

<LoadPlugin intel_pmu>
  Interval 1
</LoadPlugin>
<Plugin "intel_pmu">
  ReportHardwareCacheEvents true
  ReportKernelPMUEvents true
  ReportSoftwareEvents true
</Plugin>

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

Note

The plugin opens file descriptors whose quantity depends on number of monitored CPUs and number of monitored counters. Depending on configuration, it might be required to increase the limit on the number of open file descriptors allowed. This can be done using ‘ulimit -n’ command. If collectd is executed as a service ‘LimitNOFILE=’ directive should be defined in [Service] section of collectd.service file.

Intel RDT Plugin

Repo: https://github.com/collectd/collectd

Branch: master

Dependencies:

Building and installing PQoS/Intel RDT library:

$ git clone https://github.com/01org/intel-cmt-cat.git
$ cd intel-cmt-cat
$ make
$ make install PREFIX=/usr

You will need to insert the msr kernel module:

$ modprobe msr

Building and installing collectd:

$ git clone https://github.com/collectd/collectd.git
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --with-libpqos=/usr/ --enable-debug
$ make
$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc. To configure the RDT plugin you need to modify the configuration file to include:

<LoadPlugin intel_rdt>
  Interval 1
</LoadPlugin>
<Plugin "intel_rdt">
  Cores ""
</Plugin>

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

IPMI Plugin

Repo: https://github.com/collectd/collectd

Branch: feat_ipmi_events, feat_ipmi_analog

Dependencies: OpenIPMI library (http://openipmi.sourceforge.net/)

The IPMI plugin is already implemented in the latest collectd and sensors like temperature, voltage, fanspeed, current are already supported there. The list of supported IPMI sensors has been extended and sensors like flow, power are supported now. Also, a System Event Log (SEL) notification feature has been introduced.

  • The feat_ipmi_events branch includes new SEL feature support in collectd IPMI plugin. If this feature is enabled, the collectd IPMI plugin will dispatch notifications about new events in System Event Log.
  • The feat_ipmi_analog branch includes the support of extended IPMI sensors in collectd IPMI plugin.

Install dependencies

On Centos, install OpenIPMI library:

$ sudo yum install OpenIPMI ipmitool

Anyway, it’s recommended to use the latest version of the OpenIPMI library as it includes fixes of known issues which aren’t included in standard OpenIPMI library package. The latest version of the library can be found at https://sourceforge.net/p/openipmi/code/ci/master/tree/. Steps to install the library from sources are described below.

Remove old version of OpenIPMI library:

$ sudo yum remove OpenIPMI ipmitool

Build and install OpenIPMI library:

$ git clone https://git.code.sf.net/p/openipmi/code openipmi-code
$ cd openipmi-code
$ autoreconf --install
$ ./configure --prefix=/usr
$ make
$ sudo make install

Add the directory containing OpenIPMI*.pc files to the PKG_CONFIG_PATH environment variable:

export PKG_CONFIG_PATH=/usr/lib/pkgconfig

Enable IPMI support in the kernel:

$ sudo modprobe ipmi_devintf
$ sudo modprobe ipmi_si

Note

If HW supports IPMI, the /dev/ipmi0 character device will be created.

Clone and install the collectd IPMI plugin:

$ git clone https://github.com/collectd/collectd
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --enable-debug
$ make
$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc. To configure the IPMI plugin you need to modify the file to include:

LoadPlugin ipmi
<Plugin ipmi>
   <Instance "local">
     SELEnabled true # only feat_ipmi_events branch supports this
   </Instance>
</Plugin>

Note

By default, IPMI plugin will read all available analog sensor values, dispatch the values to collectd and send SEL notifications.

For more information on the IPMI plugin parameters and SEL feature configuration, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

Extended analog sensors support doesn’t require additional configuration. The usual collectd IPMI documentation can be used:

IPMI documentation:

Mcelog Plugin

Repo: https://github.com/collectd/collectd

Branch: master

Dependencies: mcelog

Start by installing mcelog.

Note

The kernel has to have CONFIG_X86_MCE enabled. For 32bit kernels you need atleast a 2.6,30 kernel.

On Centos:

$ sudo yum install mcelog

Or build from source

$ git clone https://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git
$ cd mcelog
$ make
... become root ...
$ make install
$ cp mcelog.service /etc/systemd/system/
$ systemctl enable mcelog.service
$ systemctl start mcelog.service

Verify you got a /dev/mcelog. You can verify the daemon is running completely by running:

$ mcelog --client

This should query the information in the running daemon. If it prints nothing that is fine (no errors logged yet). More info @ http://www.mcelog.org/installation.html

Modify the mcelog configuration file “/etc/mcelog/mcelog.conf” to include or enable:

socket-path = /var/run/mcelog-client
[dimm]
dimm-tracking-enabled = yes
dmi-prepopulate = yes
uc-error-threshold = 1 / 24h
ce-error-threshold = 10 / 24h

[socket]
socket-tracking-enabled = yes
mem-uc-error-threshold = 100 / 24h
mem-ce-error-threshold = 100 / 24h
mem-ce-error-log = yes

[page]
memory-ce-threshold = 10 / 24h
memory-ce-log = yes
memory-ce-action = soft

[trigger]
children-max = 2
directory = /etc/mcelog

Clone and install the collectd mcelog plugin:

$ git clone https://github.com/collectd/collectd
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --enable-debug
$ make
$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc. To configure the mcelog plugin you need to modify the configuration file to include:

<LoadPlugin mcelog>
  Interval 1
</LoadPlugin>
<Plugin mcelog>
  <Memory>
    McelogClientSocket "/var/run/mcelog-client"
    PersistentNotification false
  </Memory>
  #McelogLogfile "/var/log/mcelog"
</Plugin>

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

Simulating a Machine Check Exception can be done in one of 3 ways:

  • Running $make test in the mcelog cloned directory - mcelog test suite
  • using mce-inject
  • using mce-test

mcelog test suite:

It is always a good idea to test an error handling mechanism before it is really needed. mcelog includes a test suite. The test suite relies on mce-inject which needs to be installed and in $PATH.

You also need the mce-inject kernel module configured (with CONFIG_X86_MCE_INJECT=y), compiled, installed and loaded:

$ modprobe mce-inject

Then you can run the mcelog test suite with

$ make test

This will inject different classes of errors and check that the mcelog triggers runs. There will be some kernel messages about page offlining attempts. The test will also lose a few pages of memory in your system (not significant).

Note

This test will kill any running mcelog, which needs to be restarted manually afterwards.

mce-inject:

A utility to inject corrected, uncorrected and fatal machine check exceptions

$ git clone https://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
$ cd mce-inject
$ make
$ modprobe mce-inject

Modify the test/corrected script to include the following:

CPU 0 BANK 0
STATUS 0xcc00008000010090
ADDR 0x0010FFFFFFF

Inject the error: .. code:: bash

$ ./mce-inject < test/corrected

Note

The uncorrected and fatal scripts under test will cause a platform reset. Only the fatal script generates the memory errors**. In order to quickly emulate uncorrected memory errors and avoid host reboot following test errors from mce-test suite can be injected:

$ mce-inject  mce-test/cases/coverage/soft-inj/recoverable_ucr/data/srao_mem_scrub

mce-test:

In addition a more in-depth test of the Linux kernel machine check facilities can be done with the mce-test test suite. mce-test supports testing uncorrected error handling, real error injection, handling of different soft offlining cases, and other tests.

Corrected memory error injection:

To inject corrected memory errors:

  • Remove sb_edac and edac_core kernel modules: rmmod sb_edac rmmod edac_core
  • Insert einj module: modprobe einj param_extension=1
  • Inject an error by specifying details (last command should be repeated at least two times):
$ APEI_IF=/sys/kernel/debug/apei/einj
$ echo 0x8 > $APEI_IF/error_type
$ echo 0x01f5591000 > $APEI_IF/param1
$ echo 0xfffffffffffff000 > $APEI_IF/param2
$ echo 1 > $APEI_IF/notrigger
$ echo 1 > $APEI_IF/error_inject
  • Check the MCE statistic: mcelog –client. Check the mcelog log for injected error details: less /var/log/mcelog.
Open vSwitch Plugins

OvS Plugins Repo: https://github.com/collectd/collectd

OvS Plugins Branch: master

OvS Events MIBs: The SNMP OVS interface link status is provided by standard IF-MIB (http://www.net-snmp.org/docs/mibs/IF-MIB.txt)

Dependencies: Open vSwitch, Yet Another JSON Library (https://github.com/lloyd/yajl)

On Centos, install the dependencies and Open vSwitch:

$ sudo yum install yajl-devel

Steps to install Open vSwtich can be found at http://docs.openvswitch.org/en/latest/intro/install/fedora/

Start the Open vSwitch service:

$ sudo service openvswitch-switch start

Configure the ovsdb-server manager:

$ sudo ovs-vsctl set-manager ptcp:6640

Clone and install the collectd ovs plugin:

$ git clone $REPO
$ cd collectd
$ git checkout master
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --enable-debug
$ make
$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc. To configure the OVS events plugin you need to modify the configuration file to include:

<LoadPlugin ovs_events>
   Interval 1
</LoadPlugin>
<Plugin ovs_events>
   Port "6640"
   Address "127.0.0.1"
   Socket "/var/run/openvswitch/db.sock"
   Interfaces "br0" "veth0"
   SendNotification true
</Plugin>

To configure the OVS stats plugin you need to modify the configuration file to include:

<LoadPlugin ovs_stats>
   Interval 1
</LoadPlugin>
<Plugin ovs_stats>
   Port "6640"
   Address "127.0.0.1"
   Socket "/var/run/openvswitch/db.sock"
   Bridges "br0"
</Plugin>

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

OVS PMD stats

Repo: https://gerrit.opnfv.org/gerrit/barometer

Prequistes: 1. Open vSwitch dependencies are installed. 2. Open vSwitch service is running. 3. Ovsdb-server manager is configured. You can refer Open vSwitch Plugins section above for each one of them.

OVS PMD stats application is run through the exec plugin.

To configure the OVS PMD stats application you need to modify the exec plugin configuration to include:

<LoadPlugin exec>
   Interval 1
</LoadPlugin
<Plugin exec>
    Exec "user:group" "<path to ovs_pmd_stat.sh>"
</Plugin>

Note

Exec plugin configuration has to be changed to use appropriate user before starting collectd service.

ovs_pmd_stat.sh calls the script for OVS PMD stats application with its argument:

sudo python /usr/local/src/ovs_pmd_stats.py" "--socket-pid-file"
"/var/run/openvswitch/ovs-vswitchd.pid"
SNMP Agent Plugin

Repo: https://github.com/collectd/collectd

Branch: master

Dependencies: NET-SNMP library

Start by installing net-snmp and dependencies.

On Centos 7:

$ sudo yum install net-snmp net-snmp-libs net-snmp-utils net-snmp-devel
$ sudo systemctl start snmpd.service

go to the snmp configuration steps.

From source:

Clone and build net-snmp:

$ git clone https://github.com/haad/net-snmp.git
$ cd net-snmp
$ ./configure --with-persistent-directory="/var/net-snmp" --with-systemd --enable-shared --prefix=/usr
$ make

Become root

$ make install

Copy default configuration to persistent folder:

$ cp EXAMPLE.conf /usr/share/snmp/snmpd.conf

Set library path and default MIB configuration:

$ cd ~/
$ echo export LD_LIBRARY_PATH=/usr/lib >> .bashrc
$ net-snmp-config --default-mibdirs
$ net-snmp-config --snmpconfpath

Configure snmpd as a service:

$ cd net-snmp
$ cp ./dist/snmpd.service /etc/systemd/system/
$ systemctl enable snmpd.service
$ systemctl start snmpd.service

Add the following line to snmpd.conf configuration file /etc/snmp/snmpd.conf to make all OID tree visible for SNMP clients:

view    systemview    included   .1

To verify that SNMP is working you can get IF-MIB table using SNMP client to view the list of Linux interfaces:

$ snmpwalk -v 2c -c public localhost IF-MIB::interfaces

Get the default MIB location:

$ net-snmp-config --default-mibdirs
/opt/stack/.snmp/mibs:/usr/share/snmp/mibs

Install Intel specific MIBs (if needed) into location received by net-snmp-config command (e.g. /usr/share/snmp/mibs).

$ git clone https://gerrit.opnfv.org/gerrit/barometer.git
$ sudo cp -f barometer/mibs/*.txt /usr/share/snmp/mibs/
$ sudo systemctl restart snmpd.service

Clone and install the collectd snmp_agent plugin:

$ cd ~
$ git clone https://github.com/collectd/collectd
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --enable-debug --enable-snmp --with-libnetsnmp
$ make
$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf) can be found at /opt/collectd/etc.

SNMP Agent plugin is a generic plugin and cannot work without configuration. To configure the snmp_agent plugin you need to modify the configuration file to include OIDs mapped to collectd types. The following example maps scalar memAvailReal OID to value represented as free memory type of memory plugin:

LoadPlugin snmp_agent
<Plugin "snmp_agent">
  <Data "memAvailReal">
    Plugin "memory"
    Type "memory"
    TypeInstance "free"
    OIDs "1.3.6.1.4.1.2021.4.6.0"
  </Data>
</Plugin>

The snmpwalk command can be used to validate the collectd configuration:

$ snmpwalk -v 2c -c public localhost 1.3.6.1.4.1.2021.4.6.0
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 135237632 kB

Limitations

  • Object instance with Counter64 type is not supported in SNMPv1. When GetNext request is received, Counter64 type objects will be skipped. When Get request is received for Counter64 type object, the error will be returned.
  • Interfaces that are not visible to Linux like DPDK interfaces cannot be retreived using standard IF-MIB tables.

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

For more details on AgentX subagent, please see: http://www.net-snmp.org/tutorial/tutorial-5/toolkit/demon/

virt plugin

Repo: https://github.com/collectd/collectd

Branch: master

Dependencies: libvirt (https://libvirt.org/), libxml2

On Centos, install the dependencies:

$ sudo yum install libxml2-dev libpciaccess-devel yajl-devel device-mapper-devel

Install libvirt:

libvirt version in package manager might be quite old and offer only limited functionality. Hence, building and installing libvirt from sources is recommended. Detailed instructions can bet found at: https://libvirt.org/compiling.html

Certain metrics provided by the plugin have a requirement on a minimal version of the libvirt API. File system information statistics require a Guest Agent (GA) to be installed and configured in a VM. User must make sure that installed GA version supports retrieving file system information. Number of Performance monitoring events metrics depends on running libvirt daemon version.

Note

Please keep in mind that RDT metrics (part of Performance monitoring events) have to be supported by hardware. For more details on hardware support, please see: https://github.com/01org/intel-cmt-cat

Additionally perf metrics cannot be collected if Intel RDT plugin is enabled.

libvirt version can be checked with following commands:

$ virsh --version
$ libvirtd --version
Extended statistics requirements
Statistic Min. libvirt API version Requires GA
Domain reason 0.9.2 No
Disk errors 0.9.10 No
Job statistics 1.2.9 No
File system information 1.2.11 Yes
Performance monitoring events 1.3.3 No

Start libvirt daemon:

$ systemctl start libvirtd

Create domain (VM) XML configuration file. For more information on domain XML format and examples, please see: https://libvirt.org/formatdomain.html

Note

Installing additional hypervisor dependencies might be required before deploying virtual machine.

Create domain, based on created XML file:

$ virsh define DOMAIN_CFG_FILE.xml

Start domain:

$ virsh start DOMAIN_NAME

Check if domain is running:

$ virsh list

Check list of available Performance monitoring events and their settings:

$ virsh perf DOMAIN_NAME

Enable or disable Performance monitoring events for domain:

$ virsh perf DOMAIN_NAME [--enable | --disable] EVENT_NAME --live

Clone and install the collectd virt plugin:

$ git clone $REPO
$ cd collectd
$ ./build.sh
$ ./configure --enable-syslog --enable-logfile --enable-debug
$ make
$ sudo make install

Where $REPO is equal to information provided above.

This will install collectd to /opt/collectd. The collectd configuration file collectd.conf can be found at /opt/collectd/etc. To load the virt plugin user needs to modify the configuration file to include:

LoadPlugin virt

Additionally, user can specify plugin configuration parameters in this file, such as connection URL, domain name and much more. By default extended virt plugin statistics are disabled. They can be enabled with ExtraStats option.

<Plugin virt>
   RefreshInterval 60
   ExtraStats "cpu_util disk disk_err domain_state fs_info job_stats_background pcpu perf vcpupin"
</Plugin>

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod

Installing collectd as a service

NOTE: In an OPNFV installation, collectd is installed and configured as a service.

Collectd service scripts are available in the collectd/contrib directory. To install collectd as a service:

$ sudo cp contrib/systemd.collectd.service /etc/systemd/system/
$ cd /etc/systemd/system/
$ sudo mv systemd.collectd.service collectd.service
$ sudo chmod +x collectd.service

Modify collectd.service

[Service]
ExecStart=/opt/collectd/sbin/collectd
EnvironmentFile=-/opt/collectd/etc/
EnvironmentFile=-/opt/collectd/etc/
CapabilityBoundingSet=CAP_SETUID CAP_SETGID

Reload

$ sudo systemctl daemon-reload
$ sudo systemctl start collectd.service
$ sudo systemctl status collectd.service should show success
Additional useful plugins

Exec Plugin : Can be used to show you when notifications are being generated by calling a bash script that dumps notifications to file. (handy for debug). Modify /opt/collectd/etc/collectd.conf:

LoadPlugin exec
<Plugin exec>
#   Exec "user:group" "/path/to/exec"
   NotificationExec "user" "<path to barometer>/barometer/src/collectd/collectd_sample_configs/write_notification.sh"
</Plugin>

write_notification.sh (just writes the notification passed from exec through STDIN to a file (/tmp/notifications)):

#!/bin/bash
rm -f /tmp/notifications
while read x y
do
  echo $x$y >> /tmp/notifications
done

output to /tmp/notifications should look like:

Severity:WARNING
Time:1479991318.806
Host:localhost
Plugin:ovs_events
PluginInstance:br-ex
Type:gauge
TypeInstance:link_status
uuid:f2aafeec-fa98-4e76-aec5-18ae9fc74589

linkstate of "br-ex" interface has been changed to "DOWN"
  • logfile plugin: Can be used to log collectd activity. Modify /opt/collectd/etc/collectd.conf to include:
LoadPlugin logfile
<Plugin logfile>
    LogLevel info
    File "/var/log/collectd.log"
    Timestamp true
    PrintSeverity false
</Plugin>
Monitoring Interfaces and Openstack Support
_images/monitoring_interfaces.png

Monitoring Interfaces and Openstack Support

The figure above shows the DPDK L2 forwarding application running on a compute node, sending and receiving traffic. Collectd is also running on this compute node retrieving the stats periodically from DPDK through the dpdkstat plugin and publishing the retrieved stats to OpenStack through the collectd-ceilometer-plugin.

To see this demo in action please checkout: Barometer OPNFV Summit demo

For more information on configuring and installing OpenStack plugins for collectd, check out the collectd-ceilometer-plugin GSG.

Security
VES Application User Guide

The Barometer repository contains a python based application for VES (VNF Event Stream) which receives the collectd specific metrics via Kafka bus, normalizes the metric data into the VES message and sends it into the VES collector.

The application currently supports pushing platform relevant metrics through the additional measurements field for VES.

Collectd has a write_kafka plugin that sends collectd metrics and values to a Kafka Broker. The VES application uses Kafka Consumer to receive metrics from the Kafka Broker.

Install Kafka Broker
  1. Dependencies: install JAVA & Zookeeper.

    JAVA:

    $ sudo apt install default-jre
    

    CentOS 7.x use:

    $ sudo yum install java-1.6.0-openjdk
    

    Zookeeper:

    $ sudo apt install zookeeperd
    

    CentOS 7.x use:

    $ sudo yum install zookeeper
    

    Note

    You may need to add the repository that contains zookeeper. To do so, follow the step below and try to install zookeeper again using steps above. Otherwise, skip next step.

    $ sudo yum install
    https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
    

    CentOS 7.x start zookeeper:

    $ sudo zookeeper-server start
    

    if you get the error message like ZooKeeper data directory is missing at /var/lib/zookeeper during the start of zookeeper, initialize zookeeper data directory using the command below and start zookeeper again, otherwise skip the next step.

    $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
    No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
    

    To test if Zookeeper is running as a daemon.

    $ telnet localhost 2181
    

    Type ‘ruok’ & hit enter. Expected response is ‘imok’. Zookeeper is running fine.

    Note

    VES doesn’t work with version 0.9.4 of kafka-python. The recommended/tested version is 1.3.3.

    $ sudo yum install python-pip
    $ sudo pip install kafka-python
    
  2. Download Kafka:

    $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
    
  3. Extract the archive:

    $ tar -xvzf kafka_2.11-0.11.0.0.tgz
    
  4. Configure Kafka Server:

    $ vi kafka_2.11-0.11.0.0/config/server.properties
    

    By default Kafka does not allow you to delete topics. Please uncomment:

    delete.topic.enable=true
    
  5. Start the Kafka Server.

    Run ‘kafka-server-start.sh’ with nohup as a background process:

    $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
      kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
    
  6. Test Kafka Broker Installation

    To test if the installation worked correctly there are two scripts, producer and consumer scripts. These will allow you to see messages pushed to broker and receive from broker.

    Producer (Publish “Hello World”):

    $ echo "Hello, World" | kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh \
      --broker-list localhost:9092 --topic TopicTest > /dev/null
    

    Consumer (Receive “Hello World”):

    $ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
      localhost:2181 --topic TopicTest --from-beginning
    
Install collectd

Install development tools:

$ sudo yum group install 'Development Tools'

Install Apache Kafka C/C++ client library:

$ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka
$ cd ~/librdkafka
$ git checkout -b v0.9.5 v0.9.5
$ ./configure --prefix=/usr
$ make
$ sudo make install

Build collectd with Kafka support:

$ git clone https://github.com/collectd/collectd.git ~/collectd
$ cd ~/collectd
$ ./build.sh
$ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no
$ make && sudo make install

Configure and start collectd. Create /opt/collectd/etc/collectd.conf collectd configuration file as following:

Note

The following collectd configuration file allows user to run VES application in the guest mode. To run the VES in host mode, please follow the Configure VES in host mode steps.

LoadPlugin logfile
<Plugin logfile>
  LogLevel info
  File "/opt/collectd/var/log/collectd.log"
  Timestamp true
  PrintSeverity false
</Plugin>

LoadPlugin cpu
<Plugin cpu>
  ReportByCpu true
  ReportByState true
  ValuesPercentage true
</Plugin>

LoadPlugin interface
LoadPlugin memory
LoadPlugin load
LoadPlugin disk
LoadPlugin uuid

LoadPlugin write_kafka
<Plugin write_kafka>
  Property "metadata.broker.list" "localhost:9092"
  <Topic "collectd">
    Format JSON
  </Topic>
</Plugin>

Start collectd process as a service as described in Installing collectd as a service.

Setup VES Test Collector

Note

Test Collector setup is required only for VES application testing purposes.

Install dependencies:

$ sudo pip install jsonschema

Clone VES Test Collector:

$ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector

Modify VES Test Collector config file to point to existing log directory and schema file:

$ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
$ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf

Start VES Test Collector:

$ cd ~/evel-test-collector/code/collector
$ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
Setup VES application (guest mode)

Install dependencies:

$ sudo pip install pyyaml

Clone Barometer repo and start the VES application:

$ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer
$ cd ~/barometer/3rd_party/collectd-ves-app/ves_app
$ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log &

Note

The above configuration is used for a localhost. The VES application can be configured to use remote real VES collector and remote Kafka server. To do so, the IP addresses/host names needs to be changed in collector.conf and ves_app_config.conf files accordingly.

Configure VES in host mode

Running the VES in host mode looks like steps described in Setup VES application (guest mode) but with the following exceptions:

  • The host.yaml configuration file should be used instead of guest.yaml file when VES application is running.

  • Collectd should be running on host machine only.

  • Addition libvirtd dependencies needs to be installed on a host where collectd daemon is running. To install those dependencies, see virt plugin section of Barometer user guide.

  • At least one VM instance should be up and running by hypervisor on the host.

  • The next (minimum) configuration needs to be provided to collectd to be able to generate the VES message to VES collector.

    LoadPlugin logfile
    <Plugin logfile>
      LogLevel info
      File "/opt/collectd/var/log/collectd.log"
      Timestamp true
      PrintSeverity false
    </Plugin>
    
    LoadPlugin cpu
    
    LoadPlugin virt
    <Plugin virt>
      Connection "qemu:///system"
      RefreshInterval 60
      HostnameFormat uuid
      PluginInstanceFormat name
      ExtraStats "cpu_util"
    </Plugin>
    
    LoadPlugin write_kafka
    <Plugin write_kafka>
      Property "metadata.broker.list" "localhost:9092"
      <Topic "collectd">
        Format JSON
      </Topic>
    </Plugin>
    

    to apply this configuration, the /opt/collectd/etc/collectd.conf file needs to be modified based on example above and collectd daemon needs to be restarted using the command below:

    $ sudo systemctl restart collectd
    

Note

The list of the plugins can be extented depends on your needs.

VES application configuration description

Details of the Vendor Event Listener REST service

REST resources are defined with respect to a ServerRoot:

ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}

REST resources are of the form:

{ServerRoot}/eventListener/v{apiVersion}`
{ServerRoot}/eventListener/v{apiVersion}/{topicName}`
{ServerRoot}/eventListener/v{apiVersion}/eventBatch`

Within the VES directory (3rd_party/collectd-ves-app/ves_app) there is a configuration file called ves_app.conf. The description of the configuration options are described below:

Domain “host”
VES domain name. It can be IP address or hostname of VES collector (default: 127.0.0.1)
Port port
VES port (default: 30000)
Path “path”
Used as the “optionalRoutingPath” element in the REST path (default: empty)
Topic “path”
Used as the “topicName” element in the REST path (default: empty)
UseHttps true|false
Allow application to use HTTPS instead of HTTP (default: false)
Username “username”
VES collector user name (default: empty)
Password “passwd”
VES collector password (default: empty)
SendEventInterval interval
This configuration option controls how often (sec) collectd data is sent to Vendor Event Listener (default: 20)
ApiVersion version
Used as the “apiVersion” element in the REST path (default: 5.1)
KafkaPort port
Kafka Port (Default 9092)
KafkaBroker host
Kafka Broker domain name. It can be an IP address or hostname of local or remote server (default: localhost)
VES YAML configuration format

The format of the VES message is generated by the VES application based on the YAML schema configuration file provided by user via --events-schema command-line option of the application.

Note

Use --help option of VES application to see the description of all available options

Note

The detailed installation guide of the VES application is described in the VES Application User Guide.

The message defined by YAML schema should correspond to format defined in VES shema definition.

Warning

This section doesn’t explain the YAML language itself, so the knowledge of the YAML language is required before continue reading it!

Since, the VES message output is a JSON format, it’s recommended to understand how YAML document is converted to JSON before starting to create a new YAML definition for the VES. E.g.:

The following YAML document:

---
additionalMeasurements:
  plugin: somename
  instance: someinstance

will be converted to JSON like this:

{
  "additionalMeasurements": {
    "instance": "someinstance",
    "plugin": "somename"
  }
}

Note

The YAML syntax section of PyYAML documentation can be used as a reference for this.

VES message types

The VES agent can generate two type of messages which are sent to the VES collector. Each message type must be specified in the YAML configuration file using a specific YAML tag.

Measurements

This type is used to send a message defined in YAML configuration to the VES collector with a specified interval (default is 20 sec and it’s configurable via command line option of the application). This type can be specified in the configuration using !Measurements tag. For instance:

---
# My comments
My Host Measurements: !Measurements
  ... # message definition
Events

This type is used to send a message defined in YAML configuration to the VES collector when collectd notification is received from Kafka bus (collectd write_kafka plugin). This type can be specified in the configuration using !Events tag. For instance:

---
# My comments
My Events: !Events
  ... # event definition
Collectd metrics in VES

The VES application caches collectd metrics received via Kafka bus. The data is stored in a table format. It’s important to know it before mapping collectd metric values to message defined in YAML configuration file.

VES collectd metric cache example:

host plugin plugin_instance type type_instace time value ds_name interval
localhost cpu   percent user 1509535512.30567 16 value 10
localhost memory   memory free 1509535573.448014 798456 value 10
localhost interface   eth0 if_packets 1509544183.956496 253 rx 10
7ec333e7 virt Ubuntu-12.04.5-LTS percent virt_cpu_total 1509544402.378035 0.2 value 10
7ec333e7 virt Ubuntu-12.04.5-LTS memory rss 1509544638.55119 123405 value 10
7ec333e7 virt Ubuntu-12.04.5-LTS if_octets vnet1 1509544646.27108 67 tx 10
cc659a52 virt Ubuntu-16.04 percent virt_cpu_total 1509544745.617207 0.3 value 10
cc659a52 virt Ubuntu-16.04 memory rss 1509544754.329411 4567 value 10
cc659a52 virt Ubuntu-16.04 if_octets vnet0 1509544760.720381 0 rx 10

It’s possible from YAML configuration file to refer to any field of any row of the table via special YAML tags like ValueItem or ArrayItem. See the Collectd metric reference reference for more details.

Note

The collectd data types file contains map of type to ds_name field. It can be used to get possible value for ds_name field. See the collectd data types description for more details on collectd data types.

Aging of collectd metric

If the metric will not be updated by the collectd during the double metric interval time, it will be removed (aged) from VES internal cache.

VES YAML references

There are four type of references supported by the YAML format: System, Config, Collectd metric and Collectd notification. The format of the reference is the following:

"{<ref type>.<attribute name>}"

Note

Possible values for <ref type> and <attribute name> are described in farther sections.

System reference

This reference is used to get system statistics like time, date etc. The system reference (<ref type> = system) can be used in any place of the YAML configuration file. This type of reference provides the following attributes:

hostname
The name of the host where VES application is running.
id
Unique ID during VES application runtime.
time
Current time in seconds since the Epoch. For example 1509641411.631951.
date
Current date in ISO 8601 format, YYYY-MM-DD. For instance 2017-11-02.

For example:

Date: "{system.date}"
Config reference

This reference is used to get VES configuration described in VES application configuration description sectoin. The reference (<ref type> = config) can be used in any place of the YAML configuration file. This type of reference provides the following attributes:

interval
Measurements dispatch interval. It referenses to SendEventInterval configuration of the VES application.

For example:

Interval: "{config.interval}"
Collectd metric reference

This reference is used to get the attribute value of collectd metric from the VES cache. The reference (<ref type> = vl) can be used ONLY inside Measurements, ValueItem and ArrayItem tags. Using the reference inside a helper tag is also allowed if the helper tag is located inside the tag where the reference is allowed (e.g.: ArrayItem). The <attribute name> name corresponds to the table field name described in Collectd metrics in VES section. For example:

name: "{vl.type}-{vl.type_instance}"
Collectd notification reference

This reference is used to get the attribute value of received collectd notification. The reference (<ref type> = n) can be used ONLY inside Events tag. Using the reference inside a helper tag is also allowed if the helper tag is located inside the Events tag. This type of reference provides the following attributes:

host
The hostname of received collectd notification.
plugin
The plugin name of received collectd notification.
plugin_instance
The plugin instance of the received collectd notification.
type
The type name of the received collectd notification.
type_instance
The type instance of the received collectd notification.
severity
The severity of the received collectd notification.
message
The message of the received collectd notification.

Note

The exact value for each attribute depends on the collectd plugin which may generate the notification. Please refer to the collectd plugin description document to get more details on the specific collectd plugin.

YAML config example:

sourceId: "{n.plugin_instance}"
Collectd metric mapping YAML tags

This section describes the YAML tags used to map collectd metric values in the YAML configuration file.

Measurements tag

This tag is a YAML map which is used to define the VES measurement message. It’s allowed to be used multiple times in the document (e.g.: you can define multiple VES messages in one YAML document). This tag works in the same way as ArrayItem tag does and all keys have the same description/rules.

ValueItem tag

This tag is used to select a collectd metric and get its attribute value using Collectd metric reference. The type of this tag is a YAML array of maps with the possible keys described below.

SELECT (required)
Is a YAML map which describes the select metric criteria. Each key name of the map must correspond to the table field name described in Collectd metrics in VES section.
VALUE (optional)
Describes the value to be assigned. If not provided, the default !Number "{vl.value}" expression is used.
DEFAULT (optional)
Describes the default value which will be assigned if no metric is selected by SELECT criteria.

ValueItem tag description example:

memoryFree: !ValueItem
  - SELECT:
      plugin: memory
      type: memory
      type_instance: rss
  - VALUE: !Bytes2Kibibytes "{vl.value}"
  - DEFAULT: 0

The tag process workflow is described on the figure below.

_images/value-item-parse-workflow.png

YAML ValueItem tag process workflow

ArrayItem tag

This tag is used to select a list of collectd metrics and generate a YAML array of YAML items described by ITEM-DESC key. If no collectd metrics are selected by the given criteria, the empty array will be returned.

SELECT (optional)

Is a YAML map which describes the select metrics criteria. Each key name of the map must correspond to the table field name described in Collectd metrics in VES section. The value of the key may be regular expression. To enable regular expression in the value, the YAML string containing / char at the beginning and at the end should be used. For example:

plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin

The VES application uses the python RE library to work with regular expression specified in the YAML configuration. Please refer to python regular expression syntax documentation for more details on a syntax used by the VES.

Multiple SELECT keys are allowed by the tag. If nor SELECT or INDEX-KEY key is specified, the VES error is generated.

INDEX-KEY (optional)
Is a YAML array which describes the unique fields to be selected by the tag. Each element of array is a YAML string which should be one of the table field name described in Collectd metrics in VES section. Please note, if this key is used, only fields specified by the key can be used as a collectd reference in the ITEM-DESC key.
ITEM-DESC (required)
Is a YAML map which describes each element of the YAML array generated by the tag. Using ArrayItem tags and other Helper YAML tags are also allowed in the definition of the key.

In the example below, the ArrayItem tag is used to generate an array of ITEM-DESC items for each collectd metrics except virt plugin with unique plugin, plugin_instance attribute values.

Measurements: !ArrayItem
  - SELECT:
      plugin: "/^(?!virt).*$/"
  - INDEX-KEY:
      - plugin
      - plugin_instance
  - ITEM-DESC:
      name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"

The tag process workflow is described on the figure below.

_images/array-item-parse-workflow.png

YAML ArrayItem tag process workflow

Collectd event mapping YAML tags

This section describes the YAML tags used to map the collectd notification to the VES event message in the YAML configuration file.

Events tag

This tag is a YAML map which is used to define the VES event message. It’s allowed to be used multiple times in the document (e.g.: you can map multiple collectd notification into VES message in one YAML document). The possible keys of the tag are described below.

CONDITION (optional)
Is a YAML map which describes the select metric criteria. Each key name of the map must correspond to the name of attribute provided by Collectd notification reference. If no such key provided, any collectd notification will map the defined YAML message.
ITEM-DESC (required)
Is a YAML map which describes the message generated by this tag. Only Helper YAML tags are allowed in the definition of the key.

The example of the VES event message which will be generate by the VES application when collectd notification of the virt plugin is triggered is described below.

---
Virt Event: !Events
  - ITEM-DESC:
      event:
        commonEventHeader:
          domain: fault
          eventType: Notification
          sourceId: &event_sourceId "{n.plugin_instance}"
          sourceName: *event_sourceId
          lastEpochMicrosec: !Number "{n.time}"
          startEpochMicrosec: !Number "{n.time}"
        faultFields:
          alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"
          alarmCondition: "{n.severity}"
          faultFieldsVersion: 1.1
  - CONDITION:
      plugin: virt
Helper YAML tags

This section describes the YAML tags used as utilities for formatting the output message. The YAML configuration process workflow is described on the figure below.

_images/parse-work-flow.png

YAML configuration process workflow

Convert string to number tag

The !Number tag is used in YAML configuration file to convert string value into the number type. For instance:

lastEpochMicrosec: !Number "3456"

The output of the tag will be the JSON number.

{
  lastEpochMicrosec: 3456
}
Convert bytes to Kibibytes tag

The !Bytes2Kibibytes tag is used in YAML configuration file to convert bytes into kibibytes (1 kibibyte = 1024 bytes). For instance:

memoryConfigured: !Bytes2Kibibytes 4098
memoryConfigured: !Bytes2Kibibytes "1024"

The output of the tag will be the JSON number.

{
  memoryConfigured: 4
  memoryConfigured: 1
}
Convert one value to another tag

The !MapValue tag is used in YAML configuration file to map one value into another value defined in the configuration. For instance:

Severity: !MapValue
  VALUE: Failure
  TO:
    Failure: Critical
    Error: Warnign

The output of the tag will be the mapped value.

{
  Severity: Critical
}
Strip extra dash tag

The !StripExtraDash tag is used in YAML configuration file to strip extra dashes in the string (dashes at the beginning, at the end and double dashes). For example:

name: !StripExtraDash string-with--extra-dashes-

The output of the tag will be the JSON string with extra dashes removed.

{
  name: string-with-extra-dashes
}
Limitations
  1. Only one document can be defined in the same YAML configuration file.
  2. The collectd notification is not supported by Kafka collectd plugin. Due to this limitation, the collectd notifications cannot be received by the VES application and the defined YAML event will not be generated and sent to the VES collector. Please note, that VES YAML format already supports the events definition and the format is descibed in the document.
OPNFV Barometer Docker User Guide
Barometer docker image description

The intention of this user guide is to outline how to install and test the barometer docker image that can be built from the Dockerfile available in the barometer repository.

Note

The Dockerfile is available in the docker/ directory in the barometer repo. The Dockerfile builds a CentOS 7 docker image.

The barometer docker image gives you a collectd installation that includes all the barometer plugins.

Note

The container MUST be run as a privileged container.

Collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanisms to publish the collected metrics. It supports more than 90 different input and output plugins. Input plugins retrieve metrics and publish them to the collectd deamon, while output plugins publish the data they receive to an end point. collectd also has infrastructure to support thresholding and notification.

Barometer docker image has enabled the following collectd plugins (in addition to the standard collectd plugins):

  • hugepages plugin
  • Open vSwitch events Plugin
  • Open vSwitch stats Plugin
  • mcelog plugin
  • PMU plugin
  • RDT plugin
  • virt
  • SNMP Agent
  • Kafka_write plugin

Plugins and third party applications in Barometer repository that will be available in the docker image:

  • Open vSwitch PMD stats
  • ONAP VES application
  • gnocchi plugin
  • aodh plugin
  • Legacy/IPMI
Installing Docker
On Ubuntu

Note

  • sudo permissions are required to install docker.
  • These instructions are for Ubuntu 16.10

To install docker:

$ sudo apt-get install curl
$ sudo curl -fsSL https://get.docker.com/ | sh
$ sudo usermod -aG docker <username>
$ sudo systemctl status docker

Replace <username> above with an appropriate user name.

On CentOS

Note

  • sudo permissions are required to install docker.
  • These instructions are for CentOS 7

To install docker:

$ sudo yum remove docker docker-common docker-selinux docker-engine
$ sudo yum install -y yum-utils  device-mapper-persistent-data  lvm2
$ sudo yum-config-manager   --add-repo    https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum-config-manager --enable docker-ce-edge
$ sudo yum-config-manager --enable docker-ce-test
$ sudo yum install docker-ce
$ sudo usermod -aG docker <username>
$ sudo systemctl status docker

Replace <username> above with an appropriate user name.

Note

If this is the first time you are installing a package from a recently added repository, you will be prompted to accept the GPG key, and the key’s fingerprint will be shown. Verify that the fingerprint is correct, and if so, accept the key. The fingerprint should match060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35.

Retrieving key from https://download.docker.com/linux/centos/gpg Importing GPG key 0x621E9F35:

Userid : “Docker Release (CE rpm) <docker@docker.com>” Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://download.docker.com/linux/centos/gpg

Is this ok [y/N]: y

Proxy Configuration:

Note

This applies for both CentOS and Ubuntu.

If you are behind an HTTP or HTTPS proxy server, you will need to add this configuration in the Docker systemd service file.

  1. Create a systemd drop-in directory for the docker service:
$ sudo mkdir -p /etc/systemd/system/docker.service.d

2. Create a file called /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"

Or, if you are behind an HTTPS proxy server, create a file called /etc/systemd/system/docker.service.d/https-proxy.conf that adds the HTTPS_PROXY environment variable:

[Service]
Environment="HTTPS_PROXY=https://proxy.example.com:443/"

Or create a single file with all the proxy configurations: /etc/systemd/system/docker.service.d/proxy.conf

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"
Environment="HTTPS_PROXY=https://proxy.example.com:443/"
Environment="FTP_PROXY=ftp://proxy.example.com:443/"
Environment="NO_PROXY=localhost"
  1. Flush changes:
$ sudo systemctl daemon-reload
  1. Restart Docker:
$ sudo systemctl restart docker
  1. Check docker environment variables:
sudo systemctl show --property=Environment docker
Test docker installation

Note

This applies for both CentOS and Ubuntu.

$ sudo docker run hello-world

The output should be something like:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
5b0f327be733: Pull complete
Digest: sha256:07d5f7800dfe37b8c2196c7b1c524c33808ce2e0f74e7aa00e603295ca9a0972
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:

$ docker run -it ubuntu bash
Build the barometer docker image
$ cd barometer/docker
$ sudo docker build -t barometer_image --build-arg http_proxy=`echo $http_proxy` \
  --build-arg https_proxy=`echo $https_proxy` -f Dockerfile .

Note

In the above mentioned docker build command, http_proxy & https_proxy arguments needs to be passed only if system is behind an HTTP or HTTPS proxy server.

Check the docker images:

$ sudo docker images

Output should contain a barometer image:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
barometer_image     latest              05f2a3edd96b        3 hours ago         1.2GB
centos              7                   196e0ce0c9fb        4 weeks ago         197MB
centos              latest              196e0ce0c9fb        4 weeks ago         197MB
hello-world         latest              05a3bd381fc2        4 weeks ago         1.84kB

Run the barometer docker image:

$ sudo docker run -tid --net=host -v `pwd`/../src/collectd_sample_configs:/opt/collectd/etc/collectd.conf.d \
 -v /var/run:/var/run -v /tmp:/tmp --privileged barometer_image /run_collectd.sh

Note

The docker barometer image contains configuration for all the collectd plugins. In the command above we are overriding /opt/collectd/etc/collectd.conf.d by mounting a host directory pwd/../src/collectd_sample_configs thta contains only the sample configurations we are interested in running. It’s important to do this if you don’t have DPDK, or RDT installed on the host.

To make some changes run:

sudo docker exec -ti barometer_image /bin/bash

Check your docker image is running

sudo docker ps
Build the influxdb + Grafana docker images

On the node where you want to run influxdb + grafana: 1. Start by installing docker compose

$ sudo curl -L https://github.com/docker/compose/releases/download/1.17.0/docker-compose-`uname -s`-`uname -m` -o /usr/bin/docker-compose

Note

Use the latest Compose release number in the download command. The above command is an example, and it may become out-of-date. To ensure you have the latest version, check the Compose repository release page on GitHub.

  1. Apply executable permissions to the binary:
$ sudo chmod +x /usr/bin/docker-compose
  1. Test the installation.
$ sudo docker-compose --version
  1. Run the get_types_db.sh script in barometer/docker
  2. Run the docker containers:
$ sudo docker-compose up -d
  1. Check your docker images are running
$ sudo docker ps
  1. Run the script to create the CPU dashboard barometer/docker:
$ cd dashboards && ./configure_grafana.sh
  1. Connect to <host_ip>:3000 with a browser and log into grafana: admin/admin

Compass4Nfv

Compass4NFV Development Overview
Introduction of Containerized Compass

Containerized Compass uses five compass containers instead of a single VM.

Each container stands for a micro service and compass-core function separates into these five micro services:

  • Compass-deck : RESTful API and DB Handlers for Compass
  • Compass-tasks : Registered tasks and MQ modules for Compass
  • Compass-cobbler : Cobbler container for Compass
  • Compass-db : Database for Compass
  • Compass-mq : Message Queue for Compass

Compass4nfv has several containers to satisfy OPNFV requirements:

  • Compass-tasks-osa : compass-task’s adapter for deployment OpenStack via OpenStack-ansible
  • Compass-tasks-k8s : compass-task’s adapter for deployment Kubernetes
  • Compass-repo-osa-ubuntu : optional container to support OPNFV offfline installation via OpenStack-ansible
  • Compass-repo-osa-centos : optional container to support OPNFV offfline installation via OpenStack-ansible

Picture below shows the new architecture of compass4nfv:

New Archietecture of Compass4nfv

Fig 1. New Archietecture of Compass4nfv

Compass4nfv Installation Instructions
1. Abstract

This document describes how to install the Euphrates release of OPNFV when using Compass4nfv as a deployment tool covering it’s limitations, dependencies and required system resources.

2. Features
2.1. Supported Openstack Version and OS
  OS only OpenStack Liberty OpenStack Mitaka OpenStack Newton OpenStack Ocata
CentOS 7 yes yes yes yes no
Ubuntu trusty yes yes yes no no
Ubuntu xenial yes no yes yes yes
2.2. Supported Openstack Flavor and Features
  OpenStack Liberty OpenStack Mitaka OpenStack Newton OpenStack Ocata
Virtual Deployment Yes Yes Yes Yes
Baremetal Deployment Yes Yes Yes Yes
HA Yes Yes Yes Yes
Ceph Yes Yes Yes Yes
SDN ODL/ONOS Yes Yes Yes Yes*
Compute Node Expansion Yes Yes Yes No
Multi-Nic Support Yes Yes Yes Yes
Boot Recovery Yes Yes Yes Yes
SFC No No Yes Yes
  • ONOS will not be supported in this release.
3. Compass4nfv configuration

This document describes providing guidelines on how to install and configure the Danube release of OPNFV when using Compass as a deployment tool including required software and hardware configurations.

Installation and configuration of host OS, OpenStack, OpenDaylight, ONOS, Ceph etc. can be supported by Compass on Virtual nodes or Bare Metal nodes.

The audience of this document is assumed to have good knowledge in networking and Unix/Linux administration.

3.1. Preconditions

Before starting the installation of the Euphrates release of OPNFV, some planning must be done.

3.1.1. Retrieving the installation tarball

First of all, The installation tarball is needed for deploying your OPNFV environment, it included packages of Compass, OpenStack, OpenDaylight, ONOS and so on.

The stable release tarball can be retrieved via OPNFV software download page

The daily build tarball can be retrieved via OPNFV artifacts repository:

http://artifacts.opnfv.org/compass4nfv.html

NOTE: Search the keyword “compass4nfv/Euphrates” to locate the tarball.

E.g. compass4nfv/euphrates/opnfv-2017-03-29_08-55-09.tar.gz

The name of tarball includes the time of tarball building, you can get the daily tarball according the building time. The git url and sha1 of Compass4nfv are recorded in properties files, According these, the corresponding deployment scripts can be retrieved.

3.1.2. Getting the deployment scripts

To retrieve the repository of Compass4nfv on Jumphost use the following command:

NOTE: PLEASE DO NOT GIT CLONE COMPASS4NFV IN ROOT DIRECTORY(INCLUDE SUBFOLDERS).

To get stable/euphrates release, you can use the following command:

  • git checkout Euphrates.1.0
3.2. Setup Requirements

If you have only 1 Bare Metal server, Virtual deployment is recommended. if more than or equal 3 servers, the Bare Metal deployment is recommended. The minimum number of servers for Bare metal deployment is 3, 1 for JumpServer(Jumphost), 1 for controller, 1 for compute.

3.2.1. Jumphost Requirements

The Jumphost requirements are outlined below:

  1. Ubuntu 14.04 (Pre-installed).
  2. Root access.
  3. libvirt virtualization support.
  4. Minimum 2 NICs.
    • PXE installation Network (Receiving PXE request from nodes and providing OS provisioning)
    • IPMI Network (Nodes power control and set boot PXE first via IPMI interface)
    • External Network (Optional: Internet access)
  5. 16 GB of RAM for a Bare Metal deployment, 64 GB of RAM for a Virtual deployment.
  6. CPU cores: 32, Memory: 64 GB, Hard Disk: 500 GB, (Virtual Deployment needs 1 TB Hard Disk)
3.3. Bare Metal Node Requirements

Bare Metal nodes require:

  1. IPMI enabled on OOB interface for power control.
  2. BIOS boot priority should be PXE first then local hard disk.
  3. Minimum 3 NICs.
    • PXE installation Network (Broadcasting PXE request)
    • IPMI Network (Receiving IPMI command from Jumphost)
    • External Network (OpenStack mgmt/external/storage/tenant network)
3.4. Network Requirements

Network requirements include:

  1. No DHCP or TFTP server running on networks used by OPNFV.
  2. 2-6 separate networks with connectivity between Jumphost and nodes.
    • PXE installation Network
    • IPMI Network
    • Openstack mgmt Network*
    • Openstack external Network*
    • Openstack tenant Network*
    • Openstack storage Network*
  3. Lights out OOB network access from Jumphost with IPMI node enabled (Bare Metal deployment only).
  4. External network has Internet access, meaning a gateway and DNS availability.

The networks with(*) can be share one NIC(Default configuration) or use an exclusive NIC(Reconfigurated in network.yml).

3.5. Execution Requirements (Bare Metal Only)

In order to execute a deployment, one must gather the following information:

  1. IPMI IP addresses of the nodes.
  2. IPMI login information for the nodes (user/pass).
  3. MAC address of Control Plane / Provisioning interfaces of the Bare Metal nodes.
3.6. Configurations

There are three configuration files a user needs to modify for a cluster deployment. network_cfg.yaml for openstack networks on hosts. dha file for host role, IPMI credential and host nic idenfitication (MAC address). deploy.sh for os and openstack version.

4. Configure network

network_cfg.yaml file describes networks configuration for openstack on hosts. It specifies host network mapping and ip assignment of networks to be installed on hosts. Compass4nfv includes a sample network_cfg.yaml under compass4nfv/deploy/conf/network_cfg.yaml

There are three openstack networks to be installed: external, mgmt and storage. These three networks can be shared on one physical nic or on separate nics (multi-nic). The sample included in compass4nfv uses one nic. For multi-nic configuration, see multi-nic configuration.

4.1. Configure openstack network

**! All interface name in network_cfg.yaml must be identified in dha file by mac address !**

Compass4nfv will install networks on host as described in this configuration. It will look for physical nic on host by mac address from dha file and rename nic to the name with that mac address. Therefore, any network interface name that is not identified by mac address in dha file will not be installed correctly as compass4nfv cannot find the nic.

Configure provider network

provider_net_mappings:
  - name: br-prv
    network: physnet
    interface: eth1
    type: ovs
    role:
      - controller
      - compute

The external nic in dha file must be named eth1 with mac address. If user uses a different interface name in dha file, change eth1 to that name here. Note: User cannot use eth0 for external interface name as install/pxe network is named as such.

Configure openstack mgmt&storage network:

sys_intf_mappings:
  - name: mgmt
    interface: eth1
    vlan_tag: 101
    type: vlan
    role:
      - controller
      - compute
- name: storage
    interface: eth1
    vlan_tag: 102
    type: vlan
    role:
      - controller
      - compute

Change vlan_tag of mgmt and storage to corresponding vlan tag configured on switch.

Note: for virtual deployment, there is no need to modify mgmt&storage network.

If using multi-nic feature, i.e, separate nic for mgmt or storage network, user needs to change name to desired nic name (need to match dha file). Please see multi-nic configuration.

4.2. Assign IP address to networks

ip_settings section specifics ip assignment for openstack networks.

User can use default ip range for mgmt&storage network.

for external networks:

- name: external
   ip_ranges:
   - - "192.168.50.210"
     - "192.168.50.220"
   cidr: "192.168.50.0/24"
   gw: "192.168.50.1"
   role:
     - controller
     - compute

Provide at least number of hosts available ip for external IP range(these ips will be assigned to each host). Provide actual cidr and gateway in cidr and gw fields.

configure public IP for horizon dashboard

public_vip:
 ip: 192.168.50.240
 netmask: "24"
 interface: external

Provide an external ip in ip field. This ip cannot be within the ip range assigned to external network configured in pervious section. It will be used for horizon address.

See section 6.2 (Vitual) and 7.2 (BareMetal) for graphs illustrating network topology.

5. Installation on Bare Metal
5.1. Nodes Configuration (Bare Metal Deployment)

The below file is the inventory template of deployment nodes:

“compass4nfv/deploy/conf/hardware_environment/huawei-pod1/dha.yml”

The “dha.yml” is a collectively name for “os-nosdn-nofeature-ha.yml os-ocl-nofeature-ha.yml os-odl_l2-moon-ha.yml etc”.

You can write your own IPMI IP/User/Password/Mac address/roles reference to it.

  • name – Host name for deployment node after installation.
  • ipmiVer – IPMI interface version for deployment node support. IPMI 1.0 or IPMI 2.0 is available.
  • ipmiIP – IPMI IP address for deployment node. Make sure it can access from Jumphost.
  • ipmiUser – IPMI Username for deployment node.
  • ipmiPass – IPMI Password for deployment node.
  • mac – MAC Address of deployment node PXE NIC.
  • interfaces – Host NIC renamed according to NIC MAC addresses when OS provisioning.
  • roles – Components deployed.

Set TYPE/FLAVOR and POWER TOOL

E.g. .. code-block:: yaml

TYPE: baremetal FLAVOR: cluster POWER_TOOL: ipmitool

Set ipmiUser/ipmiPass and ipmiVer

E.g.

ipmiUser: USER
ipmiPass: PASSWORD
ipmiVer: '2.0'

Assignment of different roles to servers

E.g. Openstack only deployment roles setting

hosts:
  - name: host1
    mac: 'F8:4A:BF:55:A2:8D'
    interfaces:
       - eth1: 'F8:4A:BF:55:A2:8E'
    ipmiIp: 172.16.130.26
    roles:
      - controller
      - ha

  - name: host2
    mac: 'D8:49:0B:DA:5A:B7'
    interfaces:
      - eth1: 'D8:49:0B:DA:5A:B8'
    ipmiIp: 172.16.130.27
    roles:
      - compute

NOTE: THE ‘ha’ role MUST BE SELECTED WITH CONTROLLERS, EVEN THERE IS ONLY ONE CONTROLLER NODE.

E.g. Openstack and ceph deployment roles setting

hosts:
  - name: host1
    mac: 'F8:4A:BF:55:A2:8D'
    interfaces:
       - eth1: 'F8:4A:BF:55:A2:8E'
    ipmiIp: 172.16.130.26
    roles:
      - controller
      - ha
      - ceph-adm
      - ceph-mon

  - name: host2
    mac: 'D8:49:0B:DA:5A:B7'
    interfaces:
      - eth1: 'D8:49:0B:DA:5A:B8'
    ipmiIp: 172.16.130.27
    roles:
      - compute
      - ceph-osd

E.g. Openstack and ODL deployment roles setting

hosts:
  - name: host1
    mac: 'F8:4A:BF:55:A2:8D'
    interfaces:
       - eth1: 'F8:4A:BF:55:A2:8E'
    ipmiIp: 172.16.130.26
    roles:
      - controller
      - ha
      - odl

  - name: host2
    mac: 'D8:49:0B:DA:5A:B7'
    interfaces:
      - eth1: 'D8:49:0B:DA:5A:B8'
    ipmiIp: 172.16.130.27
    roles:
      - compute

E.g. Openstack and ONOS deployment roles setting

hosts:
  - name: host1
    mac: 'F8:4A:BF:55:A2:8D'
    interfaces:
       - eth1: 'F8:4A:BF:55:A2:8E'
    ipmiIp: 172.16.130.26
    roles:
      - controller
      - ha
      - onos

  - name: host2
    mac: 'D8:49:0B:DA:5A:B7'
    interfaces:
      - eth1: 'D8:49:0B:DA:5A:B8'
    ipmiIp: 172.16.130.27
    roles:
      - compute
5.2. Network Configuration (Bare Metal Deployment)

Before deployment, there are some network configuration to be checked based on your network topology.Compass4nfv network default configuration file is “compass4nfv/deploy/conf/hardware_environment/huawei-pod1/network.yml”. This file is an example, you can customize by yourself according to specific network environment.

In this network.yml, there are several config sections listed following(corresponed to the ordre of the config file):

5.2.1. Provider Mapping
  • name – provider network name.
  • network – default as physnet, do not change it.
  • interfaces – the NIC or Bridge attached by the Network.
  • type – the type of the NIC or Bridge(vlan for NIC and ovs for Bridge, either).
  • roles – all the possible roles of the host machines which connected by this network(mostly put both controller and compute).
5.2.2. System Interface
  • name – Network name.
  • interfaces – the NIC or Bridge attached by the Network.
  • vlan_tag – if type is vlan, add this tag before ‘type’ tag.
  • type – the type of the NIC or Bridge(vlan for NIC and ovs for Bridge, either).
  • roles – all the possible roles of the host machines which connected by this network(mostly put both controller and compute).
5.2.3. IP Settings
  • name – network name corresponding the the network name in System Interface section one by one.
  • ip_ranges – ip addresses range provided for this network.
  • cidr – the IPv4 address and its associated routing prefix and subnet mask?
  • gw – need to add this line only if network is external.
  • roles – all the possible roles of the host machines which connected by this network(mostly put both controller and compute).
5.2.4. Internal VIP(virtual or proxy IP)
  • ip – virtual or proxy ip address, must be in the same subnet with mgmt network but must not be in the range of mgmt network.
  • netmask – the length of netmask
  • interface – mostly mgmt.
5.2.5. Public VIP
  • ip – virtual or proxy ip address, must be in the same subnet with external network but must not be in the range of external network.
  • netmask – the length of netmask
  • interface – mostly external.
5.2.6. Public Network
  • enable – must be True(if False, you need to set up provider network manually).
  • network – leave it ext-net.
  • type – the type of the ext-net above, such as flat or vlan.
  • segment_id – when the type is vlan, this should be id of vlan.
  • subnet – leave it ext-subnet.
  • provider_network – leave it physnet.
  • router – leave it router-ext.
  • enable_dhcp – must be False.
  • no_gateway – must be False.
  • external_gw – same as gw in ip_settings.
  • floating_ip_cidr – cidr for floating ip, see explanation in ip_settings.
  • floating_ip_start – define range of floating ip with floating_ip_end(this defined range must not be included in ip range of external configured in ip_settings section).
  • floating_ip_end – define range of floating ip with floating_ip_start.

The following figure shows the default network configuration.

+--+                          +--+     +--+
|  |                          |  |     |  |
|  |      +------------+      |  |     |  |
|  +------+  Jumphost  +------+  |     |  |
|  |      +------+-----+      |  |     |  |
|  |             |            |  |     |  |
|  |             +------------+  +-----+  |
|  |                          |  |     |  |
|  |      +------------+      |  |     |  |
|  +------+    host1   +------+  |     |  |
|  |      +------+-----+      |  |     |  |
|  |             |            |  |     |  |
|  |             +------------+  +-----+  |
|  |                          |  |     |  |
|  |      +------------+      |  |     |  |
|  +------+    host2   +------+  |     |  |
|  |      +------+-----+      |  |     |  |
|  |             |            |  |     |  |
|  |             +------------+  +-----+  |
|  |                          |  |     |  |
|  |      +------------+      |  |     |  |
|  +------+    host3   +------+  |     |  |
|  |      +------+-----+      |  |     |  |
|  |             |            |  |     |  |
|  |             +------------+  +-----+  |
|  |                          |  |     |  |
|  |                          |  |     |  |
+-++                          ++-+     +-++
  ^                            ^         ^
  |                            |         |
  |                            |         |
+-+-------------------------+  |         |
|      External Network     |  |         |
+---------------------------+  |         |
       +-----------------------+---+     |
       |       IPMI Network        |     |
       +---------------------------+     |
               +-------------------------+-+
               | PXE(Installation) Network |
               +---------------------------+

The following figure shows the interfaces and nics of JumpHost and deployment nodes in huawei-pod1 network configuration(default one nic for openstack networks).

Single nic scenario

Fig 1. Single nic scenario

The following figure shows the interfaces and nics of JumpHost and deployment nodes in intel-pod8 network configuration(openstack networks are seperated by multiple NICs).

Multiple nics scenario

Fig 2. Multiple nics scenario

5.3. Start Deployment (Bare Metal Deployment)
  1. Edit deploy.sh
1.1. Set OS version for deployment nodes.
Compass4nfv supports ubuntu and centos based openstack newton.

E.g.

# Set OS version for target hosts
# Ubuntu16.04 or CentOS7
export OS_VERSION=xenial
or
export OS_VERSION=centos7

1.2. Set tarball corresponding to your code

E.g.

# Set ISO image corresponding to your code
export ISO_URL=file:///home/compass/compass4nfv.tar.gz
1.3. Set hardware deploy jumpserver PXE NIC. (set eth1 E.g.)
You do not need to set it when virtual deploy.

E.g.

# Set hardware deploy jumpserver PXE NIC
# you need to comment out it when virtual deploy
export INSTALL_NIC=eth1

1.4. Set scenario that you want to deploy

E.g.

nosdn-nofeature scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-nosdn-nofeature-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml

odl_l2-moon scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-odl_l2-moon-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml

odl_l2-nofeature scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-odl_l2-nofeature-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml

odl_l3-nofeature scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-odl_l3-nofeature-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml

odl-sfc deploy scenario sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/os-odl-sfc-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
  1. Run deploy.sh
./deploy.sh
6. Installation on virtual machines
6.1. Quick Start

Only 1 command to try virtual deployment, if you have Internet access. Just Paste it and Run.

curl https://raw.githubusercontent.com/opnfv/compass4nfv/euphrates/quickstart.sh | bash

If you want to deploy noha with1 controller and 1 compute, run the following command

export SCENARIO=os-nosdn-nofeature-noha.yml
export VIRT_NUMBER=2
curl https://raw.githubusercontent.com/opnfv/compass4nfv/stable/euphrates/quickstart.sh | bash
6.2. Nodes Configuration (Virtual Deployment)
6.2.1. virtual machine setting
  • VIRT_NUMBER – the number of nodes for virtual deployment.
  • VIRT_CPUS – the number of CPUs allocated per virtual machine.
  • VIRT_MEM – the memory size(MB) allocated per virtual machine.
  • VIRT_DISK – the disk size allocated per virtual machine.
export VIRT_NUMBER=${VIRT_NUMBER:-5}
export VIRT_CPUS=${VIRT_CPU:-4}
export VIRT_MEM=${VIRT_MEM:-16384}
export VIRT_DISK=${VIRT_DISK:-200G}
6.2.2. roles setting

The below file is the inventory template of deployment nodes:

”./deploy/conf/vm_environment/huawei-virtual1/dha.yml”

The “dha.yml” is a collectively name for “os-nosdn-nofeature-ha.yml os-ocl-nofeature-ha.yml os-odl_l2-moon-ha.yml etc”.

You can write your own address/roles reference to it.

  • name – Host name for deployment node after installation.
  • roles – Components deployed.

Set TYPE and FLAVOR

E.g.

TYPE: virtual
FLAVOR: cluster

Assignment of different roles to servers

E.g. Openstack only deployment roles setting

hosts:
  - name: host1
    roles:
      - controller
      - ha

  - name: host2
    roles:
      - compute

NOTE: IF YOU SELECT MUTIPLE NODES AS CONTROLLER, THE ‘ha’ role MUST BE SELECT, TOO.

E.g. Openstack and ceph deployment roles setting

hosts:
  - name: host1
    roles:
      - controller
      - ha
      - ceph-adm
      - ceph-mon

  - name: host2
    roles:
      - compute
      - ceph-osd

E.g. Openstack and ODL deployment roles setting

hosts:
  - name: host1
    roles:
      - controller
      - ha
      - odl

  - name: host2
    roles:
      - compute

E.g. Openstack and ONOS deployment roles setting

hosts:
  - name: host1
    roles:
      - controller
      - ha
      - onos

  - name: host2
    roles:
      - compute
6.3. Network Configuration (Virtual Deployment)

The same with Baremetal Deployment.

6.4. Start Deployment (Virtual Deployment)
  1. Edit deploy.sh
1.1. Set OS version for deployment nodes.
Compass4nfv supports ubuntu and centos based openstack ocata.

E.g.

# Set OS version for target hosts
# Ubuntu16.04 or CentOS7
export OS_VERSION=xenial
or
export OS_VERSION=centos7

1.2. Set ISO image corresponding to your code

E.g.

# Set ISO image corresponding to your code
export ISO_URL=file:///home/compass/compass4nfv.tar.gz

1.3. Set scenario that you want to deploy

E.g.

nosdn-nofeature scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-nosdn-nofeature-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml

odl_l2-moon scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-odl_l2-moon-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml

odl_l2-nofeature scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-odl_l2-nofeature-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml

odl_l3-nofeature scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-odl_l3-nofeature-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml

odl-sfc deploy scenario sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/vm_environment/os-odl-sfc-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/vm_environment/huawei-virtual1/network.yml
  1. Run deploy.sh
./deploy.sh
7. K8s introduction
7.1. Kubernetes Architecture

Currently Compass can deploy kubernetes as NFVI in 3+2 mode by default.

The following figure shows a typical architecture of Kubernetes.

K8s architecture

Fig 3. K8s architecture

7.1.1. Kube-apiserver

Kube-apiserver exposes the Kubernetes API. It is the front-end for the Kubernetes control plane. It is designed to scale horizontally, that is, it scales by deploying more instances.

7.1.2. Etcd

Etcd is used as Kubernetes’ backing store. All cluster data is stored here. Always have a backup plan for etcd’s data for your Kubernetes cluster.

7.1.3. Kube-controller-manager

Kube-controller-manager runs controllers, which are the background threads that handle routine tasks in the cluster. Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.

These controllers include:

  • Node Controller: Responsible for noticing and responding when nodes go down.
  • Replication Controller: Responsible for maintaining the correct number of pods for every replication controller object in the system.
  • Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods).
  • Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces.
7.1.4. kube-scheduler

Kube-scheduler watches newly created pods that have no node assigned, and selects a node for them to run on.

7.1.5. Kubelet

Kubelet is the primary node agent. It watches for pods that have been assigned to its node (either by apiserver or via local configuration file) and:

  • Mounts the pod’s required volumes.
  • Downloads the pod’s secrets.
  • Runs the pod’s containers via docker (or, experimentally, rkt).
  • Periodically executes any requested container liveness probes.
  • Reports the status of the pod back to the rest of the system, by creating a mirror pod if necessary.
  • Reports the status of the node back to the rest of the system.
7.1.6. Kube-proxy

Kube-proxy enables the Kubernetes service abstraction by maintaining network rules on the host and performing connection forwarding.

7.1.7. Docker

Docker is used for running containers.

7.1.8. POD

A pod is a collection of containers and its storage inside a node of a Kubernetes cluster. It is possible to create a pod with multiple containers inside it. For example, keeping a database container and data container in the same pod.

7.2. Understand Kubernetes Networking in Compass configuration

The following figure shows the Kubernetes Networking in Compass configuration.

Kubernetes Networking in Compass

Fig 4. Kubernetes Networking in Compass

8. Installation of K8s on virtual machines
8.1. Quick Start

Only 1 command to try virtual deployment, if you have Internet access. Just Paste it and Run.

curl https://raw.githubusercontent.com/opnfv/compass4nfv/master/quickstart_k8s.sh | bash

If you want to deploy noha with1 controller and 1 compute, run the following command

export SCENARIO=k8-nosdn-nofeature-noha.yml
export VIRT_NUMBER=2
curl https://raw.githubusercontent.com/opnfv/compass4nfv/stable/euphrates/quickstart_k8s.sh | bash
9. Installation of K8s on Bare Metal
9.1. Nodes Configuration (Bare Metal Deployment)

The below file is the inventory template of deployment nodes:

“compass4nfv/deploy/conf/hardware_environment/huawei-pod1/k8-nosdn-nofeature-ha.yml”

You can write your own IPMI IP/User/Password/Mac address/roles reference to it.

  • name – Host name for deployment node after installation.
  • ipmiVer – IPMI interface version for deployment node support. IPMI 1.0 or IPMI 2.0 is available.
  • ipmiIP – IPMI IP address for deployment node. Make sure it can access from Jumphost.
  • ipmiUser – IPMI Username for deployment node.
  • ipmiPass – IPMI Password for deployment node.
  • mac – MAC Address of deployment node PXE NIC.
  • interfaces – Host NIC renamed according to NIC MAC addresses when OS provisioning.
  • roles – Components deployed.

Set TYPE/FLAVOR and POWER TOOL

E.g. .. code-block:: yaml

TYPE: baremetal FLAVOR: cluster POWER_TOOL: ipmitool

Set ipmiUser/ipmiPass and ipmiVer

E.g.

ipmiUser: USER
ipmiPass: PASSWORD
ipmiVer: '2.0'

Assignment of different roles to servers

E.g. K8s only deployment roles setting

hosts:
  - name: host1
    mac: 'F8:4A:BF:55:A2:8D'
    interfaces:
       - eth1: 'F8:4A:BF:55:A2:8E'
    ipmiIp: 172.16.130.26
    roles:
      - kube_master
      - etcd

  - name: host2
    mac: 'D8:49:0B:DA:5A:B7'
    interfaces:
      - eth1: 'D8:49:0B:DA:5A:B8'
    ipmiIp: 172.16.130.27
    roles:
      - kube_node
9.2. Network Configuration (Bare Metal Deployment)

Before deployment, there are some network configuration to be checked based on your network topology.Compass4nfv network default configuration file is “compass4nfv/deploy/conf/hardware_environment/huawei-pod1/network.yml”. This file is an example, you can customize by yourself according to specific network environment.

In this network.yml, there are several config sections listed following(corresponed to the ordre of the config file):

9.2.1. Provider Mapping
  • name – provider network name.
  • network – default as physnet, do not change it.
  • interfaces – the NIC or Bridge attached by the Network.
  • type – the type of the NIC or Bridge(vlan for NIC and ovs for Bridge, either).
  • roles – all the possible roles of the host machines which connected by this network(mostly put both controller and compute).
9.2.2. System Interface
  • name – Network name.
  • interfaces – the NIC or Bridge attached by the Network.
  • vlan_tag – if type is vlan, add this tag before ‘type’ tag.
  • type – the type of the NIC or Bridge(vlan for NIC and ovs for Bridge, either).
  • roles – all the possible roles of the host machines which connected by this network(mostly put both controller and compute).
9.2.3. IP Settings
  • name – network name corresponding the the network name in System Interface section one by one.
  • ip_ranges – ip addresses range provided for this network.
  • cidr – the IPv4 address and its associated routing prefix and subnet mask?
  • gw – need to add this line only if network is external.
  • roles – all the possible roles of the host machines which connected by this network(mostly put both controller and compute).
9.2.4. Internal VIP(virtual or proxy IP)
  • ip – virtual or proxy ip address, must be in the same subnet with mgmt network but must not be in the range of mgmt network.
  • netmask – the length of netmask
  • interface – mostly mgmt.
9.2.5. Public VIP
  • ip – virtual or proxy ip address, must be in the same subnet with external network but must not be in the range of external network.
  • netmask – the length of netmask
  • interface – mostly external.
9.2.6. Public Network
  • enable – must be True(if False, you need to set up provider network manually).
  • network – leave it ext-net.
  • type – the type of the ext-net above, such as flat or vlan.
  • segment_id – when the type is vlan, this should be id of vlan.
  • subnet – leave it ext-subnet.
  • provider_network – leave it physnet.
  • router – leave it router-ext.
  • enable_dhcp – must be False.
  • no_gateway – must be False.
  • external_gw – same as gw in ip_settings.
  • floating_ip_cidr – cidr for floating ip, see explanation in ip_settings.
  • floating_ip_start – define range of floating ip with floating_ip_end(this defined range must not be included in ip range of external configured in ip_settings section).
  • floating_ip_end – define range of floating ip with floating_ip_start.

The following figure shows the default network configuration.

Kubernetes network configuration

Fig 5. Kubernetes network configuration

9.3. Start Deployment (Bare Metal Deployment)
  1. Edit deploy.sh
1.1. Set OS version for deployment nodes.
Compass4nfv supports ubuntu and centos based openstack newton.

E.g.

# Set OS version for target hosts
# Only CentOS7 supported now
export OS_VERSION=centos7

1.2. Set tarball corresponding to your code

E.g.

# Set ISO image corresponding to your code
export ISO_URL=file:///home/compass/compass4nfv.tar.gz
1.3. Set hardware deploy jumpserver PXE NIC. (set eth1 E.g.)
You do not need to set it when virtual deploy.

E.g.

# Set hardware deploy jumpserver PXE NIC
# you need to comment out it when virtual deploy
export INSTALL_NIC=eth1

1.4. K8s scenario that you want to deploy

E.g.

nosdn-nofeature scenario deploy sample

# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/huawei-pod1/k8-nosdn-nofeature-ha.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
  1. Run deploy.sh
./deploy.sh
10. Offline Deploy

Compass4nfv uses a repo docker container as distro and pip package source to deploy cluster and support complete offline deployment on a jumphost without access internet. Here is the offline deployment instruction:

10.1. Preparation for offline deploy
  1. Download compass.tar.gz from OPNFV artifacts repository (Search compass4nfv in http://artifacts.opnfv.org/ and download an appropriate tarball. Tarball can also be generated by script build.sh in compass4nfv root directory.)
  2. Download the Jumphost preparation package from our httpserver. (Download the jumphost environment package from here. It should be awared that currently we only support ubuntu trusty as offline jumphost OS.)
  3. Clone the compass4nfv code repository.
10.2. Steps of offline deploy
  1. Copy the compass.tar.gz, jh_env_package.tar.gz and the compass4nfv code repository to your jumphost.
  2. Export the local path of the compass.tar.gz and jh_env_package.tar.gz on jumphost. Then you can perform deployment on a offline jumphost.

E.g.

Export the compass4nfv.iso and jh_env_package.tar.gz path

# ISO_URL and JHPKG_URL should be absolute path
export ISO_URL=file:///home/compass/compass4nfv.iso
export JHPKG_URL=file:///home/compass/jh_env_package.tar.gz
  1. Open the OSA offline deployment switch on jumphost.
export OFFLINE_DEPLOY=Enable
  1. Run deploy.sh
./deploy.sh
11. Expansion Guide
11.1. Edit NETWORK File

The below file is the inventory template of deployment nodes:

”./deploy/conf/hardware_environment/huawei-pod1/network.yml”

You need to edit the network.yml which you had edited the first deployment.

NOTE: External subnet’s ip_range should exclude the IPs those have already been used.

11.2. Edit DHA File

The below file is the inventory template of deployment nodes:

”./deploy/conf/hardware_environment/expansion-sample/hardware_cluster_expansion.yml”

You can write your own IPMI IP/User/Password/Mac address/roles reference to it.

  • name – Host name for deployment node after installation.
  • ipmiIP – IPMI IP address for deployment node. Make sure it can access from Jumphost.
  • ipmiUser – IPMI Username for deployment node.
  • ipmiPass – IPMI Password for deployment node.
  • mac – MAC Address of deployment node PXE NIC .

Set TYPE/FLAVOR and POWER TOOL

E.g.

TYPE: baremetal
FLAVOR: cluster
POWER_TOOL: ipmitool

Set ipmiUser/ipmiPass and ipmiVer

E.g.

ipmiUser: USER
ipmiPass: PASSWORD
ipmiVer: '2.0'

Assignment of roles to servers

E.g. Only increase one compute node

hosts:
   - name: host6
     mac: 'E8:4D:D0:BA:60:45'
     interfaces:
        - eth1: '08:4D:D0:BA:60:44'
     ipmiIp: 172.16.131.23
     roles:
       - compute

E.g. Increase two compute nodes

hosts:
   - name: host6
     mac: 'E8:4D:D0:BA:60:45'
     interfaces:
        - eth1: '08:4D:D0:BA:60:44'
     ipmiIp: 172.16.131.23
     roles:
       - compute

   - name: host6
     mac: 'E8:4D:D0:BA:60:78'
     interfaces:
        - eth1: '08:4D:56:BA:60:83'
     ipmiIp: 172.16.131.23
     roles:
       - compute
11.2.1. Start Expansion
  1. Edit network.yml and dha.yml file

    You need to Edit network.yml and virtual_cluster_expansion.yml or hardware_cluster_expansion.yml. Edit the DHA and NETWORK envionment variables. External subnet’s ip_range and management ip should be changed as the first 6 IPs are already taken by the first deployment.

E.g.

--- network.yml     2017-02-16 20:07:10.097878150 +0800
+++ network-expansion.yml   2017-05-03 10:01:34.537379013 +0800
@@ -38,7 +38,7 @@
 ip_settings:
   - name: mgmt
     ip_ranges:
-      - - "172.16.1.1"
+      - - "172.16.1.6"
         - "172.16.1.254"
     cidr: "172.16.1.0/24"
     role:
@@ -47,7 +47,7 @@

   - name: storage
     ip_ranges:
-      - - "172.16.2.1"
+      - - "172.16.2.6"
         - "172.16.2.254"
     cidr: "172.16.2.0/24"
     role:
@@ -56,7 +56,7 @@

   - name: external
     ip_ranges:
-      - - "192.168.116.201"
+      - - "192.168.116.206"
         - "192.168.116.221"
     cidr: "192.168.116.0/24"
     gw: "192.168.116.1"
  1. Edit deploy.sh
2.1. Set EXPANSION and VIRT_NUMBER.
VIRT_NUMBER decide how many virtual machines needs to expand when virtual expansion

E.g.

export EXPANSION="true"
export MANAGEMENT_IP_START="10.1.0.55"
export VIRT_NUMBER=1
export DEPLOY_FIRST_TIME="false"

2.2. Set scenario that you need to expansion

E.g.

# DHA is your dha.yml's path
export DHA=./deploy/conf/hardware_environment/expansion-sample/hardware_cluster_expansion.yml

# NETWORK is your network.yml's path
export NETWORK=./deploy/conf/hardware_environment/huawei-pod1/network.yml
Note: Other environment variable shoud be same as your first deployment.
Please check the environment variable before you run deploy.sh.
  1. Run deploy.sh
./deploy.sh
Compass4nfv Design Guide
1. How to integrate a feature into compass4nfv

This document describes how to integrate a feature (e.g. sdn, moon, kvm, sfc) into compass installer. Follow the steps below, you can achieve the goal.

1.1. Create a role for the feature

Currently Ansible is the main packages installation plugin in the adapters of Compass4nfv, which is used to deploy all the roles listed in the playbooks. (More details about ansible and playbook can be achieved according to the Reference.) The mostly used playbook in compass4nfv is named “HA-ansible-multinodes.yml” located in “your_path_to_compass4nfv/compass4nfv/deploy/ adapters/ansible/openstack/”.

Before you add your role into the playbook, create your role under the directory of “your_path_to_compass4nfv/compass4nfv/deploy/adapters/ansible/roles/”. For example Fig 1 shows some roles currently existed in compass4nfv.

Existed roles in compass4nfv

Fig 1. Existed roles in compass4nfv

Let’s take a look at “moon” and understand the construction of a role. Fig 2 below presents the tree of “moon”.

Tree of moon role

Fig 2. Tree of moon role

There are five directories in moon, which are files, handlers, tasks, templates and vars. Almost every role has such five directories.

For “files”, it is used to store the files you want to copy to the hosts without any modification. These files can be configuration files, code files and etc. Here in moon’s files directory, there are two python files and one configuration file. All of the three files will be copied to controller nodes for some purposes.

For “handlers”, it is used to store some operations frequently used in your tasks. For example, restart the service daemon.

For “tasks”, it is used to store the task yaml files. You need to add the yaml files including the tasks you write to deploy your role on the hosts. Please attention that a main.yml should be existed as the entrance of running tasks. In Fig 2, you can find that there are four yaml files in the tasks directory of moon. The main.yml is the entrance which will call the other three yaml files.

For “templates”, it is used to store the files that you want to replace some variables in them before copying to hosts. These variables are usually defined in “vars” directory. This can avoid hard coding.

For “vars”, it is used to store the yaml files in which the packages and variables are defined. The packages defined here are some generic debian or rpm packages. The script of making repo will scan the packages names here and download them into related PPA. For some special packages, section “Build packages for the feature” will introduce how to handle with special packages. The variables defined here are used in the files in “templates” and “tasks”.

Note: you can get the special packages in the tasks like this:

- name: get the special packages' http server
  shell: awk -F'=' '/compass_server/ {print $2}' /etc/compass.conf
  register: http_server

- name: download odl package
  get_url:
    url: "http://{{ http_server.stdout_lines[0] }}/packages/odl/{{ odl_pkg_url }}"
    dest: /opt/
1.2. Build packages for the feature

In the previous section, we have explained how to build the generic packages for your feature. In this section, we will talk about how to build the special packages used by your feature.

Features building directory in compass4nfv

Fig 3. Features building directory in compass4nfv

Fig 3 shows the tree of “your_path_to_compass4nfv/compass4nfv/repo/features/”. Dockerfile is used to start a docker container to run the scripts in scripts directory. These scripts will download the special feature related packages into the container. What you need to do is to write a shell script to download or build the package you want. And then put the script into “your_path_to_compass4nfv/compass4nfv/repo/features/scripts/”. Attention that, you need to make a directory under /pkg. Take opendaylight as an example:

mkdir -p /pkg/odl

After downloading or building your feature packages, please copy all of your packages into the directory you made, e.g. /pkg/odl.

Note: If you have specail requirements for the container OS or kernel vesion, etc. Please contact us.

After all of these, come back to your_path_to_compass4nfv/compass4nfv/ directory, and run the command below:

./repo/make_repo.sh feature # To get special packages

./repo/make_repo.sh openstack # To get generic packages

When execution finished, you will get a tar package named packages.tar.gz under “your_path_to_compass4nfv/compass4nfv/work/repo/”. Your feature related packages have been archived in this tar package. And you will also get the PPA packages which includes the generic packages you defined in the role directory. The PPA packages are xenial-newton-ppa.tar.gz and centos7-newton-ppa.tar.gz, also in “your_path_to_compass4nfv/compass4nfv/work/repo/”.

1.3. Build compass ISO including the feature

Before you deploy a cluster with your feature installed, you need an ISO with feature packages, generic packages and role included. This section introduces how to build the ISO you want. What you need to do are two simple things:

Configure the build configuration file

The build configuration file is located in “your_path_to_compass4nfv/compass4nfv/build/”. There are lines in the file like this:

export APP_PACKAGE=${APP_PACKAGE:-$FEATURE_URL/packages.tar.gz}

export XENIAL_NEWTON_PPA=${XENIAL_NEWTON_PPA:-$PPA_URL/xenial-newton-ppa.tar.gz}

export CENTOS7_NEWTON_PPA=${CENTOS7_NEWTON_PPA:-$PPA_URL/centos7-newton-ppa.tar.gz}

Just replace the $FEATURE_URL and $PPA_URL to the directory where your packages.tar.gz located in. For example:

export APP_PACKAGE=${APP_PACKAGE:-file:///home/opnfv/compass4nfv/work/repo/packages.tar.gz}

export XENIAL_NEWTON_PPA=${XENIAL_NEWTON_PPA:-file:///home/opnfv/compass4nfv/work/repo/xenial-newton-ppa.tar.gz}

export CENTOS7_NEWTON_PPA=${CENTOS7_NEWTON_PPA:-file:///home/opnfv/compass4nfv/work/repo/centos7-newton-ppa.tar.gz}

Build the ISO

After the configuration, just run the command below to build the ISO you want for deployment.

./build.sh
1.4. References

Ansible documentation: http://docs.ansible.com/ansible/index.html>

Copper

OPNFV Copper Danube Overview
OPNFV Danube Copper Overview
Introduction

The OPNFV Copper project aims to help ensure that virtualized infrastructure and application deployments comply with goals of the NFV service provider or the VNF designer/user.

This is the third (“Danube”) release of the Copper project. The documentation provided here focuses on the overall goals of the Copper project and the specific features supported in the Colorado release.

Overall Goals for Configuration Policy

As focused on by Copper, configuration policy helps ensure that the NFV service environment meets the requirements of the variety of stakeholders which will provide or use NFV platforms.

These requirements can be expressed as an intent of the stakeholder, in specific terms or more abstractly, but at the highest level they express:

  • what I want
  • what I don’t want

Using road-based transportation as an analogy, some examples of this are shown below:

Configuration Intent Example
Who I Am What I Want What I Don’t Want
user a van, wheelchair-accessible, electric powered someone driving off with my van
road provider keep drivers moving at an optimum safe speed four-way stops
public safety shoulder warning strips, center media barriers speeding, tractors on the freeway

According to their role, service providers may apply more specific configuration requirements than users, since service providers are more likely to be managing specific types of infrastructure capabilities.

Developers and users may also express their requirements more specifically, based upon the type of application or how the user intends to use it.

For users, a high-level intent can be also translated into a more or less specific configuration capability by the service provider, taking into consideration aspects such as the type of application or its constraints.

Examples of such translation are:

Intent Translation into Configuration Capability
Intent Configuration Capability
network security firewall, DPI, private subnets
compute/storage security vulnerability monitoring, resource access controls
high availability clustering, auto-scaling, anti-affinity, live migration
disaster recovery geo-diverse anti-affinity
high compute/storage performance clustering, affinity
high network performance data plane acceleration
resource reclamation low-usage monitoring

Although such intent-to-capability translation is conceptually useful, it is unclear how it can address the variety of aspects that may affect the choice of an applicable configuration capability.

For that reason, the Copper project will initially focus on more specific configuration requirements as fulfilled by specific configuration capabilities, as well as how those requirements and capabilities are expressed in VNF and service design and packaging or as generic policies for the NFV Infrastructure.

OPNFV Copper Danube Requirements
Requirements

This section outlines general requirements for configuration policies, per the two main aspects in the Copper project scope:

  • Ensuring resource requirements of VNFs and services are applied per VNF designer, service, and tenant intent
  • Ensuring that generic policies are not violated, e.g. networks connected to VMs must either be public or owned by the VM owner
Resource Requirements

Resource requirements describe the characteristics of virtual resources (compute, storage, network) that are needed for VNFs and services, and how those resources should be managed over the lifecycle of a VNF/service. Upstream projects already include multiple ways in which resource requirements can be expressed and fulfilled, e.g.:

  • OpenStack Nova
    • the Image feature, enabling “VM templates” to be defined for NFs and referenced by name as a specific NF version to be used
    • the Flavor feature, addressing basic compute and storage requirements, with extensibility for custom attributes
  • OpenStack Heat
    • the Heat Orchestration Template feature, enabling a variety of VM aspects to be defined and managed by Heat throughout the VM lifecycle, notably
      • alarm handling (requires Ceilometer)
      • attached volumes (requires Cinder)
      • domain name assignment (requires Designate)
      • images (requires Glance)
      • autoscaling
      • software configuration associated with VM lifecycle hooks (CREATE, UPDATE, SUSPEND, RESUME, DELETE)
      • wait conditions and signaling for sequencing orchestration steps
      • orchestration service user management (requires Keystone)
      • shared storage (requires Manila)
      • load balancing (requires Neutron LBaaS)
      • firewalls (requires Neutron FWaaS)
      • various Neutron-based network and security configuration items
      • Nova flavors
      • Nova server attributes including access control
      • Nova server group affinity and anti-affinity
      • “Data-intensive application clustering” (requires Sahara)
      • DBaaS (requires Trove)
      • “multi-tenant cloud messaging and notification service” (requires Zaqar)
  • OpenStack Group-Based Policy
    • API-based grouping of endpoints with associated contractual expectations for data flow processing and service chaining
  • OpenStack Tacker
    • “a fully functional ETSI MANO based general purpose NFV Orchestrator and VNF Manager for OpenStack”
  • OpenDaylight Group-Based Policy
    • model-based grouping of endpoints with associated contractual expectations for data flow processing
  • OpenDaylight Service Function Chaining (SFC)
    • model-based management of “service chains” and the infrastucture that enables them
  • Additional projects that are commonly used for configuration management, implemented as client-server frameworks using model-based, declarative, or scripted configuration management data.
Generic Policy Requirements

Generic policy requirements address conditions related to resource state and events which need to be monitored for, and optionally responded to or prevented. These conditions are typically expected to be VNF/service-independent, as VNF/service-dependent condition handling (e.g. scale in/out) are considered to be addressed by VNFM/NFVO/VIM functions as described under Resource Requirements or as FCAPS related functions. However the general capabilities below can be applied to VNF/service-specific policy handling as well, or in particular to invocation of VNF/service-specific management/orchestration actions. The high-level required capabilities include:

  • Polled monitoring: Exposure of state via request-response APIs.
  • Notifications: Exposure of state via pub-sub APIs.
  • Realtime/near-realtime notifications: Notifications that occur in actual or near realtime.
  • Delegated policy: CRUD operations on policies that are distributed to specific components for local handling, including one/more of monitoring, violation reporting, and enforcement.
  • Violation reporting: Reporting of conditions that represent a policy violation.
  • Reactive enforcement: Enforcement actions taken in response to policy violation events.
  • Proactive enforcement: Enforcement actions taken in advance of policy violation events, e.g. blocking actions that could result in a policy violation.
  • Compliance auditing: Periodic auditing of state against policies.

Upstream projects already include multiple ways in which configuration conditions can be monitored and responded to:

  • OpenStack Congress provides a table-based mechanism for state monitoring and proactive/reactive policy enforcement, including data obtained from internal databases of OpenStack core and optional services. The Congress design approach is also extensible to other VIMs (e.g. SDNCs) through development of data source drivers for the new monitored state information.
  • OpenStack Aodh provides means to trigger alarms upon a wide variety of conditions derived from its monitored OpenStack analytics.
  • Nagios “offers complete monitoring and alerting for servers, switches, applications, and services”.
Requirements Validation Approach

The Copper project will assess the completeness of the upstream project solutions for requirements in scope though a process of:

  • developing configuration policy use cases to focus solution assessment tests
  • integrating the projects into the OPNFV platform for testing
  • executing functional and performance tests for the solutions
  • assessing overall requirements coverage and gaps in the most complete upstream solutions

Depending upon the priority of discovered gaps, new requirements will be submitted to upstream projects for the next available release cycle.

OPNFV Copper Installation
OPNFV Copper Installation Guide

This document describes how to install Copper, its dependencies and required system resources.

Version History
Date Ver. Author Comment
2017 Feb 7 1.0 Bryan Sullivan  
Introduction

The Congress service is automatically configured as required by the JOID and Apex installers, including creation of datasources per the installed datasource drivers. This release includes default support for the following datasource drivers:

  • nova
  • neutronv2
  • ceilometer
  • cinder
  • glancev2
  • keystone

For JOID, Congress is installed through a JuJu Charm, and for Apex through a Puppet Module. Both the Charm and Module are being upstreamed to OpenStack for future maintenance.

Other project installer support (e.g. Doctor) may install additional datasource drivers once Congress is installed.

Manual Installation

NOTE: This section describes a manual install procedure that had been tested under the JOID and Apex base installs prior to the integration of native installer support through JuJu (JOID) and Puppet (Apex). This procedure is being maintained as a basis for additional installer support in future releases. However, since Congress is pre-installed for JOID and Apex, this procedure is not necessary and not recommended for use if Congress is already installed.

Copper provides a set of bash scripts to automatically install Congress based upon a JOID or Apex install which does not already have Congress installed. These scripts are in the Copper repo at:

  • components/congress/install/bash/install_congress_1.sh
  • components/congress/install/bash/install_congress_2.sh

Prerequisites to using these scripts:

  • OPFNV installed via JOID or Apex
  • For Apex installs, on the jumphost, ssh to the undercloud VM and “su stack”.
  • For JOID installs, admin-openrc.sh saved from Horizon to ~/admin-openrc.sh
  • Retrieve the copper install script as below, optionally specifying the branch to use as a URL parameter, e.g. ?h=stable%2Fbrahmaputra

To invoke the procedure, enter the following shell commands, optionally specifying the branch identifier to use for OpenStack.

cd ~
wget https://git.opnfv.org/cgit/copper/plain/components/congress/install/bash/install_congress_1.sh
wget https://git.opnfv.org/cgit/copper/plain/components/congress/install/bash/install_congress_2.sh
bash install_congress_1.sh [openstack-branch]
OPNFV Copper Configuration Guide
Copper Configuration

This release includes installer support for the OpenStack Congress service under JOID and Apex installers. Congress is installed by default for all JOID and Apex scenarios. Support for other OPNFV installer deployed environments is planned for the next release.

Hardware Configuration

There is no specific hardware configuration required for the Copper project.

Copper Post Installation Procedures

This section describes optional procedures for verifying that the Congress service is operational.

Copper Functional Tests

This release includes the following test cases which are integrated into OPNFV Functest for the JOID and Apex installers:

  • DMZ Placement: dmz.sh
  • SMTP Ingress: smtp_ingress.sh
  • Reserved Subnet: reserved_subnet.sh

These scripts, related scripts that clean up the OpenStack environment afterward, and a combined test runner (run.sh) are in the Copper repo under the “tests” folder. Instructions for using the tests are provided as script comments.

Further description of the tests is provided on the Copper wiki at https://wiki.opnfv.org/display/copper/testing.

Congress Test Webapp

This release also provides a webapp that can be automatically installed in a Docker container on the OPNFV jumphost. This script is in the Copper repo at:

  • components/congress/test-webapp/setup/install_congress_testserver.sh

Prerequisites for using this script:

  • OPFNV installed per JOID or Apex installer
  • For Apex installs, on the jumphost, ssh to the undercloud VM and “su stack”

To invoke the procedure, enter the following shell commands, optionally specifying the branch identifier to use for Copper:

wget https://git.opnfv.org/cgit/copper/plain/components/congress/test-webapp/setup/install_congress_testserver.sh
bash install_congress_testserver.sh [copper-branch]
Using the Test Webapp

Browse to the webapp IP address provided at the end of the install procedure.

Interactive options are meant to be self-explanatory given a basic familiarity with the Congress service and data model.

Removing the Test Webapp

The webapp can be removed by running this script from the Copper repo:

  • components/congress/test-webapp/setup/clean_congress_testserver.sh
OPNFV Copper User Guide
OPNFV Copper User Guide

This release focuses on use of the OpenStack Congress service for managing configuration policy. See the Congress intro guide for general information on the capabilities and usage of Congress.

Examples of Congress API usage can be found in the Copper tests as described on the OPNFV wiki at https://wiki.opnfv.org/display/copper/testing.

OPNFV Copper Danube Design
Definitions
Definitions
Term Meaning
State Information that can be used to convey or imply the state of something, e.g. an application, resource, entity, etc. This can include data held inside OPNFV components, “events” that have occurred (e.g. “policy violation”), etc.
Event An item of significance to the policy engine, for which the engine has become aware through some method of discovery e.g. polling or notification.
Abbreviations
Abbreviations
Term Meaning
CRUD Create, Read, Update, Delete (database operation types)
FCAPS Fault, Configuration, Accounting, Performance, Security
NF Network Function
SFC Service Function Chaining
VNF Virtual Network Function
NFVI Network Function Virtualization Infrastructure
Architecture
Architectural Concept

The following example diagram illustrates a “relationship diagram” type view of an NFVI platform, in which the roles of components focused on policy management, services, and infrastructure are shown.

This view illustrates that a large-scale deployment of NFVI may leverage multiple components of the same “type” (e.g. SDN Controller), which fulfill specific purposes for which they are optimized. For example, a global SDN controller and cloud orchestrator can act as directed by a service orchestrator in the provisioning of VNFs per intent, while various components at a local and global level handle policy-related events directly and/or feed them back through a closed-loop policy design that responds as needed, directly or through the service orchestrator.

policy_architecture.png

(source of the diagram above: https://git.opnfv.org/cgit/copper/plain/design_docs/images/policy_architecture.pptx)

Architectural Aspects
  • Policies are reflected in two high-level goals
    • Ensure resource requirements of VNFs and services are applied per VNF designer, service, and tenant intent
    • Ensure that generic policies are not violated, e.g. networks connected to VMs must either be public or owned by the VM owner
  • Policies are distributed through two main means
    • As part of VNF packages, customized if needed by Service Design tools, expressing intent of the VNF designer and service provider, and possibly customized or supplemented by service orchestrators per the intent of specific tenants
    • As generic policies provisioned into VIMs (SDN controllers and cloud orchestrators), expressing intent of the service provider re what states/events need to be policy-governed independently of specific VNFs
  • Policies are applied locally and in closed-loop systems per the capabilities of the local policy enforcer and the impact of the related state/event conditions
    • VIMs should be able to execute most policies locally
    • VIMs may need to pass policy-related state/events to a closed-loop system, where those events are relevant to other components in the architecture (e.g. service orchestrator), or some additional data/arbitration is needed to resolve the state/event condition
  • Policies are localized as they are distributed/delegated
    • High-level policies (e.g. expressing “intent”) can be translated into VNF package elements or generic policies, perhaps using distinct syntaxes
    • Delegated policy syntaxes are likely VIM-specific, e.g. Datalog (Congress)
  • Closed-loop policy and VNF-lifecycle event handling are //somewhat// distinct
    • Closed-loop policy is mostly about resolving conditions that can’t be handled locally, but as above in some cases the conditions may be of relevance and either delivered directly or forwarded to service orchestrators
    • VNF-lifecycle events that can’t be handled by the VIM locally are delivered directly to the service orchestrator
  • Some events/analytics need to be collected into a more “open-loop” system which can enable other actions, e.g.
    • audits and manual interventions
    • machine-learning focused optimizations of policies (largely a future objective)

Issues to be investigated as part of establishing an overall cohesive/adaptive policy architecture:

  • For the various components which may fulfill a specific purpose, what capabilities (e.g. APIs) do they have/need to
    • handle events locally
    • enable closed-loop policy handling components to subscribe/optimize policy-related events that are of interest
  • For global controllers and cloud orchestrators
    • How do they support correlation of events impacting resources in different scopes (network and cloud)
    • What event/response flows apply to various policy use cases
  • What specific policy use cases can/should fall into each overall class
    • locally handled by NFVI components
    • handled by a closed-loop policy system, either VNF/service-specific or VNF-independent

Daisy4NFV

Design Docs for Daisy4nfv
1. CI Job Introduction
1.2. Project Gating And Daily Deployment Test

To save time, currently, Daisy4NFV does not run deployment test in gate job which simply builds and uploads artifacts to low confidence level repo. The project deployment test is triggered on a daily basis. If the artifact passes the test, then it will be promoted to the high confidence level repo.

The low confidence level artifacts are bin files in http://artifacts.opnfv.org/daisy.html named like “daisy/opnfv-Gerrit-39495.bin”, while the high confidence level artifacts are named like “daisy/opnfv-2017-08-20_08-00-04.bin”.

The daily project deployment status can be found at

https://build.opnfv.org/ci/job/daisy-daily-master/

2. Deployment Steps

This document takes VM all-in-one environment as example to show what ci/deploy/deploy.sh really do.

  1. On jump host, clean up all-in-one vm and networks.
  2. On jump host, clean up daisy vm and networks.
  3. On jump host, create and start daisy vm and networks.
  4. In daisy vm, Install daisy artifact.
  5. In daisy vm, config daisy and OpenStack default options.

6. In daisy vm, create cluster, update network and build PXE server for the bootstrap kernel. In short, be ready for discovering target nodes. These tasks are done by running the following command.

python /home/daisy/deploy/tempest.py –dha /home/daisy/labs/zte/virtual1/daisy/config/deploy.yml –network /home/daisy/labs/zte/virtual1/daisy/config/network.yml –cluster ‘yes’

  1. On jump host, create and start all-in-one vm and networks.
  2. On jump host, after all-in-one vm is up, get its mac address and write into /home/daisy/labs/zte/virtual1/daisy/config/deploy.yml.

9. In daisy vm, check if all-in-one vm was discovered, if it was, then update its network assignment and config OpenStack according to OPNFV scenario and setup PXE for OS installaion. These tasks are done by running the following command.

python /home/daisy/deploy/tempest.py –dha /home/daisy/labs/zte/virtual1/daisy/config/deploy.yml –network /home/daisy/labs/zte/virtual1/daisy/config/network.yml –host yes –isbare 0 –scenario os-nosdn-nofeature-noha

Note: Current host status: os_status is “init”.

  1. On jump host, restart all_in_one vm to install OS.

11. In daisy vm, continue to intall OS by running the following command which for VM environment only.

python /home/daisy/deploy/tempest.py –dha /home/daisy/labs/zte/virtual1/daisy/config/deploy.yml –network /home/daisy/labs/zte/virtual1/daisy/config/network.yml –install ‘yes’

12. In daisy vm, run the following command to check OS intallation progress. /home/daisy/deploy/check_os_progress.sh -d 0 -n 1

Note: Current host status: os_status is “installing” during installation, then os_status becomes “active” after OS was succesfully installed.

  1. On jump host, reboot all-in-one vm again to get a fresh and first booted OS.

14. In daisy vm, run the following command to check OpenStack/ODL/... intallation progress.

/home/daisy/deploy/check_openstack_progress.sh -n 1

3. Kolla Image Multicast Design
3.1. Protocol Design
  1. All Protocol headers are 1 byte long or align to 4 bytes.

2. Packet size should not exceed above 1500(MTU) bytes including UDP/IP header and should be align to 4 bytes. In future, MTU can be modified larger than 1500(Jumbo Frame) through cmd line option to enlarge the data throughput.

/* Packet header definition (align to 4 bytes) */ struct packet_ctl {

uint32_t seq; // packet seq number start from 0, unique in server life cycle. uint32_t crc; // checksum uint32_t data_size; // payload length uint8_t data[0];

};

/* Buffer info definition (align to 4 bytes) */ struct buffer_ctl {

uint32_t buffer_id; // buffer seq number start from 0, unique in server life cycle. uint32_t buffer_size; // payload total length of a buffer uint32_t packet_id_base; // seq number of the first packet in this buffer. uint32_t pkt_count; // number of packet in this buffer, 0 means EOF.

};

  1. 1-byte-long header definition

Signals such as the four below are 1 byte long, to simplify the receive process(since it cannot be spitted ).

#define CLIENT_READY 0x1 #define CLIENT_REQ 0x2 #define CLIENT_DONE 0x4 #define SERVER_SENT 0x8

Note: Please see the collaboration diagram for their meanings.

  1. Retransmission Request Header

/* Retransmition Request Header (align to 4 bytes) */ struct request_ctl {

uint32_t req_count; // How many seqs below. uint32_t seqs[0]; // packet seqs.

};

  1. Buffer operations

void buffer_init(); // Init the buffer_ctl structure and all(say 1024) packet_ctl structures. Allocate buffer memory. long buffer_fill(int fd); // fill a buffer from fd, such as stdin long buffer_flush(int fd); // flush a buffer to fd, say stdout struct packet_ctl *packet_put(struct packet_ctl *new_pkt);// put a packet to a buffer and return a free memory slot for the next packet. struct packet_ctl *packet_get(uint32_t seq);// get a packet data in buffer by indicating the packet seq.

3.2. How to sync between server threads

If children’s aaa() operation need to wait the parents’s init() to be done, then do it literally like this:

UDP Server TCP Server1 = spawn( )—-> TCP Server1

init()
TCP Server2 = spawn( )—–> TCP Server2
V(sem)———————-> P(sem) // No child any more

V(sem)———————> P(sem) aaa() // No need to V(sem), for no child

aaa()

If parent’s send() operation need to wait the children’s ready() done, then do it literally too, but is a reverse way:

UDP Server TCP Server1 TCP Server2
// No child any more

ready() ready() P(sem) <——————— V(sem)

P(sem) <—————— V(sem) send()

Note that the aaa() and ready() operations above run in parallel. If this is not the case due to race condition, the sequence above can be modified into this below:

UDP Server TCP Server1 TCP Server2
// No child any more
ready()

P(sem) <——————— V(sem) ready()

P(sem) <——————- V(sem) send()

In order to implement such chained/zipper sync pattern, a pair of semaphores is needed between the parent and the child. One is used by child to wait parent , the other is used by parent to wait child. semaphore pair can be allocated by parent and pass the pointer to the child over spawn() operation such as pthread_create().

/* semaphore pair definition */ struct semaphores {

sem_t wait_parent; sem_t wait_child;

};

Then the semaphore pair can be recorded by threads by using the semlink struct below: struct semlink {

struct semaphores this; / used by parent to point to the struct semaphores
which it created during spawn child. */
struct semaphores parent; / used by child to point to the struct
semaphores which it created by parent */

};

chained/zipper sync API:

void sl_wait_child(struct semlink *sl); void sl_release_child(struct semlink *sl); void sl_wait_parent(struct semlink *sl); void sl_release_parent(struct semlink *sl);

API usage is like this.

Thread1(root parent) Thread2(child) Thread3(grandchild) sl_wait_parent(noop op) sl_release_child

+———->sl_wait_parent
sl_release_child
+———–> sl_wait_parent
sl_release_child(noop op) ... sl_wait_child(noop op)
  • sl_release_parent

sl_wait_child <————-

  • sl_release_parent

sl_wait_child <———— sl_release_parent(noop op)

API implementation:

void sl_wait_child(struct semlink *sl) {

if (sl->this) {
P(sl->this->wait_child);

}

}

void sl_release_child(struct semlink *sl) {

if (sl->this) {
V(sl->this->wait_parent);

}

}

void sl_wait_parent(struct semlink *sl) {

if (sl->parent) {
P(sl->parent->wait_parent);

}

}

void sl_release_parent(struct semlink *sl) {

if (sl->parent) {
V(sl->parent->wait_child);

}

}

3.3. Client flow chart

See Collaboration Diagram

3.4. UDP thread flow chart

See Collaboration Diagram

3.5. TCP thread flow chart
S_INIT — (UDP initialized) —> S_ACCEPT — (accept clients) –+

/—————————————————————-/ V

S_PREP — (UDP prepared abuffer)

^ | | –> S_SYNC — (clients ClIENT_READY) | | | –> S_SEND — (clients CLIENT_DONE) | | | V —————(bufferctl.pkt_count != 0)———————–+


V

exit() <— (bufferctl.pkt_count == 0)

3.6. TCP using poll and message queue

TCP uses poll() to sync with client’s events as well as output event from itself, so that we can use non-block socket operations to reduce the latency. POLLIN means there are message from client and POLLOUT means we are ready to send message/retransmission packets to client.

poll main loop pseudo code: void check_clients(struct server_status_data *sdata) {

poll_events = poll(&(sdata->ds[1]), sdata->ccount - 1, timeout);

/* check all connected clients */ for (sdata->cindex = 1; sdata->cindex < sdata->ccount; sdata->cindex++) {

ds = &(sdata->ds[sdata->cindex]); if (!ds->revents) {

continue;

}

if (ds->revents & (POLLERR|POLLHUP|POLLNVAL)) {
handle_error_event(sdata);
} else if (ds->revents & (POLLIN|POLLPRI)) {
handle_pullin_event(sdata); // may set POLLOUT into ds->events
// to trigger handle_pullout_event().
} else if (ds->revents & POLLOUT) {
handle_pullout_event(sdata);

}

}

}

For TCP, since the message from client may not complete and send data may be also interrupted due to non-block fashion, there should be one send message queue and a receive message queue on the server side for each client (client do not use non-block operations).

TCP message queue definition:

struct tcpq {
struct qmsg head, *tail; long count; / message count in a queue / long size; / Total data size of a queue */

};

TCP message queue item definition:

struct qmsg {
struct qmsg *next; void *data; long size;

};

TCP message queue API:

// Allocate and init a queue. struct tcpq * tcpq_queue_init(void);

// Free a queue. void tcpq_queue_free(struct tcpq *q);

// Return queue length. long tcpq_queue_dsize(struct tcpq *q);

// queue new message to tail. void tcpq_queue_tail(struct tcpq *q, void *data, long size);

// queue message that cannot be sent currently back to queue head. void tcpq_queue_head(struct tcpq *q, void *data, long size);

// get one piece from queue head. void * tcpq_dequeue_head(struct tcpq *q, long *size);

// Serialize all pieces of a queue, and move it out of queue, to ease the further //operation on it. void * tcpq_dqueue_flat(struct tcpq *q, long *size);

// Serialize all pieces of a queue, do not move it out of queue, to ease the further //operation on it. void * tcpq_queue_flat_peek(struct tcpq *q, long *size);

Release notes for Daisy4nfv
1. Abstract

This document compiles the release notes for the E 1.0 release of OPNFV when using Daisy as a deployment tool.

1.1. Configuration Guide

Before installing Daisy4NFV on jump server,you have to configure the daisy.conf file.Then put the right configured daisy.conf file in the /home/daisy_install/ dir.

  1. you have to supplement the “daisy_management_ip” field with the ip of management ip of your Daisy server vm.
  2. Now the backend field “default_backend_types” just support the “kolla”.
  3. “os_install_type” field just support “pxe” for now.
  4. Daisy now use pxe server to install the os, the “build_pxe” item must set to “no”.
  5. “eth_name” field is the pxe server interface, and this field is required when the “build_pxe” field set to “yes”.This should be set to the interface (in Daisy Server VM) which will be used for communicating with other target nodes on management/PXE net plane. Default is ens3.
  6. “ip_address” field is the ip address of pxe server interface.
  7. “net_mask” field is the netmask of pxe server,which is required when the “build_pxe” is set to “yes”
  8. “client_ip_begin” and “client_ip_end” field are the dhcp range of the pxe server.
  9. If you want to use the multicast type to deliver the kolla image to target node, set the “daisy_conf_mcast_enabled” field to “True”
2. OpenStack Configuration Guide
2.1. Before The First Deployment

When executing deploy.sh, before doing real deployment, Daisy utilizes Kolla’s service configuration functionality [1] to specify the following changes to the default OpenStack configuration which comes from Kolla as default.

a) If is it is a VM deployment, set virt_type=qemu amd cpu_mode=none for nova-compute.conf.

b) In nova-api.conf set default_floating_pool to the name of the external network which will be created by Daisy after deployment for nova-api.conf.

c) In heat-api.conf and heat-engine.conf, set deferred_auth_method to trusts and unset trusts_delegated_roles.

Those above changes are requirements of OPNFV or environment’s constraints. So it is not recommended to change them. But if the user wants to add more specific configurations to OpenStack services before doing real deployment, we suggest to do it in the same way as deploy.sh do. Currently, this means hacking into deploy/prepare.sh or deploy/prepare/execute.py then add config file as described in [1].

Notes: Suggest to pass the first deployment first, then reconfigure and deploy again.

2.2. After The First Deployment

After the first time of deployment of OpenStack, its configurations can also be changed and applied by using Kolla’s service configuration functionality [1]. But user has to issue Kolla’s command to do it in this release:

[1] https://docs.openstack.org/kolla-ansible/latest/advanced-configuration.html#openstack-service-configuration-in-kolla

OPNFV Daisy4nfv Installation Guide
Abstract

This document describes how to install the Euphrates release of OPNFV when using Daisy4nfv as a deployment tool covering it’s limitations, dependencies and required resources.

Version history
Date Ver. Author Comment
2017-02-07 0.0.1 Zhijiang Hu (ZTE) Initial version
Daisy4nfv configuration

This document provides guidelines on how to install and configure the Euphrates release of OPNFV when using Daisy as a deployment tool including required software and hardware configurations.

Installation and configuration of host OS, OpenStack etc. can be supported by Daisy on Virtual nodes and Bare Metal nodes.

The audience of this document is assumed to have good knowledge in networking and Unix/Linux administration.

Prerequisites

Before starting the installation of the Euphrates release of OPNFV, some plannings must be done.

Retrieve the installation iso image

First of all, the installation iso which includes packages of Daisy, OS, OpenStack, and so on is needed for deploying your OPNFV environment.

The stable release iso image can be retrieved via OPNFV software download page

The daily build iso image can be retrieved via OPNFV artifact repository:

http://artifacts.opnfv.org/daisy.html

NOTE: Search the keyword “daisy/Euphrates” to locate the iso image.

E.g. daisy/opnfv-2017-10-06_09-50-23.iso

Download the iso file, then mount it to a specified directory and get the opnfv-*.bin from that directory.

The git url and sha512 checksum of iso image are recorded in properties files. According to these, the corresponding deployment scripts can be retrieved.

Retrieve the deployment scripts

To retrieve the repository of Daisy on Jumphost use the following command:

To get stable Euphrates release, you can use the following command:

  • git checkout euphrates.1.0
Setup Requirements

If you have only 1 Bare Metal server, Virtual deployment is recommended. if you have more than 3 servers, the Bare Metal deployment is recommended. The minimum number of servers for each role in Bare metal deployment is listed below.

Role Number of Servers
Jump Host 1
Controller 1
Compute 1
Jumphost Requirements

The Jumphost requirements are outlined below:

  1. CentOS 7.2 (Pre-installed).
  2. Root access.
  3. Libvirt virtualization support(For virtual deployment).
  4. Minimum 1 NIC(or 2 NICs for virtual deployment).
    • PXE installation Network (Receiving PXE request from nodes and providing OS provisioning)
    • IPMI Network (Nodes power control and set boot PXE first via IPMI interface)
    • Internet access (For getting latest OS updates)
    • External Interface(For virtual deployment, exclusively used by instance traffic to access the rest of the Internet)
  5. 16 GB of RAM for a Bare Metal deployment, 64 GB of RAM for a Virtual deployment.
  6. CPU cores: 32, Memory: 64 GB, Hard Disk: 500 GB, (Virtual deployment needs 1 TB Hard Disk)
Bare Metal Node Requirements

Bare Metal nodes require:

  1. IPMI enabled on OOB interface for power control.
  2. BIOS boot priority should be PXE first then local hard disk.
  3. Minimum 1 NIC for Compute nodes, 2 NICs for Controller nodes.
    • PXE installation Network (Broadcasting PXE request)
    • IPMI Network (Receiving IPMI command from Jumphost)
    • Internet access (For getting latest OS updates)
    • External Interface(For virtual deployment, exclusively used by instance traffic to access the rest of the Internet)
Network Requirements

Network requirements include:

  1. No DHCP or TFTP server running on networks used by OPNFV.
  2. 2-7 separate networks with connectivity between Jumphost and nodes.
    • PXE installation Network
    • IPMI Network
    • Internet access Network
    • OpenStack Public API Network
    • OpenStack Private API Network
    • OpenStack External Network
    • OpenStack Tenant Network(currently, VxLAN only)
  3. Lights out OOB network access from Jumphost with IPMI node enabled (Bare Metal deployment only).
  4. Internet access Network has Internet access, meaning a gateway and DNS availability.
  5. OpenStack External Network has Internet access too if you want instances to access the Internet.

Note: All networks except OpenStack External Network can share one NIC(Default configuration) or use an exclusive NIC(Reconfigurated in network.yml).

Execution Requirements (Bare Metal Only)

In order to execute a deployment, one must gather the following information:

  1. IPMI IP addresses of the nodes.
  2. IPMI login information for the nodes (user/password).
Installation Guide (Bare Metal Deployment)
Nodes Configuration (Bare Metal Deployment)

The below file is the inventory template of deployment nodes:

”./deploy/config/bm_environment/zte-baremetal1/deploy.yml”

You can write your own name/roles reference into it.

  • name – Host name for deployment node after installation.
  • roles – Components deployed. CONTROLLER_LB is for Controller,

COMPUTER is for Compute role. Currently only these two role is supported. The first CONTROLLER_LB is also used for ODL controller. 3 hosts in inventory will be chosen to setup the Ceph storage cluster.

Set TYPE and FLAVOR

E.g.

TYPE: virtual
FLAVOR: cluster

Assignment of different roles to servers

E.g. OpenStack only deployment roles setting

hosts:
  - name: host1
    roles:
      - CONTROLLER_LB
  - name: host2
    roles:
      - COMPUTER
  - name: host3
    roles:
      - COMPUTER

NOTE: For B/M, Daisy uses MAC address defined in deploy.yml to map discovered nodes to node items definition in deploy.yml, then assign role described by node item to the discovered nodes by name pattern. Currently, controller01, controller02, and controller03 will be assigned with Controler role while computer01, ‘computer02, computer03, and computer04 will be assigned with Compute role.

NOTE: For V/M, There is no MAC address defined in deploy.yml for each virtual machine. Instead, Daisy will fill that blank by getting MAC from “virsh dump-xml”.

Network Configuration (Bare Metal Deployment)

Before deployment, there are some network configurations to be checked based on your network topology. The default network configuration file for Daisy is ”./deploy/config/bm_environment/zte-baremetal1/network.yml”. You can write your own reference into it.

The following figure shows the default network configuration.

+-B/M--------+------------------------------+
|Jumperserver+                              |
+------------+                       +--+   |
|                                    |  |   |
|                +-V/M--------+      |  |   |
|                | Daisyserver+------+  |   |
|                +------------+      |  |   |
|                                    |  |   |
+------------------------------------|  |---+
                                     |  |
                                     |  |
      +--+                           |  |
      |  |       +-B/M--------+      |  |
      |  +-------+ Controller +------+  |
      |  |       | ODL(Opt.)  |      |  |
      |  |       | Network    |      |  |
      |  |       | CephOSD1   |      |  |
      |  |       +------------+      |  |
      |  |                           |  |
      |  |                           |  |
      |  |                           |  |
      |  |       +-B/M--------+      |  |
      |  +-------+  Compute1  +------+  |
      |  |       |  CephOSD2  |      |  |
      |  |       +------------+      |  |
      |  |                           |  |
      |  |                           |  |
      |  |                           |  |
      |  |       +-B/M--------+      |  |
      |  +-------+  Compute2  +------+  |
      |  |       |  CephOSD3  |      |  |
      |  |       +------------+      |  |
      |  |                           |  |
      |  |                           |  |
      |  |                           |  |
      +--+                           +--+
        ^                             ^
        |                             |
        |                             |
       /---------------------------\  |
       |      External Network     |  |
       \---------------------------/  |
              /-----------------------+---\
              |    Installation Network   |
              |    Public/Private API     |
              |      Internet Access      |
              |      Tenant Network       |
              |     Storage Network       |
              |     HeartBeat Network     |
              \---------------------------/

Note: For Flat External networks(which is used by default), a physical interface is needed on each compute node for ODL NetVirt recent versions. HeartBeat network is selected,and if it is configured in network.yml,the keepalived interface will be the heartbeat interface.

Start Deployment (Bare Metal Deployment)
  1. Git clone the latest daisy4nfv code from opnfv: “git clone https://gerrit.opnfv.org/gerrit/daisy

(2) Download latest bin file(such as opnfv-2017-06-06_23-00-04.bin) of daisy from http://artifacts.opnfv.org/daisy.html and change the bin file name(such as opnfv-2017-06-06_23-00-04.bin) to opnfv.bin. Check the https://build.opnfv.org/ci/job/daisy-os-odl-nofeature-ha-baremetal-daily-master/, and if the ‘snaps_health_check’ of functest result is ‘PASS’, you can use this verify-passed bin to deploy the openstack in your own environment

(3) Assumed cloned dir is $workdir, which laid out like below: [root@daisyserver daisy]# ls ci deploy docker INFO LICENSE requirements.txt templates tests tox.ini code deploy.log docs known_hosts setup.py test-requirements.txt tools Make sure the opnfv.bin file is in $workdir

(4) Enter into the $workdir, which laid out like below: [root@daisyserver daisy]# ls ci code deploy docker docs INFO LICENSE requirements.txt setup.py templates test-requirements.txt tests tools tox.ini Create folder of labs/zte/pod2/daisy/config in $workdir

(5) Move the ./deploy/config/bm_environment/zte-baremetal1/deploy.yml and ./deploy/config/bm_environment/zte-baremetal1/network.yml to labs/zte/pod2/daisy/config dir.

Note: If selinux is disabled on the host, please delete all xml files section of below lines in dir templates/physical_environment/vms/

<seclabel type=’dynamic’ model=’selinux’ relabel=’yes’>
<label>system_u:system_r:svirt_t:s0:c182,c195</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c182,c195</imagelabel>

</seclabel>

(6) Config the bridge in jumperserver,make sure the daisy vm can connect to the targetnode,use the command below: brctl addbr br7 brctl addif br7 enp3s0f3(the interface for jumperserver to connect to daisy vm) ifconfig br7 10.20.7.1 netmask 255.255.255.0 up service network restart

(7) Run the script deploy.sh in daisy/ci/deploy/ with command: sudo ./ci/deploy/deploy.sh -L $(cd ./;pwd) -l zte -p pod2 -s os-nosdn-nofeature-noha

Note: The value after -L should be a absolute path which points to the directory which contents labs/zte/pod2/daisy/config directory. The value after -p parameter(pod2) comes from path “labs/zte/pod2” The value after -l parameter(zte) comes from path “labs/zte” The value after -s “os-nosdn-nofeature-ha” used for deploy multinode openstack The value after -s “os-nosdn-nofeature-noha” used for deploy all-in-one openstack

(8) When deploy successfully,the floating ip of openstack is 10.20.7.11, the login account is “admin” and the password is “keystone”

Installation Guide (Virtual Deployment)
Nodes Configuration (Virtual Deployment)

The below file is the inventory template of deployment nodes:

”./deploy/conf/vm_environment/zte-virtual1/deploy.yml”

You can write your own name/roles reference into it.

  • name – Host name for deployment node after installation.
  • roles – Components deployed.

Set TYPE and FLAVOR

E.g.

TYPE: virtual
FLAVOR: cluster

Assignment of different roles to servers

E.g. OpenStack only deployment roles setting

hosts:
  - name: host1
    roles:
      - controller

  - name: host2
    roles:
      - compute

NOTE: For B/M, Daisy uses MAC address defined in deploy.yml to map discovered nodes to node items definition in deploy.yml, then assign role described by node item to the discovered nodes by name pattern. Currently, controller01, controller02, and controller03 will be assigned with Controller role while computer01, computer02, computer03, and computer04 will be assigned with Compute role.

NOTE: For V/M, There is no MAC address defined in deploy.yml for each virtual machine. Instead, Daisy will fill that blank by getting MAC from “virsh dump-xml”.

E.g. OpenStack and ceph deployment roles setting

hosts:
  - name: host1
    roles:
      - controller

  - name: host2
    roles:
      - compute
Network Configuration (Virtual Deployment)

Before deployment, there are some network configurations to be checked based on your network topology. The default network configuration file for Daisy is “daisy/deploy/config/vm_environment/zte-virtual1/network.yml”. You can write your own reference into it.

The following figure shows the default network configuration.

+-B/M--------+------------------------------+
|Jumperserver+                              |
+------------+                       +--+   |
|                                    |  |   |
|                +-V/M--------+      |  |   |
|                | Daisyserver+------+  |   |
|                +------------+      |  |   |
|                                    |  |   |
|     +--+                           |  |   |
|     |  |       +-V/M--------+      |  |   |
|     |  +-------+ Controller +------+  |   |
|     |  |       | ODL(Opt.)  |      |  |   |
|     |  |       | Network    |      |  |   |
|     |  |       | Ceph1      |      |  |   |
|     |  |       +------------+      |  |   |
|     |  |                           |  |   |
|     |  |                           |  |   |
|     |  |                           |  |   |
|     |  |       +-V/M--------+      |  |   |
|     |  +-------+  Compute1  +------+  |   |
|     |  |       |  Ceph2     |      |  |   |
|     |  |       +------------+      |  |   |
|     |  |                           |  |   |
|     |  |                           |  |   |
|     |  |                           |  |   |
|     |  |       +-V/M--------+      |  |   |
|     |  +-------+  Compute2  +------+  |   |
|     |  |       |  Ceph3     |      |  |   |
|     |  |       +------------+      |  |   |
|     |  |                           |  |   |
|     |  |                           |  |   |
|     |  |                           |  |   |
|     +--+                           +--+   |
|       ^                             ^     |
|       |                             |     |
|       |                             |     |
|      /---------------------------\  |     |
|      |      External Network     |  |     |
|      \---------------------------/  |     |
|             /-----------------------+---\ |
|             |    Installation Network   | |
|             |    Public/Private API     | |
|             |      Internet Access      | |
|             |      Tenant Network       | |
|             |     Storage Network       | |
|             |     HeartBeat Network     | |
|             \---------------------------/ |
+-------------------------------------------+

Note: For Flat External networks(which is used by default), a physical interface is needed on each compute node for ODL NetVirt recent versions. HeartBeat network is selected,and if it is configured in network.yml,the keepalived interface will be the heartbeat interface.

Start Deployment (Virtual Deployment)

(1) Git clone the latest daisy4nfv code from opnfv: “git clone https://gerrit.opnfv.org/gerrit/daisy”, make sure the current branch is master

(2) Download latest bin file(such as opnfv-2017-06-06_23-00-04.bin) of daisy from http://artifacts.opnfv.org/daisy.html and change the bin file name(such as opnfv-2017-06-06_23-00-04.bin) to opnfv.bin. Check the https://build.opnfv.org/ci/job/daisy-os-odl-nofeature-ha-baremetal-daily-master/, and if the ‘snaps_health_check’ of functest result is ‘PASS’, you can use this verify-passed bin to deploy the openstack in your own environment

(3) Assumed cloned dir is $workdir, which laid out like below: [root@daisyserver daisy]# ls ci code deploy docker docs INFO LICENSE requirements.txt setup.py templates test-requirements.txt tests tools tox.ini Make sure the opnfv.bin file is in $workdir

  1. Enter into $workdir, Create folder of labs/zte/virtual1/daisy/config in $workdir

(5) Move the deploy/config/vm_environment/zte-virtual1/deploy.yml and deploy/config/vm_environment/zte-virtual1/network.yml to labs/zte/virtual1/daisy/config dir.

Note: zte-virtual1 config file deploy openstack with five nodes(3 lb nodes and 2 computer nodes), if you want to deploy an all-in-one openstack, change the zte-virtual1 to zte-virtual2

Note: If selinux is disabled on the host, please delete all xml files section of below lines in dir templates/virtual_environment/vms/

<seclabel type=’dynamic’ model=’selinux’ relabel=’yes’>
<label>system_u:system_r:svirt_t:s0:c182,c195</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c182,c195</imagelabel>

</seclabel>

(6) Run the script deploy.sh in daisy/ci/deploy/ with command: sudo ./ci/deploy/deploy.sh -L $(cd ./;pwd) -l zte -p virtual1 -s os-nosdn-nofeature-ha

Note: The value after -L should be a absolute path which points to the directory which contents labs/zte/virtual1/daisy/config directory. The value after -p parameter(virtual1) is get from labs/zte/virtual1/daisy/config/ The value after -l parameter(zte) is get from labs/ The value after -s “os-nosdn-nofeature-ha” used for deploy multinode openstack The value after -s “os-nosdn-nofeature-noha” used for deploy all-in-one openstack

(7) When deploy successfully,the floating ip of openstack is 10.20.11.11, the login account is “admin” and the password is “keystone”

Deployment Error Recovery Guide

Deployment may fail due to different kinds of reasons, such as Daisy VM creation error, target nodes failure during OS installation, or Kolla deploy command error. Different errors can be grouped into several error levels. We define Recovery Levels below to fulfill recover requirements in different error levels.

1. Recovery Level 0

This level restart whole deployment again. Mainly to retry to solve errors such as Daisy VM creation failed. For example we use the following command to do virtual deployment(in the jump host):

sudo ./ci/deploy/deploy.sh -b ./ -l zte -p virtual1 -s os-nosdn-nofeature-ha

If command failed because of Daisy VM creation error, then redo above command will restart whole deployment which includes rebuild the daisy VM image and restart Daisy VM.

2. Recovery Level 1

If Daisy VM was created successfully, but bugs was encountered in Daisy code or software of target OS which prevent deployment from being done, in this case, the user or the developer does not want to recreate the Daisy VM again during next deployment process but just to modify some pieces of code in it. To achieve this, he/she can redo deployment by deleting all clusters and hosts first(in the Daisy VM):

source /root/daisyrc_admin
for i in `daisy cluster-list | awk -F "|" '{print $2}' | sed -n '4p' | tr -d " "`;do daisy cluster-delete $i;done
for i in `daisy host-list | awk -F "|" '{print $2}'| grep -o "[^ ]\+\( \+[^ ]\+\)*"|tail -n +2`;do daisy host-delete $i;done

Then, adjust deployment command as below and run it again(in the jump host):

sudo ./ci/deploy/deploy.sh -S -b ./ -l zte -p virtual1 -s os-nosdn-nofeature-ha

Pay attention to the “-S” argument above, it lets the deployment process to skip re-creating Daisy VM and use the existing one.

3. Recovery Level 2

If both Daisy VM and target node’s OS are OK, but error ocurred when doing OpenStack deployment, then there is even no need to re-install target OS for the deployment retrying. In this level, all we need to do is just retry the Daisy deployment command as follows(in the Daisy VM):

source /root/daisyrc_admin
daisy uninstall <cluster-id>
daisy install <cluster-id>

This basically do kolla-ansible destroy and kolla-asnible deploy.

OpenStack Minor Version Update Guide

Thanks for the Kolla’s kolla-ansible upgrade function, Daisy enable to update OpenStack minor version as the follows:

1. Get new version file only from Daisy team. Since Daisy’s Kolla images are build by meeting the OPNFV requirements and have their own file packaging layout, Daisy requires user to always use Kolla image file built by Daisy team. Currently, it can be got from http://artifacts.opnfv.org/daisy/upstream, or please see this chapter for how to build your own image.

2. Put new version file into /var/lib/daisy/versionfile/kolla/, for example: /var/lib/daisy/versionfile/kolla/kolla-image-ocata-170811155446.tgz

3. Add version file to Daisy’s version management database then get the version ID.

[root@daisy ~]# source /root/daisyrc_admin
[root@daisy ~]# daisy version-add kolla-image-ocata-170811155446.tgz kolla
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| checksum    | None                                 |
| created_at  | 2017-08-28T06:45:25.000000           |
| description | None                                 |
| id          | 8be92587-34d7-43e8-9862-a5288c651079 |
| name        | kolla-image-ocata-170811155446.tgz   |
| owner       | None                                 |
| size        | 0                                    |
| status      | unused                               |
| target_id   | None                                 |
| type        | kolla                                |
| updated_at  | 2017-08-28T06:45:25.000000           |
| version     | None                                 |
+-------------+--------------------------------------+
  1. Get cluster ID
[root@daisy ~]# daisy cluster-list
+--------------------------------------+-------------+...
| ID                                   | Name        |...
+--------------------------------------+-------------+...
| d4c1e0d3-c4b8-4745-aab0-0510e62f0ebb | clustertest |...
+--------------------------------------+-------------+...
  1. Issuing update command passing cluster ID and version ID
[root@daisy ~]# daisy update d4c1e0d3-c4b8-4745-aab0-0510e62f0ebb --update-object kolla --version-id 8be92587-34d7-43e8-9862-a5288c651079
+----------+--------------+
| Property | Value        |
+----------+--------------+
| status   | begin update |
+----------+--------------+

6. Since step 5’s command is non-blocking, the user need to run the following command to get updating progress.

[root@daisy ~]# daisy host-list --cluster-id d4c1e0d3-c4b8-4745-aab0-0510e62f0ebb
...+---------------+-------------+-------------------------+
...| Role_progress | Role_status | Role_messages           |
...+---------------+-------------+-------------------------+
...| 0             | updating    | prechecking envirnoment |
...+---------------+-------------+-------------------------+

Notes. The above command returns many fields. User only have to take care about the Role_xxx fields in this case.

Build Your Own Kolla Image For Daisy

The following command will build Ocata Kolla image for Daisy based on Daisy’s fork of openstack/kolla project. This is also the method Daisy used for the Euphrates release.

The reason why here use fork of openstack/kolla project is to backport ODL support from pike branch to ocata branch.

cd ./ci
./kolla-build.sh

After building, the above command will put Kolla image into /tmp/kolla-build-output directory and the image version will be 4.0.2.

If you want to build an image which can update 4.0.2, run the following command:

cd ./ci
./kolla-build.sh -e 1

This time the image version will be 4.0.2.1 which is higher than 4.0.2 so that it can be used to replace the old version.

Deployment Test Guide

After successful deployment of openstack, daisy4nfv use Functest to test the api of openstack. You can follow below instruction to test the successfully deployed openstack on jumperserver.

1.docker pull opnfv/functest run ‘docker images’ command to make sure have the latest functest images.

2.docker run -ti –name functest -e INSTALLER_TYPE=”daisy”-e INSTALLER_IP=”10.20.11.2” -e NODE_NAME=”zte-vtest” -e DEPLOY_SCENARIO=”os-nosdn-nofeature-ha” -e BUILD_TAG=”jenkins-functest-daisy-virtual-daily-master-1259” -e DEPLOY_TYPE=”virt” opnfv/functest:latest /bin/bash Before run above command change below parameters: DEPLOY_SCENARIO: indicate the scenario DEPLOY_TYPE: virt/baremetal NODE_NAME: pod name INSTALLER_IP: daisy vm node ip

3.Log in the daisy vm node to get the /etc/kolla/admin-openrc.sh file, and write them in /home/opnfv/functest/conf/openstack.creds file of functest container.

4.Run command ‘functest env prepare’ to prepare the functest env.

5.Run command ‘functest testcase list’ to list all the testcase can be run.

6.Run command ‘functest testcase run testcase_name’ to run the testcase_name testcase of functest.

Doctor

Doctor Development Guide
Testing Doctor

You have two options to test Doctor functions with the script developed for doctor CI.

You need to install OpenStack and other OPNFV components except Doctor Sample Inspector, Sample Monitor and Sample Consumer, as these will be launched in this script. You are encouraged to use OPNFV offcial installers, but you can also deploy all components with other installers such as devstack or manual operation. In those cases, the versions of all components shall be matched with the versions of them in OPNFV specific release.

Run Test Script

Doctor project has own testing script under doctor/tests. This test script can be used for functional testing agained an OPNFV deployment.

Before running this script, make sure OpenStack env parameters are set properly following OpenStack CLI manual, so that Doctor Inspector can operate OpenStack services.

Run Bash Test Script

You can run the bash script as follows:

git clone https://gerrit.opnfv.org/gerrit/doctor
cd doctor/tests
export INSTALLER_TYPE=local
export INSPECTOR_TYPE=sample
./run.sh

INSTALLER_TYPE can be ‘apex’, ‘fuel’, ‘joid’ and ‘local’(default). If you are not using OPNFV installers in this option, chose ‘local’. INSPECTOR_TYPE can be specified either ‘sample’(default) or ‘congress’.

For testing with stable version, checkout stable branch of doctor repo before ‘./run.sh’.

The bash test script will be deprecated(only bug fixes) after E Release.

Run Python Test Script

You can run the python script as follows:

git clone https://gerrit.opnfv.org/gerrit/doctor
cd doctor && tox

You can see all the configurations with default values in sample configuration file doctor.sample.conf. And you can also modify the file to meet your environment and then run the test.

Run Functest Suite

Functest supports Doctor testing by triggering the test script above in a Functest container. You can run the Doctor test with the following steps:

DOCKER_TAG=latest
docker pull docker.io/opnfv/functest-features:${DOCKER_TAG}
docker run --privileged=true -id \
    -e INSTALLER_TYPE=${INSTALLER_TYPE} \
    -e INSTALLER_IP=${INSTALLER_IP} \
    -e INSPECTOR_TYPE=sample \
    docker.io/opnfv/functest-features:${DOCKER_TAG} /bin/bash
docker exec <container_id> functest env prepare
docker exec <container_id> functest testcase run doctor-notification

See Functest Userguide for more information.

For testing with stable version, change DOCKER_TAG to ‘stable’ or other release tag identifier.

Tips
Doctor: Fault Management and Maintenance
Project:

Doctor, https://wiki.opnfv.org/doctor

Editors:

Ashiq Khan (NTT DOCOMO), Gerald Kunzmann (NTT DOCOMO)

Authors:

Ryota Mibu (NEC), Carlos Goncalves (NEC), Tomi Juvonen (Nokia), Tommy Lindgren (Ericsson), Bertrand Souville (NTT DOCOMO), Balazs Gibizer (Ericsson), Ildiko Vancsa (Ericsson) and others.

Abstract:

Doctor is an OPNFV requirement project [DOCT]. Its scope is NFVI fault management, and maintenance and it aims at developing and realizing the consequent implementation for the OPNFV reference platform.

This deliverable is introducing the use cases and operational scenarios for Fault Management considered in the Doctor project. From the general features, a high level architecture describing logical building blocks and interfaces is derived. Finally, a detailed implementation is introduced, based on available open source components, and a related gap analysis is done as part of this project. The implementation plan finally discusses an initial realization for a NFVI fault management and maintenance solution in open source software.

Definition of terms

Different SDOs and communities use different terminology related to NFV/Cloud/SDN. This list tries to define an OPNFV terminology, mapping/translating the OPNFV terms to terminology used in other contexts.

ACT-STBY configuration
Failover configuration common in Telco deployments. It enables the operator to use a standby (STBY) instance to take over the functionality of a failed active (ACT) instance.
Administrator
Administrator of the system, e.g. OAM in Telco context.
Consumer
User-side Manager; consumer of the interfaces produced by the VIM; VNFM, NFVO, or Orchestrator in ETSI NFV [ENFV] terminology.
EPC
Evolved Packet Core, the main component of the core network architecture of 3GPP’s LTE communication standard.
MME
Mobility Management Entity, an entity in the EPC dedicated to mobility management.
NFV
Network Function Virtualization
NFVI
Network Function Virtualization Infrastructure; totality of all hardware and software components which build up the environment in which VNFs are deployed.
S/P-GW
Serving/PDN-Gateway, two entities in the EPC dedicated to routing user data packets and providing connectivity from the UE to external packet data networks (PDN), respectively.
Physical resource
Actual resources in NFVI; not visible to Consumer.
VNFM
Virtualized Network Function Manager; functional block that is responsible for the lifecycle management of VNF.
NFVO
Network Functions Virtualization Orchestrator; functional block that manages the Network Service (NS) lifecycle and coordinates the management of NS lifecycle, VNF lifecycle (supported by the VNFM) and NFVI resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity.
VIM
Virtualized Infrastructure Manager; functional block that is responsible for controlling and managing the NFVI compute, storage and network resources, usually within one operator’s Infrastructure Domain, e.g. NFVI Point of Presence (NFVI-PoP).
Virtual Machine (VM)
Virtualized computation environment that behaves very much like a physical computer/server.
Virtual network
Virtual network routes information among the network interfaces of VM instances and physical network interfaces, providing the necessary connectivity.
Virtual resource
A Virtual Machine (VM), a virtual network, or virtualized storage; Offered resources to “Consumer” as result of infrastructure virtualization; visible to Consumer.
Virtual Storage
Virtualized non-volatile storage allocated to a VM.
VNF
Virtualized Network Function. Implementation of a Network Function that can be deployed on a Network Function Virtualization Infrastructure (NFVI).
Introduction

The goal of this project is to build an NFVI fault management and maintenance framework supporting high availability of the Network Services on top of the virtualized infrastructure. The key feature is immediate notification of unavailability of virtualized resources from VIM, to support failure recovery, or failure avoidance of VNFs running on them. Requirement survey and development of missing features in NFVI and VIM are in scope of this project in order to fulfil requirements for fault management and maintenance in NFV.

The purpose of this requirement project is to clarify the necessary features of NFVI fault management, and maintenance, identify missing features in the current OpenSource implementations, provide a potential implementation architecture and plan, provide implementation guidelines in relevant upstream projects to realize those missing features, and define the VIM northbound interfaces necessary to perform the task of NFVI fault management, and maintenance in alignment with ETSI NFV [ENFV].

Problem description

A Virtualized Infrastructure Manager (VIM), e.g. OpenStack [OPSK], cannot detect certain Network Functions Virtualization Infrastructure (NFVI) faults. This feature is necessary to detect the faults and notify the Consumer in order to ensure the proper functioning of EPC VNFs like MME and S/P-GW.

  • EPC VNFs are often in active standby (ACT-STBY) configuration and need to switch from STBY mode to ACT mode as soon as relevant faults are detected in the active (ACT) VNF.
  • NFVI encompasses all elements building up the environment in which VNFs are deployed, e.g., Physical Machines, Hypervisors, Storage, and Network elements.

In addition, VIM, e.g. OpenStack, needs to receive maintenance instructions from the Consumer, i.e. the operator/administrator of the VNF.

  • Change the state of certain Physical Machines (PMs), e.g. empty the PM, so that maintenance work can be performed at these machines.

Note: Although fault management and maintenance are different operations in NFV, both are considered as part of this project as – except for the trigger – they share a very similar work and message flow. Hence, from implementation perspective, these two are kept together in the Doctor project because of this high degree of similarity.

Use cases and scenarios

Telecom services often have very high requirements on service performance. As a consequence they often utilize redundancy and high availability (HA) mechanisms for both the service and the platform. The HA support may be built-in or provided by the platform. In any case, the HA support typically has a very fast detection and reaction time to minimize service impact. The main changes proposed in this document are about making a clear distinction between fault management and recovery a) within the VIM/NFVI and b) High Availability support for VNFs on the other, claiming that HA support within a VNF or as a service from the platform is outside the scope of Doctor and is discussed in the High Availability for OPNFV project. Doctor should focus on detecting and remediating faults in the NFVI. This will ensure that applications come back to a fully redundant configuration faster than before.

As an example, Telecom services can come with an Active-Standby (ACT-STBY) configuration which is a (1+1) redundancy scheme. ACT and STBY nodes (aka Physical Network Function (PNF) in ETSI NFV terminology) are in a hot standby configuration. If an ACT node is unable to function properly due to fault or any other reason, the STBY node is instantly made ACT, and affected services can be provided without any service interruption.

The ACT-STBY configuration needs to be maintained. This means, when a STBY node is made ACT, either the previously ACT node, after recovery, shall be made STBY, or, a new STBY node needs to be configured. The actual operations to instantiate/configure a new STBY are similar to instantiating a new VNF and therefore are outside the scope of this project.

The NFVI fault management and maintenance requirements aim at providing fast failure detection of physical and virtualized resources and remediation of the virtualized resources provided to Consumers according to their predefined request to enable applications to recover to a fully redundant mode of operation.

  1. Fault management/recovery using ACT-STBY configuration (Triggered by critical error)
  2. Preventive actions based on fault prediction (Preventing service stop by handling warnings)
  3. VM Retirement (Managing service during NFVI maintenance, i.e. H/W, Hypervisor, Host OS, maintenance)
Faults
Fault management using ACT-STBY configuration

In figure1, a system-wide view of relevant functional blocks is presented. OpenStack is considered as the VIM implementation (aka Controller) which has interfaces with the NFVI and the Consumers. The VNF implementation is represented as different virtual resources marked by different colors. Consumers (VNFM or NFVO in ETSI NFV terminology) own/manage the respective virtual resources (VMs in this example) shown with the same colors.

The first requirement in this use case is that the Controller needs to detect faults in the NFVI (“1. Fault Notification” in figure1) affecting the proper functioning of the virtual resources (labelled as VM-x) running on top of it. It should be possible to configure which relevant fault items should be detected. The VIM (e.g. OpenStack) itself could be extended to detect such faults. Alternatively, a third party fault monitoring tool could be used which then informs the VIM about such faults; this third party fault monitoring element can be considered as a component of VIM from an architectural point of view.

Once such fault is detected, the VIM shall find out which virtual resources are affected by this fault. In the example in figure1, VM-4 is affected by a fault in the Hardware Server-3. Such mapping shall be maintained in the VIM, depicted as the “Server-VM info” table inside the VIM.

Once the VIM has identified which virtual resources are affected by the fault, it needs to find out who is the Consumer (i.e. the owner/manager) of the affected virtual resources (Step 2). In the example shown in figure1, the VIM knows that for the red VM-4, the manager is the red Consumer through an Ownership info table. The VIM then notifies (Step 3 “Fault Notification”) the red Consumer about this fault, preferably with sufficient abstraction rather than detailed physical fault information.

_images/figure1.png

Fault management/recovery use case

The Consumer then switches to STBY configuration by switching the STBY node to ACT state (Step 4). It further initiates a process to instantiate/configure a new STBY. However, switching to STBY mode and creating a new STBY machine is a VNFM/NFVO level operation and therefore outside the scope of this project. Doctor project does not create interfaces for such VNFM level configuration operations. Yet, since the total failover time of a consumer service depends on both the delay of such processes as well as the reaction time of Doctor components, minimizing Doctor’s reaction time is a necessary basic ingredient to fast failover times in general.

Once the Consumer has switched to STBY configuration, it notifies (Step 5 “Instruction” in figure1) the VIM. The VIM can then take necessary (e.g. pre-determined by the involved network operator) actions on how to clean up the fault affected VMs (Step 6 “Execute Instruction”).

The key issue in this use case is that a VIM (OpenStack in this context) shall not take a standalone fault recovery action (e.g. migration of the affected VMs) before the ACT-STBY switching is complete, as that might violate the ACT-STBY configuration and render the node out of service.

As an extension of the 1+1 ACT-STBY resilience pattern, a STBY instance can act as backup to N ACT nodes (N+1). In this case, the basic information flow remains the same, i.e., the consumer is informed of a failure in order to activate the STBY node. However, in this case it might be useful for the failure notification to cover a number of failed instances due to the same fault (e.g., more than one instance might be affected by a switch failure). The reaction of the consumer might depend on whether only one active instance has failed (similar to the ACT-STBY case), or if more active instances are needed as well.

Preventive actions based on fault prediction

The fault management scenario explained in Fault management using ACT-STBY configuration can also be performed based on fault prediction. In such cases, in VIM, there is an intelligent fault prediction module which, based on its NFVI monitoring information, can predict an imminent fault in the elements of NFVI. A simple example is raising temperature of a Hardware Server which might trigger a pre-emptive recovery action. The requirements of such fault prediction in the VIM are investigated in the OPNFV project “Data Collection for Failure Prediction” [PRED].

This use case is very similar to Fault management using ACT-STBY configuration. Instead of a fault detection (Step 1 “Fault Notification in” figure1), the trigger comes from a fault prediction module in the VIM, or from a third party module which notifies the VIM about an imminent fault. From Step 2~5, the work flow is the same as in the “Fault management using ACT-STBY configuration” use case, except in this case, the Consumer of a VM/VNF switches to STBY configuration based on a predicted fault, rather than an occurred fault.

NFVI Maintenance
VM Retirement

All network operators perform maintenance of their network infrastructure, both regularly and irregularly. Besides the hardware, virtualization is expected to increase the number of elements subject to such maintenance as NFVI holds new elements like the hypervisor and host OS. Maintenance of a particular resource element e.g. hardware, hypervisor etc. may render a particular server hardware unusable until the maintenance procedure is complete.

However, the Consumer of VMs needs to know that such resources will be unavailable because of NFVI maintenance. The following use case is again to ensure that the ACT-STBY configuration is not violated. A stand-alone action (e.g. live migration) from VIM/OpenStack to empty a physical machine so that consequent maintenance procedure could be performed may not only violate the ACT-STBY configuration, but also have impact on real-time processing scenarios where dedicated resources to virtual resources (e.g. VMs) are necessary and a pause in operation (e.g. vCPU) is not allowed. The Consumer is in a position to safely perform the switch between ACT and STBY nodes, or switch to an alternative VNF forwarding graph so the hardware servers hosting the ACT nodes can be emptied for the upcoming maintenance operation. Once the target hardware servers are emptied (i.e. no virtual resources are running on top), the VIM can mark them with an appropriate flag (i.e. “maintenance” state) such that these servers are not considered for hosting of virtual machines until the maintenance flag is cleared (i.e. nodes are back in “normal” status).

A high-level view of the maintenance procedure is presented in figure2. VIM/OpenStack, through its northbound interface, receives a maintenance notification (Step 1 “Maintenance Request”) from the Administrator (e.g. a network operator) including information about which hardware is subject to maintenance. Maintenance operations include replacement/upgrade of hardware, update/upgrade of the hypervisor/host OS, etc.

The consequent steps to enable the Consumer to perform ACT-STBY switching are very similar to the fault management scenario. From VIM/OpenStack’s internal database, it finds out which virtual resources (VM-x) are running on those particular Hardware Servers and who are the managers of those virtual resources (Step 2). The VIM then informs the respective Consumer (VNFMs or NFVO) in Step 3 “Maintenance Notification”. Based on this, the Consumer takes necessary actions (Step 4, e.g. switch to STBY configuration or switch VNF forwarding graphs) and then notifies (Step 5 “Instruction”) the VIM. Upon receiving such notification, the VIM takes necessary actions (Step 6 “Execute Instruction” to empty the Hardware Servers so that consequent maintenance operations could be performed. Due to the similarity for Steps 2~6, the maintenance procedure and the fault management procedure are investigated in the same project.

_images/figure2.png

Maintenance use case

High level architecture and general features
Functional overview

The Doctor project circles around two distinct use cases: 1) management of failures of virtualized resources and 2) planned maintenance, e.g. migration, of virtualized resources. Both of them may affect a VNF/application and the network service it provides, but there is a difference in frequency and how they can be handled.

Failures are spontaneous events that may or may not have an impact on the virtual resources. The Consumer should as soon as possible react to the failure, e.g., by switching to the STBY node. The Consumer will then instruct the VIM on how to clean up or repair the lost virtual resources, i.e. restore the VM, VLAN or virtualized storage. How much the applications are affected varies. Applications with built-in HA support might experience a short decrease in retainability (e.g. an ongoing session might be lost) while keeping availability (establishment or re-establishment of sessions are not affected), whereas the impact on applications without built-in HA may be more serious. How much the network service is impacted depends on how the service is implemented. With sufficient network redundancy the service may be unaffected even when a specific resource fails.

On the other hand, planned maintenance impacting virtualized resources are events that are known in advance. This group includes e.g. migration due to software upgrades of OS and hypervisor on a compute host. Some of these might have been requested by the application or its management solution, but there is also a need for coordination on the actual operations on the virtual resources. There may be an impact on the applications and the service, but since they are not spontaneous events there is room for planning and coordination between the application management organization and the infrastructure management organization, including performing whatever actions that would be required to minimize the problems.

Failure prediction is the process of pro-actively identifying situations that may lead to a failure in the future unless acted on by means of maintenance activities. From applications’ point of view, failure prediction may impact them in two ways: either the warning time is so short that the application or its management solution does not have time to react, in which case it is equal to the failure scenario, or there is sufficient time to avoid the consequences by means of maintenance activities, in which case it is similar to planned maintenance.

Architecture Overview

NFV and the Cloud platform provide virtual resources and related control functionality to users and administrators. figure3 shows the high level architecture of NFV focusing on the NFVI, i.e., the virtualized infrastructure. The NFVI provides virtual resources, such as virtual machines (VM) and virtual networks. Those virtual resources are used to run applications, i.e. VNFs, which could be components of a network service which is managed by the consumer of the NFVI. The VIM provides functionalities of controlling and viewing virtual resources on hardware (physical) resources to the consumers, i.e., users and administrators. OpenStack is a prominent candidate for this VIM. The administrator may also directly control the NFVI without using the VIM.

Although OpenStack is the target upstream project where the new functional elements (Controller, Notifier, Monitor, and Inspector) are expected to be implemented, a particular implementation method is not assumed. Some of these elements may sit outside of OpenStack and offer a northbound interface to OpenStack.

General Features and Requirements

The following features are required for the VIM to achieve high availability of applications (e.g., MME, S/P-GW) and the Network Services:

  1. Monitoring: Monitor physical and virtual resources.
  2. Detection: Detect unavailability of physical resources.
  3. Correlation and Cognition: Correlate faults and identify affected virtual resources.
  4. Notification: Notify unavailable virtual resources to their Consumer(s).
  5. Fencing: Shut down or isolate a faulty resource.
  6. Recovery action: Execute actions to process fault recovery and maintenance.

The time interval between the instant that an event is detected by the monitoring system and the Consumer notification of unavailable resources shall be < 1 second (e.g., Step 1 to Step 4 in figure4).

_images/figure3.png

High level architecture

Monitoring

The VIM shall monitor physical and virtual resources for unavailability and suspicious behavior.

Detection

The VIM shall detect unavailability and failures of physical resources that might cause errors/faults in virtual resources running on top of them. Unavailability of physical resource is detected by various monitoring and managing tools for hardware and software components. This may include also predicting upcoming faults. Note, fault prediction is out of scope of this project and is investigated in the OPNFV “Data Collection for Failure Prediction” project [PRED].

The fault items/events to be detected shall be configurable.

The configuration shall enable Failure Selection and Aggregation. Failure aggregation means the VIM determines unavailability of physical resource from more than two non-critical failures related to the same resource.

There are two types of unavailability - immediate and future:

  • Immediate unavailability can be detected by setting traps of raw failures on hardware monitoring tools.
  • Future unavailability can be found by receiving maintenance instructions issued by the administrator of the NFVI or by failure prediction mechanisms.
Correlation and Cognition

The VIM shall correlate each fault to the impacted virtual resource, i.e., the VIM shall identify unavailability of virtualized resources that are or will be affected by failures on the physical resources under them. Unavailability of a virtualized resource is determined by referring to the mapping of physical and virtualized resources.

VIM shall allow configuration of fault correlation between physical and virtual resources. VIM shall support correlating faults:

  • between a physical resource and another physical resource
  • between a physical resource and a virtual resource
  • between a virtual resource and another virtual resource

Failure aggregation is also required in this feature, e.g., a user may request to be only notified if failures on more than two standby VMs in an (N+M) deployment model occurred.

Notification

The VIM shall notify the alarm, i.e., unavailability of virtual resource(s), to the Consumer owning it over the northbound interface, such that the Consumers impacted by the failure can take appropriate actions to recover from the failure.

The VIM shall also notify the unavailability of physical resources to its Administrator.

All notifications shall be transferred immediately in order to minimize the stalling time of the network service and to avoid over assignment caused by delay of capability updates.

There may be multiple consumers, so the VIM has to find out the owner of a faulty resource. Moreover, there may be a large number of virtual and physical resources in a real deployment, so polling the state of all resources to the VIM would lead to heavy signaling traffic. Thus, a publication/subscription messaging model is better suited for these notifications, as notifications are only sent to subscribed consumers.

Notifications will be send out along with the configuration by the consumer. The configuration includes endpoint(s) in which the consumers can specify multiple targets for the notification subscription, so that various and multiple receiver functions can consume the notification message. Also, the conditions for notifications shall be configurable, such that the consumer can set according policies, e.g. whether it wants to receive fault notifications or not.

Note: the VIM should only accept notification subscriptions for each resource by its owner or administrator. Notifications to the Consumer about the unavailability of virtualized resources will include a description of the fault, preferably with sufficient abstraction rather than detailed physical fault information.

Fencing

Recovery actions, e.g. safe VM evacuation, have to be preceded by fencing the failed host. Fencing hereby means to isolate or shut down a faulty resource. Without fencing – when the perceived disconnection is due to some transient or partial failure – the evacuation might lead into two identical instances running together and having a dangerous conflict.

There is a cross-project definition in OpenStack of how to implement fencing, but there has not been any progress. The general description is available here: https://wiki.openstack.org/wiki/Fencing_Instances_of_an_Unreachable_Host

OpenStack provides some mechanisms that allow fencing of faulty resources. Some are automatically invoked by the platform itself (e.g. Nova disables the compute service when libvirtd stops running, preventing new VMs to be scheduled to that node), while other mechanisms are consumer trigger-based actions (e.g. Neutron port admin-state-up). For other fencing actions not supported by OpenStack, the Doctor project may suggest ways to address the gap (e.g. through means of resourcing to external tools and orchestration methods), or documenting or implementing them upstream.

The Doctor Inspector component will be responsible of marking resources down in the OpenStack and back up if necessary.

Recovery Action

In the basic Fault management using ACT-STBY configuration use case, no automatic actions will be taken by the VIM, but all recovery actions executed by the VIM and the NFVI will be instructed and coordinated by the Consumer.

In a more advanced use case, the VIM may be able to recover the failed virtual resources according to a pre-defined behavior for that resource. In principle this means that the owner of the resource (i.e., its consumer or administrator) can define which recovery actions shall be taken by the VIM. Examples are a restart of the VM or migration/evacuation of the VM.

High level northbound interface specification
Fault Management

This interface allows the Consumer to subscribe to fault notification from the VIM. Using a filter, the Consumer can narrow down which faults should be notified. A fault notification may trigger the Consumer to switch from ACT to STBY configuration and initiate fault recovery actions. A fault query request/response message exchange allows the Consumer to find out about active alarms at the VIM. A filter can be used to narrow down the alarms returned in the response message.

_images/figure4.png

High-level message flow for fault management

The high level message flow for the fault management use case is shown in figure4. It consists of the following steps:

  1. The VIM monitors the physical and virtual resources and the fault management workflow is triggered by a monitored fault event.
  2. Event correlation, fault detection and aggregation in VIM. Note: this may also happen after Step 3.
  3. Database lookup to find the virtual resources affected by the detected fault.
  4. Fault notification to Consumer.
  5. The Consumer switches to standby configuration (STBY).
  6. Instructions to VIM requesting certain actions to be performed on the affected resources, for example migrate/update/terminate specific resource(s). After reception of such instructions, the VIM is executing the requested action, e.g., it will migrate or terminate a virtual resource.
NFVI Maintenance

The NFVI maintenance interface allows the Administrator to notify the VIM about a planned maintenance operation on the NFVI. A maintenance operation may for example be an update of the server firmware or the hypervisor. The MaintenanceRequest message contains instructions to change the state of the physical resource from ‘enabled’ to ‘going-to-maintenance’ and a timeout [1]. After receiving the MaintenanceRequest,the VIM decides on the actions to be taken based on maintenance policies predefined by the affected Consumer(s).

[1]Timeout is set by the Administrator and corresponds to the maximum time to empty the physical resources.
_images/figure5a.png

High-level message flow for maintenance policy enforcement

The high level message flow for the NFVI maintenance policy enforcement is shown in figure5a. It consists of the following steps:

  1. Maintenance trigger received from Administrator.
  2. VIM switches the affected physical resources to “going-to-maintenance” state e.g. so that no new VM will be scheduled on the physical servers.
  3. Database lookup to find the Consumer(s) and virtual resources affected by the maintenance operation.
  4. Maintenance policies are enforced in the VIM, e.g. affected VM(s) are shut down on the physical server(s), or affected Consumer(s) are notified about the planned maintenance operation (steps 4a/4b).

Once the affected Consumer(s) have been notified, they take specific actions (e.g. switch to standby (STBY) configuration, request to terminate the virtual resource(s)) to allow the maintenance action to be executed. After the physical resources have been emptied, the VIM puts the physical resources in “in-maintenance” state and sends a MaintenanceResponse back to the Administrator.

_images/figure5b.png

Successful NFVI maintenance

The high level message flow for a successful NFVI maintenance is show in figure5b. It consists of the following steps:

  1. The Consumer C3 switches to standby configuration (STBY).
  2. Instructions from Consumers C2/C3 are shared to VIM requesting certain actions to be performed (steps 6a, 6b). After receiving such instructions, the VIM executes the requested action in order to empty the physical resources (step 6c) and informs the Consumer about the result of the actions (steps 6d, 6e).
  3. The VIM switches the physical resources to “in-maintenance” state
  4. Maintenance response is sent from VIM to inform the Administrator that the physical servers have been emptied.
  5. The Administrator is coordinating and executing the maintenance operation/work on the NFVI. Note: this step is out of scope of Doctor project.

The requested actions to empty the physical resources may not be successful (e.g. migration fails or takes too long) and in such a case, the VIM puts the physical resources back to ‘enabled’ and informs the Administrator about the problem.

_images/figure5c.png

Example of failed NFVI maintenance

An example of a high level message flow to cover the failed NFVI maintenance case is shown in figure5c. It consists of the following steps:

  1. The Consumer C3 switches to standby configuration (STBY).
  2. Instructions from Consumers C2/C3 are shared to VIM requesting certain actions to be performed (steps 6a, 6b). The VIM executes the requested actions and sends back a NACK to consumer C2 (step 6d) as the migration of the virtual resource(s) is not completed by the given timeout.
  3. The VIM switches the physical resources to “enabled” state.
  4. MaintenanceNotification is sent from VIM to inform the Administrator that the maintenance action cannot start.
Gap analysis in upstream projects

This section presents the findings of gaps on existing VIM platforms. The focus was to identify gaps based on the features and requirements specified in Section 3.3. The analysis work determined gaps that are presented here.

VIM Northbound Interface
Immediate Notification
  • Type: ‘deficiency in performance’
  • Description
    • To-be
      • VIM has to notify unavailability of virtual resource (fault) to VIM user immediately.
      • Notification should be passed in ‘1 second’ after fault detected/notified by VIM.
      • Also, the following conditions/requirement have to be met:
        • Only the owning user can receive notification of fault related to owned virtual resource(s).
    • As-is
      • OpenStack Metering ‘Ceilometer’ can notify unavailability of virtual resource (fault) to the owner of virtual resource based on alarm configuration by the user.
      • Alarm notifications are triggered by alarm evaluator instead of notification agents that might receive faults
      • Evaluation interval should be equal to or larger than configured pipeline interval for collection of underlying metrics.
      • The interval for collection has to be set large enough which depends on the size of the deployment and the number of metrics to be collected.
      • The interval may not be less than one second in even small deployments. The default value is 60 seconds.
      • Alternative: OpenStack has a message bus to publish system events. The operator can allow the user to connect this, but there are no functions to filter out other events that should not be passed to the user or which were not requested by the user.
    • Gap
      • Fault notifications cannot be received immediately by Ceilometer.
  • Solved by
Maintenance Notification
VIM Southbound interface
Normalization of data collection models
  • Type: ‘missing’
  • Description
    • To-be
      • A normalized data format needs to be created to cope with the many data models from different monitoring solutions.
    • As-is
      • Data can be collected from many places (e.g. Zabbix, Nagios, Cacti, Zenoss). Although each solution establishes its own data models, no common data abstraction models exist in OpenStack.
    • Gap
      • Normalized data format does not exist.
  • Solved by
OpenStack
Ceilometer

OpenStack offers a telemetry service, Ceilometer, for collecting measurements of the utilization of physical and virtual resources [CEIL]. Ceilometer can collect a number of metrics across multiple OpenStack components and watch for variations and trigger alarms based upon the collected data.

Scalability of fault aggregation
  • Type: ‘scalability issue’
  • Description
    • To-be
      • Be able to scale to a large deployment, where thousands of monitoring events per second need to be analyzed.
    • As-is
      • Performance issue when scaling to medium-sized deployments.
    • Gap
      • Ceilometer seems to be unsuitable for monitoring medium and large scale NFVI deployments.
  • Solved by
Monitoring of hardware and software
  • Type: ‘missing (lack of functionality)’
  • Description
    • To-be
      • OpenStack (as VIM) should monitor various hardware and software in NFVI to handle faults on them by Ceilometer.
      • OpenStack may have monitoring functionality in itself and can be integrated with third party monitoring tools.
      • OpenStack need to be able to detect the faults listed in the Annex.
    • As-is
      • For each deployment of OpenStack, an operator has responsibility to configure monitoring tools with relevant scripts or plugins in order to monitor hardware and software.
      • OpenStack Ceilometer does not monitor hardware and software to capture faults.
    • Gap
      • Ceilometer is not able to detect and handle all faults listed in the Annex.
  • Solved by
Nova

OpenStack Nova [NOVA] is a mature and widely known and used component in OpenStack cloud deployments. It is the main part of an “infrastructure-as-a-service” system providing a cloud computing fabric controller, supporting a wide diversity of virtualization and container technologies.

Nova has proven throughout these past years to be highly available and fault-tolerant. Featuring its own API, it also provides a compatibility API with Amazon EC2 APIs.

Correct states when compute host is down
  • Type: ‘missing (lack of functionality)’
  • Description
    • To-be
      • The API shall support to change VM power state in case host has failed.
      • The API shall support to change nova-compute state.
      • There could be single API to change different VM states for all VMs belonging to a specific host.
      • Support external systems that are monitoring the infrastructure and resources that are able to call the API fast and reliable.
      • Resource states are reliable such that correlation actions can be fast and automated.
      • User shall be able to read states from OpenStack and trust they are correct.
    • As-is
      • When a VM goes down due to a host HW, host OS or hypervisor failure, nothing happens in OpenStack. The VMs of a crashed host/hypervisor are reported to be live and OK through the OpenStack API.
      • nova-compute state might change too slowly or the state is not reliable if expecting also VMs to be down. This leads to ability to schedule VMs to a failed host and slowness blocks evacuation.
    • Gap
      • OpenStack does not change its states fast and reliably enough.
      • The API does not support to have an external system to change states and to trust the states are reliable (external system has fenced failed host).
      • User cannot read all the states from OpenStack nor trust they are right.
  • Solved by
Evacuate VMs in Maintenance mode
  • Type: ‘missing’
  • Description
    • To-be
      • When maintenance mode for a compute host is set, trigger VM evacuation to available compute nodes before bringing the host down for maintenance.
    • As-is
      • If setting a compute node to a maintenance mode, OpenStack only schedules evacuation of all VMs to available compute nodes if in-maintenance compute node runs the XenAPI and VMware ESX hypervisors. Other hypervisors (e.g. KVM) are not supported and, hence, guest VMs will likely stop running due to maintenance actions administrator may perform (e.g. hardware upgrades, OS updates).
    • Gap
      • Nova libvirt hypervisor driver does not implement automatic guest VMs evacuation when compute nodes are set to maintenance mode ($ nova host-update --maintenance enable <hostname>).
Monasca

Monasca is an open-source monitoring-as-a-service (MONaaS) solution that integrates with OpenStack. Even though it is still in its early days, it is the interest of the community that the platform be multi-tenant, highly scalable, performant and fault-tolerant. It provides a streaming alarm engine, a notification engine, and a northbound REST API users can use to interact with Monasca. Hundreds of thousands of metrics per second can be processed [MONA].

Anomaly detection
  • Type: ‘missing (lack of functionality)’
  • Description
    • To-be
      • Detect the failure and perform a root cause analysis to filter out other alarms that may be triggered due to their cascading relation.
    • As-is
      • A mechanism to detect root causes of failures is not available.
    • Gap
      • Certain failures can trigger many alarms due to their dependency on the underlying root cause of failure. Knowing the root cause can help filter out unnecessary and overwhelming alarms.
  • Status
    • Monasca as of now lacks this feature, although the community is aware and working toward supporting it.
Sensor monitoring
  • Type: ‘missing (lack of functionality)’
  • Description
    • To-be
      • It should support monitoring sensor data retrieval, for instance, from IPMI.
    • As-is
      • Monasca does not monitor sensor data
    • Gap
      • Sensor monitoring is very important. It provides operators status on the state of the physical infrastructure (e.g. temperature, fans).
  • Addressed by
    • Monasca can be configured to use third-party monitoring solutions (e.g. Nagios, Cacti) for retrieving additional data.
Hardware monitoring tools
Zabbix

Zabbix is an open-source solution for monitoring availability and performance of infrastructure components (i.e. servers and network devices), as well as applications [ZABB]. It can be customized for use with OpenStack. It is a mature tool and has been proven to be able to scale to large systems with 100,000s of devices.

Delay in execution of actions
  • Type: ‘deficiency in performance’
  • Description
    • To-be
      • After detecting a fault, the monitoring tool should immediately execute the appropriate action, e.g. inform the manager through the NB I/F
    • As-is
      • A delay of around 10 seconds was measured in two independent testbed deployments
    • Gap
Detailed architecture and interface specification

This section describes a detailed implementation plan, which is based on the high level architecture introduced in Section 3. Section 5.1 describes the functional blocks of the Doctor architecture, which is followed by a high level message flow in Section 5.2. Section 5.3 provides a mapping of selected existing open source components to the building blocks of the Doctor architecture. Thereby, the selection of components is based on their maturity and the gap analysis executed in Section 4. Sections 5.4 and 5.5 detail the specification of the related northbound interface and the related information elements. Finally, Section 5.6 provides a first set of blueprints to address selected gaps required for the realization functionalities of the Doctor project.

Functional Blocks

This section introduces the functional blocks to form the VIM. OpenStack was selected as the candidate for implementation. Inside the VIM, 4 different building blocks are defined (see figure6).

_images/figure6.png

Functional blocks

Monitor

The Monitor module has the responsibility for monitoring the virtualized infrastructure. There are already many existing tools and services (e.g. Zabbix) to monitor different aspects of hardware and software resources which can be used for this purpose.

Inspector

The Inspector module has the ability a) to receive various failure notifications regarding physical resource(s) from Monitor module(s), b) to find the affected virtual resource(s) by querying the resource map in the Controller, and c) to update the state of the virtual resource (and physical resource).

The Inspector has drivers for different types of events and resources to integrate any type of Monitor and Controller modules. It also uses a failure policy database to decide on the failure selection and aggregation from raw events. This failure policy database is configured by the Administrator.

The reason for separation of the Inspector and Controller modules is to make the Controller focus on simple operations by avoiding a tight integration of various health check mechanisms into the Controller.

Controller

The Controller is responsible for maintaining the resource map (i.e. the mapping from physical resources to virtual resources), accepting update requests for the resource state(s) (exposing as provider API), and sending all failure events regarding virtual resources to the Notifier. Optionally, the Controller has the ability to force the state of a given physical resource to down in the resource mapping when it receives failure notifications from the Inspector for that given physical resource. The Controller also re-calculates the capacity of the NVFI when receiving a failure notification for a physical resource.

In a real-world deployment, the VIM may have several controllers, one for each resource type, such as Nova, Neutron and Cinder in OpenStack. Each controller maintains a database of virtual and physical resources which shall be the master source for resource information inside the VIM.

Notifier

The focus of the Notifier is on selecting and aggregating failure events received from the controller based on policies mandated by the Consumer. Therefore, it allows the Consumer to subscribe for alarms regarding virtual resources using a method such as API endpoint. After receiving a fault event from a Controller, it will notify the fault to the Consumer by referring to the alarm configuration which was defined by the Consumer earlier on.

To reduce complexity of the Controller, it is a good approach for the Controllers to emit all notifications without any filtering mechanism and have another service (i.e. Notifier) handle those notifications properly. This is the general philosophy of notifications in OpenStack. Note that a fault message consumed by the Notifier is different from the fault message received by the Inspector; the former message is related to virtual resources which are visible to users with relevant ownership, whereas the latter is related to raw devices or small entities which should be handled with an administrator privilege.

The northbound interface between the Notifier and the Consumer/Administrator is specified in Detailed northbound interface specification.

Sequence
Fault Management

The detailed work flow for fault management is as follows (see also figure7):

  1. Request to subscribe to monitor specific virtual resources. A query filter can be used to narrow down the alarms the Consumer wants to be informed about.
  2. Each subscription request is acknowledged with a subscribe response message. The response message contains information about the subscribed virtual resources, in particular if a subscribed virtual resource is in “alarm” state.
  3. The NFVI sends monitoring events for resources the VIM has been subscribed to. Note: this subscription message exchange between the VIM and NFVI is not shown in this message flow.
  4. Event correlation, fault detection and aggregation in VIM.
  5. Database lookup to find the virtual resources affected by the detected fault.
  6. Fault notification to Consumer.
  7. The Consumer switches to standby configuration (STBY)
  8. Instructions to VIM requesting certain actions to be performed on the affected resources, for example migrate/update/terminate specific resource(s). After reception of such instructions, the VIM is executing the requested action, e.g. it will migrate or terminate a virtual resource.
  1. Query request from Consumer to VIM to get information about the current status of a resource.
  2. Response to the query request with information about the current status of the queried resource. In case the resource is in “fault” state, information about the related fault(s) is returned.

In order to allow for quick reaction to failures, the time interval between fault detection in step 3 and the corresponding recovery actions in step 7 and 8 shall be less than 1 second.

_images/figure7.png

Fault management work flow

_images/figure8.png

Fault management scenario

figure8 shows a more detailed message flow (Steps 4 to 6) between the 4 building blocks introduced in Functional Blocks.

  1. The Monitor observed a fault in the NFVI and reports the raw fault to the Inspector. The Inspector filters and aggregates the faults using pre-configured failure policies.
  2. a) The Inspector queries the Resource Map to find the virtual resources affected by the raw fault in the NFVI. b) The Inspector updates the state of the affected virtual resources in the Resource Map. c) The Controller observes a change of the virtual resource state and informs the Notifier about the state change and the related alarm(s). Alternatively, the Inspector may directly inform the Notifier about it.
  3. The Notifier is performing another filtering and aggregation of the changes and alarms based on the pre-configured alarm configuration. Finally, a fault notification is sent to northbound to the Consumer.
NFVI Maintenance
_images/figure9.png

NFVI maintenance work flow

The detailed work flow for NFVI maintenance is shown in figure9 and has the following steps. Note that steps 1, 2, and 5 to 8a in the NFVI maintenance work flow are very similar to the steps in the fault management work flow and share a similar implementation plan in Release 1.

  1. Subscribe to fault/maintenance notifications.
  2. Response to subscribe request.
  3. Maintenance trigger received from administrator.
  4. VIM switches NFVI resources to “maintenance” state. This, e.g., means they should not be used for further allocation/migration requests
  5. Database lookup to find the virtual resources affected by the detected maintenance operation.
  6. Maintenance notification to Consumer.
  7. The Consumer switches to standby configuration (STBY)
  8. Instructions from Consumer to VIM requesting certain recovery actions to be performed (step 8a). After reception of such instructions, the VIM is executing the requested action in order to empty the physical resources (step 8b).
  9. Maintenance response from VIM to inform the Administrator that the physical machines have been emptied (or the operation resulted in an error state).
  10. Administrator is coordinating and executing the maintenance operation/work on the NFVI.
  1. Query request from Administrator to VIM to get information about the current state of a resource.
  2. Response to the query request with information about the current state of the queried resource(s). In case the resource is in “maintenance” state, information about the related maintenance operation is returned.
_images/figure10.png

NFVI Maintenance scenario

figure10 shows a more detailed message flow (Steps 3 to 6 and 9) between the 4 building blocks introduced in Section 5.1..

  1. The Administrator is sending a StateChange request to the Controller residing in the VIM.

  2. The Controller queries the Resource Map to find the virtual resources affected by the planned maintenance operation.

  3. a) The Controller updates the state of the affected virtual resources in the Resource Map database.

    b) The Controller informs the Notifier about the virtual resources that will be affected by the maintenance operation.

  4. A maintenance notification is sent to northbound to the Consumer.

...

  1. The Controller informs the Administrator after the physical resources have been freed.
Information elements

This section introduces all attributes and information elements used in the messages exchange on the northbound interfaces between the VIM and the VNFO and VNFM.

Note: The information elements will be aligned with current work in ETSI NFV IFA working group.

Simple information elements:

  • SubscriptionID (Identifier): identifies a subscription to receive fault or maintenance notifications.
  • NotificationID (Identifier): identifies a fault or maintenance notification.
  • VirtualResourceID (Identifier): identifies a virtual resource affected by a fault or a maintenance action of the underlying physical resource.
  • PhysicalResourceID (Identifier): identifies a physical resource affected by a fault or maintenance action.
  • VirtualResourceState (String): state of a virtual resource, e.g. “normal”, “maintenance”, “down”, “error”.
  • PhysicalResourceState (String): state of a physical resource, e.g. “normal”, “maintenance”, “down”, “error”.
  • VirtualResourceType (String): type of the virtual resource, e.g. “virtual machine”, “virtual memory”, “virtual storage”, “virtual CPU”, or “virtual NIC”.
  • FaultID (Identifier): identifies the related fault in the underlying physical resource. This can be used to correlate different fault notifications caused by the same fault in the physical resource.
  • FaultType (String): Type of the fault. The allowed values for this parameter depend on the type of the related physical resource. For example, a resource of type “compute hardware” may have faults of type “CPU failure”, “memory failure”, “network card failure”, etc.
  • Severity (Integer): value expressing the severity of the fault. The higher the value, the more severe the fault.
  • MinSeverity (Integer): value used in filter information elements. Only faults with a severity higher than the MinSeverity value will be notified to the Consumer.
  • EventTime (Datetime): Time when the fault was observed.
  • EventStartTime and EventEndTime (Datetime): Datetime range that can be used in a FaultQueryFilter to narrow down the faults to be queried.
  • ProbableCause (String): information about the probable cause of the fault.
  • CorrelatedFaultID (Integer): list of other faults correlated to this fault.
  • isRootCause (Boolean): Parameter indicating if this fault is the root for other correlated faults. If TRUE, then the faults listed in the parameter CorrelatedFaultID are caused by this fault.
  • FaultDetails (Key-value pair): provides additional information about the fault, e.g. information about the threshold, monitored attributes, indication of the trend of the monitored parameter.
  • FirmwareVersion (String): current version of the firmware of a physical resource.
  • HypervisorVersion (String): current version of a hypervisor.
  • ZoneID (Identifier): Identifier of the resource zone. A resource zone is the logical separation of physical and software resources in an NFVI deployment for physical isolation, redundancy, or administrative designation.
  • Metadata (Key-value pair): provides additional information of a physical resource in maintenance/error state.

Complex information elements (see also UML diagrams in figure13 and figure14):

  • VirtualResourceInfoClass:
    • VirtualResourceID [1] (Identifier)
    • VirtualResourceState [1] (String)
    • Faults [0..*] (FaultClass): For each resource, all faults including detailed information about the faults are provided.
  • FaultClass: The parameters of the FaultClass are partially based on ETSI TS 132 111-2 (V12.1.0) [*], which is specifying fault management in 3GPP, in particular describing the information elements used for alarm notifications.
    • FaultID [1] (Identifier)
    • FaultType [1] (String)
    • Severity [1] (Integer)
    • EventTime [1] (Datetime)
    • ProbableCause [1] (String)
    • CorrelatedFaultID [0..*] (Identifier)
    • FaultDetails [0..*] (Key-value pair)
[*]http://www.etsi.org/deliver/etsi_ts/132100_132199/13211102/12.01.00_60/ts_13211102v120100p.pdf
  • SubscribeFilterClass
    • VirtualResourceType [0..*] (String)
    • VirtualResourceID [0..*] (Identifier)
    • FaultType [0..*] (String)
    • MinSeverity [0..1] (Integer)
  • FaultQueryFilterClass: narrows down the FaultQueryRequest, for example it limits the query to certain physical resources, a certain zone, a given fault type/severity/cause, or a specific FaultID.
    • VirtualResourceType [0..*] (String)
    • VirtualResourceID [0..*] (Identifier)
    • FaultType [0..*] (String)
    • MinSeverity [0..1] (Integer)
    • EventStartTime [0..1] (Datetime)
    • EventEndTime [0..1] (Datetime)
  • PhysicalResourceStateClass:
    • PhysicalResourceID [1] (Identifier)
    • PhysicalResourceState [1] (String): mandates the new state of the physical resource.
    • Metadata [0..*] (Key-value pair)
  • PhysicalResourceInfoClass:
    • PhysicalResourceID [1] (Identifier)
    • PhysicalResourceState [1] (String)
    • FirmwareVersion [0..1] (String)
    • HypervisorVersion [0..1] (String)
    • ZoneID [0..1] (Identifier)
    • Metadata [0..*] (Key-value pair)
  • StateQueryFilterClass: narrows down a StateQueryRequest, for example it limits the query to certain physical resources, a certain zone, or a given resource state (e.g., only resources in “maintenance” state).
    • PhysicalResourceID [1] (Identifier)
    • PhysicalResourceState [1] (String)
    • ZoneID [0..1] (Identifier)
Detailed northbound interface specification

This section is specifying the northbound interfaces for fault management and NFVI maintenance between the VIM on the one end and the Consumer and the Administrator on the other ends. For each interface all messages and related information elements are provided.

Note: The interface definition will be aligned with current work in ETSI NFV IFA working group .

All of the interfaces described below are produced by the VIM and consumed by the Consumer or Administrator.

Fault management interface

This interface allows the VIM to notify the Consumer about a virtual resource that is affected by a fault, either within the virtual resource itself or by the underlying virtualization infrastructure. The messages on this interface are shown in figure13 and explained in detail in the following subsections.

Note: The information elements used in this section are described in detail in Section 5.4.

_images/figure13.png

Fault management NB I/F messages

SubscribeRequest (Consumer -> VIM)

Subscription from Consumer to VIM to be notified about faults of specific resources. The faults to be notified about can be narrowed down using a subscribe filter.

Parameters:

  • SubscribeFilter [1] (SubscribeFilterClass): Optional information to narrow down the faults that shall be notified to the Consumer, for example limit to specific VirtualResourceID(s), severity, or cause of the alarm.
SubscribeResponse (VIM -> Consumer)

Response to a subscribe request message including information about the subscribed resources, in particular if they are in “fault/error” state.

Parameters:

  • SubscriptionID [1] (Identifier): Unique identifier for the subscription. It can be used to delete or update the subscription.
  • VirtualResourceInfo [0..*] (VirtualResourceInfoClass): Provides additional information about the subscribed resources, i.e., a list of the related resources, the current state of the resources, etc.
FaultNotification (VIM -> Consumer)

Notification about a virtual resource that is affected by a fault, either within the virtual resource itself or by the underlying virtualization infrastructure. After reception of this request, the Consumer will decide on the optimal action to resolve the fault. This includes actions like switching to a hot standby virtual resource, migration of the fault virtual resource to another physical machine, termination of the faulty virtual resource and instantiation of a new virtual resource in order to provide a new hot standby resource. In some use cases the Consumer can leave virtual resources on failed host to be booted up again after fault is recovered. Existing resource management interfaces and messages between the Consumer and the VIM can be used for those actions, and there is no need to define additional actions on the Fault Management Interface.

Parameters:

  • NotificationID [1] (Identifier): Unique identifier for the notification.
  • VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of faulty resources with detailed information about the faults.
FaultQueryRequest (Consumer -> VIM)

Request to find out about active alarms at the VIM. A FaultQueryFilter can be used to narrow down the alarms returned in the response message.

Parameters:

  • FaultQueryFilter [1] (FaultQueryFilterClass): narrows down the FaultQueryRequest, for example it limits the query to certain physical resources, a certain zone, a given fault type/severity/cause, or a specific FaultID.
FaultQueryResponse (VIM -> Consumer)

List of active alarms at the VIM matching the FaultQueryFilter specified in the FaultQueryRequest.

Parameters:

  • VirtualResourceInfo [0..*] (VirtualResourceInfoClass): List of faulty resources. For each resource all faults including detailed information about the faults are provided.
NFVI maintenance

The NFVI maintenance interfaces Consumer-VIM allows the Consumer to subscribe to maintenance notifications provided by the VIM. The related maintenance interface Administrator-VIM allows the Administrator to issue maintenance requests to the VIM, i.e. requesting the VIM to take appropriate actions to empty physical machine(s) in order to execute maintenance operations on them. The interface also allows the Administrator to query the state of physical machines, e.g., in order to get details in the current status of the maintenance operation like a firmware update.

The messages defined in these northbound interfaces are shown in figure14 and described in detail in the following subsections.

_images/figure14.png

NFVI maintenance NB I/F messages

SubscribeRequest (Consumer -> VIM)

Subscription from Consumer to VIM to be notified about maintenance operations for specific virtual resources. The resources to be informed about can be narrowed down using a subscribe filter.

Parameters:

  • SubscribeFilter [1] (SubscribeFilterClass): Information to narrow down the faults that shall be notified to the Consumer, for example limit to specific virtual resource type(s).
SubscribeResponse (VIM -> Consumer)

Response to a subscribe request message, including information about the subscribed virtual resources, in particular if they are in “maintenance” state.

Parameters:

  • SubscriptionID [1] (Identifier): Unique identifier for the subscription. It can be used to delete or update the subscription.
  • VirtualResourceInfo [0..*] (VirtalResourceInfoClass): Provides additional information about the subscribed virtual resource(s), e.g., the ID, type and current state of the resource(s).
MaintenanceNotification (VIM -> Consumer)

Notification about a physical resource switched to “maintenance” state. After reception of this request, the Consumer will decide on the optimal action to address this request, e.g., to switch to the standby (STBY) configuration.

Parameters:

  • VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of virtual resources where the state has been changed to maintenance.
StateChangeRequest (Administrator -> VIM)

Request to change the state of a list of physical resources, e.g. to “maintenance” state, in order to prepare them for a planned maintenance operation.

Parameters:

  • PhysicalResourceState [1..*] (PhysicalResourceStateClass)
StateChangeResponse (VIM -> Administrator)

Response message to inform the Administrator that the requested resources are now in maintenance state (or the operation resulted in an error) and the maintenance operation(s) can be executed.

Parameters:

  • PhysicalResourceInfo [1..*] (PhysicalResourceInfoClass)
StateQueryRequest (Administrator -> VIM)

In this procedure, the Administrator would like to get the information about physical machine(s), e.g. their state (“normal”, “maintenance”), firmware version, hypervisor version, update status of firmware and hypervisor, etc. It can be used to check the progress during firmware update and the confirmation after update. A filter can be used to narrow down the resources returned in the response message.

Parameters:

  • StateQueryFilter [1] (StateQueryFilterClass): narrows down the StateQueryRequest, for example it limits the query to certain physical resources, a certain zone, or a given resource state.
StateQueryResponse (VIM -> Administrator)

List of physical resources matching the filter specified in the StateQueryRequest.

Parameters:

  • PhysicalResourceInfo [0..*] (PhysicalResourceInfoClass): List of physical resources. For each resource, information about the current state, the firmware version, etc. is provided.
NFV IFA, OPNFV Doctor and AODH alarms

This section compares the alarm interfaces of ETSI NFV IFA with the specifications of this document and the alarm class of AODH.

ETSI NFV specifies an interface for alarms from virtualised resources in ETSI GS NFV-IFA 005 [ENFV]. The interface specifies an Alarm class and two notifications plus operations to query alarm instances and to subscribe to the alarm notifications.

The specification in this document has a structure that is very similar to the ETSI NFV specifications. The notifications differ in that an alarm notification in the NFV interface defines a single fault for a single resource while the notification specified in this document can contain multiple faults for multiple resources. The Doctor specification is lacking the detailed time stamps of the NFV specification essential for synchronizaion of the alarm list using the query operation. The detailed time stamps are also of value in the event and alarm history DBs.

AODH defines a base class for alarms, not the notifications. This means that some of the dynamic attributes of the ETSI NFV alarm type, like alarmRaisedTime, are not applicable to the AODH alarm class but are attributes of in the actual notifications. (Description of these attributes will be added later.) The AODH alarm class is lacking some attributes present in the NFV specification, fault details and correlated alarms. Instead the AODH alarm class has attributes for actions, rules and user and project id.

ETSI NFV Alarm Type OPNFV Doctor Requirement Specs AODH Event Alarm Notification Description / Comment Recommendations
alarmId FaultId alarm_id Identifier of an alarm. -
- - alarm_name Human readable alarm name. May be added in ETSI NFV Stage 3.
managedObjectId VirtualResourceId (reason) Identifier of the affected virtual resource is part of the AODH reason parameter. -
- - user_id, project_id User and project identifiers. May be added in ETSI NFV Stage 3.
alarmRaisedTime - - Timestamp when alarm was raised. To be added to Doctor and AODH. May be derived (e.g. in a shimlayer) from the AODH alarm history.
alarmChangedTime - - Timestamp when alarm was changed/updated. see above
alarmClearedTime - - Timestamp when alarm was cleared. see above
eventTime - - Timestamp when alarm was first observed by the Monitor. see above
- EventTime generated Timestamp of the Notification. Update parameter name in Doctor spec. May be added in ETSI NFV Stage 3.
state: E.g. Fired, Updated Cleared VirtualResourceState: E.g. normal, down maintenance, error current: ok, alarm, insufficient_data ETSI NFV IFA 005/006 lists example alarm states. Maintenance state is missing in AODH. List of alarm states will be specified in ETSI NFV Stage 3.
perceivedSeverity: E.g. Critical, Major, Minor, Warning, Indeterminate, Cleared Severity (Integer) Severity: low (default), moderate, critical ETSI NFV IFA 005/006 lists example perceived severity values.

List of alarm states will be specified in ETSI NFV Stage 3.

OPNFV: Severity (Integer):
  • update OPNFV Doctor specification to Enum
perceivedSeverity=Indetermined:
  • remove value Indetermined in IFA and map undefined values to “minor” severity, or
  • add value indetermined in AODH and make it the default value.
perceivedSeverity=Cleared:
  • remove value Cleared in IFA as the information about a cleared alarm alarm can be derived from the alarm state parameter, or
  • add value cleared in AODH and set a rule that the severity is “cleared” when the state is ok.
faultType FaultType event_type in reason_data Type of the fault, e.g. “CPU failure” of a compute resource, in machine interpretable format. OpenStack Alarming (Aodh) can use a fuzzy matching with wildcard string, “compute.cpu.failure”.
N/A N/A type = “event” Type of the notification. For fault notifications the type in AODH is “event”. -
probableCause ProbableCause - Probable cause of the alarm. May be provided (e.g. in a shimlayer) based on Vitrage topology awareness / root-cause-analysis.
isRootCause IsRootCause - Boolean indicating whether the fault is the root cause of other faults. see above
correlatedAlarmId CorrelatedFaultId - List of IDs of correlated faults. see above
faultDetails FaultDetails - Additional details about the fault/alarm. FaultDetails information element will be specified in ETSI NFV Stage 3.
- - action, previous Additional AODH alarm related parameters. -

Table: Comparison of alarm attributes

The primary area of improvement should be alignment of the perceived severity. This is important for a quick and accurate evaluation of the alarm. AODH thus should support also the X.733 values Critical, Major, Minor, Warning and Indeterminate.

The detailed time stamps (raised, changed, cleared) which are essential for synchronizing the alarm list using a query operation should be added to the Doctor specification.

Other areas that need alignment is the so called alarm state in NFV. Here we must however consider what can be attributes of the notification vs. what should be a property of the alarm instance. This will be analyzed later.

Detailed southbound interface specification

This section is specifying the southbound interfaces for fault management between the Monitors and the Inspector. Although southbound interfaces should be flexible to handle various events from different types of Monitors, we define unified event API in order to improve interoperability between the Monitors and the Inspector. This is not limiting implementation of Monitor and Inspector as these could be extended in order to support failures from intelligent inspection like prediction.

Note: The interface definition will be aligned with current work in ETSI NFV IFA working group.

Fault event interface

This interface allows the Monitors to notify the Inspector about an event which was captured by the Monitor and may effect resources managed in the VIM.

EventNotification

Event notification including fault description. The entity of this notification is event, and not fault or error specifically. This allows us to use generic event format or framework build out of Doctor project. The parameters below shall be mandatory, but keys in ‘Details’ can be optional.

Parameters:

  • Time [1]: Datetime when the fault was observed in the Monitor.
  • Type [1]: Type of event that will be used to process correlation in Inspector.
  • Details [0..1]: Details containing additional information with Key-value pair style. Keys shall be defined depending on the Type of the event.

E.g.:

{
    'event': {
        'time': '2016-04-12T08:00:00',
        'type': 'compute.host.down',
        'details': {
            'hostname': 'compute-1',
            'source': 'sample_monitor',
            'cause': 'link-down',
            'severity': 'critical',
            'status': 'down',
            'monitor_id': 'monitor-1',
            'monitor_event_id': '123',
        }
    }
}

Optional parameters in ‘Details’:

  • Hostname: the hostname on which the event occurred.
  • Source: the display name of reporter of this event. This is not limited to monitor, other entity can be specified such as ‘KVM’.
  • Cause: description of the cause of this event which could be different from the type of this event.
  • Severity: the severity of this event set by the monitor.
  • Status: the status of target object in which error occurred.
  • MonitorID: the ID of the monitor sending this event.
  • MonitorEventID: the ID of the event in the monitor. This can be used by operator while tracking the monitor log.
  • RelatedTo: the array of IDs which related to this event.

Also, we can have bulk API to receive multiple events in a single HTTP POST message by using the ‘events’ wrapper as follows:

{
    'events': [
        'event': {
            'time': '2016-04-12T08:00:00',
            'type': 'compute.host.down',
            'details': {},
        },
        'event': {
            'time': '2016-04-12T08:00:00',
            'type': 'compute.host.nic.error',
            'details': {},
        }
    ]
}
Blueprints

This section is listing a first set of blueprints that have been proposed by the Doctor project to the open source community. Further blueprints addressing other gaps identified in Section 4 will be submitted at a later stage of the OPNFV. In this section the following definitions are used:

  • “Event” is a message emitted by other OpenStack services such as Nova and Neutron and is consumed by the “Notification Agents” in Ceilometer.
  • “Notification” is a message generated by a “Notification Agent” in Ceilometer based on an “event” and is delivered to the “Collectors” in Ceilometer that store those notifications (as “sample”) to the Ceilometer “Databases”.
Instance State Notification (Ceilometer) [†]

The Doctor project is planning to handle “events” and “notifications” regarding Resource Status; Instance State, Port State, Host State, etc. Currently, Ceilometer already receives “events” to identify the state of those resources, but it does not handle and store them yet. This is why we also need a new event definition to capture those resource states from “events” created by other services.

This BP proposes to add a new compute notification state to handle events from an instance (server) from nova. It also creates a new meter “instance.state” in OpenStack.

[†]https://etherpad.opnfv.org/p/doctor_bps
Event Publisher for Alarm (Ceilometer) [‡]

Problem statement:

The existing “Alarm Evaluator” in OpenStack Ceilometer is periodically querying/polling the databases in order to check all alarms independently from other processes. This is adding additional delay to the fault notification send to the Consumer, whereas one requirement of Doctor is to react on faults as fast as possible.

The existing message flow is shown in figure12: after receiving an “event”, a “notification agent” (i.e. “event publisher”) will send a “notification” to a “Collector”. The “collector” is collecting the notifications and is updating the Ceilometer “Meter” database that is storing information about the “sample” which is capured from original “event”. The “Alarm Evaluator” is periodically polling this databases then querying “Meter” database based on each alarm configuration.

_images/figure12.png

Implementation plan in Ceilometer architecture

In the current Ceilometer implementation, there is no possibility to directly trigger the “Alarm Evaluator” when a new “event” was received, but the “Alarm Evaluator” will only find out that requires firing new notification to the Consumer when polling the database.

Change/feature request:

This BP proposes to add a new “event publisher for alarm”, which is bypassing several steps in Ceilometer in order to avoid the polling-based approach of the existing Alarm Evaluator that makes notification slow to users. See figure12.

After receiving an “(alarm) event” by listening on the Ceilometer message queue (“notification bus”), the new “event publisher for alarm” immediately hands a “notification” about this event to a new Ceilometer component “Notification-driven alarm evaluator” proposed in the other BP (see Section 5.6.3).

Note, the term “publisher” refers to an entity in the Ceilometer architecture (it is a “notification agent”). It offers the capability to provide notifications to other services outside of Ceilometer, but it is also used to deliver notifications to other Ceilometer components (e.g. the “Collectors”) via the Ceilometer “notification bus”.

Implementation detail

  • “Event publisher for alarm” is part of Ceilometer
  • The standard AMQP message queue is used with a new topic string.
  • No new interfaces have to be added to Ceilometer.
  • “Event publisher for Alarm” can be configured by the Administrator of Ceilometer to be used as “Notification Agent” in addition to the existing “Notifier”
  • Existing alarm mechanisms of Ceilometer can be used allowing users to configure how to distribute the “notifications” transformed from “events”, e.g. there is an option whether an ongoing alarm is re-issued or not (“repeat_actions”).
[‡]https://etherpad.opnfv.org/p/doctor_bps
Notification-driven alarm evaluator (Ceilometer) [§]

Problem statement:

The existing “Alarm Evaluator” in OpenStack Ceilometer is periodically querying/polling the databases in order to check all alarms independently from other processes. This is adding additional delay to the fault notification send to the Consumer, whereas one requirement of Doctor is to react on faults as fast as possible.

Change/feature request:

This BP is proposing to add an alternative “Notification-driven Alarm Evaluator” for Ceilometer that is receiving “notifications” sent by the “Event Publisher for Alarm” described in the other BP. Once this new “Notification-driven Alarm Evaluator” received “notification”, it finds the “alarm” configurations which may relate to the “notification” by querying the “alarm” database with some keys i.e. resource ID, then it will evaluate each alarm with the information in that “notification”.

After the alarm evaluation, it will perform the same way as the existing “alarm evaluator” does for firing alarm notification to the Consumer. Similar to the existing Alarm Evaluator, this new “Notification-driven Alarm Evaluator” is aggregating and correlating different alarms which are then provided northbound to the Consumer via the OpenStack “Alarm Notifier”. The user/administrator can register the alarm configuration via existing Ceilometer API [¶]. Thereby, he can configure whether to set an alarm or not and where to send the alarms to.

Implementation detail

  • The new “Notification-driven Alarm Evaluator” is part of Ceilometer.
  • Most of the existing source code of the “Alarm Evaluator” can be re-used to implement this BP
  • No additional application logic is needed
  • It will access the Ceilometer Databases just like the existing “Alarm evaluator”
  • Only the polling-based approach will be replaced by a listener for “notifications” provided by the “Event Publisher for Alarm” on the Ceilometer “notification bus”.
  • No new interfaces have to be added to Ceilometer.
[§]https://etherpad.opnfv.org/p/doctor_bps
[¶]https://wiki.openstack.org/wiki/Ceilometer/Alerting
Report host fault to update server state immediately (Nova) [#]

Problem statement:

  • Nova state change for failed or unreachable host is slow and does not reliably state host is down or not. This might cause same server instance to run twice if action taken to evacuate instance to another host.
  • Nova state for server(s) on failed host will not change, but remains active and running. This gives the user false information about server state.
  • VIM northbound interface notification of host faults towards VNFM and NFVO should be in line with OpenStack state. This fault notification is a Telco requirement defined in ETSI and will be implemented by OPNFV Doctor project.
  • Openstack user cannot make HA actions fast and reliably by trusting server state and host state.

Proposed change:

There needs to be a new API for Admin to state host is down. This API is used to mark services running in host down to reflect the real situation.

Example on compute node is:

  • When compute node is up and running::

    vm_state: activeand power_state: running
    nova-compute state: up status: enabled
    
  • When compute node goes down and new API is called to state host is down::

    vm_state: stopped power_state: shutdown
    nova-compute state: down status: enabled
    

Alternatives:

There is no attractive alternative to detect all different host faults than to have an external tool to detect different host faults. For this kind of tool to exist there needs to be new API in Nova to report fault. Currently there must be some kind of workarounds implemented as cannot trust or get the states from OpenStack fast enough.

[#]https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately
Summary and conclusion

The Doctor project aimed at detailing NFVI fault management and NFVI maintenance requirements. These are indispensable operations for an Operator, and extremely necessary to realize telco-grade high availability. High availability is a large topic; the objective of Doctor is not to realize a complete high availability architecture and implementation. Instead, Doctor limited itself to addressing the fault events in NFVI, and proposes enhancements necessary in VIM, e.g. OpenStack, to ensure VNFs availability in such fault events, taking a Telco VNFs application level management system into account.

The Doctor project performed a robust analysis of the requirements from NFVI fault management and NFVI maintenance operation, concretely found out gaps in between such requirements and the current implementation of OpenStack. A detailed architecture and interface specification has been described in this document and work to realize Doctor features and fill out the identified gaps in upstream communities is in the final stages of development.

Annex: NFVI Faults

Faults in the listed elements need to be immediately notified to the Consumer in order to perform an immediate action like live migration or switch to a hot standby entity. In addition, the Administrator of the host should trigger a maintenance action to, e.g., reboot the server or replace a defective hardware element.

Faults can be of different severity, i.e., critical, warning, or info. Critical faults require immediate action as a severe degradation of the system has happened or is expected. Warnings indicate that the system performance is going down: related actions include closer (e.g. more frequent) monitoring of that part of the system or preparation for a cold migration to a backup VM. Info messages do not require any action. We also consider a type “maintenance”, which is no real fault, but may trigger maintenance actions like a re-boot of the server or replacement of a faulty, but redundant HW.

Faults can be gathered by, e.g., enabling SNMP and installing some open source tools to catch and poll SNMP. When using for example Zabbix one can also put an agent running on the hosts to catch any other fault. In any case of failure, the Administrator should be notified. The following tables provide a list of high level faults that are considered within the scope of the Doctor project requiring immediate action by the Consumer.

Compute/Storage

Fault Severity How to detect? Comment Immediate action to recover
Processor/CPU failure, CPU condition not ok Critical Zabbix   Switch to hot standby
Memory failure/ Memory condition not ok Critical Zabbix (IPMI)   Switch to hot standby
Network card failure, e.g. network adapter connectivity lost Critical Zabbix/ Ceilometer   Switch to hot standby
Disk crash Info RAID monitoring Network storage is very redundant (e.g. RAID system) and can guarantee high availability Inform OAM
Storage controller Critical Zabbix (IPMI)   Live migration if storage is still accessible; otherwise hot standby
PDU/power failure, power off, server reset Critical Zabbix/ Ceilometer   Switch to hot standby
Power degration, power redundancy lost, power threshold exceeded Warning SNMP   Live migration
Chassis problem (e.g. fan degraded/failed, chassis power degraded), CPU fan problem, temperature/ thermal condition not ok Warning SNMP   Live migration
Mainboard failure Critical Zabbix (IPMI) e.g. PCIe, SAS link failure Switch to hot standby
OS crash (e.g. kernel panic) Critical Zabbix   Switch to hot standby

Hypervisor

Fault Severity How to detect? Comment Immediate action to recover
System has restarted Critical Zabbix   Switch to hot standby
Hypervisor failure Warning/ Critical Zabbix/ Ceilometer   Evacuation/switch to hot standby
Hypervisor status not retrievable after certain period Warning Alarming service Zabbix/ Ceilometer unreachable Rebuild VM

Network

Fault Severity How to detect? Comment Immediate action to recover
SDN/OpenFlow switch, controller degraded/failed Critical Ceilo- meter   Switch to hot standby or reconfigure virtual network topology
Hardware failure of physical switch/router Warning SNMP Redundancy of physical infrastructure is reduced or no longer available Live migration if possible otherwise evacuation
References and bibliography
[DOCT]OPNFV, “Doctor” requirements project, [Online]. Available at https://wiki.opnfv.org/doctor
[PRED]OPNFV, “Data Collection for Failure Prediction” requirements project [Online]. Available at https://wiki.opnfv.org/prediction
[OPSK]OpenStack, [Online]. Available at https://www.openstack.org/
[CEIL]OpenStack Telemetry (Ceilometer), [Online]. Available at https://wiki.openstack.org/wiki/Ceilometer
[NOVA]OpenStack Nova, [Online]. Available at https://wiki.openstack.org/wiki/Nova
[NEUT]OpenStack Neutron, [Online]. Available at https://wiki.openstack.org/wiki/Neutron
[CIND]OpenStack Cinder, [Online]. Available at https://wiki.openstack.org/wiki/Cinder
[MONA]OpenStack Monasca, [Online], Available at https://wiki.openstack.org/wiki/Monasca
[OSAG]OpenStack Cloud Administrator Guide, [Online]. Available at http://docs.openstack.org/admin-guide-cloud/content/
[ZABB]ZABBIX, the Enterprise-class Monitoring Solution for Everyone, [Online]. Available at http://www.zabbix.com/
[ENFV]ETSI NFV, [Online]. Available at http://www.etsi.org/technologies-clusters/technologies/nfv
Doctor Installation Guide
Doctor Configuration

OPNFV installers install most components of Doctor framework including OpenStack Nova, Neutron and Cinder (Doctor Controller) and OpenStack Ceilometer and Aodh (Doctor Notifier) except Doctor Monitor.

After major components of OPNFV are deployed, you can setup Doctor functions by following instructions in this section. You can also learn detailed steps in setup_installer() under doctor/tests.

Doctor Inspector

You need to configure one of Doctor Inspector below.

Doctor Sample Inspector

Sample Inspector is intended to show minimum functions of Doctor Inspector.

Doctor Sample Inspector suggested to be placed in one of the controller nodes, but it can be put on any host where Doctor Monitor can reach and access the OpenStack Controller (Nova).

Make sure OpenStack env parameters are set properly, so that Doctor Inspector can issue admin actions such as compute host force-down and state update of VM.

Then, you can configure Doctor Inspector as follows:

git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
cd doctor/tests
INSPECTOR_PORT=12345
python inspector.py $INSPECTOR_PORT > inspector.log 2>&1 &

Congress

OpenStack Congress is a Governance as a Service (previously Policy as a Service). Congress implements Doctor Inspector as it can inspect a fault situation and propagate errors onto other entities.

Congress is deployed by OPNFV installers. You need to enable doctor datasource driver and set policy rules. By the example configuration below, Congress will force down nova compute service when it received a fault event of that compute host. Also, Congress will set the state of all VMs running on that host from ACTIVE to ERROR state.

openstack congress datasource create doctor doctor

openstack congress policy rule create \
    --name host_down classification \
    'host_down(host) :-
        doctor:events(hostname=host, type="compute.host.down", status="down")'

openstack congress policy rule create \
    --name active_instance_in_host classification \
    'active_instance_in_host(vmid, host) :-
        nova:servers(id=vmid, host_name=host, status="ACTIVE")'

openstack congress policy rule create \
    --name host_force_down classification \
    'execute[nova:services.force_down(host, "nova-compute", "True")] :-
        host_down(host)'

openstack congress policy rule create \
    --name error_vm_states classification \
    'execute[nova:servers.reset_state(vmid, "error")] :-
        host_down(host),
        active_instance_in_host(vmid, host)'

Vitrage

OpenStack Vitrage is an RCA (Root Cause Analysis) service for organizing, analyzing and expanding OpenStack alarms & events. Vitrage implements Doctor Inspector, as it receives a notification that a host is down and calls Nova force-down API. In addition, it raises alarms on the instances running on this host.

Vitrage is not deployed by OPNFV installers yet. It can be installed either on top of a devstack environment, or on top of a real OpenStack environment. See Vitrage Installation

Doctor SB API and a Doctor datasource were implemented in Vitrage in the Ocata release. The Doctor datasource is enabled by default.

After Vitrage is installed and configured, there is a need to configure it to support the Doctor use case. This can be done in a few steps:

  1. Make sure that ‘aodh’ and ‘doctor’ are included in the list of datasource types in /etc/vitrage/vitrage.conf:
[datasources]
types = aodh,doctor,nova.host,nova.instance,nova.zone,static,cinder.volume,neutron.network,neutron.port,heat.stack
  1. Enable the Vitrage Nova notifier. Set the following line in /etc/vitrage/vitrage.conf:
[DEFAULT]
notifiers = nova
  1. Add a template that is responsible to call Nova force-down if Vitrage receives a ‘compute.host.down’ alarm. Copy template and place it under /etc/vitrage/templates
  1. Restart the vitrage-graph and vitrage-notifier services
Doctor Monitor

Doctor Sample Monitor

Doctor Monitors are suggested to be placed in one of the controller nodes, but those can be put on any host which is reachable to target compute host and accessible by the Doctor Inspector. You need to configure Monitors for all compute hosts one by one.

Make sure OpenStack env parameters are set properly, so that Doctor Inspector can issue admin actions such as compute host force-down and state update of VM.

Then, you can configure the Doctor Monitor as follows (Example for Apex deployment):

git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
cd doctor/tests
INSPECTOR_PORT=12345
COMPUTE_HOST='overcloud-novacompute-1.localdomain.com'
COMPUTE_IP=192.30.9.5
sudo python monitor.py "$COMPUTE_HOST" "$COMPUTE_IP" \
    "http://127.0.0.1:$INSPECTOR_PORT/events" > monitor.log 2>&1 &
Doctor User Guide
Doctor capabilities and usage

figure1 shows the currently implemented and tested architecture of Doctor. The implementation is based on OpenStack and related components. The Monitor can be realized by a sample Python-based implementation provided in the Doctor code repository. The Controller is realized by OpenStack Nova, Neutron and Cinder for compute, network and storage, respectively. The Inspector can be realized by OpenStack Congress or a sample Python-based implementation also available in the code repository of Doctor. The Notifier is realized by OpenStack Aodh.

_images/figure11.png

Implemented and tested architecture

Immediate Notification

Immediate notification can be used by creating ‘event’ type alarm via OpenStack Alarming (Aodh) API with relevant internal components support.

See, upstream spec document: http://specs.openstack.org/openstack/ceilometer-specs/specs/liberty/event-alarm-evaluator.html

An example of a consumer of this notification can be found in the Doctor repository. It can be executed as follows:

git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
cd doctor/tests
CONSUMER_PORT=12346
python consumer.py "$CONSUMER_PORT" > consumer.log 2>&1 &
Consistent resource state awareness

Resource state of compute host can be changed/updated according to a trigger from a monitor running outside of OpenStack Compute (Nova) by using force-down API.

See http://artifacts.opnfv.org/doctor/danube/manuals/mark-host-down_manual.html for more detail.

Valid compute host status given to VM owner

The resource state of a compute host can be retrieved by a user with the OpenStack Compute (Nova) servers API.

See http://artifacts.opnfv.org/doctor/danube/manuals/get-valid-server-state.html for more detail.

Design Documents

This is the directory to store design documents which may include draft versions of blueprints written before proposing to upstream OSS communities such as OpenStack, in order to keep the original blueprint as reviewed in OPNFV. That means there could be out-dated blueprints as result of further refinements in the upstream OSS community. Please refer to the link in each document to find the latest version of the blueprint and status of development in the relevant OSS community.

See also https://wiki.opnfv.org/requirements_projects .

Note

This is a specification draft of a blueprint proposed for OpenStack Nova Liberty. It was written by project member(s) and agreed within the project before submitting it upstream. No further changes to its content will be made here anymore; please follow it upstream:

Original draft is as follow:

Report host fault to update server state immediately

https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately

A new API is needed to report a host fault to change the state of the instances and compute node immediately. This allows usage of evacuate API without a delay. The new API provides the possibility for external monitoring system to detect any kind of host failure fast and reliably and inform OpenStack about it. Nova updates the compute node state and states of the instances. This way the states in the Nova DB will be in sync with the real state of the system.

Problem description
  • Nova state change for failed or unreachable host is slow and does not reliably state compute node is down or not. This might cause same instance to run twice if action taken to evacuate instance to another host.
  • Nova state for instances on failed compute node will not change, but remains active and running. This gives user a false information about instance state. Currently one would need to call “nova reset-state” for each instance to have them in error state.
  • OpenStack user cannot make HA actions fast and reliably by trusting instance state and compute node state.
  • As compute node state changes slowly one cannot evacuate instances.
Use Cases

Use case in general is that in case there is a host fault one should change compute node state fast and reliably when using DB servicegroup backend. On top of this here is the use cases that are not covered currently to have instance states changed correctly: * Management network connectivity lost between controller and compute node. * Host HW failed.

Generic use case flow:

  • The external monitoring system detects a host fault.
  • The external monitoring system fences the host if not down already.
  • The external system calls the new Nova API to force the failed compute node into down state as well as instances running on it.
  • Nova updates the compute node state and state of the effected instances to Nova DB.

Currently nova-compute state will be changing “down”, but it takes a long time. Server state keeps as “vm_state: active” and “power_state: running”, which is not correct. By having external tool to detect host faults fast, fence host by powering down and then report host down to OpenStack, all these states would reflect to actual situation. Also if OpenStack will not implement automatic actions for fault correlation, external tool can do that. This could be configured for example in server instance METADATA easily and be read by external tool.

Project Priority

Liberty priorities have not yet been defined.

Proposed change

There needs to be a new API for Admin to state host is down. This API is used to mark compute node and instances running on it down to reflect the real situation.

Example on compute node is:

  • When compute node is up and running: vm_state: active and power_state: running nova-compute state: up status: enabled
  • When compute node goes down and new API is called to state host is down: vm_state: stopped power_state: shutdown nova-compute state: down status: enabled

vm_state values: soft-delete, deleted, resized and error should not be touched. task_state effect needs to be worked out if needs to be touched.

Alternatives

There is no attractive alternatives to detect all different host faults than to have a external tool to detect different host faults. For this kind of tool to exist there needs to be new API in Nova to report fault. Currently there must have been some kind of workarounds implemented as cannot trust or get the states from OpenStack fast enough.

Data model impact

None

REST API impact
  • Update CLI to report host is down

    nova host-update command

    usage: nova host-update [–status <enable|disable>]

    [–maintenance <enable|disable>] [–report-host-down] <hostname>

    Update host settings.

    Positional arguments

    <hostname> Name of host.

    Optional arguments

    –status <enable|disable> Either enable or disable a host.

    –maintenance <enable|disable> Either put or resume host to/from maintenance.

    –down Report host down to update instance and compute node state in db.

  • Update Compute API to report host is down:

    /v2.1/{tenant_id}/os-hosts/{host_name}

    Normal response codes: 200 Request parameters

    Parameter Style Type Description host_name URI xsd:string The name of the host of interest to you.

    {
    “host”: {

    “status”: “enable”, “maintenance_mode”: “enable” “host_down_reported”: “true”

    }

    }

    {
    “host”: {

    “host”: “65c5d5b7e3bd44308e67fc50f362aee6”, “maintenance_mode”: “enabled”, “status”: “enabled” “host_down_reported”: “true”

    }

    }

  • New method to nova.compute.api module HostAPI class to have a to mark host related instances and compute node down: set_host_down(context, host_name)

  • class novaclient.v2.hosts.HostManager(api) method update(host, values) Needs to handle reporting host down.

  • Schema does not need changes as in db only service and server states are to be changed.

Security impact

API call needs admin privileges (in the default policy configuration).

Notifications impact

None

Other end user impact

None

Performance Impact

Only impact is that user can get information faster about instance and compute node state. This also gives possibility to evacuate faster. No impact that would slow down. Host down should be rare occurrence.

Other deployer impact

Developer can make use of any external tool to detect host fault and report it to OpenStack.

Developer impact

None

Implementation
Assignee(s)

Primary assignee: Tomi Juvonen Other contributors: Ryota Mibu

Work Items
  • Test cases.
  • API changes.
  • Documentation.
Dependencies

None

Testing

Test cases that exists for enabling or putting host to maintenance should be altered or similar new cases made test new functionality.

Documentation Impact

New API needs to be documented:

References
Notification Alarm Evaluator

Note

This is spec draft of blueprint for OpenStack Ceilomter Liberty. To see current version: https://review.openstack.org/172893 To track development activity: https://blueprints.launchpad.net/ceilometer/+spec/notification-alarm-evaluator

https://blueprints.launchpad.net/ceilometer/+spec/notification-alarm-evaluator

This blueprint proposes to add a new alarm evaluator for handling alarms on events passed from other OpenStack services, that provides event-driven alarm evaluation which makes new sequence in Ceilometer instead of the polling-based approach of the existing Alarm Evaluator, and realizes immediate alarm notification to end users.

Problem description

As an end user, I need to receive alarm notification immediately once Ceilometer captured an event which would make alarm fired, so that I can perform recovery actions promptly to shorten downtime of my service. The typical use case is that an end user set alarm on “compute.instance.update” in order to trigger recovery actions once the instance status has changed to ‘shutdown’ or ‘error’. It should be nice that an end user can receive notification within 1 second after fault observed as the same as other helth- check mechanisms can do in some cases.

The existing Alarm Evaluator is periodically querying/polling the databases in order to check all alarms independently from other processes. This is good approach for evaluating an alarm on samples stored in a certain period. However, this is not efficient to evaluate an alarm on events which are emitted by other OpenStack servers once in a while.

The periodical evaluation leads delay on sending alarm notification to users. The default period of evaluation cycle is 60 seconds. It is recommended that an operator set longer interval than configured pipeline interval for underlying metrics, and also longer enough to evaluate all defined alarms in certain period while taking into account the number of resources, users and alarms.

Proposed change

The proposal is to add a new event-driven alarm evaluator which receives messages from Notification Agent and finds related Alarms, then evaluates each alarms;

  • New alarm evaluator could receive event notification from Notification Agent by which adding a dedicated notifier as a publisher in pipeline.yaml (e.g. notifier://?topic=event_eval).
  • When new alarm evaluator received event notification, it queries alarm database by Project ID and Resource ID written in the event notification.
  • Found alarms are evaluated by referring event notification.
  • Depending on the result of evaluation, those alarms would be fired through Alarm Notifier as the same as existing Alarm Evaluator does.

This proposal also adds new alarm type “notification” and “notification_rule”. This enables users to create alarms on events. The separation from other alarm types (such as “threshold” type) is intended to show different timing of evaluation and different format of condition, since the new evaluator will check each event notification once it received whereas “threshold” alarm can evaluate average of values in certain period calculated from multiple samples.

The new alarm evaluator handles Notification type alarms, so we have to change existing alarm evaluator to exclude “notification” type alarms from evaluation targets.

Alternatives

There was similar blueprint proposal “Alarm type based on notification”, but the approach is different. The old proposal was to adding new step (alarm evaluations) in Notification Agent every time it received event from other OpenStack services, whereas this proposal intends to execute alarm evaluation in another component which can minimize impact to existing pipeline processing.

Another approach is enhancement of existing alarm evaluator by adding notification listener. However, there are two issues; 1) this approach could cause stall of periodical evaluations when it receives bulk of notifications, and 2) this could break the alarm portioning i.e. when alarm evaluator received notification, it might have to evaluate some alarms which are not assign to it.

Data model impact

Resource ID will be added to Alarm model as an optional attribute. This would help the new alarm evaluator to filter out non-related alarms while querying alarms, otherwise it have to evaluate all alarms in the project.

REST API impact

Alarm API will be extended as follows;

  • Add “notification” type into alarm type list
  • Add “resource_id” to “alarm”
  • Add “notification_rule” to “alarm”

Sample data of Notification-type alarm:

{
    "alarm_actions": [
        "http://site:8000/alarm"
    ],
    "alarm_id": null,
    "description": "An alarm",
    "enabled": true,
    "insufficient_data_actions": [
        "http://site:8000/nodata"
    ],
    "name": "InstanceStatusAlarm",
    "notification_rule": {
        "event_type": "compute.instance.update",
        "query" : [
            {
                "field" : "traits.state",
                "type" : "string",
                "value" : "error",
                "op" : "eq",
            },
        ]
    },
    "ok_actions": [],
    "project_id": "c96c887c216949acbdfbd8b494863567",
    "repeat_actions": false,
    "resource_id": "153462d0-a9b8-4b5b-8175-9e4b05e9b856",
    "severity": "moderate",
    "state": "ok",
    "state_timestamp": "2015-04-03T17:49:38.406845",
    "timestamp": "2015-04-03T17:49:38.406839",
    "type": "notification",
    "user_id": "c96c887c216949acbdfbd8b494863567"
}

“resource_id” will be refered to query alarm and will not be check permission and belonging of project.

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

When Ceilomter received a number of events from other OpenStack services in short period, this alarm evaluator can keep working since events are queued in a messaging queue system, but it can cause delay of alarm notification to users and increase the number of read and write access to alarm database.

“resource_id” can be optional, but restricting it to mandatory could be reduce performance impact. If user create “notification” alarm without “resource_id”, those alarms will be evaluated every time event occurred in the project. That may lead new evaluator heavy.

Other deployer impact

New service process have to be run.

Developer impact

Developers should be aware that events could be notified to end users and avoid passing raw infra information to end users, while defining events and traits.

Implementation
Assignee(s)
Primary assignee:
r-mibu
Other contributors:
None
Ongoing maintainer:
None
Work Items
  • New event-driven alarm evaluator
  • Add new alarm type “notification” as well as AlarmNotificationRule
  • Add “resource_id” to Alarm model
  • Modify existing alarm evaluator to filter out “notification” alarms
  • Add new config parameter for alarm request check whether accepting alarms without specifying “resource_id” or not
Future lifecycle

This proposal is key feature to provide information of cloud resources to end users in real-time that enables efficient integration with user-side manager or Orchestrator, whereas currently those information are considered to be consumed by admin side tool or service. Based on this change, we will seek orchestrating scenarios including fault recovery and add useful event definition as well as additional traits.

Dependencies

None

Testing

New unit/scenario tests are required for this change.

Documentation Impact
  • Proposed evaluator will be described in the developer document.
  • New alarm type and how to use will be explained in user guide.
References
Neutron Port Status Update

Note

This document represents a Neutron RFE reviewed in the Doctor project before submitting upstream to Launchpad Neutron space. The document is not intended to follow a blueprint format or to be an extensive document. For more information, please visit http://docs.openstack.org/developer/neutron/policies/blueprints.html

The RFE was submitted to Neutron. You can follow the discussions in https://bugs.launchpad.net/neutron/+bug/1598081

Neutron port status field represents the current status of a port in the cloud infrastructure. The field can take one of the following values: ‘ACTIVE’, ‘DOWN’, ‘BUILD’ and ‘ERROR’.

At present, if a network event occurs in the data-plane (e.g. virtual or physical switch fails or one of its ports, cable gets pulled unintentionally, infrastructure topology changes, etc.), connectivity to logical ports may be affected and tenants’ services interrupted. When tenants/cloud administrators are looking up their resources’ status (e.g. Nova instances and services running in them, network ports, etc.), they will wrongly see everything looks fine. The problem is that Neutron will continue reporting port ‘status’ as ‘ACTIVE’.

Many SDN Controllers managing network elements have the ability to detect and report network events to upper layers. This allows SDN Controllers’ users to be notified of changes and react accordingly. Such information could be consumed by Neutron so that Neutron could update the ‘status’ field of those logical ports, and additionally generate a notification message to the message bus.

However, Neutron misses a way to be able to receive such information through e.g. ML2 driver or the REST API (‘status’ field is read-only). There are pros and cons on both of these approaches as well as other possible approaches. This RFE intends to trigger a discussion on how Neutron could be improved to receive fault/change events from SDN Controllers or even also from 3rd parties not in charge of controlling the network (e.g. monitoring systems, human admins).

Port data plane status

https://bugs.launchpad.net/neutron/+bug/1598081

Neutron does not detect data plane failures affecting its logical resources. This spec addresses that issue by means of allowing external tools to report to Neutron about faults in the data plane that are affecting the ports. A new REST API field is proposed to that end.

Problem Description

An initial description of the problem was introduced in bug #159801 [1]. This spec focuses on capturing one (main) part of the problem there described, i.e. extending Neutron’s REST API to cover the scenario of allowing external tools to report network failures to Neutron. Out of scope of this spec are works to enable port status changes to be received and managed by mechanism drivers.

This spec also tries to address bug #1575146 [2]. Specifically, and argued by the Neutron driver team in [3]:

  • Neutron should not shut down the port completly upon detection of physnet failure; connectivity between instances on the same node may still be reachable. Externals tools may or may not want to trigger a status change on the port based on their own logic and orchestration.
  • Port down is not detected when an uplink of a switch is down;
  • The physnet bridge may have multiple physical interfaces plugged; shutting down the logical port may not be needed in case network redundancy is in place.
Proposed Change

A couple of possible approaches were proposed in [1] (comment #3). This spec proposes tackling the problema via a new extension API to the port resource. The extension adds a new attribute ‘dp-down’ (data plane down) to represent the status of the data plane. The field should be read-only by tenants and read-write by admins.

Neutron should send out an event to the message bus upon toggling the data plane status value. The event is relevant for e.g. auditing.

Data Model Impact

A new attribute as extension will be added to the ‘ports’ table.

Attribute Name Type Access Default Value Validation/ Conversion Description
dp_down boolean RO, tenant RW, admin False True/False  
REST API Impact

A new API extension to the ports resource is going to be introduced.

EXTENDED_ATTRIBUTES_2_0 = {
    'ports': {
        'dp_down': {'allow_post': False, 'allow_put': True,
                    'default': False, 'convert_to': convert_to_boolean,
                    'is_visible': True},
    },
}
Examples

Updating port data plane status to down:

PUT /v2.0/ports/<port-uuid>
Accept: application/json
{
    "port": {
        "dp_down": true
    }
}
Command Line Client Impact
neutron port-update [--dp-down <True/False>] <port>
openstack port set [--dp-down <True/False>] <port>

Argument –dp-down is optional. Defaults to False.

Security Impact

None

Notifications Impact

A notification (event) upon toggling the data plane status (i.e. ‘dp-down’ attribute) value should be sent to the message bus. Such events do not happen with high frequency and thus no negative impact on the notification bus is expected.

Performance Impact

None

IPv6 Impact

None

Other Deployer Impact

None

Developer Impact

None

Implementation
Assignee(s)
  • cgoncalves
Work Items
  • New ‘dp-down’ attribute in ‘ports’ database table
  • API extension to introduce new field to port
  • Client changes to allow for data plane status (i.e. ‘dp-down’ attribute’) being set
  • Policy (tenants read-only; admins read-write)
Documentation Impact

Documentation for both administrators and end users will have to be contemplated. Administrators will need to know how to set/unset the data plane status field.

References
[1]RFE: Port status update, https://bugs.launchpad.net/neutron/+bug/1598081
[2]RFE: ovs port status should the same as physnet https://bugs.launchpad.net/neutron/+bug/1575146
[3]Neutron Drivers meeting, July 21, 2016 http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-07-21-22.00.html
Inspector Design Guideline

Note

This is spec draft of design guideline for inspector component. JIRA ticket to track the update and collect comments: DOCTOR-73.

This document summarize the best practise in designing a high performance inspector to meet the requirements in OPNFV Doctor project.

Problem Description

Some pitfalls has be detected during the development of sample inspector, e.g. we suffered a significant performance degrading in listing VMs in a host.

A patch set for caching the list has been committed to solve issue. When a new inspector is integrated, it would be nice to have an evaluation of existing design and give recommendations for improvements.

This document can be treated as a source of related blueprints in inspector projects.

Guidelines
Host specific VMs list

While requirement in doctor project is to have alarm about fault to consumer in one second, it is just a limit we have set in requirements. When talking about fault management in Telco, the implementation needs to be by all means optimal and the one second is far from traditional Telco requirements.

One thing to be optimized in inspector is to eliminate the need to read list of host specific VMs from Nova API, when it gets a host specific failure event. Optimal way of implementation would be to initialize this list when Inspector start by reading from Nova API and after this list would be kept up-to-date by instance.update notifications received from nova. Polling Nova API can be used as a complementary channel to make snapshot of hosts and VMs list in order to keep the data consistent with reality.

This is enhancement and not perhaps something needed to keep under one second in a small system. Anyhow this would be something needed in case of production use.

This guideline can be summarized as following:

  • cache the host VMs mapping instead of reading it on request
  • subscribe and handle update notifications to keep the list up to date
  • make snapshot periodically to ensure data consistency
Parallel execution

In doctor’s architecture, the inspector is responsible to set error state for the affected VMs in order to notify the consumers of such failure. This is done by calling the nova reset-state API. However, this action is a synchronous request with many underlying steps and cost typically hundreds of milliseconds. According to the discussion in mailing list, this time cost will grow linearly if the requests are sent one by one. It will become a critical issue in large scale system.

It is recommended to introduce parallel execution for actions like reset-state that takes a list of targets.

Shortcut notification

An alternative way to improve notification performance is to take a shortcut from inspector to notifier instead of triggering it from controller. The difference between the two workflow is shown below:

conservative notification

Conservative Notification

shortcut notification

Shortcut Notification

It worth noting that the shortcut notification has a side effect that cloud resource states could still be out-of-sync by the time consumer processes the alarm notification. This is out of scope of inspector design but need to be taken consideration in system level.

Also the call of “reset servers state to error” is not necessary in the alternative notification case where the “host forced down” is still called. “get-valid-server-state” was implemented to have valid server state while earlier one couldn’t get it unless calling “reset servers state to error”. When not having “reset servers state to error”, states are more unlikely to be out of sync while notification and force down host would be parallel.

Appendix

A study has been made to evaluate the effect of parallel execution and shortcut notification on OPNFV Beijing Summit 2017.

notification time

Notification Time

Download the full presentation slides here.

Performance Profiler

https://goo.gl/98Osig

This blueprint proposes to create a performance profiler for doctor scenarios.

Problem Description

In the verification job for notification time, we have encountered some performance issues, such as

1. In environment deployed by APEX, it meets the criteria while in the one by Fuel, the performance is much more poor. 2. Signification performance degradation was spotted when we increase the total number of VMs

It takes time to dig the log and analyse the reason. People have to collect timestamp at each checkpoints manually to find out the bottleneck. A performance profiler will make this process automatic.

Proposed Change

Current Doctor scenario covers the inspector and notifier in the whole fault management cycle:

start                                          end
  +       +         +        +       +          +
  |       |         |        |       |          |
  |monitor|inspector|notifier|manager|controller|
  +------>+         |        |       |          |
occurred  +-------->+        |       |          |
  |     detected    +------->+       |          |
  |       |     identified   +-------+          |
  |       |               notified   +--------->+
  |       |                  |    processed  resolved
  |       |                  |                  |
  |       +<-----doctor----->+                  |
  |                                             |
  |                                             |
  +<---------------fault management------------>+

The notification time can be split into several parts and visualized as a timeline:

start                                         end
  0----5---10---15---20---25---30---35---40---45--> (x 10ms)
  +    +   +   +   +    +      +   +   +   +   +
0-hostdown |   |   |    |      |   |   |   |   |
  +--->+   |   |   |    |      |   |   |   |   |
  |  1-raw failure |    |      |   |   |   |   |
  |    +-->+   |   |    |      |   |   |   |   |
  |    | 2-found affected      |   |   |   |   |
  |    |   +-->+   |    |      |   |   |   |   |
  |    |     3-marked host down|   |   |   |   |
  |    |       +-->+    |      |   |   |   |   |
  |    |         4-set VM error|   |   |   |   |
  |    |           +--->+      |   |   |   |   |
  |    |           |  5-notified VM error  |   |
  |    |           |    +----->|   |   |   |   |
  |    |           |    |    6-transformed event
  |    |           |    |      +-->+   |   |   |
  |    |           |    |      | 7-evaluated event
  |    |           |    |      |   +-->+   |   |
  |    |           |    |      |     8-fired alarm
  |    |           |    |      |       +-->+   |
  |    |           |    |      |         9-received alarm
  |    |           |    |      |           +-->+
sample | sample    |    |      |           |10-handled alarm
monitor| inspector |nova| c/m  |    aodh   |
  |                                        |
  +<-----------------doctor--------------->+

Note: c/m = ceilometer

And a table of components sorted by time cost from most to least

Component Time Cost Percentage
inspector 160ms 40%
aodh 110ms 30%
monitor 50ms 14%
...    
...    

Note: data in the table is for demonstration only, not actual measurement

Timestamps can be collected from various sources

  1. log files
  2. trace point in code

The performance profiler will be integrated into the verification job to provide detail result of the test. It can also be deployed independently to diagnose performance issue in specified environment.

Working Items
  1. PoC with limited checkpoints
  2. Integration with verification job
  3. Collect timestamp at all checkpoints
  4. Display the profiling result in console
  5. Report the profiling result to test database
  6. Independent package which can be installed to specified environment
Planned Maintenance Design Guideline

Note

This is spec draft of design guideline for planned maintenance. JIRA ticket to track the update and collect comments: DOCTOR-52.

This document describes how one can implement planned maintenance by utilizing the OPNFV Doctor project. framework and to meet the set requirements.

Problem Description

Telco application need to know when planned maintenance is going to happen in order to guarantee zero down time in its operation. It needs to be possible to make own actions to have application running on not affected resource or give guidance to admin actions like migration. More details are defined in requirement documentation: use cases, architecture and implementation. Also discussion in the OPNFV summit about planned maintenance session.

Guidelines

Cloud admin needs to make a notification about planned maintenance including all details that application needs in order to make decisions upon his affected service. This notification payload can be consumed by application by subscribing to corresponding event alarm trough alarming service like OpenStack AODH.

Before maintenance starts application needs to be able to make switch over for his ACT-STBY service affected, do operation to move service to not effected part of infra or give a hint for admin operation like migration that can be automatically issued by admin tool according to agreed policy.

Flow diagram:

admin alarming project  controller  inspector
  |   service  app manager   |           |
  |  1.   |         |        |           |
  +------------------------->+           |
  +<-------------------------+           |
  |  2.   |         |        |           |
  +------>+    3.   |        |           |
  |       +-------->+   4.   |           |
  |       |         +------->+           |
  |       |    5.   +<-------+           |
  +<----------------+        |           |
  |                 |   6.   |           |
  +------------------------->+           |
  +<-------------------------+     7.    |
  +------------------------------------->+
  |   8.  |         |        |           |
  +------>+    9.   |        |           |
  |       +-------->+        |           |
  +--------------------------------------+
  |                10.                   |
  +--------------------------------------+
  |  11.  |         |        |           |
  +------------------------->+           |
  +<-------------------------+           |
  |  12.  |         |        |           |
  +------>+-------->+        |    13.    |
  +------------------------------------->+
  +-------+---------+--------+-----------+

Concepts used below:

  • full maintenance: This means maintenance will take a longer time and resource should be emptied, meaning container or VM need to be moved or deleted. Admin might need to test resource to work after maintenance.
  • reboot: Only a reboot is needed and admin does not need separate testing after that. Container or VM can be left in place if so wanted.
  • notification: Notification to rabbitmq.

Admin makes a planned maintenance session where he sets a maintenance_session_id that is a unique ID for all the hardware resources he is going to have the maintenance at the same time. Mostly maintenance should be done node by node, meaning a single compute node at a time would be in single planned maintenance session having unique maintenance_session_id. This ID will be carried trough the whole session in all places and can be used to query maintenance in admin tool API. Project running a Telco application should set a specific role for admin tool to know it cannot do planned maintenance unless project has agreed actions to be done for its VMs or containers. This means the project has configured itself to get alarms upon planned maintenance and it is capable of agreeing needed actions. Admin is supposed to use an admin tool to automate maintenance process partially or entirely.

The flow of a successful planned maintenance session as in OpenStack example case:

  1. Admin disables nova-compute in order to do planned maintenance on a compute host and gets ACK from the API call. This action needs to be done to ensure no thing will be placed in this compute host by any user. Action is always done regardless the whole compute will be affected or not.
  2. Admin sends a project specific maintenance notification with state planned maintenance. This includes detailed information about maintenance, like when it is going to start, is it reboot or full maintenance including the information about project containers or VMs running on host or the part of it that will need maintenance. Also default action like migration will be mentioned that will be issued by admin before maintenance starts if no other action is set by project. In case project has a specific role set, planned maintenance cannot start unless project has agreed the admin action. Available admin actions are also listed in notification.
  3. Application manager of the project receives AODH alarm about the same.
  4. Application manager can do switch over to his ACT-STBY service, delete and re-instantiate his service on not affected resource if so wanted.
  5. Application manager may call admin tool API to give preferred instructions for leaving VMs and containers in place or do admin action to migrate them. In case admin does not receive this instruction before maintenance is to start it will do the pre-configured default action like migration to projects without a specific role to say project need to agree the action. VMs or Containers can be left on host if type of maintenance is just reboot.
  6. Admin does possible actions to VMs and containers and receives an ACK.
  7. In case everything went ok, Admin sends admin type of maintenance notification with state in maintenance. This notification can be consumed by Inspector and other cloud services to know there is ongoing maintenance which means things like automatic fault management actions for the hardware resources should be disabled.
  8. If maintenance type is reboot and project is still having containers or VMs running on affected hardware resource, Admin sends project specific maintenance notification with state updated to in maintenance. If project do not have anything left running on affected hardware resource, state will be maintenance over instead. If maintenance can not be performed for some reason state should be maintenance cancelled. In this case last operation remaining for admin is to re-enable nova-compute service, ensure everything is running and not to proceed any further steps.
  9. Application manager of the project receives AODH alarm about the same.
  10. Admin will do the maintenance. This is out of Doctor scope.
  11. Admin enables nova-compute service when maintenance is over and host can be put back to production. An ACK is received from API call.
  12. In case project had left containers or VMs on hardware resource over maintenance, Admin sends project specific maintenance notification with state updated to maintenance over.
  13. Admin sends admin type of maintenance notification with state updated to maintenance over. Inspector and other cloud services can consume this to know hardware resource is back in use.
POC

There was a Maintenance POC for planned maintenance in the OPNFV Beijing summit to show the basic concept of using framework defined by the project.

Inspector Design Guideline

Note

This is spec draft of design guideline for inspector component. JIRA ticket to track the update and collect comments: DOCTOR-73.

This document summarize the best practise in designing a high performance inspector to meet the requirements in OPNFV Doctor project.

Problem Description

Some pitfalls has be detected during the development of sample inspector, e.g. we suffered a significant performance degrading in listing VMs in a host.

A patch set for caching the list has been committed to solve issue. When a new inspector is integrated, it would be nice to have an evaluation of existing design and give recommendations for improvements.

This document can be treated as a source of related blueprints in inspector projects.

Guidelines
Host specific VMs list

While requirement in doctor project is to have alarm about fault to consumer in one second, it is just a limit we have set in requirements. When talking about fault management in Telco, the implementation needs to be by all means optimal and the one second is far from traditional Telco requirements.

One thing to be optimized in inspector is to eliminate the need to read list of host specific VMs from Nova API, when it gets a host specific failure event. Optimal way of implementation would be to initialize this list when Inspector start by reading from Nova API and after this list would be kept up-to-date by instance.update notifications received from nova. Polling Nova API can be used as a complementary channel to make snapshot of hosts and VMs list in order to keep the data consistent with reality.

This is enhancement and not perhaps something needed to keep under one second in a small system. Anyhow this would be something needed in case of production use.

This guideline can be summarized as following:

  • cache the host VMs mapping instead of reading it on request
  • subscribe and handle update notifications to keep the list up to date
  • make snapshot periodically to ensure data consistency
Parallel execution

In doctor’s architecture, the inspector is responsible to set error state for the affected VMs in order to notify the consumers of such failure. This is done by calling the nova reset-state API. However, this action is a synchronous request with many underlying steps and cost typically hundreds of milliseconds. According to the discussion in mailing list, this time cost will grow linearly if the requests are sent one by one. It will become a critical issue in large scale system.

It is recommended to introduce parallel execution for actions like reset-state that takes a list of targets.

Shortcut notification

An alternative way to improve notification performance is to take a shortcut from inspector to notifier instead of triggering it from controller. The difference between the two workflow is shown below:

conservative notification

Conservative Notification

shortcut notification

Shortcut Notification

It worth noting that the shortcut notification has a side effect that cloud resource states could still be out-of-sync by the time consumer processes the alarm notification. This is out of scope of inspector design but need to be taken consideration in system level.

Also the call of “reset servers state to error” is not necessary in the alternative notification case where the “host forced down” is still called. “get-valid-server-state” was implemented to have valid server state while earlier one couldn’t get it unless calling “reset servers state to error”. When not having “reset servers state to error”, states are more unlikely to be out of sync while notification and force down host would be parallel.

Appendix

A study has been made to evaluate the effect of parallel execution and shortcut notification on OPNFV Beijing Summit 2017.

notification time

Notification Time

Download the full presentation slides here.

Manuals
OpenStack NOVA API for marking host down.
What the API is for
This API will give external fault monitoring system a possibility of telling OpenStack Nova fast that compute host is down. This will immediately enable calling of evacuation of any VM on host and further enabling faster HA actions.
What this API does
In OpenStack the nova-compute service state can represent the compute host state and this new API is used to force this service down. It is assumed that the one calling this API has made sure the host is also fenced or powered down. This is important, so there is no chance same VM instance will appear twice in case evacuated to new compute host. When host is recovered by any means, the external system is responsible of calling the API again to disable forced_down flag and let the host nova-compute service report again host being up. If network fenced host come up again it should not boot VMs it had if figuring out they are evacuated to other compute host. The decision of deleting or booting VMs there used to be on host should be enhanced later to be more reliable by Nova blueprint: https://blueprints.launchpad.net/nova/+spec/robustify-evacuate
REST API for forcing down:

Parameter explanations: tenant_id: Identifier of the tenant. binary: Compute service binary name. host: Compute host name. forced_down: Compute service forced down flag. token: Token received after successful authentication. service_host_ip: Serving controller node ip.

request: PUT /v2.1/{tenant_id}/os-services/force-down { “binary”: “nova-compute”, “host”: “compute1”, “forced_down”: true }

response: 200 OK { “service”: { “host”: “compute1”, “binary”: “nova-compute”, “forced_down”: true } }

Example: curl -g -i -X PUT http://{service_host_ip}:8774/v2.1/{tenant_id}/os-services /force-down -H “Content-Type: application/json” -H “Accept: application/json ” -H “X-OpenStack-Nova-API-Version: 2.11” -H “X-Auth-Token: {token}” -d ‘{“b inary”: “nova-compute”, “host”: “compute1”, “forced_down”: true}’

CLI for forcing down:

nova service-force-down <hostname> nova-compute

Example: nova service-force-down compute1 nova-compute

REST API for disabling forced down:

Parameter explanations: tenant_id: Identifier of the tenant. binary: Compute service binary name. host: Compute host name. forced_down: Compute service forced down flag. token: Token received after successful authentication. service_host_ip: Serving controller node ip.

request: PUT /v2.1/{tenant_id}/os-services/force-down { “binary”: “nova-compute”, “host”: “compute1”, “forced_down”: false }

response: 200 OK { “service”: { “host”: “compute1”, “binary”: “nova-compute”, “forced_down”: false } }

Example: curl -g -i -X PUT http://{service_host_ip}:8774/v2.1/{tenant_id}/os-services /force-down -H “Content-Type: application/json” -H “Accept: application/json ” -H “X-OpenStack-Nova-API-Version: 2.11” -H “X-Auth-Token: {token}” -d ‘{“b inary”: “nova-compute”, “host”: “compute1”, “forced_down”: false}’

CLI for disabling forced down:

nova service-force-down –unset <hostname> nova-compute

Example: nova service-force-down –unset compute1 nova-compute

Get valid server state
Problem description

Previously when the owner of a VM has queried his VMs, he has not received enough state information, states have not changed fast enough in the VIM and they have not been accurate in some scenarios. With this change this gap is now closed.

A typical case is that, in case of a fault of a host, the user of a high availability service running on top of that host, needs to make an immediate switch over from the faulty host to an active standby host. Now, if the compute host is forced down [1] as a result of that fault, the user has to be notified about this state change such that the user can react accordingly. Similarly, a change of the host state to “maintenance” should also be notified to the users.

What is changed

A new host_status parameter is added to the /servers/{server_id} and /servers/detail endpoints in microversion 2.16. By this new parameter user can get additional state information about the host.

host_status possible values where next value in list can override the previous:

  • UP if nova-compute is up.
  • UNKNOWN if nova-compute status was not reported by servicegroup driver within configured time period. Default is within 60 seconds, but can be changed with service_down_time in nova.conf.
  • DOWN if nova-compute was forced down.
  • MAINTENANCE if nova-compute was disabled. MAINTENANCE in API directly means nova-compute service is disabled. Different wording is used to avoid the impression that the whole host is down, as only scheduling of new VMs is disabled.
  • Empty string indicates there is no host for server.

host_status is returned in the response in case the policy permits. By default the policy is for admin only in Nova policy.json:

"os_compute_api:servers:show:host_status": "rule:admin_api"

For an NFV use case this has to also be enabled for the owner of the VM:

"os_compute_api:servers:show:host_status": "rule:admin_or_owner"
REST API examples:

Case where nova-compute is enabled and reporting normally:

GET /v2.1/{tenant_id}/servers/{server_id}

200 OK
{
  "server": {
    "host_status": "UP",
    ...
  }
}

Case where nova-compute is enabled, but not reporting normally:

GET /v2.1/{tenant_id}/servers/{server_id}

200 OK
{
  "server": {
    "host_status": "UNKNOWN",
    ...
  }
}

Case where nova-compute is enabled, but forced_down:

GET /v2.1/{tenant_id}/servers/{server_id}

200 OK
{
  "server": {
    "host_status": "DOWN",
    ...
  }
}

Case where nova-compute is disabled:

GET /v2.1/{tenant_id}/servers/{server_id}

200 OK
{
  "server": {
    "host_status": "MAINTENANCE",
    ...
  }
}

Host Status is also visible in python-novaclient:

+-------+------+--------+------------+-------------+----------+-------------+
| ID    | Name | Status | Task State | Power State | Networks | Host Status |
+-------+------+--------+------------+-------------+----------+-------------+
| 9a... | vm1  | ACTIVE | -          | RUNNING     | xnet=... | UP          |
+-------+------+--------+------------+-------------+----------+-------------+

Domino

Domino Project Overview
Domino Description

Domino provides a distribution service for Network Service Descriptors (NSDs) and Virtual Network Function Descriptors (VNFDs) that are composed using Tosca Simple Profile for Network Functions Virtualization (http://docs.oasis-open.org/tosca/tosca-nfv/v1.0/tosca-nfv-v1.0.html). Domino service is targeted towards supporting many Software Defined Network (SDN) controllers, Service Orchestrators (SOs), VNF Managers (VNFMs), Virtual Infastructure Managers (VIMs), Operation and Business Support Systems that produce and/or consume NSDs and VNFDs.

Producers of NSDs and VNFDs use Domino Service through Service Access Points (SAPs) or End Points (EPs) to publish these descriptors. Consumers of NSDs and VNFDs subscribe with the Domino Service through the same SAPs/EPs and declare their resource capabilities to onboard and perform Life Cycle Management (LCM) for Network Services (NSs) and Virtual Network Functions (VNFs). Thus, Domino acts as a service broker for NSs and VNFs modeled in a Tosca template.

Domino Capabilities and Usage
Labels in Domino

Domino’s pub/sub architecture is based on labels (see Fig. 1 below). Each Template Producer and Template Consumer is expected to run a local Domino Client to publish templates and subscribe for labels.

alternate text

Domino provides a pub/sub server for NSDs and VNFDs

Domino Service does not interpret what the labels mean. Domino derives labels directly from the normative definitions in TOSCA Simple YAML Profile for NFV. Domino parses the policy rules included in the NSD/VNFD, form “policy” labels, and determine which resources are associated with which set of labels. Domino identifies which Domino Clients can host which resource based on the label subscriptions by these clients. Once mapping of resources to the clients are done, new NSDs/VNFDs are created based on the mapping. These new NSDs/VNFDs are translated and delivered to the clients.

Label Format and Examples

Domino supports policy labels in the following form:

<policytype>:properties:<key:value>

Orchestrators, controllers, and managers use Domino service to announce their capabilities by defining labels in this form and subscribing for these labels with the Domino Server.

For instance a particular VIM that is capable of performing an affinity based VNF or VDU placement at host machine granularity can specify a label in the form:

tosca.policies.Placement.affinity:properties:granularity:hostlevel

When the VIM registers with the Domino Service and subscribed for that label, Domino views this VIM as a candidate location that can host a VNF or VDU requesting affinity based placement policy at host machine granularity.

Another use case is the announcement of lifecycle management capabilities for VNFs and VNF Forwarding Graphs (VNFFG) by different SDN Controllers (SDN-Cs), VNFMs, or VIMs. For instance

tosca.policies.Scaling.VNFFG:properties:session_continuity:true

can be used as a label to indicate that when a scaling operation on a VNFFG (e.g., add more VNFs into the graph) is requested, existing session can still be enforced to go through the same chain of VNF instances.

To utilize Domino’s domain mapping services for virtual network resources (e.g., VNF, VDU, VNFFG, CP, VL, etc.), a network service or network function request must include policy rules that are composed of policy types and property values that match to the label announcements made by these domains. For instance, when a TOSCA template includes a policy rule with type “tosca.policies.Scaling.VNFFG” and property field “session_continuity” set as “true” targeting one or more VNFFGs, this serves as the hint for the Domino Server to identify all the Domain Clients that subscribed the label “tosca.policies.Scaling.VNFFG:properties:session_continuity:true”.

Template Example for Label Extraction

Consider the following NSD TOSCA template:

tosca_definitions_version: tosca_simple_profile_for_nfv_1_0_0
description: Template for deploying a single server with predefined properties.
metadata:
  template_name: TOSCA NFV Sample Template
policy_types:
  tosca.policies.Placement.Geolocation:
    description: Geolocation policy
    derived_from: tosca.policies.Placement
topology_template:
  node_templates:
    VNF1:
      type: tosca.nodes.nfv.VNF
      properties:
        id: vnf1
        vendor: acmetelco
        version: 1.0
    VNF2:
      type: tosca.nodes.nfv.VNF
      properties:
        id: vnf2
        vendor: ericsson
        version: 1.0
    VNF3:
      type: tosca.nodes.nfv.VNF
      properties:
        id: vnf3
        vendor: huawei
        version: 1.0
  policies:
    - rule1:
        type: tosca.policies.Placement.Geolocation
        targets: [ VNF1 ]
        properties:
          region: [ us-west-1 ]
    - rule2:
        type: tosca.policies.Placement.Geolocation
        targets: [ VNF2, VNF3 ]
        properties:
          region: [ us-west-1 , us-west-2 ]

Domino Server extracts all possible policy labels by exhaustively concatenating key-value pairs under the properties section of the policy rules to the policy type of these rules:

tosca.policies.Placement.Geolocation:properties:region:us-west-1
tosca.policies.Placement.Geolocation:properties:region:us-west-2

Furthermore, Domino Server iterates over the targets specified under policy rules to generate a set of labels for each target node:

required_labels['VNF1'] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 }
required_labels['VNF2'] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 , tosca.policies.Placement.Geolocation:properties:region:us-west-2}
required_labels['VNF3'] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 , tosca.policies.Placement.Geolocation:properties:region:us-west-2}

When a Template Consuming site (e.g., VNFM or VIM) registers with the Domino Server using Domino Client, it becomes an eligible candidate for template distribution with an initially empty set of label subscriptions. Suppose three different Domino Clients register with the Domino Server and subscribe for some or none of the policy labels such that the Domino Server has the current subscription state as follows:

subscribed_labels[site-1] = { } #this is empty set
subscribed_labels[site-2] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 }
subscribed_labels[site-3] = { tosca.policies.Placement.Geolocation:properties:region:us-west-1 ,  tosca.policies.Placement.Geolocation:properties:region:us-west-2}

Based on the TOSCA example and hypothetical label subscriptions above, Domino Server identifies all the VNFs can be hosted by Site-3, while VNF1 can be hosted by both Site-2 and Site-3. Note that Site-1 cannot host any of the VNFs listed in the TOSCA file. When a VNF can be hosted by multiple sites, Domino Server picks the site that can host the most number of VNFs. When not all VNFs can be hosted on the same site, the TOSCA file is partitioned into multiple files, one for each site. These files share a common part (e.g, meta-data, policy-types, version, description, virtual resources that are not targeted by any policy rule, etc.). Each site specific file has also a non-common part that only appears in that file (i.e., virtual resources explicitly assigned to that site and the policy rules that accompany those virtual resources.

In the current Domino convention, if a VNF (or any virtual resource) does not have a policy rule (i.e., it is not specified as a target in any of the policy rules) and it also is not dependent on any VNF (or any virtual resource) that is assigned to another site, that resource is wild carded by default and treated as part of the “common part”. Also note that currently Domino does not support all or nothing semantics: if some of the virtual resources are not mappable to any domain because they are targets of policy rules that are not supported by any site, these portions will be excluded while the remaining virtual resources will be still be part of one or more template files to be distributed to hosting sites. When NSDs and VNFDs are prepared, these conventions must be kept in mind. In the future releases, these conventions can change based on the new use cases.

For the example above, no partitioning would occur as all VNFs are mapped onto site-3; Domino Server simply delivers the Tosca file to Domino Client hosted on site-3. When TOSCA cannot be consumed by a particular site directly, Domino Server can utilize existing translators (e.g., heat-translator) to first translate the template before delivery.

Internal Processing Pipeline at Domino Server

Fig. 2 shows the block diagram for the processing stages of a published TOSCA template. Domino Client issues an RPC call publish(tosca file). Domino Server passes the received tosca file to Label Extractor that outputs resource labels. Domain Mapper uses the extracted labels and tosca file to find mappings from resources to domains as well as the resource dependencies. Resource to domain mappings and resource dependencies are utilized to partition the orchestration template into individual resource orchestration templates (one for each domain). If a translation is required (e.g., TOSCA to HOT), individual resource orchestration templates are first translated and then placed on a template distribution workflow based on resource dependencies. Message Sender block in the server takes one distribution task at a time from the workflow generator and pushes the orchestration template to the corresponding Domino Client.

alternate text

Domino Service Processing Pipeline

Resource Scheduling

Domino Service currently supports maximum packing strategy when a virtual resource type can be hosted on multiple candidate sites. Initially, Domino Scheduler identifies virtual resources that has only one feasible site for hosting. Each such virtual resource is trivially assigned to its only feasible site. The remaining virtual resources with multiple candidate locations are sequentially allocated to one of their candidate locations that has the most virtual resource assignments so far. Note that wildcarded resources are assigned to all sites. To prevent wildcarding within the current release, (i) all sites must subscribed to a base policy with a dummy key-value pair defined under the properties tab and (ii) all the independent resources must be specified as target of that policy in NSD or VNFD file.

Domino Installation Instruction
Domino Installation

Note: The steps below are tested for Ubuntu (16.04, 14.04) and OS X El Capitan.

Prerequisites
  • git
  • python-pip
  • python (version =2.7)
  • tosca-parser (version >=0.4.0)
  • heat-translator (version >=0.5.0)
Installation Steps (Single Node)
  • Step-0: Prepare Environment
> $sudo pip install tosca-parser
> $sudo pip install heat-translator
> $sudo pip install requests
  • Step-1: Get the Domino code
git clone https://gerrit.opnfv.org/gerrit/domino -b stable/danube
  • Step-2: Go to the main domino directory
cd domino

You should see DominoClient.py, DominoServer.py, and domino-cli.py as executables.

Installation Steps (Multiple Node)

Repeat the installation steps for single node on each node. The script run_on_remotenodes.sh under ./domino/tests directory deploys the Domino Code on three hosts from a deployment node and tests RPC calls. In the script, the private key location and remote host IP addresses must be manually entered. IS_IPandKEY_CONFIGURED should be set true, i.e., IS_IPandKEY_CONFIGURED=true.

Domino Configuration Guide
Domino Configuration

Domino Server and Clients can be configured via (i) passing command line options (see API documentation) and (ii) the configuration file “domino_conf.py” under the main directory.

  • The default file for logging is set as none and log level set as “WARNING”.
Domino Server
  • The default server unique user ID is set as 0 in the configuration file.
  • The default TCP port for RPC calls is set as 9090 in the configuration file.
  • The default database file for Domino Server is set as “dominoserver.db” under the main directory
  • The default folder for keeping published TOSCA files and pushed parts is set as “toscafiles” in the configuration file via variable TOSCADIR.
  • The default log level (WARNING) can be changed by passing the flags –log or -l followed by a log level, e.g., ERROR, WARNING, INFO, or DEBUG.
Domino Client
  • The default mode of CLI is non-interactive (i.e., Domino CLI Utility is used). Passing –iac=TRUE would begin the client in interactive mode.
  • The default log level (WARNING) can be changed by passing the flags –log or -l followed by a log level, e.g., ERROR, WARNING, INFO, or DEBUG.
  • The default Domino Server IP is set as “localhost”. This can be overwritten at the time of launching DominoClient via the option flags -i or –ipaddr followed by the IP address of the actual server hosting the Domino Server.
  • The default Domino Client TCP port for RPC calls is set as 9091 in the configuration file. It can be overwritten when the DominoClient is launched by passing the flags –port or -p followed by the port number.
  • The default folder for keeping preceived TOSCA files is set as “toscafiles” in the configuration file via variable TOSCA_RX_DIR.
Domino User Guide
Domino API Usage Guidelines and Examples
Using domino-cli Client

Prerequisites:

  1. Make sure that domino-cli.py is in +x mode.
  2. Change directory to where domino-cli.py, DominoClient.py and DominoServer.py are located or include file path in the PATH environment variable.
  3. Start the Domino Server:
./DominoServer.py --log=debug
  1. Start the Domino Client:
./DominoClient.py -p <portnumber> --cliport <cli-portnumber> --log=debug

Note1: The default log level is WARNING and omitting –log option will lead to minimal/no logging on the console

Note2: domino_conf.py file includes most of the default values

  • Registration Command

Command line input:

./domino-cli.py <cli-portnumber> register

This message has the following fields that are automatically filled in.

Message Type (= REGISTER)
DESIRED UDID (= if not allocated, this will be assigned as Unique Domino ID)
Sequence Number (=incremented after each RPC call)
IP ADDR (= IP address of DOMINO Client to be used by DOMINO Server for future RPC Calls to this client)
TCP PORT (= TCP port of DOMINO Client to be used by DOMINO Server for future RPC Calls to this client)
Supported Templates (= Null, this field not used currently)
  • Heart Beat Command

Command line input:

./domino-cli.py <cli-portnumber> heartbeat

This message has the following fields that are automatically filled in.

Message Type (= HEART_BEAT)
UDID (= Unique Domino ID assigned during registration)
Sequence Number (=incremented after each RPC call)
  • Label and Template Type Subscription Command
./domino-cli.py <cli-portnumber> subscribe -l <labelname> -t <templatetype>

Note that -l can be substituted by –label and -t can be substituted by –ttype.

More than one label or template type can be subscribed within the same command line as comma separated labels or template types

./domino-cli.py <cli-portnumber> subscribe -l <label1>,<label2>,<labeln> -t <ttype1>,<ttype2>,<ttypen>

To subscribe more than one label or template type, one can also repeat the options -l and -t, e.g.:

./domino-cli.py <cli-portnumber> subscribe -l <label1> -l <label2> -l <labeln> -t <ttype1> -t <ttype2> -t <ttypen>

It is safe to call subscribe command multiple times with duplicate labels.

This message has the following fields that are automatically filled in.

Message Type (= SUBSCRIBE)
UDID (= Unique Domino ID assigned during registration)
Sequence Number (=incremented after each RPC call)
Template Operation (= APPEND)
Label Operation (= APPEND)

The following fields are filled in based on arguments passed on via -l/–label and -t/–ttype flags

Subscribe RPC also supports options for label using
–lop=APPEND/DELETE/OVERWRITE
and for supported template types using
–top=APPEND/DELETE/OVERWRITE.

When unspecified, the default is APPEND. DELETE deletes existing labels (template types) specified in the current call via key -l/–label (-t/–ttype). OVERWRITE removes the current set of labels (template types) and sets it to the new set of values passed in the same RPC call.

By default, no translation service is provided. Currently, only TOSCA to Heat Orchestration Template (HOT) translation is supported using OpenStack heat-translator library. A domain that requires HOT files must subscribe HOT template type using

./domino-cli.py <cli-portnumber> subscribe -t hot
  • Template Publishing Command
./domino-cli.py <cli-portnumber> publish -t <toscafile>

Note that -t can be substituted with –tosca-file.

If -t or –tosca-file flag is used multiple times, the last tosca file passed as input will be used. This usage is not recommended as undefined/unintended results may emerge as the Domino client will continue to publish.

This message has the following fields that are automatically filled in.

Message Type (= PUBLISH)
UDID (= Unique Domino ID assigned during registration)
Sequence Number (=incremented after each RPC call)
Template Type (= TOSCA)
Template File

Since Danube release, Domino Server supports stateful updates for template publishing. The following command can be used to update the service template for an existing Template Unique ID (TUID):

./domino-cli.py <cli-portnumber> publish -t <toscafile> -k <TUID>

Note that -k can be substituted with –tuid. When Domino Server receives this command, it verifies whether the client previously published the provided TUID. If such TUID does not exist, then Domino Server returns FAILED response back to the client. If such TUID exists, Domino Server recomputes which resources are mapped onto which domains and updates each domain with the new VNF and NS descriptors. If a previously utilized domain is no longer targeted, it is updated with a null descriptor.

  • Template Listing Command
./domino-cli.py <cli-portnumber> list-tuids

Queries all the Template Unique IDs (TUIDs) published by the Domino Client from the Domino Server.

Interactive CLI mode

To enter this mode, start Domino Client with interactive console option set as true, i.e., –iac=true:

./DominoClient -p <portnumber> --iax=true --log=DEBUG

The rest of the API calls are the same as in the case of using domino-cli.py except that at the prompt there is no need to write “domino-cli.py <cli-portnumber>, e.g.,:

>>register
>>heartbeat
>>subscribe -l <label1> -t <ttype1>
>>publish -t <toscafile>

The interactive CLI mode is mainly supported for manual testing.

IPV6

IPv6 Installation Procedure
Abstract:

This document provides the users with the Installation Procedure to install OPNFV Euphrates Release on IPv6-only Infrastructure.

1. Install OPNFV on IPv6-Only Infrastructure

This section provides instructions to install OPNFV on IPv6-only Infrastructure. All underlay networks and API endpoints will be IPv6-only except:

  1. “admin” network in underlay/undercloud still has to be IPv4, due to lack of support of IPMI over IPv6 or PXE over IPv6.
  2. Metadata server is still IPv4 only.

Except the limitations above, the use case scenario of the IPv6-only infrastructure includes:

  1. Support OPNFV deployment on an IPv6 only infrastructure.
  2. Horizon/ODL-DLUX access using IPv6 address from an external host.
  3. OpenStack API access using IPv6 addresses from various python-clients.
  4. Ability to create Neutron Routers, IPv6 subnets (e.g. SLAAC/DHCPv6-Stateful/ DHCPv6-Stateless) to support North-South traffic.
  5. Inter VM communication (East-West routing) when VMs are spread across two compute nodes.
  6. VNC access into a VM using IPv6 addresses.
  7. IPv6 support in OVS VxLAN (and/or GRE) tunnel endpoints with OVS 2.6+ (NEW)
1.1. Install OPNFV in OpenStack-Only Environment

Apex Installer:

# HA, Virtual deployment in OpenStack-only environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-nosdn-nofeature-ha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml

# HA, Bare Metal deployment in OpenStack-only environment
./opnfv-deploy -d /etc/opnfv-apex/os-nosdn-nofeature-ha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml

# Non-HA, Virtual deployment in OpenStack-only environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-nosdn-nofeature-noha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml

# Non-HA, Bare Metal deployment in OpenStack-only environment
./opnfv-deploy -d /etc/opnfv-apex/os-nosdn-nofeature-noha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml

# Note:
#
# 1. Parameter ""-v" is mandatory for Virtual deployment
# 2. Parameter "-i <inventory file>" is mandatory for Bare Metal deployment
# 2.1 Refer to https://git.opnfv.org/cgit/apex/tree/config/inventory for examples of inventory file
# 3. You can use "-n /etc/opnfv-apex/network_settings.yaml" for deployment in IPv4 infrastructure

Please NOTE that:

  • You need to refer to installer’s documentation for other necessary parameters applicable to your deployment.
  • You need to refer to Release Notes and installer’s documentation if there is any issue in installation.
1.2. Install OPNFV in OpenStack with ODL-L3 Environment

Apex Installer:

# HA, Virtual deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-odl-nofeature-ha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml

# HA, Bare Metal deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -d /etc/opnfv-apex/os-odl-nofeature-ha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml

# Non-HA, Virtual deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-odl-nofeature-noha.yaml \
-n /etc/opnfv-apex/network_settings_v6.yaml

# Non-HA, Bare Metal deployment in OpenStack with Open Daylight L3 environment
./opnfv-deploy -d /etc/opnfv-apex/os-odl-nofeature-noha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_settings_v6.yaml

# Note:
#
# 1. Parameter ""-v" is mandatory for Virtual deployment
# 2. Parameter "-i <inventory file>" is mandatory for Bare Metal deployment
# 2.1 Refer to https://git.opnfv.org/cgit/apex/tree/config/inventory for examples of inventory file
# 3. You can use "-n /etc/opnfv-apex/network_settings.yaml" for deployment in IPv4 infrastructure

Please NOTE that:

  • You need to refer to installer’s documentation for other necessary parameters applicable to your deployment.
  • You need to refer to Release Notes and installer’s documentation if there is any issue in installation.
1.3. Testing Methodology

There are 2 levels of testing to validate the deployment.

1.3.1. Underlay Testing for OpenStack API Endpoints

Underlay Testing is to validate that API endpoints are listening on IPv6 addresses. Currently, we are only considering the Underlay Testing for OpenStack API endpoints. The Underlay Testing for Open Daylight API endpoints is for future release.

The Underlay Testing for OpenStack API endpoints can be as simple as validating Keystone service, and as complete as validating each API endpoint. It is important to reuse Tempest API testing. Currently:

  • Apex Installer will change OS_AUTH_URL in overcloudrc during installation process. For example: export OS_AUTH_URL=http://[2001:db8::15]:5000/v2.0. OS_AUTH_URL points to Keystone and Keystone catalog.
  • When FuncTest runs Tempest for the first time, the OS_AUTH_URL is taken from the environment and placed automatically in Tempest.conf.
  • Under this circumstance, openstack catalog list will return IPv6 URL endpoints for all the services in catalog, including Nova, Neutron, etc, and covering public URLs, private URLs and admin URLs.
  • Thus, as long as the IPv6 URL is given in the overclourc, all the tests will use that (including Tempest).

Therefore Tempest API testing is reused to validate API endpoints are listening on IPv6 addresses as stated above. They are part of OpenStack default Smoke Tests, run in FuncTest and integrated into OPNFV’s CI/CD environment.

1.3.2. Overlay Testing

Overlay Testing is to validate that IPv6 is supported in tenant networks, subnets and routers. Both Tempest API testing and Tempest Scenario testing are used in our Overlay Testing.

Tempest API testing validates that the Neutron API supports the creation of IPv6 networks, subnets, routers, etc:

tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
tempest.api.network.test_networks.NetworksIpV6Test.test_create_update_delete_network_subnet
tempest.api.network.test_networks.NetworksIpV6Test.test_external_network_visibility
tempest.api.network.test_networks.NetworksIpV6Test.test_list_networks
tempest.api.network.test_networks.NetworksIpV6Test.test_list_subnets
tempest.api.network.test_networks.NetworksIpV6Test.test_show_network
tempest.api.network.test_networks.NetworksIpV6Test.test_show_subnet
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_create_update_delete_network_subnet
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_external_network_visibility
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_networks
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_subnets
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_show_network
tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_show_subnet
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_update_delete_port
tempest.api.network.test_ports.PortsIpV6TestJSON.test_list_ports
tempest.api.network.test_ports.PortsIpV6TestJSON.test_show_port
tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces
tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id
tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id
tempest.api.network.test_routers.RoutersIpV6Test.test_create_show_list_update_delete_router
tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_list_update_show_delete_security_group
tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_show_delete_security_group_rule
tempest.api.network.test_security_groups.SecGroupIPv6Test.test_list_security_groups

Tempest Scenario testing validates some specific overlay IPv6 scenarios (i.e. use cases) as follows:

tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_slaac_from_os
tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_dhcpv6_stateless
tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac
tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os

The above Tempest API testing and Scenario testing are quite comprehensive to validate overlay IPv6 tenant networks. They are part of OpenStack default Smoke Tests, run in FuncTest and integrated into OPNFV’s CI/CD environment.

IPv6 Configuration Guide
Abstract:

This document provides the users with the Configuration Guide to set up a service VM as an IPv6 vRouter using OPNFV Euphrates Release.

1. IPv6 Configuration - Setting Up a Service VM as an IPv6 vRouter

This section provides instructions to set up a service VM as an IPv6 vRouter using OPNFV Euphrates Release installers. Because Open Daylight no longer supports L2-only option, and there is only limited support of IPv6 in L3 option of Open Daylight, setup of service VM as an IPv6 vRouter is only available under pure/native OpenStack environment. The deployment model may be HA or non-HA. The infrastructure may be bare metal or virtual environment.

1.1. Pre-configuration Activities

The configuration will work only in OpenStack-only environment.

Depending on which installer will be used to deploy OPNFV, each environment may be deployed on bare metal or virtualized infrastructure. Each deployment may be HA or non-HA.

Refer to the previous installer configuration chapters, installations guide and release notes.

1.2. Setup Manual in OpenStack-Only Environment

If you intend to set up a service VM as an IPv6 vRouter in OpenStack-only environment of OPNFV Euphrates Release, please NOTE that:

  • Because the anti-spoofing rules of Security Group feature in OpenStack prevents a VM from forwarding packets, we need to disable Security Group feature in the OpenStack-only environment.
  • The hostnames, IP addresses, and username are for exemplary purpose in instructions. Please change as needed to fit your environment.
  • The instructions apply to both deployment model of single controller node and HA (High Availability) deployment model where multiple controller nodes are used.
1.2.1. Install OPNFV and Preparation

OPNFV-NATIVE-INSTALL-1: To install OpenStack-only environment of OPNFV Euphrates Release:

Apex Installer:

# HA, Virtual deployment in OpenStack-only environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-nosdn-nofeature-ha.yaml \
-n /etc/opnfv-apex/network_setting.yaml

# HA, Bare Metal deployment in OpenStack-only environment
./opnfv-deploy -d /etc/opnfv-apex/os-nosdn-nofeature-ha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_setting.yaml

# Non-HA, Virtual deployment in OpenStack-only environment
./opnfv-deploy -v -d /etc/opnfv-apex/os-nosdn-nofeature-noha.yaml \
-n /etc/opnfv-apex/network_setting.yaml

# Non-HA, Bare Metal deployment in OpenStack-only environment
./opnfv-deploy -d /etc/opnfv-apex/os-nosdn-nofeature-noha.yaml \
-i <inventory file> -n /etc/opnfv-apex/network_setting.yaml

# Note:
#
# 1. Parameter ""-v" is mandatory for Virtual deployment
# 2. Parameter "-i <inventory file>" is mandatory for Bare Metal deployment
# 2.1 Refer to https://git.opnfv.org/cgit/apex/tree/config/inventory for examples of inventory file
# 3. You can use "-n /etc/opnfv-apex/network_setting_v6.yaml" for deployment in IPv6-only infrastructure

Compass Installer:

# HA deployment in OpenStack-only environment
export ISO_URL=file://$BUILD_DIRECTORY/compass.iso
export OS_VERSION=${{COMPASS_OS_VERSION}}
export OPENSTACK_VERSION=${{COMPASS_OPENSTACK_VERSION}}
export CONFDIR=$WORKSPACE/deploy/conf/vm_environment
./deploy.sh --dha $CONFDIR/os-nosdn-nofeature-ha.yml \
--network $CONFDIR/$NODE_NAME/network.yml

# Non-HA deployment in OpenStack-only environment
# Non-HA deployment is currently not supported by Compass installer

Fuel Installer:

# HA deployment in OpenStack-only environment
# Scenario Name: os-nosdn-nofeature-ha
# Scenario Configuration File: ha_heat_ceilometer_scenario.yaml
# You can use either Scenario Name or Scenario Configuration File Name in "-s" parameter
sudo ./deploy.sh -b <stack-config-uri> -l <lab-name> -p <pod-name> \
-s os-nosdn-nofeature-ha -i <iso-uri>

# Non-HA deployment in OpenStack-only environment
# Scenario Name: os-nosdn-nofeature-noha
# Scenario Configuration File: no-ha_heat_ceilometer_scenario.yaml
# You can use either Scenario Name or Scenario Configuration File Name in "-s" parameter
sudo ./deploy.sh -b <stack-config-uri> -l <lab-name> -p <pod-name> \
-s os-nosdn-nofeature-noha -i <iso-uri>

# Note:
#
# 1. Refer to http://git.opnfv.org/cgit/fuel/tree/deploy/scenario/scenario.yaml for scenarios
# 2. Refer to http://git.opnfv.org/cgit/fuel/tree/ci/README for description of
#    stack configuration directory structure
# 3. <stack-config-uri> is the base URI of stack configuration directory structure
# 3.1 Example: http://git.opnfv.org/cgit/fuel/tree/deploy/config
# 4. <lab-name> and <pod-name> must match the directory structure in stack configuration
# 4.1 Example of <lab-name>: -l devel-pipeline
# 4.2 Example of <pod-name>: -p elx
# 5. <iso-uri> could be local or remote ISO image of Fuel Installer
# 5.1 Example: http://artifacts.opnfv.org/fuel/euphrates/opnfv-euphrates.1.0.iso
#
# Please refer to Fuel Installer's documentation for further information and any update

Joid Installer:

# HA deployment in OpenStack-only environment
./deploy.sh -o mitaka -s nosdn -t ha -l default -f ipv6

# Non-HA deployment in OpenStack-only environment
./deploy.sh -o mitaka -s nosdn -t nonha -l default -f ipv6

Please NOTE that:

  • You need to refer to installer’s documentation for other necessary parameters applicable to your deployment.
  • You need to refer to Release Notes and installer’s documentation if there is any issue in installation.

OPNFV-NATIVE-INSTALL-2: Clone the following GitHub repository to get the configuration and metadata files

git clone https://github.com/sridhargaddam/opnfv_os_ipv6_poc.git \
/opt/stack/opnfv_os_ipv6_poc
1.2.2. Disable Security Groups in OpenStack ML2 Setup

Please NOTE that although Security Groups feature has been disabled automatically through local.conf configuration file by some installers such as devstack, it is very likely that other installers such as Apex, Compass, Fuel or Joid will enable Security Groups feature after installation.

Please make sure that Security Groups are disabled in the setup

In order to disable Security Groups globally, please make sure that the settings in OPNFV-NATIVE-SEC-1 and OPNFV-NATIVE-SEC-2 are applied, if they are not there by default.

OPNFV-NATIVE-SEC-1: Change the settings in /etc/neutron/plugins/ml2/ml2_conf.ini as follows, if they are not there by default

# /etc/neutron/plugins/ml2/ml2_conf.ini
[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.firewall.NoopFirewallDriver
[ml2]
extension_drivers = port_security
[agent]
prevent_arp_spoofing = False

OPNFV-NATIVE-SEC-2: Change the settings in /etc/nova/nova.conf as follows, if they are not there by default.

# /etc/nova/nova.conf
[DEFAULT]
security_group_api = neutron
firewall_driver = nova.virt.firewall.NoopFirewallDriver

OPNFV-NATIVE-SEC-3: After updating the settings, you will have to restart the Neutron and Nova services.

Please note that the commands of restarting Neutron and Nova would vary depending on the installer. Please refer to relevant documentation of specific installers

1.2.3. Set Up Service VM as IPv6 vRouter

OPNFV-NATIVE-SETUP-1: Now we assume that OpenStack multi-node setup is up and running. We have to source the tenant credentials in OpenStack controller node in this step. Please NOTE that the method of sourcing tenant credentials may vary depending on installers. For example:

Apex installer:

# On jump host, source the tenant credentials using /bin/opnfv-util provided by Apex installer
opnfv-util undercloud "source overcloudrc; keystone service-list"

# Alternatively, you can copy the file /home/stack/overcloudrc from the installer VM called "undercloud"
# to a location in controller node, for example, in the directory /opt, and do:
# source /opt/overcloudrc

Compass installer:

# source the tenant credentials using Compass installer of OPNFV
source /opt/admin-openrc.sh

Fuel installer:

# source the tenant credentials using Fuel installer of OPNFV
source /root/openrc

Joid installer:

# source the tenant credentials using Joid installer of OPNFV
source $HOME/joid_config/admin-openrc

devstack:

# source the tenant credentials in devstack
source openrc admin demo

Please refer to relevant documentation of installers if you encounter any issue.

OPNFV-NATIVE-SETUP-2: Download fedora22 image which would be used for vRouter

wget https://download.fedoraproject.org/pub/fedora/linux/releases/22/Cloud/x86_64/\
Images/Fedora-Cloud-Base-22-20150521.x86_64.qcow2

OPNFV-NATIVE-SETUP-3: Import Fedora22 image to glance

glance image-create --name 'Fedora22' --disk-format qcow2 --container-format bare \
--file ./Fedora-Cloud-Base-22-20150521.x86_64.qcow2

OPNFV-NATIVE-SETUP-4: This step is Informational. OPNFV Installer has taken care of this step during deployment. You may refer to this step only if there is any issue, or if you are using other installers.

We have to move the physical interface (i.e. the public network interface) to br-ex, including moving the public IP address and setting up default route. Please refer to OS-NATIVE-SETUP-4 and OS-NATIVE-SETUP-5 in our more complete instruction.

OPNFV-NATIVE-SETUP-5: Create Neutron routers ipv4-router and ipv6-router which need to provide external connectivity.

neutron router-create ipv4-router
neutron router-create ipv6-router

OPNFV-NATIVE-SETUP-6: Create an external network/subnet ext-net using the appropriate values based on the data-center physical network setup.

Please NOTE that you may only need to create the subnet of ext-net because OPNFV installers should have created an external network during installation. You must use the same name of external network that installer creates when you create the subnet. For example:

  • Apex installer: external
  • Compass installer: ext-net
  • Fuel installer: admin_floating_net
  • Joid installer: ext-net

Please refer to the documentation of installers if there is any issue

# This is needed only if installer does not create an external work
# Otherwise, skip this command "net-create"
neutron net-create --router:external ext-net

# Note that the name "ext-net" may work for some installers such as Compass and Joid
# Change the name "ext-net" to match the name of external network that an installer creates
neutron subnet-create --disable-dhcp --allocation-pool start=198.59.156.251,\
end=198.59.156.254 --gateway 198.59.156.1 ext-net 198.59.156.0/24

OPNFV-NATIVE-SETUP-7: Create Neutron networks ipv4-int-network1 and ipv6-int-network2 with port_security disabled

neutron net-create ipv4-int-network1
neutron net-create ipv6-int-network2

OPNFV-NATIVE-SETUP-8: Create IPv4 subnet ipv4-int-subnet1 in the internal network ipv4-int-network1, and associate it to ipv4-router.

neutron subnet-create --name ipv4-int-subnet1 --dns-nameserver 8.8.8.8 \
ipv4-int-network1 20.0.0.0/24

neutron router-interface-add ipv4-router ipv4-int-subnet1

OPNFV-NATIVE-SETUP-9: Associate the ext-net to the Neutron routers ipv4-router and ipv6-router.

# Note that the name "ext-net" may work for some installers such as Compass and Joid
# Change the name "ext-net" to match the name of external network that an installer creates
neutron router-gateway-set ipv4-router ext-net
neutron router-gateway-set ipv6-router ext-net

OPNFV-NATIVE-SETUP-10: Create two subnets, one IPv4 subnet ipv4-int-subnet2 and one IPv6 subnet ipv6-int-subnet2 in ipv6-int-network2, and associate both subnets to ipv6-router

neutron subnet-create --name ipv4-int-subnet2 --dns-nameserver 8.8.8.8 \
ipv6-int-network2 10.0.0.0/24

neutron subnet-create --name ipv6-int-subnet2 --ip-version 6 --ipv6-ra-mode slaac \
--ipv6-address-mode slaac ipv6-int-network2 2001:db8:0:1::/64

neutron router-interface-add ipv6-router ipv4-int-subnet2
neutron router-interface-add ipv6-router ipv6-int-subnet2

OPNFV-NATIVE-SETUP-11: Create a keypair

nova keypair-add vRouterKey > ~/vRouterKey

OPNFV-NATIVE-SETUP-12: Create ports for vRouter (with some specific MAC address - basically for automation - to know the IPv6 addresses that would be assigned to the port).

neutron port-create --name eth0-vRouter --mac-address fa:16:3e:11:11:11 ipv6-int-network2
neutron port-create --name eth1-vRouter --mac-address fa:16:3e:22:22:22 ipv4-int-network1

OPNFV-NATIVE-SETUP-13: Create ports for VM1 and VM2.

neutron port-create --name eth0-VM1 --mac-address fa:16:3e:33:33:33 ipv4-int-network1
neutron port-create --name eth0-VM2 --mac-address fa:16:3e:44:44:44 ipv4-int-network1

OPNFV-NATIVE-SETUP-14: Update ipv6-router with routing information to subnet 2001:db8:0:2::/64

neutron router-update ipv6-router --routes type=dict list=true \
destination=2001:db8:0:2::/64,nexthop=2001:db8:0:1:f816:3eff:fe11:1111

OPNFV-NATIVE-SETUP-15: Boot Service VM (vRouter), VM1 and VM2

nova boot --image Fedora22 --flavor m1.small \
--user-data /opt/stack/opnfv_os_ipv6_poc/metadata.txt \
--availability-zone nova:opnfv-os-compute \
--nic port-id=$(neutron port-list | grep -w eth0-vRouter | awk '{print $2}') \
--nic port-id=$(neutron port-list | grep -w eth1-vRouter | awk '{print $2}') \
--key-name vRouterKey vRouter

nova list

# Please wait for some 10 to 15 minutes so that necessary packages (like radvd)
# are installed and vRouter is up.
nova console-log vRouter

nova boot --image cirros-0.3.4-x86_64-uec --flavor m1.tiny \
--user-data /opt/stack/opnfv_os_ipv6_poc/set_mtu.sh \
--availability-zone nova:opnfv-os-controller \
--nic port-id=$(neutron port-list | grep -w eth0-VM1 | awk '{print $2}') \
--key-name vRouterKey VM1

nova boot --image cirros-0.3.4-x86_64-uec --flavor m1.tiny
--user-data /opt/stack/opnfv_os_ipv6_poc/set_mtu.sh \
--availability-zone nova:opnfv-os-compute \
--nic port-id=$(neutron port-list | grep -w eth0-VM2 | awk '{print $2}') \
--key-name vRouterKey VM2

nova list # Verify that all the VMs are in ACTIVE state.

OPNFV-NATIVE-SETUP-16: If all goes well, the IPv6 addresses assigned to the VMs would be as shown as follows:

# vRouter eth0 interface would have the following IPv6 address:
#     2001:db8:0:1:f816:3eff:fe11:1111/64
# vRouter eth1 interface would have the following IPv6 address:
#     2001:db8:0:2::1/64
# VM1 would have the following IPv6 address:
#     2001:db8:0:2:f816:3eff:fe33:3333/64
# VM2 would have the following IPv6 address:
#     2001:db8:0:2:f816:3eff:fe44:4444/64

OPNFV-NATIVE-SETUP-17: Now we need to disable eth0-VM1, eth0-VM2, eth0-vRouter and eth1-vRouter port-security

for port in eth0-VM1 eth0-VM2 eth0-vRouter eth1-vRouter
do
    neutron port-update --no-security-groups $port
    neutron port-update $port --port-security-enabled=False
    neutron port-show $port | grep port_security_enabled
done

OPNFV-NATIVE-SETUP-18: Now we can SSH to VMs. You can execute the following command.

# 1. Create a floatingip and associate it with VM1, VM2 and vRouter (to the port id that is passed).
#    Note that the name "ext-net" may work for some installers such as Compass and Joid
#    Change the name "ext-net" to match the name of external network that an installer creates
neutron floatingip-create --port-id $(neutron port-list | grep -w eth0-VM1 | \
awk '{print $2}') ext-net
neutron floatingip-create --port-id $(neutron port-list | grep -w eth0-VM2 | \
awk '{print $2}') ext-net
neutron floatingip-create --port-id $(neutron port-list | grep -w eth1-vRouter | \
awk '{print $2}') ext-net

# 2. To know / display the floatingip associated with VM1, VM2 and vRouter.
neutron floatingip-list -F floating_ip_address -F port_id | grep $(neutron port-list | \
grep -w eth0-VM1 | awk '{print $2}') | awk '{print $2}'
neutron floatingip-list -F floating_ip_address -F port_id | grep $(neutron port-list | \
grep -w eth0-VM2 | awk '{print $2}') | awk '{print $2}'
neutron floatingip-list -F floating_ip_address -F port_id | grep $(neutron port-list | \
grep -w eth1-vRouter | awk '{print $2}') | awk '{print $2}'

# 3. To ssh to the vRouter, VM1 and VM2, user can execute the following command.
ssh -i ~/vRouterKey fedora@<floating-ip-of-vRouter>
ssh -i ~/vRouterKey cirros@<floating-ip-of-VM1>
ssh -i ~/vRouterKey cirros@<floating-ip-of-VM2>

If everything goes well, ssh will be successful and you will be logged into those VMs. Run some commands to verify that IPv6 addresses are configured on eth0 interface.

OPNFV-NATIVE-SETUP-19: Show an IPv6 address with a prefix of 2001:db8:0:2::/64

ip address show

OPNFV-NATIVE-SETUP-20: ping some external IPv6 address, e.g. ipv6-router

ping6 2001:db8:0:1::1

If the above ping6 command succeeds, it implies that vRouter was able to successfully forward the IPv6 traffic to reach external ipv6-router.

2. IPv6 Post Installation Procedures

Congratulations, you have completed the setup of using a service VM to act as an IPv6 vRouter. You have validated the setup based on the instruction in previous sections. If you want to further test your setup, you can ping6 among VM1, VM2, vRouter and ipv6-router.

This setup allows further open innovation by any 3rd-party.

2.1. Automated post installation activities

Refer to the relevant testing guides, results, and release notes of Yardstick Project.

Using IPv6 Feature of Euphrates Release
Abstract:

This section provides the users with gap analysis regarding IPv6 feature requirements with OpenStack Ocata Official Release and Open Daylight Carbon Official Release. The gap analysis serves as feature specific user guides and references when as a user you may leverage the IPv6 feature in the platform and need to perform some IPv6 related operations.

For more information, please find Neutron’s IPv6 document for Ocata Release.

1. IPv6 Gap Analysis with OpenStack Ocata

This section provides users with IPv6 gap analysis regarding feature requirement with OpenStack Neutron in Ocata Official Release. The following table lists the use cases / feature requirements of VIM-agnostic IPv6 functionality, including infrastructure layer and VNF (VM) layer, and its gap analysis with OpenStack Neutron in Ocata Official Release.

Please NOTE that in terms of IPv6 support in OpenStack Neutron, there is no difference between Ocata release and Newton release.

Use Case / Requirement Supported in Ocata Notes
All topologies work in a multi-tenant environment Yes The IPv6 design is following the Neutron tenant networks model; dnsmasq is being used inside DHCP network namespaces, while radvd is being used inside Neutron routers namespaces to provide full isolation between tenants. Tenant isolation can be based on VLANs, GRE, or VXLAN encapsulation. In case of overlays, the transport network (and VTEPs) must be IPv4 based as of today.
IPv6 VM to VM only Yes It is possible to assign IPv6-only addresses to VMs. Both switching (within VMs on the same tenant network) as well as east/west routing (between different networks of the same tenant) are supported.
IPv6 external L2 VLAN directly attached to a VM Yes IPv6 provider network model; RA messages from upstream (external) router are forwarded into the VMs

IPv6 subnet routed via L3 agent to an external IPv6 network

  1. Both VLAN and overlay (e.g. GRE, VXLAN) subnet attached to VMs;
  2. Must be able to support multiple L3 agents for a given external network to support scaling (neutron scheduler to assign vRouters to the L3 agents)
  1. Yes
  2. Yes
Configuration is enhanced since Kilo to allow easier setup of the upstream gateway, without the user being forced to create an IPv6 subnet for the external network.

Ability for a NIC to support both IPv4 and IPv6 (dual stack) address.

  1. VM with a single interface associated with a network, which is then associated with two subnets.
  2. VM with two different interfaces associated with two different networks and two different subnets.
  1. Yes
  2. Yes
Dual-stack is supported in Neutron with the addition of Multiple IPv6 Prefixes Blueprint

Support IPv6 Address assignment modes.

  1. SLAAC
  2. DHCPv6 Stateless
  3. DHCPv6 Stateful
  1. Yes
  2. Yes
  3. Yes
 
Ability to create a port on an IPv6 DHCPv6 Stateful subnet and assign a specific IPv6 address to the port and have it taken out of the DHCP address pool. Yes  
Ability to create a port with fixed_ip for a SLAAC/DHCPv6-Stateless Subnet. No The following patch disables this operation: https://review.openstack.org/#/c/129144/
Support for private IPv6 to external IPv6 floating IP; Ability to specify floating IPs via Neutron API (REST and CLI) as well as via Horizon, including combination of IPv6/IPv4 and IPv4/IPv6 floating IPs if implemented. Rejected Blueprint proposed in upstream and got rejected. General expectation is to avoid NAT with IPv6 by assigning GUA to tenant VMs. See https://review.openstack.org/#/c/139731/ for discussion.
Provide IPv6/IPv4 feature parity in support for pass-through capabilities (e.g., SR-IOV). To-Do The L3 configuration should be transparent for the SR-IOV implementation. SR-IOV networking support introduced in Juno based on the sriovnicswitch ML2 driver is expected to work with IPv4 and IPv6 enabled VMs. We need to verify if it works or not.
Additional IPv6 extensions, for example: IPSEC, IPv6 Anycast, Multicast No It does not appear to be considered yet (lack of clear requirements)
VM access to the meta-data server to obtain user data, SSH keys, etc. using cloud-init with IPv6 only interfaces. No This is currently not supported. Config-drive or dual-stack IPv4 / IPv6 can be used as a workaround (so that the IPv4 network is used to obtain connectivity with the metadata service). The following blog How to Use Config-Drive for Metadata with IPv6 Network provides a neat summary on how to use config-drive for metadata with IPv6 network.
Full support for IPv6 matching (i.e., IPv6, ICMPv6, TCP, UDP) in security groups. Ability to control and manage all IPv6 security group capabilities via Neutron/Nova API (REST and CLI) as well as via Horizon. Yes Both IPTables firewall driver and OVS firewall driver support IPv6 Security Group API.
During network/subnet/router create, there should be an option to allow user to specify the type of address management they would like. This includes all options including those low priority if implemented (e.g., toggle on/off router and address prefix advertisements); It must be supported via Neutron API (REST and CLI) as well as via Horizon Yes

Two new Subnet attributes were introduced to control IPv6 address assignment options:

  • ipv6-ra-mode: to determine who sends Router Advertisements;
  • ipv6-address-mode: to determine how VM obtains IPv6 address, default gateway, and/or optional information.
Security groups anti-spoofing: Prevent VM from using a source IPv6/MAC address which is not assigned to the VM Yes  
Protect tenant and provider network from rogue RAs Yes When using a tenant network, Neutron is going to automatically handle the filter rules to allow connectivity of RAs to the VMs only from the Neutron router port; with provider networks, users are required to specify the LLA of the upstream router during the subnet creation, or otherwise manually edit the security-groups rules to allow incoming traffic from this specific address.
Support the ability to assign multiple IPv6 addresses to an interface; both for Neutron router interfaces and VM interfaces. Yes  
Ability for a VM to support a mix of multiple IPv4 and IPv6 networks, including multiples of the same type. Yes  
IPv6 Support in “Allowed Address Pairs” Extension Yes  
Support for IPv6 Prefix Delegation. Yes Partial support in Ocata
Distributed Virtual Routing (DVR) support for IPv6 No In Ocata DVR implementation, IPv6 works. But all the IPv6 ingress/ egress traffic is routed via the centralized controller node, i.e. similar to SNAT traffic. A fully distributed IPv6 router is not yet supported in Neutron.
VPNaaS Yes VPNaaS supports IPv6. But this feature is not extensively tested.
FWaaS Yes  
BGP Dynamic Routing Support for IPv6 Prefixes Yes BGP Dynamic Routing supports peering via IPv6 and advertising IPv6 prefixes.
VxLAN Tunnels with IPv6 endpoints. Yes Neutron ML2/OVS supports configuring local_ip with IPv6 address so that VxLAN tunnels are established with IPv6 addresses. This feature requires OVS 2.6 or higher version.
IPv6 First-Hop Security, IPv6 ND spoofing Yes  
IPv6 support in Neutron Layer3 High Availability (keepalived+VRRP). Yes  
2. IPv6 Gap Analysis with Open Daylight Carbon

This section provides users with IPv6 gap analysis regarding feature requirement with Open Daylight Carbon Official Release. The following table lists the use cases / feature requirements of VIM-agnostic IPv6 functionality, including infrastructure layer and VNF (VM) layer, and its gap analysis with Open Daylight Carbon Official Release.

Open Daylight Carbon Status

In Open Daylight Carbon official release, the legacy Old Netvirt identified by feature odl-ovsdb-openstack is deprecated and no longer supported. The New Netvirt identified by feature odl-netvirt-openstack is used.

Use Case / Requirement Supported in ODL Carbon Notes
REST API support for IPv6 subnet creation in ODL Yes

Yes, it is possible to create IPv6 subnets in ODL using Neutron REST API.

For a network which has both IPv4 and IPv6 subnets, ODL mechanism driver will send the port information which includes IPv4/v6 addresses to ODL Neutron northbound API. When port information is queried, it displays IPv4 and IPv6 addresses.

IPv6 Router support in ODL:

  1. Communication between VMs on same network
Yes  

IPv6 Router support in ODL:

  1. Communication between VMs on different networks connected to the same router (east-west)
Yes  

IPv6 Router support in ODL:

  1. External routing (north-south)
Work in Progress Work in progress.

IPAM: Support for IPv6 Address assignment modes.

  1. SLAAC
  2. DHCPv6 Stateless
  3. DHCPv6 Stateful
Yes ODL IPv6 Router supports all the IPv6 Address assignment modes along with Neutron DHCP Agent.
When using ODL for L2 forwarding/tunneling, it is compatible with IPv6. Yes  
Full support for IPv6 matching (i.e. IPv6, ICMPv6, TCP, UDP) in security groups. Ability to control and manage all IPv6 security group capabilities via Neutron/Nova API (REST and CLI) as well as via Horizon Yes  
Shared Networks support Yes  
IPv6 external L2 VLAN directly attached to a VM. ToDo  
ODL on an IPv6 only Infrastructure. Work in Progress Deploying OpenStack with ODL on an IPv6 only infrastructure where the API endpoints are all IPv6 addresses.
VxLAN Tunnels with IPv6 Endpoints Yes  

Joid

JOID installation instruction
1. Abstract

This document will explain how to install the Euphrates release of OPNFV with JOID including installing JOID, configuring JOID for your environment, and deploying OPNFV with different SDN solutions in HA, or non-HA mode.

2. Introduction
2.1. JOID in brief

JOID as Juju OPNFV Infrastructure Deployer allows you to deploy different combinations of OpenStack release and SDN solution in HA or non-HA mode. For OpenStack, JOID currently supports Newton and Ocata. For SDN, it supports Openvswitch, OpenContrail, OpenDayLight, and ONOS. In addition to HA or non-HA mode, it also supports deploying from the latest development tree.

JOID heavily utilizes the technology developed in Juju and MAAS.

Juju is a state-of-the-art, open source modelling tool for operating software in the cloud. Juju allows you to deploy, configure, manage, maintain, and scale cloud applications quickly and efficiently on public clouds, as well as on physical servers, OpenStack, and containers. You can use Juju from the command line or through its beautiful GUI. (source: Juju Docs)

MAAS is Metal As A Service. It lets you treat physical servers like virtual machines (instances) in the cloud. Rather than having to manage each server individually, MAAS turns your bare metal into an elastic cloud-like resource. Machines can be quickly provisioned and then destroyed again as easily as you can with instances in a public cloud. ... In particular, it is designed to work especially well with Juju, the service and model management service. It’s a perfect arrangement: MAAS manages the machines and Juju manages the services running on those machines. (source: MAAS Docs)

2.2. Typical JOID Architecture

The MAAS server is installed and configured on Jumphost with Ubuntu 16.04 LTS server with access to the Internet. Another VM is created to be managed by MAAS as a bootstrap node for Juju. The rest of the resources, bare metal or virtual, will be registered and provisioned in MAAS. And finally the MAAS environment details are passed to Juju for use.

3. Setup Requirements
3.1. Network Requirements

Minimum 2 Networks:

  • One for the administrative network with gateway to access the Internet
  • One for the OpenStack public network to access OpenStack instances via floating IPs

JOID supports multiple isolated networks for data as well as storage based on your network requirement for OpenStack.

No DHCP server should be up and configured. Configure gateways only on eth0 and eth1 networks to access the network outside your lab.

3.2. Jumphost Requirements

The Jumphost requirements are outlined below:

  • OS: Ubuntu 16.04 LTS Server
  • Root access.
  • CPU cores: 16
  • Memory: 32GB
  • Hard Disk: 1× (min. 250 GB)
  • NIC: eth0 (admin, management), eth1 (external connectivity)
3.3. Physical nodes requirements (bare metal deployment)

Besides Jumphost, a minimum of 5 physical servers for bare metal environment.

  • CPU cores: 16
  • Memory: 32GB
  • Hard Disk: 2× (500GB) prefer SSD
  • NIC: eth0 (Admin, Management), eth1 (external network)

NOTE: Above configuration is minimum. For better performance and usage of the OpenStack, please consider higher specs for all nodes.

Make sure all servers are connected to top of rack switch and configured accordingly.

4. Bare Metal Installation

Before proceeding, make sure that your hardware infrastructure satisfies the Setup Requirements.

4.1. Networking

Make sure you have at least two networks configured:

  1. Admin (management) network with gateway to access the Internet (for downloading installation resources).
  2. public/floating network to consume by tenants for floating IPs.

You may configure other networks, e.g. for data or storage, based on your network options for Openstack.

4.2. Jumphost installation and configuration
  1. Install Ubuntu 16.04 (Xenial) LTS server on Jumphost (one of the physical nodes).

    Tip

    Use ubuntu as username as password, as this matches the MAAS credentials installed later.

    During the OS installation, install the OpenSSH server package to allow SSH connections to the Jumphost.

    If the data size of the image is too big or slow (e.g. when mounted through a slow virtual console), you can also use the Ubuntu mini ISO. Install packages: standard system utilities, basic Ubuntu server, OpenSSH server, Virtual Machine host.

    If you have issues with blank console after booting, see this SO answer and set nomodeset, (removing quiet splash can also be useful to see log during booting) either through console in recovery mode or via SSH (if installed).

  2. Install git and bridge-utils packages

    sudo apt install git bridge-utils
    
  3. Configure bridges for each network to be used.

    Example /etc/network/interfaces file:

    source /etc/network/interfaces.d/*
    
    # The loopback network interface (set by Ubuntu)
    auto lo
    iface lo inet loopback
    
    # Admin network interface
    iface eth0 inet manual
    auto brAdmin
    iface brAdmin inet static
            bridge_ports eth0
            address 10.5.1.1
            netmask 255.255.255.0
    
    # Ext. network for floating IPs
    iface eth1 inet manual
    auto brExt
    iface brExt inet static
            bridge_ports eth1
            address 10.5.15.1
            netmask 255.255.255.0
    

    Note

    If you choose to use the separate network for management, public, data and storage, then you need to create bridge for each interface. In case of VLAN tags, use the appropriate network on Jumphost depending on the VLAN ID on the interface.

    Note

    Both of the networks need to have Internet connectivity. If only one of your interfaces has Internet access, you can setup IP forwarding. For an example how to accomplish that, see the script in Nokia pod 1 deployment (labconfig/nokia/pod1/setup_ip_forwarding.sh).

4.3. Configure JOID for your lab

All configuration for the JOID deployment is specified in a labconfig.yaml file. Here you describe all your physical nodes, their roles in OpenStack, their network interfaces, IPMI parameters etc. It’s also where you configure your OPNFV deployment and MAAS networks/spaces. You can find example configuration files from already existing nodes in the repository.

First of all, download JOID to your Jumphost. We recommend doing this in your home directory.

git clone https://gerrit.opnfv.org/gerrit/p/joid.git

Tip

You can select the stable version of your choice by specifying the git branch, for example:

git clone -b stable/danube https://gerrit.opnfv.org/gerrit/p/joid.git

Create a directory in joid/labconfig/<company_name>/<pod_number>/ and create or copy a labconfig.yaml configuration file to that directory. For example:

# All JOID actions are done from the joid/ci directory
cd joid/ci
mkdir -p ../labconfig/your_company/pod1
cp ../labconfig/nokia/pod1/labconfig.yaml ../labconfig/your_company/pod1/

Example labconfig.yaml configuration file:

lab:
  location: your_company
  racks:
  - rack: pod1
    nodes:
    - name: rack-1-m1
      architecture: x86_64
      roles: [network,control]
      nics:
      - ifname: eth0
        spaces: [admin]
        mac: ["12:34:56:78:9a:bc"]
      - ifname: eth1
        spaces: [floating]
        mac: ["12:34:56:78:9a:bd"]
      power:
        type: ipmi
        address: 192.168.10.101
        user: admin
        pass: admin
    - name: rack-1-m2
      architecture: x86_64
      roles: [compute,control,storage]
      nics:
      - ifname: eth0
        spaces: [admin]
        mac: ["23:45:67:89:ab:cd"]
      - ifname: eth1
        spaces: [floating]
        mac: ["23:45:67:89:ab:ce"]
      power:
        type: ipmi
        address: 192.168.10.102
        user: admin
        pass: admin
    - name: rack-1-m3
      architecture: x86_64
      roles: [compute,control,storage]
      nics:
      - ifname: eth0
        spaces: [admin]
        mac: ["34:56:78:9a:bc:de"]
      - ifname: eth1
        spaces: [floating]
        mac: ["34:56:78:9a:bc:df"]
      power:
        type: ipmi
        address: 192.168.10.103
        user: admin
        pass: admin
    - name: rack-1-m4
      architecture: x86_64
      roles: [compute,storage]
      nics:
      - ifname: eth0
        spaces: [admin]
        mac: ["45:67:89:ab:cd:ef"]
      - ifname: eth1
        spaces: [floating]
        mac: ["45:67:89:ab:ce:f0"]
      power:
        type: ipmi
        address: 192.168.10.104
        user: admin
        pass: admin
    - name: rack-1-m5
      architecture: x86_64
      roles: [compute,storage]
      nics:
      - ifname: eth0
        spaces: [admin]
        mac: ["56:78:9a:bc:de:f0"]
      - ifname: eth1
        spaces: [floating]
        mac: ["56:78:9a:bc:df:f1"]
      power:
        type: ipmi
        address: 192.168.10.105
        user: admin
        pass: admin
    floating-ip-range: 10.5.15.6,10.5.15.250,10.5.15.254,10.5.15.0/24
    ext-port: "eth1"
    dns: 8.8.8.8
opnfv:
    release: d
    distro: xenial
    type: noha
    openstack: ocata
    sdncontroller:
    - type: nosdn
    storage:
    - type: ceph
      disk: /dev/sdb
    feature: odl_l2
    spaces:
    - type: admin
      bridge: brAdmin
      cidr: 10.5.1.0/24
      gateway:
      vlan:
    - type: floating
      bridge: brExt
      cidr: 10.5.15.0/24
      gateway: 10.5.15.1
      vlan:

Once you have prepared the configuration file, you may begin with the automatic MAAS deployment.

4.4. MAAS Install

This section will guide you through the MAAS deployment. This is the first of two JOID deployment steps.

Note

For all the commands in this document, please do not use a root user account to run but instead use a non-root user account. We recommend using the ubuntu user as described above.

If you have already enabled maas for your environment and installed it then there is no need to enabled it again or install it. If you have patches from previous MAAS install, then you can apply them here.

Pre-installed MAAS without using the 03-maasdeploy.sh script is not supported. We strongly suggest to use 03-maasdeploy.sh script to deploy the MAAS and JuJu environment.

With the labconfig.yaml configuration file ready, you can start the MAAS deployment. In the joid/ci directory, run the following command:

# in joid/ci directory
./03-maasdeploy.sh custom <absolute path of config>/labconfig.yaml

If you prefer, you can also host your labconfig.yaml file remotely and JOID will download it from there. Just run

# in joid/ci directory
./03-maasdeploy.sh custom http://<web_site_location>/labconfig.yaml

This step will take approximately 30 minutes to a couple of hours depending on your environment. This script will do the following:

  • If this is your first time running this script, it will download all the required packages.
  • Install MAAS on the Jumphost.
  • Configure MAAS to enlist and commission a VM for Juju bootstrap node.
  • Configure MAAS to enlist and commission bare metal servers.
  • Download and load Ubuntu server images to be used by MAAS.

Already during deployment, once MAAS is installed, configured and launched, you can visit the MAAS Web UI and observe the progress of the deployment. Simply open the IP of your jumphost in a web browser and navigate to the /MAAS directory (e.g. http://10.5.1.1/MAAS in our example). You can login with username ubuntu and password ubuntu. In the Nodes page, you can see the bootstrap node and the bare metal servers and their status.

Hint

If you need to re-run this step, first undo the performed actions by running

# in joid/ci
./cleanvm.sh
./cleanmaas.sh
# now you can run the ./03-maasdeploy.sh script again
4.5. Juju Install

This section will guide you through the Juju an OPNFV deployment. This is the second of two JOID deployment steps.

JOID allows you to deploy different combinations of OpenStack and SDN solutions in HA or no-HA mode. For OpenStack, it supports Newton and Ocata. For SDN, it supports Open vSwitch, OpenContrail, OpenDaylight and ONOS (Open Network Operating System). In addition to HA or no-HA mode, it also supports deploying the latest from the development tree (tip).

To deploy OPNFV on the previously deployed MAAS system, use the deploy.sh script. For example:

# in joid/ci directory
./deploy.sh -d xenial -m openstack -o ocata -s nosdn -f none -t noha -l custom

The above command starts an OPNFV deployment with Ubuntu Xenial (16.04) distro, OpenStack model, Ocata version of OpenStack, Open vSwitch (and no other SDN), no special features, no-HA OpenStack mode and with custom labconfig. I.e. this corresponds to the os-nosdn-nofeature-noha OPNFV deployment scenario.

Note

You can see the usage info of the script by running

./deploy.sh --help

Possible script arguments are as follows.

Ubuntu distro to deploy

[-d <trusty|xenial>]
  • trusty: Ubuntu 16.04.
  • xenial: Ubuntu 17.04.

Model to deploy

[-m <openstack|kubernetes>]

JOID introduces two various models to deploy.

  • openstack: Openstack, which will be used for KVM/LXD container-based workloads.
  • kubernetes: Kubernetes model will be used for docker-based workloads.

Version of Openstack deployed

[-o <newton|mitaka>]
  • newton: Newton version of OpenStack.
  • ocata: Ocata version of OpenStack.

SDN controller

[-s <nosdn|odl|opencontrail|onos>]
  • nosdn: Open vSwitch only and no other SDN.
  • odl: OpenDayLight Boron version.
  • opencontrail: OpenContrail SDN.
  • onos: ONOS framework as SDN.

Feature to deploy (comma separated list)

[-f <lxd|dvr|sfc|dpdk|ipv6|none>]
  • none: No special feature will be enabled.
  • ipv6: IPv6 will be enabled for tenant in OpenStack.
  • lxd: With this feature hypervisor will be LXD rather than KVM.
  • dvr: Will enable distributed virtual routing.
  • dpdk: Will enable DPDK feature.
  • sfc: Will enable sfc feature only supported with ONOS deployment.
  • lb: Load balancing in case of Kubernetes will be enabled.

Mode of Openstack deployed

[-t <noha|ha|tip>]
  • noha: No High Availability.
  • ha: High Availability.
  • tip: The latest from the development tree.

Where to deploy

[-l <custom|default|...>]
  • custom: For bare metal deployment where labconfig.yaml was provided externally and not part of JOID package.
  • default: For virtual deployment where installation will be done on KVM created using 03-maasdeploy.sh.

Architecture

[-a <amd64|ppc64el|aarch64>]
  • amd64: Only x86 architecture will be used. Future version will support arm64 as well.

This step may take up to a couple of hours, depending on your configuration, internet connectivity etc. You can check the status of the deployment by running this command in another terminal:

watch juju status --format tabular

Hint

If you need to re-run this step, first undo the performed actions by running

# in joid/ci
./clean.sh
# now you can run the ./deploy.sh script again
4.6. OPNFV Scenarios in JOID

Following OPNFV scenarios can be deployed using JOID. Separate yaml bundle will be created to deploy the individual scenario.

Scenario Owner Known Issues
os-nosdn-nofeature-ha Joid  
os-nosdn-nofeature-noha Joid  
os-odl_l2-nofeature-ha Joid Floating ips are not working on this deployment.
os-nosdn-lxd-ha Joid Yardstick team is working to support.
os-nosdn-lxd-noha Joid Yardstick team is working to support.
os-onos-nofeature-ha ONOSFW  
os-onos-sfc-ha ONOSFW  
k8-nosdn-nofeature-noha Joid No support from Functest and Yardstick
k8-nosdn-lb-noha Joid No support from Functest and Yardstick
4.7. Troubleshoot

By default debug is enabled in script and error messages will be printed on ssh terminal where you are running the scripts.

Logs are indispensable when it comes time to troubleshoot. If you want to see all the service unit deployment logs, you can run juju debug-log in another terminal. The debug-log command shows the consolidated logs of all Juju agents (machine and unit logs) running in the environment.

To view a single service unit deployment log, use juju ssh to access to the deployed unit. For example to login into nova-compute unit and look for /var/log/juju/unit-nova-compute-0.log for more info:

ubuntu@R4N4B1:~$ juju ssh nova-compute/0
Warning: Permanently added '172.16.50.60' (ECDSA) to the list of known hosts.
Warning: Permanently added '3-r4n3b1-compute.maas' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 3.13.0-77-generic x86_64)

* Documentation:  https://help.ubuntu.com/
<skipped>
Last login: Tue Feb  2 21:23:56 2016 from bootstrap.maas
ubuntu@3-R4N3B1-compute:~$ sudo -i
root@3-R4N3B1-compute:~# cd /var/log/juju/
root@3-R4N3B1-compute:/var/log/juju# ls
machine-2.log  unit-ceilometer-agent-0.log  unit-ceph-osd-0.log  unit-neutron-contrail-0.log  unit-nodes-compute-0.log  unit-nova-compute-0.log  unit-ntp-0.log
root@3-R4N3B1-compute:/var/log/juju#

Note

By default Juju will add the Ubuntu user keys for authentication into the deployed server and only ssh access will be available.

Once you resolve the error, go back to the jump host to rerun the charm hook with

$ juju resolved --retry <unit>

If you would like to start over, run juju destroy-environment <environment name> to release the resources, then you can run deploy.sh again.

To access of any of the nodes or containers, use

juju ssh <service name>/<instance id>

For example:

juju ssh openstack-dashboard/0
juju ssh nova-compute/0
juju ssh neutron-gateway/0

You can see the available nodes and containers by running

juju status

All charm log files are available under /var/log/juju.


If you have questions, you can join the JOID channel #opnfv-joid on Freenode.

4.8. Common Issues

The following are the common issues we have collected from the community:

  • The right variables are not passed as part of the deployment procedure.

    ./deploy.sh -o newton -s nosdn -t ha -l custom -f none
    
  • If you have not setup MAAS with 03-maasdeploy.sh then the ./clean.sh command could hang, the juju status command may hang because the correct MAAS API keys are not mentioned in cloud listing for MAAS.

    _Solution_: Please make sure you have an MAAS cloud listed using juju clouds and the correct MAAS API key has been added.

  • Deployment times out: use the command juju status and make sure all service containers receive an IP address and they are executing code. Ensure there is no service in the error state.

  • In case the cleanup process hangs,run the juju destroy-model command manually.

Direct console access via the OpenStack GUI can be quite helpful if you need to login to a VM but cannot get to it over the network. It can be enabled by setting the console-access-protocol in the nova-cloud-controller to vnc. One option is to directly edit the juju-deployer bundle and set it there prior to deploying OpenStack.

nova-cloud-controller:
  options:
    console-access-protocol: vnc

To access the console, just click on the instance in the OpenStack GUI and select the Console tab.

5. Virtual Installation

The virtual deployment of JOID is very simple and does not require any special configuration. To deploy a virtual JOID environment follow these few simple steps:

  1. Install a clean Ubuntu 16.04 (Xenial) server on the machine. You can use the tips noted in the first step of the Jumphost installation and configuration for bare metal deployment. However, no specialized configuration is needed, just make sure you have Internet connectivity.

  2. Run the MAAS deployment for virtual deployment without customized labconfig file:

    # in joid/ci directory
    ./03-maasdeploy.sh
    
  3. Run the Juju/OPNFV deployment with your desired configuration parameters, but with -l default -i 1 for virtual deployment. For example to deploy the Kubernetes model:

    # in joid/ci directory
    ./deploy.sh -d xenial -s nosdn -t noha -f none -m kubernetes -l default -i 1
    

Now you should have a working JOID deployment with three virtual nodes. In case of any issues, refer to the Troubleshoot section.

6. Post Installation
6.1. Testing Your Deployment

Once Juju deployment is complete, use juju status to verify that all deployed units are in the _Ready_ state.

Find the OpenStack dashboard IP address from the juju status output, and see if you can login via a web browser. The domain, username and password are admin_domain, admin and openstack.

Optionally, see if you can log in to the Juju GUI. Run juju gui to see the login details.

If you deploy OpenDaylight, OpenContrail or ONOS, find the IP address of the web UI and login. Please refer to each SDN bundle.yaml for the login username/password.

Note

If the deployment worked correctly, you can get easier access to the web dashboards with the setupproxy.sh script described in the next section.

6.2. Create proxies to the dashboards

MAAS, Juju and OpenStack/Kubernetes all come with their own web-based dashboards. However, they might be on private networks and require SSH tunnelling to see them. To simplify access to them, you can use the following script to configure the Apache server on Jumphost to work as a proxy to Juju and OpenStack/Kubernetes dashboards. Furthermore, this script also creates JOID deployment homepage with links to these dashboards, listing also their access credentials.

Simply run the following command after JOID has been deployed.

# run in joid/ci directory
# for OpenStack model:
./setupproxy.sh openstack
# for Kubernetes model:
./setupproxy.sh kubernetes

You can also use the -v argument for more verbose output with xtrace.

After the script has finished, it will print out the addresses and credentials to the dashboards. You can also find the JOID deployment homepage if you open the Jumphost’s IP address in your web browser.

6.3. Configuring OpenStack

At the end of the deployment, the admin-openrc with OpenStack login credentials will be created for you. You can source the file and start configuring OpenStack via CLI.

. ~/joid_config/admin-openrc

The script openstack.sh under joid/ci can be used to configure the OpenStack after deployment.

./openstack.sh <nosdn> custom xenial newton

Below commands are used to setup domain in heat.

juju run-action heat/0 domain-setup

Upload cloud images and creates the sample network to test.

joid/juju/get-cloud-images
joid/juju/joid-configure-openstack
6.4. Configuring Kubernetes

The script k8.sh under joid/ci would be used to show the Kubernetes workload and create sample pods.

./k8.sh
6.5. Configuring OpenStack

At the end of the deployment, the admin-openrc with OpenStack login credentials will be created for you. You can source the file and start configuring OpenStack via CLI.

cat ~/joid_config/admin-openrc
export OS_USERNAME=admin
export OS_PASSWORD=openstack
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://172.16.50.114:5000/v2.0
export OS_REGION_NAME=RegionOne

We have prepared some scripts to help your configure the OpenStack cloud that you just deployed. In each SDN directory, for example joid/ci/opencontrail, there is a ‘scripts’ folder where you can find the scripts. These scripts are created to help you configure a basic OpenStack Cloud to verify the cloud. For more information on OpenStack Cloud configuration, please refer to the OpenStack Cloud Administrator Guide: http://docs.openstack.org/user-guide-admin/. Similarly, for complete SDN configuration, please refer to the respective SDN administrator guide.

Each SDN solution requires slightly different setup. Please refer to the README in each SDN folder. Most likely you will need to modify the openstack.sh and cloud-setup.sh scripts for the floating IP range, private IP network, and SSH keys. Please go through openstack.sh, glance.sh and cloud-setup.sh and make changes as you see fit.

Let’s take a look at those for the Open vSwitch and briefly go through each script so you know what you need to change for your own environment.

$ ls ~/joid/juju
configure-juju-on-openstack  get-cloud-images  joid-configure-openstack
6.6. openstack.sh

Let’s first look at openstack.sh. First there are 3 functions defined, configOpenrc(), unitAddress(), and unitMachine().

configOpenrc() {
  cat <<-EOF
      export SERVICE_ENDPOINT=$4
      unset SERVICE_TOKEN
      unset SERVICE_ENDPOINT
      export OS_USERNAME=$1
      export OS_PASSWORD=$2
      export OS_TENANT_NAME=$3
      export OS_AUTH_URL=$4
      export OS_REGION_NAME=$5
EOF
}

unitAddress() {
  if [[ "$jujuver" < "2" ]]; then
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"services\"][\"$1\"][\"units\"][\"$1/$2\"][\"public-address\"]" 2> /dev/null
  else
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"applications\"][\"$1\"][\"units\"][\"$1/$2\"][\"public-address\"]" 2> /dev/null
  fi
}

unitMachine() {
  if [[ "$jujuver" < "2" ]]; then
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"services\"][\"$1\"][\"units\"][\"$1/$2\"][\"machine\"]" 2> /dev/null
  else
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"applications\"][\"$1\"][\"units\"][\"$1/$2\"][\"machine\"]" 2> /dev/null
  fi
}

The function configOpenrc() creates the OpenStack login credentials, the function unitAddress() finds the IP address of the unit, and the function unitMachine() finds the machine info of the unit.

create_openrc() {
   keystoneIp=$(keystoneIp)
   if [[ "$jujuver" < "2" ]]; then
       adminPasswd=$(juju get keystone | grep admin-password -A 5 | grep value | awk '{print $2}' 2> /dev/null)
   else
       adminPasswd=$(juju config keystone | grep admin-password -A 5 | grep value | awk '{print $2}' 2> /dev/null)
   fi

   configOpenrc admin $adminPasswd admin http://$keystoneIp:5000/v2.0 RegionOne > ~/joid_config/admin-openrc
   chmod 0600 ~/joid_config/admin-openrc
}

This finds the IP address of the keystone unit 0, feeds in the OpenStack admin credentials to a new file name ‘admin-openrc’ in the ‘~/joid_config/’ folder and change the permission of the file. It’s important to change the credentials here if you use a different password in the deployment Juju charm bundle.yaml.

neutron net-show ext-net > /dev/null 2>&1 || neutron net-create ext-net \
                                               --router:external=True \
                                               --provider:network_type flat \
                                               --provider:physical_network physnet1
neutron subnet-show ext-subnet > /dev/null 2>&1 || neutron subnet-create ext-net \
  --name ext-subnet --allocation-pool start=$EXTNET_FIP,end=$EXTNET_LIP \
  --disable-dhcp --gateway $EXTNET_GW $EXTNET_NET

This section will create the ext-net and ext-subnet for defining the for floating ips.

openstack congress datasource create nova "nova" \
  --config username=$OS_USERNAME \
  --config tenant_name=$OS_TENANT_NAME \
  --config password=$OS_PASSWORD \
  --config auth_url=http://$keystoneIp:5000/v2.0

This section will create the congress datasource for various services. Each service datasource will have entry in the file.

6.7. get-cloud-images
folder=/srv/data/
sudo mkdir $folder || true

if grep -q 'virt-type: lxd' bundles.yaml; then
   URLS=" \
   http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-lxc.tar.gz \
   http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz "

else
   URLS=" \
   http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img \
   http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img \
   http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img \
   http://mirror.catn.com/pub/catn/images/qcow2/centos6.4-x86_64-gold-master.img \
   http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 \
   http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img "
fi

for URL in $URLS
do
FILENAME=${URL##*/}
if [ -f $folder/$FILENAME ];
then
   echo "$FILENAME already downloaded."
else
   wget  -O  $folder/$FILENAME $URL
fi
done

This section of the file will download the images to jumphost if not found to be used with openstack VIM.

Note

The image downloading and uploading might take too long and time out. In this case, use juju ssh glance/0 to log in to the glance unit 0 and run the script again, or manually run the glance commands.

6.8. joid-configure-openstack
source ~/joid_config/admin-openrc

First, source the the admin-openrc file.

::
#Upload images to glance glance image-create –name=”Xenial LXC x86_64” –visibility=public –container-format=bare –disk-format=root-tar –property architecture=”x86_64” < /srv/data/xenial-server-cloudimg-amd64-root.tar.gz glance image-create –name=”Cirros LXC 0.3” –visibility=public –container-format=bare –disk-format=root-tar –property architecture=”x86_64” < /srv/data/cirros-0.3.4-x86_64-lxc.tar.gz glance image-create –name=”Trusty x86_64” –visibility=public –container-format=ovf –disk-format=qcow2 < /srv/data/trusty-server-cloudimg-amd64-disk1.img glance image-create –name=”Xenial x86_64” –visibility=public –container-format=ovf –disk-format=qcow2 < /srv/data/xenial-server-cloudimg-amd64-disk1.img glance image-create –name=”CentOS 6.4” –visibility=public –container-format=bare –disk-format=qcow2 < /srv/data/centos6.4-x86_64-gold-master.img glance image-create –name=”Cirros 0.3” –visibility=public –container-format=bare –disk-format=qcow2 < /srv/data/cirros-0.3.4-x86_64-disk.img

Upload the images into Glance to be used for creating the VM.

# adjust tiny image
nova flavor-delete m1.tiny
nova flavor-create m1.tiny 1 512 8 1

Adjust the tiny image profile as the default tiny instance is too small for Ubuntu.

# configure security groups
neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol icmp --remote-ip-prefix 0.0.0.0/0 default
neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol tcp --port-range-min 22 --port-range-max 22 --remote-ip-prefix 0.0.0.0/0 default

Open up the ICMP and SSH access in the default security group.

# import key pair
keystone tenant-create --name demo --description "Demo Tenant"
keystone user-create --name demo --tenant demo --pass demo --email demo@demo.demo

nova keypair-add --pub-key id_rsa.pub ubuntu-keypair

Create a project called ‘demo’ and create a user called ‘demo’ in this project. Import the key pair.

# configure external network
neutron net-create ext-net --router:external --provider:physical_network external --provider:network_type flat --shared
neutron subnet-create ext-net --name ext-subnet --allocation-pool start=10.5.8.5,end=10.5.8.254 --disable-dhcp --gateway 10.5.8.1 10.5.8.0/24

This section configures an external network ‘ext-net’ with a subnet called ‘ext-subnet’. In this subnet, the IP pool starts at 10.5.8.5 and ends at 10.5.8.254. DHCP is disabled. The gateway is at 10.5.8.1, and the subnet mask is 10.5.8.0/24. These are the public IPs that will be requested and associated to the instance. Please change the network configuration according to your environment.

# create vm network
neutron net-create demo-net
neutron subnet-create --name demo-subnet --gateway 10.20.5.1 demo-net 10.20.5.0/24

This section creates a private network for the instances. Please change accordingly.

neutron router-create demo-router

neutron router-interface-add demo-router demo-subnet

neutron router-gateway-set demo-router ext-net

This section creates a router and connects this router to the two networks we just created.

# create pool of floating ips
i=0
while [ $i -ne 10 ]; do
  neutron floatingip-create ext-net
  i=$((i + 1))
done

Finally, the script will request 10 floating IPs.

6.8.1. configure-juju-on-openstack

This script can be used to do juju bootstrap on openstack so that Juju can be used as model tool to deploy the services and VNF on top of openstack using the JOID.

7. Appendices
7.1. Appendix A: Single Node Deployment

By default, running the script ./03-maasdeploy.sh will automatically create the KVM VMs on a single machine and configure everything for you.

if [ ! -e ./labconfig.yaml ]; then
    virtinstall=1
    labname="default"
    cp ../labconfig/default/labconfig.yaml ./
    cp ../labconfig/default/deployconfig.yaml ./

Please change joid/ci/labconfig/default/labconfig.yaml accordingly. The MAAS deployment script will do the following: 1. Create bootstrap VM. 2. Install MAAS on the jumphost. 3. Configure MAAS to enlist and commission VM for Juju bootstrap node.

Later, the 03-massdeploy.sh script will create three additional VMs and register them into the MAAS Server:

if [ "$virtinstall" -eq 1 ]; then
          sudo virt-install --connect qemu:///system --name $NODE_NAME --ram 8192 --cpu host --vcpus 4 \
                   --disk size=120,format=qcow2,bus=virtio,io=native,pool=default \
                   $netw $netw --boot network,hd,menu=off --noautoconsole --vnc --print-xml | tee $NODE_NAME

          nodemac=`grep  "mac address" $NODE_NAME | head -1 | cut -d '"' -f 2`
          sudo virsh -c qemu:///system define --file $NODE_NAME
          rm -f $NODE_NAME
          maas $PROFILE machines create autodetect_nodegroup='yes' name=$NODE_NAME \
              tags='control compute' hostname=$NODE_NAME power_type='virsh' mac_addresses=$nodemac \
              power_parameters_power_address='qemu+ssh://'$USER'@'$MAAS_IP'/system' \
              architecture='amd64/generic' power_parameters_power_id=$NODE_NAME
          nodeid=$(maas $PROFILE machines read | jq -r '.[] | select(.hostname == '\"$NODE_NAME\"').system_id')
          maas $PROFILE tag update-nodes control add=$nodeid || true
          maas $PROFILE tag update-nodes compute add=$nodeid || true

fi
7.2. Appendix B: Automatic Device Discovery

If your bare metal servers support IPMI, they can be discovered and enlisted automatically by the MAAS server. You need to configure bare metal servers to PXE boot on the network interface where they can reach the MAAS server. With nodes set to boot from a PXE image, they will start, look for a DHCP server, receive the PXE boot details, boot the image, contact the MAAS server and shut down.

During this process, the MAAS server will be passed information about the node, including the architecture, MAC address and other details which will be stored in the database of nodes. You can accept and commission the nodes via the web interface. When the nodes have been accepted the selected series of Ubuntu will be installed.

7.3. Appendix C: Machine Constraints

Juju and MAAS together allow you to assign different roles to servers, so that hardware and software can be configured according to their roles. We have briefly mentioned and used this feature in our example. Please visit Juju Machine Constraints https://jujucharms.com/docs/stable/charms-constraints and MAAS tags https://maas.ubuntu.com/docs/tags.html for more information.

7.4. Appendix D: Offline Deployment

When you have limited access policy in your environment, for example, when only the Jump Host has Internet access, but not the rest of the servers, we provide tools in JOID to support the offline installation.

The following package set is provided to those wishing to experiment with a ‘disconnected from the internet’ setup when deploying JOID utilizing MAAS. These instructions provide basic guidance as to how to accomplish the task, but it should be noted that due to the current reliance of MAAS and DNS, that behavior and success of deployment may vary depending on infrastructure setup. An official guided setup is in the roadmap for the next release:

  1. Get the packages from here: https://launchpad.net/~thomnico/+archive/ubuntu/ubuntu-cloud-mirrors

    Note

    The mirror is quite large 700GB in size, and does not mirror SDN repo/ppa.

  2. Additionally to make juju use a private repository of charms instead of using an external location are provided via the following link and configuring environments.yaml to use cloudimg-base-url: https://github.com/juju/docs/issues/757

JOID Configuration guide
JOID Configuration
Scenario 1: Nosdn

./deploy.sh -o ocata -s nosdn -t ha -l custom -f none -d xenial -m openstack

Scenario 2: Kubernetes core

./deploy.sh -l custom -f none -m kubernetes

Scenario 3: Kubernetes Load Balancer

./deploy.sh -l custom -f lb -m kubernetes

Scenario 4: Kubernetes with OVN

./deploy.sh -s ovn -l custom -f lb -m kubernetes

Scenario 5: Openstack with Opencontrail

./deploy.sh -o ocata -s ocl -t ha -l custom -f none -d xenial -m openstack

JOID User Guide
1. Introduction

This document will explain how to install OPNFV Euphrates with JOID including installing JOID, configuring JOID for your environment, and deploying OPNFV with different SDN solutions in HA, or non-HA mode. Prerequisites include

  • An Ubuntu 16.04 LTS Server Jumphost
  • Minimum 2 Networks per Pharos requirement
    • One for the administrative network with gateway to access the Internet
    • One for the OpenStack public network to access OpenStack instances via floating IPs
    • JOID supports multiple isolated networks for data as well as storage based on your network requirement for OpenStack.
  • Minimum 6 Physical servers for bare metal environment
    • Jump Host x 1, minimum H/W configuration:
      • CPU cores: 16
      • Memory: 32GB
      • Hard Disk: 1 (250GB)
      • NIC: eth0 (Admin, Management), eth1 (external network)
    • Control and Compute Nodes x 5, minimum H/W configuration:
      • CPU cores: 16
      • Memory: 32GB
      • Hard Disk: 2 (500GB) prefer SSD
      • NIC: eth0 (Admin, Management), eth1 (external network)

NOTE: Above configuration is minimum. For better performance and usage of the OpenStack, please consider higher specs for all nodes.

Make sure all servers are connected to top of rack switch and configured accordingly. No DHCP server should be up and configured. Configure gateways only on eth0 and eth1 networks to access the network outside your lab.

2. Orientation
2.1. JOID in brief

JOID as Juju OPNFV Infrastructure Deployer allows you to deploy different combinations of OpenStack release and SDN solution in HA or non-HA mode. For OpenStack, JOID supports Juno and Liberty. For SDN, it supports Openvswitch, OpenContrail, OpenDayLight, and ONOS. In addition to HA or non-HA mode, it also supports deploying from the latest development tree.

JOID heavily utilizes the technology developed in Juju and MAAS. Juju is a state-of-the-art, open source, universal model for service oriented architecture and service oriented deployments. Juju allows you to deploy, configure, manage, maintain, and scale cloud services quickly and efficiently on public clouds, as well as on physical servers, OpenStack, and containers. You can use Juju from the command line or through its powerful GUI. MAAS (Metal-As-A-Service) brings the dynamism of cloud computing to the world of physical provisioning and Ubuntu. Connect, commission and deploy physical servers in record time, re-allocate nodes between services dynamically, and keep them up to date; and in due course, retire them from use. In conjunction with the Juju service orchestration software, MAAS will enable you to get the most out of your physical hardware and dynamically deploy complex services with ease and confidence.

For more info on Juju and MAAS, please visit https://jujucharms.com/ and http://maas.ubuntu.com.

2.2. Typical JOID Setup

The MAAS server is installed and configured on Jumphost with Ubuntu 16.04 LTS with access to the Internet. Another VM is created to be managed by MAAS as a bootstrap node for Juju. The rest of the resources, bare metal or virtual, will be registered and provisioned in MAAS. And finally the MAAS environment details are passed to Juju for use.

3. Installation

We will use 03-maasdeploy.sh to automate the deployment of MAAS clusters for use as a Juju provider. MAAS-deployer uses a set of configuration files and simple commands to build a MAAS cluster using virtual machines for the region controller and bootstrap hosts and automatically commission nodes as required so that the only remaining step is to deploy services with Juju. For more information about the maas-deployer, please see https://launchpad.net/maas-deployer.

3.1. Configuring the Jump Host

Let’s get started on the Jump Host node.

The MAAS server is going to be installed and configured on a Jumphost machine. We need to create bridges on the Jump Host prior to setting up the MAAS.

NOTE: For all the commands in this document, please do not use a ‘root’ user account to run. Please create a non root user account. We recommend using the ‘ubuntu’ user.

Install the bridge-utils package on the Jump Host and configure a minimum of two bridges, one for the Admin network, the other for the Public network:

$ sudo apt-get install bridge-utils

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

iface p1p1 inet manual

auto brAdm
iface brAdm inet static
    address 172.16.50.51
    netmask 255.255.255.0
    bridge_ports p1p1

iface p1p2 inet manual

auto brPublic
iface brPublic inet static
    address 10.10.15.1
    netmask 255.255.240.0
    gateway 10.10.10.1
    dns-nameservers 8.8.8.8
    bridge_ports p1p2

NOTE: If you choose to use separate networks for management, data, and storage, then you need to create a bridge for each interface. In case of VLAN tags, make the appropriate network on jump-host depend upon VLAN ID on the interface.

NOTE: The Ethernet device names can vary from one installation to another. Please change the Ethernet device names according to your environment.

MAAS has been integrated in the JOID project. To get the JOID code, please run

$ sudo apt-get install git
$ git clone https://gerrit.opnfv.org/gerrit/p/joid.git
3.2. Setting Up Your Environment for JOID

To set up your own environment, create a directory in joid/ci/maas/<company name>/<pod number>/ and copy an existing JOID environment over. For example:

$ cd joid/ci
$ mkdir -p ../labconfig/myown/pod
$ cp ../labconfig/cengn/pod2/labconfig.yaml ../labconfig/myown/pod/

Now let’s configure labconfig.yaml file. Please modify the sections in the labconfig as per your lab configuration.

lab:

## Change the name of the lab you want maas name will get firmat as per location and rack name ## location: myown racks: - rack: pod

## based on your lab hardware please fill it accoridngly. ##

# Define one network and control and two control, compute and storage # and rest for compute and storage for backward compaibility. again # server with more disks should be used for compute and storage only. nodes: # DCOMP4-B, 24cores, 64G, 2disk, 4TBdisk - name: rack-2-m1

architecture: x86_64 roles: [network,control] nics: - ifname: eth0

spaces: [admin] mac: [“0c:c4:7a:3a:c5:b6”]
  • ifname: eth1 spaces: [floating] mac: [“0c:c4:7a:3a:c5:b7”]
power:
type: ipmi address: <bmc ip> user: <bmc username> pass: <bmc password>

## repeate the above section for number of hardware nodes you have it.

## define the floating IP range along with gateway IP to be used during the instance floating ips ##
floating-ip-range: 172.16.120.20,172.16.120.62,172.16.120.254,172.16.120.0/24 # Mutiple MACs seperated by space where MACs are from ext-ports across all network nodes.
## interface name to be used for floating ips ##
# eth1 of m4 since tags for networking are not yet implemented. ext-port: “eth1” dns: 8.8.8.8 osdomainname:
opnfv:
release: d distro: xenial type: noha openstack: newton sdncontroller: - type: nosdn storage: - type: ceph
## define the maximum disk possible in your environment ##
disk: /dev/sdb

feature: odl_l2

## Ensure the following configuration matches the bridge configuration on your jumphost

spaces: - type: admin

bridge: brAdm cidr: 10.120.0.0/24 gateway: 10.120.0.254 vlan:
  • type: floating bridge: brPublic cidr: 172.16.120.0/24 gateway: 172.16.120.254

Next we will use the 03-maasdeploy.sh in joid/ci to kick off maas deployment.

3.3. Starting MAAS depoyment

Now run the 03-maasdeploy.sh script with the environment you just created

~/joid/ci$ ./03-maasdeploy.sh custom ../labconfig/mylab/pod/labconfig.yaml

This will take approximately 30 minutes to couple of hours depending on your environment. This script will do the following: 1. Create 1 VM (KVM). 2. Install MAAS on the Jumphost. 3. Configure MAAS to enlist and commission a VM for Juju bootstrap node. 4. Configure MAAS to enlist and commission bare metal servers. 5. Download and load 16.04 images to be used by MAAS.

When it’s done, you should be able to view the MAAS webpage (in our example http://172.16.50.2/MAAS) and see 1 bootstrap node and bare metal servers in the ‘Ready’ state on the nodes page.

3.4. Troubleshooting MAAS deployment

During the installation process, please carefully review the error messages.

Join IRC channel #opnfv-joid on freenode to ask question. After the issues are resolved, re-running 03-maasdeploy.sh will clean up the VMs created previously. There is no need to manually undo what’s been done.

3.5. Deploying OPNFV

JOID allows you to deploy different combinations of OpenStack release and SDN solution in HA or non-HA mode. For OpenStack, it supports Juno and Liberty. For SDN, it supports Open vSwitch, OpenContrail, OpenDaylight and ONOS (Open Network Operating System). In addition to HA or non-HA mode, it also supports deploying the latest from the development tree (tip).

The deploy.sh script in the joid/ci directoy will do all the work for you. For example, the following deploys OpenStack Newton with OpenvSwitch in a HA mode.

~/joid/ci$  ./deploy.sh -o newton -s nosdn -t ha -l custom -f none -m openstack

The deploy.sh script in the joid/ci directoy will do all the work for you. For example, the following deploys Kubernetes with Load balancer on the pod.

~/joid/ci$  ./deploy.sh -m openstack -f lb

Take a look at the deploy.sh script. You will find we support the following for each option:

[-s]
  nosdn: Open vSwitch.
  odl: OpenDayLight Lithium version.
  opencontrail: OpenContrail.
  onos: ONOS framework as SDN.
[-t]
  noha: NO HA mode of OpenStack.
  ha: HA mode of OpenStack.
  tip: The tip of the development.
[-o]
  mitak: OpenStack Mitaka version.
  newton: OpenStack Newton version.
[-l]
  default: For virtual deployment where installation will be done on KVM created using ./03-maasdeploy.sh
  custom: Install on bare metal OPNFV defined by labconfig.yaml
[-f]
  none: no special feature will be enabled.
  ipv6: IPv6 will be enabled for tenant in OpenStack.
  dpdk: dpdk will be enabled.
  lxd: virt-type will be lxd.
  dvr: DVR will be enabled.
  lb: Load balancing in case of Kubernetes will be enabled.
[-d]
  xenial: distro to be used is Xenial 16.04
[-a]
  amd64: Only x86 architecture will be used. Future version will support arm64 as well.
[-m]
  openstack: Openstack model will be deployed.
  kubernetes: Kubernetes model will be deployed.

The script will call 01-bootstrap.sh to bootstrap the Juju VM node, then it will call 02-deploybundle.sh with the corrosponding parameter values.

./02-deploybundle.sh $opnfvtype $openstack $opnfvlab $opnfvsdn $opnfvfeature $opnfvdistro

Python script GenBundle.py would be used to create bundle.yaml based on the template defined in the config_tpl/juju2/ directory.

By default debug is enabled in the deploy.sh script and error messages will be printed on the SSH terminal where you are running the scripts. It could take an hour to a couple of hours (maximum) to complete.

You can check the status of the deployment by running this command in another terminal:

$ watch juju status --format tabular

This will refresh the juju status output in tabular format every 2 seconds.

Next we will show you what Juju is deploying and to where, and how you can modify based on your own needs.

3.6. OPNFV Juju Charm Bundles

The magic behind Juju is a collection of software components called charms. They contain all the instructions necessary for deploying and configuring cloud-based services. The charms publicly available in the online Charm Store represent the distilled DevOps knowledge of experts.

A bundle is a set of services with a specific configuration and their corresponding relations that can be deployed together in a single step. Instead of deploying a single service, they can be used to deploy an entire workload, with working relations and configuration. The use of bundles allows for easy repeatability and for sharing of complex, multi-service deployments.

For OPNFV, we have created the charm bundles for each SDN deployment. They are stored in each directory in ~/joid/ci.

We use Juju to deploy a set of charms via a yaml configuration file. You can find the complete format guide for the Juju configuration file here: http://pythonhosted.org/juju-deployer/config.html

In the ‘services’ subsection, here we deploy the ‘Ubuntu Xenial charm from the charm store,’ You can deploy the same charm and name it differently such as the second service ‘nodes-compute.’ The third service we deploy is named ‘ntp’ and is deployed from the NTP Trusty charm from the Charm Store. The NTP charm is a subordinate charm, which is designed for and deployed to the running space of another service unit.

The tag here is related to what we define in the deployment.yaml file for the MAAS. When ‘constraints’ is set, Juju will ask its provider, in this case MAAS, to provide a resource with the tags. In this case, Juju is asking one resource tagged with control and one resource tagged with compute from MAAS. Once the resource information is passed to Juju, Juju will start the installation of the specified version of Ubuntu.

In the next subsection, we define the relations between the services. The beauty of Juju and charms is you can define the relation of two services and all the service units deployed will set up the relations accordingly. This makes scaling out a very easy task. Here we add the relation between NTP and the two bare metal services.

Once the relations are established, Juju considers the deployment complete and moves to the next.

juju  deploy bundles.yaml

It will start the deployment , which will retry the section,

nova-cloud-controller:
  branch: lp:~openstack-charmers/charms/trusty/nova-cloud-controller/next
  num_units: 1
  options:
    network-manager: Neutron
  to:
    - "lxc:nodes-api=0"

We define a service name ‘nova-cloud-controller,’ which is deployed from the next branch of the nova-cloud-controller Trusty charm hosted on the Launchpad openstack-charmers team. The number of units to be deployed is 1. We set the network-manager option to ‘Neutron.’ This 1-service unit will be deployed to a LXC container at service ‘nodes-api’ unit 0.

To find out what other options there are for this particular charm, you can go to the code location at http://bazaar.launchpad.net/~openstack-charmers/charms/trusty/nova-cloud-controller/next/files and the options are defined in the config.yaml file.

Once the service unit is deployed, you can see the current configuration by running juju get:

$ juju config nova-cloud-controller

You can change the value with juju config, for example:

$ juju config nova-cloud-controller network-manager=’FlatManager’

Charms encapsulate the operation best practices. The number of options you need to configure should be at the minimum. The Juju Charm Store is a great resource to explore what a charm can offer you. Following the nova-cloud-controller charm example, here is the main page of the recommended charm on the Charm Store: https://jujucharms.com/nova-cloud-controller/trusty/66

If you have any questions regarding Juju, please join the IRC channel #opnfv-joid on freenode for JOID related questions or #juju for general questions.

3.7. Testing Your Deployment

Once juju-deployer is complete, use juju status –format tabular to verify that all deployed units are in the ready state.

Find the Openstack-dashboard IP address from the juju status output, and see if you can login via a web browser. The username and password is admin/openstack.

Optionally, see if you can log in to the Juju GUI. The Juju GUI is on the Juju bootstrap node, which is the second VM you define in the 03-maasdeploy.sh file. The username and password is admin/admin.

If you deploy OpenDaylight, OpenContrail or ONOS, find the IP address of the web UI and login. Please refer to each SDN bundle.yaml for the login username/password.

3.8. Troubleshooting

Logs are indispensable when it comes time to troubleshoot. If you want to see all the service unit deployment logs, you can run juju debug-log in another terminal. The debug-log command shows the consolidated logs of all Juju agents (machine and unit logs) running in the environment.

To view a single service unit deployment log, use juju ssh to access to the deployed unit. For example to login into nova-compute unit and look for /var/log/juju/unit-nova-compute-0.log for more info.

$ juju ssh nova-compute/0

Example:

ubuntu@R4N4B1:~$ juju ssh nova-compute/0
Warning: Permanently added '172.16.50.60' (ECDSA) to the list of known hosts.
Warning: Permanently added '3-r4n3b1-compute.maas' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 3.13.0-77-generic x86_64)

* Documentation:  https://help.ubuntu.com/
<skipped>
Last login: Tue Feb  2 21:23:56 2016 from bootstrap.maas
ubuntu@3-R4N3B1-compute:~$ sudo -i
root@3-R4N3B1-compute:~# cd /var/log/juju/
root@3-R4N3B1-compute:/var/log/juju# ls
machine-2.log  unit-ceilometer-agent-0.log  unit-ceph-osd-0.log  unit-neutron-contrail-0.log  unit-nodes-compute-0.log  unit-nova-compute-0.log  unit-ntp-0.log
root@3-R4N3B1-compute:/var/log/juju#

NOTE: By default Juju will add the Ubuntu user keys for authentication into the deployed server and only ssh access will be available.

Once you resolve the error, go back to the jump host to rerun the charm hook with:

$ juju resolved --retry <unit>

If you would like to start over, run juju destroy-environment <environment name> to release the resources, then you can run deploy.sh again.

The following are the common issues we have collected from the community:

  • The right variables are not passed as part of the deployment procedure.
./deploy.sh -o newton -s nosdn -t ha -l custom -f none
  • If you have setup maas not with 03-maasdeploy.sh then the ./clean.sh command could hang, the juju status command may hang because the correct MAAS API keys are not mentioned in cloud listing for MAAS. Solution: Please make sure you have an MAAS cloud listed using juju clouds. and the correct MAAS API key has been added.

  • Deployment times out:

    use the command juju status –format=tabular and make sure all service containers receive an IP address and they are executing code. Ensure there is no service in the error state.

  • In case the cleanup process hangs,run the juju destroy-model command manually.

Direct console access via the OpenStack GUI can be quite helpful if you need to login to a VM but cannot get to it over the network. It can be enabled by setting the console-access-protocol in the nova-cloud-controller to vnc. One option is to directly edit the juju-deployer bundle and set it there prior to deploying OpenStack.

nova-cloud-controller:
options:
  console-access-protocol: vnc

To access the console, just click on the instance in the OpenStack GUI and select the Console tab.

4. Post Installation Configuration
4.1. Configuring OpenStack

At the end of the deployment, the admin-openrc with OpenStack login credentials will be created for you. You can source the file and start configuring OpenStack via CLI.

~/joid_config$ cat admin-openrc
export OS_USERNAME=admin
export OS_PASSWORD=openstack
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://172.16.50.114:5000/v2.0
export OS_REGION_NAME=RegionOne

We have prepared some scripts to help your configure the OpenStack cloud that you just deployed. In each SDN directory, for example joid/ci/opencontrail, there is a ‘scripts’ folder where you can find the scripts. These scripts are created to help you configure a basic OpenStack Cloud to verify the cloud. For more information on OpenStack Cloud configuration, please refer to the OpenStack Cloud Administrator Guide: http://docs.openstack.org/user-guide-admin/. Similarly, for complete SDN configuration, please refer to the respective SDN administrator guide.

Each SDN solution requires slightly different setup. Please refer to the README in each SDN folder. Most likely you will need to modify the openstack.sh and cloud-setup.sh scripts for the floating IP range, private IP network, and SSH keys. Please go through openstack.sh, glance.sh and cloud-setup.sh and make changes as you see fit.

Let’s take a look at those for the Open vSwitch and briefly go through each script so you know what you need to change for your own environment.

~/joid/juju$ ls
configure-juju-on-openstack  get-cloud-images  joid-configure-openstack
4.1.1. openstack.sh

Let’s first look at ‘openstack.sh’. First there are 3 functions defined, configOpenrc(), unitAddress(), and unitMachine().

configOpenrc() {
  cat <<-EOF
      export SERVICE_ENDPOINT=$4
      unset SERVICE_TOKEN
      unset SERVICE_ENDPOINT
      export OS_USERNAME=$1
      export OS_PASSWORD=$2
      export OS_TENANT_NAME=$3
      export OS_AUTH_URL=$4
      export OS_REGION_NAME=$5
EOF
}

unitAddress() {
  if [[ "$jujuver" < "2" ]]; then
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"services\"][\"$1\"][\"units\"][\"$1/$2\"][\"public-address\"]" 2> /dev/null
  else
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"applications\"][\"$1\"][\"units\"][\"$1/$2\"][\"public-address\"]" 2> /dev/null
  fi
}

unitMachine() {
  if [[ "$jujuver" < "2" ]]; then
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"services\"][\"$1\"][\"units\"][\"$1/$2\"][\"machine\"]" 2> /dev/null
  else
      juju status --format yaml | python -c "import yaml; import sys; print yaml.load(sys.stdin)[\"applications\"][\"$1\"][\"units\"][\"$1/$2\"][\"machine\"]" 2> /dev/null
  fi
}

The function configOpenrc() creates the OpenStack login credentials, the function unitAddress() finds the IP address of the unit, and the function unitMachine() finds the machine info of the unit.

create_openrc() {
   keystoneIp=$(keystoneIp)
   if [[ "$jujuver" < "2" ]]; then
       adminPasswd=$(juju get keystone | grep admin-password -A 5 | grep value | awk '{print $2}' 2> /dev/null)
   else
       adminPasswd=$(juju config keystone | grep admin-password -A 5 | grep value | awk '{print $2}' 2> /dev/null)
   fi

   configOpenrc admin $adminPasswd admin http://$keystoneIp:5000/v2.0 RegionOne > ~/joid_config/admin-openrc
   chmod 0600 ~/joid_config/admin-openrc
}

This finds the IP address of the keystone unit 0, feeds in the OpenStack admin credentials to a new file name ‘admin-openrc’ in the ‘~/joid_config/’ folder and change the permission of the file. It’s important to change the credentials here if you use a different password in the deployment Juju charm bundle.yaml.

neutron net-show ext-net > /dev/null 2>&1 || neutron net-create ext-net \
                                               --router:external=True \
                                               --provider:network_type flat \
                                               --provider:physical_network physnet1
::
neutron subnet-show ext-subnet > /dev/null 2>&1 || neutron subnet-create ext-net
–name ext-subnet –allocation-pool start=$EXTNET_FIP,end=$EXTNET_LIP –disable-dhcp –gateway $EXTNET_GW $EXTNET_NET

This section will create the ext-net and ext-subnet for defining the for floating ips.

openstack congress datasource create nova "nova" \
 --config username=$OS_USERNAME \
 --config tenant_name=$OS_TENANT_NAME \
 --config password=$OS_PASSWORD \
 --config auth_url=http://$keystoneIp:5000/v2.0

This section will create the congress datasource for various services. Each service datasource will have entry in the file.

4.1.2. get-cloud-images
folder=/srv/data/
sudo mkdir $folder || true

if grep -q 'virt-type: lxd' bundles.yaml; then
   URLS=" \
   http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-lxc.tar.gz \
   http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz "

else
   URLS=" \
   http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img \
   http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img \
   http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img \
   http://mirror.catn.com/pub/catn/images/qcow2/centos6.4-x86_64-gold-master.img \
   http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 \
   http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img "
fi

for URL in $URLS
do
FILENAME=${URL##*/}
if [ -f $folder/$FILENAME ];
then
   echo "$FILENAME already downloaded."
else
   wget  -O  $folder/$FILENAME $URL
fi
done

This section of the file will download the images to jumphost if not found to be used with openstack VIM.

NOTE: The image downloading and uploading might take too long and time out. In this case, use juju ssh glance/0 to log in to the glance unit 0 and run the script again, or manually run the glance commands.

4.1.3. joid-configure-openstack
source ~/joid_config/admin-openrc

First, source the the admin-openrc file.

::
#Upload images to glance
glance image-create –name=”Xenial LXC x86_64” –visibility=public –container-format=bare –disk-format=root-tar –property architecture=”x86_64” < /srv/data/xenial-server-cloudimg-amd64-root.tar.gz glance image-create –name=”Cirros LXC 0.3” –visibility=public –container-format=bare –disk-format=root-tar –property architecture=”x86_64” < /srv/data/cirros-0.3.4-x86_64-lxc.tar.gz glance image-create –name=”Trusty x86_64” –visibility=public –container-format=ovf –disk-format=qcow2 < /srv/data/trusty-server-cloudimg-amd64-disk1.img glance image-create –name=”Xenial x86_64” –visibility=public –container-format=ovf –disk-format=qcow2 < /srv/data/xenial-server-cloudimg-amd64-disk1.img glance image-create –name=”CentOS 6.4” –visibility=public –container-format=bare –disk-format=qcow2 < /srv/data/centos6.4-x86_64-gold-master.img glance image-create –name=”Cirros 0.3” –visibility=public –container-format=bare –disk-format=qcow2 < /srv/data/cirros-0.3.4-x86_64-disk.img

upload the images into glane to be used for creating the VM.

# adjust tiny image
nova flavor-delete m1.tiny
nova flavor-create m1.tiny 1 512 8 1

Adjust the tiny image profile as the default tiny instance is too small for Ubuntu.

# configure security groups
neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol icmp --remote-ip-prefix 0.0.0.0/0 default
neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol tcp --port-range-min 22 --port-range-max 22 --remote-ip-prefix 0.0.0.0/0 default

Open up the ICMP and SSH access in the default security group.

# import key pair
keystone tenant-create --name demo --description "Demo Tenant"
keystone user-create --name demo --tenant demo --pass demo --email demo@demo.demo

nova keypair-add --pub-key id_rsa.pub ubuntu-keypair

Create a project called ‘demo’ and create a user called ‘demo’ in this project. Import the key pair.

# configure external network
neutron net-create ext-net --router:external --provider:physical_network external --provider:network_type flat --shared
neutron subnet-create ext-net --name ext-subnet --allocation-pool start=10.5.8.5,end=10.5.8.254 --disable-dhcp --gateway 10.5.8.1 10.5.8.0/24

This section configures an external network ‘ext-net’ with a subnet called ‘ext-subnet’. In this subnet, the IP pool starts at 10.5.8.5 and ends at 10.5.8.254. DHCP is disabled. The gateway is at 10.5.8.1, and the subnet mask is 10.5.8.0/24. These are the public IPs that will be requested and associated to the instance. Please change the network configuration according to your environment.

# create vm network
neutron net-create demo-net
neutron subnet-create --name demo-subnet --gateway 10.20.5.1 demo-net 10.20.5.0/24

This section creates a private network for the instances. Please change accordingly.

neutron router-create demo-router
neutron router-interface-add demo-router demo-subnet
neutron router-gateway-set demo-router ext-net

This section creates a router and connects this router to the two networks we just created.

# create pool of floating ips
i=0
while [ $i -ne 10 ]; do
  neutron floatingip-create ext-net
  i=$((i + 1))
done

Finally, the script will request 10 floating IPs.

4.1.4. configure-juju-on-openstack

This script can be used to do juju bootstrap on openstack so that Juju can be used as model tool to deploy the services and VNF on top of openstack using the JOID.

5. Appendix A: Single Node Deployment

By default, running the script ./03-maasdeploy.sh will automatically create the KVM VMs on a single machine and configure everything for you.

if [ ! -e ./labconfig.yaml ]; then
    virtinstall=1
    labname="default"
    cp ../labconfig/default/labconfig.yaml ./
    cp ../labconfig/default/deployconfig.yaml ./

Please change joid/ci/labconfig/default/labconfig.yaml accordingly. The MAAS deployment script will do the following: 1. Create bootstrap VM. 2. Install MAAS on the jumphost. 3. Configure MAAS to enlist and commission VM for Juju bootstrap node.

Later, the 03-massdeploy.sh script will create three additional VMs and register them into the MAAS Server:

if [ "$virtinstall" -eq 1 ]; then
          sudo virt-install --connect qemu:///system --name $NODE_NAME --ram 8192 --cpu host --vcpus 4 \
                   --disk size=120,format=qcow2,bus=virtio,io=native,pool=default \
                   $netw $netw --boot network,hd,menu=off --noautoconsole --vnc --print-xml | tee $NODE_NAME

          nodemac=`grep  "mac address" $NODE_NAME | head -1 | cut -d '"' -f 2`
          sudo virsh -c qemu:///system define --file $NODE_NAME
          rm -f $NODE_NAME
          maas $PROFILE machines create autodetect_nodegroup='yes' name=$NODE_NAME \
              tags='control compute' hostname=$NODE_NAME power_type='virsh' mac_addresses=$nodemac \
              power_parameters_power_address='qemu+ssh://'$USER'@'$MAAS_IP'/system' \
              architecture='amd64/generic' power_parameters_power_id=$NODE_NAME
          nodeid=$(maas $PROFILE machines read | jq -r '.[] | select(.hostname == '\"$NODE_NAME\"').system_id')
          maas $PROFILE tag update-nodes control add=$nodeid || true
          maas $PROFILE tag update-nodes compute add=$nodeid || true

fi
6. Appendix B: Automatic Device Discovery

If your bare metal servers support IPMI, they can be discovered and enlisted automatically by the MAAS server. You need to configure bare metal servers to PXE boot on the network interface where they can reach the MAAS server. With nodes set to boot from a PXE image, they will start, look for a DHCP server, receive the PXE boot details, boot the image, contact the MAAS server and shut down.

During this process, the MAAS server will be passed information about the node, including the architecture, MAC address and other details which will be stored in the database of nodes. You can accept and commission the nodes via the web interface. When the nodes have been accepted the selected series of Ubuntu will be installed.

7. Appendix C: Machine Constraints

Juju and MAAS together allow you to assign different roles to servers, so that hardware and software can be configured according to their roles. We have briefly mentioned and used this feature in our example. Please visit Juju Machine Constraints https://jujucharms.com/docs/stable/charms-constraints and MAAS tags https://maas.ubuntu.com/docs/tags.html for more information.

8. Appendix D: Offline Deployment

When you have limited access policy in your environment, for example, when only the Jump Host has Internet access, but not the rest of the servers, we provide tools in JOID to support the offline installation.

The following package set is provided to those wishing to experiment with a ‘disconnected from the internet’ setup when deploying JOID utilizing MAAS. These instructions provide basic guidance as to how to accomplish the task, but it should be noted that due to the current reliance of MAAS and DNS, that behavior and success of deployment may vary depending on infrastructure setup. An official guided setup is in the roadmap for the next release:

  1. Get the packages from here: https://launchpad.net/~thomnico/+archive/ubuntu/ubuntu-cloud-mirrors
NOTE: The mirror is quite large 700GB in size, and does not mirror SDN repo/ppa.
  1. Additionally to make juju use a private repository of charms instead of using an external location are provided via the following link and configuring environments.yaml to use cloudimg-base-url: https://github.com/juju/docs/issues/757

Kvmfornfv

KVM4NFV Requirements
1. Kvm4nfv Requirements
1.1. Introduction

The NFV hypervisors provide crucial functionality in the NFV Infrastructure(NFVI).The existing hypervisors, however, are not necessarily designed or targeted to meet the requirements for the NFVI.

This document specifies the list of requirements that need to be met as part of this “NFV Hypervisors-KVM” project in Euphrates release.

As part of this project we need to make collaborative efforts towards enabling the NFV features.

1.2. Scope and Purpose

The main purpose of this project is to enhance the KVM hypervisor for NFV, by looking at the following areas initially:

  • Minimal Interrupt latency variation for data plane VNFs:
    • Minimal Timing Variation for Timing correctness of real-time VNFs
    • Minimal packet latency variation for data-plane VNFs
  • Inter-VM communication

  • Fast live migration

The output of this project would be list of the performance goals,comprehensive instructions for the system configurations,tools to measure Performance and interrupt latency.

1.3. Methods and Instrumentation

The above areas would require software development and/or specific hardware features, and some need just configurations information for the system (hardware, BIOS, OS, etc.).

A right configuration is critical for improving the NFV performance/latency. Even working on the same code base, different configurations can make completely different performance/latency result. Configurations that can be made as part of this project to tune a specific scenario are:

  1. Platform Configuration : Some hardware features like Power management, Hyper-Threading,Legacy USB Support/Port 60/64 Emulation,SMI can be configured.
  2. Operating System Configuration : Some configuration features like CPU isolation,Memory allocation,IRQ affinity,Device assignment for VM,Tickless, TSC,Idle,_RCU_NOCB_,Disable the RT throttling,NUMA can be configured.
  3. Performance/Latency Tuning : Application level configurations like timers,Making vfio MSI interrupt as non-threaded,Cache Allocation Technology(CAT) enabling can be tuned to improve the NFV performance/latency.
1.4. Features to be tested
The tests that need to be conducted to make sure that latency is addressed are:
  1. Timer test
  2. Device Interrupt Test
  3. Packet forwarding (DPDK OVS)
  4. Packet Forwarding (SR-IOV)
  5. Bare-metal Packet Forwarding
1.5. Dependencies
  1. OPNFV Project: “Characterize vSwitch Performance for Telco NFV Use Cases” (VSPERF) for performance evaluation of ivshmem vs. vhost-user.
  2. OPNFV Project: “Pharos” for Test Bed Infrastructure, and possibly “Yardstick” for infrastructure verification.
  3. There are currently no similar projects underway in OPNFV or in an upstream project
  4. The relevant upstream project to be influenced here is QEMU/KVM and libvirt.
  5. In terms of HW dependencies, the aim is to use standard IA Server hardware for this project, as provided by OPNFV Pharos.
KVM4NFV Installation instruction
1. Abstract

This document will give the instructions to user on how to deploy available KVM4NFV build scenario verfied for the Euphrates release of the OPNFV platform.

2. KVM4NFV Installation Instruction
2.1. Preparing the installation

The OPNFV project- KVM4NFV (https://gerrit.opnfv.org/gerrit/kvmfornfv.git) is cloned first, to make the build scripts for Qemu & Kernel, Rpms and Debians available.

2.2. HW requirements

These build scripts are triggered on the Jenkins-Slave build server. Currently Intel POD10 is used as test environment for kvm4nfv to execute cyclictest. As part of this test environment Intel pod10-jump is configured as jenkins slave and all the latest build artifacts are downloaded on to it. Intel pod10-node1 is the host on which a guest vm will be launched as a part of running cylictest through yardstick.

2.3. Build instructions

Builds are possible for the following packages-

kvmfornfv source code

The ./ci/build.sh is the main script used to trigger the Rpms (on ‘centos’) and Debians (on ‘ubuntu’) builds in this case.

  • How to build Kernel/Qemu Rpms- To build rpm packages, build.sh script is run with -p and -o option (i.e. if -p package option is passed as “centos” or in default case). Example:
cd kvmfornfv/

For Kernel/Qemu RPMs,
sh ./ci/build.sh -p centos -o build_output
  • How to build Kernel/Qemu Debians- To build debian packages, build.sh script is run with -p and -o option (i.e. if -p package option is passed as “ubuntu”). Example:
cd kvmfornfv/

For Kernel/Qemu Debians,
sh ./ci/build.sh -p ubuntu -o build_output
  • How to build all Kernel & Qemu, Rpms & Debians- To build both debian and rpm packages, build.sh script is run with -p and -o option (i.e. if -p package option is passed as “both”). Example:
cd kvmfornfv/

For Kernel/Qemu RPMs and Debians,
sh ./ci/build.sh -p both -o build_output

Note

Kvm4nfv can be installed in two ways

  1. As part of a scenario deployment
  2. As a stand alone component

For installation of kvmfornfv as part of scenario deployment use this `link`_

http://artifacts.opnfv.org/kvmfornfv/docs/index.html#document-scenarios/kvmfornfv.scenarios.description
2.4. Installation instructions

Installation can be done in the following ways-

1. From kvmfornfv source code- The build packages that are prepared in the above section, are installed differently depending on the platform.

Please visit the links for each-

2. Using Fuel installer-

  • Please refer to the document present at /fuel-plugin/README.md
2.5. Post-installation activities

After the packages are built, test these packages by executing the scripts present in ci/envs for configuring the host and guest respectively.

3. Release Note for KVM4NFV CICD
3.1. Abstract

This document contains the release notes for the Euphrates release of OPNFV when using KVM4NFV CICD process.

3.2. Introduction

Provide a brief introduction of how this configuration is used in OPNFV release using KVM4VFV CICD as scenario.

Be sure to reference your scenario installation instruction.

3.3. Release Data
Project NFV Hypervisors-KVM
Repo/tag kvmfornfv
Release designation  
Release date 2017-10-06
Purpose of the delivery
  • Automate the KVM4VFV CICD scenario
  • Executing latency test cases
  • Collection of logs for debugging
3.4. Document version change
The following documents are added-
  • configurationguide
  • installationprocedure
  • userguide
  • overview
  • glossary
  • releasenotes
3.5. Reason for new version
3.5.1. Feature additions
JIRA REFERENCE SLOGAN
JIRA: NFV Hypervisors-KVMFORNFV-72
JIRA: NFV Hypervisors-KVMFORNFV-73
JIRA: NFV Hypervisors-KVMFORNFV-78
JIRA: NFV Hypervisors-KVMFORNFV-86
JIRA: NFV Hypervisors-KVMFORNFV-87
JIRA: NFV Hypervisors-KVMFORNFV-88
JIRA: NFV Hypervisors-KVMFORNFV-89
JIRA: VSPERF-510
JIRA: YARDSTICK-783
JIRA: YARDSTICK-815
3.6. Known issues

JIRA TICKETS:

JIRA REFERENCE SLOGAN
   
3.7. Workarounds

See JIRA: https://jira.opnfv.org/projects

For more information on the OPNFV Euphrates release, please visit http://www.opnfv.org/euphrates

Kvm4nfv Configuration Guide
Euphrates 1.0
1. Configuration Abstract

This document provides guidance for the configurations available in the Euphrates release of OPNFV

The release includes four installer tools leveraging different technologies; Apex, Compass4nfv, Fuel and JOID, which deploy components of the platform.

This document also includes the selection of tools and components including guidelines for how to deploy and configure the platform to an operational state.

2. Configuration Options

OPNFV provides a variety of virtual infrastructure deployments called scenarios designed to host virtualised network functions (VNF’s). KVM4NFV scenarios provide specific capabilities and/or components aimed to solve specific problems for the deployment of VNF’s. KVM4NFV scenario includes components such as OpenStack,KVM etc. which includes different source components or configurations.

Note

  • Each KVM4NFV scenario provides unique features and capabilities, it is important to understand your target platform capabilities before installing and configuring. This configuration guide outlines how to configure components in order to enable the features required.
  • More deatils of kvm4nfv scenarios installation and description can be found in the scenario guide of kvm4nfv docs
4. Low Latency Feature Configuration Description
4.1. Introduction

In KVM4NFV project, we focus on the KVM hypervisor to enhance it for NFV, by looking at the following areas initially

  • Minimal Interrupt latency variation for data plane VNFs:
    • Minimal Timing Variation for Timing correctness of real-time VNFs
    • Minimal packet latency variation for data-plane VNFs
  • Inter-VM communication,

  • Fast live migration

4.2. Configuration of Cyclictest

Cyclictest measures Latency of response to a stimulus. Achieving low latency with the KVM4NFV project requires setting up a special test environment. This environment includes the BIOS settings, kernel configuration, kernel parameters and the run-time environment.

4.2.1. Pre-configuration activities

Intel POD10 is currently used as OPNFV-KVM4NFV test environment. The rpm packages from the latest build are downloaded onto Intel-Pod10 jump server from artifact repository. Yardstick running in a ubuntu docker container on Intel Pod10-jump server will configure the host(intel pod10 node1/node2 based on job type), the guest and triggers the cyclictest on the guest using below sample yaml file.

For IDLE-IDLE test,

host_setup_seqs:
- "host-setup0.sh"
- "reboot"
- "host-setup1.sh"
- "host-run-qemu.sh"

guest_setup_seqs:
- "guest-setup0.sh"
- "reboot"
- "guest-setup1.sh"
_images/idle-idle-test.png
For [CPU/Memory/IO]Stress-IDLE tests,

host_setup_seqs:
- "host-setup0.sh"
- "reboot"
- "host-setup1.sh"
- "stress_daily.sh" [cpustress/memory/io]
- "host-run-qemu.sh"

guest_setup_seqs:
- "guest-setup0.sh"
- "reboot"
- "guest-setup1.sh"
_images/stress-idle-test.png

The following scripts are used for configuring host and guest to create a special test environment and achieve low latency.

Note: host-setup0.sh, host-setup1.sh and host-run-qemu.sh are run on the host, followed by guest-setup0.sh and guest-setup1.sh scripts on the guest VM.

host-setup0.sh: Running this script will install the latest kernel rpm on host and will make necessary changes as following to create special test environment.

  • Isolates CPUs from the general scheduler
  • Stops timer ticks on isolated CPUs whenever possible
  • Stops RCU callbacks on isolated CPUs
  • Enables intel iommu driver and disables DMA translation for devices
  • Sets HugeTLB pages to 1GB
  • Disables machine check
  • Disables clocksource verification at runtime

host-setup1.sh: Running this script will make the following test environment changes.

  • Disabling watchdogs to reduce overhead
  • Disabling RT throttling
  • Reroute interrupts bound to isolated CPUs to CPU 0
  • Change the iptable so that we can ssh to the guest remotely

stress_daily.sh: Scripts gets triggered only for stress-idle tests. Running this script make the following environment changes.

  • Triggers stress_script.sh, which runs the stress command with necessary options
  • CPU,Memory or IO stress can be applied based on the test type
  • Applying stress only on the Host is handled in D-Release
  • For Idle-Idle test the stress script is not triggered
  • Stress is applied only on the free cores to prevent load on qemu process
Note:
  • On Numa Node 1: 22,23 cores are allocated for QEMU process
  • 24-43 are used for applying stress
host-run-qemu.sh: Running this script will launch a guest vm on the host.
Note: download guest disk image from artifactory.

guest-setup0.sh: Running this scrcipt on the guest vm will install the latest build kernel rpm, cyclictest and make the following configuration on guest vm.

  • Isolates CPUs from the general scheduler
  • Stops timer ticks on isolated CPUs whenever possible
  • Uses polling idle loop to improve performance
  • Disables clocksource verification at runtime

guest-setup1.sh: Running this script on guest vm will do the following configurations.

  • Disable watchdogs to reduce overhead
  • Routes device interrupts to non-RT CPU
  • Disables RT throttling
4.2.2. Hardware configuration

Currently Intel POD10 is used as test environment for kvm4nfv to execute cyclictest. As part of this test environment Intel pod10-jump is configured as jenkins slave and all the latest build artifacts are downloaded on to it.

3. Scenariomatrix

Scenarios are implemented as deployable compositions through integration with an installation tool. OPNFV supports multiple installation tools and for any given release not all tools will support all scenarios. While our target is to establish parity across the installation tools to ensure they can provide all scenarios, the practical challenge of achieving that goal for any given feature and release results in some disparity.

3.1. Euphrates scenario overeview

The following table provides an overview of the installation tools and available scenario’s in the Euphrates release of OPNFV.

Scenario status is indicated by a weather pattern icon. All scenarios listed with a weather pattern are possible to deploy and run in your environment or a Pharos lab, however they may have known limitations or issues as indicated by the icon.

Weather pattern icon legend:

Weather Icon Scenario Status
_images/weather-clear.jpg Stable, no known issues
_images/weather-few-clouds.jpg Stable, documented limitations
_images/weather-overcast.jpg Deployable, stability or feature limitations
_images/weather-dash.jpg Not deployed with this installer

Scenarios that are not yet in a state of “Stable, no known issues” will continue to be stabilised and updates will be made on the stable/euphrates branch. While we intend that all Euphrates scenarios should be stable it is worth checking regularly to see the current status. Due to our dependency on upstream communities and code, some issues may not be resolved prior to E release.

3.2. Scenario Naming

In OPNFV scenarios are identified by short scenario names, these names follow a scheme that identifies the key components and behaviours of the scenario. The rules for scenario naming are as follows:

os-[controller]-[feature]-[mode]-[option]

Details of the fields are

  • [os]: mandatory

    • Refers to the platform type used
    • possible value: os (OpenStack)
  • [controller]: mandatory

    • Refers to the SDN controller integrated in the platform
    • example values: nosdn, ocl, odl, onos
  • [feature]: mandatory

    • Refers to the feature projects supported by the scenario
    • example values: nofeature, kvm, ovs, sfc
  • [mode]: mandatory

    • Refers to the deployment type, which may include for instance high availability
    • possible values: ha, noha
  • [option]: optional

    • Used for the scenarios those do not fit into naming scheme.
    • The optional field in the short scenario name should not be included if there is no optional

    scenario.

Some examples of supported scenario names are:

  • os-nosdn-kvm-noha

    • This is an OpenStack based deployment using neutron including the OPNFV enhanced KVM hypervisor
  • os-onos-nofeature-ha

    • This is an OpenStack deployment in high availability mode including ONOS as the SDN controller
  • os-odl_l2-sfc

    • This is an OpenStack deployment using OpenDaylight and OVS enabled with SFC features
  • os-nosdn-kvm_ovs_dpdk-ha

    • This is an Openstack deployment with high availability using OVS, DPDK including the OPNFV

    enhanced KVM hypervisor * This deployment has 3-Contoller and 2-Compute nodes

  • os-nosdn-kvm_ovs_dpdk-noha

    • This is an Openstack deployment without high availability using OVS, DPDK including the OPNFV

    enhanced KVM hypervisor * This deployment has 1-Contoller and 3-Compute nodes

  • os-nosdn-kvm_ovs_dpdk_bar-ha

    • This is an Openstack deployment with high availability using OVS, DPDK including the OPNFV
    enhanced KVM hypervisor

    and Barometer

    • This deployment has 3-Contoller and 2-Compute nodes
  • os-nosdn-kvm_ovs_dpdk_bar-noha

    • This is an Openstack deployment without high availability using OVS, DPDK including the OPNFV
    enhanced KVM hypervisor

    and Barometer

    • This deployment has 1-Contoller and 3-Compute nodes
3.3. Installing your scenario

There are two main methods of deploying your target scenario, one method is to follow this guide which will walk you through the process of deploying to your hardware using scripts or ISO images, the other method is to set up a Jenkins slave and connect your infrastructure to the OPNFV Jenkins master.

For the purposes of evaluation and development a number of Euphrates scenarios are able to be deployed virtually to mitigate the requirements on physical infrastructure. Details and instructions on performing virtual deployments can be found in the installer specific installation instructions.

To set up a Jenkins slave for automated deployment to your lab, refer to the Jenkins slave connect guide.

KVM4NFV User Guide
1. Userguide Abstract

In KVM4NFV project, we focus on the KVM hypervisor to enhance it for NFV, by looking at the following areas initially-

  • Minimal Interrupt latency variation for data plane VNFs:
    • Minimal Timing Variation for Timing correctness of real-time VNFs
    • Minimal packet latency variation for data-plane VNFs
  • Inter-VM communication

  • Fast live migration

2. Userguide Introduction
2.1. Overview

The project “NFV Hypervisors-KVM” makes collaborative efforts to enable NFV features for existing hypervisors, which are not necessarily designed or targeted to meet the requirements for the NFVI.The KVM4NFV scenario consists of Continuous Integration builds, deployments and testing combinations of virtual infrastructure components.

2.2. KVM4NFV Features

Using this project, the following areas are targeted-

  • Minimal Interrupt latency variation for data plane VNFs:
    • Minimal Timing Variation for Timing correctness of real-time VNFs
    • Minimal packet latency variation for data-plane VNFs
  • Inter-VM communication

  • Fast live migration

Some of the above items would require software development and/or specific hardware features, and some need just configurations information for the system (hardware, BIOS, OS, etc.).

We include a requirements gathering stage as a formal part of the project. For each subproject, we will start with an organized requirement stage so that we can determine specific use cases (e.g. what kind of VMs should be live migrated) and requirements (e.g. interrupt latency, jitters, Mpps, migration-time, down-time, etc.) to set out the performance goals.

Potential future projects would include:

  • Dynamic scaling (via scale-out) using VM instantiation
  • Fast live migration for SR-IOV

The user guide outlines how to work with key components and features in the platform, each feature description section will indicate the scenarios that provide the components and configurations required to use it.

The configuration guide details which scenarios are best for you and how to install and configure them.

2.3. General usage guidelines

The user guide for KVM4NFV features and capabilities provide step by step instructions for using features that have been configured according to the installation and configuration instructions.

2.4. Scenarios User Guide

The procedure to deploy/test KVM4NFV scenarios in a nested virtualization or on bare-metal environment is mentioned in the below link. The kvm4nfv user guide can be found at docs/scenarios

http://artifacts.opnfv.org/kvmfornfv/docs/index.html#kvmfornfv-scenarios-overview-and-description

The deployment has been verified for os-nosdn-kvm-ha, os-nosdn-kvm-noha, os-nosdn-kvm_ovs_dpdk-ha, os-nosdn-kvm_ovs_dpdk-noha and os-nosdn-kvm_ovs_dpdk_bar-ha, os-nosdn-kvm_ovs_dpdk_bar-noha test scenarios.

For brief view of the above scenarios use:

http://artifacts.opnfv.org/kvmfornfv/docs/index.html#scenario-abstract
3. Using common platform components

This section outlines basic usage principals and methods for some of the commonly deployed components of supported OPNFV scenario’s in Euphrates. The subsections provide an outline of how these components are commonly used and how to address them in an OPNFV deployment.The components derive from autonomous upstream communities and where possible this guide will provide direction to the relevant documentation made available by those communities to better help you navigate the OPNFV deployment.

4. Using Euphrates Features

The following sections of the user guide provide feature specific usage guidelines and references for KVM4NFV project.

  • <project>/docs/userguide/low_latency.userguide.rst
  • <project>/docs/userguide/live_migration.userguide.rst
  • <project>/docs/userguide/tuning.userguide.rst
5. FTrace Debugging Tool
5.1. About Ftrace

Ftrace is an internal tracer designed to find what is going on inside the kernel. It can be used for debugging or analyzing latencies and performance related issues that take place outside of user-space. Although ftrace is typically considered the function tracer, it is really a frame work of several assorted tracing utilities.

One of the most common uses of ftrace is the event tracing.

Note: - For KVM4NFV, Ftrace is preferred as it is in-built kernel tool - More stable compared to other debugging tools

5.2. Version Features
Release Features
Colorado
  • Ftrace Debugging tool is not implemented in Colorado release of KVM4NFV
Danube
  • Ftrace aids in debugging the KVM4NFV 4.4-linux-kernel level issues
  • Option to disable if not required
Euphrates
  • Breaktrace option is implemented.
  • Implemented post-execute script option to disable the ftrace when it is enabled.
5.3. Implementation of Ftrace

Ftrace uses the debugfs file system to hold the control files as well as the files to display output.

When debugfs is configured into the kernel (which selecting any ftrace option will do) the directory /sys/kernel/debug will be created. To mount this directory, you can add to your /etc/fstab file:

debugfs       /sys/kernel/debug          debugfs defaults        0       0

Or you can mount it at run time with:

mount -t debugfs nodev /sys/kernel/debug

Some configurations for Ftrace are used for other purposes, like finding latency or analyzing the system. For the purpose of debugging, the kernel configuration parameters that should be enabled are:

CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_FTRACE=y

The above parameters must be enabled in /boot/config-4.4.0-el7.x86_64 i.e., kernel config file for ftrace to work. If not enabled, change the parameter to y and run.,

On CentOS
grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

Re-check the parameters after reboot before running ftrace.

5.4. Files in Ftrace:

The below is a list of few major files in Ftrace.

current_tracer:

This is used to set or display the current tracer that is configured.

available_tracers:

This holds the different types of tracers that have been compiled into the kernel. The tracers listed here can be configured by echoing their name into current_tracer.

tracing_on:

This sets or displays whether writing to the tracering buffer is enabled. Echo 0 into this file to disable the tracer or 1 to enable it.

trace:

This file holds the output of the trace in a human readable format.

tracing_cpumask:

This is a mask that lets the user only trace on specified CPUs. The format is a hex string representing the CPUs.

events:

It holds event tracepoints (also known as static tracepoints) that have been compiled into the kernel. It shows what event tracepoints exist and how they are grouped by system.
5.5. Avaliable Tracers

Here is the list of current tracers that may be configured based on usage.

  • function
  • function_graph
  • irqsoff
  • preemptoff
  • preemptirqsoff
  • wakeup
  • wakeup_rt

Brief about a few:

function:

Function call tracer to trace all kernel functions.

function_graph:

Similar to the function tracer except that the function tracer probes the functions on their entry whereas the function graph tracer traces on both entry and exit of the functions.

nop:

This is the “trace nothing” tracer. To remove tracers from tracing simply echo “nop” into current_tracer.

Examples:

To list available tracers:
[tracing]# cat available_tracers
function_graph function wakeup wakeup_rt preemptoff irqsoff preemptirqsoff nop

Usage:
[tracing]# echo function > current_tracer
[tracing]# cat current_tracer
function

To view output:
[tracing]# cat trace | head -10

To Stop tracing:
[tracing]# echo 0 > tracing_on

To Start/restart tracing:
[tracing]# echo 1 > tracing_on;
5.6. Ftrace in KVM4NFV

Ftrace is part of KVM4NFV D-Release. KVM4NFV built 4.4-linux-Kernel will be tested by executing cyclictest and analyzing the results/latency values (max, min, avg) generated. Ftrace (or) function tracer is a stable kernel inbuilt debugging tool which tests real time kernel and outputs a log as part of the code. These output logs are useful in following ways.

  • Kernel Debugging.
  • Helps in Kernel code optimization and
  • Can be used to better understand the kernel level code flow

Ftrace logs for KVM4NFV can be found here:

5.7. Ftrace Usage in KVM4NFV Kernel Debugging:

Kvm4nfv has two scripts in /ci/envs to provide ftrace tool:

  • enable_trace.sh
  • disable_trace.sh
Found at.,
$ cd kvmfornfv/ci/envs
5.8. Enabling Ftrace in KVM4NFV

The enable_trace.sh script is triggered by changing ftrace_enable value in test_kvmfornfv.sh script to 1 (which is zero by default). Change as below to enable Ftrace.

ftrace_enable=1

Note:

  • Ftrace is enabled before
5.9. Details of enable_trace script
  • CPU coremask is calculated using getcpumask()

  • All the required events are enabled by,

    echoing “1” to $TRACEDIR/events/event_name/enable file

Example,

$TRACEDIR = /sys/kernel/debug/tracing/
sudo bash -c "echo 1 > $TRACEDIR/events/irq/enable"
sudo bash -c "echo 1 > $TRACEDIR/events/task/enable"
sudo bash -c "echo 1 > $TRACEDIR/events/syscalls/enable"

The set_event file contains all the enabled events list

  • Function tracer is selected. May be changed to other avaliable tracers based on requirement
sudo bash -c "echo function > $TRACEDIR/current_tracer
  • When tracing is turned ON by setting tracing_on=1, the trace file keeps getting append

with the traced data until tracing_on=0 and then ftrace_buffer gets cleared.

To Stop/Pause,
echo 0 >tracing_on;

To Start/Restart,
echo 1 >tracing_on;
  • Once tracing is disabled, disable_trace.sh script is triggered.
5.10. BREAKTRACE
  • Send break trace command when latency > USEC. This is a debugging option to control the latency

tracer in the realtime preemption patch. It is useful to track down unexpected large latencies on a system. This option does only work with following kernel config options.

For kernel < 2.6.24: * CONFIG_PREEMPT_RT=y * CONFIG_WAKEUP_TIMING=y * CONFIG_LATENCY_TRACE=y * CONFIG_CRITICAL_PREEMPT_TIMING=y * CONFIG_CRITICAL_IRQSOFF_TIMING=y

For kernel >= 2.6.24: * CONFIG_PREEMPT_RT=y * CONFIG_FTRACE * CONFIG_IRQSOFF_TRACER=y * CONFIG_PREEMPT_TRACER=y * CONFIG_SCHED_TRACER=y * CONFIG_WAKEUP_LATENCY_HIST

  • Kernel configuration options enabled. The USEC parameter to the -b option defines a maximum

latency value, which is compared against the actual latencies of the test. Once the measured latency is higher than the given maximum, the kernel tracer and cyclictest is stopped. The trace can be read from /proc/latency_trace. Please be aware that the tracer adds significant overhead to the kernel, so the latencies will be much higher than on a kernel with latency tracing disabled.

  • Breaktrace option will enable the trace by default, suppress the tracing by using –notrace option.
5.11. Post-execute scripts

post-execute script to yardstick node context teardown is added to disable the ftrace soon after the completion of cyclictest execution throughyardstick. This option is implemented to collect only required ftrace logs for effective debugging if needed.

5.12. Details of disable_trace Script

In disable trace script the following are done:

  • The trace file is copied and moved to /tmp folder based on timestamp
  • The current tracer file is set to nop
  • The set_event file is cleared i.e., all the enabled events are disabled
  • Kernel Ftrace is disabled/unmounted
5.13. Publishing Ftrace logs:

The generated trace log is pushed to artifacts by kvmfornfv-upload-artifact.sh script available in releng which will be triggered as a part of kvm4nfv daily job. The trigger in the script is.,

echo "Uploading artifacts for future debugging needs...."
gsutil cp -r $WORKSPACE/build_output/log-*.tar.gz $GS_LOG_LOCATION > $WORKSPACE/gsutil.log 2>&1
6. KVM4NFV Dashboard Guide
6.1. Dashboard for KVM4NFV Daily Test Results
6.2. Abstract

This chapter explains the procedure to configure the InfluxDB and Grafana on Node1 or Node2 depending on the testtype to publish KVM4NFV test results. The cyclictest cases are executed and results are published on Yardstick Dashboard(Grafana). InfluxDB is the database which will store the cyclictest results and Grafana is a visualisation suite to view the maximum,minimum and average values of the time series data of cyclictest results.The framework is shown in below image.

_images/dashboard-architecture.png
6.3. Version Features
Release Features
Colorado
  • Data published in Json file format
  • No database support to store the test’s latency values of cyclictest
  • For each run, the previous run’s output file is replaced with a new file with currents latency values.
Danube
  • Test results are stored in Influxdb
  • Graphical representation of the latency values using Grafana suite. (Dashboard)
  • Supports graphical view for multiple testcases and test-types (Stress/Idle)
6.4. Installation Steps:

To configure Yardstick, InfluxDB and Grafana for KVM4NFV project following sequence of steps are followed:

Note:

All the below steps are done as per the script, which is a part of CICD integration of kvmfornfv.

For Yardstick:
git clone https://gerrit.opnfv.org/gerrit/yardstick

For InfluxDB:
docker pull tutum/influxdb
docker run -d --name influxdb -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 tutum/influxdb
docker exec -it influxdb bash
$influx
>CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
>CREATE DATABASE yardstick;
>use yardstick;
>show MEASUREMENTS;

For Grafana:
docker pull grafana/grafana
docker run -d --name grafana -p 3000:3000 grafana/grafana

The Yardstick document for Grafana and InfluxDB configuration can be found here.

6.5. Configuring the Dispatcher Type:

Need to configure the dispatcher type in /etc/yardstick/yardstick.conf depending on the dispatcher methods which are used to store the cyclictest results. A sample yardstick.conf can be found at /yardstick/etc/yardstick.conf.sample, which can be copied to /etc/yardstick.

mkdir -p /etc/yardstick/
cp /yardstick/etc/yardstick.conf.sample /etc/yardstick/yardstick.conf

Dispatcher Modules:

Three type of dispatcher methods are available to store the cyclictest results.

  • File
  • InfluxDB
  • HTTP

1. File: Default Dispatcher module is file. If the dispatcher module is configured as a file, then the test results are stored in a temporary file yardstick.out(default path: /tmp/yardstick.out). Dispatcher module of “Verify Job” is “Default”. So,the results are stored in Yardstick.out file for verify job. Storing all the verify jobs in InfluxDB database causes redundancy of latency values. Hence, a File output format is prefered.

[DEFAULT]
debug = False
dispatcher = file

[dispatcher_file]
file_path = /tmp/yardstick.out
max_bytes = 0
backup_count = 0

2. Influxdb: If the dispatcher module is configured as influxdb, then the test results are stored in Influxdb. Users can check test resultsstored in the Influxdb(Database) on Grafana which is used to visualizethe time series data.

To configure the influxdb, the following content in /etc/yardstick/yardstick.conf need to updated

[DEFAULT]
debug = False
dispatcher = influxdb

[dispatcher_influxdb]
timeout = 5
target = http://127.0.0.1:8086  ##Mention the IP where influxdb is running
db_name = yardstick
username = root
password = root

Dispatcher module of “Daily Job” is Influxdb. So, the results are stored in influxdb and then published to Dashboard.

3. HTTP: If the dispatcher module is configured as http, users can check test result on OPNFV testing dashboard which uses MongoDB as backend.

[DEFAULT]
debug = False
dispatcher = http

[dispatcher_http]
timeout = 5
target = http://127.0.0.1:8000/results
_images/UseCaseDashboard.png
6.5.1. Detailing the dispatcher module in verify and daily Jobs:

KVM4NFV updates the dispatcher module in the yardstick configuration (/etc/yardstick/yardstick.conf) file depending on the Job type(Verify/Daily). Once the test is completed, results are published to the respective dispatcher modules.

Dispatcher module is configured for each Job type as mentioned below.

1. Verify Job : Default “DISPATCHER_TYPE” i.e. file(/tmp/yardstick.out) is used. User can also see the test results on Jenkins console log.

*"max": "00030", "avg": "00006", "min": "00006"*
  1. Daily Job : Opnfv Influxdb url is configured as dispatcher module.
DISPATCHER_TYPE=influxdb
DISPATCHER_INFLUXDB_TARGET="http://104.197.68.199:8086"

Influxdb only supports line protocol, and the json protocol is deprecated.

For example, the raw_result of cyclictest in json format is:
"benchmark": {
     "timestamp": 1478234859.065317,
     "errors": "",
     "data": {
        "max": "00012",
        "avg": "00008",
        "min": "00007"
     },
   "sequence": 1
   },
  "runner_id": 23
}
With the help of “influxdb_line_protocol”, the json is transformed as a line string:
'kvmfornfv_cyclictest_idle_idle,deploy_scenario=unknown,host=kvm.LF,
installer=unknown,pod_name=unknown,runner_id=23,scenarios=Cyclictest,
task_id=e7be7516-9eae-406e-84b6-e931866fa793,version=unknown
avg="00008",max="00012",min="00007" 1478234859065316864'

Influxdb api which is already implemented in Influxdb will post the data in line format into the database.

Displaying Results on Grafana dashboard:

  • Once the test results are stored in Influxdb, dashboard configuration file(Json) which used to

display the cyclictest results on Grafana need to be created by following the Grafana-procedure and then pushed into yardstick-repo

  • Grafana can be accessed at Login using credentials opnfv/opnfv and used for visualizing the

collected test data as shown in Visual

_images/Dashboard-screenshot-1.png
_images/Dashboard-screenshot-2.png
6.6. Understanding Kvm4nfv Grafana Dashboard

The Kvm4nfv dashboard found at http://testresults.opnfv.org/ currently supports graphical view of cyclictest. For viewing Kvm4nfv dashboarduse,

http://testresults.opnfv.org/grafana/dashboard/db/kvmfornfv-cyclictest

The login details are:

    Username: opnfv
    Password: opnfv
The JSON of the kvmfonfv-cyclictest dashboard can be found at.,

$ git clone https://gerrit.opnfv.org/gerrit/yardstick.git
$ cd yardstick/dashboard
$ cat KVMFORNFV-Cyclictest

The Dashboard has four tables, each representing a specific test-type of cyclictest case,

  • Kvmfornfv_Cyclictest_Idle-Idle
  • Kvmfornfv_Cyclictest_CPUstress-Idle
  • Kvmfornfv_Cyclictest_Memorystress-Idle
  • Kvmfornfv_Cyclictest_IOstress-Idle

Note:

  • For all graphs, X-axis is marked with time stamps, Y-axis with value in microsecond units.

A brief about what each graph of the dashboard represents:

6.6.1. 1. Idle-Idle Graph

Idle-Idle graph displays the Average, Maximum and Minimum latency values obtained by running Idle_Idle test-type of the cyclictest. Idle_Idle implies that no stress is applied on the Host or the Guest.

_images/Idle-Idle.png
6.6.2. 2. CPU_Stress-Idle Graph

Cpu_Stress-Idle graph displays the Average, Maximum and Minimum latency values obtained by running Cpu-stress_Idle test-type of the cyclictest. Cpu-stress_Idle implies that CPU stress is applied on the Host and no stress on the Guest.

_images/Cpustress-Idle.png
6.6.3. 3. Memory_Stress-Idle Graph

Memory_Stress-Idle graph displays the Average, Maximum and Minimum latency values obtained by running Memory-stress_Idle test-type of the Cyclictest. Memory-stress_Idle implies that Memory stress is applied on the Host and no stress on the Guest.

_images/Memorystress-Idle.png
6.6.4. 4. IO_Stress-Idle Graph

IO_Stress-Idle graph displays the Average, Maximum and Minimum latency values obtained by running IO-stress_Idle test-type of the Cyclictest. IO-stress_Idle implies that IO stress is applied on the Host and no stress on the Guest.

_images/IOstress-Idle.png
6.6.5. Packet Forwarding Results
6.7. Understanding Kvm4nfv Grafana Dashboard

The Kvm4nfv dashboard found at http://testresults.opnfv.org/grafana/ currently supports graphical view of packet forwarding as well. For viewing Kvm4nfv packet forwarding dashboard use,

http://testresults.opnfv.org/grafana/dashboard/db/kvmfornfv-packet-forwarding

The login details are:

    Username: opnfv
    Password: opnfv
The JSON of the KVMFORNFV-Packet-Forwarding dashboard can be found at.,

$ git clone https://gerrit.opnfv.org/gerrit/yardstick.git
$ cd yardstick/dashboard
$ cat KVMFORNFV-Packet-Forwarding

The Dashboard has five tables for each specific test of packet forwarding, one for each frame size.

  • KVM4NFV-PHY2PHY-TPUT-OVS_WITH_DPDK_AND_VHOST_USER
  • KVM4NFV-PVP-TPUT-OVS_WITH_DPDK_AND_VHOST_USER
  • KVM4NFV-PVP-TPUT-SRIOV
  • KVM4NFV-PVVP-TPUT-OVS_WITH_DPDK_AND_VHOST_USER
  • KVM4NFV-PVVP-TPUT-OVS_WITH_DPDK_AND_VHOST_USER

Note:

  • For all graphs, X-axis is marked with time stamps, Y-axis with value in microsecond units.
6.8. Future Scope

The future work will include adding new tables to packet forwarding Grafana dashboard to publish the results of new packet forwarding test cases to be added if any.

7. Low Latency Environment

Achieving low latency with the KVM4NFV project requires setting up a special test environment. This environment includes the BIOS settings, kernel configuration, kernel parameters and the run-time environment.

7.1. Hardware Environment Description

BIOS setup plays an important role in achieving real-time latency. A collection of relevant settings, used on the platform where the baseline performance data was collected, is detailed below:

7.1.1. CPU Features

Some special CPU features like TSC-deadline timer, invariant TSC and Process posted interrupts, etc, are helpful for latency reduction.

7.1.2. CPU Topology

NUMA topology is also important for latency reduction.

7.1.3. BIOS Setup

Careful BIOS setup is important in achieving real time latency. Different platforms have different BIOS setups, below are the important BIOS settings on the platform used to collect the baseline performance data.

7.2. Software Environment Setup

Both the host and the guest environment need to be configured properly to reduce latency variations. Below are some suggested kernel configurations. The ci/envs/ directory gives detailed implementation on how to setup the environment.

7.2.1. Kernel Parameter

Please check the default kernel configuration in the source code at: kernel/arch/x86/configs/opnfv.config.

Below is host kernel boot line example:

isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35
iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll
intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable

Below is guest kernel boot line example

isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G
hugepagesz=1G

Please refer to tuning.userguide for more explanation.

7.2.2. Run-time Environment Setup

Not only are special kernel parameters needed but a special run-time environment is also required. Please refer to tunning.userguide for more explanation.

7.3. Test cases to measure Latency

The performance of the kvm4nfv is assesed by the latency values. Cyclictest and Packet forwarding Test cases result in real time latency values of average, minimum and maximum.

  • Cyclictest
  • Packet Forwarding test
7.4. 1. Cyclictest case

Cyclictest results are the most frequently cited real-time Linux metric. The core concept of Cyclictest is very simple. In KVM4NFV cyclictest is implemented on the Guest-VM with 4.4-Kernel RPM installed. It generated Max,Min and Avg values which help in assesing the kernel used. Cyclictest in currently divided into the following test types,

  • Idle-Idle
  • CPU_stress-Idle
  • Memory_stress-Idle
  • IO_stress-Idle

Future scope of work may include the below test-types,

  • CPU_stress-CPU_stress
  • Memory_stress-Memory_stress
  • IO_stress-IO_stress
7.4.1. Understanding the naming convention
[Host-Type ] - [Guest-Type]
  • Host-Type : Mentions the type of stress applied on the kernel of the Host
  • Guest-Type : Mentions the type of stress applied on the kernel of the Guest

Example.,

Idle - CPU_stress

The above name signifies that,

  • No Stress is applied on the Host kernel
  • CPU Stress is applied on the Guest kernel

Note:

  • Stress is applied using the stress which is installed as part of the deployment. Stress can be applied on CPU, Memory and Input-Output (Read/Write) operations using the stress tool.
7.4.2. Version Features
Test Name Colorado Danube Euphrates
  • Idle - Idle
Y Y Y
  • Cpustress - Idle
  Y Y
  • Memorystress - Idle
  Y Y
  • IOstress - Idle
  Y Y
7.4.3. Idle-Idle test-type

Cyclictest in run on the Guest VM when Host,Guest are not under any kind of stress. This is the basic cyclictest of the KVM4NFV project. Outputs Avg, Min and Max latency values.

_images/idle-idle-test-type.png
7.4.4. CPU_Stress-Idle test-type

Here, the host is under CPU stress, where multiple times sqrt() function is called on kernel which results increased CPU load. The cyclictest will run on the guest, where the guest is under no stress. Outputs Avg, Min and Max latency values.

_images/cpu-stress-idle-test-type.png
7.4.5. Memory_Stress-Idle test-type

In this type, the host is under memory stress where continuos memory operations are implemented to increase the Memory stress (Buffer stress).The cyclictest will run on the guest, where the guest is under no stress. It outputs Avg, Min and Max latency values.

_images/memory-stress-idle-test-type.png
7.4.6. IO_Stress-Idle test-type

The host is under constant Input/Output stress .i.e., multiple read-write operations are invoked to increase stress. Cyclictest will run on the guest VM that is launched on the same host, where the guest is under no stress. It outputs Avg, Min and Max latency values.

_images/io-stress-idle-test-type.png
7.4.7. CPU_Stress-CPU_Stress test-type

Not implemented for Euphrates release.

7.4.8. Memory_Stress-Memory_Stress test-type

Not implemented for Euphrates release.

7.4.9. IO_Stress-IO_Stress test type

Not implemented for Euphrates release.

7.5. 2. Packet Forwarding Test cases

Packet forwarding is an other test case of Kvm4nfv. It measures the time taken by a packet to return to source after reaching its destination. This test case uses automated test-framework provided by OPNFV VSWITCHPERF project and a traffic generator (IXIA is used for kvm4nfv). Only latency results generating test cases are triggered as a part of kvm4nfv daily job.

Latency test measures the time required for a frame to travel from the originating device through the network to the destination device. Please note that RFC2544 Latency measurement will be superseded with a measurement of average latency over all successfully transferred packets or frames.

Packet forwarding test cases currently supports the following test types:

  • Packet forwarding to Host
  • Packet forwarding to Guest
  • Packet forwarding to Guest using SRIOV

The testing approach adoped is black box testing, meaning the test inputs can be generated and the outputs captured and completely evaluated from the outside of the System Under Test(SUT).

7.5.1. Packet forwarding to Host

This is also known as Physical port → vSwitch → physical port deployment. This test measures the time taken by the packet/frame generated by traffic generator(phy) to travel through the network to the destination device(phy). This test results min,avg and max latency values. This value signifies the performance of the installed kernel.

Packet flow,

_images/host_pk_fw.png
7.5.2. Packet forwarding to Guest

This is also known as Physical port → vSwitch → VNF → vSwitch → physical port deployment.

This test measures the time taken by the packet/frame generated by traffic generator(phy) to travel through the network involving a guest to the destination device(phy). This test results min,avg and max latency values. This value signifies the performance of the installed kernel.

Packet flow,

_images/guest_pk_fw.png
7.5.3. Packet forwarding to Guest using SRIOV

This test is used to verify the VNF and measure the base performance (maximum forwarding rate in fps and latency) that can be achieved by the VNF without a vSwitch. The performance metrics collected by this test will serve as a key comparison point for NIC passthrough technologies and vSwitches. VNF in this context refers to the hypervisor and the VM.

Note: The Vsperf running on the host is still required.

Packet flow,

_images/sriov_pk_fw.png
8. Fast Live Migration

The NFV project requires fast live migration. The specific requirement is total live migration time < 2Sec, while keeping the VM down time < 10ms when running DPDK L2 forwarding workload.

We measured the baseline data of migrating an idle 8GiB guest running a DPDK L2 forwarding work load and observed that the total live migration time was 2271ms while the VM downtime was 26ms. Both of these two indicators failed to satisfy the requirements.

8.1. Current Challenges

The following 4 features have been developed over the years to make the live migration process faster.

  • XBZRLE:

    Helps to reduce the network traffic by just sending the compressed data.

  • RDMA:

    Uses a specific NIC to increase the efficiency of data transmission.

  • Multi thread compression:

    Compresses the data before transmission.

  • Auto convergence:

    Reduces the data rate of dirty pages.

Tests show none of the above features can satisfy the requirement of NFV. XBZRLE and Multi thread compression do the compression entirely in software and they are not fast enough in a 10Gbps network environment. RDMA is not flexible because it has to transport all the guest memory to the destination without zero page optimization. Auto convergence is not appropriate for NFV because it will impact guest’s performance.

So we need to find other ways for optimization.

8.2. Optimizations
  1. Delay non-emergency operations By profiling, it was discovered that some of the cleanup operations during the stop and copy stage are the main reason for the long VM down time. The cleanup operation includes stopping the dirty page logging, which is a time consuming operation. By deferring these operations until the data transmission is completed the VM down time is reduced to about 5-7ms.
  2. Optimize zero page checking Currently QEMU uses the SSE2 instruction to optimize the zero pages checking. The SSE2 instruction can process 16 bytes per instruction. By using the AVX2 instruction, we can process 32 bytes per instruction. Testing shows that using AVX2 can speed up the zero pages checking process by about 25%.
  3. Remove unnecessary context synchronization. The CPU context was being synchronized twice during live migration. Removing this unnecessary synchronization shortened the VM downtime by about 100us.
8.3. Test Environment

The source and destination host have the same hardware and OS: :: Host: HSW-EP CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz RAM: 64G OS: RHEL 7.1 Kernel: 4.2 QEMU v2.4.0

Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

Vhost-user with OVS/DPDK as backend: :: The goal is to connect guests’ virtio-net devices having vhost-user backend to OVS dpdkvhostuser ports and be able to run any kind of network traffic between them.

Installation of OVS and DPDK: :: Using vsperf,installing the OVS and DPDk. Prepare the directories

mkdir -p /var/run/openvswitch
mount -t hugetlbfs -o pagesize=2048k none /dev/hugepages

Load Kernel modules

modprobe openvswitch

For OVS setup, clean the environment

rm -f /usr/local/var/run/openvswitch/vhost-user*
rm -f /usr/local/etc/openvswitch/conf.db

Start database server

ovsdb-tool create /usr/local/etc/openvswitch/conf.db $VSPERF/src/ovs/ovs/vswitchd/vswitch.ovsschema
ovsdb-server --remote=punix:$DB_SOCK --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach

Start OVS

ovs-vsctl --no-wait init
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xf
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

Configure the bridge

ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
ovs-vsctl add-port ovsbr0 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser

QEMU parameters: :: qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -chardev socket,id=char1,path=/usr/local/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=net1,chardev=char1,vhostforce -device virtio-net-pci,netdev=net1,mac=52:54:00:12:34:56 -chardev socket,id=char2,path=/usr/local/var/run/openvswitch/vhost-user2-netdev type=vhost-user,id=net2,chardev=char2,vhostforce -device virtio-net-pci,netdev=net2,mac=54:54:00:12:34:56 -m 1024 -mem-path /dev/hugepages -mem-prealloc -realtime mlock=on -monitor unix:/tmp/qmp-sock-src,server,nowait -balloon virtio -drive file=/root/guest1.qcow2 -vnc :1 &

Run the standby qemu with -incoming tcp:-incoming tcp:${incoming_ip}:${migrate_port}:${migrate_port}

For local live migration

incoming ip=0

For peer -peer live migration

incoming ip=dest_host

Network connection

live migration network connection

Commands for performing live migration:

.. code:: bash
echo “migrate_set_speed 0” |nc -U /tmp/qmp-sock-src echo “migrate_set_downtime 0.10” |nc -U /tmp/qmp-sock-src echo “migrate -d tcp:0:4444|nc -U /tmp/qmp-sock-src #Wait till livemigration completed echo “info migrate” | nc -U /tmp/qmp-sock-src
8.4. Test Result

The down time is set to 10ms when doing the test. We use pktgen to send the packages to guest, the package size is 64 bytes, and the line rate is 2013 Mbps.

  1. Total live migration time

    The total live migration time before and after optimization is shown in the chart below. For an idle guest, we can reduce the total live migration time from 2070ms to 401ms. For a guest running the DPDK L2 forwarding workload, the total live migration time is reduced from 2271ms to 654ms.

total live migration time
  1. VM downtime

    The VM down time before and after optimization is shown in the chart below. For an idle guest, we can reduce the VM down time from 29ms to 9ms. For a guest running the DPDK L2 forwarding workload, the VM down time is reduced from 26ms to 5ms.

vm downtime
9. Euphrates OpenStack User Guide

OpenStack is a cloud operating system developed and released by the OpenStack project. OpenStack is used in OPNFV for controlling pools of compute, storage, and networking resources in a Pharos compliant infrastructure.

OpenStack is used in Euphrates to manage tenants (known in OpenStack as projects),users, services, images, flavours, and quotas across the Pharos infrastructure.The OpenStack interface provides the primary interface for an operational Euphrates deployment and it is from the “horizon console” that an OPNFV user will perform the majority of administrative and operational activities on the deployment.

9.1. OpenStack references

The OpenStack user guide provides details and descriptions of how to configure and interact with the OpenStack deployment.This guide can be used by lab engineers and operators to tune the OpenStack deployment to your liking.

Once you have configured OpenStack to your purposes, or the Euphrates deployment meets your needs as deployed, an operator, or administrator, will find the best guidance for working with OpenStack in the OpenStack administration guide.

9.2. Connecting to the OpenStack instance

Once familiar with the basic of working with OpenStack you will want to connect to the OpenStack instance via the Horizon Console. The Horizon console provide a Web based GUI that will allow you operate the deployment. To do this you should open a browser on the JumpHost to the following address and enter the username and password:

http://{Controller-VIP}:80/index.html> username: admin password: admin

Other methods of interacting with and configuring OpenStack,, like the REST API and CLI are also available in the Euphrates deployment, see the OpenStack administration guide for more information on using those interfaces.

10. Packet Forwarding
10.1. About Packet Forwarding

Packet Forwarding is a test suite of KVM4NFV. These latency tests measures the time taken by a Packet generated by the traffic generator to travel from the originating device through the network to the destination device. Packet Forwarding is implemented using test framework implemented by OPNFV VSWITCHPERF project and an IXIA Traffic Generator.

10.2. Version Features
Release Features
Colorado
  • Packet Forwarding is not part of Colorado release of KVM4NFV
Danube
  • Packet Forwarding is a testcase in KVM4NFV
  • Implements three scenarios (Host/Guest/SRIOV) as part of testing in KVM4NFV
  • Uses automated test framework of OPNFV VSWITCHPERF software (PVP/PVVP)
  • Works with IXIA Traffic Generator
Euphrates
  • Test cases involving multiple guests (PVVP/PVPV) included.
  • Implemented Yardstick Grafana dashboard to publish results of packet forwarding test cases
10.3. VSPERF

VSPerf is an OPNFV testing project. VSPerf will develop a generic and architecture agnostic vSwitch testing framework and associated tests, that will serve as a basis for validating the suitability of different vSwitch implementations in a Telco NFV deployment environment. The output of this project will be utilized by the OPNFV Performance and Test group and its associated projects, as part of OPNFV Platform and VNF level testing and validation.

For complete VSPERF documentation go to link.

10.3.1. Installation

Guidelines of installating VSPERF.

10.3.2. Supported Operating Systems
  • CentOS 7
  • Fedora 20
  • Fedora 21
  • Fedora 22
  • RedHat 7.2
  • Ubuntu 14.04
10.3.3. Supported vSwitches

The vSwitch must support Open Flow 1.3 or greater.

  • OVS (built from source).
  • OVS with DPDK (built from source).
10.3.4. Supported Hypervisors
  • Qemu version 2.6.
10.3.5. Other Requirements

The test suite requires Python 3.3 and relies on a number of other packages. These need to be installed for the test suite to function.

Installation of required packages, preparation of Python 3 virtual environment and compilation of OVS, DPDK and QEMU is performed by script systems/build_base_machine.sh. It should be executed under user account, which will be used for vsperf execution.

Please Note: Password-less sudo access must be configured for given user before script is

executed.

Execution of installation script:

$ cd vswitchperf
$ cd systems
$ ./build_base_machine.sh

Script build_base_machine.sh will install all the vsperf dependencies in terms of system packages, Python 3.x and required Python modules. In case of CentOS 7 it will install Python 3.3 from an additional repository provided by Software Collections (a link). In case of RedHat 7 it will install Python 3.4 as an alternate installation in /usr/local/bin. Installation script will also use virtualenv to create a vsperf virtual environment, which is isolated from the default Python environment. This environment will reside in a directory called vsperfenv in $HOME.

You will need to activate the virtual environment every time you start a new shell session. Its activation is specific to your OS:

For running testcases VSPERF is installed on Intel pod1-node2 in which centos operating system is installed. Only VSPERF installion on Centos is discussed here. For installation steps on other operating systems please refer to here.

10.3.6. For CentOS 7

## Python 3 Packages

To avoid file permission errors and Python version issues, use virtualenv to create an isolated environment with Python3. The required Python 3 packages can be found in the requirements.txt file in the root of the test suite. They can be installed in your virtual environment like so:

scl enable python33 bash
# Create virtual environment
virtualenv vsperfenv
cd vsperfenv
source bin/activate
pip install -r requirements.txt

You need to activate the virtual environment every time you start a new shell session. To activate, simple run:

scl enable python33 bash
cd vsperfenv
source bin/activate
10.3.7. Working Behind a Proxy

If you’re behind a proxy, you’ll likely want to configure this before running any of the above. For example:

export http_proxy="http://<username>:<password>@<proxy>:<port>/";
export https_proxy="https://<username>:<password>@<proxy>:<port>/";
export ftp_proxy="ftp://<username>:<password>@<proxy>:<port>/";
export socks_proxy="socks://<username>:<password>@<proxy>:<port>/";

For other OS specific activation click this link:

http://artifacts.opnfv.org/vswitchperf/colorado/configguide/installation.html#other-requirements

10.4. Traffic-Generators

VSPERF supports many Traffic-generators. For configuring VSPERF to work with the available traffic generator go through this.

VSPERF supports the following traffic generators:

  • Dummy (DEFAULT): Allows you to use your own external traffic generator.
  • IXIA (IxNet and IxOS)
  • Spirent TestCenter
  • Xena Networks
  • MoonGen

To see the list of traffic gens from the cli:

$ ./vsperf --list-trafficgens

This guide provides the details of how to install and configure the various traffic generators.

As KVM4NFV uses only IXIA traffic generator, it is discussed here. For complete documentation regarding traffic generators please follow this link.

10.5. IXIA Setup
10.5.1. Hardware Requirements

VSPERF requires the following hardware to run tests: IXIA traffic generator (IxNetwork), a machine that runs the IXIA client software and a CentOS Linux release 7.1.1503 (Core) host.

10.5.2. Installation

Follow the installation instructions to install.

10.5.3. On the CentOS 7 system

You need to install IxNetworkTclClient$(VER_NUM)Linux.bin.tgz.

10.5.4. On the IXIA client software system
Find the IxNetwork TCL server app
  • (start -> All Programs -> IXIA -> IxNetwork -> IxNetwork_$(VER_NUM) -> IxNetwork TCL Server)
  • Right click on IxNetwork TCL Server, select properties
  • Under shortcut tab in the Target dialogue box make sure there is the argument “-tclport xxxx”

where xxxx is your port number (take note of this port number you will need it for the 10_custom.conf file).

_images/IXIA1.png
  • Hit Ok and start the TCL server application
10.6. VSPERF configuration

There are several configuration options specific to the IxNetworks traffic generator from IXIA. It is essential to set them correctly, before the VSPERF is executed for the first time.

Detailed description of options follows:

  • TRAFFICGEN_IXNET_MACHINE - IP address of server, where IxNetwork TCL Server is running
  • TRAFFICGEN_IXNET_PORT - PORT, where IxNetwork TCL Server is accepting connections from TCL clients
  • TRAFFICGEN_IXNET_USER - username, which will be used during communication with IxNetwork TCL Server and IXIA chassis
  • TRAFFICGEN_IXIA_HOST - IP address of IXIA traffic generator chassis
  • TRAFFICGEN_IXIA_CARD - identification of card with dedicated ports at IXIA chassis
  • TRAFFICGEN_IXIA_PORT1 - identification of the first dedicated port at TRAFFICGEN_IXIA_CARD at IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of unidirectional traffic, it is essential to correctly connect 1st IXIA port to the 1st NIC at DUT, i.e. to the first PCI handle from WHITELIST_NICS list. Otherwise traffic may not be able to pass through the vSwitch.
  • TRAFFICGEN_IXIA_PORT2 - identification of the second dedicated port at TRAFFICGEN_IXIA_CARD at IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of unidirectional traffic, it is essential to correctly connect 2nd IXIA port to the 2nd NIC at DUT, i.e. to the second PCI handle from WHITELIST_NICS list. Otherwise traffic may not be able to pass through the vSwitch.
  • TRAFFICGEN_IXNET_LIB_PATH - path to the DUT specific installation of IxNetwork TCL API
  • TRAFFICGEN_IXNET_TCL_SCRIPT - name of the TCL script, which VSPERF will use for communication with IXIA TCL server
  • TRAFFICGEN_IXNET_TESTER_RESULT_DIR - folder accessible from IxNetwork TCL server, where test results are stored, e.g. c:/ixia_results; see test-results-share
  • TRAFFICGEN_IXNET_DUT_RESULT_DIR - directory accessible from the DUT, where test results from IxNetwork TCL server are stored, e.g. /mnt/ixia_results; see test-results-share
10.6.1. Test results share

VSPERF is not able to retrieve test results via TCL API directly. Instead, all test results are stored at IxNetwork TCL server. Results are stored at folder defined by TRAFFICGEN_IXNET_TESTER_RESULT_DIR configuration parameter. Content of this folder must be shared (e.g. via samba protocol) between TCL Server and DUT, where VSPERF is executed. VSPERF expects, that test results will be available at directory configured by TRAFFICGEN_IXNET_DUT_RESULT_DIR configuration parameter.

Example of sharing configuration:

  • Create a new folder at IxNetwork TCL server machine, e.g. c:\ixia_results

  • Modify sharing options of ixia_results folder to share it with everybody

  • Create a new directory at DUT, where shared directory with results will be mounted, e.g. /mnt/ixia_results

  • Update your custom VSPERF configuration file as follows:

    TRAFFICGEN_IXNET_TESTER_RESULT_DIR = 'c:/ixia_results'
    TRAFFICGEN_IXNET_DUT_RESULT_DIR = '/mnt/ixia_results'
    

    Note: It is essential to use slashes ‘/’ also in path configured by TRAFFICGEN_IXNET_TESTER_RESULT_DIR parameter.

  • Install cifs-utils package.

    e.g. at rpm based Linux distribution:

yum install cifs-utils
  • Mount shared directory, so VSPERF can access test results.

    e.g. by adding new record into /etc/fstab

mount -t cifs //_TCL_SERVER_IP_OR_FQDN_/ixia_results /mnt/ixia_results
      -o file_mode=0777,dir_mode=0777,nounix

It is recommended to verify, that any new file inserted into c:/ixia_results folder is visible at DUT inside /mnt/ixia_results directory.

10.6.2. Cloning and building src dependencies

In order to run VSPERF, you will need to download DPDK and OVS. You can do this manually and build them in a preferred location, or you could use vswitchperf/src. The vswitchperf/src directory contains makefiles that will allow you to clone and build the libraries that VSPERF depends on, such as DPDK and OVS. To clone and build simply:

cd src
make

To delete a src subdirectory and its contents to allow you to re-clone simply use:

make cleanse
10.6.3. Configure the ./conf/10_custom.conf file

The supplied 10_custom.conf file must be modified, as it contains configuration items for which there are no reasonable default values.

The configuration items that can be added is not limited to the initial contents. Any configuration item mentioned in any .conf file in ./conf directory can be added and that item will be overridden by the custom configuration value.

10.6.4. Using a custom settings file

Alternatively a custom settings file can be passed to vsperf via the –conf-file argument.

./vsperf --conf-file <path_to_settings_py> ...

Note that configuration passed in via the environment (–load-env) or via another command line argument will override both the default and your custom configuration files. This “priority hierarchy” can be described like so (1 = max priority):

  1. Command line arguments
  2. Environment variables
  3. Configuration file(s)
10.6.5. vloop_vnf

VSPERF uses a VM image called vloop_vnf for looping traffic in the deployment scenarios involving VMs. The image can be downloaded from http://artifacts.opnfv.org/.

Please see the installation instructions for information on vloop-vnf images.

10.6.6. l2fwd Kernel Module

A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd

10.6.7. Executing tests

Before running any tests make sure you have root permissions by adding the following line to /etc/sudoers:

username ALL=(ALL)       NOPASSWD: ALL

username in the example above should be replaced with a real username.

To list the available tests:

./vsperf --list-tests

To run a group of tests, for example all tests with a name containing ‘RFC2544’:

./vsperf --conf-file=user_settings.py --tests="RFC2544"

To run all tests:

./vsperf --conf-file=user_settings.py

Some tests allow for configurable parameters, including test duration (in seconds) as well as packet sizes (in bytes).

./vsperf --conf-file user_settings.py
    --tests RFC2544Tput
    --test-param` "rfc2544_duration=10;packet_sizes=128"

For all available options, check out the help dialog:

./vsperf --help
10.7. Testcases

Available Tests in VSPERF are:

  • phy2phy_tput
  • phy2phy_forwarding
  • back2back
  • phy2phy_tput_mod_vlan
  • phy2phy_cont
  • pvp_cont
  • pvvp_cont
  • pvpv_cont
  • phy2phy_scalability
  • pvp_tput
  • pvp_back2back
  • pvvp_tput
  • pvvp_back2back
  • phy2phy_cpu_load
  • phy2phy_mem_load
10.8. VSPERF modes of operation

VSPERF can be run in different modes. By default it will configure vSwitch, traffic generator and VNF. However it can be used just for configuration and execution of traffic generator. Another option is execution of all components except traffic generator itself.

Mode of operation is driven by configuration parameter -m or –mode

-m MODE, --mode MODE  vsperf mode of operation;
   Values:
        "normal" - execute vSwitch, VNF and traffic generator
        "trafficgen" - execute only traffic generator
        "trafficgen-off" - execute vSwitch and VNF
        "trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission

In case, that VSPERF is executed in “trafficgen” mode, then configuration of traffic generator can be modified through TRAFFIC dictionary passed to the --test-params option. It is not needed to specify all values of TRAFFIC dictionary. It is sufficient to specify only values, which should be changed. Detailed notes on TRAFFIC dictionary can be found at: ref:configuration-of-traffic-dictionary.

Example of execution of VSPERF in “trafficgen” mode:

$ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \
    --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}"
10.9. Packet Forwarding Test Scenarios

KVM4NFV currently implements three scenarios as part of testing:

  • Host Scenario
  • Guest Scenario.
  • SR-IOV Scenario.
10.9.1. Packet Forwarding Host Scenario

Here host DUT has VSPERF installed in it and is properly configured to use IXIA Traffic-generator by providing IXIA CARD, PORTS and Lib paths along with IP. please refer to figure.2

_images/Host_Scenario.png
10.9.2. Packet Forwarding Guest Scenario (PXP Deployment)

Here the guest is a Virtual Machine (VM) launched by using vloop_vnf provided by vsperf project on host/DUT using Qemu. In this latency test the time taken by the frame/packet to travel from the originating device through network involving a guest to destination device is calculated. The resulting latency values will define the performance of installed kernel.

_images/Guest_Scenario.png

Every testcase uses one of the supported deployment scenarios to setup test environment. The controller responsible for a given scenario configures flows in the vswitch to route traffic among physical interfaces connected to the traffic generator and virtual machines. VSPERF supports several deployments including PXP deployment, which can setup various scenarios with multiple VMs.

These scenarios are realized by VswitchControllerPXP class, which can configure and execute given number of VMs in serial or parallel configurations. Every VM can be configured with just one or an even number of interfaces. In case that VM has more than 2 interfaces, then traffic is properly routed among pairs of interfaces.

Example of traffic routing for VM with 4 NICs in serial configuration:

         +------------------------------------------+
         |  VM with 4 NICs                          |
         |  +---------------+    +---------------+  |
         |  |  Application  |    |  Application  |  |
         |  +---------------+    +---------------+  |
         |      ^       |            ^       |      |
         |      |       v            |       v      |
         |  +---------------+    +---------------+  |
         |  | logical ports |    | logical ports |  |
         |  |   0       1   |    |   2       3   |  |
         +--+---------------+----+---------------+--+
                ^       :            ^       :
                |       |            |       |
                :       v            :       v
+-----------+---------------+----+---------------+----------+
| vSwitch   |   0       1   |    |   2       3   |          |
|           | logical ports |    | logical ports |          |
| previous  +---------------+    +---------------+   next   |
| VM or PHY     ^       |            ^       |     VM or PHY|
|   port   -----+       +------------+       +--->   port   |
+-----------------------------------------------------------+

It is also possible to define different number of interfaces for each VM to better simulate real scenarios.

The number of VMs involved in the test and the type of their connection is defined by deployment name as follows:

  • pvvp[number] - configures scenario with VMs connected in series with optional number of VMs. In case that number is not specified, then 2 VMs will be used.

    Example of 2 VMs in a serial configuration:

    +----------------------+  +----------------------+
    |   1st VM             |  |   2nd VM             |
    |   +---------------+  |  |   +---------------+  |
    |   |  Application  |  |  |   |  Application  |  |
    |   +---------------+  |  |   +---------------+  |
    |       ^       |      |  |       ^       |      |
    |       |       v      |  |       |       v      |
    |   +---------------+  |  |   +---------------+  |
    |   | logical ports |  |  |   | logical ports |  |
    |   |   0       1   |  |  |   |   0       1   |  |
    +---+---------------+--+  +---+---------------+--+
            ^       :                 ^       :
            |       |                 |       |
            :       v                 :       v
    +---+---------------+---------+---------------+--+
    |   |   0       1   |         |   3       4   |  |
    |   | logical ports | vSwitch | logical ports |  |
    |   +---------------+         +---------------+  |
    |       ^       |                 ^       |      |
    |       |       +-----------------+       v      |
    |   +----------------------------------------+   |
    |   |              physical ports            |   |
    |   |      0                         1       |   |
    +---+----------------------------------------+---+
               ^                         :
               |                         |
               :                         v
    +------------------------------------------------+
    |                                                |
    |                traffic generator               |
    |                                                |
    +------------------------------------------------+
    
  • pvpv[number] - configures scenario with VMs connected in parallel with

    optional number of VMs. In case that number is not specified, then 2 VMs will be used. Multistream feature is used to route traffic to particular VMs (or NIC pairs of every VM). It means, that VSPERF will enable multistream feaure and sets the number of streams to the number of VMs and their NIC pairs. Traffic will be dispatched based on Stream Type, i.e. by UDP port, IP address or MAC address.

    Example of 2 VMs in a parallel configuration, where traffic is dispatched

    based on the UDP port.

    +----------------------+  +----------------------+
    |   1st VM             |  |   2nd VM             |
    |   +---------------+  |  |   +---------------+  |
    |   |  Application  |  |  |   |  Application  |  |
    |   +---------------+  |  |   +---------------+  |
    |       ^       |      |  |       ^       |      |
    |       |       v      |  |       |       v      |
    |   +---------------+  |  |   +---------------+  |
    |   | logical ports |  |  |   | logical ports |  |
    |   |   0       1   |  |  |   |   0       1   |  |
    +---+---------------+--+  +---+---------------+--+
            ^       :                 ^       :
            |       |                 |       |
            :       v                 :       v
    +---+---------------+---------+---------------+--+
    |   |   0       1   |         |   3       4   |  |
    |   | logical ports | vSwitch | logical ports |  |
    |   +---------------+         +---------------+  |
    |      ^         |                 ^       :     |
    |      |     ......................:       :     |
    |  UDP | UDP :   |                         :     |
    |  port| port:   +--------------------+    :     |
    |   0  |  1  :                        |    :     |
    |      |     :                        v    v     |
    |   +----------------------------------------+   |
    |   |              physical ports            |   |
    |   |    0                               1   |   |
    +---+----------------------------------------+---+
             ^                               :
             |                               |
             :                               v
    +------------------------------------------------+
    |                                                |
    |                traffic generator               |
    |                                                |
    +------------------------------------------------+
    

PXP deployment is backward compatible with PVP deployment, where pvp is an alias for pvvp1 and it executes just one VM.

The number of interfaces used by VMs is defined by configuration option GUEST_NICS_NR. In case that more than one pair of interfaces is defined for VM, then:

  • for pvvp (serial) scenario every NIC pair is connected in serial before connection to next VM is created
  • for pvpv (parallel) scenario every NIC pair is directly connected to the physical ports and unique traffic stream is assigned to it

Examples:

  • Deployment pvvp10 will start 10 VMs and connects them in series
  • Deployment pvpv4 will start 4 VMs and connects them in parallel
  • Deployment pvpv1 and GUEST_NICS_NR = [4] will start 1 VM with 4 interfaces and every NIC pair is directly connected to the physical ports
  • Deployment pvvp and GUEST_NICS_NR = [2, 4] will start 2 VMs; 1st VM will have 2 interfaces and 2nd VM 4 interfaces. These interfaces will be connected in serial, i.e. traffic will flow as follows: PHY1 -> VM1_1 -> VM1_2 -> VM2_1 -> VM2_2 -> VM2_3 -> VM2_4 -> PHY2

Note: In case that only 1 or more than 2 NICs are configured for VM, then testpmd should be used as forwarding application inside the VM. As it is able to forward traffic between multiple VM NIC pairs.

Note: In case of linux_bridge, all NICs are connected to the same bridge inside the VM.

10.9.3. Packet Forwarding SRIOV Scenario

In this test the packet generated at the IXIA is forwarded to the Guest VM launched on Host by implementing SR-IOV interface at NIC level of host .i.e., DUT. The time taken by the packet to travel through the network to the destination the IXIA traffic-generator is calculated and published as a test result for this scenario.

SRIOV-support is given below, it details how to use SR-IOV.

_images/SRIOV_Scenario.png
10.9.4. Using vfio_pci with DPDK

To use vfio with DPDK instead of igb_uio add into your custom configuration file the following parameter:

PATHS['dpdk']['src']['modules'] = ['uio', 'vfio-pci']

NOTE: In case, that DPDK is installed from binary package, then please

set PATHS['dpdk']['bin']['modules'] instead.

NOTE: Please ensure that Intel VT-d is enabled in BIOS.

NOTE: Please ensure your boot/grub parameters include the following:

iommu=pt intel_iommu=on

To check that IOMMU is enabled on your platform:

 $ dmesg | grep IOMMU
 [    0.000000] Intel-IOMMU: enabled
 [    0.139882] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
 [    0.139888] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
 [    0.139893] IOAPIC id 2 under DRHD base  0xfbffe000 IOMMU 0
 [    0.139894] IOAPIC id 0 under DRHD base  0xebffc000 IOMMU 1
 [    0.139895] IOAPIC id 1 under DRHD base  0xebffc000 IOMMU 1
 [    3.335744] IOMMU: dmar0 using Queued invalidation
 [    3.335746] IOMMU: dmar1 using Queued invalidation
....
10.9.5. Using SRIOV support

To use virtual functions of NIC with SRIOV support, use extended form of NIC PCI slot definition:

WHITELIST_NICS = ['0000:03:00.0|vf0', '0000:03:00.1|vf3']

Where vf is an indication of virtual function usage and following number defines a VF to be used. In case that VF usage is detected, then vswitchperf will enable SRIOV support for given card and it will detect PCI slot numbers of selected VFs.

So in example above, one VF will be configured for NIC ‘0000:05:00.0’ and four VFs will be configured for NIC ‘0000:05:00.1’. Vswitchperf will detect PCI addresses of selected VFs and it will use them during test execution.

At the end of vswitchperf execution, SRIOV support will be disabled.

SRIOV support is generic and it can be used in different testing scenarios. For example:

  • vSwitch tests with DPDK or without DPDK support to verify impact of VF usage on vSwitch performance
  • tests without vSwitch, where traffic is forwared directly between VF interfaces by packet forwarder (e.g. testpmd application)
  • tests without vSwitch, where VM accesses VF interfaces directly by PCI-passthrough to measure raw VM throughput performance.
10.9.5.1. Using QEMU with PCI passthrough support

Raw virtual machine throughput performance can be measured by execution of PVP test with direct access to NICs by PCI passthrough. To execute VM with direct access to PCI devices, enable vfio-pci. In order to use virtual functions, SRIOV-support must be enabled.

Execution of test with PCI passthrough with vswitch disabled:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
           --vswitch none --vnf QemuPciPassthrough pvp_tput

Any of supported guest-loopback-application can be used inside VM with PCI passthrough support.

Note: Qemu with PCI passthrough support can be used only with PVP test deployment.

10.9.5.2. Guest Core and Thread Binding

VSPERF provides options to achieve better performance by guest core binding and guest vCPU thread binding as well. Core binding is to bind all the qemu threads. Thread binding is to bind the house keeping threads to some CPU and vCPU thread to some other CPU, this helps to reduce the noise from qemu house keeping threads.

GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]

NOTE By default the GUEST_THREAD_BINDING will be none, which means same as the GUEST_CORE_BINDING, i.e. the vcpu threads are sharing the physical CPUs with the house keeping threads. Better performance using vCPU thread binding can be achieved by enabling affinity in the custom configuration file.

For example, if an environment requires 28,29 to be core binded and 30,31 for guest thread binding to achieve better performance.

VNF_AFFINITIZATION_ON = True
GUEST_CORE_BINDING = [('28','29')]
GUEST_THREAD_BINDING = [('30', '31')]
10.9.5.3. Qemu CPU features

QEMU default to a compatible subset of performance enhancing cpu features. To pass all available host processor features to the guest.

GUEST_CPU_OPTIONS = ['host,migratable=off']

NOTE To enhance the performance, cpu features tsc deadline timer for guest, the guest PMU, the invariant TSC can be provided in the custom configuration file.

10.9.5.4. Selection of loopback application for tests with VMs

To select the loopback applications which will forward packets inside VMs, the following parameter should be configured:

GUEST_LOOPBACK = ['testpmd']

or use --test-params CLI argument:

$ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
      --test-params "GUEST_LOOPBACK=['testpmd']"

Supported loopback applications are:

'testpmd'       - testpmd from dpdk will be built and used
'l2fwd'         - l2fwd module provided by Huawei will be built and used
'linux_bridge'  - linux bridge will be configured
'buildin'       - nothing will be configured by vsperf; VM image must
                  ensure traffic forwarding between its interfaces

Guest loopback application must be configured, otherwise traffic will not be forwarded by VM and testcases with VM related deployments will fail. Guest loopback application is set to ‘testpmd’ by default.

NOTE: In case that only 1 or more than 2 NICs are configured for VM, then ‘testpmd’ should be used. As it is able to forward traffic between multiple VM NIC pairs.

NOTE: In case of linux_bridge, all guest NICs are connected to the same bridge inside the guest.

10.9.6. Results

The results for the packet forwarding test cases are uploaded to artifacts and also published on Yardstick Grafana dashboard. The links for the same can be found below

http://artifacts.opnfv.org/kvmfornfv.html
http://testresults.opnfv.org/KVMFORNFV-Packet-Forwarding
11. PCM Utility in KVM4NFV
11.1. Collecting Memory Bandwidth Information using PCM utility

This chapter includes how the PCM utility is used in kvm4nfv to collect memory bandwidth information

11.2. About PCM utility

The Intel® Performance Counter Monitor provides sample C++ routines and utilities to estimate the internal resource utilization of the latest Intel® Xeon® and Core™ processors and gain a significant performance boost.In Intel PCM toolset,there is a pcm-memory.x tool which is used for observing the memory traffic intensity

11.3. Version Features
Release Features
Colorado
  • In Colorado release,we don’t have memory bandwidth information collected through the cyclic testcases.
Danube
  • pcm-memory.x will be executed before the execution of every testcase
  • pcm-memory.x provides the memory bandwidth data throughout out the testcases
  • used for all test-types (stress/idle)
  • Generated memory bandwidth logs are published to the KVMFORFNV artifacts
11.3.1. Implementation of pcm-memory.x:

The tool measures the memory bandwidth observed for every channel reporting seperate throughput for reads from memory and writes to the memory. pcm-memory.x tool tends to report values slightly higher than the application’s own measurement.

Command:

sudo ./pcm-memory.x  [Delay]/[external_program]

Parameters

  • pcm-memory can called with either delay or external_program/application as a parameter
  • If delay is given as 5,then the output will be produced with refresh of every 5 seconds.
  • If external_program is script/application,then the output will produced after the execution of the application or the script passed as a parameter.

Sample Output:

The output produced with default refresh of 1 second.
Socket 0 Socket 1
Memory Performance Monitoring Memory Performance Monitoring
Mem Ch 0: Reads (MB/s): 6870.81
Writes(MB/s): 1805.03
Mem Ch 1: Reads (MB/s): 6873.91
Writes(MB/s): 1810.86
Mem Ch 2: Reads (MB/s): 6866.77
Writes(MB/s): 1804.38
Mem Ch 3: Reads (MB/s): 6867.47
Writes(MB/s): 1805.53

NODE0 Mem Read (MB/s) : 27478.96 NODE0 Mem Write (MB/s): 7225.79 NODE0 P. Write (T/s) : 214810 NODE0 Memory (MB/s) : 34704.75

Mem Ch 0: Reads (MB/s): 7406.36
Writes(MB/s): 1951.25
Mem Ch 1: Reads (MB/s): 7411.11
Writes(MB/s): 1957.73
Mem Ch 2: Reads (MB/s): 7403.39
Writes(MB/s): 1951.42
Mem Ch 3: Reads (MB/s): 7403.66
Writes(MB/s): 1950.95

NODE1 Mem Read (MB/s) : 29624.51 NODE1 Mem Write (MB/s): 7811.36 NODE1 P. Write (T/s) : 238294 NODE1 Memory (MB/s) : 37435.87

  • System Read Throughput(MB/s): 57103.47
  • System Write Throughput(MB/s): 15037.15
  • System Memory Throughput(MB/s): 72140.62
11.3.2. pcm-memory.x in KVM4NFV:

pcm-memory is a part of KVM4NFV in D release.pcm-memory.x will be executed with delay of 60 seconds before starting every testcase to monitor the memory traffic intensity which was handled in collect_MBWInfo function .The memory bandwidth information will be collected into the logs through the testcase updating every 60 seconds.

Pre-requisites:

1.Check for the processors supported by PCM .Latest pcm utility version (2.11)support Intel® Xeon® E5 v4 processor family.

2.Disabling NMI Watch Dog

3.Installing MSR registers

Memory Bandwidth logs for KVM4NFV can be found here:

http://artifacts.opnfv.org/kvmfornfv.html

Details of the function implemented:

In install_Pcm function, it handles the installation of pcm utility and the required prerequisites for pcm-memory.x tool to execute.

$ git clone https://github.com/opcm/pcm
$ cd pcm
$ make

In collect_MBWInfo Function,the below command is executed on the node which was collected to the logs with the timestamp and testType.The function will be called at the begining of each testcase and signal will be passed to terminate the pcm-memory process which was executing throughout the cyclic testcase.

$ pcm-memory.x 60 &>/root/MBWInfo/MBWInfo_${testType}_${timeStamp}

where,
${testType} = verify (or) daily
11.4. Future Scope

PCM information will be added to cyclictest of kvm4nfv in yardstick.

12. Low Latency Tunning Suggestion

The correct configuration is critical for improving the NFV performance/latency.Even working on the same codebase, configurations can cause wildly different performance/latency results.

There are many combinations of configurations, from hardware configuration to Operating System configuration and application level configuration. And there is no one simple configuration that works for every case. To tune a specific scenario, it’s important to know the behaviors of different configurations and their impact.

12.1. Platform Configuration

Some hardware features can be configured through firmware interface(like BIOS) but others may not be configurable (e.g. SMI on most platforms).

  • Power management: Most power management related features save power at the expensive of latency. These features include: Intel®Turbo Boost Technology, Enhanced Intel®SpeedStep, Processor C state and P state. Normally they should be disabled but, depending on the real-time application design and latency requirements, there might be some features that can be enabled if the impact on deterministic execution of the workload is small.
  • Hyper-Threading: The logic cores that share resource with other logic cores can introduce latency so the recommendation is to disable this feature for realtime use cases.
  • Legacy USB Support/Port 60/64 Emulation: These features involve some emulation in firmware and can introduce random latency. It is recommended that they are disabled.
  • SMI (System Management Interrupt): SMI runs outside of the kernel code and can potentially cause latency. It is a pity there is no simple way to disable it. Some vendors may provide related switches in BIOS but most machines do not have this capability.
12.2. Operating System Configuration
  • CPU isolation: To achieve deterministic latency, dedicated CPUs should be allocated for realtime application. This can be achieved by isolating cpus from kernel scheduler. Please refer to http://lxr.free-electrons.com/source/Documentation/kernel-parameters.txt#L1608 for more information.
  • Memory allocation: Memory shoud be reserved for realtime applications and usually hugepage should be used to reduce page fauts/TLB misses.
  • IRQ affinity: All the non-realtime IRQs should be affinitized to non realtime CPUs to reduce the impact on realtime CPUs. Some OS distributions contain an irqbalance daemon which balances the IRQs among all the cores dynamically. It should be disabled as well.
  • Device assignment for VM: If a device is used in a VM, then device passthrough is desirable. In this case,the IOMMU should be enabled.
  • Tickless: Frequent clock ticks cause latency. CONFIG_NOHZ_FULL should be enabled in the linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger many fewer clock tick interrupts(currently, 1 tick per second). This can reduce latency because each host timer interrupt triggers a VM exit from guest to host which causes performance/latency impacts.
  • TSC: Mark TSC clock source as reliable. A TSC clock source that seems to be unreliable causes the kernel to continuously enable the clock source watchdog to check if TSC frequency is still correct. On recent Intel platforms with Constant TSC/Invariant TSC/Synchronized TSC, the TSC is reliable so the watchdog is useless but cause latency.
  • Idle: The poll option forces a polling idle loop that can slightly improve the performance of waking up an idle CPU.
  • RCU_NOCB: RCU is a kernel synchronization mechanism. Refer to http://lxr.free-electrons.com/source/Documentation/RCU/whatisRCU.txt for more information. With RCU_NOCB, the impact from RCU to the VNF will be reduced.
  • Disable the RT throttling: RT Throttling is a Linux kernel mechanism that occurs when a process or thread uses 100% of the core, leaving no resources for the Linux scheduler to execute the kernel/housekeeping tasks. RT Throttling increases the latency so should be disabled.
  • NUMA configuration: To achieve the best latency. CPU/Memory and device allocated for realtime application/VM should be in the same NUMA node.
OPNFV Glossary
Euphrates 1.0
Contents

This glossary provides a common definition of phrases and words commonly used in OPNFV.


A

Arno

A river running through Tuscany and the name of the first OPNFV release.

Apex

OPNFV Installation and Deployment tool based on the RDO Project’s Triple-O OpenStack installation tool.

API

Application Programming Interface

AVX2

Advanced Vector Extensions 2 is an instruction set extension for x86.

B

Brahmaputra

A river running through Asia and the name of the Second OPNFV release.

Bios

Basic Input/Output System

Builds

Build in Jenkins is a version of a program.

Bogomips

Bogomips is the number of million times per second a processor can do absolutely nothing.

C

CAT

Cache Automation Technology

CentOS

Community Enterprise Operating System is a Linux distribution

CICD

Continuous Integration and Continuous Deployment

CLI

Command Line Interface

Colorado

A river in Argentina and the name of the Third OPNFV release.

Compute

Compute is an OpenStack service which offers many configuration options which may be deployment specific.

Console

Console is display screen.
CPU
Central Processing Unit

D

Danube

A river in Europe and name of the Fourth OPNFV release.

Data plane

The data plane is the part of a network that carries user traffic.

Debian/deb

Debian is a Unix-like computer operating system that is composed entirely of free software.

Docs

Documentation/documents

DPDK

Data Plane Development Kit

DPI

Deep Packet Inspection

DSCP

Differentiated Services Code Point

E

Euphrates

Longest river of Western Asia and name of the fifth OPNFV release.

F

Flavors

Flavors are templates used to define VM configurations.

Fuel

Provides an intuitive, GUI-driven experience for deployment and management of OpenStack

H

Horizon

Horizon is an OpenStack service which serves as an UI.

Hypervisor

A hypervisor, also called a virtual machine manager, is a program that allows multiple operating systems to share a single hardware host.

I

IGMP

Internet Group Management Protocol

IOMMU

Input-Output Memory Management Unit

IOPS

Input/Output Operations Per Second

IRQ

Interrupt ReQuest is an interrupt request sent from the hardware level to the CPU.

IRQ affinity

IRQ affinity is the set of CPU cores that can service that interrupt.

J

Jenkins

Jenkins is an open source continuous integration tool written in Java.

JIRA

JIRA is a bug tracking software.

Jitter

Time difference in packet inter-arrival time to their destination can be called jitter.

JumpHost

A jump host or jump server or jumpbox is a computer on a network typically used to manage devices in a separate security zone.

K

Kernel

The kernel is a computer program that constitutes the central core of a computer’s operating system.

L

Latency

The amount of time it takes a packet to travel from source to destination is Latency.

libvirt

libvirt is an open source API, daemon and management tool for managing platform virtualization.

M

Migration

Migration is the process of moving from the use of one operating environment to another operating environment.

N

NFV

Network Functions Virtualisation, an industry initiative to leverage virtualisation technologies in carrier networks.

NFVI

Network Function Virtualization Infrastructure

NIC

Network Interface Controller

NUMA

Non-Uniform Memory Access

O

OPNFV

Open Platform for NFV, an open source project developing an NFV reference platform and features.

P

Pharos

Is a lighthouse and is a project deals with developing an OPNFV lab infrastructure that is geographically and technically diverse.

Pipeline

A suite of plugins in Jenkins that lets you orchestrate automation.

Platform

OPNFV provides an open source platform for deploying NFV solutions that leverages investments from a community of developers and solution providers.

Pools

A Pool is a set of resources that are kept ready to use, rather than acquired on use and released afterwards.

Q

Qemu

QEMU is a free and open-source hosted hypervisor that performs hardware virtualization.

R

RDMA

Remote Direct Memory Access (RDMA)

Rest-Api

REST (REpresentational State Transfer) is an architectural style, and an approach to communications that is often used in the development of web services

S

Scaling

Refers to altering the size.

Slave

Works with/for master.where master has unidirectional control over one or more other devices.

SR-IOV

Single root IO- Virtualization.

Spin locks

A spinlock is a lock which causes a thread trying to acquire it to simply wait in a loop while repeatedly checking if the lock is available.

Storage

Refers to computer components which store some data.

T

Tenant

A Tenant is a group of users who share a common access with specific privileges to the software instance.

Tickless

A tickless kernel is an operating system kernel in which timer interrupts do not occur at regular intervals, but are only delivered as required.

TSC

Technical Steering Committee

V

VLAN

A virtual local area network, typically an isolated ethernet network.

VM

Virtual machine, an emulation in software of a computer system.

VNF

Virtual network function, typically a networking application or function running in a virtual environment.

X

XBZRLE

Helps to reduce the network traffic by just sending the updated data

Y

Yardstick

Yardstick is an infrastructure verification. It is an OPNFV testing project.
KVM4NFV Design
1. KVM4NFV design description

This design focuses on the enhancement of following area for KVM Hypervisor

  • Minimal Interrupt latency variation for data plane VNFs:
    • Minimal Timing Variation for Timing correctness of real-time VNFs
    • Minimal packet latency variation for data-plane VNFs
  • Fast live migration

Minimal Interrupt latency variation for data plane VNFs

Processing performance and latency depend on a number of factors, including the CPUs (frequency, power management features, etc.), micro-architectural resources, the cache hierarchy and sizes, memory (and hierarchy, such as NUMA) and speed, inter-connects, I/O and I/O NUMA, devices, etc.

There are two separate types of latencies to minimize:

  1. Minimal Timing Variation for Timing correctness of real-time VNFs – timing correctness for scheduling operations(such as Radio scheduling)
  2. Minimal packet latency variation for data-plane VNFs – packet delay variation, which applies to packet processing.

For a VM, interrupt latency (time between arrival of H/W interrupt and invocation of the interrupt handler in the VM), for example, can be either of the above or both, depending on the type of the device. Interrupt latency with a (virtual) timer can cause timing correctness issues with real-time VNFs even if they only use polling for packet processing.

We assume that the VNFs are implemented properly to minimize interrupt latency variation within the VMs, but we have additional causes of latency variation on KVM:

  • Asynchronous (e.g. external interrupts) and synchronous(e.g. instructions) VM exits and handling in KVM (and kernel routines called), which may have loops and spin locks
  • Interrupt handling in the host Linux and KVM, scheduling and virtual interrupt delivery to VNFs
  • Potential VM exit (e.g. EOI) in the interrupt service routines in VNFs
  • Exit to the user-level (e.g. QEMU)
_images/kvm1.png
1.1. Design Considerations

The latency variation and jitters can be minimized with the below steps (with some in parallel):

  1. Statically and exclusively assign hardware resources (CPUs, memory, caches,) to the VNFs.
  2. Pre-allocate huge pages (e.g. 1 GB/2MB pages) and guest-to-host mapping, e.g. EPT (Extended Page Table) page tables, to minimize or mitigate latency from misses in caches,
  3. Use the host Linux configured for hard real-time and packet latency, Check the set of virtual devices used by the VMs to optimize or eliminate virtualization overhead if applicable
  4. Use advanced hardware virtualization features that can reduce or eliminate VM exits, if present, and
  5. Inspect the code paths in KVM and associated kernel services to eliminate code that can cause latencies (e.g. loops and spin locks).
  6. Measure latencies intensively. We leverage the existing testing methods. OSADL, for example, defines industry tests for timing correctness.
1.2. Goals and Guidelines

The output of this project will provide :

  1. A list of the performance goals, which will be obtained by the OPNFV members (as described above)
  2. A set of comprehensive instructions for the system configurations (hardware features, BIOS setup, kernel parameters, VM configuration, options to QEMU/KVM, etc.)
  3. The above features to the upstream of Linux, the real-time patch set, KVM, QEMU, libvirt, and
  4. Performance and interrupt latency measurement tools
1.3. Test plan

The tests that need to be conducted to make sure that all components from OPNFV meet the requirement are mentioned below:

Timer test:This test utilize the cyclictest (https://rt.wiki.kernel.org/index.php/Cyclictest) to test the guest timer latency (the latency from the time that the guest timer should be triggered to the time the guest timer is really triggered).

_images/TimerTest.png

Device Interrupt Test:A device on the hardware platform trigger interrupt every one ms and the device interrupt will be delivered to the VNF. This test cover the latency from the interrupt happened on the hardware to the time the interrupt handled in the VNF.

_images/DeviceInterruptTest.png

Packet forwarding (DPDK OVS):A packet is sent from TG (Traffic Generator) to a VNF. The VNF, after processing the packet, forwards the packet to another NIC and in the end the packet is received by the traffic generator. The test check the latency from the packet is sent out by the TC to the time the packet is received by the TC.

_images/PacketforwardingDPDK_OVS.png

Packet Forwarding (SR-IOV):This test is similar to Packet Forwarding (DPDK OVS). However, instead of using virtio NIC devices on the guest, a PCI NIC or a PCI VF NIC is assigned to the guest for network acess.

Bare-metal Packet Forwarding:This is used to compare with the above packet forwarding scenario.

_images/Bare-metalPacketForwarding.png

Multisite

Multisite Installation procedure
1. Kingbird installation instruction
1.1. Abstract

This document will give the user instructions on how to deploy available scenarios verified for the Danube release of OPNFV platform.

1.2. Preparing the installation

Kingbird is centralized synchronization service for multi-region OpenStack deployments. Kingbird provides centralized quota management feature. At least two OpenStack regions with shared KeyStone should be installed first.

Kingbird includes kingbird-api and kingbird-engine, kingbird-api and kingbird-engine which talk to each other through message bus, and both services access the database. Kingbird-api receives the RESTful API request for quota management and forward the request to kingbird-engine to do quota synchronization etc task.

Therefore install Kingbird on the controller nodes of one of the OpenStack region, these two services could be deployed in same node or different node. Both kingbird-api and kingbird-engine can run in multiple nodes with multi-workers mode. It’s up to you how many nodes you want to deploy kingbird-api and kingbird-engine and they can work in same node or different nodes.

1.3. HW requirements

No special hardware requirements

1.4. Installation instruction

In colorado release, Kingbird is recommended to be installed in a python virtual environment. So install and activate virtualenv first.

sudo pip install virtualenv
virtualenv venv
source venv/bin/activate

Get the latest code of Kingbird from git repository:

git clone https://github.com/openstack/kingbird.git
cd kingbird/
pip install -e .

or get the stable release from PyPI repository:

pip install kingbird

In case of the database package are not installed, you may need to install:

pip install mysql
pip install pymysql

In the Kingbird root folder, where you can find the source code of Kingbird, generate the configuration sample file for Kingbird:

oslo-config-generator --config-file=./tools/config-generator.conf

prepare the folder used for cache, log and configuration for Kingbird:

sudo rm -rf /var/cache/kingbird
sudo mkdir -p /var/cache/kingbird
sudo chown `whoami` /var/cache/kingbird
sudo rm -rf /var/log/kingbird
sudo mkdir -p /var/log/kingbird
sudo chown `whoami` /var/log/kingbird
sudo rm -rf /etc/kingbird
sudo mkdir -p /etc/kingbird
sudo chown `whoami` /etc/kingbird

Copy the sample configuration to the configuration folder /etc/kingbird:

cp etc/kingbird/kingbird.conf.sample /etc/kingbird/kingbird.conf

Before editing the configuration file, prepare the database info for Kingbird.

mysql -uroot -e "CREATE DATABASE $kb_db CHARACTER SET utf8;"
mysql -uroot -e "GRANT ALL PRIVILEGES ON $kb_db.* TO '$kb_db_user'@'%' IDENTIFIED BY '$kb_db_pwd';"

For example, the following command will create database “kingbird”, and grant the privilege for the db user “kingbird” with password “password”:

mysql -uroot -e "CREATE DATABASE kingbird CHARACTER SET utf8;"
mysql -uroot -e "GRANT ALL PRIVILEGES ON kingbird.* TO 'kingbird'@'%' IDENTIFIED BY 'password';"

Create the service user in OpenStack:

source openrc admin admin
openstack user create --project=service --password=$kb_svc_pwd $kb_svc_user
openstack role add --user=$kb_svc_user --project=service admin

For example, the following command will create service user “kingbird”, and grant the user “kingbird” with password “password” the role of admin in service project:

source openrc admin admin
openstack user create --project=service --password=password kingbird
openstack role add --user=kingbird --project=service admin

Then edit the configuration file for Kingbird:

vim /etc/kingbird/kingbird.conf

By default, the bind_host of kingbird-api is local_host(127.0.0.1), and the port for the service is 8118, you can leave it as the default if no port conflict happened.

Please replace the address of Kingbird service “127.0.0.1” which is mentioned below to the address you get from OpenStack Kingbird endpoint.

To make the Kingbird work normally, you have to edit these configuration items. The [cache] section is used by kingbird engine to access the quota information of Nova, Cinder, Neutron in each region, replace the auth_uri to the keystone service in your environment, especially if the keystone service is not located in the same node, and also for the account to access the Nova, Cinder, Neutron in each region, in the following configuration, user “admin” with password “password” of the tenant “admin” is configured to access other Nova, Cinder, Neutron in each region:

[cache]
auth_uri = http://127.0.0.1:5000/v3
admin_tenant = admin
admin_password = password
admin_username = admin

Configure the database section with the service user “kingbird” and its password, to access database “kingbird”. For detailed database section configuration, please refer to http://docs.openstack.org/developer/oslo.db/opts.html, and change the following configuration accordingly based on your environment.

[database]
connection = mysql+pymysql://$kb_db_user:$kb_db_pwd@127.0.0.1/$kb_db?charset=utf8

For example, if the database is “kingbird”, and the db user “kingbird” with password “password”, then the configuration is as following:

[database]
connection = mysql+pymysql://kingbird:password@127.0.0.1/kingbird?charset=utf8

The [keystone_authtoken] section is used by keystonemiddleware for token validation during the API request to the kingbird-api, please refer to http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html on how to configure the keystone_authtoken section for the keystonemiddleware in detail, and change the following configuration accordingly based on your environment:

please specify the region_name where you want the token will be validated if the KeyStone is deployed in multiple regions

[keystone_authtoken]
signing_dir = /var/cache/kingbird
cafile = /opt/stack/data/ca-bundle.pem
auth_uri = http://127.0.0.1:5000/v3
project_domain_name = Default
project_name = service
user_domain_name = Default
password = $kb_svc_pwd
username = $kb_svc_user
auth_url = http://127.0.0.1:35357/v3
auth_type = password
region_name = RegionOne

For example, if the service user is “kingbird, and the password for the user is “password”, then the configuration will look like this:

[keystone_authtoken]
signing_dir = /var/cache/kingbird
cafile = /opt/stack/data/ca-bundle.pem
auth_uri = http://127.0.0.1:5000/v3
project_domain_name = Default
project_name = service
user_domain_name = Default
password = password
username = kingbird
auth_url = http://127.0.0.1:35357/v3
auth_type = password
region_name = RegionOne

And also configure the message bus connection, you can refer to the message bus configuration in Nova, Cinder, Neutron configuration file.

[DEFAULT]
transport_url = rabbit://stackrabbit:password@127.0.0.1:5672/

After these basic configuration items configured, now the database schema of “kingbird” should be created:

python kingbird/cmd/manage.py --config-file=/etc/kingbird/kingbird.conf db_sync

And create the service and endpoint for Kingbird, please change the endpoint url according to your cloud planning:

openstack service create --name=kingbird synchronization
openstack endpoint create --region=RegionOne kingbird public http://127.0.0.1:8118/v1.0
openstack endpoint create --region=RegionOne kingbird admin http://127.0.0.1:8118/v1.0
openstack endpoint create --region=RegionOne kingbird internal http://127.0.0.1:8118/v1.0

Now it’s ready to run kingbird-api and kingbird-engine:

nohup python kingbird/cmd/api.py --config-file=/etc/kingbird/kingbird.conf &
nohup python kingbird/cmd/engine.py --config-file=/etc/kingbird/kingbird.conf &

Run the following command to check whether kingbird-api and kingbird-engine are running:

ps aux|grep python
1.5. Post-installation activities

Run the following commands to check whether kingbird-api is serving, please replace $mytoken to the token you get from “openstack token issue”:

openstack token issue
curl  -H "Content-Type: application/json"  -H "X-Auth-Token: $mytoken" \
http://127.0.0.1:8118/

If the response looks like following: {“versions”: [{“status”: “CURRENT”, “updated”: “2016-03-07”, “id”: “v1.0”, “links”: [{“href”: “http://127.0.0.1:8118/v1.0/”, “rel”: “self”}]}]}, then that means the kingbird-api is working normally.

Run the following commands to check whether kingbird-engine is serving, please replace $mytoken to the token you get from “openstack token issue”, and the $admin_project_id to the admin project id in your environment:

curl  -H "Content-Type: application/json"  -H "X-Auth-Token: $mytoken" \
-X PUT \
http://127.0.0.1:8118/v1.0/$admin_project_id/os-quota-sets/$admin_project_id/sync

If the response looks like following: “triggered quota sync for 0320065092b14f388af54c5bd18ab5da”, then that means the kingbird-engine is working normally.

Multisite Configuration Guide
1. Multisite identity service management
1.1. Goal

A user should, using a single authentication point be able to manage virtual resources spread over multiple OpenStack regions.

1.2. Before you read

This chapter does not intend to cover all configuration of KeyStone and other OpenStack services to work together with KeyStone.

This chapter focuses only on the configuration part should be taken into account in multi-site scenario.

Please read the configuration documentation related to identity management of OpenStack for all configuration items.

http://docs.openstack.org/liberty/config-reference/content/ch_configuring-openstack-identity.html

How to configure the database cluster for synchronization or asynchrounous repliation in multi-site scenario is out of scope of this document. The only remainder is that for the synchronization or replication, only Keystone database is required. If you are using MySQL, you can configure like this:

In the master:

binlog-do-db=keystone

In the slave:

replicate-do-db=keystone
1.3. Deployment options

For each detail description of each deployment option, please refer to the admin-user-guide.

  • Distributed KeyStone service with PKI token

    In KeyStone configuration file, PKI token format should be configured

    provider = pki
    

    or

    provider = pkiz
    

    In the [keystone_authtoken] section of each OpenStack service configuration file in each site, configure the identity_url and auth_uri to the address of KeyStone service

    identity_uri = https://keystone.your.com:35357/
    auth_uri = http://keystone.your.com:5000/v2.0
    

    It’s better to use domain name for the KeyStone service, but not to use IP address directly, especially if you deployed KeyStone service in at least two sites for site level high availability.

  • Distributed KeyStone service with Fernet token

  • Distributed KeyStone service with Fernet token + Async replication ( star-mode).

    In these two deployment options, the token validation is planned to be done in local site.

    In KeyStone configuration file, Fernet token format should be configured

    provider = fernet
    

    In the [keystone_authtoken] section of each OpenStack service configuration file in each site, configure the identity_url and auth_uri to the address of local KeyStone service

    identity_uri = https://local-keystone.your.com:35357/
    auth_uri = http://local-keystone.your.com:5000/v2.0
    

    and especially, configure the region_name to your local region name, for example, if you are configuring services in RegionOne, and there is local KeyStone service in RegionOne, then

    region_name = RegionOne
    
2. Configuration of Multisite.Kingbird

A brief introduction to configure Multisite Kingbird service. Only the configuration items for Kingbird will be described here. Logging, messaging, database, keystonemiddleware etc configuration which are generated from OpenStack OSLO libary, will not be described here, for these configuration items are common to Nova, Cinder, Neutron. So please refer to corresponding description from Nova or Cinder or Neutron.

2.1. Configuration in [DEFAULT]
2.1.1. configuration items for kingbird-api
2.1.1.1. bind_host
  • default value: bind_host = 0.0.0.0
  • description: The host IP to bind for kingbird-api service
2.1.1.2. bind_port
  • default value: bind_port = 8118
  • description: The port to bind for kingbird-api service
2.1.1.3. api_workers
  • default value: api_workers = 2
  • description: Number of kingbird-api workers
2.1.2. configuration items for kingbird-engine
2.1.2.1. host
  • default value: host = localhost
  • description: The host name kingbird-engine service is running on
2.1.2.2. workers
  • default value: workers = 1
  • description: Number of kingbird-engine workers
2.1.2.3. report_interval
  • default value: report_interval = 60
  • description: Seconds between running periodic reporting tasks to keep the engine alive in the DB. If the engine doesn’t report its aliveness to the DB more than two intervals, then the lock accquired by the engine will be removed by other engines.
2.1.3. common configuration items for kingbird-api and kingbird-engine
2.1.3.1. use_default_quota_class
  • default value: use_default_quota_class = true
  • description: Enables or disables use of default quota class with default quota, boolean value
2.2. Configuration in [kingbird_global_limit]

For quota limit, a negative value means unlimited.

2.2.1. configuration items for kingbird-api and kingbird-engine
2.2.1.1. quota_instances
  • default value: quota_instances = 10
  • description: Number of instances allowed per project, integer value.
2.2.1.2. quota_cores
  • default value: quota_cores = 20
  • description: Number of instance cores allowed per project, integer value.
2.2.1.3. quota_ram
  • default value: quota_ram = 512
  • description: Megabytes of instance RAM allowed per project, integer value.
2.2.1.4. quota_metadata_items
  • default value: quota_metadata_items = 128
  • description: Number of metadata items allowed per instance, integer value.
2.2.1.5. quota_key_pairs
  • default value: quota_key_pairs = 10
  • description: Number of key pairs per user, integer value.
2.2.1.6. quota_fixed_ips
  • default value: quota_fixed_ips = -1
  • description: Number of fixed IPs allowed per project, this should be at least the number of instances allowed, integer value.
2.2.1.7. quota_security_groups
  • default value: quota_security_groups = 10
  • description: Number of security groups per project, integer value.
2.2.1.8. quota_floating_ips
  • default value: quota_floating_ips = 10
  • description: Number of floating IPs allowed per project, integer value.
2.2.1.9. quota_network
  • default value: quota_network = 10
  • description: Number of networks allowed per project, integer value.
2.2.1.10. quota_subnet
  • default value: quota_subnet = 10
  • description: Number of subnets allowed per project, integer value.
2.2.1.11. quota_port
  • default value: quota_port = 50
  • description: Number of ports allowed per project, integer value.
2.2.1.12. quota_security_group
  • default value: quota_security_group = 10
  • description: Number of security groups allowed per project, integer value.
2.2.1.13. quota_security_group_rule
  • default value: quota_security_group_rule = 100
  • description: Number of security group rules allowed per project, integer value.
2.2.1.14. quota_router
  • default value: quota_router = 10
  • description: Number of routers allowed per project, integer value.
2.2.1.15. quota_floatingip
  • default value: quota_floatingip = 50
  • description: Number of floating IPs allowed per project, integer value.
2.2.1.16. quota_volumes
  • default value: quota_volumes = 10
  • description: Number of volumes allowed per project, integer value.
2.2.1.17. quota_snapshots
  • default value: quota_snapshots = 10
  • description: Number of snapshots allowed per project, integer value.
2.2.1.18. quota_gigabytes
  • default value: quota_gigabytes = 1000
  • description: Total amount of storage, in gigabytes, allowed for volumes and snapshots per project, integer value.
2.2.1.19. quota_backups
  • default value: quota_backups = 10
  • description: Number of volume backups allowed per project, integer value.
2.2.1.20. quota_backup_gigabytes
  • default value: quota_backup_gigabytes = 1000
  • description: Total amount of storage, in gigabytes, allowed for volume backups per project, integer value.
2.3. Configuration in [cache]

The [cache] section is used by kingbird engine to access the quota information for Nova, Cinder, Neutron in each region in order to reduce the KeyStone load while retrieving the endpoint information each time.

2.3.1. configuration items for kingbird-engine
2.3.1.1. auth_uri
2.3.1.2. admin_username
  • default value:
  • description: Username of admin account, for example, admin.
2.3.1.3. admin_password
  • default value:
  • description: Password for admin account, for example, password.
2.3.1.4. admin_tenant
  • default value:
  • description: Tenant name of admin account, for example, admin.
2.3.1.5. admin_user_domain_name
  • default value: admin_user_domain_name = Default
  • description: User domain name of admin account.
2.3.1.6. admin_project_domain_name
  • default value: admin_project_domain_name = Default
  • description: Project domain name of admin account.
2.4. Configuration in [scheduler]

The [scheduler] section is used by kingbird engine to periodically synchronize and rebalance the quota for each project.

2.4.1. configuration items for kingbird-engine
2.4.1.1. periodic_enable
  • default value: periodic_enable = True
  • description: Boolean value for enable/disable periodic tasks.
2.4.1.2. periodic_interval
  • default value: periodic_interval = 900
  • description: Periodic time interval for automatic quota sync job, unit is seconds.
2.5. Configuration in [batch]

The [batch] section is used by kingbird engine to periodicly synchronize and rebalance the quota for each project.

  • default value: batch_size = 3
  • description: Batch size number of projects will be synced at a time.
2.6. Configuration in [locks]

The [locks] section is used by kingbird engine to periodically synchronize and rebalance the quota for each project.

  • default value: lock_retry_times = 3
  • description: Number of times trying to grab a lock.
  • default value: lock_retry_interval =10
  • description: Number of seconds between lock retries.
Multisite Admin User Guide
1. Multisite identity service management
1.1. Goal

A user should, using a single authentication point be able to manage virtual resources spread over multiple OpenStack regions.

1.2. Token Format

There are 3 types of token format supported by OpenStack KeyStone

  • FERNET
  • UUID
  • PKI/PKIZ

It’s very important to understand these token format before we begin the mutltisite identity service management. Please refer to the OpenStack official site for the identity management. http://docs.openstack.org/admin-guide-cloud/identity_management.html

Please note that PKI/PKIZ token format has been deprecated.

1.3. Key consideration in multisite scenario

A user is provided with a single authentication URL to the Identity (Keystone) service. Using that URL, the user authenticates with Keystone by requesting a token typically using username/password credentials. Keystone server validates the credentials, possibly with an external LDAP/AD server and returns a token to the user. The user sends a request to a service in a selected region including the token. Now the service in the region, say Nova needs to validate the token. The service uses its configured keystone endpoint and service credentials to request token validation from Keystone. After the token is validated by KeyStone, the user is authorized to use the service.

The key considerations for token validation in multisite scenario are:
  • Site level failure: impact on authN and authZ shoulde be as minimal as possible
  • Scalable: as more and more sites added, no bottleneck in token validation
  • Amount of inter region traffic: should be kept as little as possible

Hence, Keystone token validation should preferably be done in the same region as the service itself.

The challenge to distribute KeyStone service into each region is the KeyStone backend. Different token format has different data persisted in the backend.

  • Fernet: Tokens are non persistent cryptographic based tokens and validated online by the Keystone service. Fernet tokens are more lightweight than PKI tokens and have a fixed size. Fernet tokens require Keystone deployed in a distributed manner, again to avoid inter region traffic. The data synchronization cost for the Keystone backend is smaller due to the non- persisted token.
  • UUID: UUID tokens have a fixed size. Tokens are persistently stored and create a lot of database traffic, the persistence of token is for the revoke purpose. UUID tokens are validated online by Keystone, call to service will request keystone for token validation. Keystone can become a bottleneck in a large system. Due to this, UUID token type is not suitable for use in multi region clouds, no matter the Keystone database replicates or not.

Cryptographic tokens bring new (compared to UUID tokens) issues/use-cases like key rotation, certificate revocation. Key management is out of scope for this use case.

1.4. Database deployment as the backend for KeyStone service
Database replication:
  • Master/slave asynchronous: supported by the database server itself (mysql/mariadb etc), works over WAN, it’s more scalable. But only master will provide write functionality, domain/project/role provisioning.
  • Multi master synchronous: Galera(others like percona), not so scalable, for multi-master writing, and need more parameter tunning for WAN latency.It can provide the capability for limited multi-sites multi-write function for distributed KeyStone service.
  • Symmetrical/asymmetrical: data replicated to all regions or a subset, in the latter case it means some regions needs to access Keystone in another region.

Database server sharing: In an OpenStack controller, normally many databases from different services are provided from the same database server instance. For HA reasons, the database server is usually synchronously replicated to a few other nodes (controllers) to form a cluster. Note that _all_ database are replicated in this case, for example when Galera sync repl is used.

Only the Keystone database can be replicated to other sites. Replicating databases for other services will cause those services to get of out sync and malfunction.

Since only the Keystone database is to be replicated sync. or async. to another region/site, it’s better to deploy Keystone database into its own database server with extra networking requirement, cluster or replication configuration. How to support this by installer is out of scope.

The database server can be shared when async master/slave replication is used, if global transaction identifiers GTID is enabled.

1.5. Deployment options

Distributed KeyStone service with Fernet token

Fernet token is a very new format, and just introduced recently,the biggest gain for this token format is :1) lightweight, size is small to be carried in the API request, not like PKI token( as the sites increased, the endpoint-list will grows and the token size is too long to carry in the API request) 2) no token persistence, this also make the DB not changed too much and with light weight data size (just project, Role, domain, endpoint etc). The drawback for the Fernet token is that token has to be validated by KeyStone for each API request.

This makes that the DB of KeyStone can work as a cluster in multisite (for example, using MySQL galera cluster). That means install KeyStone API server in each site, but share the same the backend DB cluster.Because the DB cluster will synchronize data in real time to multisite, all KeyStone server can see the same data.

Because each site with KeyStone installed, and all data kept same, therefore all token validation could be done locally in the same site.

The challenge for this solution is how many sites the DB cluster can support. Question is aksed to MySQL galera developers, their answer is that no number/distance/network latency limitation in the code. But in the practice, they have seen a case to use MySQL cluster in 5 data centers, each data centers with 3 nodes.

This solution will be very good for limited sites which the DB cluster can cover very well.

Distributed KeyStone service with Fernet token + Async replication (star-mode)

One master KeyStone cluster with Fernet token in one or two sites(for site level high availability purpose), other sites will be installed with at least 2 slave nodes where the node is configured with DB async replication from the master cluster members. The async. replication data source is better to be from different member of the master cluster, if there are two sites for the KeyStone cluster, it’ll be better that source members for async. replication are located in different site.

Only the master cluster nodes are allowed to write, other slave nodes waiting for replication from the master cluster member( very little delay).

Pros:
  • Deploy database cluster in the master sites is to provide more master nodes, in order to provide more slaves could be done with async. replication in parallel. Two sites for the master cluster is to provide higher reliability (site level) for writing request, but reduce the maintaince challenge at the same time by limiting the cluster spreading over too many sites.
  • Multi-slaves in other sites is because of the slave has no knowledge of other slaves, so easy to manage multi-slaves in one site than a cluster, and multi-slaves work independently but provide multi-instance redundancy(like a cluster, but independent).
Cons:
  • Need to be aware of the chanllenge of key distribution and rotation for Fernet token.
2. Multisite VNF Geo site disaster recovery
2.1. Goal

A VNF (telecom application) should, be able to restore in another site for catastrophic failures happened.

2.2. Key consideration in multisite scenario

Geo site disaster recovery is to deal with more catastrophic failures (flood, earthquake, propagating software fault), and that loss of calls, or even temporary loss of service, is acceptable. It is also seems more common to accept/expect manual / administrator intervene into drive the process, not least because you don’t want to trigger the transfer by mistake.

In terms of coordination/replication or backup/restore between geographic sites, discussion often (but not always) seems to focus on limited application level data/config replication, as opposed to replication backup/restore between of cloud infrastructure between different sites.

And finally, the lack of a requirement to do fast media transfer (without resignalling) generally removes the need for special networking behavior, with slower DNS-style redirection being acceptable.

Here is more concerns about cloud infrastructure level capability to support VNF geo site disaster recovery

2.3. Option1, Consistency application backup

The disater recovery process will work like this:

  1. DR(Geo site disaster recovery )software get the volumes for each VM in the VNF from Nova
  2. DR software call Nova quiesce API to quarantee quiecing VMs in desired order
  3. DR software takes snapshots of these volumes in Cinder (NOTE: Because storage often provides fast snapshot, so the duration between quiece and unquiece is a short interval)
  4. DR software call Nova unquiece API to unquiece VMs of the VNF in reverse order
  5. DR software create volumes from the snapshots just taken in Cinder
  6. DR software create backup (incremental) for these volumes to remote backup storage ( swift or ceph, or.. ) in Cinder
  7. If this site failed,
  1. DR software restore these backup volumes in remote Cinder in the backup site.
  2. DR software boot VMs from bootable volumes from the remote Cinder in the backup site and attach the regarding data volumes.

Note: Quiesce/Unquiesce spec was approved in Mitaka, but code not get merged in time, https://blueprints.launchpad.net/nova/+spec/expose-quiesce-unquiesce-api The spec was rejected in Newton when it was reproposed: https://review.openstack.org/#/c/295595/. So this option will not work any more.

2.4. Option2, Vitrual Machine Snapshot
  1. DR software create VM snapshot in Nova
  2. Nova quiece the VM internally (NOTE: The upper level application or DR software should take care of avoiding infra level outage induced VNF outage)
  3. Nova create image in Glance
  4. Nova create a snapshot of the VM, including volumes

5) If the VM is volume backed VM, then create volume snapshot in Cinder 5) No image uploaded to glance, but add the snapshot in the meta data of the

image in Glance
  1. DR software to get the snapshot information from the Glance

7) DR software create volumes from these snapshots 9) DR software create backup (incremental) for these volumes to backup storage

( swift or ceph, or.. ) in Cinder
  1. If this site failed,
  1. DR software restore these backup volumes to Cinder in the backup site.
  2. DR software boot vm from bootable volume from Cinder in the backup site and attach the data volumes.

This option only provides single VM level consistency disaster recovery.

This feature is already available in current OPNFV release.

2.5. Option3, Consistency volume replication
  1. DR software creates datastore (Block/Cinder, Object/Swift, App Custom storage) with replication enabled at the relevant scope, for use to selectively backup/replicate desire data to GR backup site
  2. DR software get the reference of storage in the remote site storage
  3. If primary site failed,
  1. DR software managing recovery in backup site gets references to relevant storage and passes to new software instances
  2. Software attaches (or has attached) replicated storage, in the case of volumes promoting to writable.
Pros:
  • Replication will be done in the storage level automatically, no need to create backup regularly, for example, daily.
  • Application selection of limited amount of data to replicate reduces risk of replicating failed state and generates less overhear.
  • Type of replication and model (active/backup, active/active, etc) can be tailored to application needs
Cons:
  • Applications need to be designed with support in mind, including both selection of data to be replicated and consideration of consistency
  • “Standard” support in Openstack for Disaster Recovery currently fairly limited, though active work in this area.

Note: Volume replication v2.1 support project level replication.

3. VNF high availability across VIM
3.1. Goal

A VNF (telecom application) should, be able to realize high availability deloyment across OpenStack instances.

3.2. Key consideration in multisite scenario

Most of telecom applications have already been designed as Active-Standby/Active-Active/N-Way to achieve high availability (99.999%, corresponds to 5.26 minutes of unplanned downtime in a year), typically state replication or heart beat between Active-Active/Active-Active/N-Way (directly or via replicated database services, or via private designed message format) are required.

We have to accept the currently limited availability ( 99.99%) of a given OpenStack instance, and intend to provide the availability of the telecom application by spreading its function across multiple OpenStack instances.To help with this, many people appear willing to provide multiple “independent” OpenStack instances in a single geographic site, with special networking (L2/L3) between clouds in that physical site.

The telecom application often has different networking plane for different purpose:

  1. external network plane: using for communication with other telecom application.
  2. components inter-communication plane: one VNF often consisted of several components, this plane is designed for components inter-communication with each other
  3. backup plane: this plane is used for the heart beat or state replication between the component’s active/standby or active/active or N-way cluster.
  4. management plane: this plane is mainly for the management purpose, like configuration

Generally these planes are separated with each other. And for legacy telecom application, each internal plane will have its fixed or flexble IP addressing plan.

There are some interesting/hard requirements on the networking (L2/L3) between OpenStack instances, at lease the backup plane across different OpenStack instances:

To make the VNF can work with HA mode across different OpenStack instances in one site (but not limited to), need to support at lease the backup plane across different OpenStack instances:

1) L2 networking across OpenStack instance for heartbeat or state replication. Overlay L2 networking or shared L2 provider networks can work as the backup plance for heartbeat or state replication. Overlay L2 network is preferred, the reason is:

  1. Support legacy compatibility: Some telecom app with built-in internal L2 network, for easy to move these app to VNF, it would be better to provide L2 network.
  2. Isolated L2 network will simplify the security management between different network planes.
  3. Easy to support IP/mac floating across OpenStack.
  4. Support IP overlapping: multiple VNFs may have overlaping IP address for cross OpenStack instance networking.

Therefore, over L2 networking across Neutron feature is required in OpenStack.

2) L3 networking across OpenStack instance for heartbeat or state replication. For L3 networking, we can leverage the floating IP provided in current Neutron, or use VPN or BGPVPN(networking-bgpvpn) to setup the connection.

L3 networking to support the VNF HA will consume more resources and need to take more security factors into consideration, this make the networking more complex. And L3 networking is also not possible to provide IP floating across OpenStack instances.

3) The IP address used for VNF to connect with other VNFs should be able to be floating cross OpenStack instance. For example, if the master failed, the IP address should be used in the standby which is running in another OpenStack instance. There are some method like VRRP/GARP etc can help the movement of the external IP, so no new feature will be added to OpenStack.

Several projects are addressing the networking requirements, deployment should consider the factors mentioned above.

4. Multisite.Kingbird user guide
4.1. Quota management for OpenStack multi-region deployments

Kingbird is centralized synchronization service for multi-region OpenStack deployments. In OPNFV Colorado release, Kingbird provides centralized quota management feature. Administrator can set quota per project based in Kingbird and sync the quota limit to multi-region OpenStack periodiclly or on-demand. The tenant can check the total quota limit and usage from Kingbird for all regions. Administrator can also manage the default quota by quota class setting.

Following quota items are supported to be managed in Kingbird:

  • instances: Number of instances allowed per project.
  • cores: Number of instance cores allowed per project.
  • ram: Megabytes of instance RAM allowed per project.
  • metadata_items: Number of metadata items allowed per instance.
  • key_pairs: Number of key pairs per user.
  • fixed_ips: Number of fixed IPs allowed per project, valid if Nova Network is used.
  • security_groups: Number of security groups per project, valid if Nova Network is used.
  • floating_ips: Number of floating IPs allowed per project, valid if Nova Network is used.
  • network: Number of networks allowed per project, valid if Neutron is used.
  • subnet: Number of subnets allowed per project, valid if Neutron is used.
  • port: Number of ports allowed per project, valid if Neutron is used.
  • security_group: Number of security groups allowed per project, valid if Neutron is used.
  • security_group_rule: Number of security group rules allowed per project, valid if Neutron is used.
  • router: Number of routers allowed per project, valid if Neutron is used.
  • floatingip: Number of floating IPs allowed per project, valid if Neutron is used.
  • volumes: Number of volumes allowed per project.
  • snapshots: Number of snapshots allowed per project.
  • gigabytes: Total amount of storage, in gigabytes, allowed for volumes and snapshots per project.
  • backups: Number of volume backups allowed per project.
  • backup_gigabytes: Total amount of storage, in gigabytes, allowed for volume backups per project.

Key pair is the only resource type supported in resource synchronization.

Only restful APIs are provided for Kingbird in Colorado release, so curl or other http client can be used to call Kingbird API.

Before use the following command, get token, project id, and kingbird service endpoint first. Use $kb_token to repesent the token, and $admin_tenant_id as administrator project_id, and $tenant_id as the target project_id for quota management and $kb_ip_addr for the kingbird service endpoint ip address.

Note: To view all tenants (projects), run:

openstack project list

To get token, run:

openstack token issue

To get Kingbird service endpoint, run:

openstack endpoint list
4.2. Quota Management API
  1. Update global limit for a tenant

    Use python-kingbirdclient:

    kingbird quota update b8eea2ceda4c47f1906fda7e7152a322 --port 10 --security_groups 10
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X PUT \
    -d '{"quota_set":{"cores": 10,"ram": 51200, "metadata_items": 100,"key_pairs": 100, "network":20,"security_group": 20,"security_group_rule": 20}}' \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
    
  2. Get global limit for a tenant

    Use python-kingbirdclient:

    kingbird quota show --tenant $tenant_id
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
    
  3. A tenant can also get the global limit by himself

    Use python-kingbirdclient:

    kingbird quota show
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    http://$kb_ip_addr:8118/v1.0/$tenant_id/os-quota-sets/$tenant_id
    
  4. Get defaults limits

    Use python-kingbirdclient:

    kingbird quota defaults
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/defaults
    
  5. Get total usage for a tenant

    Use python-kingbirdclient:

    kingbird quota detail --tenant $tenant_id
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X GET \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id/detail
    
  6. A tenant can also get the total usage by himself

    Use python-kingbirdclient:

    kingbird quota detail
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X GET \
    http://$kb_ip_addr:8118/v1.0/$tenant_id/os-quota-sets/$tenant_id/detail
    
  7. On demand quota sync

    Use python-kingbirdclient:

    kingbird quota sync $tenant_id
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X PUT \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id/sync
    
  8. Delete specific global limit for a tenant

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X DELETE \
    -d '{"quota_set": [ "cores", "ram"]}' \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
    
  9. Delete all kingbird global limit for a tenant

    Use python-kingbirdclient:

    kingbird quota delete $tenant_id
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X DELETE \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-sets/$tenant_id
    
4.3. Quota Class API
  1. Update default quota class

    Use python-kingbirdclient:

    kingbird quota-class update --port 10 --security_groups 10 <quota class>
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X PUT \
    -d '{"quota_class_set":{"cores": 100, "network":50,"security_group": 50,"security_group_rule": 50}}' \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-class-sets/default
    
  2. Get default quota class

    Use python-kingbirdclient:

    kingbird quota-class show default
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-class-sets/default
    
  3. Delete default quota class

    Use python-kingbirdclient:

    kingbird quota-class delete default
    

    Use curl:

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X DELETE \
    http://$kb_ip_addr:8118/v1.0/$admin_tenant_id/os-quota-class-sets/default
    
4.4. Resource Synchronization API
  1. Create synchronization job

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X POST -d \
    '{"resource_set":{"resources": ["<Keypair_name>"],"force":<True/False>,"resource_type": "keypair","source": <"Source_Region">,"target": [<"List_of_target_regions">]}}' \
    http://$kb_ip_addr:8118/v1.0/$tenant_id/os-sync
    
  2. Get synchronization job

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    http://$kb_ip_addr:8118/v1.0/$tenant_id/os-sync/
    
  3. Get active synchronization job

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    http://$kb_ip_addr:8118/v1.0/$tenant_id/os-sync/active
    
  4. Get detail information of a synchronization job

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    http://$kb_ip_addr:8118/v1.0/$tenant_id/os-sync/$job_id
    
  5. Delete a synchronization job

    curl \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $kb_token" \
    -X DELETE \
     http://$kb_ip_addr:8118/v1.0/$tenant_id/os-sync/job_id
    
5. Multisite.Tricircle user guide

Tricircle is one OpenStack big-tent project. All user guide related documents could be found from OpenStack website:

Netready

NetReady: Network Readiness
Project:

NetReady, https://wiki.opnfv.org/display/netready/NetReady

Editors:

Georg Kunz (Ericsson)

Authors:

Bin Hu (AT&T), Gergely Csatari (Nokia), Georg Kunz (Ericsson) and others

Abstract:

OPNFV provides an infrastructure with different SDN controller options to realize NFV functionality on the platform it builds. As OPNFV uses OpenStack as a VIM, we need to analyze the capabilities this component offers us. The networking functionality is provided by a component called Neutron, which provides a pluggable architecture and specific APIs for integrating different networking backends, for instance SDN controllers. As NFV wasn’t taken into consideration at the time when Neutron was designed we are already facing several bottlenecks and architectural shortcomings while implementing NFV use cases.

The NetReady project aims at evolving OpenStack networking step-by-step to find the most efficient way to fulfill the requirements of the identified NFV use cases, taking into account the NFV mindset and the capabilities of SDN controllers.

History:
Date Description
22.03.2016 Project creation
19.04.2016 Initial version of the deliverable uploaded to Gerrit
22.07.2016 First version ready for sharing with the community
22.09.2016 Version accompanying the OPNFV Colorado release
1. Introduction

This document represents and describes the results of the OPNFV NetReady (Network Readiness) project. Specifically, the document comprises a selection of NFV-related networking use cases and their networking requirements. For every use case, it furthermore presents a gap analysis of the aforementioned requirements with respect to the current OpenStack networking architecture. Finally it provides a description of potential solutions and improvements.

1.1. Scope

NetReady is a project within the OPNFV initiative. Its focus is on NFV (Network Function Virtualization) related networking use cases and their requirements on the underlying NFVI (Network Function Virtualization Infrastructure).

The NetReady project addresses the OpenStack networking architecture, specifically OpenStack Neutron, from a NFV perspective. Its goal is to identify gaps in the current OpenStack networking architecture with respect to NFV requirements and to propose and evaluate improvements and potential complementary solutions.

1.2. Problem Description

Telco ecosystem’s movement towards the cloud domain results in Network Function Virtualization that is discussed and specified in ETSI NFV. This movement opens up many green field areas which are full of potential growth in both business and technology. This new NFV domain brings new business opportunities and new market segments as well as emerging technologies that are exploratory and experimental in nature, especially in NFV networking.

It is often stated that NFV imposes additional requirements on the networking architecture and feature set of the underlying NFVI beyond those of data center networking. For instance, the NFVI needs to establish and manage connectivity beyond the data center to the WAN (Wide Area Network). Moreover, NFV networking use cases often abstract from L2 connectivity and instead focus on L3-only connectivity. Hence, the NFVI networking architecture needs to be flexible enough to be able to meet the requirements of NFV-related use cases in addition to traditional data center networking.

Traditionally, OpenStack networking, represented typically by the OpenStack Neutron project, targets virtualized data center networking. This comprises originally establishing and managing layer 2 network connectivity among VMs (Virtual Machines). Over the past releases of OpenStack, Neutron has grown to provide an extensive feature set, covering both L2 as well as L3 networking services such as virtual routers, NATing, VPNaaS and BGP VPNs.

It is an ongoing debate how well the current OpenStack networking architecture can meet the additional requirements of NFV networking. Hence, a thorough analysis of NFV networking requirements and their relation to the OpenStack networking architecture is needed.

Besides current additional use cases and requirements of NFV networking, more importantly, because of the green field nature of NFV, it is foreseen that there will be more and more new NFV networking use cases and services, which will bring new business, in near future. The challenges for telco ecosystem are to:

  • Quickly catch the new business opportunity;
  • Execute it in agile way so that we can accelerate the time-to-market and improve the business agility in offering our customers with innovative NFV services.

Therefore, it is critically important for telco ecosystem to quickly develop and deploy new NFV networking APIs on-demand based on market need.

1.3. Goals

The goals of the NetReady project and correspondingly this document are the following:

  • This document comprises a collection of relevant NFV networking use cases and clearly describes their requirements on the NFVI. These requirements are stated independently of a particular implementation, for instance OpenStack Neutron. Instead, requirements are formulated in terms of APIs (Application Programming Interfaces) and data models needed to realize a given NFV use case.
  • The list of use cases is not considered to be all-encompassing but it represents a carefully selected set of use cases that are considered to be relevant at the time of writing. More use cases may be added over time. The authors are very open to suggestions, reviews, clarifications, corrections and feedback in general.
  • This document contains a thorough analysis of the gaps in the current OpenStack networking architecture with respect to the requirements imposed by the selected NFV use cases. To this end, we analyze existing functionality in OpenStack networking.
  • Beyond current list of use cases and gap analysis in the document, more importantly, it is the future of NFV networking that needs to be made easy to innovate, quick to develop, and agile to deploy and operate. A model-driven, extensible framework is expected to achieve agility for innovations in NFV networking.
  • This document will in future revisions describe the proposed improvements and complementary solutions needed to enable OpenStack to fulfill the identified NFV requirements.
2. Use Cases

The following sections address networking use cases that have been identified to be relevant in the scope of NFV and NetReady.

2.1. Multiple Networking Backends
2.1.1. Description

Network Function Virtualization (NFV) brings the need of supporting multiple networking back-ends in virtualized infrastructure environments.

First of all, a Service Providers’ virtualized network infrastructure will consist of multiple SDN Controllers from different vendors for obvious business reasons. Those SDN Controllers may be managed within one cloud or multiple clouds. Jointly, those VIMs (e.g. OpenStack instances) and SDN Controllers need to work together in an interoperable framework to create NFV services in the Service Providers’ virtualized network infrastructure. It is needed that one VIM (e.g. OpenStack instance) shall be able to support multiple SDN Controllers as back-end.

Secondly, a Service Providers’ virtualized network infrastructure will serve multiple, heterogeneous administrative domains, such as mobility domain, access networks, edge domain, core networks, WAN, enterprise domain, etc. The architecture of virtualized network infrastructure needs different types of SDN Controllers that are specialized and targeted for specific features and requirements of those different domains. The architectural design may also include global and local SDN Controllers. Importantly, multiple local SDN Controllers may be managed by one VIM (e.g. OpenStack instance).

Furthermore, even within one administrative domain, NFV services could also be quite diversified. Specialized NFV services require specialized and dedicated SDN Controllers. Thus a Service Provider needs to use multiple APIs and back-ends simultaneously in order to provide users with diversified services at the same time. At the same time, for a particular NFV service, the new networking APIs need to be agnostic of the back-ends.

2.1.2. Requirements

Based on the use cases described above, we derive the following requirements.

It is expected that in NFV networking service domain:

  • One OpenStack instance shall support multiple SDN Controllers simultaneously
  • New networking API shall be integrated flexibly and quickly
  • New NFV Networking APIs shall be agnostic of back-ends
  • Interoperability is needed among multi-vendor SDN Controllers at back-end
2.1.3. Current Implementation

In the current implementation of OpenStack networking, SDN controllers are hooked up to Neutron by means of dedicated plugins. A plugin translates requests coming in through the Neutron northbound API, e.g. the creation of a new network, into the appropriate northbound API calls of the corresponding SDN controller.

There are multiple different plugin mechanisms currently available in Neutron, each targeting a different purpose. In general, there are core plugins, covering basic networking functionality and service plugins, providing layer 3 connectivity and advanced networking services such as FWaaS or LBaaS.

2.1.3.1. Core and ML2 Plugins

The Neutron core plugins cover basic Neutron functionality, such as creating networks and ports. Every core plugin implements the functionality needed to cover the full range of the Neutron core API. A special instance of a core plugin is the ML2 core plugin, which in turn allows for using sub-drivers - separated again into type drivers (VLAN, VxLAN, GRE) or mechanism drivers (OVS, OpenDaylight, etc.). This allows to using dedicated sub-drivers for dedicated functionality.

In practice, different SDN controllers use both plugin mechanisms to integrate with Neutron. For instance OpenDaylight uses a ML2 mechanism plugin driver whereas OpenContrail integrated by means of a full core plugin.

In its current implementation, only one Neutron core plugin can be active at any given time. This means that if a SDN controller utilizes a dedicated core plugin, no other SDN controller can be used at the same time for the same type of service.

In contrast, the ML2 plugin allows for using multiple mechanism drivers simultaneously. In principle, this enables a parallel deployment of multiple SDN controllers if and only if all SDN controllers integrate through a ML2 mechanism driver.

2.1.3.2. Neutron Service Plugins

Neutron service plugins target L3 services and advanced networking services, such as BGPVPN or LBaaS. Typically, a service itself provides a driver plugin mechanism which needs to be implemented for every SDN controller. As the architecture of the driver mechanism is up to the community developing the service plugin, it needs to be analyzed for every driver plugin mechanism individually if and how multiple back-ends are supported.

2.1.4. Gaps in the current solution

Given the use case description and the current implementation of OpenStack Neutron, we identify the following gaps:

  • [MB-GAP1] Limited support for multiple back-ends

    As pointed out above, the Neutron core plugin mechanism only allows for one active plugin at a time. The ML2 plugin allows for running multiple mechanism drivers in parallel, however, successful inter-working strongly depends on the individual driver.

    Moreover, the ML2 plugin and its API is - by design - very layer 2 focused. For NFV networking use cases beyond layer 2, for instance L3VPNs, a more flexible API is required.

2.1.5. Conclusion

We conclude that a complementary method of integrating multiple SDN controllers into a single OpenStack deployment is needed to fulfill the needs of operators.

2.2. L3VPN Use Cases

L3VPNs are virtual layer 3 networks described in multiple standards and RFCs, such as [RFC4364] and [RFC7432]. Connectivity as well as traffic separation is achieved by exchanging routes between VRFs (Virtual Routing and Forwarding).

Moreover, a Service Providers’ virtualized network infrastructure may consist of one or more SDN Controllers from different vendors. Those SDN Controllers may be managed within one cloud or multiple clouds. Jointly, those VIMs (e.g. OpenStack instances) and SDN Controllers work together in an interoperable framework to create L3 services in the Service Providers’ virtualized network infrastructure.

While interoperability between SDN controllers and the corresponding data planes is ensured based on standardized protocols (e.g., [RFC4364] and [RFC7432]), the integration and management of different SDN domains from the VIM is not clearly defined. Hence, this section analyses three L3VPN use cases involving multiple SDN Controllers.

2.2.1. Any-to-Any Base Case
2.2.1.1. Description

This any-to-any use case is the base scenario, providing layer 3 connectivity between VNFs in the same L3VPN while separating the traffic and IP address spaces of different L3VPNs belonging to different tenants.

There are 2 hosts (compute nodes). SDN Controller A and vForwarder A are provided by Vendor A and run on host A. SDN Controller B and vForwarder B are provided by Vendor B, and run on host B.

There are 2 tenants. Tenant 1 creates L3VPN Blue with 2 subnets: 10.1.1.0/24 and 10.3.7.0/24. Tenant 2 creates L3VPN Red with 1 subnet and an overlapping address space: 10.1.1.0/24. The network topology is shown in l3vpn-any2any-figure.

_images/l3vpn-any2any.png

In L3VPN Blue, VMs G1 (10.1.1.5) and G2 (10.3.7.9) are spawned on host A, and attached to 2 subnets (10.1.1.0/24 and 10.3.7.0/24) and assigned IP addresses respectively. VMs G3 (10.1.1.6) and G4 (10.3.7.10) are spawned on host B, and attached to 2 subnets (10.1.1.0/24 and 10.3.7.0/24) and assigned IP addresses respectively.

In L3VPN Red, VM G5 (10.1.1.5) is spawned on host A, and attached to subnet 10.1.1.0/24. VM G6 (10.1.1.6) is spawned on host B, and attached to the same subnet 10.1.1.0/24.

2.2.1.2. Derived Requirements
2.2.1.2.1. Northbound API / Workflow

An example of the desired workflow is as follows:

  1. Create Network
  2. Create Network VRF Policy Resource Any-to-Any

2.1. This policy causes the following configuration when a VM of this tenant is spawned on a host:

2.1.1. There will be a RD assigned per VRF

2.1.2. There will be a RT used for the common any-to-any communication

  1. Create Subnet
  2. Create Port (subnet, network VRF policy resource). This causes the controller to:

4.1. Create a VRF in vForwarder’s FIB, or update VRF if it already exists

4.2. Install an entry for the guest’s host route in FIBs of the vForwarder serving this tenant’s virtual network

4.3. Announce guest host route to WAN-GW via MP-BGP

2.2.1.3. Current implementation

Support for creating and managing L3VPNs is available in OpenStack Neutron by means of the [BGPVPN] project. In order to create the L3VPN network configuration described above using the API [BGPVPN] API, the following workflow is needed:

  1. Create Neutron networks for tenant “Blue”

neutron net-create --tenant-id Blue net1

neutron net-create --tenant-id Blue net2

  1. Create subnets for the Neutron networks for tenant “Blue”

neutron subnet-create --tenant-id Blue --name subnet1 net1 10.1.1.0/24

neutron subnet-create --tenant-id Blue --name subnet2 net2 10.3.7.0/24

  1. Create Neutron ports in the corresponding networks for tenant “Blue”

    neutron port-create --tenant-id Blue --name G1 --fixed-ip subnet_id=subnet1,ip_address=10.1.1.5 net1

    neutron port-create --tenant-id Blue --name G2 --fixed-ip subnet_id=subnet1,ip_address=10.1.1.6 net1

    neutron port-create --tenant-id Blue --name G3 --fixed-ip subnet_id=subnet2,ip_address=10.3.7.9 net2

    neutron port-create --tenant-id Blue --name G4 --fixed-ip subnet_id=subnet2,ip_address=10.3.7.10 net2

  2. Create Neutron network for tenant “Red”

neutron net-create --tenant-id Red net3
  1. Create subnet for the Neutron network of tenant “Red”
neutron subnet-create --tenant-id Red --name subnet3 net3 10.1.1.0/24
  1. Create Neutron ports in the networks of tenant “Red”

neutron port-create --tenant-id Red --name G5 --fixed-ip subnet_id=subnet3,ip_address=10.1.1.5 net3

neutron port-create --tenant-id Red --name G7 --fixed-ip subnet_id=subnet3,ip_address=10.1.1.6 net3

  1. Create a L3VPN by means of the BGPVPN API for tenant “Blue”
neutron bgpvpn-create --tenant-id Blue --route-targets AS:100 --name vpn1
  1. Associate the L3VPN of tenant “Blue” with the previously created networks

neutron bgpvpn-net-assoc-create --tenant-id Blue --network net1 --name vpn1

neutron bgpvpn-net-assoc-create --tenant-id Blue --network net2 --name vpn1

  1. Create a L3VPN by means of the BGPVPN API for tenant “Red”
neutron bgpvpn-create --tenant-id Red --route-targets AS:200 --name vpn2
  1. Associate the L3VPN of tenant “Red” with the previously created networks
neutron bgpvpn-net-assoc-create --tenant-id Red --network net3 --name vpn2

Comments:

  • In this configuration only one BGPVPN for each tenant is created.
  • The ports are associated indirectly to the VPN through their networks.
  • The BGPVPN backend takes care of distributing the /32 routes to the vForwarder instances and assigning appropriate RD values.
2.2.1.4. Gaps in the current solution

In terms of the functionality provided by the BGPVPN project, there are no gaps preventing this particular use case from a L3VPN perspective.

However, in order to support the multi-vendor aspects of this use case, a better support for integrating multiple backends is needed (see previous use case).

2.2.2. L3VPN: ECMP Load Splitting Case (Anycast)
2.2.2.1. Description

In this use case, multiple instances of a VNF are reachable through the same IP. The networking infrastructure is then responsible for spreading the network load across the VNF instances using Equal-Cost Multi-Path (ECMP) or perform a fail-over in case of a VNF failure.

There are 2 hosts (compute nodes). SDN Controller A and vForwarder A are provided by Vendor A, and run on host A. SDN Controller B and vForwarder B are provided by Vendor B, and run on host B.

There is one tenant. Tenant 1 creates L3VPN Blue with subnet 10.1.1.0/24.

The network topology is shown in l3vpn-ecmp-figure:

_images/l3vpn-ecmp.png

In L3VPN Blue, VNF1.1 and VNF1.2 are spawned on host A, attached to subnet 10.1.1.0/24 and assigned the same IP address 10.1.1.5. VNF1.3 is spawned on host B, attached to subnet 10.1.1.0/24 and assigned the same IP addresses 10.1.1.5. VNF 2 and VNF 3 are spawned on host A and B respectively, attached to subnet 10.1.1.0/24, and assigned different IP addresses 10.1.1.6 and 10.1.1.3 respectively.

Here, the Network VRF Policy Resource is ECMP/AnyCast. Traffic to the anycast IP 10.1.1.5 can be load split from either WAN GW or another VM like G5.

2.2.2.2. Current implementation

Support for creating and managing L3VPNs is, in general, available in OpenStack Neutron by means of the BGPVPN project [BGPVPN]. However, the BGPVPN project does not yet fully support ECMP as described in the following.

There are (at least) two different approached to configuring ECMP:

  1. Using Neutron ports with identical IP addresses, or
  2. Using Neutron ports with unique IPs addresses and creating static routes to a common IP prefix with next hops pointing to the unique IP addresses.
2.2.2.2.1. Ports with identical IP addresses

In this approach, multiple Neutron ports using the same IP address are created. In the current Neutron architecture, a port has to reside in a specific Neutron network. However, re-using the same IP address multiple times in a given Neutron network is not possible as this would create an IP collision. As a consequence, creating one Neutron network for each port is required.

Given multiple Neutron networks, the BGPVPN API allows for associating those networks with the same VPN. It is then up to the networking backend to implement ECMP load balancing. This behavior and the corresponding API for configuring the behavior is currently not available. It is nevertheless on the road map of the BGPVPN project.

2.2.2.2.2. Static Routes to ports with unique IP addresses

In this approach, Neutron ports are assigned unique IPs and static routes pointing to the same ECMP load-balanced prefix are created. The static routes define the unique Neutron port IPs as next-hop addresses.

Currently, the API for static routes is not yet available in the BGPVPN project, but it is on the road map. The following work flow shows how to realize this particular use case under the assumption that support for static routes is available in the BGPVPN API.

  1. Create Neutron network for tenant “Blue”
neutron net-create --tenant-id Blue net1
  1. Create subnet for the network of tenant “Blue”
neutron subnet-create --tenant-id Blue --name subnet1 net1 5.1.1.0/24
  1. Create Neutron ports in the network of tenant “Blue”

neutron port-create --tenant-id Blue --name G1 --fixed-ip subnet_id=subnet1,ip_address=5.1.1.1 net1

neutron port-create --tenant-id Blue --name G2 --fixed-ip subnet_id=subnet1,ip_address=5.1.1.2 net1

neutron port-create --tenant-id Blue --name G3 --fixed-ip subnet_id=subnet1,ip_address=5.1.1.3 net1

neutron port-create --tenant-id Blue --name G4 --fixed-ip subnet_id=subnet1,ip_address=5.1.1.4 net1

neutron port-create --tenant-id Blue --name G5 --fixed-ip subnet_id=subnet1,ip_address=5.1.1.5 net1

neutron port-create --tenant-id Blue --name G6 --fixed-ip subnet_id=subnet1,ip_address=5.1.1.6 net1

  1. Create a L3VPN for tenant “Blue”
neutron bgpvpn-create --tenant-id Blue --route-target AS:100 vpn1
  1. Associate the BGPVPN with the network of tenant “Blue”
neutron bgpvpn-network-associate --tenant-id Blue --network-id net1 vpn1
  1. Create static routes which point to the same target

neutron bgpvpn-static-route-add --tenant-id Blue --cidr 10.1.1.5/32 --nexthop-ip 5.1.1.1 vpn1

neutron bgpvpn-static-route-add --tenant-id Blue --cidr 10.1.1.5/32 --nexthop-ip 5.1.1.2 vpn1

neutron bgpvpn-static-route-add --tenant-id Blue --cidr 10.1.1.5/32 --nexthop-ip 5.1.1.3 vpn1

2.2.2.3. Gaps in the current solution

Given the use case description and the currently available implementation in OpenStack provided by BGPVPN project, we identify the following gaps:

  • [L3VPN-ECMP-GAP1] Static routes are not yet supported by the BGPVPN project.

    Currently, no API for configuring static routes is available in the BGPVPN project. This feature is on the road map, however.

  • [L3VPN-ECMP-GAP2] Behavior not defined for multiple Neutron ports of the same IP

    The Neutron and BGPVPN API allow for creating multiple ports with the same IP in different networks and associating the networks with the same VPN. The exact behavior of this configuration is however not defined and an API for configuring the behavior (load-balancing or fail-over) is missing. Development of this feature is on the road map of the project, however.

  • [L3VPN-ECMP-GAP3] It is not possible to assign the same IP to multiple Neutron ports within the same Neutron subnet.

    This is due to the fundamental requirement of avoiding IP collisions within the L2 domain which is a Neutron network.

2.2.2.4. Conclusions

In the context of the ECMP use case, three gaps have been identified. Gap [L3VPN-ECMP-GAP1] and [L3VPN-ECMP-GAP2] are missing or undefined functionality in the BGPVPN project. There is no architectural hindrance preventing the implementation of the missing features in the BGPVPN project as well as in Neutron.

The third gap [L3VPN-ECMP-GAP3] is based on the fact that Neutron ports always have to exist in a Neutron network. As a consequence, in order to create ports with the same IP, multiple networks must be used. This port-network binding will most likely not be relaxed in future releases of Neutron to retain backwards compatibility. A clean alternative to Neutron can instead provide more modeling flexibility.

2.2.3. Hub and Spoke Case
2.2.3.1. Description

In a traditional Hub-and-spoke topology there are two types of network entities: a central hub and multiple spokes. The corresponding VRFs of the hub and the spokes are configured to import and export routes such that all traffic is directed through the hub. As a result, spokes cannot communicate with each other directly, but only indirectly via the central hub. Hence, the hub typically hosts central network functions such firewalls.

Furthermore, there is no layer 2 connectivity between the VNFs.

In addition, in this use case, the deployed network infrastructure comprises equipment from two different vendors, Vendor A and Vendor B. There are 2 hosts (compute nodes). SDN Controller A and vForwarder A are provided by Vendor A, and run on host A. SDN Controller B and vForwarder B are provided by Vendor B, and run on host B.

There is 1 tenant. Tenant 1 creates L3VPN Blue with 2 subnets: 10.1.1.0/24 and 10.3.7.0/24.

The network topology is shown in l3vpn-hub-spoke-figure:

_images/l3vpn-hub-spoke.png

In L3VPN Blue, vFW(H) is acting the role of hub (a virtual firewall). The other 3 VNF VMs are spoke. vFW(H) and VNF1(S) are spawned on host A, and VNF2(S) and VNF3(S) are spawned on host B. vFW(H) (10.1.1.5) and VNF2(S) (10.1.1.6) are attached to subnet 10.1.1.0/24. VNF1(S) (10.3.7.9) and VNF3(S) (10.3.7.10) are attached to subnet 10.3.7.0/24.

2.2.3.2. Derived Requirements
2.2.3.2.1. Northbound API / Workflow

Exemplary workflow is described as follows:

  1. Create Network
  2. Create VRF Policy Resource
2.1. Hub and Spoke
  1. Create Subnet
  2. Create Port

4.1. Subnet

4.2. VRF Policy Resource, [H | S]

2.2.3.2.2. Current implementation

Different APIs have been developed to support creating a L3 network topology and directing network traffic through specific network elements in specific order, for example, [BGPVPN] and [NETWORKING-SFC]. We analyzed those APIs regarding the Hub-and-Spoke use case.

2.2.3.2.2.1. BGPVPN

Support for creating and managing L3VPNs is in general available in OpenStack Neutron by means of the BGPVPN API [BGPVPN]. The [BGPVPN] API currently supports the concepts of network- and router-associations. An association maps Neutron network objects (networks and routers) to a VRF with the following semantics:

  • A network association interconnects all subnets and ports of a Neutron network by binding them to a given VRF
  • a router association interconnects all networks, and hence indirectly all ports, connected to a Neutron router by binding them to a given VRF

It is important to notice that these associations apply to entire Neutron networks including all ports connected to a network. This is due to the fact that in the Neutron, ports can only exist within a network but not individually. Furthermore, Neutron networks were originally designed to represent layer 2 domains. As a result, ports within the same Neutron network typically have layer connectivity among each other. There are efforts to relax this original design assumption, e.g. routed networks, which however do not solve the problem at hand here (see the gap analysis further down below).

In order to realize the hub-and-spoke topology outlined above, VRFs need to be created on a per port basis. Specifically, ports belonging to the same network should not be interconnected except through a corresponding configuration of a per-port-VRF. This configuration includes setting up next-hop routing table, labels, I-RT and E-RT etc. in order to enable traffic direction from hub to spokes.

It may be argued that given the current network- and router-association mechanisms, the following workflow establishes a network topology which aims to achieve the desired traffic flow from Hub to Spokes. The basic idea is to model separate VRFs per VM by creating a dedicated Neutron network with two subnets for each VRF in the Hub-and-Spoke topology.

  1. Create Neutron network “hub”
neutron net-create --tenant-id Blue hub
  1. Create a separate Neutron network for every “spoke”
neutron net-create --tenant-id Blue spoke-i
  1. For every network (hub and spokes), create two subnets

neutron subnet-create <hub/spoke-i UUID> --tenant-id Blue 10.1.1.0/24

neutron subnet-create <hub/spoke-i UUID> --tenant-id Blue 10.3.7.0/24

  1. Create the Neutron ports in the corresponding networks

neutron port-create --tenant-id Blue --name vFW(H) --fixed-ip subnet_id=<hub UUID>,ip_address=10.1.1.5

neutron port-create --tenant-id Blue --name VNF1(S) --fixed-ip subnet_id=<spoke-i UUID>,ip_address=10.3.7.9

neutron port-create --tenant-id Blue --name VNF2(S) --fixed-ip subnet_id=<spoke-i UUID>,ip_address=10.1.1.6

neutron port-create --tenant-id Blue --name VNF3(S) --fixed-ip subnet_id=<spoke-i UUID>,ip_address=10.3.7.10

  1. Create a BGPVPN object (VRF) for the hub network with the corresponding import and export targets
neutron bgpvpn-create --name hub-vrf --import-targets <RT-hub RT-spoke> --export-targets <RT-hub>
  1. Create a BGPVPN object (VRF) for every spoke network with the corresponding import and export targets
neutron bgpvpn-create --name spoke-i-vrf --import-targets <RT-hub> --export-targets <RT-spoke>
  1. Associate the hub network with the hub VRF
bgpvpn-net-assoc-create hub --network <hub network-UUID>
  1. Associate each spoke network with the corresponding spoke VRF
bgpvpn-net-assoc-create spoke-i --network <spoke-i network-UUID>
  1. Add static route to direct all traffic to vFW VNF running at the hub.

    Note: Support for static routes not yet available.

neutron bgpvpn-static-route-add --tenant-id Blue --cidr 0/0 --nexthop-ip 10.1.1.5 hub

After step 9, VMs can be booted with the corresponding ports.

The resulting network topology intents to resemble the target topology as shown in l3vpn-hub-spoke-figure, and achieve the desired traffic direction from Hub to Spoke. However, it deviates significantly from the essence of the Hub-and-Spoke use case as described above in terms of desired network topology, i.e. one L3VPN with multiple VRFs associated with vFW(H) and other VNFs(S) separately. And this method of using the current network- and router-association mechanism is not scalable when there are large number of Spokes, and in case of scale-in and scale-out of Hub and Spokes.

The gap analysis in the next section describes the technical reasons for this.

2.2.3.2.2.2. Network SFC

Support of Service Function Chaining is in general available in OpenStack Neutron through the Neutron API for Service Insertion and Chaining project [NETWORKING-SFC]. However, the [NETWORKING-SFC] API is focused on creating service chaining through NSH at L2, although it intends to be agnostic of backend implementation. It is unclear whether or not the service chain from vFW(H) to VNFs(S) can be created in the way of L3VPN-based VRF policy approach using [NETWORKING-SFC] API.

Hence, it is currently not possible to configure the networking use case as described above.

2.2.3.2.3. Gaps in the Current Solution

Given the use case description and the currently available implementation in OpenStack provided by [BGPVPN] project and [NETWORKING-SFC] project, we identify the following gaps:

  • [L3VPN-HS-GAP1] No means to disable layer 2 semantic of Neutron networks

    Neutron networks were originally designed to represent layer 2 broadcast domains. As such, all ports connected to a network are in principle inter-connected on layer 2 (not considering security rules here). In contrast, in order to realize L3VPN use cases such as the hub-and-spoke topology, connectivity among ports must be controllable on a per port basis on layer 3.

    There are ongoing efforts to relax this design assumption, for instance by means of routed networks ([NEUTRON-ROUTED-NETWORKS]). In a routed network, a Neutron network is a layer 3 domain which is composed of multiple layer 2 segments. A routed network only provides layer 3 connectivity across segments, but layer 2 connectivity across segments is optional. This means, depending on the particular networking backend and segmentation technique used, there might be layer 2 connectivity across segments or not. A new flag l2_adjacency indicates whether or not a user can expect layer 2 connectivity or not across segments.

    This flag, however, is ready-only and cannot be used to overwrite or disable the layer 2 semantics of a Neutron network.

  • [L3VPN-HS-GAP2] No port-association available in the BGPVPN project yet

    Due to gap [L3VPN-HS-GAP1], the [BGPVPN] project was not yet able to implement the concept of a port association. A port association would allow to associate individual ports with VRFs and thereby control layer 3 connectivity on a per port basis.

    The workflow described above intents to mimic port associations by means of separate Neutron networks. Hence, the resulting workflow is overly complicated and not intuitive by requiring to create additional Neutron entities (networks) which are not present in the target topology. Moreover, creating large numbers of Neutron networks limits scalability.

    Port associations are on the road map of the [BGPVPN] project, however, no design that overcomes the problems outlined above has been specified yet. Consequently, the time-line for this feature is unknown.

    As a result, creating a clean Hub-and-Spoke topology is current not yet supported by the [BGPVPN] API.

  • [L3VPN-HS-GAP3] No support for static routes in the BGPVPN project yet

    In order to realize the hub-and-spoke use case, a static route is needed to attract the traffic at the hub to the corresponding VNF (direct traffic to the firewall). Support for static routes in the BGPVPN project is available for the router association by means of the Neutron router extra routes feature. However, there is no support for static routes for network and port associations yet.

    Design work for supporting static routes for network associations has started, but no final design has been proposed yet.

2.2.4. Conclusion

Based on the gap analyses of the three specific L3VPN use cases we conclude that there are gaps in both the functionality provided by the BGPVPN project as well as the support for multiple backends in Neutron.

Some of the identified gaps [L3VPN-ECMP-GAP1, L3VPN-ECMP-GAP2, L3VPN-HS-GAP3] in the BGPVPN project are merely missing functionality which can be integrated in the existing OpenStack networking architecture.

Other gaps, such as the inability to explicitly disable the layer 2 semantics of Neutron networks [L3VPN-HS-GAP1] or the tight integration of ports and networks [L3VPN-HS-GAP2] hinder a clean integration of the needed functionality. In order to close these gaps, fundamental changes in Neutron or alternative approaches need to be investigated.

2.3. Service Binding Design Pattern
2.3.1. Description

This use case aims at binding multiple networks or network services to a single vNIC (port) of a given VM. There are several specific application scenarios for this use case:

  • Shared Service Functions: A service function connects to multiple networks of a tenant by means of a single vNIC.

    Typically, a vNIC is bound to a single network. Hence, in order to directly connect a service function to multiple networks at the same time, multiple vNICs are needed - each vNIC binds the service function to a separate network. For service functions requiring connectivity to a large number of networks, this approach does not scale as the number of vNICs per VM is limited and additional vNICs occupy additional resources on the hypervisor.

    A more scalable approach is to bind multiple networks to a single vNIC and let the service function, which is now shared among multiple networks, handle the separation of traffic itself.

  • Multiple network services: A service function connects to multiple different network types such as a L2 network, a L3(-VPN) network, a SFC domain or services such as DHCP, IPAM, firewall/security, etc.

In order to achieve a flexible binding of multiple services to vNICs, a logical separation between a vNIC (instance port) - that is, the entity that is used by the compute service as hand-off point between the network and the VM - and a service interface - that is, the interface a service binds to - is needed.

Furthermore, binding network services to service interfaces instead of to the vNIC directly enables a more dynamic management of the network connectivity of network functions as there is no need to add or remove vNICs.

2.3.2. Requirements
2.3.2.1. Data model

This section describes a general concept for a data model and a corresponding API. It is not intended that these entities are to be implemented exactly as described. Instead, they are meant to show a design pattern for future network service models and their corresponding APIs. For example, the “service” entity should hold all required attributes for a specific service, for instance a given L3VPN service. Hence, there would be no entity “service” but rather “L3VPN”.

  • instance-port

    An instance port object represents a vNIC which is bindable to an OpenStack instance by the compute service (Nova).

    Attributes: Since an instance-port is a layer 2 device, its attributes include the MAC address, MTU and others.

  • interface

    An interface object is a logical abstraction of an instance-port. It allows to build hierarchies of interfaces by means of a reference to a parent interface. Each interface represents a subset of the packets traversing a given port or parent interface after applying a layer 2 segmentation mechanism specific to the interface type.

    Attributes: The attributes are specific to the type of interface.

    Examples: trunk interface, VLAN interface, VxLAN interface, MPLS interface

  • service

    A service object represents a specific networking service.

    Attributes: The attributes of the service objects are service specific and valid for given service instance.

    Examples: L2, L3VPN, SFC

  • service-port

    A service port object binds an interface to a service.

    Attributes: The attributes of a service-port are specific for the bound service.

    Examples: port services (IPAM, DHCP, security), L2 interfaces, L3VPN interfaces, SFC interfaces.

2.3.2.2. Northbound API

An exemplary API for manipulating the data model is described below. As for the data model, this API is not intended to be a concrete API, but rather an example for a design pattern that clearly separates ports from services and service bindings.

  • instance-port-{create,delete} <name>

    Creates or deletes an instance port object that represents a vNIC in a VM.

  • interface-{create,delete} <name> [interface type specific parameters]

    Creates or deletes an interface object.

  • service-{create,delete} <name> [service specific parameters]

    Create a specific service object, for instance a L3VPN, a SFC domain, or a L2 network.

  • service-port-{create,delete} <service-id> <interface-id> [service specific parameters]

    Creates a service port object, thereby binding an interface to a given service.

2.3.2.3. Orchestration

None.

2.3.2.4. Dependencies on other resources

The compute service needs to be able to consume instance ports instead of classic Neutron ports.

2.3.3. Current Implementation

The core Neutron API does not follow the service binding design pattern. For example, a port has to exist in a Neutron network - specifically it has to be created for a particular Neutron network. It is not possible to create just a port and assign it to a network later on as needed. As a result, a port cannot be moved from one network to another, for instance.

Regarding the shared service function use case outlined above, there is an ongoing activity in Neutron [VLAN-AWARE-VMs]. The solution proposed by this activity allows for creating a trunk-port and multiple sub-ports per Neutron port which can be bound to multiple networks (one network per sub-port). This allows for binding a single VNIC to multiple networks and allow the corresponding VMs to handle the network segmentation (VLAN tagged traffic) itself. While this is a step in the direction of binding multiple services (networks) to a port, it is limited by the fundamental assumption of Neutron that a port has to exist on a given network.

There are extensions of Neutron that follow the service binding design pattern more closely. An example is the BGPVPN project. A rough mapping of the service binding design pattern to the data model of the BGPVPN project is as follows:

  • instance-port -> Neutron port
  • service -> VPN
  • service-port -> network association

This example shows that extensions of Neutron can in fact follow the described design pattern in their respective data model and APIs.

2.3.4. Conclusions

In conclusion, the design decisions taken for the core Neutron API and data model do not follow the service binding model. As a result, it is hard to implement certain use cases which rely on a flexible binding of services to ports. Due to the backwards compatibility to the large amount of existing Neutron code, it is unlikely that the core Neutron API will adapt to this design pattern.

New extension to Neutron however are relatively free to choose their data model and API - within the architectural boundaries of Neutron of course. In order to provide the flexibility needed, extensions shall aim for following the service binding design pattern if possible.

For the same reason, new networking frameworks complementing Neutron, such as Gluon, shall follow this design pattern and create the foundation for implementing networking services accordingly.

2.4. Georedundancy

Georedundancy refers to a configuration which ensures the service continuity of the VNFs even if a whole datacenter fails.

It is possible that the VNF application layer provides additional redundancy with VNF pooling on top of the georedundancy functionality described here.

It is possible that either the VNFCs of a single VNF are spread across several datacenters (this case is covered by the OPNFV multi-site project [MULTISITE] or different, redundant VNFs are started in different datacenters.

When the different VNFs are started in different datacenters the redundancy can be achieved by redundant VNFs in a hot (spare VNF is running its configuration and internal state is synchronized to the active VNF), warm (spare VNF is running, its configuration is synchronized to the active VNF) or cold (spare VNF is not running, active VNFs configuration is stored in a persistent, central store and configured to the spare VNF during its activation) standby state in a different datacenter from where the active VNFs are running. The synchronization and data transfer can be handled by the application or by the infrastructure.

In all of these georedundancy setups there is a need for a network connection between the datacenter running the active VNF and the datacenter running the spare VNF.

In case of a distributed cloud it is possible that the georedundant cloud of an application is not predefined or changed and the change requires configuration in the underlay networks when the network operator uses network isolation. Isolation of the traffic between the datacenters might be needed due to the multi-tenant usage of NFVI/VIM or due to the IP pool management of the network operator.

This set of georedundancy use cases is about enabling the possibility to select a datacenter as backup datacenter and build the connectivity between the NFVIs in the different datacenters in a programmable way.

The focus of these uses cases is on the functionality of OpenStack. It is not considered how the provisioning of physical resources is handled by the SDN controllers to interconnect the two datacenters.

As an example the following picture (georedundancy-before) shows a multi-cell cloud setup where the underlay network is not fully meshed.

_images/georedundancy-before.png

Each datacenter (DC) is a separate OpenStack cell, region or instance. Let’s assume that a new VNF is started in DC b with a Redundant VNF in DC d. In this case a direct underlay network connection is needed between DC b and DC d. The configuration of this connection should be programmable in both DC b and DC d. The result of the deployment is shown in the following figure (georedundancy-after):

_images/georedundancy-after.png
2.4.1. Connection between different OpenStack cells
2.4.1.1. Description

There should be an API to manage the infrastructure networks between two OpenStack cells. (Note: In the Mitaka release of OpenStack cells v1 are considered as experimental, while cells v2 functionality is under implementation). Cells are considered to be problematic from maintainability perspective as the sub-cells are using only the internal message bus and there is no API (and CLI) to do maintenance actions in case of a network connectivity problem between the main cell and the sub cells.

The following figure (cells-architecture) shows the architecture of the most relevant OpenStack components in multi cell OpenStack environment.

_images/cells-architecture.png

The functionality behind the API depends on the underlying network providers (SDN controllers) and the networking setup. (For example OpenDaylight has an API to add new BGP neighbor.)

OpenStack Neutron should provide an abstracted API for this functionality what calls the underlying SDN controllers API.

2.4.1.2. Derived Requirements
  • Possibility to define a remote and a local endpoint
  • As in case of cells the nova-api service is shared. It should be possible to identify the cell in the API calls
2.4.1.2.1. Northbound API / Workflow
  • An infrastructure network management API is needed
  • API call to define the remote and local infrastructure endpoints
  • When the endpoints are created neutron is configured to use the new network.
2.4.1.2.2. Dependencies on compute services
None.
2.4.1.2.3. Data model objects
  • local and remote endpoint objects (Most probably IP addresses with some additional properties).
2.4.1.3. Current implementation
Current OpenStack implementation provides no way to set up the underlay network connection. OpenStack Tricircle project [TRICIRCLE] has plans to build up inter datacenter L2 and L3 networks.
2.4.1.4. Gaps in the current solution
An infrastructure management API is missing from Neutron where the local and remote endpoints of the underlay network could be configured.
2.4.2. Connection between different OpenStack regions or cloud instances
2.4.2.1. Description

There should be an API to manage the infrastructure networks between two OpenStack regions or instances.

The following figure (instances-architecture) shows the architecture of the most relevant OpenStack components in multi instance OpenStack environment.

_images/instances-architecture.png

The functionality behind the API depends on the underlying network providers (SDN controllers) and the networking setup. (For example both OpenDaylight and ONOS have an API to add new BGP neighbors.)

OpenStack Neutron should provide an abstracted API for this functionality what calls the underlying SDN controllers API.

2.4.2.2. Derived Requirements
  • Possibility to define a remote and a local endpoint
  • As in case of cells the nova-api service is shared. It should be possible to identify the cell in the API calls
2.4.2.2.1. Northbound API / Workflow
  • An infrastructure network management API is needed
  • API call to define the remote and local infrastructure endpoints
  • When the endpoints are created Neutron is configured to use the new network.
2.4.2.2.2. Data model objects
  • local and remote endpoint objects (Most probably IP addresses with some additional properties, like local or remote Autonomus Systems (AS)).
2.4.2.3. Current implementation
Current OpenStack implementation provides no way to set up the underlay network connection. OpenStack Tricircle project [TRICIRCLE] has plans to build up inter datacenter L2 and L3 networks.
2.4.2.4. Gaps in the current solution
An infrastructure management API is missing from Neutron where the local and remote endpoints of the underlay network could be configured.
2.4.3. Conclusion
An API is needed what provides possibility to set up the local and remote endpoints for the underlay network. This API present in the SDN solutions, but OpenStack does not provide an abstracted API for this functionality to hide the differences of the SDN solutions.
3. Retired Use Cases

The following use cases have previously been analyzed in OPNFV Netready. Since then, the identified gaps have been addressed and/or closed in the upstream community.

These use cases are not removed from the document for the sake of completeness, but moved to a separate chapter to keep the document structure clean.

3.1. Programmable Provisioning of Provider Networks
3.1.1. Description

In a NFV environment the VNFMs (Virtual Network Function Manager) are consumers of the OpenStack IaaS API. They are often deployed without administrative rights on top of the NFVI platform. Furthermore, in the telco domain provider networks are often used. However, when a provider network is created administrative rights are needed what in the case of a VNFM without administrative rights requires additional manual configuration work. It shall be possible to configure provider networks without administrative rights. It should be possible to assign the capability to create provider networks to any roles.

The following figure (api-users) shows the possible users of an OpenStack API and the relation of OpenStack and ETSI NFV components. Boxes with solid line are the ETSI NFV components while the boxes with broken line are the OpenStack components.

_images/api-users.png
3.1.2. Requirements
  • Authorize the possibility of provider network creation based on policy
  • There should be a new entry in policy.json which controls the provider network creation
  • Default policy of this new entry should be rule:admin_or_owner.
  • This policy should be respected by the Neutron API
3.1.2.1. Northbound API / Workflow
  • No changes in the API
3.1.2.2. Data model objects
  • No changes in the data model
3.1.3. Current implementation

Only admin users can manage provider networks [OS-NETWORKING-GUIDE-ML2].

3.1.4. Potential implementation
  • Policy engine shall be able to handle a new provider network creation and modification related policy.
  • When a provider network is created or modified neutron should check the authority with the policy engine instead of requesting administrative rights.
3.1.5. Solution in upstream community

A bug report has been submitted to the upstream OpenStack community to highlight this gap: https://bugs.launchpad.net/neutron/+bug/1630880

This bug report revealed that this use case has already been addressed in the upstream community. Specifically, it is possible to specify the roles (e.g., admin, regular user) in the Neutron policy.json file which are able to create and update provider networks.

However, the OpenStack user guide wrongly stated that only administrators can create and update provider type networks. Hence, a correction has been submitted to the OpenStack documentation repository, clarifying the possibility to change this behavior based on policies: https://review.openstack.org/#/c/390359/

In conclusion, this use case has been retired as the corresponding gaps have been closed in the upstream community.

4. Summary and Conclusion

This document presented the results of the OPNFV NetReady (Network Readiness) project ([NETREADY]). It described a selection of NFV-related networking use cases and their corresponding networking requirements. Moreover, for every use case, it describes an associated gap analysis which analyses the aforementioned networking requirements with respect to the current OpenStack networking architecture.

The contents of the current document are the selected use cases and their derived requirements and identified gaps for OPNFV C release.

OPNFV NetReady is open to take any further use cases under analysis in later OPNFV releases. The project backlog ([NETREADY-JIRA]) lists the use cases and topics planned to be developed in future releases of OPNFV.

Based on the gap analyses, we draw the following conclusions:

  • Besides current requirements and gaps identified in support of NFV networking, more and more new NFV networking services are to be innovated in the near future. Those innovations will bring additional requirements, and more significant gaps will be expected. On the other hand, NFV networking business requires it to be made easy to innovate, quick to develop, and agile to deploy and operate. Therefore, a model-driven, extensible framework is expected to support NFV networking on-demand in order to accelerate time-to-market and achieve business agility for innovations in NFV networking business.
  • Neutron networks are implicitly, because of their reliance on subnets, L2 domains. L2 network overlays are the only way to implement Neutron networks because of their semantics. However, L2 networks are inefficient ways to implement cloud networking, and while this is not necessarily a problem for enterprise use cases with moderate traffic it can add expense to the infrastructure of NFV cases where networking is heavily used and efficient use of capacity is key.
  • In NFV environment it should be possible to execute network administrator tasks without OpenStack administrator rights.
  • In a multi-site setup it should be possible to manage the connection between the sites in a programmable way.

The latest version of this document can be found at [SELF].

5. Definition of terms

Different standards developing organizations and communities use different terminology related to Network Function Virtualization, Cloud Computing, and Software Defined Networking. This list defines the terminology in the contexts of this document.

API
Application Programming Interface.
Cloud Computing
A model that enables access to a shared pool of configurable computing resources, such as networks, servers, storage, applications, and services, that can be rapidly provisioned and released with minimal management effort or service provider interaction.
Edge Computing
Edge computing pushes applications, data and computing power (services) away from centralized points to the logical extremes of a network.
Instance
Refers in OpenStack terminology to a running VM, or a VM in a known state such as suspended, that can be used like a hardware server.
NFV
Network Function Virtualization.
NFVI
Network Function Virtualization Infrastructure. Totality of all hardware and software components which build up the environment in which VNFs are deployed.
SDN
Software-Defined Networking. Emerging architecture that decouples the network control and forwarding functions, enabling the network control to become directly programmable and the underlying infrastructure to be abstracted for applications and network services.
Server
Computer that provides explicit services to the client software running on that system, often managing a variety of computer operations. In OpenStack terminology, a server is a VM instance.
vForwarder
vForwarder is used as generic and vendor neutral term for a software packet forwarder. Concrete examples includes OpenContrail vRouter, OpenvSwitch, Cisco VTF.
VIM
Virtualized Infrastructure Manager. Functional block that is responsible for controlling and managing the NFVI compute, storage and network resources, usually within one operator’s Infrastructure Domain, e.g. NFVI Point of Presence (NFVI-PoP).
Virtual network
Virtual network routes information among the network interfaces of VM instances and physical network interfaces, providing the necessary connectivity.
VM
Virtual Machine. Virtualized computation environment that behaves like a physical computer/server by modeling the computing architecture of a real or hypothetical computer.
VNF
Virtualized Network Function. Implementation of an Network Function that can be deployed on a Network Function Virtualization Infrastructure (NFVI).
VNFC
Virtualized Network Function Component. A VNF may be composed of multiple components, jointly providing the functionality of the VNF.
WAN
Wide Area Network.
OPNFV NetReady Installation
OPNFV Gluon Installation Guide

The Gluon framework can be installed by means of the os-odl-gluon-noha scenario and the Apex installer. Please visit the Apex installer documentation for details on how to install the os-odl-gluon-noha scenario in a virtual or a bare-metal environment.

Quick start guide

The easiest way to set up Gluon is to create a virtual deployment. In a nutshell, these are the installation steps:

  1. install a bare-metal CentOS jumphost
  2. install the Apex RPM packages
  3. create the virtual deployment by running the following command
opnfv-deploy -v -n network_settings.yaml  \
             -d os-odl-gluon-noha.yaml \
             --virtual-computes 3
OPNFV NetReady Configuration Guide
Gluon Configuration

The configuration of the Gluon framework is entirely handled by the corresponding scenario os-odl-gluon-noha available for the Apex installer. In general, the installer installs and configures all components so that no additional configuration steps are needed after installing the aforementioned scenario.

Pre-configuration activities

No pre-configuration steps are needed in addition to the pre-configuration needed for an Apex virtual or bare metal deployment. Please review the Apex installation instructions for further details.

Hardware configuration

No specific hardware configuration is needed for running the os-odl-gluon-noha scenario providing the Gluon framework in addition to the hardware requirements listed for Apex-based scenarios.

Feature configuration

No specific additional configuration is needed after installing the os-odl-gluon-noha scenario using the Apex installer.

Gluon Post Installation Procedure

The configuration of the Gluon framework is entirely handled by the corresponding scenario os-odl-gluon-noha available for the Apex installer. In general, Apex installs and configures all components so that no additional configuration steps are needed after deploying the aforementioned scenario.

Automated post installation activities

An overview of all test suites run by the OPNFV pipeline against the os-odl-gluon-noha scenario as well as the test results can be found at the Functest test result overview page.

Gluon post configuration procedures

No post configuration procedures need to be performed after deploying the os-odl-gluon-noha scenario using the Apex installer.

Platform components validation

As described in the Gluon scenario description, the Gluon framework consists of five software components. This section describes how to validate their successful installation.

  • Gluon core plugin: Check in the file /etc/neutron/neutron.conf that the Neutron core plugin is set to gluon.
  • Proton server: Check that the process proton-server is running.
  • Proton client: Verify that the protonclient tool is installed and executable.
  • etcd: Verify that the etcd key-value-store is installed and running by means of the etcdctl tool.
  • Proton shim layer for OpenDaylight: Verify that the proton-shim-server process is running.
NetReady User Guide
OPNFV NetReady User Guide
Gluon Description

Gluon brings a Networking Service Framework that enables Telecom Service Providers to provide their customers with networking services on-demand. Gluon uses a model-driven approach to generate Networking Service APIs (including objects, database schema, and RESTful API endpoints) from a YAML file which models the Networking Service. When a Telecom Service Provider needs to launch a new Networking Service, it only needs to model the new service in a YAML file. The Gluon framework generates the APIs accordingly. Thus Gluon helps Telecom Service Providers accelerate the time-to-market and achieve business agility through its extensibility and scalability in generating APIs for new use-cases and services.

Gluon Capabilities and Usage

Gluon is the port arbiter that maintains a list of ports and bindings of different networking backends. A Proton is a set of APIs of a particular NFV Networking Service. A Proton Server is the API server that hosts multiple Protons, i.e. multiple sets of APIs. Gluon uses backend drivers to interact with the Proton Server for port binding and other operations.

A Proton is created by a Particle Generator based on a YAML file modeled for this particular NFV Networking Service. When a Proton is created, the objects, database schema, and RESTful APIs of this Proton are created. Then the Proton specific driver would be loaded into Gluon.

When the Proton Server receives port binding and other operation requests, it broadcasts those requests to etcd. The Shim Layers of respective SDN Controllers listen to etcd, and get the notification from etcd. Based on the type of operations, parameter data, and its own deployment and policy configuration, SDN Controllers act upon accordingly. This mechanism is similar to Neutron’s Hierarchical Port Binding (HPB), and provides the flexibility and scalability when a port operation needs to be supported by multiple SDN Controllers in collaborative and interoperable way.

Gluon API Guidelines and Examples

This section shows you how to use Proton to create the needed objects, and then use nova boot to bind the port to a VM. It is assumed that you have already installed Gluon package, including etcd and Gluon Plugin, and started Proton Server. If not, please refer to the Installation guide.

Getting Help

Just typing the protonclient --help command gives you general help information:

$ protonclient --help

Usage: protonclient --api <api_name> [OPTIONS] COMMAND[ARGS]...

Options:
--api TEXT      Name of API, one of ['net-l3vpn', 'test']
--port INTEGER  Port of endpoint (OS_PROTON_PORT)
--host TEXT     Host of endpoint (OS_PROTON_HOST)
--help          Show this message and exit.
Mandatory Parameters

--api <api_name> is a mandatory parameter. For example, --api net-l3vpn.

Just typing the protonclient command shows you that those mandatory parameters are required, and gives you general help information too.

$ protonclient
--api is not specified!

Usage: protonclient --api <api_name> [OPTIONS] COMMAND[ARGS]...

Options:
--api TEXT      Name of API, one of ['net-l3vpn', 'test']
--port INTEGER  Port of endpoint (OS_PROTON_PORT)
--host TEXT     Host of endpoint (OS_PROTON_HOST)
--help          Show this message and exit.
Using L3VPN Proton

NOTE that there is a KNOWN BUG in the Usage message where the mandatory parameters --api net-l3vpn are missing.

$ protonclient --api net-l3vpn
Usage: protonclient [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  interface-create
  interface-delete
  interface-list
  interface-show
  interface-update
  port-create
  port-delete
  port-list
  port-show
  port-update
  vpn-create
  vpn-delete
  vpn-list
  vpn-show
  vpn-update
  vpnafconfig-create
  vpnafconfig-delete
  vpnafconfig-list
  vpnafconfig-show
  vpnafconfig-update
  vpnbinding-create
  vpnbinding-delete
  vpnbinding-list
  vpnbinding-show
  vpnbinding-update

The following sections give you the general work flow of how to use Proton to create and configure an L3VPN.

For more details and examples, please refer to the Gluon upstream user guide.

Work Flow of Using L3VPN

The work flow of using L3VPN includes:

  • Step 1: Create ``Port`` Object
$ protonclient --api net-l3vpn port-create --help
$ protonclient --api net-l3vpn port-create [ARGS] ...

Please NOTE: a default interface object is automatically created too when a Port is created, and this default interface object is attached to this Port object. The UUID of this default Interface object will be the same as the UUID of the parent Port object.

  • Step 2 (Optional): Create ``Interface`` Object
$ protonclient --api net-l3vpn interface-create --help
$ protonclient --api net-l3vpn interface-create [ARGS] ...

Please NOTE: This step is optional because a default Interface object was already automatically created when a Port object was created at Step 1.

  • For example: list the default ``Interface`` Object:
$ protonclient --api net-l3vpn interface-list
  • Step 3 (Optional): Create ``VPNAFConfig`` Object
$ protonclient --api net-l3vpn vpnafconfig-create --help
$ protonclient --api net-l3vpn vpnafconfig-create [ARGS] ...

Please NOTE: This step is optional because all parameters needed for an L3VPN (route specifiers) are also present in creating a VPN service object at Step 4. This part of the API needs to be aligned in the future.

  • Step 4: Create ``VPN`` Object
$ protonclient --api net-l3vpn vpn-create --help
$ protonclient --api net-l3vpn vpn-create [ARGS] ...

At this point you have a Port object, default Interface object and a VPN service object created.

  • View VPN and Port Objects

You can view the values with the following commands:

$ protonclient --api net-l3vpn vpn-list
$ protonclient --api net-l3vpn port-list
  • Step 5: Create ``VPNBinding`` Object

You need to create a VPNBinding object to tie the Interface and the Service together in order to achieve service binding.

  $ protonclient --api net-l3vpn vpnbinding-create --help
  $ protonclient --api net-l3vpn vpnbinding-create [ARGS] ...

* View ``VPNBinding`` Objects
$ protonclient --api net-l3vpn vpnbinding-list

At this point you have had all of the information needed for an L3VPN Port in Proton.

  • Step 6: Create VM and Bind our L3VPN Port
$ nova --debug boot --flavor 1 --image cirros --nic port-id=<port-id> <VM-Name>

Opera

OPNFV Opera Overview
1. OPERA Project Overview

Since OPNFV board expanded its scope to include NFV MANO last year, several upstream open source projects have been created to develop MANO solutions. Each solution has demonstrated its unique value in specific area. Open-Orchestrator (OPEN-O) project is one of such communities. Opera seeks to develop requirements for OPEN-O MANO support in the OPNFV reference platform, with the plan to eventually integrate OPEN-O in OPNFV as a non-exclusive upstream MANO. The project will definitely benefit not only OPNFV and Open-O, but can be referenced by other MANO integration as well. In particular, this project is basically use case driven. Based on that, it will focus on the requirement of interfaces/data models for integration among various components and OPNFV platform. The requirement is designed to support integration among Open-O as NFVO with Juju as VNFM and OpenStack as VIM.

Currently OPNFV has already included upstream OpenStack as VIM, and Juju and Tacker have been being considered as gVNFM by different OPNFV projects. OPEN-O as NFVO part of MANO will interact with OpenStack and Juju. The key items required for the integration can be described as follows.

key item

Fig 1. Key Item for Integration

2. Open-O is scoped for the integration

OPEN-O includes various components for OPNFV MANO integration. The initial release of integration will be focusing on NFV-O, Common service and Common TOSCA. Other components of Open-O will be gradually integrated to OPNFV reference platform in later release.

openo component

Fig 2. Deploy Overview

3. The vIMS is used as initial use case

based on which test cases will be created and aligned with Open-O first release for OPNFV D release.

  • Creatting scenario (os-nosdn-openoe-ha) to integrate Open-O with OpenStack Newton.
  • Integrating with COMPASS as installer, FuncTest as testing framework
  • Clearwater vIMS is used as VNFs, Juju is used as VNFM.
  • Use Open-O as Orchestrator to deploy vIMS to do end-2-end test with the following steps.
  1. deploy Open-O as orchestrator
  2. create tenant by Open-O to OpenStack
  3. deploy vIMS VNFs from orchestrator based on TOSCA blueprintn and create VNFs
  4. launch test suite
  5. collect results and clean up
vIMS deploy

Fig 3. vIMS Deploy

OPNFV Opera Installation Instructions
1. Abstract

This document describes how to install Open-O in an OpenStack deployed environment using Opera project.

2. Version history
Date Ver. Author Comment
2017-02-16 0.0.1 Harry Huang (HUAWEI) First draft
3. Opera Installation Instructions

This document providing guidelines on how to deploy a working Open-O environment using opera project.

The audience of this document is assumed to have good knowledge in OpenStack and Linux.

3.1. Preconditions

There are some preconditions before starting the Opera deployment

3.1.1. A functional OpenStack environment

OpenStack should be deployed before opera deploy.

3.1.2. Getting the deployment scripts

Retrieve the repository of Opera using the following command:

3.2. Machine requirements
  1. Ubuntu OS (Pre-installed).
  2. Root access.
  3. Minimum 1 NIC (internet access)
  4. CPU cores: 32
  5. 64 GB free memory
  6. 100G free disk
3.3. Deploy Instruction

After opera deployment, Open-O dockers will be launched on local server as orchestrator and juju vm will be launched on OpenStack as VNFM.

3.3.1. Add OpenStack Admin Openrc file

Add the admin openrc file of your local openstack into opera/conf directory with the name of admin-openrc.sh.

3.3.2. Config open-o.yml

Set openo_version to specify Open-O version.

Set openo_ip to specify an external ip to access Open-O services. (leave the value unset will use local server’s external ip)

Set ports in openo_docker_net to specify Open-O’s exposed service ports.

Set enable_sdno to specify if use Open-O ‘s sdno services. (set this value false will not launch Open-O sdno dockers and reduce deploy duration)

Set vnf_type to specify the vnf type need to be deployed. (currently only support clearwater deployment, leave this unset will not deploy any vnf)

3.3.3. Run opera_launch.sh
./opera_launch.sh
OPNFV Opera Config Instructions
1. Config Guide
1.1. Add OpenStack Admin Openrc file

Add the admin openrc file of your local openstack into opera/conf directory with the name of admin-openrc.sh.

1.2. Config open-o.yml

Set openo_version to specify Open-O version.

Set openo_ip to specify an external ip to access Open-O services. (leave the value unset will use local server’s external ip)

Set ports in openo_docker_net to specify Open-O’s exposed service ports.

Set enable_sdno to specify if use Open-O ‘s sdno services. (set this value false will not launch Open-O sdno dockers and reduce deploy duration)

Set vnf_type to specify the vnf type need to be deployed. (currently only support clearwater deployment, leave this unset will not deploy any vnf)

OPNFV Opera Design
1. OPERA Requirement and Design
  • Define Scenario OS-NOSDN-OPENO-HA and Integrate OPEN-O M Release with OPNFV D Release (with OpenStack Newton)

  • Integrate OPEN-O to OPNFV CI Process
    • Integrate automatic Open-O and Juju installation
  • Deploy Clearwater vIMS through OPEN-O
    • Test case to simulate SIP clients voice call
  • Integrate vIMS test scripts to FuncTest

2. OS-NOSDN-OPENO-HA Scenario Definition
2.1. Compass4NFV supports Open-O NFV Scenario
  • Scenario name: os-nosdn-openo-ha

  • Deployment: OpenStack + Open-O + JuJu

  • Setups:
    • Virtual deployment (one physical server as Jump Server with OS ubuntu)
    • Physical Deployment (one physical server as Jump Server, ubuntu + 5 physical Host Server)
deploy overview

Fig 1. Deploy Overview

3. Open-O is participating OPNFV CI Process
  • All steps are linked to OPNFV CI Process
  • Jenkins jobs remotely access OPEN-O NEXUS repository to fetch binaries
  • COMPASS is to deploy scenario based on OpenStack Newton release.
  • OPEN-O and JuJu installation scripts will be triggered in Jenkins job after COMPASS finish deploying OpenStack
  • Clearwater vIMS deploy scripts will be integrated into FuncTest
  • Clearwater vIMS test scripts will be integrated into FuncTest
opera ci

Fig 2. Opera Ci

4. The vIMS is used as initial use case

based on which test cases will be created and aligned with Open-O first release for OPNFV D release.

  • Creatting scenario (os-nosdn-openoe-ha) to integrate Open-O with OpenStack Newton.
  • Integrating with COMPASS as installer, FuncTest as testing framework
  • Clearwater vIMS is used as VNFs, Juju is used as VNFM.
  • Use Open-O as Orchestrator to deploy vIMS to do end-2-end test with the following steps.
  1. deploy Open-O as orchestrator
  2. create tenant by Open-O to OpenStack
  3. deploy vIMS VNFs from orchestrator based on TOSCA blueprintn and create VNFs
  4. launch test suite
  5. collect results and clean up
vIMS deploy

Fig 3. vIMS Deploy

5. Requirement and Tasks
5.1. OPERA Deployment Key idea
  • Keep OPEN-O deployment agnostic from an installer perspective (Apex, Compass, Fuel, Joid)
  • Breakdown deployments in single scripts (isolation)
  • Have OPNFV CI Process (Jenkins) control and monitor the execution
5.2. Tasks need to be done for OPNFV CD process
  1. Compass to deploy scenario of os-nosdn-openo-noha

  2. Automate OPEN-O installation (deployment) process

  3. Automate JuJu installation process

  4. Create vIMS TOSCA blueprint (for vIMS deployment)

  5. Automate vIMS package deployment (need helper/OPEN-O M)
    • (a)Jenkins to invoke OPEN-O Restful API to import & deploy vIMS ackage
  6. Integrate scripts of step 2,3,4,5 with OPNFV CD Jenkins Job

5.3. FUNCTEST
  1. test case automation
    • (a)Invoke URL request to vIMS services to test deployment is successfully done.
  2. Integrate test scripts with FuncTest
    • (a)trigger these test scripts
    • (b)record test result to DB
functest

Fig 4. Functest

Parser

OPNFV Parser Installation Instruction
Parser tosca2heat Installation

Please follow the below installation steps to install tosca2heat submodule in parser.

Step 1: Clone the parser project.

git clone https://gerrit.opnfv.org/gerrit/parser

Step 2: Install the heat-translator sub project.

# uninstall pre-installed tosca-parser
pip uninstall -y heat-translator

# change directory to heat-translator
cd parser/tosca2heat/heat-translator

# install requirements
pip install -r requirements.txt

# install heat-translator
python setup.py install

Step 3: Install the tosca-parser sub project.

# uninstall pre-installed tosca-parser
pip uninstall -y tosca-parser

# change directory to tosca-parser
cd parser/tosca2heat/tosca-parser

# install requirements
pip install -r requirements.txt

# install tosca-parser
python setup.py install

Notes: It must uninstall pre-installed tosca-parser and heat-translator before install the two components, and install heat-translator before installing tosca-parser, which is sure to use the OPNFV version of tosca-parser and heat-translator other than openstack’s components.

Parser yang2tosca Installation

Parser yang2tosca requires the following to be installed.

Step 1: Clone the parser project.

git clone https://gerrit.opnfv.org/gerrit/parser

Step 2: Clone pyang tool or download the zip file from the following link.

git clone https://github.com/mbj4668/pyang.git

OR

wget https://github.com/mbj4668/pyang/archive/master.zip

Step 3: Change directory to the downloaded directory and run the setup file.

cd pyang
python setup.py
Step 4: install python-lxml

Please follow the below installation link. http://lxml.de/installation.html

Parser policy2tosca installation

Please follow the below installation steps to install parser - POLICY2TOSCA.

Step 1: Clone the parser project.

git clone https://gerrit.opnfv.org/gerrit/parser

Step 2: Install the policy2tosca module.

cd parser/policy2tosca
python setup.py install
Parser verigraph installation

In the present release, verigraph requires that the following software is also installed:

Please follow the below installation steps to install verigraph.

Step 1: Clone the parser project.

git clone https://gerrit.opnfv.org/gerrit/parser

Step 2: Go to the verigraph directory.

cd parser/verigraph

Step3: Set up the execution environment, based on your operating system.

VeriGraph deployment on Apache Tomcat (Windows):

  • set JAVA HOME environment variable to where you installed the jdk (e.g. C:\Program Files\Java\jdk1.8.XYY);
  • set CATALINA HOME ambient variable to the directory where you installed Apache (e.g. C:\Program Files\Java\apache-tomcat-8.0.30);
  • open the file %CATALINA_HOME%\conf\tomcat-users.xml and under the tomcat-users tag place, initialize an user with roles “tomcat, manager-gui, manager-script”. An example is the following content: xml   <role rolename="tomcat"/>   <role rolename="role1"/>   <user username="tomcat" password="tomcat" roles="tomcat,manager-gui"/>   <user username="both" password="tomcat" roles="tomcat,role1"/>   <user username="role1" password="tomcat" roles="role1"/>
  • edit the “to_be_defined” fields in tomcat-build.xml with the username and password previously configured in Tomcat(e.g. name="tomcatUsername" value="tomcat" and name="tomcatPassword" value="tomcat" the values set in ‘tomcat-users’). Set server.location property to the directory where you installed Apache (e.g. C:\Program Files\Java\apache-tomcat-8.0.30);

VeriGraph deployment on Apache Tomcat (Unix):

  • sudo nano ~/.bashrc
  • set a few environment variables by paste the following content at the end of the file export CATALINA_HOME='/path/to/apache/tomcat/folder' export JRE_HOME='/path/to/jdk/folder' export JDK_HOME='/path/to/jdk/folder'
  • exec bash
  • open the file $CATALINA_HOME\conf\tomcat-users.xml and under the tomcat-users tag place, initialize an user with roles “tomcat, manager-gui, manager-script”. An example is the following content: xml <role rolename="tomcat"/>   <role rolename="role1"/>   <user username="tomcat" password="tomcat" roles="tomcat,manager-gui"/>   <user username="both" password="tomcat" roles="tomcat,role1"/>   <user username="role1" password="tomcat" roles="role1"/>
  • edit the “to_be_defined” fields in tomcat-build.xml with the username and password previously configured in Tomcat(e.g. name="tomcatUsername" value="tomcat" and name="tomcatPassword" value="tomcat" the values set in ‘tomcat-users’). Set server.location property to the directory where you installed Apache (e.g. C:\Program Files\Java\apache-tomcat-8.0.30);

Step4a: Deploy Verigraph in Tomcat.

ant -f build.xml deployWS

Use the Ant script build.xml to manage Verigraph webservice with the following targets:

  • generate-war: it generates the war file;
  • generate-binding: it generates the JAXB classes from the XML Schema file xml_components.xsd;
  • start-tomcat : it starts the Apache Tomcat;
  • deployWS: it deploys the verigraph.war file contained in verigraph/war folder;
  • startWS: it starts the webservice;
  • run-test: it runs the tests in tester folder. It is possible to choose the iterations number for each verification request by launching the test with “-Diteration=n run-test” where n is the number of iterations you want;
  • stopWS: it stops the webservice;
  • undeployWS: it undeploys the webservice from Apache Tomcat;
  • stop-tomcat: it stops Apache Tomcat.

Step4b: Deploy Verigraph with gRPC interface.

ant -f build.xml generate-binding
ant -f gRPC-build.xml run-server

Use the Ant script gRPC-build.xml to manage Verigraph with the following targets:

  • build: compile the program;
  • run: run both client and server;
  • run-client : run only client;
  • run-server : run only server;
  • run-test : launch all tests that are present in the package;
Parser apigateway Installation

In the present release, apigateway requires that the following software is also installed:

Please follow the below installation steps to install apigateway submodule in parser.

Step 1: Clone the parser project.

git clone https://gerrit.opnfv.org/gerrit/parser

Step 2: Install the apigateway submodule.

# change directory to apigateway
cd parser/apigateway

# install requirements
pip install -r requirements.txt

# install apigateway
python setup.py install

Notes: In release D, apigateway submodule is only initial framework code, and more feature will be provided in the next release.

OPNFV Parser Configuration Guide
Parser configuration

Parser can be configured with any installer in current OPNFV, it only depends on openstack.

Pre-configuration activities

For parser, there is not specific pre-configuration activities.

Hardware configuration

For parser, there is not hardware configuration needed for any current feature.

Feature configuration

For parser, there is not specific configure on openstack.

Parser Post Installation Procedure

Add a brief introduction to the methods of validating the installation according to this specific installer or feature.

Automated post installation activities

Describe specific post installation activities performed by the OPNFV deployment pipeline including testing activities and reports. Refer to the relevant testing guides, results, and release notes.

note: this section should be singular and derived from the test projects once we have one test suite to run for all deploy tools. This is not the case yet so each deploy tool will need to provide (hopefully very simillar) documentation of this.

Parser post configuration procedures

Describe any deploy tool or feature specific scripts, tests or procedures that should be carried out on the deployment post install and configuration in this section.

Platform components validation

Describe any component specific validation procedures necessary for your deployment tool in this section.

OPNFV Parser User Guide
Parser tosca2heat Execution

Step 1: Change directory to where the tosca yaml files are present, example is below with vRNC definiton.

cd parser/tosca2heat/tosca-parser/toscaparser/extensions/nfv/tests/data/vRNC/Definitions

Step 2: Run the python command heat-translator with the TOSCA yaml file as an input option.

heat-translator --template-file=<input file> --template-type=tosca
                --outpurt-file=<output hot file>

Example:

heat-translator --template-file=vRNC.yaml \
    --template-type=tosca --output-file=vRNC_hot.yaml

Notes: heat-translator will call class of ToscaTemplate in tosca-parser firstly to validate and parse input yaml file, then tranlate the file into hot file, if you only want to validate or check the input file and don’t want to translate, please use tosaca-parser as following:

tosca-parser --template-file=<path to the YAML template>  [--nrpv]  [--debug]
or
    tosca-parser --template-file=<path to the CSAR zip file> [--nrpv]  [--debug]
or
    tosca-parser --template-file=<URL to the template or CSAR>  [--nrpv]  [--debug]
options:
  --nrpv Ignore input parameter validation when parse template.
  --debug debug mode for print more details other than raise exceptions when errors happen

Example:

tosca-parser --template-file=vRNC.yaml --nrpv
Parser yang2tosca Execution

Step 1: Change directory to where the scripts are present.

cd parser/yang2tosca
Step 2: Copy the YANG file which needs to be converted into TOSCA to
current (parser/yang2tosca) folder.

Step 3: Run the python script “parser.py” with the YANG file as an input option.

python parser.py -n "YANG filename"

Example:

python parser.py -n example.yaml
Step 4: Verify the TOSCA YAMl which file has been created with the same name
as the YANG file with a “_tosca” suffix.
cat "YANG filename_tosca.yaml"

Example:

cat example_tosca.yaml
Parser policy2tosca Execution

Step 1: To see a list of commands available.

policy2tosca --help

Step 2: To see help for an individual command, include the command name on the command line

policy2tosca help <service>

Step 3: To inject/remove policy types/policy definitions provide the TOSCA file as input to policy2tosca command line.

policy2tosca <service> [arguments]

Example:

policy2tosca add-definition \
    --policy_name rule2 --policy_type  tosca.policies.Placement.Geolocation \
    --description "test description" \
    --properties region:us-north-1,region:us-north-2,min_inst:2 \
    --targets VNF2,VNF4 \
    --metadata "map of strings" \
    --triggers "1,2,3,4" \
    --source example.yaml

Step 4: Verify the TOSCA YAMl updated with the injection/removal executed.

cat "<source tosca file>"

Example:

cat example_tosca.yaml
Parser verigraph Execution

VeriGraph is accessible via both a RESTful API and a gRPC interface.

REST API

Step 1. Change directory to where the service graph examples are present

cd parser/verigraph/examples

Step 2. Use a REST client (e.g., cURL) to send a POST request (whose body is one of the JSON file in the directory)

curl -X POST -d @<file_name>.json http://<server_address>:<server_port>/verify/api/graphs
--header "Content-Type:application/json"

Step 3. Use a REST client to send a GET request to check a reachability-based property between two nodes of the service graph created in the previous step.

curl -X GET http://<server_addr>:<server_port>/verify/api/graphs/<graphID>/
policy?source=<srcNodeID>&destination=<dstNodeID>&type=<propertyType>

where:

  • <graphID> is the identifier of the service graph created at Step 2
  • <srcNodeID> is the name of the source node
  • <dstNodeID> is the name of the destination node
  • <propertyType> can be reachability, isolation or traversal

Step 4. the output is a JSON with the overall result of the verification process and the partial result for each path that connects the source and destination nodes in the service graph.

gRPC API

VeriGraph exposes a gRPC interface that is self-descriptive by its Protobuf file (parser/verigraph/src/main/proto/verigraph.proto). In the current release, Verigraph misses a module that receives service graphs in format of JSON and sends the proper requests to the gRPC server. A testing client has been provided to have an example of how to create a service graph using the gRPC interface and to trigger the verification step.

  1. Run the testing client
cd parser/verigraph
#Client souce code in ``parser/verigraph/src/it/polito/verigraph/grpc/client/Client.java``
ant -f buildVeriGraph_gRPC.xml run-client

Promise

Promise: Resource Management
Project:Promise, https://wiki.opnfv.org/promise
Editors:Ashiq Khan (NTT DOCOMO), Bertrand Souville (NTT DOCOMO)
Authors:Ravi Chunduru (ClearPath Networks), Peter Lee (ClearPath Networks), Gerald Kunzmann (NTT DOCOMO), Ryota Mibu (NEC), Carlos Goncalves (NEC), Arturo Martin De Nicolas (Ericsson)
Abstract:Promise is an OPNFV requirement project. Its objective is to realize ETSI NFV defined resource reservation and NFVI capacity features within the scope of OPNFV. Promise provides the details of the requirements on resource reservation, NFVI capacity management at VIM, specification of the northbound interfaces from VIM relevant to these features, and implementation plan to realize these features in OPNFV.
1. Definition of terms

Different SDOs and communities use different terminology related to NFV/Cloud/SDN. This list tries to define an OPNFV terminology, mapping/translating the OPNFV terms to terminology used in other contexts.

Administrator
Administrator of the system, e.g. OAM in Telco context.
Consumer
User-side Manager; consumer of the interfaces produced by the VIM; VNFM, NFVO, or Orchestrator in ETSI NFV [NFV003] terminology.
NFV
Network Function Virtualization
NFVI
Network Function Virtualization Infrastructure; totality of all hardware and software components which build up the environment in which VNFs are deployed.
NFVO
Network Functions Virtualization Orchestrator; functional block that manages the Network Service (NS) lifecycle and coordinates the management of NS lifecycle, VNF lifecycle (supported by the VNFM) and NFVI resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity.
Physical resource
Actual resources in NFVI; not visible to Consumer.
Resource zone
A set of NFVI hardware and software resources logically grouped according to physical isolation and redundancy capabilities or to certain administrative policies for the NFVI [NFVIFA010]
VIM
Virtualized Infrastructure Manager; functional block that is responsible for controlling and managing the NFVI compute, storage and network resources, usually within one operator’s Infrastructure Domain, e.g. NFVI Point of Presence (NFVI-PoP).
Virtual Machine (VM)
Virtualized computation environment that behaves very much like a physical computer/server.
Virtual network
Virtual network routes information among the network interfaces of VM instances and physical network interfaces, providing the necessary connectivity.
Virtual resource
A Virtual Machine (VM), a virtual network, or virtualized storage; Offered resources to “Consumer” as result of infrastructure virtualization; visible to Consumer.
Virtual Storage
Virtualized non-volatile storage allocated to a VM.
VNF
Virtualized Network Function. Implementation of an Network Function that can be deployed on a Network Function Virtualization Infrastructure (NFVI).
VNFM
Virtualized Network Function Manager; functional block that is responsible for the lifecycle management of VNF.
2. Introduction

Resource reservation is a basic function for the operation of a virtualized telecom network. In resource reservation, VIM reserves resources for a certain period as requested by the NFVO. A resource reservation will have a start time which could be into the future. Therefore, the reserved resources shall be available for the NFVO requested purpose (e.g. for a VNF) at the start time for the duration asked by NFVO. Resources include all three resource types in an NFVI i.e. compute, storage and network.

Besides, NFVO requires abstracted NFVI resource capacity information in order to take decisions on VNF placement and other operations related to the virtual resources. VIM is required to inform the NFVO of NFVI resource state information for this purpose. Promise project aims at delivering the detailed requirements on these two features defined in ETSI NFV MAN GS [NFVMAN], the list of gaps in upstream projects, potential implementation architecture and plan, and the VIM northbound interface specification for resource reservation and capacity management.

2.1. Problem description

OpenStack, a prominent candidate for the VIM, cannot reserve resources for future use. OpenStack requires immediate instantiation of Virtual Machines (VMs) in order to occupy resources intended to be reserved. Blazar can reserve compute resources for future by keeping the VMs in shelved mode. However, such reserved resources can also be used for scaling out rather than new VM instantiation. Blazar does not support network and storage resource reservation yet.

Besides, OpenStack does not provide a northbound interface through which it can notify an upper layer management entity e.g. NFVO about capacity changes in its NFVI, periodically or in an event driven way. Capacity management is a feature defined in ETSI NFV MAN GS [NFVMAN] and is required in network operation.

3. Use cases and scenarios
3.1. Use cases

Resource reservation is a basic feature in any virtualization-based network operation. In order to perform such resource reservation from NFVO to VIM, NFVI capacity information is also necessary at the NFVO side. Below, four use cases to show typical requirements and solutions for capacity management and resource reservation is presented. A typical use case as considered for the Brahmaputra release is described in ANNEX A: Use case for OPNFV Brahmaputra.

  1. Resource capacity management
  2. Resource reservation for immediate use
  3. Resource reservation for future use
  4. Co-existence of reservations and allocation requests without reservation
3.1.1. Resource capacity management

NFVO takes the first decision on in which NFVI it would instantiate a VNF. Along with NFVIs resource attributes (e.g. availability of hardware accelerators, particular CPU architectures etc.), NFVO needs to know available capacity of an NFVI in order to make an informed decision on selecting a particular NFVI. Such capacity information shall be in a coarser granularity than the respective VIM, as VIM maintains capacity information of its NFVI in fine details. However a very coarse granularity, like simply the number of available virtual CPU cores, may not be sufficient. In order to allow the NFVO to make well founded allocation decisions, an appropriate level to expose the available capacity may be per flavor. Capacity information may be required for the complete NFVI, or per partition or availability zone, or other granularities. Therefore, VIM requires to inform the NFVO about available capacity information regarding its NFVI at a pre-determined abstraction, either by a query-response, or in an event-based, or in a periodical way.

3.1.2. Resource reservation for immediate use

Reservation is inherently for the future. Even if some reserved resources are to be consumed instantly, there is a network latency between the issuance of a resource reservation request from the NFVO, a response from the VIM, and actual allocation of the requested resources to a VNF/VNFM. Within such latency, resource capacity in the NFVI in question could change, e.g., due to failure, allocation to a different request. Therefore, the response from a VIM to the NFVO to a resource reservation request for immediate use should have a validity period which shows until when this VIM can hold the requested resources. During this time, the NFVO should proceed to allocation if it wishes to consume the reserved requested. If allocation is not performed within the validity period, the response from VIM for a particular resource reservation request becomes invalid and VIM is not liable to provide those resources to NFVO/VNFM anymore. Reservations requests for immediate use do not have a start time but may have an end time.

3.1.3. Resource reservation for future use

Network operators may want to reserve extra resources for future use. Such necessity could arise from predicted congestion in telecom nodes e.g. due to local traffic spikes for concerts, natural disasters etc. In such a case, the NFVO, while sending a resource reservation request to the VIM, shall include a start time (and an end time if necessary). The start time indicates at what time the reserved resource shall be available to a designated consumer e.g. a VNF/VNFM. Here, the requirement is that the reserved resources shall be available when the start time arrives. After the start time has arrived, the reserved resources are allocated to the designated consumer(s). An explicit allocation request is needed. How actually these requested resources are held by the VIM for the period in between the arrival of the resource reservation request and the actual allocation is outside the scope of this requirement project.

3.1.4. Co-existence of reservations and allocation requests without reservation

In a real environment VIM will have to handle allocation requests without any time reference, i.e. time-unbound, together with time-bound reservations and allocation requests with an explicitly indicated end-time. A granted reservation for the future will effectively reduce the available capacity for any new time-unbound allocation request. The consequence is that reservations, even those far in the future, may result in denial of service for new allocation requests.

To alleviate this problem several approaches can be taken. They imply an implicit or explicit priority scheme:

  • Allocation requests without reservation and which are time-unbound will be granted resources in a best-effort way: if there is instant capacity, but the resources may be later withdrawn due to the start time of a previously granted reservation
  • Both allocation requests and reservation requests contain a priority which may be related to SLAs and contractual conditions between the tenant and the NFVI provider. Interactions may look like:
    • A reservation request for future use may cancel another, not yet started, reservation with lower priority
    • An allocation request without reservations and time-unbound [1] may be granted resources and prevent a future reservation with lower priority from getting resources at start time
    • A reservation request may result in terminating resources allocated to a request with no reservation, if the latter has lower priority
[1]In this case, the consumer (VNFM or NFVO) requests to immediately instantiate and assign virtualized resources without having reserved the resources beforehand
3.2. Scenarios

This section describes the expected behavior of the system in different scenarios.

As we are targeting a cloud platform with the above use cases, it is crucial to keep the flexibility in the system. By this means it is hard to explicitely block hardware resources for the future and ensure their availablility for allocation therefore it can happen that the reservation plan cannot be fulfilled due to certain changes in the underlying environment. We need to ensure that we are prepared for error and edge cases during the design period.

4. High level architecture and general features
4.1. Architecture Overview
_images/figure16.png

Resource Reservation Architecture

figure1 shows the high level architecture for the resource reservation use cases. Reserved resources are guaranteed for a given user/client for the period expressed by start and end time. User/client represents the requestor and the consequent consumer of the reserved resources and correspond to the NFVO or VNFM in ETSI NFV terminology.

Note: in this document only reservation requests from NFVO are considered.

4.2. General Features

This section provides a list of features that need to be developed in the Promise project.

  • Resource capacity management
    • Discovery of available resource capacity in resource providers
    • Monitoring of available resource capacity in resource providers
    • Update available resource capacity as a result of new or expired reservations, addition/removal of resources. Note: this is a VIM internal function, not an operation in the VIM northbound interface.
  • Resource reservation
    • Set start time and end time for allocation
    • Increase/decrease reserved resource’s capacity
    • Update resource reservations, e.g. add/remove reserved resources
    • Terminate an allocated resource due to the end time of a reservation
  • VIM northbound interfaces
    • Receive/Reply resource reservation requests
    • Receive/Reply resource capacity management requests
    • Receive/Reply resource allocation requests for reserved resources when start time arrives
    • Subscribe/Notify resource reservation event
      • Notify reservation error or process completion prior to reservation start
      • Notify remaining time until termination of a resource due to the end time of a reservation
      • Notify termination of a resource due to the end time of a reservation
    • Receive/Reply queries on available resource capacity
    • Subscribe/Notify changes in available resource capacity
4.3. High level northbound interface specification
4.3.1. Resource Capacity Management
_images/figure22.png

Resource capacity management message flow: notification of capacity change

figure2 shows a high level flow for a use case of resource capacity management. In this example, the VIM notifies the NFVO of capacity change after having received an event regarding a change in capacity (e.g. a fault notification) from the NFVI. The NFVO can also retrieve detailed capacity information using the Query Capacity Request interface operation.

_images/figure32.png

Resource capacity management message flow: query of capacity density

figure3 shows a high level flow for another use case of resource capacity management. In this example, the NFVO queries the VIM about the actual capacity to instantiate a certain resource according to a certain template, for example a VM according to a certain flavor. In this case the VIM responds with the number of VMs that could be instantiated according to that flavor with the currently available capacity.

4.3.2. Resource Reservation
_images/figure42.png

Resource reservation flow

figure4 shows a high level flow for a use case of resource reservation. The main steps are:

  • The NFVO sends a resource reservation request to the VIM using the Create Resource Reservation Request interface operation.
  • The NFVO gets a reservation identifier reservation associated with this request in the reply message
  • Using the reservation identifier reservation, the NFVO can query/update/terminate a resource reservation using the corresponding interface operations
  • The NFVO is notified that the resource reservation is terminated due to the end time of the reservation
4.4. Information elements
4.4.1. Resource Capacity Management
4.4.1.1. Notify Capacity Change Event

The notification change message shall include the following information elements:

Name Type Description
Notification Identifier Identifier issued by the VIM for the capacity change event notification
Zone Identifier Identifier of the zone where capacity has changed
Used/Reserved/Total Capacity List Used, reserved and total capacity information regarding the resource items subscribed for notification for which capacity change event occurred
4.4.1.2. Query Resource Capacity Request

The capacity management query request message shall include the following information elements:

Name Type Description
Zone Identifier Identifier of the zone where capacity is requested
Attributes List Attributes of resource items to be notified regarding capacity change events
Resources List Identifiers of existing resource items to be queried regarding capacity info (such as images, flavors, virtual containers, networks, physical machines, etc.)

The capacity management query request message may also include the following information element:

Name Type Description
Flavor Identifier Identifier that is passed in the request to obtain information of the number of virtual resources that can be instantiated according to this flavor with the available capacity
4.4.1.3. Query Resource Capacity Reply

The capacity management query reply message shall include the following information elements:

Name Type Description
Zone Identifier Identifier of the zone where capacity is requested
Used/Reserved/Total Capacity List Used, reserved and total capacity information regarding each of the resource items requested to check for capacity

The detailed specification of the northbound interface for Capacity Management in provided in section 5.1.1.

4.4.2. Resource Reservation
4.4.2.1. Create Resource Reservation Request

The create resource reservation request message shall include the following information elements:

Name Type Description
Start Timestamp Start time for consumption of the reserved resources
End Timestamp End time for consumption of the reserved resources
Expiry Timestamp If not all reserved resources are allocated between start time and expiry, the VIM shall release the corresponding resources [1]
Amount Number Amount of the resources per resource item type (i.e. compute/network/storage) that need to be reserved
Zone Identifier The zone where the resources need(s) to be reserved
Attributes List Attributes of the resources to be reserved such as DPDK support, hypervisor, network link bandwidth, affinity rules, etc.
Resources List Identifiers of existing resource items to be reserved (such as images, flavors, virtual containers, networks, physical machines, etc.)
[1]Expiry is a period around start time within which, the allocation process must take place. If allocation process does not start within the expiry period, the reservation becomes invalid and VIM should release the resources
4.4.2.2. Create Resource Reservation Reply

The create resource reservation reply message shall include the following information elements:

Name Type Description
Reservation Identifier Identification of the reservation instance. It can be used by a consumer to modify the reservation later, and to request the allocation of the reserved resources.
Message Text Output message that provides additional information about the create resource reservation request (e.g. may be a simple ACK if the request is being background processed by the VIM)
4.4.2.3. Notify Reservation Event

The notification reservation event message shall include the following information elements:

Name Type Description
Reservation Identifier Identification of the reservation instance triggering the event
Notification Identifier Identification of the resource event notification issued by the VIM
Message Text Message describing the event

The detailed specification of the northbound interface for Resource Reservation is provided in section 5.1.2.

5. Gap analysis in upstream projects

This section provides a list of gaps in upstream projects for realizing resource reservation and management. The gap analysis work focuses on the current OpenStack Blazar project [BLAZAR] in this first release.

5.1. OpenStack
5.1.1. Resource reservation for future use
5.1.2. Resource reservation update
  • Category: Blazar
  • Type: ‘missing’ (lack of functionality)
  • Description:
    • To-be: Have the possibility of adding/removing resources to an existing reservation, e..g in case of NFVI failure
    • As-is: Currently in Blazar, a reservation can only be modified in terms of start/end time
  • Related blueprints: N/A
5.1.3. Give me an offer
  • Category: Blazar
  • Type: ‘missing’ (lack of functionality)
  • Description:
    • To-be: To have the possibility of giving a quotation to a requesting user and an expiration time. Reserved resources shall be released if they are not claimed before this expiration time.
    • As-is: Blazar can already send notification e.g. to inform a given user that a reservation is about to expire
  • Related blueprints: N/A
5.1.4. StormStack StormForge
5.1.4.1. Stormify
  • Stormify enables rapid web applications construction
  • Based on Ember.js style Data stores
  • Developed on Node.js using coffeescript/javascript
  • Auto RESTful API generation based on Data Models
  • Development starts with defining Data Models
  • Code hosted at github : http://github.com/stormstack/stormify
5.1.4.2. StormForge
  • Data Model driven management of Resource Providers
  • Based on Stormify Framework and implemented as per the OPNFV Promise requirements
  • Data Models are auto generated and RESTful API code from YANG schema
  • Currently planned key services include Resource Capacity Management Service and Resource Reservation Service
  • List of YANG schemas for Promise project is attached in the Appendix
  • Code hosted at github: http://github.com/stormstack/stormforge
5.1.4.3. Resource Discovery
  • Category: StormForge
  • Type: ‘planning’ (lack of functionality)
  • Description
    • To-be: To be able to discover resources in real time from OpenStack components. Planning to add OpenStack Project to interface with Promise for real time updates on capacity or any failures
    • As-is: Currently, resource capacity is learnt using NB APIs related to quota
  • Related Blueprints: N/A
6. Detailed architecture and message flows

Within the Promise project we consider two different architectural options, i.e. a shim-layer based architecture and an architecture targeting at full OpenStack integration.

6.1. Shim-layer architecture

The shim-layer architecture is using a layer on top of OpenStack to provide the capacity management, resource reservation, and resource allocation features.

6.1.1. Detailed Message Flows

Note, that only selected parameters for the messages are shown. Refer to Detailed northbound interface specification and Annex ANNEX B: Promise YANG schema based on YangForge for a full set of message parameters.

6.1.1.1. Resource Capacity Management
_images/figure5_new.png

Capacity Management Scenario

figure5 shows a detailed message flow between the consumers and the capacity management functional blocks inside the shim-layer. It has the following steps:

  • Step 1a: The Consumer sends a query-capacity request to Promise using some filter like time-windows or resource type. The capacity is looked up in the shim-layer capacity map.
  • Step 1b: The shim-layer will respond with information about the total, available, reserved, and used (allocated) capacities matching the filter.
  • Step 2a: The Consumer can send increase/decrease-capacity requests to update the capacity available to the reservation system. It can be 100% of available capacity in the given provider/source or only a subset, i.e., it can allow for leaving some “buffer” in the actual NFVI to be used outside the Promise shim-layer or for a different reservation service instance. It can also be used to inform the reservation system that from a certain time in the future, additional resources can be reserved (e.g. due to a planned upgrade of the capacity), or the available capacity will be reduced (e.g. due to a planned downtime of some of the resources).
  • Step 2b: The shim-layer will respond with an ACK/NACK message.
  • Step 3a: Consumers can subscribe for capacity-change events using a filter.
  • Step 3b: Each successful subscription is responded with a subscription_id.
  • Step 4: The shim-layer monitors the capacity information for the various types of resources by periodically querying the various Controllers (e.g. Nova, Neutron, Cinder) or by creating event alarms in the VIM (e.g. with Ceilometer for OpenStack) and updates capacity information in its capacity map.
  • Step 5: Capacity changes are notified to the Consumer.
6.1.1.2. Resource Reservation
_images/figure6_new.png

Resource Reservation for Future Use Scenario

figure6 shows a detailed message flow between the Consumer and the resource reservation functional blocks inside the shim-layer. It has the following steps:

  • Step 1a: The Consumer creates a resource reservation request for future use by setting a start and end time for the reservation as well as more detailed information about the resources to be reserved. The Promise shim-layer will check the free capacity in the given time window and in case sufficient capacity exists to meet the reservation request, will mark those resources “reserved” in its reservation map.
  • Step 1b: If the reservation was successful, a reservation_id and status of the reservation will be returned to the Consumer. In case the reservation cannot be met, the shim-layer may return information about the maximum capacity that could be reserved during the requested time window and/or a potential time window where the requested (amount of) resources would be available.
  • Step 2a: Reservations can be updated using an update-reservation, providing the reservation_id and the new reservation_data. Promise Reservation Manageer will check the feasibility to update the reservation as requested.
  • Step 2b: If the reservation was updated successfully, a reservation_id and status of the reservation will be returned to the Consumer. Otherwise, an appropriate error message will be returned.
  • Step 3a: A cancel-reservation request can be used to withdraw an existing reservation. Promise will update the reservation map by removing the reservation as well as the capacity map by adding the freed capacity.
  • Step 3b: The response message confirms the cancelation.
  • Step 4a: Consumers can also issue query-reservation requests to receive a list of reservation. An input filter can be used to narrow down the query, e.g., only provide reservations in a given time window. Promise will query its reservation map to identify reservations matching the input filter.
  • Step 4b: The response message contains information about all reservations matching the input filter. It also provides information about the utilization in the requested time window.
  • Step 5a: Consumers can subscribe for reservation-change events using a filter.
  • Step 5b: Each successful subscription is responded with a subscription_id.
  • Step 6a: Promise synchronizes the available and used capacity with the underlying VIM.
  • Step 6b: In certain cases, e.g., due a failure in the underlying hardware, some reservations cannot be kept up anymore and have to be updated or canceled. The shim-layer will identify affected reservations among its reservation records.
  • Step 7: Subscribed Consumers will be informed about the updated reservations. The notification contains the updated reservation_data and new status of the reservation. It is then up to the Consumer to take appropriate actions in order to ensure high priority reservations are favored over lower priority reservations.
6.1.1.3. Resource Allocation
_images/figure7_new.png

Resource Allocation

figure7 shows a detailed message flow between the Consumer, the functional blocks inside the shim-layer, and the VIM. It has the following steps:

  • Step 1a: The Consumer sends a create-instance request providing information about the resources to be reserved, i.e., provider_id (optional in case of only one provider), name of the instance, the requested flavour and image, etc. If the allocation is against an existing reservation, the reservation_id has to be provided.
  • Step 1b: If a reservation_id was provided, Promise checks if a reservation with that ID exists, the reservation start time has arrived (i.e. the reservation is active), and the required capacity for the requested flavor is within the available capacity of the reservation. If those conditions are met, Promise creates a record for the allocation (VMState=”INITIALIZED”) and update its databases. If no reservation_id was provided in the allocation request, Promise checks whether the required capacity to meet the request can be provided from the available, non-reserved capacity. If yes, Promise creates a record for the allocation and update its databases. In any other case, Promise rejects the create-instance request.
  • Step 2: In the case the create-instance request was rejected, Promise responds with a “status=rejected” providing the reason of the rejection. This will help the Consumer to take appropriate actions, e.g., send an updated create-instance request. The allocation work flow will terminate at this step and the below steps are not executed.
  • Step 3a: If the create-instance request was accepted and a related allocation record has been created, the shim-layer issues a createServer request to the VIM Controller providing all information to create the server instance.
  • Step 3b: The VIM Controller sends an immediate reply with an instance_id and starts the VIM-internal allocation process.
  • Step 4: The Consumer gets an immediate response message with allocation status “in progress” and the assigned instance_id.
  • Step 5a+b: The consumer subscribes to receive notifications about allocation events related to the requested instance. Promise responds with an acknowledgment including a subscribe_id.
  • Step 6: In parallel to the previous step, Promise shim-layer creates an alarm in Aodh to receive notifications about all changes to the VMState for instance_id.
  • Step 7a: The VIM Controller notifies all instance related events to Ceilometer. After the allocation has been completed or failed, it sends an event to Ceilometer. This triggers the OpenStack alarming service Aodh to notify the new VMState (e.g. ACTIVE and ERROR) to the shim-layer that updates its internal allocation records.
  • Step 7b: Promise sends a notification message to the subscribed Consumer with information on the allocated resources including their new VMState.
  • Step 8a+b: Allocated instances can be terminated by the Consumer by sending a destroy-instance request to the shim-layer. Promise responds with an acknowledgment and the new status “DELETING” for the instance.
  • Step 9a: Promise sends a deleteServer request for the instance_id to the VIM Controller.
  • Step 10a: After the instance has been deleted, an event alarm is sent to the shim-layer that updates its internal allocation records and capacity utilization.
  • Step 10b: The shim-layer also notifies the subscribed Consumer about the successfully destroyed instance.
6.1.2. Internal operations

Note

This section is to be updated

In the following, the internal logic and operations of the shim-layer will be explained in more detail, e.g. the “check request” (step 1b in figure7 of the allocation work flow).

6.2. Integrated architecture

The integrated architecture aims at full integration with OpenStack. This means that it is planned to use the already existing OpenStack APIs extended with the reservation capabilities.

The advantage of this approach is that we don’t need to re-model the complex resource structure we have for the virtual machines and the corresponding infrastructure.

The atomic item is the virtual machine with the minimum set of resources it requires to be able to start it up. It is important to state that resource reservation is handled on VM instance level as opposed to standalone resources like CPU, memory and so forth. As the placement is an important aspect in order to be able to use the reserved resources it provides the constraint to handle resources in groups.

The placement constraint also makes it impossible to use a quota management system to solve the base use case described earlier in this document.

OpenStack had a project called Blazar, which was created in order to provide resource reservation functionality in cloud environments. It uses the Shelve API of Nova, which provides a sub-optimal solution. Due to the fact that this feature blocks the reserved resources this solution cannot be considered to be final. Further work is needed to reach a more optimal stage, where the Nova scheduler is intended to be used to schedule the resources for future use to make the reservations.

6.2.1. Phases of the work

The work has two main stages to reach the final solution. The following main work items are on the roadmap for this approach:

  1. Sub-optimal solution by using the shelve API of Nova through the Blazar project:

    • Fix the code base of the Blazar project:

      Due to integration difficulties the Blazar project got suspended. Since the last activities in that repository the OpenStack code base and environment changed significantly, which means that the project’s code base needs to be updated to the latest standards and has to be able to interact with the latest version of the other OpenStack services.

    • Update the Blazar API:

      The REST API needs to be extended to contain the attributes for the reservation defined in this document. This activity shall include testing towards the new API.

  2. Use Nova scheduler to avoid blocking the reserved resources:

    • Analyze the Nova scheduler:

      The status and the possible interface between the resource reservation system and the Nova scheduler needs to be identified. It is crucial to achieve a much more optimal solution than what the current version of Blazar can provide. The goal is to be able to use the reserved resources before the reservation starts. In order to be able to achieve this we need the scheduler to do scheduling for the future considering the reservation intervals that are specified in the request.

    • Define a new design based on the analysis and start the work on it:

      The design for the more optimal solution can be defined only after analyzing the structure and capabilities of the Nova scheduler.

    • This phase can be started in parallel with the previous one.

6.2.2. Detailed Message Flows

Note

to be done

6.2.2.1. Resource Reservation

Note

to be specified

7. Detailed northbound interface specification

Note

This is Work in Progress.

7.1. ETSI NFV IFA Information Models
7.1.1. Compute Flavor

A compute flavor includes information about number of virtual CPUs, size of virtual memory, size of virtual storage, and virtual network interfaces [NFVIFA005].

_images/computeflavor.png
7.2. Virtualised Compute Resources
7.2.1. Compute Capacity Management
7.2.1.1. Subscribe Compute Capacity Change Event

Subscription from Consumer to VIM to be notified about compute capacity changes

POST /capacity/compute/subscribe

Example request:

POST /capacity/compute/subscribe HTTP/1.1
Accept: application/json

{
   "zoneId": "12345",
   "computeResourceTypeId": "vcInstances",
   "threshold": {
      "thresholdType" : "absoluteValue",
      "threshold": {
          "capacity_info": "available",
          "condition": "lt",
          "value": 5
      }
   }
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
   "created": "2015-09-21T00:00:00Z",
   "capacityChangeSubscriptionId": "abcdef-ghijkl-123456789"
}
Status Codes:
7.2.1.2. Query Compute Capacity for a defined resource type

Request to find out about available, reserved, total and allocated compute capacity.

GET /capacity/compute/query

Example request:

GET /capacity/compute/query HTTP/1.1
Accept: application/json

{
  "zoneId": "12345",
  "computeResourceTypeId": "vcInstances",
  "timePeriod":  {
       "startTime": "2015-09-21T00:00:00Z",
       "stopTime": "2015-09-21T00:05:30Z"
  }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
   "zoneId": "12345",
   "lastUpdate": "2015-09-21T00:03:20Z",
   "capacityInformation": {
      "available": 4,
      "reserved": 17,
      "total": 50,
      "allocated": 29
   }
}
Query Parameters:
 
  • limit – Default is 10.
Status Codes:
7.2.1.3. Query Compute Capacity with required attributes

Request to find out available compute capacity with given characteristics

GET /capacity/compute/query

Example request:

GET /capacity/compute/query HTTP/1.1
Accept: application/json

{
  "zoneId": "12345",
  "resourceCriteria":  {
       "virtualCPU": {
           "cpuArchitecture": "x86",
           "numVirtualCpu": 8
       }
  },
  "attributeSelector":  "available",
  "timePeriod":  {
       "startTime": "2015-09-21T00:00:00Z",
       "stopTime": "2015-09-21T00:05:30Z"
  }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
   "zoneId": "12345",
   "lastUpdate": "2015-09-21T00:03:20Z",
   "capacityInformation": {
      "available": 50
   }
}
Query Parameters:
 
  • limit – Default is 10.
Status Codes:
7.2.1.4. Notify Compute Capacity Change Event

Notification about compute capacity changes

POST /capacity/compute/notification

Example notification:

Content-Type: application/json

{
     "zoneId": "12345",
     "notificationId": "zyxwvu-tsrqpo-987654321",
     "capacityChangeTime": "2015-09-21T00:03:20Z",
     "resourceDescriptor": {
        "computeResourceTypeId": "vcInstances"
     },
     "capacityInformation": {
        "available": 4,
        "reserved": 17,
        "total": 50,
        "allocated": 29
     }
}
7.2.2. Compute Resource Reservation
7.2.2.1. Create Compute Resource Reservation

Request the reservation of compute resource capacity

POST /reservation/compute/create

Example request:

POST /reservation/compute/create HTTP/1.1
Accept: application/json

{
    "startTime": "2015-09-21T01:00:00Z",
    "computePoolReservation": {
        "numCpuCores": 20,
        "numVcInstances": 5,
        "virtualMemSize": 10
    }
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
   "reservationData": {
      "startTime": "2015-09-21T01:00:00Z",
      "reservationStatus": "initialized",
      "reservationId": "xxxx-yyyy-zzzz",
      "computePoolReserved": {
          "numCpuCores": 20,
          "numVcInstances": 5,
          "virtualMemSize": 10,
          "zoneId": "23456"
      }
   }
}

or virtualization containers

POST reservation/compute/create

Example request:

POST /reservation/compute/create HTTP/1.1
Accept: application/json

{
  "startTime": "2015-10-05T15:00:00Z",
  "virtualizationContainerReservation": [
    {
       "containerId": "myContainer",
       "containerFlavor": {
          "flavorId": "myFlavor",
          "virtualCpu": {
             "numVirtualCpu": 2,
             "cpuArchitecture": "x86"
          },
          "virtualMemory": {
              "numaEnabled": "False",
              "virtualMemSize": 16
          },
          "storageAttributes": {
              "typeOfStorage": "volume",
              "sizeOfStorage": 16
          }
       }
    }
  ]
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
   "reservationData": {
      "startTime": "2015-10-05T15:00:00Z",
      "reservationId": "aaaa-bbbb-cccc",
      "reservationStatus": "initialized",
      "virtualizationContainerReserved": [
          {
             "containerId": "myContainer",
             "flavorId": "myFlavor",
             "virtualCpu": {
                 "numVirtualCpu": 2,
                 "cpuArchitecture": "x86"
             },
             "virtualMemory": {
                 "numaEnabled": "False",
                 "virtualMemSize": 16
             },
             "virtualDisks": {
                 "storageId": "myStorage",
                 "flavourId": "myStorageFlavour",
                 "typeOfStorage": "volume",
                 "sizeOfStorage": 16,
                 "operationalState": "enabled"
             }
          }
      ]
   }
}
7.2.2.2. Query Compute Resource Reservation

Request to find out about reserved compute resources that the consumer has access to.

GET /reservation/compute/query

Example request:

GET /reservation/compute/query HTTP/1.1
Accept: application/json

{
   "queryReservationFilter": [
       {
           "reservationId": "xxxx-yyyy-zzzz"
       }
   ]

}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
   "queryResult":
   {
      "startTime": "2015-09-21T01:00:00Z",
      "reservationStatus": "active",
      "reservationId": "xxxx-yyyy-zzzz",
      "computePoolReserved":
      {
          "numCpuCores": 20,
          "numVcInstances": 5,
          "virtualMemSize": 10,
          "zoneId": "23456"
      }
   }
}
Status Codes:
7.2.2.3. Update Compute Resource Reservation

Request to update compute resource reservation

POST /reservation/compute/update

Example request:

POST /reservation/compute/update HTTP/1.1
Accept: application/json

{
    "startTime": "2015-09-14T16:00:00Z",
    "reservationId": "xxxx-yyyy-zzzz"
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
  "reservationData": {
      "startTime": "2015-09-14TT16:00:00Z",
      "reservationStatus": "active",
      "reservationId": "xxxx-yyyy-zzzz",
      "computePoolReserved": {
          "numCpuCores": 20,
          "numVcInstances": 5,
          "virtualMemSize": 10,
          "zoneId": "23456"
      }
   }
}
7.2.2.4. Terminate Compute Resource Reservation

Request to terminate a compute resource reservation

DELETE /reservation/compute/(reservation_id)

Example response:

HTTP/1.1 200
Content-Type: application/json

{
   "reservationId": "xxxx-yyyy-zzzz",
}
7.2.2.5. Subscribe Resource Reservation Change Event

Subscription from Consumer to VIM to be notified about changes related to a reservation or to the resources associated to it.

POST /reservation/subscribe

Example request:

 POST /reservation/subscribe HTTP/1.1
 Accept: application/json

 {
    "inputFilter": [
        {
           "reservationId": "xxxx-yyyy-zzzz",
        }
    ]
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
   "created": "2015-09-21T00:00:00Z",
   "reservationChangeSubscriptionId": "abcdef-ghijkl-123456789"
}
Status Codes:
7.2.2.6. Notify Resource Reservation Change Event

Notification about changes in a compute resource reservation

POST /capacity/compute/notification

Example notification:

Content-Type: application/json

{
     "changeId": "aaaaaa-btgxxx-987654321",
     "reservationId": "xxxx-yyyy-zzzz",
     "vimId": "vim-CX-03"
     "changeType": "Reservation time change"
     "changedReservationData": {
        "endTime": "2015-10-14TT16:00:00Z",
     }
}
7.3. Virtualised Network Resources
7.3.1. Network Capacity Management
7.3.1.1. Subscribe Network Capacity Change Event

Susbcription from Consumer to VIM to be notified about network capacity changes

POST /capacity/network/subscribe

Example request:

POST /capacity/network/subscribe HTTP/1.1
Accept: application/json

{
    "networkResourceTypeId": "publicIps",
    "threshold": {
       "thresholdType": "absoluteValue",
       "threshold": {
           "capacity_info": "available",
           "condition": "lt",
           "value": 5
       }
    }
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
   "created": "2015-09-28T00:00:00Z",
   "capacityChangeSubscriptionId": "bcdefg-hijklm-234567890"
}
7.3.1.2. Query Network Capacity

Request to find out about available, reserved, total and allocated network capacity.

GET /capacity/network/query

Example request:

GET /capacity/network/query HTTP/1.1
Accept: application/json

{
    "networkResourceTypeId": "publicIps",
    "timePeriod":  {
        "startTime": "2015-09-28T00:00:00Z",
        "stopTime": "2015-09-28T00:05:30Z"
    }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "lastUpdate": "2015-09-28T00:02:10Z",
    "capacityInformation": {
        "available": 4,
        "reserved": 10,
        "total": 64,
        "allocated": 50
    }
}
7.3.1.3. Notify Network Capacity Change Event

Notification about network capacity changes

POST /capacity/network/notification

Example notification:

Content-Type: application/json

{
    "notificationId": "yxwvut-srqpon-876543210",
    "capacityChangeTime": "2015-09-28T00:02:10Z",
    "resourceDescriptor": {
        "networkResourceTypeId": "publicIps"
    },
    "capacityInformation": {
        "available": 4,
        "reserved": 10,
        "total": 64,
        "allocated": 50
    }
}
7.3.2. Network Resource Reservation
7.3.2.1. Create Network Resource Reservation

Request the reservation of network resource capacity and/or virtual networks, network ports

POST /reservation/network/create

Example request:

POST /reservation/network/create HTTP/1.1
Accept: application/json

{
    "startTime": "2015-09-28T01:00:00Z",
    "networkReservation": {
        "numPublicIps": 2
    }
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
    "reservationData": {
        "startTime": "2015-09-28T01:00:00Z",
        "reservationStatus": "initialized",
        "reservationId": "wwww-xxxx-yyyy",
        "publicIps": [
            "10.2.91.60",
            "10.2.91.61"
        ]
    }
}
7.3.2.2. Query Network Resource Reservation

Request to find out about reserved network resources that the consumer has access to.

GET /reservation/network/query

Example request:

GET /reservation/network/query HTTP/1.1
Accept: application/json

{
    "queryReservationFilter": [
        {
            "reservationId": "wwww-xxxx-yyyy"
        }
    ]
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "queryResult": {
        "startTime": "2015-09-28T01:00:00Z",
        "reservationStatus": "active",
        "reservationId": "wwww-xxxx-yyyy",
        "publicIps": [
            "10.2.91.60",
            "10.2.91.61"
        ]
    }
}
7.3.2.3. Update Network Resource Reservation

Request to update network resource reservation

POST /reservation/network/update

Example request:

POST /reservation/network/update HTTP/1.1
Accept: application/json

{
    "startTime": "2015-09-21T16:00:00Z",
    "reservationId": "wwww-xxxx-yyyy"
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
    "reservationData": {
        "startTime": "2015-09-21T16:00:00Z",
        "reservationStatus": "active",
        "reservationId": "wwww-xxxx-yyyy",
        "publicIps": [
           "10.2.91.60",
           "10.2.91.61"
        ]
    }
}
7.3.2.4. Terminate Network Resource Reservation

Request to terminate a network resource reservation

DELETE /reservation/network/(reservation_id)

Example response:

HTTP/1.1 200
Content-Type: application/json

{
   "reservationId": "xxxx-yyyy-zzzz",
}
7.4. Virtualised Storage Resources
7.4.1. Storage Capacity Management
7.4.1.1. Subscribe Storage Capacity Change Event

Subscription from Consumer to VIM to be notified about storage capacity changes

POST /capacity/storage/subscribe

Example request:

POST /capacity/storage/subscribe HTTP/1.1
Accept: application/json

{
   "storageResourceTypeId": "volumes",
   "threshold": {
      "thresholdType": "absoluteValue",
      "threshold": {
          "capacity_info": "available",
          "condition": "lt",
          "value": 3
       }
   }
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
    "created": "2015-09-28T12:00:00Z",
    "capacityChangeSubscriptionId": "cdefgh-ijklmn-345678901"
}
7.4.1.2. Query Storage Capacity for a defined resource type

Request to find out about available, reserved, total and allocated storage capacity.

GET /capacity/storage/query

Example request:

GET /capacity/storage/query HTTP/1.1
Accept: application/json

{
    "storageResourceTypeId": "volumes",
    "timePeriod":  {
        "startTime": "2015-09-28T12:00:00Z",
        "stopTime": "2015-09-28T12:04:45Z"
    }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "lastUpdate": "2015-09-28T12:01:35Z",
    "capacityInformation": {
        "available": 2,
        "reserved": 4,
        "total": 10,
        "allocated": 4
    }
}
7.4.1.3. Query Storage Capacity with required attributes

Request to find out available capacity.

GET /capacity/storage/query

Example request:

GET /capacity/storage/query HTTP/1.1
Accept: application/json

{
    "resourceCriteria": {
        "typeOfStorage" : "volume",
        "sizeOfStorage" : 200,
        "rdmaSupported" : "True",
    },
    "attributeSelector": "available",
    "timePeriod":  {
        "startTime": "2015-09-28T12:00:00Z",
        "stopTime": "2015-09-28T12:04:45Z"
    }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "lastUpdate": "2015-09-28T12:01:35Z",
    "capacityInformation": {
        "available": 2
    }
}
7.4.1.4. Notify Storage Capacity Change Event

Notification about storage capacity changes

POST /capacity/storage/notification

Example notification:

 Content-Type: application/json

 {
     "notificationId": "xwvuts-rqponm-765432109",
     "capacityChangeTime": "2015-09-28T12:01:35Z",
     "resourceDescriptor": {
         "storageResourceTypeId": "volumes"
     },
     "capacityInformation": {
         "available": 2,
         "reserved": 4,
         "total": 10,
         "allocated": 4
     }
}
7.4.2. Storage Resource Reservation
7.4.2.1. Create Storage Resource Reservation

Request the reservation of storage resource capacity

POST /reservation/storage/create

Example request:

POST /reservation/storage/create HTTP/1.1
Accept: application/json

{
    "startTime": "2015-09-28T13:00:00Z",
    "storagePoolReservation": {
        "storageSize": 10,
        "numSnapshots": 3,
        "numVolumes": 2
    }
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
    "reservationData": {
        "startTime": "2015-09-28T13:00:00Z",
        "reservationStatus": "initialized",
        "reservationId": "vvvv-wwww-xxxx",
        "storagePoolReserved": {
            "storageSize": 10,
            "numSnapshots": 3,
            "numVolumes": 2
        }
    }
}
7.4.2.2. Query Storage Resource Reservation

Request to find out about reserved storage resources that the consumer has access to.

GET /reservation/storage/query

Example request:

GET /reservation/storage/query HTTP/1.1
Accept: application/json

{
    "queryReservationFilter": [
        {
            "reservationId": "vvvv-wwww-xxxx"
        }
    ]
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
    "queryResult": {
        "startTime": "2015-09-28T13:00:00Z",
        "reservationStatus": "active",
        "reservationId": "vvvv-wwww-xxxx",
        "storagePoolReserved": {
            "storageSize": 10,
            "numSnapshots": 3,
            "numVolumes": 2
        }
    }
}
7.4.2.3. Update Storage Resource Reservation

Request to update storage resource reservation

POST /reservation/storage/update

Example request:

POST /reservation/storage/update HTTP/1.1
Accept: application/json


{
    "startTime": "2015-09-20T23:00:00Z",
    "reservationId": "vvvv-wwww-xxxx"

}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
    "reservationData": {
        "startTime": "2015-09-20T23:00:00Z",
        "reservationStatus": "active",
        "reservationId": "vvvv-wwww-xxxx",
        "storagePoolReserved": {
            "storageSize": 10,
            "numSnapshots": 3,
            "numVolumes": 2
        }
    }
}
7.4.2.4. Terminate Storage Resource Reservation

Request to terminate a storage resource reservation

DELETE /reservation/storage/(reservation_id)

Example response:

HTTP/1.1 200
Content-Type: application/json

{
   "reservationId": "xxxx-yyyy-zzzz",
}
8. Summary and conclusion

Resource Reservation and Resource Capacity Management are features to be supported by the VIM and exposed to the consumer via the VIM NBI. These features have been specified by ETSI NFV.

This document has described several use cases and corresponding high level flows where Resource Reservation and Capacity Management are of great benefit for the consumer of the virtualised resource management interface: the NFVO or the VNFM. The use cases include:

  • Notification of changes in capacity in the NFVI
  • Query of available resource capacity
  • Reservation of a resource or set of resources for immediate use
  • Reservation of a resource or set of resources for future use

The Promise project has performed a gap analysis in order to fulfill the required functionality. Based on the gap analysis an implementation plan an architecture, information model and northbound interface has been specified.

9. References and bibliography
[PROMISE]OPNFV, “Promise” requirements project, [Online]. Available at https://wiki.opnfv.org/promise
[BLAZAR]OpenStack Blazar Project, [Online]. Available at https://wiki.openstack.org/wiki/Blazar
[PROMOSS]Promise reference implementation, [Online]. Available at https://github.com/opnfv/promise
[YANGFO]Yangforge Project, [Online]. Available at https://github.com/opnfv/yangforge
[NFVMAN]ETSI GS NFV MAN 001, “Network Function Virtualisation (NFV); Management and Orchestration”
[NFV003]ETSI GS NFV 003, “Network Function Virtualisation (NFV); Terminology for Main Concepts in NFV”
[NFVIFA010]ETSI GS NFV IFA 010, “Network Function Virtualisation (NFV); Management and Orchestration; Functional Requirements Specification”
[NFVIFA005]ETSI GS NFV IFA 005, “Network Function Virtualisation (NFV); Management and Orchestration; Or-Vi reference point - Interface and Information Model Specification”
[NFVIFA006]ETSI GS NFV IFA 006 “Network Function Virtualisation (NFV); Management and Orchestration; Vi-Vnfm reference point - Interface and Information Model Specification”
[ETSINFV]ETSI NFV, [Online]. Available at http://www.etsi.org/technologies-clusters/technologies/nfv
10. ANNEX A: Use case for OPNFV Brahmaputra

A basic resource reservation use case to be realized for OPNFV B-release may look as follows:

  • Step 0: Shim-layer is monitoring/querying available capacity at NFVI
    • Step 0a: Cloud operator creates a new OpenStack tenant user and updates quota values for this user
    • Step 0b: The tenant user is creating and instantiating a simple VNF (e.g. 1 network, 2 VMs)
    • Step 0c: OpenStack is notifying shim-layer about capacity change for this new tenant user
    • Step 0d: Cloud operator can visualize the changes using the GUI
  • Step 1: Consumer(NFVO) is sending a reservation request for future use to shim-layer
  • Step 2: Shim-layer is checking for available capacity at the given time window
  • Step 3: Shim-layer is responding with reservation identifier
  • Step 4 (optional): Consumer(NFVO) is sending an update reservation request to shim-layer (startTime set to now) -> continue with Steps 2 and 3.
  • Step 5: Consumer(VNFM) is requesting the allocation of virtualised resources using the reservation identifier in Step 3
11. ANNEX B: Promise YANG schema based on YangForge
module opnfv-promise {
namespace "urn:opnfv:promise";
prefix promise;

import complex-types { prefix ct; }
import ietf-yang-types { prefix yang; }
import ietf-inet-types { prefix inet; }
import access-control-models { prefix acm; }
import nfv-infrastructure { prefix nfvi; }
import nfv-mano { prefix mano; }

description
  "OPNFV Promise Resource Reservation/Allocation controller module";

revision 2015-10-05 {
  description "Complete coverage of reservation related intents";
}

revision 2015-08-06 {
  description "Updated to incorporate YangForge framework";
}

revision 2015-04-16 {
  description "Initial revision.";
}

feature reservation-service {
  description "When enabled, provides resource reservation service";
}

feature multi-provider {
  description "When enabled, provides resource management across multiple providers";
}

typedef reference-identifier {
  description "defines valid formats for external reference id";
  type union {
    type yang:uuid;
    type inet:uri;
    type uint32;
  }
}

grouping resource-utilization {
  container capacity {
    container total     { description 'Conceptual container that should be extended'; }
    container reserved  { description 'Conceptual container that should be extended';
                          config false; }
    container usage     { description 'Conceptual container that should be extended';
                          config false; }
    container available { description 'Conceptual container that should be extended';
                          config false; }
  }
}

grouping temporal-resource-collection {
  description
    "Information model capturing resource-collection with start/end time window";

  leaf start { type yang:date-and-time; }
  leaf end   { type yang:date-and-time; }

  uses nfvi:resource-collection;
}

grouping resource-usage-request {
  description
    "Information model capturing available parameters to make a resource
     usage request.";
  reference "OPNFV-PROMISE, Section 3.4.1";

  uses temporal-resource-collection {
    refine elements {
      description
        "Reference to a list of 'pre-existing' resource elements that are
       required for fulfillment of the resource-usage-request.

       It can contain any instance derived from ResourceElement,
       such as ServerInstances or even other
       ResourceReservations. If the resource-usage-request is
       accepted, the ResourceElement(s) listed here will be placed
       into 'protected' mode as to prevent accidental removal.

       If any of these resource elements become 'unavailable' due to
       environmental or administrative activity, a notification will
       be issued informing of the issue.";
    }
  }

  leaf zone {
    description "Optional identifier to an Availability Zone";
    type instance-identifier { ct:instance-type nfvi:AvailabilityZone; }
  }
}

grouping query-start-end-window {
  container window {
    description "Matches entries that are within the specified start/end time window";
    leaf start { type yang:date-and-time; }
    leaf end   { type yang:date-and-time; }
    leaf scope {
      type enumeration {
        enum "exclusive" {
          description "Matches entries that start AND end within the window";
        }
        enum "inclusive" {
          description "Matches entries that start OR end within the window";
        }
      }
      default "inclusive";
    }
  }
}

grouping query-resource-collection {
  uses query-start-end-window {
    description "Match for ResourceCollection(s) that are within the specified
                 start/end time window";
  }
  leaf-list without {
    description "Excludes specified collection identifiers from the result";
    type instance-identifier { ct:instance-type ResourceCollection; }
  }
  leaf show-utilization { type boolean; default true; }
  container elements {
    leaf-list some {
      description "Query for ResourceCollection(s) that contain some or more of
                   these element(s)";
      type instance-identifier { ct:instance-type nfvi:ResourceElement; }
    }
    leaf-list every {
      description "Query for ResourceCollection(s) that contain all of
                   these element(s)";
      type instance-identifier { ct:instance-type nfvi:ResourceElement; }
    }
  }
}

grouping common-intent-output {
  leaf result {
    type enumeration {
      enum "ok";
      enum "conflict";
      enum "error";
    }
  }
  leaf message { type string; }
}

grouping utilization-output {
  list utilization {
    key 'timestamp';
    leaf timestamp { type yang:date-and-time; }
    leaf count { type int16; }
    container capacity { uses nfvi:resource-capacity; }
  }
}

ct:complex-type ResourceCollection {
  ct:extends nfvi:ResourceContainer;
  ct:abstract true;

  description
    "Describes an abstract ResourceCollection data model, which represents
     a grouping of capacity and elements available during a given
     window in time which must be extended by other resource
     collection related models";

  leaf start { type yang:date-and-time; }
  leaf end   { type yang:date-and-time; }

  leaf active {
    config false;
    description
      "Provides current state of this record whether it is enabled and within
       specified start/end time";
    type boolean;
  }
}

ct:complex-type ResourcePool {
  ct:extends ResourceCollection;

  description
    "Describes an instance of an active ResourcePool record, which
     represents total available capacity and elements from a given
     source.";

  leaf source {
    type instance-identifier {
      ct:instance-type nfvi:ResourceContainer;
      require-instance true;
    }
    mandatory true;
  }

  refine elements {
    // following 'must' statement applies to each element
    // NOTE: just a non-working example for now...
    must "boolean(/source/elements/*[@id=id])" {
      error-message "One or more of the ResourceElement(s) does not exist in
                     the provider to be reserved";
    }
  }
}

ct:complex-type ResourceReservation {
  ct:extends ResourceCollection;

  description
    "Describes an instance of an accepted resource reservation request,
     created usually as a result of 'create-reservation' request.

     A ResourceReservation is a derived instance of a generic
     ResourceCollection which has additional parameters to map the
     pool(s) that were referenced to accept this reservation as well
     as to track allocations made referencing this reservation.

     Contains the capacities of various resource attributes being
     reserved along with any resource elements that are needed to be
     available at the time of allocation(s).";

  reference "OPNFV-PROMISE, Section 3.4.1";

  leaf created-on  { type yang:date-and-time; config false; }
  leaf modified-on { type yang:date-and-time; config false; }

  leaf-list pools {
    config false;
    description
      "Provides list of one or more pools that were referenced for providing
       the requested resources for this reservation.  This is an
       important parameter for informing how/where allocation
       requests can be issued using this reservation since it is
       likely that the total reserved resource capacity/elements are
       made availble from multiple sources.";
    type instance-identifier {
      ct:instance-type ResourcePool;
      require-instance true;
    }
  }

  container remaining {
    config false;
    description
      "Provides visibility into total remaining capacity for this
       reservation based on allocations that took effect utilizing
       this reservation ID as a reference.";

    uses nfvi:resource-capacity;
  }

  leaf-list allocations {
    config false;
    description
      "Reference to a collection of consumed allocations referencing
       this reservation.";
    type instance-identifier {
      ct:instance-type ResourceAllocation;
      require-instance true;
    }
  }
}

ct:complex-type ResourceAllocation {
  ct:extends ResourceCollection;

  description
    "A ResourceAllocation record denotes consumption of resources from a
     referenced ResourcePool.

     It does not reflect an accepted request but is created to
     represent the actual state about the ResourcePool. It is
     created once the allocation(s) have successfully taken effect
     on the 'source' of the ResourcePool.

     The 'priority' state indicates the classification for dealing
     with resource starvation scenarios. Lower priority allocations
     will be forcefully terminated to allow for higher priority
     allocations to be fulfilled.

     Allocations without reference to an existing reservation will
     receive the lowest priority.";

  reference "OPNFV-PROMISE, Section 3.4.3";

  leaf reservation {
    description "Reference to an existing reservation identifier (optional)";

    type instance-identifier {
      ct:instance-type ResourceReservation;
      require-instance true;
    }
  }

  leaf pool {
    description "Reference to an existing resource pool from which allocation is drawn";

    type instance-identifier {
      ct:instance-type ResourcePool;
      require-instance true;
    }
  }

  container instance-ref {
    config false;
    description
      "Reference to actual instance identifier of the provider/server
      for this allocation";
    leaf provider {
      type instance-identifier { ct:instance-type ResourceProvider; }
    }
    leaf server { type yang:uuid; }
  }

  leaf priority {
    config false;
    description
      "Reflects current priority level of the allocation according to
       classification rules";
    type enumeration {
      enum "high"   { value 1; }
      enum "normal" { value 2; }
      enum "low"    { value 3; }
    }
    default "normal";
  }
}

ct:complex-type ResourceProvider {
  ct:extends nfvi:ResourceContainer;

  key "name";
  leaf token { type string; mandatory true; }

  container services { // read-only
    config false;
    container compute {
      leaf endpoint { type inet:uri; }
      ct:instance-list flavors { ct:instance-type nfvi:ComputeFlavor; }
    }
  }

  leaf-list pools {
    config false;
    description
      "Provides list of one or more pools that are referencing this provider.";

    type instance-identifier {
      ct:instance-type ResourcePool;
      require-instance true;
    }
  }
}

// MAIN CONTAINER
container promise {

  uses resource-utilization {
    description "Describes current state info about capacity utilization info";

    augment "capacity/total"     { uses nfvi:resource-capacity; }
    augment "capacity/reserved"  { uses nfvi:resource-capacity; }
    augment "capacity/usage"     { uses nfvi:resource-capacity; }
    augment "capacity/available" { uses nfvi:resource-capacity; }
  }

  ct:instance-list providers {
    if-feature multi-provider;
    description "Aggregate collection of all registered ResourceProvider instances
                 for Promise resource management service";
    ct:instance-type ResourceProvider;
  }

  ct:instance-list pools {
    if-feature reservation-service;
    description "Aggregate collection of all ResourcePool instances";
    ct:instance-type ResourcePool;
  }

  ct:instance-list reservations {
    if-feature reservation-service;
    description "Aggregate collection of all ResourceReservation instances";
    ct:instance-type ResourceReservation;
  }

  ct:instance-list allocations {
    description "Aggregate collection of all ResourceAllocation instances";
    ct:instance-type ResourceAllocation;
  }

  container policy {
    container reservation {
      leaf max-future-start-range {
        description
          "Enforce reservation request 'start' time is within allowed range from now";
        type uint16 { range 0..365; }
        units "days";
      }
      leaf max-future-end-range {
        description
          "Enforce reservation request 'end' time is within allowed range from now";
        type uint16 { range 0..365; }
        units "days";
      }
      leaf max-duration {
        description
          "Enforce reservation duration (end-start) does not exceed specified threshold";
        type uint16;
        units "hours";
        default 8760; // for now cap it at max one year as default
      }
      leaf expiry {
        description
          "Duration in minutes from start when unallocated reserved resources
           will be released back into the pool";
        type uint32;
        units "minutes";
      }
    }
  }
}

//-------------------
// INTENT INTERFACE
//-------------------

// RESERVATION INTENTS
rpc create-reservation {
  if-feature reservation-service;
  description "Make a request to the reservation system to reserve resources";
  input {
    uses resource-usage-request;
  }
  output {
    uses common-intent-output;
    leaf reservation-id {
      type instance-identifier { ct:instance-type ResourceReservation; }
    }
  }
}

rpc update-reservation {
  description "Update reservation details for an existing reservation";
  input {
    leaf reservation-id {
      type instance-identifier {
        ct:instance-type ResourceReservation;
        require-instance true;
      }
      mandatory true;
    }
    uses resource-usage-request;
  }
  output {
    uses common-intent-output;
  }
}

rpc cancel-reservation {
  description "Cancel the reservation and be a good steward";
  input {
    leaf reservation-id {
      type instance-identifier { ct:instance-type ResourceReservation; }
      mandatory true;
    }
  }
  output {
    uses common-intent-output;
  }
}

rpc query-reservation {
  if-feature reservation-service;
  description "Query the reservation system to return matching reservation(s)";
  input {
    leaf zone { type instance-identifier { ct:instance-type nfvi:AvailabilityZone; } }
    uses query-resource-collection;
  }
  output {
    leaf-list reservations { type instance-identifier
                             { ct:instance-type ResourceReservation; } }
    uses utilization-output;
  }
}

// CAPACITY INTENTS
rpc increase-capacity {
  description "Increase total capacity for the reservation system
               between a window in time";
  input {
    uses temporal-resource-collection;
    leaf source {
      type instance-identifier {
        ct:instance-type nfvi:ResourceContainer;
      }
    }
  }
  output {
    uses common-intent-output;
    leaf pool-id {
      type instance-identifier { ct:instance-type ResourcePool; }
    }
  }
}

rpc decrease-capacity {
  description "Decrease total capacity for the reservation system
               between a window in time";
  input {
    uses temporal-resource-collection;
    leaf source {
      type instance-identifier {
        ct:instance-type nfvi:ResourceContainer;
      }
    }
  }
  output {
    uses common-intent-output;
    leaf pool-id {
      type instance-identifier { ct:instance-type ResourcePool; }
    }
  }
}

rpc query-capacity {
  description "Check available capacity information about a specified
               resource collection";
  input {
    leaf capacity {
      type enumeration {
        enum 'total';
        enum 'reserved';
        enum 'usage';
        enum 'available';
      }
      default 'available';
    }
    leaf zone { type instance-identifier { ct:instance-type nfvi:AvailabilityZone; } }
    uses query-resource-collection;
    // TBD: additional parameters for query-capacity
  }
  output {
    leaf-list collections { type instance-identifier
                            { ct:instance-type ResourceCollection; } }
    uses utilization-output;
  }
}

// ALLOCATION INTENTS (should go into VIM module in the future)
rpc create-instance {
  description "Create an instance of specified resource(s) utilizing capacity
               from the pool";
  input {
    leaf provider-id {
      if-feature multi-provider;
      type instance-identifier { ct:instance-type ResourceProvider;
                                 require-instance true; }
    }
    leaf name   { type string; mandatory true; }
    leaf image  {
      type reference-identifier;
      mandatory true;
    }
    leaf flavor {
      type reference-identifier;
      mandatory true;
    }
    leaf-list networks {
      type reference-identifier;
      description "optional, will assign default network if not provided";
    }

    // TODO: consider supporting a template-id (such as HEAT) for more complex instantiation

    leaf reservation-id {
      type instance-identifier { ct:instance-type ResourceReservation;
                                 require-instance true; }
    }
  }
  output {
    uses common-intent-output;
    leaf instance-id {
      type instance-identifier { ct:instance-type ResourceAllocation; }
    }
  }
}

rpc destroy-instance {
  description "Destroy an instance of resource utilization and release it
               back to the pool";
  input {
    leaf instance-id {
      type instance-identifier { ct:instance-type ResourceAllocation;
                                 require-instance true; }
    }
  }
  output {
    uses common-intent-output;
  }
}

// PROVIDER INTENTS (should go into VIM module in the future)
rpc add-provider {
  description "Register a new resource provider into reservation system";
  input {
    leaf provider-type {
      description "Select a specific resource provider type";
      mandatory true;
      type enumeration {
        enum openstack;
        enum hp;
        enum rackspace;
        enum amazon {
          status planned;
        }
        enum joyent {
          status planned;
        }
        enum azure {
          status planned;
        }
      }
      default openstack;
    }
    uses mano:provider-credentials {
      refine endpoint {
        default "http://localhost:5000/v2.0/tokens";
      }
    }
    container tenant {
      leaf id { type string; }
      leaf name { type string; }
    }
  }
  output {
    uses common-intent-output;
    leaf provider-id {
      type instance-identifier { ct:instance-type ResourceProvider; }
    }
  }
}

// TODO...
notification reservation-event;
notification capacity-event;
notification allocation-event;
}
12. ANNEX C: Supported APIS
12.1. Add Provider

Register a new resource provider (e.g. OpenStack) into reservation system.

Request parameters

Name Type Description
provider-type Enumeration Name of the resource provider
endpoint URI Targer URL end point for the resource provider
username String User name
password String Password
region String Specified region for the provider
tenant.id String Id of the tenant
tenant.name String Name of the tenant

Response parameters

Name Type Description
provider-id String Id of the new resource provider
result Enumeration Result info
POST /add-provider

Example request:

POST /add-provider HTTP/1.1
Accept: application/json

{
  "provider-type": "openstack",
  "endpoint": "http://10.0.2.15:5000/v2.0/tokens",
  "username": "promise_user",
  "password": "******",
  "tenant": {
     "name": "promise"
  }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "provider-id": "f25ed9cb-de57-43d5-9b4a-a389a1397302",
  "result": "ok"
}
12.2. Create Reservation

Make a request to the reservation system to reserve resources.

Request parameters

Name Type Description
zone String Id to an availability zone
start DateTime Timestamp when the consumption of reserved resources can begin
end DateTime Timestamp when the consumption of reserved resources should end
capacity.cores int16 Amount of cores to be reserved
capacity.ram int32 Amount of RAM to be reserved
capacity.instances int16 Amount of instances to be reserved
capacity.addresses int32 Amount of public IP addresses to be reserved
elements ResourceElement List of pre-existing resource elements to be reserved

Response parameters

Name Type Description
reservation-id String Id of the reservation
result Enumeration Result info
message String Output message
POST /create-reservation

Example request:

POST /create-reservation HTTP/1.1
Accept: application/json

{
   "capacity": {
      "cores": "5",
      "ram": "25600",
      "addresses": "3",
      "instances": "3"
   },
   "start": "2016-02-02T00:00:00Z",
   "end": "2016-02-03T00:00:00Z"
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "reservation-id": "269b2944-9efc-41e0-b067-6898221e8619",
  "result": "ok",
  "message": "reservation request accepted"
}
12.3. Update Reservation

Update reservation details for an existing reservation.

Request parameters

Name Type Description
reservation-id String Id of the reservation to be updated
zone String Id to an availability zone
start DateTime Updated timestamp when the consumption of reserved resources can begin
end DateTime Updated timestamp when the consumption of reserved resources should end
capacity.cores int16 Updated amount of cores to be reserved
capacity.ram int32 Updated amount of RAM to be reserved
capacity.instances int16 Updated amount of instances to be reserved
capacity.addresses int32 Updated amount of public IP addresses to be reserved
elements ResourceElement Updated list of pre-existing resource elements to be reserved

Response parameters

Name Type Description
result Enumeration Result info
message String Output message
POST /update-reservation

Example request:

POST /update-reservation HTTP/1.1
Accept: application/json

{
   "reservation-id": "269b2944-9efv-41e0-b067-6898221e8619",
   "capacity": {
      "cores": "1",
      "ram": "5120",
      "addresses": "1",
      "instances": "1"
   }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "result": "ok",
  "message": "reservation update successful"
}
12.4. Cancel Reservation

Cancel the reservation.

Request parameters

Name Type Description
reservation-id String Id of the reservation to be canceled

Response parameters

Name Type Description
result Enumeration Result info
message String Output message
POST /cancel-reservation

Example request:

POST /cancel-reservation HTTP/1.1
Accept: application/json

{
  "reservation-id": "269b2944-9efv-41e0-b067-6898221e8619"
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "result": "ok",
  "message": "reservation canceled"
}
12.5. Query Reservation

Query the reservation system to return matching reservation(s).

Request parameters

Name Type Description
zone String Id to an availability zone
show-utilization Boolean Show capacity utilization
without ResourceCollection Excludes specified collection identifiers from the result
elements.some ResourceElement Query for ResourceCollection(s) that contain some or more of these element(s)
elements.every ResourceElement Query for ResourceCollection(s) that contain all of these element(s)
window.start DateTime Matches entries that are within the specified start/end window
window.end DateTime  
wndow.scope Enumeration Matches entries that start {and/or} end within the time window

Response parameters

Name Type Description
reservations ResourceReservation List of matching reservations
utilization CapacityUtilization Capacity utilization over time
POST /query-reservation

Example request:

POST /query-reservation HTTP/1.1
Accept: application/json

{
   "show-utilization": false,
   "window": {
      "start": "2016-02-01T00:00:00Z",
      "end": "2016-02-04T00:00:00Z"
   }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "reservations": [
    "269b2944-9efv-41e0-b067-6898221e8619"
  ],
  "utilization": []
}
12.6. Create Instance

Create an instance of specified resource(s) utilizing capacity from the pool.

Request parameters

Name Type Description
provider-id String Id of the resource provider
reservation-id String Id of the resource reservation
name String Name of the instance
image String Id of the image
flavor String Id of the flavor
networks Uuid List of network uuids

Response parameters

Name Type Description
instance-id String Id of the instance
result Enumeration Result info
message String Output message
POST /create-instance

Example request:

POST /create-instance HTTP/1.1
Accept: application/json

{
  "provider-id": "f25ed9cb-de57-43d5-9b4a-a389a1397302",
  "name": "vm1",
  "image": "ddffc6f5-5c86-4126-b0fb-2c71678633f8",
  "flavor": "91bfdf57-863b-4b73-9d93-fc311894b902"
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "instance-id": "82572779-896b-493f-92f6-a63008868250",
  "result": "ok",
  "message": "created-instance request accepted"
}
12.7. Destroy Instance

Destroy an instance of resource utilization and release it back to the pool.

Request parameters

Name Type Description
instance-id String Id of the instance to be destroyed

Response parameters

Name Type Description
result Enumeration Result info
message String Output message
POST /destroy-instance

Example request:

POST /destroy-instance HTTP/1.1
Accept: application/json

{
   "instance-id": "82572779-896b-493f-92f6-a63008868250"
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "result": "ok",
  "message": "instance destroyed and resource released back to pool"
}
12.8. Decrease Capacity

Decrease total capacity for the reservation system for a given time window.

Request parameters

Name Type Description
source String Id of the resource container
start DateTime Start/end defines the time window when total capacity is decreased
end DateTime  
capacity.cores int16 Decreased amount of cores
capacity.ram int32 Decreased amount of RAM
capacity.instances int16 Decreased amount of instances
capacity.addresses int32 Decreased amount of public IP addresses

Response parameters

Name Type Description
pool-id String Id of the resource pool
result Enumeration Result info
message String Output message
POST /decrease-capacity

Example request:

POST /decrease-capacity HTTP/1.1
Accept: application/json

{
   "source": "ResourcePool:4085f0da-8030-4252-a0ff-c6f93870eb5f",
   "capacity": {
      "cores": "3",
      "ram": "5120",
      "addresses": "1"
   }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
   "pool-id": "c63b2a41-bcc6-42f6-8254-89d633e1bd0b",
   "result": "ok",
   "message": "capacity decrease successful"
}
12.9. Increase Capacity

Increase total capacity for the reservation system for a given time window.

Request parameters

Name Type Description
source String Id of the resource container
start DateTime Start/end defines the time window when total capacity is increased
end DateTime  
capacity.cores int16 Increased amount of cores
capacity.ram int32 Increased amount of RAM
capacity.instances int16 Increased amount of instances
capacity.addresses int32 Increased amount of public IP addresses

Response parameters

Name Type Description
pool-id String Id of the resource pool
result Enumeration Result info
message String Output message
POST /increase-capacity

Example request:

POST /increase-capacity HTTP/1.1
Accept: application/json

{
   "source": "ResourceProvider:f6f13fe3-0126-4c6d-a84f-15f1ab685c4f",
   "capacity": {
       "cores": "20",
       "ram": "51200",
       "instances": "10",
       "addresses": "10"
   }
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
   "pool-id": "279217a4-7461-4176-bf9d-66770574ca6a",
   "result": "ok",
   "message": "capacity increase successful"
}
12.10. Query Capacity

Query for capacity information about a specified resource collection.

Request parameters

Name Type Description
capacity Enumeration Return total or reserved or available or usage capacity information
zone String Id to an availability zone
show-utilization Boolean Show capacity utilization
without ResourceCollection Excludes specified collection identifiers from the result
elements.some ResourceElement Query for ResourceCollection(s) that contain some or more of these element(s)
elements.every ResourceElement Query for ResourceCollection(s) that contain all of these element(s)
window.start DateTime Matches entries that are within the specified start/end window
window.end DateTime  
window.scope Enumeration Matches entries that start {and/or} end within the time window

Response parameters

Name Type Description
collections ResourceCollection List of matching collections
utilization CapacityUtilization Capacity utilization over time
POST /query-capacity

Example request:

POST /query-capacity HTTP/1.1
Accept: application/json

{
  "show-utilization": false
}

Example response:

HTTP/1.1 201 CREATED
Content-Type: application/json

{
  "collections": [
    "ResourcePool:279217a4-7461-4176-bf9d-66770574ca6a"
  ],
  "utilization": []
}
Promise installation and configuration
Promise installation

Install nodejs, npm and promise

curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm -g install npm@latest
git clone https://gerrit.opnfv.org/gerrit/promise
cd promise/source
npm install
npm ls

Please note that the last command ‘npm ls’ will list all needed dependencies for promise (including yangforge and mocha)

_images/screenshot_promise_install.png
Validation

Please perform the following preparation steps:

  1. Set OpenStack environment parameters properly (e.g. source openrc admin demo in DevStack)
  2. Create OpenStack project (e.g. promise) and user (e.g. myuser) in e.g. the default domain
  3. Create a flavor in Nova with 1 vCPU and 512 MB RAM
  4. Create a private network, subnet and router in Neutron
  5. Create an image in Glance

Once done, the promise test script can be invoked as follows:

export OS_PROJECT_NAME=promise
export OS_TENANT_NAME=promise
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USERNAME=myuser
export OS_USER_DOMAIN_NAME=Default
export OS_PASSWORD=<user password from Step 2>
export OS_TEST_FLAVOR=<flavor ID from Step 3>
export OS_TEST_NETWORK=<network ID from Step 4>
export OS_TEST_IMAGE=<image ID from Step 5>
npm run -s test -- --reporter json > promise-results.json

The results of the tests will be stored in the promise-results.json file.

The results can also be seen in the console (“npm run -s test”)

_images/screenshot_promise.png

All 33 tests passing?! Congratulations, Promise has been successfully installed and configured.

Promise user guide
Abstract

Promise is a resource reservation and management project to identify NFV related requirements and realize resource reservation for future usage by capacity management of resource pools regarding compute, network and storage.

The following are the key features provided by this module:

  • Capacity Management
  • Reservation Management
  • Allocation Management

The following sections provide details on the Promise capabilities and its API usage.

Promise capabilities and usage

The Euphrates implementation of Promise is built with the YangForge data modeling framework [2] , using a shim-layer on top of OpenStack to provide the Promise features. This approach requires communication between Consumers/Administrators and OpenStack to pass through the shim-layer. The shim-layer intercepts the message flow to manage the allocation requests based on existing reservations and available capacities in the providers. It also extracts information from the intercepted messages in order to update its internal databases. Furthermore, Promise provides additional intent-based APIs to allow a Consumer or Administrator to perform capacity management (i.e. add providers, update the capacity, and query the current capacity and utilization of a provider), reservation management (i.e. create, update, cancel, query reservations), and allocation management (i.e. create, destroy instances).

Detailed information about Promise use cases, features, interface specifications, work flows, and the underlying Promise YANG schema can be found in the Promise requirement document [1] .

Keystone v3 support has been added during the Euphrates release.

Promise features and API usage guidelines and examples

This section lists the Promise features and API implemented in OPNFV Euphrates.

Note: The listed parameters are optional unless explicitly marked as “mandatory”.

Reservation management

The reservation management allows a Consumer to request reservations for resource capacity. Reservations can be for now or a later time window. After the start time of a reservation has arrived, the Consumer can issue create server instance requests against the reserved capacity. Note, a reservation will expire after a predefined expiry time in case no allocation referring to the reservation is requested.

The implemented workflow is well aligned with the described workflow in the Promise requirement document [1] (Section 6.1) except for the “multi-provider” scenario as described in (Multi-)provider management .

create-reservation

This operation allows making a request to the reservation system to reserve resources.

The operation takes the following input parameters:

  • start: start time of the requested reservation
  • end: end time of the requested reservation
  • capacity.instances: amount of instances to be reserved
  • capacity.cores: amount of cores to be reserved
  • capacity.ram: amount of ram in MB to be reserved

Promise will check the available capacity in the given time window and in case sufficient capacity exists to meet the reservation request, will mark those resources “reserved” in its reservation map.

update-reservation

This operation allows to update the reservation details for an existing reservation.

It can take the same input parameters as in create-reservation but in addition requires a mandatory reference to the reservation-id of the reservation that shall be updated.

cancel-reservation

This operation is used to cancel an existing reservation.

The operation takes the following input parameter:

  • reservation-id (mandatory): identifier of the reservation to be canceled.
query-reservation

The operation queries the reservation system to return reservation(s) matching the specified query filter, e.g., reservations that are within a specified start/end time window.

The operation takes the following input parameters to narrow down the query results:

  • without: excludes specified collection identifiers from the result

  • elements.some: query for ResourceCollection(s) that contain some or more of these element(s)

  • elements.every: query for ResourceCollection(s) that contain all of these element(s)

  • window.start: matches entries that are within the specified start/

  • window.end: end time window

  • window.scope: if set to ‘exclusive’, only reservations with start AND end time

    within the time window are returned. Otherwise (‘inclusive’), all reservation starting OR ending in the time windows are returned.

  • show-utilization: boolean value that specifies whether to also return the resource utilization in the queried time window or not

Allocation management
create-instance

This operation is used to create an instance of specified resource(s) for immediate use utilizing capacity from the pool. Create-instance requests can be issued against an existing reservation, but also allocations without a reference to an existing reservation are allowed. In case the allocation request specifies a reservation identifier, Promise checks if a reservation with that ID exists, the reservation start time has arrived (i.e. the reservation is ‘active’), and the required capacity for the requested flavor is within the available capacity of the reservation. If those conditions are met, Promise creates a record for the allocation (VMState=”INITIALIZED”) and update its databases. If no reservation_id was provided in the allocation request, Promise checks whether the required capacity to meet the request can be provided from the available, non-reserved capacity. If yes, Promise creates a record for the allocation with an unique instance-id and update its databases. In any other case, Promise rejects the create-instance request.

In case the create-instance request is rejected, Promise responds with a “status=rejected” providing the reason of the rejection. This will help the Consumer to take appropriate actions, e.g., send an updated create-instance request. In case the create-instance request was accepted and a related allocation record has been created, the shim-layer issues a createServer request to the VIM Controller (i.e. Nova) providing all information to create the server instance.

The operation takes the following input parameters:

  • name (mandatory): Assigned name for the instance to be created
  • image (mandatory): the image to be booted in the new instance
  • flavor (mandatory): the flavor of the requested server instance
  • networks: the list of network uuids of the requested server instance
  • provider-id: identifier of the provider where the instance shall be created
  • reservation-id: identifier of a resource reservation the create-instance

The Euphrates implementation of Promise has the following limitations:

  • All create server instance requests shall pass through the Promise shim-layer such that Promise can keep track of all allocation requests. This is necessary as in the current release the sychronization between the VIM Controller and Promise on the available capacity is not yet implemented.
  • Create-allocation requests are limited to “simple” allocations, i.e., the current workflow only supports the Nova compute service and create-allocation requests are limited to creating one server instance at a time
  • Prioritization of reservations and allocations is yet not implemented. Future version may allow certain policy-based conflict resolution where, e.g., new allocation request with high priority can “forcefully” terminate lower priority allocations.
destroy-instance

This operation request to destroy an existing server instance and release it back to the pool.

The operation takes the following input parameter:

  • instance-id: identifier of the server instance to be destroyed
Capacity management

The capacity management feature allows the Consumer or Administrator to do capacity planning, i.e. the capacity available to the reservation management can differ from the actual capacity in the registered provider(s). This feature can, e.g., be used to limit the available capacity for a given time window due to a planned downtime of some of the resources, or increase the capacity available to the reservation system in case of a planned upgrade of the available capacity.

increase/decrease-capacity

This operations allows to increase/decrease the total capacity that is made available to the Promise reservation service between a specified window in time. It does NOT increase the actual capacity of a given resource provider, but is used for capacity management inside Promise.

This feature can be used in different ways, like

  • Limit the capacity available to the reservation system to a value below 100% of the available capacity in the VIM, e.g., in order to leave “buffer” in the actual NFVI to be used outside the Promise reservation service.
  • Inform the reservation system that, from a given time in the future, additional resources can be reserved, e.g., due to a planned upgrade of the available capacity of the provider.
  • Similarily, the “decrease-capacity” can be used to reduce the consumable resources in a given time window, e.g., to prepare for a planned downtime of some of the resources.
  • Expose multiple reservation service instances to different consumers sharing the same resource provider.

The operation takes the following input parameters:

  • start: start time for the increased/decreased capacity
  • end: end time for the increased/decreased capacity
  • capacity.cores: Increased/decreased amount of cores
  • capacity.ram: Increased/decreased amount of RAM
  • capacity.instances: Increased/decreased amount of instances

Note, increase/decreasing the capacity in Promise is completely transparent to the VIM. As such, when increasing the virtual capacity in Promise (e.g. for a planned upgrade of the capacity), it is in the responsibility of the Consumer/Administrator to ensure sufficient resources in the VIM are available at the appropriate time, in order to prevent allocations against reservations to fail due to a lack of resources. Therefore, this operations should only be used carefully.

query-capacity

This operation is used to query the available capacity information of the specified resource collection. A filter attribute can be specified to narrow down the query results.

The current implementation supports the following filter criteria:

  • time window: returns reservations matching the specified window
  • window scope: if set to ‘exclusive’, only reservations with start AND end time within the time window are returned. Otherwise, all reservation starting OR ending in the time windows are returned.
  • metric: query for one of the following capacity metrics:
    • ‘total’: resource pools
    • ‘reserved’: reserved resources
    • ‘usage’: resource allocations
    • ‘available’: remaining capacity, i.e. neither reserved nor allocated
(Multi-)provider management

This API towards OpenStack allows a Consumer/Administrator to add and remove resource providers to Promise. Note, Promise supports a multi-provider configuration, however, for Euphrates, multi-provider support is not yet fully supported.

add-provider

This operation is used to register a new resource provider into the Promise reservation system.

Note, for Euphrates, the add-provider operation should only be used to register one provider with the Promise shim-layer. Further note that currently only OpenStack is supported as a provider.

The operation takes the following input parameters:

  • provider-type = ‘openstack’: select a specific resource provider type.
  • endpoint : target URL endpoint for the resource provider.
  • username : name of the user
  • password : user password
  • user-domain-name : domain name of the user
  • project.name : name of the OpenStack project
  • project.domain-name : domain name of the OpenStack project
[1](1, 2) Promise requirement document, http://artifacts.opnfv.org/promise/docs/development_requirements/index.html
[2]YangForge framework, http://github.com/opnfv/yangforge
API / interface mapping ETSI NFV vs. OpenStack Blazar
Resource reservation mapping tables

The following tables are an attempt to map the resource reservation APIs / interfaces between ETSI NFV-IFA005 and Blazar for OpenStack.

I. Create Compute Resource Reservation
I.a) Create compute pool/resources reservation REQUEST
ETSI NFV IFA005 Blazar Description Comment
- name Name of the lease/reservation.  
computePoolReservation [1]: ComputePoolReservation reservations: resource_type = ‘virtual:instance’   Instance reservation is only available from OpenStack Pike release.
  -   amount Amount of virtual instances of a given flavour to be reserved.  
  numCpuCores [1]: Integer   vcpus Number of vCPU cores to be reserved.  
  numVcInstances [1]: Integer   - Number of virtualized container instances to be reserved (without explicitly reserved specific virtual containers).  
  virtualMemSize [1]: Number   memory_mb Size of virtual memory to be reserved.  
  -   disk_gb Size of (disk) storage to be reserved.  
computeAttributes [0..1]:
VirtualComputeAttributes
ReservationData
  - Information specifying additional attributes of the compute resource to be reserved. Not yet available in Pike release. Blueprint available in [1]
affinityConstraint [0..N]: AffinityOrAnti AffinityConstraint   affinity: Boolean Element with (anti-)affinity information of the virtualised compute resources to reserve. For the resource reservation at resource pool granularity level, it defines the (anti-)affinity information of the virtual compute pool resources to reserve. For the resource reservation at virtual container granularity level, it defines the (anti-)affinity information of the virtualisation container(s) to reserve. Only boolean (anti-)affinity rule in Pike release. Blueprint available in [2]
antiAffinityConstraint [0..N]: AffinityOrAnti AffinityConstraint
startTime [0..1]: TimeStamp start_date [0..1]: DateTime Timestamp indicating the earliest time to start the consumption of the resources.

Resources are reserved for immediate use if:

  • (Blazar) the start_date parameter is omitted
  • (ETSI NFV IFA005) the startTime value is 0
endTime [0..1]: TimeStamp end_date [0..1]: DateTime Timestamp indicating the end time of the reservation (when the issuer of the request expects that the resources will no longer be needed) and used by the VIM to schedule the reservation.
  • (ETSI NFV IFA005) If the attribute is not present, resources are reserved for unlimited usage time.
  • (Blazar) If the parameter is not present, resources are reserved for 24h after the start_date.
expiryTime [0..1]: TimeStamp - Timestamp indicating the time the VIM can release the reservation in case no allocation request against this reservation was made. Not yet available in Pike release.
- before_end_notification Timestamp indicating when the before_end action will be executed, e.g. take a snapshot of the resources of the lease.
  • (ETSI NFV IFA005) not specified
  • (Blazar) Not yet implemented for pool/ resources reservation, but only host reservations.
locationConstraints [0..1]: -tbd- - If present, it defines location constraints for the resource(s) is (are) requested to be reserved, e.g. in what particular Resource Zone. Not yet available in Pike release. Blueprint available in [3]
resourceGroupId [0..1]: Identifier project_id: Identifier Unique identifier of the “infrastructure resource group”, logical grouping of virtual resources assigned to a tenant within an Infrastructure Domain.  

Note

In Blazar reservations are encapsulated in leases, whereby one lease can have several reservations (i.e. reserved resources) with the same start and end dates.

Note

In the reservation system (e.g. Blazar) each lease/reservation has additional implementation level attributes not listed in the mapping tables in this document, e.g. lease_id, status, status reason, action, …, and which are not included in reservation requests.

I.b) Create virtualisation container reservation REQUEST
ETSI NFV IFA005 Blazar Description Comment
- name Name of the lease/reservation.  
  reservations    
    resource_type = ‘...’    
virtualisationContainerReservation [0..N]:     Virtualisation containers that need to be reserved (e.g. following a specific compute “flavour”).  
  VirtualisationContainerReservation        
  containerId [1]: Identifier     Identifier given to the compute flavour.  
  flavourId [1]: Identifier     The containerFlavour encapsulates information of the virtualisation container to be reserved.  
  accelerationCapability [0..N]: -tbd-     Selected acceleration capabilities (e.g. crypto, GPU) from the set of capabilities offered by the compute node acceleration resources. The cardinality can be 0, if no particular acceleration capability is requested.  
  virtualMemory [1]: VirtualMemoryData     Virtual memory of the virtualised compute.  
  virtualCpu [1]: VirtualCpuData     Virtual CPU(s) of the virtualised compute.  
  storageAttributes [0..N]: VirtualStorageData     Element containing information about the size of virtualised storage resource (e.g. size of volume, in GB), the type of storage (e.g., volume, object), and support for RDMA.  
  virtualNetworkInterface [0..N]: VirtualNetworkInterface     Virtual network interfaces of the virtualised compute.  
affinityConstraint [0..N]: AffinityOrAntiAffinityConstraint   affinity: Boolean Element with (anti-)affinity information of the virtualised compute resources to reserve. For the resource reservation at resource pool granularity level, it defines the (anti-)affinity information of the virtual compute pool resources to reserve. For the resource reservation at virtual container granularity level, it defines the (anti-)affinity information of the virtualisation container(s) to reserve. Affinity and AntiAffinity rules are not yet available in Pike release.
antiAffinityConstraint [0..N]: AffinityOrAntiAffinityConstraint
startTime [0..1]: TimeStamp start_date [0..1]: DateTime Timestamp indicating the earliest time to start the consumption of the resources.

Resources are reserved for immediate use if:

  • (Blazar) the start_date parameter is omitted
  • (ETSI NFV IFA005) the startTime value is 0
endTime [0..1]: TimeStamp end_date [0..1]: DateTime Timestamp indicating the end time of the reservation (when the issuer of the request expects that the resources will no longer be needed) and used by the VIM to schedule the reservation.
  • (ETSI NFV IFA005) If the attribute is not present, resources are reserved for unlimited usage time.
  • (Blazar) If the parameter is not present, resources are reserved for 24h after the start_date. to be checked
expiryTime [0..1]: TimeStamp - Timestamp indicating the time the VIM can release the reservation in case no allocation request against this reservation was made. Not yet available in Pike release.
- before_end_notification Timestamp indicating when the before_end action will be executed, e.g. take a snapshot of the resources of the lease.  
locationConstraints [0..1]: -tbd- - If present, it defines location constraints for the resource(s) is (are) requested to be reserved, e.g. in what particular Resource Zone. Not yet available in Pike release.
resourceGroupId [0..1]: Identifier project_id: Identifier Unique identifier of the “infrastructure resource group”, logical grouping of virtual resources assigned to a tenant within an Infrastructure Domain.  
I.c) Create reservation RESPONSE
ETSI NFV IFA005 Blazar Description Comment
ReservedVirtualCompute [1]: reservations    
  computePoolReserved [0..1]: ReservedComputePool   resource_type = ‘virtual:instance’ Information about compute resources that have been reserved, e.g. {“cpu_cores”:90, “vm_instances”:10, “ram”:10000}. In Blazar resource_type = ‘virtual:instance’ if the reservation was for virtual instances. Instance reservation is available from Pike release.
    -   id Identifier of the reservation.  
    -   lease-id Identifier of the corresponding lease.  
    -   resource_id ??  
    -   amount Amount of virtual instances of a given flavour that have been reserved.  
    numVcInstances [1]: Integer   - Number of virtual container instances that have been reserved.  
    numCpuCores [1]: Integer   vcpus Number of CPU cores that have been reserved.  
    virtualMemSize [1]: Number   memory_mb Size of virtual memory that has been reserved.  
    -   disk_gb Size of (disk) storage that has been reserved.  
    -   affinity: Boolean Affinity information of the reserved resources. (NFV-IFA005) no such information is returned. Recommendation to add this attribute to the response message.
    computeAttributes [0..1]: VirtualComputeAttributes ReservationData   - Information specifying additional attributes of the virtual compute resource that have been reserved.  
  virtualisationContainerReserved [0..N]: ReservedVirtualisation Container     Information about the virtualisation container(s) that have been reserved.  
    containerId [1]: Identifier     Identifier of the virtualisation container that has been reserved.  
    flavourId [1]: Identifier     Identifier of the given compute flavour used in the reserved virtualisation container.  
    accelerationCapability [0..N]: -tbd-     Selected acceleration capabilities (e.g. crypto, GPU) from the set of capabilities offered by the compute node acceleration resources. The cardinality can be 0, if no particular acceleration capability is provided.  
    virtualMemory [1]: VirtualMemoryData     Virtual memory of the reserved virtualisation container.  
    virtualCpu [1]: VirtualCpuData     Virtual CPU(s) of the reserved virtualisation container.  
    virtualDisks [0..N]: VirtualStorage     Element with information of the virtualised storage resources attached to the reserved virtualisation container.  
    virtualNetworkInterface [0..N]: VirtualNetworkInterface     Element with information of the virtual network interfaces of the reserved virtualisation container  
    zoneId [0..1]: Identifier (reference to ResoureZone)     References the resource zone where the virtualisation container has been reserved. Cardinality can be 0 to cover the case where reserved network resources are not bound to a specific resource zone.  
  reservationStatus [1]: Enum   status; status_reason Status of the compute resource reservation, e.g. to indicate if a reservation is being used  
  startTime [0..1]: TimeStamp   start_date [1] Indication when the consumption of the resources starts. If the value is 0, resources are reserved for immediate use.  
  endTime [0..1]: TimeStamp   end_date [1] Indication when the reservation ends (when it is expected that the resources will no longer be needed) and used by the VIM to schedule the reservation. If not present, resources are reserved for unlimited usage time.  
  expiryTime [0..1]: TimeStamp   - Indication when the VIM can release the reservation in case no allocation request against this reservation was made.  
  -   events    
II. Query / list compute resource reservation
II.a) REQUEST
ETSI NFV IFA005 Blazar Description Comment
queryReservationFilter [1]: Filter lease_id Query filter based on e.g. name, identifier, meta-data information or status information expressing the type of information to be retrieved. It can also be used to specify one or more reservations to be queried by providing their identifiers. Blazar does not yet allow to list leases based on a filter. In Blazar you can either list all leases registered in Blazar (GET /v1/leases) or show information about a specific lease (GET /v1/leases/{lease-id}).
II.a) RESPONSE
ETSI NFV IFA005 Blazar Description Comment
queryResult [0..N]: ReservedVirtualCompute leases{ reservations {..} } Element containing information about the reserved resource. Cardinality is 0 if the query did not return any result. For attributes of ReservedVirtualCompute see clause I.c.
III. Update reservation

Note

Compute host reservations in Blazar supports updating for name, start time, and end time.

Note

Instance reservation does not yet support the update operation. A bug report is available in [4] .

III.a) REQUEST
ETSI NFV IFA005 Blazar Description Comment
reservationId [1]: Id lease_id Identifier of the existing resource reservation to be updated.  
- name Name of the lease/reservation.  
computePoolReservation [0..1]: ComputePoolReservation - New amount of compute resources to be reserved. For attributes of ComputePoolReservation see clause I.a.
virtualisationContainer Reservation [0..N]: VirtualisationContainerReservation - New virtualisation containers to be reserved (e.g. following a specific compute “flavour”). For attributes of VirtualisationContainerReservation see clause I.b.
startTime [0..1]: TimeStamp start_date [1] Indication when the consumption of the resources resources starts. If not present, the original setting will not be changed. If present and the value is 0, resources are reserved for immediate use.  
endTime [0..1]: TimeStamp end_date [1] Indication when the reservation ends (when it is expected that the resources will no longer be needed) and used by the VIM to schedule the reservation. If not present, resources are reserved for unlimited usage time.  
expiryTime [0..1]: TimeStamp - Indication when the VIM can release the reservation in case no allocation request against this reservation was made.  
III.a) RESPONSE
ETSI NFV IFA005 Blazar Description Comment
reservationData [0..N]: ReservedVirtualCompute leases { reservations {..} } Element containing information about the updated reserved resource. For attributes of ReservedVirtualCompute and Blazar reservations see clause I.c.
IV. Terminate compute resource reservation
IV.a) REQUEST
ETSI NFV IFA005 Blazar Description Comment
reservationId [1..N]: Identifier lease_id Identifier of the resource reservation(s) to terminate.  
IV.a) RESPONSE
ETSI NFV IFA005 Blazar Description Comment
reservationId [1..N]: Identifier - Identifier of the resource reservation(s) successfullly terminated. Blazar just returns a HTTP/1.1 204 NO CONTENT response code.

SDNVPN

SDN VPN

A high-level description of the scenarios is provided in this section. For details of the scenarios and their provided capabilities refer to the scenario description document: http://artifacts.opnfv.org/danube/sdnpvn/scenarios/os-odl_l2-bgpvpn/index.html

The BGPVPN feature enables creation of BGP VPNs on the Neutron API according to the OpenStack BGPVPN blueprint at https://blueprints.launchpad.net/neutron/+spec/neutron-bgp-vpn. In a nutshell, the blueprint defines a BGPVPN object and a number of ways how to associate it with the existing Neutron object model, as well as a unique definition of the related semantics. The BGPVPN framework supports a backend driver model with currently available drivers for Bagpipe, OpenContrail, Nuage and OpenDaylight. The OPNFV scenario makes use of the OpenDaylight driver and backend implementation through the ODL NetVirt project.

SDNVPN Testing Suite

An overview of the SDNVPN Test is depicted here. More details for each test case are provided: https://wiki.opnfv.org/display/sdnvpn/SDNVPN+Testing

BGPVPN Tempest test cases
Create BGPVPN passes Create BGPVPN as non-admin fails Delete BGPVPN as non-admin fails Show BGPVPN as non-owner fails List BGPVPNs as non-owner fails Show network associated BGPVPNs as non-owner fails List network associated BGPVPNs as non-owner fails Associate/Deassociate a network to a BGPVPN resource passes Update route targets on a BGPVPN passes Update route targets on a BGPVPN as non-admin fails Reject the creation of BGPVPN with invalid route targets passes Reject the update of BGPVPN with invalid route targets passes Reject the association on an invalid network to a BGPVPN passes Reject the diassociation on an invalid network to a BGPVPN passes Associate/Deassociate a router to a BGPVPN resource passes Attach the subnet of an associated network to an associated router of the same BGVPN passes

Functest scenario specific tests:

Test Case 1 - VPN provides connectivity between subnets, using network association Name: VPN connecting Neutron networks and subnets Description: VPNs provide connectivity across Neutron networks and subnets if configured accordingly.

Test setup procedure: Set up VM1 and VM2 on Node1 and VM3 on Node2, all having ports in the same Neutron Network N1 Moreover all ports have 10.10.10/24 addresses (this subnet is denoted SN1 in the following) Set up VM4 on Node1 and VM5 on Node2, both having ports in Neutron Network N2 Moreover all ports have 10.10.11/24 addresses (this subnet is denoted SN2 in the following)

Test execution:
Create VPN1 with eRT<>iRT (so that connected subnets should not reach each other) Associate SN1 to VPN1 Ping from VM1 to VM2 should work Ping from VM1 to VM3 should work Ping from VM1 to VM4 should not work Associate SN2 to VPN1 Ping from VM4 to VM5 should work Ping from VM1 to VM4 should not work (disabled until isolation fixed upstream) Ping from VM1 to VM5 should not work (disabled until isolation fixed upstream) Change VPN 1 so that iRT=eRT Ping from VM1 to VM4 should work Ping from VM1 to VM5 should work

Test Case 2 - tenant separation Name: Using VPNs for tenant separation Description: Using VPNs to isolate tenants so that overlapping IP address ranges can be used

Test setup procedure: Set up VM1 and VM2 on Node1 and VM3 on Node2, all having ports in the same Neutron Network N1. VM1 and VM2 have IP addresses in a subnet SN1 with range 10.10.10/24

VM1: 10.10.10.11, running an HTTP server which returns “I am VM1” for any HTTP request (or something else than an HTTP server) VM2: 10.10.10.12, running an HTTP server which returns “I am VM2” for any HTTP request
VM3 has an IP address in a subnet SN2 with range 10.10.11/24
VM3: 10.10.11.13, running an HTTP server which returns “I am VM3” for any HTTP request

Set up VM4 on Node1 and VM5 on Node2, both having ports in Neutron Network N2 VM4 has an address in a subnet SN1b with range 10.10.10/24

VM4: 10.10.10.12 (the same as VM2), running an HTTP server which returns “I am VM4” for any HTTP request
VM5 has an address in a subnet SN2b with range 10.10.11/24
VM5: 10.10.11.13 (the same as VM3), running an HTTP server which returns “I am VM5” for any HTTP request
Test execution:

Create VPN 1 with iRT=eRT=RT1 and associate N1 to it HTTP from VM1 to VM2 and VM3 should work

It returns “I am VM2” and “I am VM3” respectively
HTTP from VM1 to VM4 and VM5 should not work
It never returns “I am VM4” or “I am VM5”

Create VPN2 with iRT=eRT=RT2 and associate N2 to it HTTP from VM4 to VM5 should work

It returns “I am VM5”
HTTP from VM4 to VM1 and VM3 should not work
It never returns “I am VM1” or “I am VM3”

Test Case 3 - Data Center Gateway integration Name: Data Center Gateway integration Description: Investigate the peering functionality of BGP protocol, using a Zrpcd/Quagga router and OpenDaylight Controller

Test setup procedure: Search in the pool of nodes and find one Compute node and one Controller nodes, that have OpenDaylight controller running Start an instance using ubuntu-16.04-server-cloudimg-amd64-disk1.img image and in it run the Quagga setup script Start bgp router in the Controller node, using odl:configure-bgp

Test execution: Set up a Quagga instance in a nova compute node Start a BGP router with OpenDaylight in a controller node Add the Quagga running in the instance as a neighbor Check that bgpd is running Verify that the OpenDaylight and gateway Quagga peer each other Start an instance in a second nova compute node and connect it with a new network, (Network 3-3). Create a bgpvpn (include parameters route-distinguisher and route-targets) and associate it with the network created Define the same route-distinguisher and route-targets on the simulated quagga side Check that the routes from the Network 3-3 are advertised towards simulated Quagga VM

Test Case 4 - VPN provides connectivity between subnets using router association Functest: variant of Test Case 1. Set up a Router R1 with one connected network/subnet N1/S1. Set up a second network N2. Create VPN1 and associate Router R1 and Network N2 to it.

Hosts from N2 should be able to reach hosts in N1.

Name: VPN connecting Neutron networks and subnets using router association Description: VPNs provide connectivity across Neutron networks and subnets if configured accordingly.

Test setup procedure: Set up VM1 and VM2 on Node1 and VM3 on Node2, All VMs have ports in the same Neutron Network N1 and 10.10.10/24 addresses (this subnet is denoted SN1 in the following). N1/SN1 are connected to router R1. Set up VM4 on Node1 and VM5 on Node2, Both VMs have ports in Neutron Network N2 and having 10.10.11/24 addresses (this subnet is denoted SN2 in the following)

Test execution: Create VPN1 with eRT<>iRT (so that connected subnets should not reach each other) Associate R1 to VPN1

Ping from VM1 to VM2 should work Ping from VM1 to VM3 should work Ping from VM1 to VM4 should not work
Associate SN2 to VPN1
Ping from VM4 to VM5 should work Ping from VM1 to VM4 should not work Ping from VM1 to VM5 should not work
Change VPN1 so that iRT=eRT
Ping from VM1 to VM4 should work Ping from VM1 to VM5 should work

Test Case 7 - Network associate a subnet with a router attached to a VPN and verify floating IP functionality (disabled, because of ODL Bug 6962)

A test for https://bugs.opendaylight.org/show_bug.cgi?id=6962

Setup procedure: Create VM1 in a subnet with a router attached. Create VM2 in a different subnet with another router attached. Network associate them to a VPN with iRT=eRT Ping from VM1 to VM2 should work Assign a floating IP to VM1 Pinging the floating IP should work

Test Case 8 - Router associate a subnet with a router attached to a VPN and verify floating IP functionality

Setup procedure: Create VM1 in a subnet with a router which is connected with the gateway Create VM2 in a different subnet without a router attached. Assoc the two networks in a VPN iRT=eRT One with router assoc, other with net assoc Try to ping from one VM to the other Assign a floating IP to the VM in the router assoc network Ping it

Test Case 9 - Check fail mode in OVS br-int interfaces This testcase checks if the fail mode is always “secure”. To accomplish it, a check is performed on all OVS br-int interfaces, for all OpenStack nodes. The testcase is considered as successful if all OVS br-int interfaces have fail_mode=secure

Test Case 10 - Check the communication between a group of VMs This testcase investigates if communication between a group of VMs is interrupted upon deletion and creation of VMs inside this group.

Test case flow:
Create 3 VMs: VM_1 on compute 1, VM_2 on compute 1, VM_3 on compute 2. All VMs ping each other. VM_2 is deleted. Traffic is still flying between VM_ 1 and VM_3. A new VM, VM_ 4 is added to compute 1. Traffic is not interrupted and VM_4 can be reached as well.

Testcase 11: test Opendaylight resync and group_add_mod feature mechanisms This is testcase to test Opendaylight resync and group_add_mod feature functionalities

Sub-testcase 11-1: Create and start 2 VMs, connected to a common Network.

New groups should appear in OVS dump
OVS disconnects and the VMs and the networks are cleaned.
The new groups are still in the OVS dump, cause OVS is not connected anymore, so it is not notified that the groups are deleted
OVS re-connects.
The new groups should be deleted, as Opendaylight has to resync the groups totally and should remove the groups since VMS are deleted.

Sub-testcase 11-2: Create and start 2 VMs, connected to a common Network.

New groups should appear in OVS dump
OVS disconnects.
The new groups are still in the OVS dump, cause OVS is not connected anymore, so it is not notified that the groups are deleted
OVS re-connects.
The new groups should be still there, as the topology remains. Opendaylight Carbon’s group_add_mod mechanism should handle the already existing group.
OVS re-connects.
The new groups should be still there, as the topology remains. Opendaylight Carbon’ group_add_mod mechanism should handle the already existing group.
SDN VPN
Introduction

This document provides an overview of how to work with the SDN VPN features in OPNFV.

Feature and API usage guidelines and example

For the details of using OpenStack BGPVPN API, please refer to the documentation at http://docs.openstack.org/developer/networking-bgpvpn/.

Example

In the example we will show a BGPVPN associated to 2 neutron networks. The BGPVPN will have the import and export routes in the way that it imports its own Route. The outcome will be that vms sitting on these two networks will be able to have a full L3 connectivity.

Some defines:

net_1="Network1"
net_2="Network2"
subnet_net1="10.10.10.0/24"
subnet_net2="10.10.11.0/24"

Create neutron networks and save network IDs:

neutron net-create --provider:network_type=local $net_1
export net_1_id=`echo "$rv" | grep " id " |awk '{print $4}'`
neutron net-create --provider:network_type=local $net_2
export net_2_id=`echo "$rv" | grep " id " |awk '{print $4}'`

Create neutron subnets:

neutron subnet-create $net_1 --disable-dhcp $subnet_net1
neutron subnet-create $net_2 --disable-dhcp $subnet_net2

Create BGPVPN:

neutron bgpvpn-create --route-distinguishers 100:100 --route-targets 100:2530 --name L3_VPN

Start VMs on both networks:

nova boot --flavor 1 --image <some-image> --nic net-id=$net_1_id vm1
nova boot --flavor 1 --image <some-image> --nic net-id=$net_2_id vm2

The VMs should not be able to see each other.

Associate to Neutron networks:

neutron bgpvpn-net-assoc-create L3_VPN --network $net_1_id
neutron bgpvpn-net-assoc-create L3_VPN --network $net_2_id

Now the VMs should be able to ping each other

Troubleshooting

Check neutron logs on the controller:

tail -f /var/log/neutron/server.log |grep -E "ERROR|TRACE"

Check Opendaylight logs:

tail -f /opt/opendaylight/data/logs/karaf.log

Restart Opendaylight:

service opendaylight restart
SDN VPN feature installation
Hardware requirements

The SDNVPN scenarios can be deployed as a bare-metal or a virtual environment on a single host.

Bare metal deployment on Pharos Lab

Hardware requirements for bare-metal deployments of the OPNFV infrastructure are specified by the Pharos project. The Pharos project provides an OPNFV hardware specification for configuring your hardware at: http://artifacts.opnfv.org/pharos/docs/pharos-spec.html.

Virtual deployment on a single server

To perform a virtual deployment of an OPNFV scenario on a single host, that host has to meet the hardware requirements outlined in the <missing spec>.

When ODL is used as an SDN Controller in an OPNFV virtual deployment, ODL is running on the OpenStack Controller VMs. It is therefore recommended to increase the amount of resources for these VMs.

Our recommendation is to have 2 additional virtual cores and 8GB additional virtual memory on top of the normally recommended configuration.

Together with the commonly used recommendation this sums up to:

6 virtual CPU cores
16 GB virtual memory

The installation section below has more details on how to configure this.

Installation using Fuel installer
Preparing the host to install Fuel by script

Before starting the installation of the os-odl-bgpnvp scenario some preparation of the machine that will host the Fuel VM must be done.

Installation of required packages

To be able to run the installation of the basic OPNFV fuel installation the Jumphost (or the host which serves the VMs for the virtual deployment) needs to install the following packages:

sudo apt-get install -y git make curl libvirt-bin libpq-dev qemu-kvm \
                        qemu-system tightvncserver virt-manager sshpass \
                        fuseiso genisoimage blackbox xterm python-pip \
                        python-git python-dev python-oslo.config \
                        python-pip python-dev libffi-dev libxml2-dev \
                       libxslt1-dev libffi-dev libxml2-dev libxslt1-dev \
                       expect curl python-netaddr p7zip-full

sudo pip install GitPython pyyaml netaddr paramiko lxml scp \
                 python-novaclient python-neutronclient python-glanceclient \
                 python-keystoneclient debtcollector netifaces enum
Download the source code and artifact

To be able to install the scenario os-odl-bgpvpn one can follow the way CI is deploying the scenario. First of all the opnfv-fuel repository needs to be cloned:

git clone ssh://<user>@gerrit.opnfv.org:29418/fuel

To check out a specific version of OPNFV, checkout the appropriate branch:

cd fuel
git checkout stable/<colorado|danube|euphrates>

Now download the corresponding OPNFV Fuel ISO into an appropriate folder from the website

https://www.opnfv.org/software/downloads/release-archives

Have in mind that the fuel repo version needs to map with the downloaded artifact. Note: it is also possible to build the Fuel image using the tools found in the fuel git repository, but this is out of scope of the procedure described here. Check the Fuel project documentation for more information on building the Fuel ISO.

Simplified scenario deployment procedure using Fuel

This section describes the installation of the os-odl-bgpvpn-ha or os-odl-bgpvpn-noha OPNFV reference platform stack across a server cluster or a single host as a virtual deployment.

Scenario Preparation

dea.yaml and dha.yaml need to be copied and changed according to the lab-name/host where you deploy. Copy the full lab config from:

cp <path-to-opnfv-fuel-repo>/deploy/config/labs/devel-pipeline/elx \
   <path-to-opnfv-fuel-repo>/deploy/config/labs/devel-pipeline/<your-lab-name>

Add at the bottom of dha.yaml

disks:
  fuel: 100G
  controller: 100G
  compute: 100G

define_vms:
  controller:
    vcpu:
      value: 4
    memory:
      attribute_equlas:
        unit: KiB
      value: 16388608
    currentMemory:
      attribute_equlas:
        unit: KiB
      value: 16388608

Check if the default settings in dea.yaml are in line with your intentions and make changes as required.

Installation procedures

We describe several alternative procedures in the following. First, we describe several methods that are based on the deploy.sh script, which is also used by the OPNFV CI system. It can be found in the Fuel repository.

In addition, the SDNVPN feature can also be configured manually in the Fuel GUI. This is described in the last subsection.

Before starting any of the following procedures, go to

cd <opnfv-fuel-repo>/ci
Full automatic virtual deployment High Availablity Mode

The following command will deploy the high-availability flavor of SDNVPN scenario os-odl-bgpvpn-ha in a fully automatic way, i.e. all installation steps (Fuel server installation, configuration, node discovery and platform deployment) will take place without any further prompt for user input.

sudo bash ./deploy.sh -b file://<path-to-opnfv-fuel-repo>/config/ -l devel-pipeline -p <your-lab-name> -s os-odl_l2-bgpvpn-ha -i file://<path-to-fuel-iso>
Full automatic virtual deployment NO High Availability Mode

The following command will deploy the SDNVPN scenario in its non-high-availability flavor (note the different scenario name for the -s switch). Otherwise it does the same as described above.

sudo bash ./deploy.sh -b file://<path-to-opnfv-fuel-repo>/config/ -l devel-pipeline -p <your-lab-name> -s os-odl_l2-bgpvpn-noha -i file://<path-to-fuel-iso>
Automatic Fuel installation and manual scenario deployment

A useful alternative to the full automatic procedure is to only autodeploy the Fuel host and to run host selection, role assignment and SDNVPN scenario configuration manually.

sudo bash ./deploy.sh -b file://<path-to-opnfv-fuel-repo>/config/ -l devel-pipeline -p <your-lab-name> -s os-odl_l2-bgpvpn-ha -i file://<path-to-fuel-iso> -e

With -e option the installer does not launch environment deployment, so a user can do some modification before the scenario is really deployed. Another interesting option is the -f option which deploys the scenario using an existing Fuel host.

The result of this installation is a fuel sever with the right config for BGPVPN. Now the deploy button on fuel dashboard can be used to deploy the environment. It is as well possible to do the configuration manuell.

Feature configuration on existing Fuel

If a Fuel server is already provided but the fuel plugins for Opendaylight, Openvswitch and BGPVPN are not provided install them by:

cd /opt/opnfv/
fuel plugins --install fuel-plugin-ovs-*.noarch.rpm
fuel plugins --install opendaylight-*.noarch.rpm
fuel plugins --install bgpvpn-*.noarch.rpm

If plugins are installed and you want to update them use –force flag.

Now the feature can be configured. Create a new environment with “Neutron with ML2 plugin” and in there “Neutron with tunneling segmentation”. Go to Networks/Settings/Other and check “Assign public network to all nodes”. This is required for features such as floating IP, which require the Compute hosts to have public interfaces. Then go to settings/other and check “OpenDaylight plugin”, “Use ODL to manage L3 traffic”, “BGPVPN plugin” and set the OpenDaylight package version to “5.2.0-1”. Then you should be able to check “BGPVPN extensions” in OpenDaylight plugin section.

Now the deploy button on fuel dashboard can be used to deploy the environment.

Virtual deployment using Apex installer

For Virtual Apex deployment a host with Centos 7 is needed. This installation was tested on centos-release-7-2.1511.el7.centos.2.10.x86_64 however any other Centos 7 version should be fine.

Download the Apex repo from opnfv gerrit and checkout stable/danube:

git clone ssh://<user>@gerrit.opnfv.org:29418/apex
cd apex
git checkout stable/danube

In apex/contrib you will find simple_deploy.sh:

#!/bin/bash
set -e
apex_home=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../
export CONFIG=$apex_home/build
export LIB=$apex_home/lib
export RESOURCES=$apex_home/.build/
export PYTHONPATH=$PYTHONPATH:$apex_home/lib/python
$apex_home/ci/dev_dep_check.sh || true
$apex_home/ci/clean.sh
pushd $apex_home/build
make clean
make undercloud
make overcloud-opendaylight
popd
pushd $apex_home/ci
echo "All further output will be piped to $PWD/nohup.out"
(nohup ./deploy.sh -v -n $apex_home/config/network/network_settings.yaml -d $apex_home/config/deploy/os-odl_l3-nofeature-noha.yaml &)
tail -f nohup.out
popd

This script will:

  • “dev_dep_check.sh” install all required packages.
  • “clean.sh” clean existing deployments
  • “make clean” clean existing builds
  • “make undercloud” building the undercloud image
  • “make overcloud-opendaylight” build the overcloud image and convert that to a overcloud with opendaylight image
  • “deploy.sh” deploy the os-odl_l3-nofeature-nohs.yaml scenario

Edit the script and change the scenario to os-odl-bgpvpn-noha.yaml. More scenraios can be found: ./apex/config/deploy/

Execute the script in a own screen process:

yum install -y screen
screen -S deploy
bash ./simple_deploy.sh

Determin the mac address of the undercloud vm:

# virsh domiflist undercloud
-> Default network
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      network    default    virtio      00:6a:9d:24:02:31
vnet1      bridge     admin      virtio      00:6a:9d:24:02:33
vnet2      bridge     external   virtio      00:6a:9d:24:02:35
# arp -n |grep 00:6a:9d:24:02:31
192.168.122.34           ether   00:6a:9d:24:02:31   C                     virbr0
# ssh stack@192.168.122.34
-> no password needed (password stack)

List overcloud deployment info:

# source stackrc
# # Compute and controller:
# nova list
# # Networks
# neutron net-list

List overcloud openstack info:

# source overcloudrc
# nova list
# ...

On the undercloud:

# . stackrc
# nova list
# ssh heat-admin@<ip-of-host>
-> there is no password the user has direct sudo rights.
SDN VPN
Introduction

This document provides an overview of how to work with the SDN VPN features in OPNFV.

Feature and API usage guidelines and example

For the details of using OpenStack BGPVPN API, please refer to the documentation at http://docs.openstack.org/developer/networking-bgpvpn/.

Example

In the example we will show a BGPVPN associated to 2 neutron networks. The BGPVPN will have the import and export routes in the way that it imports its own Route. The outcome will be that vms sitting on these two networks will be able to have a full L3 connectivity.

Some defines:

net_1="Network1"
net_2="Network2"
subnet_net1="10.10.10.0/24"
subnet_net2="10.10.11.0/24"

Create neutron networks and save network IDs:

neutron net-create --provider:network_type=local $net_1
export net_1_id=`echo "$rv" | grep " id " |awk '{print $4}'`
neutron net-create --provider:network_type=local $net_2
export net_2_id=`echo "$rv" | grep " id " |awk '{print $4}'`

Create neutron subnets:

neutron subnet-create $net_1 --disable-dhcp $subnet_net1
neutron subnet-create $net_2 --disable-dhcp $subnet_net2

Create BGPVPN:

neutron bgpvpn-create --route-distinguishers 100:100 --route-targets 100:2530 --name L3_VPN

Start VMs on both networks:

nova boot --flavor 1 --image <some-image> --nic net-id=$net_1_id vm1
nova boot --flavor 1 --image <some-image> --nic net-id=$net_2_id vm2

The VMs should not be able to see each other.

Associate to Neutron networks:

neutron bgpvpn-net-assoc-create L3_VPN --network $net_1_id
neutron bgpvpn-net-assoc-create L3_VPN --network $net_2_id

Now the VMs should be able to ping each other

Troubleshooting

Check neutron logs on the controller:

tail -f /var/log/neutron/server.log |grep -E "ERROR|TRACE"

Check Opendaylight logs:

tail -f /opt/opendaylight/data/logs/karaf.log

Restart Opendaylight:

service opendaylight restart
SDN VPN user guide
Introduction

This document provides an overview of how to work with the SDN VPN features in OPNFV.

Feature and API usage guidelines and example

For the details of using OpenStack BGPVPN API, please refer to the documentation at http://docs.openstack.org/developer/networking-bgpvpn/.

Example

In the example we will show a BGPVPN associated to 2 neutron networks. The BGPVPN will have the import and export routes in the way that it imports its own Route. The outcome will be that vms sitting on these two networks will be able to have a full L3 connectivity.

Some defines:

net_1="Network1"
net_2="Network2"
subnet_net1="10.10.10.0/24"
subnet_net2="10.10.11.0/24"

Create neutron networks and save network IDs:

neutron net-create --provider:network_type=local $net_1
export net_1_id=`echo "$rv" | grep " id " |awk '{print $4}'`
neutron net-create --provider:network_type=local $net_2
export net_2_id=`echo "$rv" | grep " id " |awk '{print $4}'`

Create neutron subnets:

neutron subnet-create $net_1 --disable-dhcp $subnet_net1
neutron subnet-create $net_2 --disable-dhcp $subnet_net2

Create BGPVPN:

neutron bgpvpn-create --route-distinguishers 100:100 --route-targets 100:2530 --name L3_VPN

Start VMs on both networks:

nova boot --flavor 1 --image <some-image> --nic net-id=$net_1_id vm1
nova boot --flavor 1 --image <some-image> --nic net-id=$net_2_id vm2

The VMs should not be able to see each other.

Associate to Neutron networks:

neutron bgpvpn-net-assoc-create L3_VPN --network $net_1_id
neutron bgpvpn-net-assoc-create L3_VPN --network $net_2_id

Now the VMs should be able to ping each other

Troubleshooting

Check neutron logs on the controller:

tail -f /var/log/neutron/server.log |grep -E "ERROR|TRACE"

Check Opendaylight logs:

tail -f /opt/opendaylight/data/logs/karaf.log

Restart Opendaylight:

service opendaylight restart
SDN VPN
Introduction

This document provides an overview of how to work with the SDN VPN features in OPNFV.

Feature and API usage guidelines and example

For the details of using OpenStack BGPVPN API, please refer to the documentation at http://docs.openstack.org/developer/networking-bgpvpn/.

Example

In the example we will show a BGPVPN associated to 2 neutron networks. The BGPVPN will have the import and export routes in the way that it imports its own Route. The outcome will be that vms sitting on these two networks will be able to have a full L3 connectivity.

Some defines:

net_1="Network1"
net_2="Network2"
subnet_net1="10.10.10.0/24"
subnet_net2="10.10.11.0/24"

Create neutron networks and save network IDs:

neutron net-create --provider:network_type=local $net_1
export net_1_id=`echo "$rv" | grep " id " |awk '{print $4}'`
neutron net-create --provider:network_type=local $net_2
export net_2_id=`echo "$rv" | grep " id " |awk '{print $4}'`

Create neutron subnets:

neutron subnet-create $net_1 --disable-dhcp $subnet_net1
neutron subnet-create $net_2 --disable-dhcp $subnet_net2

Create BGPVPN:

neutron bgpvpn-create --route-distinguishers 100:100 --route-targets 100:2530 --name L3_VPN

Start VMs on both networks:

nova boot --flavor 1 --image <some-image> --nic net-id=$net_1_id vm1
nova boot --flavor 1 --image <some-image> --nic net-id=$net_2_id vm2

The VMs should not be able to see each other.

Associate to Neutron networks:

neutron bgpvpn-net-assoc-create L3_VPN --network $net_1_id
neutron bgpvpn-net-assoc-create L3_VPN --network $net_2_id

Now the VMs should be able to ping each other

Troubleshooting

Check neutron logs on the controller:

tail -f /var/log/neutron/server.log |grep -E "ERROR|TRACE"

Check Opendaylight logs:

tail -f /opt/opendaylight/data/logs/karaf.log

Restart Opendaylight:

service opendaylight restart

SFC

Service Function Chaining (SFC)
1. Requirements

This section defines requirements for the initial OPNFV SFC implementation, including those requirements driving upstream project enhancements.

1.1. Minimal Viable Requirement

Deploy a complete SFC solution by integrating OpenDaylight SFC with OpenStack in an OPNFV environment.

1.2. Detailed Requirements

These are the Euphrates specific requirements:

1 The supported Service Chaining encapsulation will be NSH VXLAN-GPE.

2 The version of OVS used must support NSH.

3 The SF VM life cycle will be managed by the Tacker VNF Manager.

4 The supported classifier is OpenDaylight NetVirt.

5 ODL will be the OpenStack Neutron backend and will handle all networking
on the compute nodes.

6 Tacker will use the networking-sfc API to configure ODL

7 ODL will use flow based tunnels to create the VXLAN-GPE tunnels

1.3. Long Term Requirements

These requirements are out of the scope of the Euphrates release.

1 Dynamic movement of SFs across multiple Compute nodes.

2 Load Balancing across multiple SFs

3 Support of a different MANO component apart from Tacker

SFC installation and configuration instruction
1. Abstract

This document provides information on how to install the OpenDaylight SFC features in OPNFV with the use of os_odl-sfc-(no)ha scenario.

2. SFC feature desciription

For details of the scenarios and their provided capabilities refer to the scenario description documents:

The SFC feature enables creation of Service Fuction Chains - an ordered list of chained network funcions (e.g. firewalls, NAT, QoS)

The SFC feature in OPNFV is implemented by 3 major components:

  • OpenDaylight SDN controller
  • Tacker: Generic VNF Manager (VNFM) and a NFV Orchestrator (NFVO)
  • OpenvSwitch: The Service Function Forwarder(s)
3. Hardware requirements

The SFC scenarios can be deployed on a bare-metal OPNFV cluster or on a virtual environment on a single host.

3.1. Bare metal deployment on (OPNFV) Pharos lab

Hardware requirements for bare-metal deployments of the OPNFV infrastructure are given by the Pharos project. The Pharos project provides an OPNFV hardware specification for configuring your hardware: http://artifacts.opnfv.org/pharos/docs/pharos-spec.html

3.2. Virtual deployment

SFC scenarios can be deployed using APEX installer and xci utility. Check the requirements from those in order to be able to deploy the OPNFV-SFC:

Apex: https://wiki.opnfv.org/display/apex/Apex XCI: https://wiki.opnfv.org/display/INF/XCI+Developer+Sandbox

SFC User Guide
1. SFC description

The OPNFV SFC feature will create service chains, classifiers, and create VMs for Service Functions, allowing for client traffic intended to be sent to a server to first traverse the provisioned service chain.

The Service Chain creation consists of configuring the OpenDaylight SFC feature. This configuration will in-turn configure Service Function Forwarders to route traffic to Service Functions. A Service Function Forwarder in the context of OPNFV SFC is the “br-int” OVS bridge on an Open Stack compute node.

The classifier(s) consist of configuring the OpenDaylight Netvirt feature. Netvirt is a Neutron backend which handles the networking for VMs. Netvirt can also create simple classification rules (5-tuples) to send specific traffic to a pre-configured Service Chain. A common example of a classification rule would be to send all HTTP traffic (tcp port 80) to a pre-configured Service Chain.

Service Function VM creation is performed via a VNF Manager. Currently, OPNFV SFC is integrated with OpenStack Tacker, which in addition to being a VNF Manager, also orchestrates the SFC configuration. In OPNFV SFC Tacker creates service chains, classification rules, creates VMs in OpenStack for Service Functions, and then communicates the relevant configuration to OpenDaylight SFC.

2. SFC capabilities and usage

The OPNFV SFC feature can be deployed with either the “os-odl-sfc-ha” or the “os-odl-sfc-noha” scenario. SFC usage for both of these scenarios is the same.

As previously mentioned, Tacker is used as a VNF Manager and SFC Orchestrator. All the configuration necessary to create working service chains and classifiers can be performed using the Tacker command line. Refer to the Tacker walkthrough (step 3 and onwards) for more information.

2.1. SFC API usage guidelines and example

Refer to the Tacker walkthrough for Tacker usage guidelines and examples.

Service Function Chaining (SFC)
1. Introduction

The OPNFV Service Function Chaining (SFC) project aims to provide the ability to define an ordered list of a network services (e.g. firewalls, NAT, QoS). These services are then “stitched” together in the network to create a service chain. This project provides the infrastructure to install the upstream ODL SFC implementation project in an NFV environment.

2. Definitions

Definitions of most terms used here are provided in the IETF SFC Architecture RFC. Additional terms specific to the OPNFV SFC project are defined below.

3. Abbreviations
Abbreviations
Abbreviation Term
NS Network Service
NFVO Network Function Virtualization Orchestrator
NF Network Function
NSH Network Services Header (Service chaining encapsulation)
ODL OpenDaylight SDN Controller
RSP Rendered Service Path
SDN Software Defined Networking
SF Service Function
SFC Service Function Chain(ing)
SFF Service Function Forwarder
SFP Service Function Path
VNF Virtual Network Function
VNFM Virtual Network Function Manager
VNF-FG Virtual Network Function Forwarding Graph
VIM Virtual Infrastructure Manager
4. Use Cases

This section outlines the Danube use cases driving the initial OPNFV SFC implementation.

4.1. Use Case 1 - Two chains

This use case is targeted on creating simple Service Chains using Firewall Service Functions. As can be seen in the following diagram, 2 service chains are created, each through a different Service Function Firewall. Service Chain 1 will block HTTP, while Service Chain 2 will block SSH.

_images/OPNFV_SFC_Brahmaputra_UseCase.jpg
4.2. Use Case 2 - One chain traverses two service functions

This use case creates two service functions, and a chain that makes the traffic flow through both of them. More information is available in the OPNFV-SFC wiki:

https://wiki.opnfv.org/display/sfc/Functest+SFC-ODL+-+Test+2

5. Architecture

This section describes the architectural approach to incorporating the upstream OpenDaylight (ODL) SFC project into the OPNFV Danube platform.

5.1. Service Functions

A Service Function (SF) is a Function that provides services to flows traversing a Service Chain. Examples of typical SFs include: Firewall, NAT, QoS, and DPI. In the context of OPNFV, the SF will be a Virtual Network Function. The SFs receive data packets from a Service Function Forwarder.

5.2. Service Function Forwarders

The Service Function Forwarder (SFF) is the core element used in Service Chaining. It is an OpenFlow switch that, in the context of OPNFV, is hosted in an OVS bridge. In OPNFV there will be one SFF per Compute Node that will be hosted in the “br-int” OpenStack OVS bridge.

The responsibility of the SFF is to steer incoming packets to the corresponding Service Function, or to the SFF in the next compute node. The flows in the SFF are programmed by the OpenDaylight SFC SDN Controller.

5.3. Service Chains

Service Chains are defined in the OpenDaylight SFC Controller using the following constructs:

SFC
A Service Function Chain (SFC) is an ordered list of abstract SF types.
SFP
A Service Function Path (SFP) references an SFC, and optionally provides concrete information about the SFC, like concrete SF instances. If SF instances are not supplied, then the RSP will choose them.
RSP
A Rendered Service Path (RSP) is the actual Service Chain. An RSP references an SFP, and effectively merges the information from the SFP and SFC to create the Service Chain. If concrete SF details were not provided in the SFP, then SF selection algorithms are used to choose one. When the RSP is created, the OpenFlows will be programmed and written to the SFF(s).
5.4. Service Chaining Encapsulation

Service Chaining Encapsulation encapsulates traffic sent through the Service Chaining domain to facilitate easier steering of packets through Service Chains. If no Service Chaining Encapsulation is used, then packets much be classified at every hop of the chain, which would be slow and would not scale well.

In ODL SFC, Network Service Headers (NSH) is used for Service Chaining encapsulation. NSH is an IETF specification that uses 2 main header fields to facilitate packet steering, namely:

NSP (NSH Path)
The NSP is the Service Path ID.
NSI (NSH Index)
The NSI is the Hop in the Service Chain. The NSI starts at 255 and is decremented by every SF. If the NSI reaches 0, then the packet is dropped, which avoids loops.

NSH also has metadata fields, but that’s beyond the scope of this architecture.

In ODL SFC, NSH packets are encapsulated in VXLAN-GPE.

5.5. Classifiers

A classifier is the entry point into Service Chaining. The role of the classifier is to map incoming traffic to Service Chains. In ODL SFC, this mapping is performed by matching the packets and encapsulating the packets in a VXLAN-GPE NSH tunnel.

The packet matching is specific to the classifier implementation, but can be as simple as an ACL, or can be more complex by using PCRF information or DPI.

5.6. VNF Manager

In OPNFV SFC, a VNF Manager is needed to spin-up VMs for Service Functions. It has been decided to use the OpenStack Tacker VNF Mgr to spin-up and manage the life cycle of the SFs. Tacker will receive the ODL SFC configuration, manage the SF VMs, and forward the configuration to ODL SFC. The following sequence diagram details the interactions with the VNF Mgr:

_images/OPNFV_SFC_Brahmaputra_SfCreation.jpg
5.7. OPNFV SFC Network Topology

The following image details the Network Topology used in OPNFV Danube SFC:

_images/OPNFV_SFC_Brahmaputra_NW_Topology.jpg

Infrastructure

Infrastructure Overview

OPNFV develops, operates, and maintains infrastructure which is used by the OPNFV Community for development, integration, and testing purposes. OPNFV Infrastructure Working Group (Infra WG) oversees the OPNFV Infrastructure, ensures it is kept in a state which serves the community in best possible way and always up to date.

Infra WG is working towards a model whereby we have a seamless pipeline for handing resource requests from the OPNFV community for both development and Continuous Integration perspectives. Automation of requests and integration to existing automation tools is a primary driver in reaching this model. In the Infra WG, we imagine a model where the Infrastructure Requirements that are specified by a Feature, Installer or otherrelevant projects within OPNFV are requested, provisioned, used, reported on and subsequently torn down with no (or minimal) user intervention at the physical/infrastructure level.

Objectives of the Infra WG are

  • Deliver efficiently dimensions resources to OPNFV community needs on request in a timely manner that ensure maximum usage (capacity) and maximum density (distribution of workloads)
  • Satisfy the needs of the twice-yearly release projects, this includes being able to handle load (amount of projects and requests) as well as need (topology and different layouts)
  • Support OPNFV community users. As the INFRA group, we are integral to all aspects of the OPNFV Community (since it starts with the Hardware) - this can mean troubleshooting any element within the stack
  • Provide a method to expand and adapt as OPNFV community needs grow and provide this to Hosting Providers (lab providers) for input in growth forecast so they can better judge how best to contribute with their resources.
  • Work with reporting and other groups to ensure we have adequate feedback to the end-users of the labs on how their systems, code, feature performs.

The details of what is provided as part of the infrastructure can be seen in following chapters.

Hardware Infrastructure

TBD

Software Infrastructure

3. Software Infrastructure

OPNFV Software Infrastructure consists of set of components and tools that realize OPNFV Continuous Integration (CI) and provide means for community to contribute to OPNFV in most efficient way. OPNFV Software Infrastructure enables and orchestrates development, integration and testing activities for the components OPNFV consumes from upstream communities and for the development work done in scope of OPNFV. Apart from orchestration aspects, providing timely feedback that is fit for purpose to the OPNFV community is one of its missions.

CI is the top priority for OPNFV Software Infrastructure. Due to the importance the OPNFV community puts into it, the resulting CI machinery is highly powerful, capable and runs against distributed hardware infrastructure managed by OPNFV Pharos Project. The hardware infrastructure OPNFV CI relies on is located in 3 different continents, 5+ different countries and 10+ different member companies.

OPNFV CI is continuously evolved in order to fulfill the needs and match the expectations of the OPNFV community.

OPNFV Software Infrastructure is developed, maintained and operated by OPNFV Releng Project with the support from Linux Foundation.

3.1. Continuous Integration Server

Jenkins

3.1.1. Connecting OPNFV Community Labs to OPNFV Jenkins
3.1.1.1. Abstract

This document describes how to connect resources (servers) located in Linux Foundation (LF) lab and labs provided by the OPNFV Community to OPNFV Jenkins.

3.1.1.2. License

Connecting OPNFV Community Labs to OPNFV Jenkins (c) by Fatih Degirmenci (Ericsson AB) and others.

Connecting OPNFV Labs to OPNFV Jenkins document is licensed under a Creative Commons Attribution 4.0 International License.

You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>.

3.1.1.3. Version History
Date Version Author Comment
2015-05-05 0.1.0 Fatih Degirmenci First draft
2015-09-25 1.0.0 Fatih Degirmenci Instructions for the Arno SR1 release
2016-01-25 1.1.0 Jun Li Change the format for new doc toolchain
2016-01-27 1.2.0 Fatih Degirmenci Instructions for the Brahmaputra release
2016-05-25 1.3.0 Julien Add an additional step after step9 to output the correct monit config file
3.1.1.4. Jenkins

Jenkins is an extensible open source Continuous Integration (CI) server. [1]

Linux Foundation (LF) hosts and operates OPNFV Jenkins.

3.1.1.5. Jenkins Slaves

Slaves are computers that are set up to build projects for a Jenkins Master. [2]

Jenkins runs a separate program called “slave agent” on slaves. When slaves are registered to a master, the master starts distributing load to slaves by scheduling jobs to run on slaves if the jobs are set to run on them. [2]

Term Node is used to refer to all machines that are part of Jenkins grid, slaves and master. [2]

Two types of slaves are currently connected to OPNFV Jenkins and handling different tasks depending on the purpose of connecting the slave.

The slaves connected to OPNFV Jenkins can be seen using this link: https://build.opnfv.org/ci/computer/

Slaves without red cross next to computer icon are fully functional.

3.1.1.6. Connecting Slaves to OPNFV Jenkins

The method that is normally used for connecting slaves to Jenkins requires direct SSH access to servers. [3] This is the method that is used for connecting slaves hosted in LF Lab.

Connecting slaves using direct SSH access can become a challenge given that OPNFV Project has number of different labs provided by community as mentioned in previous section. All these labs have different security requirements which can increase the effort and the time needed for connecting slaves to Jenkins. In order to reduce the effort and the time needed for connecting slaves and streamline the process, it has been decided to connect slaves using Java Network Launch Protocol (JNLP).

3.1.1.6.1. Connecting Slaves from LF Lab to OPNFV Jenkins

Slaves hosted in LF Lab are handled by LF. All the requests and questions regarding these slaves should be submitted to OPNFV LF Helpdesk.

3.1.1.6.2. Connecting Slaves from Community Labs to OPNFV Jenkins

As noted in corresponding section, slaves from Community Labs are connected using JNLP. Via JNLP, slaves open connection towards Jenkins Master instead of Jenkins Master accessing to them directly.

Servers connecting to OPNFV Jenkins using this method must have access to internet.

Please follow below steps to connect a slave to OPNFV Jenkins.

  1. Create a user named jenkins on the machine you want to connect to OPNFV Jenkins and give the user sudo rights.
  2. Install needed software on the machine you want to connect to OPNFV Jenkins as slave.
  • openjdk 8
  • monit
  1. If the slave will be used for running virtual deployments, Functest, and Yardstick, install below software and make jenkins user the member of the groups.
  • docker
  • libvirt
  1. Create slave root in Jenkins user home directory.
mkdir -p /home/jenkins/opnfv/slave_root
  1. Clone OPNFV Releng Git repository.

mkdir -p /home/jenkins/opnfv/repos

cd /home/jenkins/opnfv/repos

git clone https://gerrit.opnfv.org/gerrit/p/releng.git

  1. Contact LF by sending mail to OPNFV LF Helpdesk and request creation of a slave on OPNFV Jenkins. Include below information in your mail.
  • Slave root (/home/jenkins/opnfv/slave_root)
  • Public IP of the slave (You can get the IP by executing curl http://icanhazip.com/)
  • PGP Key (attached to the mail or exported to a key server)
  1. Once you get confirmation from LF stating that your slave is created on OPNFV Jenkins, check if the firewall on LF is open for the server you are trying to connect to Jenkins.

cp /home/jenkins/opnfv/repos/releng/utils/jenkins-jnlp-connect.sh /home/jenkins/ cd /home/jenkins/ sudo ./jenkins-jnlp-connect.sh -j /home/jenkins -u jenkins -n  <slave name on OPNFV Jenkins> -s <the token you received from LF> -f

  • If you receive an error, follow the steps listed on the command output.
  1. Run the same script with test(-t) on foreground in order to make sure no problem on connection. You should see INFO: Connected in the console log.

sudo ./jenkins-jnlp-connect.sh -j /home/jenkins -u jenkins -n <slave name on OPNFV Jenkins> -s <the token you received from LF> -t

  • If you receive an error similar to the one shown on this link, you need to check your firewall and allow outgoing connections for the port.
  1. Kill the Java slave.jar process.
  2. Run the same script normally without test(-t) in order to get monit script created.
sudo ./jenkins-jnlp-connect.sh -j /home/jenkins -u jenkins -n <slave name on OPNFV Jenkins> -s <the token you received from LF>
  1. Edit monit configuration and enable http interface. The file to edit is /etc/monit/monitrc on Ubuntu systems. Uncomment below lines.
set httpd port 2812 and
use address localhost # only accept connection from localhost allow localhost # allow localhost to connect to the server and
  1. Restart monit service.
  • Without systemd:

    sudo service monit restart

  • With systemd: you have to enable monit service first and then restart it.

    sudo systemctl enable monit

    sudo systemctl restart monit

  1. Check to see if jenkins comes up as managed service in monit.
sudo monit status
  1. Connect slave to OPNFV Jenkins using monit.
sudo monit start jenkins
  1. Check slave on OPNFV Jenkins to verify the slave is reported as connected.
  • The slave on OPNFV Jenkins should have some executors in “Idle” state if the connection is successful.
3.1.1.7. Notes
3.1.1.7.1. PGP Key Instructions

Public PGP Key can be uploaded to public key server so it can be taken from there using your mail address. Example command to upload the key to key server is

gpg --keyserver hkp://keys.gnupg.net:80  --send-keys XXXXXXX

The Public PGP Key can also be attached to the email by storing the key in a file and then attaching it to the email.

gpg --export -a '<your email address>' > pgp.pubkey
3.1.2. Jenkins User Guide

TBD

3.1.3. Creating/Configuring/Verifying Jenkins Jobs

Clone and setup the repo:

git clone ssh://YOU@gerrit.opnfv.org:29418/releng
cd releng
git review -s

Make changes:

git commit -sv
git review
remote: Resolving deltas: 100% (3/3)
remote: Processing changes: new: 1, refs: 1, done
remote:
remote: New Changes:
remote:   https://gerrit.opnfv.org/gerrit/<CHANGE_ID>
remote:
To ssh://YOU@gerrit.opnfv.org:29418/releng.git
 * [new branch]      HEAD -> refs/publish/master

Test with tox:

tox -v -ejjb

Submit the change to gerrit:

git review -v

Follow the link given in the stdoutput to gerrit eg: https://gerrit.opnfv.org/gerrit/<CHANGE_ID> the verify job will have completed and you will see Verified +1 jenkins-ci in the gerrit ui.

If the changes pass the verify job https://build.opnfv.org/ci/job/releng-verify-jjb/ , the patch can be submitited by a committer.

Job Types

  • Verify Job
    • Trigger: recheck or reverify
  • Merge Job
    • Trigger: remerge
  • Experimental Job
    • Trigger: check-experimental

The verify and merge jobs are retriggerable in Gerrit by simply leaving a comment with one of the keywords listed above. This is useful in case you need to re-run one of those jobs in case if build issues or something changed with the environment.

The experimental jobs are not triggered automatically. You need to leave a comment with the keyword list above to trigger it manually. It is useful for trying out experimental features.

Note that, experimental jobs skip vote for verified status, which means it will reset the verified status to 0. If you want to keep the verified status, use recheck-experimental in commit message to trigger both verify and experimental jobs.

You can look in the releng/INFO file for a list of current committers to add as reviewers to your patch in order to get it reviewed and submitted.

Or Add the group releng-contributors

Or just email a request for review to helpdesk@opnfv.org

The Current merge and verify jobs for jenkins job builder can be found in releng-jobs.yaml.

3.1.4. Jenkins Node Labels

TBD

3.2. Source Control and Code Review

Gerrit

3.2.1. Gerrit User Guide
3.3. Artifact and Image Repositories

Google Storage & Docker Hub

3.3.1. Artifact Repository

TBD

3.3.2. Docker Hub

TBD

3.4. Issue and Bug Tracking

JIRA

3.5. Dashboards and Analytics

Security

Continuous Integration - CI

Please see the details of CI from the chapters below.

Cross Community Continuous Integration - XCI

Please see the details of XCI from the chapters below.