%% Generated by Sphinx.
\def\sphinxdocclass{article}
\documentclass[letterpaper,10pt,english]{sphinxhowto}
\ifdefined\pdfpxdimen
   \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
\fi \sphinxpxdimen=.75bp\relax
\ifdefined\pdfimageresolution
    \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax
\fi
%% let collapsable pdf bookmarks panel have high depth per default
\PassOptionsToPackage{bookmarksdepth=5}{hyperref}

\PassOptionsToPackage{warn}{textcomp}
\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
% support both utf8 and utf8x syntaxes
  \ifdefined\DeclareUnicodeCharacterAsOptional
    \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}}
  \else
    \let\sphinxDUC\DeclareUnicodeCharacter
  \fi
  \sphinxDUC{00A0}{\nobreakspace}
  \sphinxDUC{2500}{\sphinxunichar{2500}}
  \sphinxDUC{2502}{\sphinxunichar{2502}}
  \sphinxDUC{2514}{\sphinxunichar{2514}}
  \sphinxDUC{251C}{\sphinxunichar{251C}}
  \sphinxDUC{2572}{\textbackslash}
\fi
\usepackage{cmap}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{babel}



\usepackage{tgtermes}
\usepackage{tgheros}
\renewcommand{\ttdefault}{txtt}



\usepackage[Bjarne]{fncychap}
\usepackage[,numfigreset=2,mathnumfig]{sphinx}

\fvset{fontsize=auto}
\usepackage{geometry}


% Include hyperref last.
\usepackage{hyperref}
% Fix anchor placement for figures with captions.
\usepackage{hypcap}% it must be loaded after hyperref.
% Set up styles of URL: it should be placed after hyperref.
\urlstyle{same}


\usepackage{sphinxmessages}
\setcounter{tocdepth}{1}



\title{Anuket Reference Conformance for OpenStack (RC1)}
\date{Aug 02, 2022}
\release{}
\author{Anuket}
\newcommand{\sphinxlogo}{\vbox{}}
\renewcommand{\releasename}{}
\makeindex
\begin{document}

\pagestyle{empty}
\sphinxmaketitle
\pagestyle{plain}
\sphinxtableofcontents
\pagestyle{normal}
\phantomsection\label{\detokenize{index::doc}}



\section{Introduction}
\label{\detokenize{chapters/chapter01:introduction}}\label{\detokenize{chapters/chapter01::doc}}

\subsection{Synopsis}
\label{\detokenize{chapters/chapter01:synopsis}}
\sphinxAtStartPar
Ensure an implementation of the Anuket Reference Architecture (RA), such
as the Reference Implementation (RI), meets industry driven quality
assurance standards for conformance, verification and validation. The
OPNFV Verified Program (OVP), by Linux Foundation Networking (LFN),
overseen by the Compliance Verification Committee (CVC), will provide
tracking and governance for RC.

\sphinxAtStartPar
For the purpose of this chapter, NFVI+VNF testing will be performed to
evaluate \sphinxstylestrong{Conformance} (i.e. adherence) to, and demonstrated
proficiency with, all aspects of software delivery. More specifically,
Conformance includes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Verified implementations of NFVI+VNF match expected design
requirements

\item {} 
\sphinxAtStartPar
Clearly stated guidelines for test, badging, and lifecycle management

\item {} 
\sphinxAtStartPar
Inclusion of Operational run\sphinxhyphen{}books for 3rd party supplier
instantiation and validations of NFVI+VNF

\item {} 
\sphinxAtStartPar
Evidence, through test results, confirming delivered code matches
industrial requirements

\item {} 
\sphinxAtStartPar
Interoperability testing with Reference VNFs, ensuring integration
stability and life\sphinxhyphen{}cycle management of the Reference VNF on the
target implementation.

\end{itemize}

\sphinxAtStartPar
In summary, NFVI+VNF \sphinxstylestrong{Conformance} testing will be performed for
\sphinxstylestrong{Verification} and \sphinxstylestrong{Validation} purposes, defined further as:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Verification} will be used to indicate conformance to design
requirement specifications. Accomplished with Requirement
Traceability and Manifest Reviews to ensure the NFVI is delivered per
implementation specifications.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Validations} is used to indicate that testing performed confirms
the NFVI+VNF meets the expected, or desired outcome, or behaviour.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{All Terms utilized throughout this chapter are intended to align with
CVC definitions, and their use through CVC documentation, guidelines,
and standards.}


\subsection{Overview}
\label{\detokenize{chapters/chapter01:overview}}
\sphinxAtStartPar
\sphinxstylestrong{Chapter Purpose}

\sphinxAtStartPar
This chapter includes process flow, logistics, and requirements which
must be satisfied to ensure Network Function Virtualisation
Infrastructure (NFVI) meets the design, feature, and capability
expectations of RM and RA. Upstream projects will define
features/capabilities, test scenarios, and test cases which will be used
to augment OVP test harnesses for infrastructure verification purposes.
Existing processes, communication mediums, and related technologies will
be utilized where feasible. Ultimately, test results of certified
NFVI+VNF will reduce the amount of time and cost it takes each operator
to on\sphinxhyphen{}board and maintain vendor provided VNFs.

\sphinxAtStartPar
\sphinxstylestrong{Objective}

\sphinxAtStartPar
Perform NFVI+VNF Verification and Validations using Anuket reference
architecture, leveraging the existing Anuket and CVC Intake and
Validation Process to onboard and validate new test projects for NFVI
compliance. Upstream projects will define features/capabilities, test
scenarios, and test cases to augment existing OVP test harnesses to be
executed via the OVP Ecosystem.

\sphinxAtStartPar
\sphinxstylestrong{Test Methodology}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Verification test to make sure if the OpenStack services have been
deployed and configured correctly

\item {} 
\sphinxAtStartPar
Manifest Verifications (Termed Compliance by CVC) will ensure the
NFVI is compliant, and delivered for testing, with hardware and
software profile specifications defined by the RM and RA.

\item {} 
\sphinxAtStartPar
Empirical Validation with Reference Golden VNFs (Termed Validation by
CVC) will ensure the NFVI runs with a set of VNF Families, or
Classes, to mimic production\sphinxhyphen{}like VNF connectivity, for the purposes
of interoperability checks.

\item {} 
\sphinxAtStartPar
Candidate VNF Validation (Termed Validation \& Performance by CVC)
will ensure complete interoperablity of VNF behaviour on the NFVI
leverage VVP/VNFSDK test suites. Testing ensures VNF can be spun up,
modified, or removed, on the target NFVI (aka Interoperability).

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Different Distributions}

\sphinxAtStartPar
The three step methodology described above of verifying Manifest
compliance, executing Empirical Golden VNF transactions, and performing
Interoperability Testing is the same validation process regardless of
the Distribution used to establish a cloud topology, and the components
and services used in the client software stack.


\subsubsection{Terminology}
\label{\detokenize{chapters/chapter01:terminology}}
\sphinxAtStartPar
Terminology in this document will follow \sphinxhref{https://cntt.readthedocs.io/en/latest/common/glossary.html}{Glossary}.


\subsection{Scope}
\label{\detokenize{chapters/chapter01:scope}}
\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{rc1_scope}.png}
\caption{RC1 Scope}\label{\detokenize{chapters/chapter01:id1}}\end{figure}

\sphinxAtStartPar
This document covers the realisation aspects of conformance of both NFVI
and VNFs. The document will cover the following topics:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Identify in details the Requirements for conformance Framework.

\item {} 
\sphinxAtStartPar
Identify in details the Requirement of Test Cases (and mapping them
to requirements from The Reference Model and The OpenStack Based
Reference Architecture ).

\item {} 
\sphinxAtStartPar
Analysis of existing community projects.

\item {} 
\sphinxAtStartPar
Propose an E2E Framework for conformance of NFVI and VNFs.

\item {} 
\sphinxAtStartPar
Playbook of instructions, user manuals, steps of how to perform
verification and conformance for both NFVI and VNFs using the
proposed E2E Framework.

\item {} 
\sphinxAtStartPar
Gap analysis to identify where the Gaps are in the industry (tooling,
test cases, process, etc).

\item {} 
\sphinxAtStartPar
Identify development efforts needed to address any gaps identified.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Not in Scope}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Functional testing / validation of the application provided by the
VNF is outside the scope of this work.

\item {} 
\sphinxAtStartPar
ONAP is not used in the process flow for NFVI verifications, or
validations.

\item {} 
\sphinxAtStartPar
Upgrades to VNFs, and the respective processes of verifying upgrade
procedures and validating (testing) the success and compatibility of
upgrades is not in scope.

\end{itemize}


\subsection{Relation to other communities}
\label{\detokenize{chapters/chapter01:relation-to-other-communities}}
\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{rc1_relation}.png}
\caption{RC\sphinxhyphen{}1 Relations other communities}\label{\detokenize{chapters/chapter01:id2}}\end{figure}


\subsection{Principles and Guidelines}
\label{\detokenize{chapters/chapter01:principles-and-guidelines}}
\sphinxAtStartPar
The objectives of the verification program are to deliver a validated
implementation of reference architecture which satisfies infrastructure
needs for VNF\sphinxhyphen{}developer teams, leveraging the OVP ecosystem as the
vehicle for delivering validated NFVI.

\sphinxAtStartPar
These core principles will guide NFV verification deliverables:


\subsubsection{Overarching Objectives and Goals}
\label{\detokenize{chapters/chapter01:overarching-objectives-and-goals}}\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Deliver verified implementation of reference architecture which
satisfies infrastructure needs for VNF\sphinxhyphen{}developer teams.

\item {} 
\sphinxAtStartPar
All accomplished with augmentation to the current OVP ecosystem.

\item {} 
\sphinxAtStartPar
Increase probability VNFs will on\sphinxhyphen{}board and function with minimal
problems, or issues, during initial instantiation of VNF.

\item {} 
\sphinxAtStartPar
Test Harnesses will be portable, or compatible, across all
RAs/Distributions which already conform to standard interfaces and
services.

\end{enumerate}


\subsection{Best Practices}
\label{\detokenize{chapters/chapter01:best-practices}}
\sphinxAtStartPar
The following best practices have been adopted to ensure verification
and validation procedures are repeatable with consistent quality in test
results, and RI conformances:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Standardized test methodology / flow, Test Plan, and Test Case Suites

\item {} 
\sphinxAtStartPar
Integration with Anuket Upstream Projects and OVP flow (code, docs,
cert criteria, etc.)

\item {} 
\sphinxAtStartPar
Leverage Network and Service Models, with identified VNF\sphinxhyphen{}specific
parameters

\item {} 
\sphinxAtStartPar
Standardized conformance criteria

\item {} 
\sphinxAtStartPar
Define Anuket RA as scenarios, and have all test cases for the RA be
involved in OVP

\item {} 
\sphinxAtStartPar
Add test cases from operators, which operators already tested in
their environment

\end{itemize}


\subsection{Verification methodologies}
\label{\detokenize{chapters/chapter01:verification-methodologies}}
\sphinxAtStartPar
Perform VNF interoperability verifications against an implementation of
Anuket reference architecture, leveraging existing Anuket Intake
Process. Upstream projects will define features/capabilities, test
scenarios, and test cases to augment existing OVP test harnesses to be
executed via the OVP Ecosystem.

\sphinxAtStartPar
3rd Party test platforms may also be leveraged, if desired.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RC_certifying_methodlogy_25Nov2019}.jpg}
\caption{Conformance Methodology}\label{\detokenize{chapters/chapter01:id3}}\end{figure}


\subsection{Assumptions \& Dependencies}
\label{\detokenize{chapters/chapter01:assumptions-dependencies}}
\sphinxAtStartPar
\sphinxstylestrong{Assumptions} NFVI+VNF testing will be considered \sphinxstylestrong{Testable} if the
follow qualifiers are present in a test execution, and subsequent
result:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Ability to perform Conformance, or Verification of Artifacts to
ensure designs (RM/RA/RI) are delivered per specification

\item {} 
\sphinxAtStartPar
Ability to Control (or manipulate), manifestations of RM/RA/RI for
the purposes to adjust the test environment, and respective cases,
scenarios, and apparatus, to support actual test validations

\item {} 
\sphinxAtStartPar
Ability to monitor, measure, and report, Validations performed
against a target, controlled system under test

\end{itemize}

\sphinxAtStartPar
In addition, respective Entrance criteria is a prerequisite which needs
to be satisfied for NFVI+VNF to be considered \sphinxstylestrong{Testable}.

\sphinxAtStartPar
\sphinxstylestrong{Dependencies} NFVI+VNF verification will rely upon test harnesses,
test tools, and test suites provided by Anuket projects, including
dovetaill, yardstick, and Bottleneck. These upstream projects will be
reviewed semi\sphinxhyphen{}annually to verify they are still healthy and active
projects. Over time, the projects representing the conformance process
may change, but test parity is required if new test suites are added in
place of older, stale projects.
\begin{itemize}
\item {} 
\sphinxAtStartPar
NFVI+VNF verifications will be performed against well defined
instance types consisting of a HW and SW Profile, Configured Options,
and Applied Extensions (See image.)

\end{itemize}

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RC_NFVI_VNF_Instance_Type_25Nov2019}.jpg}
\caption{Instance Type}\label{\detokenize{chapters/chapter01:id4}}\end{figure}

\sphinxAtStartPar
\sphinxstylestrong{NFVI+VNF Instance Type:}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Standard compute flavours to be tested are defined in
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter04.html\#virtual-network-interface-specifications}{Virtual Network Interface Specifications}

\item {} 
\sphinxAtStartPar
Performance profiles come in the form of Basic, Network Intensive,
and Compute intensive. Refer to
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter02.html\#analysis}{Analysis}
for details on these profiles.

\end{itemize}


\subsection{Results Collation \& Presentation}
\label{\detokenize{chapters/chapter01:results-collation-presentation}}
\sphinxAtStartPar
Test suites will be categorized as functional or performance based.
Results reporting will be communicated as a boolean (pass/fail). The
pass/fail determination for performance\sphinxhyphen{}based test cases will be made by
comparing results against a baseline. Example performance\sphinxhyphen{}based metrics
include, but are not limited to: resource utilization, response times,
latency, and sustained throughput per second (TPS).

\sphinxAtStartPar
\sphinxstylestrong{Placeholder to document where results will be posted (e.g. Dovetail
dashboards.)}


\subsection{Governance}
\label{\detokenize{chapters/chapter01:governance}}\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Conformance badges will be presented by the CVC

\item {} 
\sphinxAtStartPar
CVC will maintain requirements for conformance

\end{enumerate}


\section{NFVI Conformance Requirements}
\label{\detokenize{chapters/chapter02:nfvi-conformance-requirements}}\label{\detokenize{chapters/chapter02::doc}}

\subsection{Synopsis}
\label{\detokenize{chapters/chapter02:synopsis}}
\sphinxAtStartPar
This chapter mainly covers the overall requirement for the Reference
Conformance test. The Conformance tests are conducted on the Cloud
Infrastructure and VNF level. Conformance on the Cloud Infrastructure
makes sure the SUT follows RM \& RA requirements, and conformance on VNF
makes sure that the VNF can be deployed and sufficiently work on the
Cloud Infrastructure that has passed the conformance test.

\sphinxAtStartPar
All Terms utilized throughout this chapter are intended to align with
CVC definitions, and their use through CVC documentation, guidelines,
and standards. This chapter will outline the Requirements, Process, and
Automation, needed to deliver the Cloud Infrastructure conformance.


\subsection{Introduction}
\label{\detokenize{chapters/chapter02:introduction}}
\sphinxAtStartPar
The Cloud Infrastructure refers to the physical and virtual resources
(compute, storage and network) on which virtual network functions (VNFs)
are deployed. Due to the differences in the API and under\sphinxhyphen{}layer
configuration and capability matrix, multiple vendor VNF deployments on
the shared Cloud Infrastructure becomes hard to predict, and requires s
amount of cross vendor interoperability tests. With the combined effort
from operators and vendors, define the RA and RM, that standardises the
under layer configuration, capability and API, so as to provide the
upper layer VNF with a ‘common cloud infrastructure’. Based on this,
Anuket also provides RC for conformance tests of SUT against RA \& RM
requirements. SUT passes the conformance test will be identified as NFVI
that can fit into the common requirements.

\sphinxAtStartPar
In the meantime, RC also provides conformance test for VNF. The
intention is to make sure VNF that passes RC test cases can be deployed
on any NFVI which also passes RC without any conformance and
interoperability issue.


\subsection{Methodology}
\label{\detokenize{chapters/chapter02:methodology}}
\sphinxAtStartPar
The Cloud Infrastructure is consumed or used by VNFs via APIs exposed by
Virtualised Infrastructure Manager (VIM). The resources created by VIM
on the NFVI use the underlying physical hardware (compute, storage and
network) either directly or indirectly. Anuket recommends RA1 to be used
as a reference architecture for the Cloud Infrastructure conformance.
This would provide a set of standard interfaces to create resources on
the Cloud Infrastructure. Below step by step process illustrates the
NFVI conformance methodology:
\begin{itemize}
\item {} 
\sphinxAtStartPar
SUT (Anuket RI1 or commercial NFVI) is deployed on hardware for
conformance test.

\item {} 
\sphinxAtStartPar
A set of tests run on SUT to determine the SUT readiness for
conformance process.

\item {} 
\sphinxAtStartPar
Golden KPIs are taken as a reference.

\item {} 
\sphinxAtStartPar
A set of tests are run on the SUT (target for conformance).

\item {} 
\sphinxAtStartPar
KPIs obtained from the SUT are collected and submitted to conformance
portal.

\item {} 
\sphinxAtStartPar
The SUT KPIs are reviewed and compared with Golden KPIs to determine
if the conformance badge is to be provided to SUT or not.

\end{itemize}

\sphinxAtStartPar
Based on a NFVI passing RC test and getting the conformance badge, VNF
conformance test can be further conducted. Such test will leverage
existing Anuket Intake Process. Upstream projects will define
features/capabilities, test scenarios, and test cases to augment
existing OVP test harnesses to be executed via the OVP Ecosystem.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RC_CertificationMethodology}.jpg}
\caption{Conformance Methodology}\label{\detokenize{chapters/chapter02:id1}}\end{figure}

\sphinxAtStartPar
Conformance methodologies to be implemented, from a process perspective
include:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Engineering package validations will be performed against targeted
infrastructure/architecture.

\item {} 
\sphinxAtStartPar
Configuration settings/features/capabilities will be baseline.

\item {} 
\sphinxAtStartPar
Entrance Criteria Guidelines will be satisfied prior to RC test
(i.e. Supplier needs to submit/agree/conform)
\begin{itemize}
\item {} 
\sphinxAtStartPar
Conform to Anuket RM

\item {} 
\sphinxAtStartPar
Conform to Anuket RA

\item {} 
\sphinxAtStartPar
Submit standard documentation

\item {} 
\sphinxAtStartPar
Adhere to security compliance

\end{itemize}

\item {} 
\sphinxAtStartPar
Exit Criteria Guidelines will be satisfied prior to issuance of
Anuket compliance badges.

\item {} 
\sphinxAtStartPar
Verification decisions will be based on data. Test harness is
compatible, or conforms to testing against standard interfaces and
services.

\item {} 
\sphinxAtStartPar
Leverage test harnesses from existing open source projects where
practical, and applicable.

\end{itemize}


\subsection{Conformance Strategy \& Vehicle}
\label{\detokenize{chapters/chapter02:conformance-strategy-vehicle}}
\sphinxAtStartPar
In order to begin the Conformance process, the Cloud Infrastructure
needs to be validated and expected to be in a required state. This state
would be determined by running tests as described in RI. Once the target
Cloud Infrastructure passes these tests, it would become a candidate for
the Cloud Infrastructure Conformance. If the Cloud Infrastructure fails
the tests, it will not be moved to the next workflow for Conformance.
The Cloud Infrastructure+VNF conformance consist of a three part process
for Compliance, Validation, and Performance. Adherence to Security
standards are equally important and addressed in
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter07.html}{Security}.

\sphinxAtStartPar
The three part conformance process includes NFVI Manifest conformance,
Empirical Baseline measurements against targeted VNF families, and
Candidate VNF validation. More specifically,
\begin{itemize}
\item {} 
\sphinxAtStartPar
NFVI conformance: NFVI is the SUT, ensuring NFVI is compliant with
specs of RM and RA accomplished with Manifest test

\item {} 
\sphinxAtStartPar
Empirical Validation with Reference VNF (Validation): NFVI is the
SUT, ensuring NFVI runs with Golden VNFs and is instrumented to
objectively validate resources through consumption and measurement

\item {} 
\sphinxAtStartPar
Candidate VNF Conformance (Validation \& Performance): VNF is the SUT,
ensuring VNFs operate with RM and RA leveraging VVP/CVP/VFN SDK Test
Suites

\item {} 
\sphinxAtStartPar
Security: Ensures NFVI+VNF is free from known security
vulnerabilities, utilizing industry standard cyber security
frameworks (Refer to Chapter 7 Security for additional
test/verification details) Validations are performed against an
Infrastructure Profile Catalog, VNF performance profile, and targeted
VNF class or family for baseline measurements.

\end{itemize}

\sphinxAtStartPar
The Infrastructure Profile Catalog contains the following attributes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Profile is a collection of (limited) options offered by the
infrastructure to the VNF
\begin{itemize}
\item {} 
\sphinxAtStartPar
Capabilities

\item {} 
\sphinxAtStartPar
Metrics

\item {} 
\sphinxAtStartPar
Compute flavors

\item {} 
\sphinxAtStartPar
Interface options

\item {} 
\sphinxAtStartPar
Storage extensions

\item {} 
\sphinxAtStartPar
Acceleration capabilities

\end{itemize}

\item {} 
\sphinxAtStartPar
Profiles are offered to VNFs as an instance types with predefined
compute flavors.
\begin{itemize}
\item {} 
\sphinxAtStartPar
A particular set of options is an instance type

\item {} 
\sphinxAtStartPar
Compute flavors: .tiny, .small etc as defined in
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter04.html\#profiles}{Profiles}

\end{itemize}

\item {} 
\sphinxAtStartPar
NFVI performance profiles, for which NFVI validations will support
and be verified against, are defined as basic and network intensive.
Details for each of these profiles can be found in
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter02.html\#analysis}{Analysis}.

\end{itemize}

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RM-ch04-node-profiles}.png}
\caption{NFVI Profiles}\label{\detokenize{chapters/chapter02:id2}}\end{figure}


\subsection{Profiles Reference}
\label{\detokenize{chapters/chapter02:profiles-reference}}
\sphinxAtStartPar
Different vendors have different types of VNFs to serve different
use\sphinxhyphen{}cases. A VNF like Broadband Network Gateway (BNG) would require high
networking throughput whereas a VNF like Mobility Management Entity
(MME) would require high computing performance. As such, BNG would
require high KPI values for network throughput and MME would require
high CPU performance KPIs like Index Score, Instructions Per Second
(IPS) etc. The target NFVI to cater these needs would have different
characteristics. Depending on VNF’s requirements, the NFVI can be
categorized into below profiles:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Basic (B) profile for standard computing and

\item {} 
\sphinxAtStartPar
Network intensive (N) profile offering predictable computing
performance along with low latency and high networking throughput
Similarly, different NFVI vendors may specialize in different
hardware profiles and some may specialize in both VNFs and NFVI.

\end{itemize}

\sphinxAtStartPar
To cater to different needs from multiple NFVI vendors, Anuket allows
different types of NFVI Conformance based on their types of
profile \sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter02.html\#analysis}{Analysis}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Certify Vendor NFVI Hardware solution: This allows for Conformance of
only NFVI.

\item {} 
\sphinxAtStartPar
Certify Vendor NFVI Hardware and Software Solution: This
allows for Conformance for NFVI running a particular VNF.

\end{itemize}


\subsection{Compliance, Verification, and Conformance}
\label{\detokenize{chapters/chapter02:compliance-verification-and-conformance}}
\sphinxAtStartPar
The below set of steps define the compliance, verification and
Conformance process for NFVI
\begin{itemize}
\item {} 
\sphinxAtStartPar
Based on VNF’s requirements, the Cloud Infrastructure profile is
selected \sphinxhyphen{} B, N

\item {} 
\sphinxAtStartPar
The Cloud Infrastructure readiness is checked for Conformance.

\item {} 
\sphinxAtStartPar
The test VNFs are on\sphinxhyphen{}boarded using automation scripts on the NFVI.

\item {} 
\sphinxAtStartPar
VNF on\sphinxhyphen{}boarding is validated by running functional tests to ensure
that the on\sphinxhyphen{}boarding is successful.

\item {} 
\sphinxAtStartPar
VNF performance tests are executed and NFVI KPIs are recorded during
the tests.

\item {} 
\sphinxAtStartPar
KPI comparison is run to compare NFVI KPIs with Golden KPIs, which
serve as a reference for NFVI Conformance.

\item {} 
\sphinxAtStartPar
If NFVI KPIs meet Golden KPIs, NFVI is certified and granted a
Conformance badge.

\item {} 
\sphinxAtStartPar
If NFVI KPIs do not meet Golden KPIs, no Conformance is provided.

\end{itemize}

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RC_Ref_NFVI_Profiles}.jpg}
\caption{Reference NFVI Profiles Implementation}\label{\detokenize{chapters/chapter02:id3}}\end{figure}


\subsection{Entry \& Exit Criteria}
\label{\detokenize{chapters/chapter02:entry-exit-criteria}}
\sphinxAtStartPar
\sphinxstylestrong{Entry criteria}: Before entering into NFVI Conformance, NFVI needs to
satisfy the following requirements as entry pass:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Design \& Requirements
\begin{itemize}
\item {} 
\sphinxAtStartPar
Design, Configuration, Features, SLAs, and Capability
documentation complete

\item {} 
\sphinxAtStartPar
Users stories / Adherence to Anuket Model principles and
guidelines

\item {} 
\sphinxAtStartPar
Chosen Reference Architecture matches the Architecture from the
product catalog

\end{itemize}

\item {} 
\sphinxAtStartPar
Environment
\begin{itemize}
\item {} 
\sphinxAtStartPar
Lab assets/resources and respective software revision levels are
specified, with confirmation of compatibility across external
systems

\item {} 
\sphinxAtStartPar
Tenant needs identified

\item {} 
\sphinxAtStartPar
All connectivity, network, image, VMs, delivered with successful
pairwise tests

\item {} 
\sphinxAtStartPar
Lab instrumented for proper monitoring

\item {} 
\sphinxAtStartPar
Lab needs to be setup according to RA1/RA2 as defined by Anuket
specifications and should be in the NFVI required state.

\end{itemize}

\item {} 
\sphinxAtStartPar
Planning \& Delivery
\begin{itemize}
\item {} 
\sphinxAtStartPar
Kickoff / Acceptance Criteria reviews performed

\item {} 
\sphinxAtStartPar
Delivery commitments, timelines, and cadence accepted

\item {} 
\sphinxAtStartPar
Confirm backward compatibility across software/flavor revision
levels

\end{itemize}

\item {} 
\sphinxAtStartPar
Data/VNFs/Security
\begin{itemize}
\item {} 
\sphinxAtStartPar
Images, Heat Templates, Preload Sheets available

\item {} 
\sphinxAtStartPar
Images uploaded to tenant space

\item {} 
\sphinxAtStartPar
External system test data needs identified

\item {} 
\sphinxAtStartPar
Owners (NFVI, VNF, PTL, etc) documented

\item {} 
\sphinxAtStartPar
Security Compliance Satisfied (Refer to Anuket specification
Chapter XXXX Security for additional tests, scans, and
vulnerabilities validations)

\end{itemize}

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Exit criteria}: NFVI Conformance testing should complete with
following exit criteria:
\begin{itemize}
\item {} 
\sphinxAtStartPar
All mandatory test cases should pass.

\item {} 
\sphinxAtStartPar
Test results collated, centralized, and normalized, with a final
report generated showing status of the test scenario/case (e.g. Pass,
Fail, Skip, Measurement Success/Fail, etc), along with trace\sphinxhyphen{}ability
to a functional, or non\sphinxhyphen{}functional, requirement.

\end{itemize}


\subsection{Framework Requirements}
\label{\detokenize{chapters/chapter02:framework-requirements}}
\sphinxAtStartPar
The NFVI Conformance framework deals with the process of testing NFVI in
below three areas:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Compliance: The Cloud Infrastructure needs to comply to Anuket
RA1/RA2.

\item {} 
\sphinxAtStartPar
Validation: Validation deals with the ability of NFVI to respond to
Cloud APIs and interfaces.

\item {} 
\sphinxAtStartPar
Performance: Performance deals with running tests on NFVI depending
on the NFVI profile and collecting KPIs.

\end{itemize}

\sphinxAtStartPar
The Cloud Infrastructure KPIs are compared with Golden KPIs, which serve
as a reference for the Cloud Infrastructure Conformance. If the Cloud
Infrastructure KPIs meet Golden KPIs, The Cloud Infrastructure is
certified and granted a Conformance badge. If the Cloud Infrastructure
KPIs do not meet Golden KPIs, no Conformance badge is provided.


\subsubsection{Best Practices (General)}
\label{\detokenize{chapters/chapter02:best-practices-general}}
\sphinxAtStartPar
The NFVI Conformance framework will be guided by the following core
principles:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Implementing, and adhering to, Standardized Test Methodology / flow,
Test Plan, and Test Case Suites, which promotes scalability using
repeatable processes.

\item {} 
\sphinxAtStartPar
Integration with Automated Tool\sphinxhyphen{}Chains, such as XTesting or Dovetail,
for continuous deployment, validation, and centralization of test
harnesses and results visualization.

\item {} 
\sphinxAtStartPar
Alliance and execution of OVP flows and methodologies, which supports
common structures for code, artifact generation and repository,
Conformance criteria, etc.)

\item {} 
\sphinxAtStartPar
Where possible, leveraging ONAP Network and Service Models, with
identified VNF\sphinxhyphen{}specific parameters

\item {} 
\sphinxAtStartPar
Utilizing Standard Conformance criteria.

\item {} 
\sphinxAtStartPar
Defining reference architecture (RA) as scenarios, and having all
test cases for the RA be involved in OVP

\item {} 
\sphinxAtStartPar
Add test cases from operators, which operators already tested in
their environment

\end{itemize}


\subsubsection{Testing}
\label{\detokenize{chapters/chapter02:testing}}
\sphinxAtStartPar
Testing for NFVI Conformance falls under three broad categories \sphinxhyphen{}
Compliance, Validation and Performance. Target NFVI for Conformance
needs to pass all these tests in order to obtain the Conformance badge.


\paragraph{Test Categories}
\label{\detokenize{chapters/chapter02:test-categories}}
\sphinxAtStartPar
The following five test categories have been identified as \sphinxstylestrong{minimal
testing required} to verify NFVI interoperability to satisfy the needs
of VNF developer teams.
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Baremetal validation: To validate control and compute nodes hardware

\item {} 
\sphinxAtStartPar
VNF Interoperability: After VNFs are on\sphinxhyphen{}boarded, Openstack resources
like Tenant, Network (L2/L3), CPU Pining, security policies, Affinity
anti\sphinxhyphen{}affinity roles and flavors etc. would be validated.

\item {} 
\sphinxAtStartPar
Compute components: Validate VMs status and connectivity result after
performing each of listed steps. Best candidate for this testing
would be identify compute node that holds VMs which has L2 and L3
connectivity.

\item {} 
\sphinxAtStartPar
Control plane components: Validations for RabbitMQ, Ceph, MariaDB
etc. and OpenStack components like Nova/Glance/Heat etc. APIs.

\item {} 
\sphinxAtStartPar
Security: Validation for use RBAC roles and user group policies. See
{\hyperref[\detokenize{chapters/chapter07:vnf-testing-cookbook}]{\sphinxcrossref{\DUrole{std,std-ref}{VNF Testing Cookbook}}}} for complete
list.

\end{enumerate}

\sphinxAtStartPar
The following \sphinxstylestrong{Optional Test Categories} which can be considered by
the Operator, or Supplier, for targeted validations to complement
required testing for Conformance:
\begin{itemize}
\item {} 
\sphinxAtStartPar
On\sphinxhyphen{}Boarding (MANO agnostic)

\item {} 
\sphinxAtStartPar
VNF Functional Testing

\item {} 
\sphinxAtStartPar
Charging / Revenue Assurance Verification

\item {} 
\sphinxAtStartPar
MicroServices Support

\item {} 
\sphinxAtStartPar
Closed Loop Testing

\item {} 
\sphinxAtStartPar
VNF Coexistence (ETSI NFV\sphinxhyphen{}TST001 “Noisy Neighbor”)

\item {} 
\sphinxAtStartPar
VNF Interactions with Extended NFVi Topology

\item {} 
\sphinxAtStartPar
VNF Interactions with Complex NFVi (Akraino)

\item {} 
\sphinxAtStartPar
Scalability Testing

\item {} 
\sphinxAtStartPar
HA Testing

\item {} 
\sphinxAtStartPar
Fault Recovery Testing

\item {} 
\sphinxAtStartPar
PM/KPI/Service Assurance Testing

\end{itemize}


\paragraph{Test Harnesses}
\label{\detokenize{chapters/chapter02:test-harnesses}}
\sphinxAtStartPar
In addition to General Best Practices for NFVI Conformance, the
following Quality Engineering (QE) standards will be applied when
defining and delivering test scenarios for Conformance:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Standardized test methodologies / flows capturing requirements from
RA’s, goals and scenarios for test execution, and normalizing test
results.

\item {} 
\sphinxAtStartPar
Establishing, and leveraging, working test\sphinxhyphen{}beds which can be
referenced in subsequent test scenario designs.

\item {} 
\sphinxAtStartPar
Leveraging standardized cloud\sphinxhyphen{}based facilities such as storage, IAM,
etc.

\item {} 
\sphinxAtStartPar
Test Script libraries need to enable Data\sphinxhyphen{}Driven testing of
On\sphinxhyphen{}Boarding, Instantiation, etc.

\item {} 
\sphinxAtStartPar
Standards base Test Plan and Test Case suite needs to include sample
VNFs, CSAR, and Automated Test Cases.

\item {} 
\sphinxAtStartPar
Documentation needs to be dynamic, and consumable.

\item {} 
\sphinxAtStartPar
Harnesses need to apply a “Just add Water” deployment strategy,
enabling test teams to readily implement test harnesses which
promotes Conformance scalability.

\end{enumerate}


\paragraph{Test Results}
\label{\detokenize{chapters/chapter02:test-results}}
\sphinxAtStartPar
\sphinxstylestrong{Categorization}. Test suites will be categorized as Functional or
Performance based.

\sphinxAtStartPar
\sphinxstylestrong{Results.} Test results reporting will be communicated as a boolean
(pass/fail), or Measurements Only.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Functional Pass/Fail} signals the assertions set in a test script
verify the Functional Requirements (FR) has met its stated objective
as delivered by the developer. This will consist of both positive
validation of expected behavior, as well as negative based testing
when to confirm error handling is working as expected.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Performance\sphinxhyphen{}based Pass/Fail} determination will be made by
comparing Non\sphinxhyphen{}Functional (NFR) NFVI KPIs (obtained after testing)
with the Golden KPIs. Some of the examples of performance KPIs
include, but not limited to: TCP bandwidth, UDP throughput, Memory
latency, Jitter, IOPS etc. See \sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter04.html}{Infrastructure Capabilities, Measurements and Catalogue}
for a complete list of metrics and requirements.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Measurement Results}. Baseline Measurements will be performed when
there are no benchmark standards to compare results, or established
FRs/NFRs for which to gauge application / platform behavior in an
integrated environment, or under load conditions. In these cases,
test results will be executed to measure the application, platform,
then prepare FRs/NFRs for subsequent enhancements and test runs.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Collation | Portal}. The following criteria will be applied to the
collation and presentation of test\sphinxhyphen{}runs seeking NFVI Conformance:
\begin{itemize}
\item {} 
\sphinxAtStartPar
RA number and name (e.g. RA\sphinxhyphen{}1 OpenStack)

\item {} 
\sphinxAtStartPar
Version of software tested (e.g. OpenStack Ocata)

\item {} 
\sphinxAtStartPar
Normalized results will be collated across all test runs
(i.e. centralized database)

\item {} 
\sphinxAtStartPar
Clear time stamps of test runs will be provided.

\item {} 
\sphinxAtStartPar
Identification of test engineer / executor.

\item {} 
\sphinxAtStartPar
Traceability to requirements.

\item {} 
\sphinxAtStartPar
Summarized conclusion if conditions warrant test Conformance (see
Badging Section).

\item {} 
\sphinxAtStartPar
Portal contains links to Conformance badge(s) received.

\end{itemize}


\subsubsection{Badging}
\label{\detokenize{chapters/chapter02:badging}}
\sphinxAtStartPar
\sphinxstylestrong{Defined}. \sphinxstyleemphasis{Badging} refers to the granting of a Conformance badge by
the OVP to Suppliers/Testers of Anuket NFVI upon demonstration the
testing performed confirms:
\begin{itemize}
\item {} 
\sphinxAtStartPar
NFVI adheres to Anuket RA/RM requirements.

\item {} 
\sphinxAtStartPar
Anuket certified VNFs functionally perform as expected (i.e. test
cases pass) on NFVI with acceptable levels of stability and
performance.

\end{itemize}

\sphinxAtStartPar
The below figure shows the targeted badge for NFVI.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{NFVI-badge}.jpg}
\caption{NFVI badge}\label{\detokenize{chapters/chapter02:id4}}\end{figure}

\sphinxAtStartPar
\sphinxstylestrong{Specifics}. More specifically, suppliers of NFVI testing seeking
infrastructure Conformance are required to furnish the following:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{OVP/CVC Expectation}\label{\detokenize{chapters/chapter02:id5}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{30}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
OVP/CVC Expectation
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Supporting Artifact(s)
\\
\hline
\sphinxAtStartPar
Lab
&
\sphinxAtStartPar
Verification that the delivered test lab conforms to RI\sphinxhyphen{}x lab
requirements for topology, \# of nodes, network fabric, etc
&
\sphinxAtStartPar
Bare\sphinxhyphen{}metal H/W Validations
\\
\hline
\sphinxAtStartPar
Compliance
&
\sphinxAtStartPar
Verification that the installed software conforms to RM/RA requirements
for required components and configured options and extensions, etc
&
\sphinxAtStartPar
Manifest S/W Validations
\\
\hline
\sphinxAtStartPar
Validation
&
\sphinxAtStartPar
FR Validation of Component and API functional behavior meets
requirements specified in RM/RA\sphinxhyphen{}x requirements documents
&
\sphinxAtStartPar
API \& Platform Test Results
\\
\hline
\sphinxAtStartPar
Performance
&
\sphinxAtStartPar
NFR Validation of Component, Interface, and API, results are within
tolerance, or achieve baseline measurements
&
\sphinxAtStartPar
Performance Test Results
\\
\hline
\sphinxAtStartPar
Results Reporting
&
\sphinxAtStartPar
Published of Test Results into centralized and common repository and
reporting portal
&
\sphinxAtStartPar
Normalized Results per Standards
\\
\hline
\sphinxAtStartPar
Release Notes
&
\sphinxAtStartPar
Supplier provides concluding remarks, links to artifacts, and
demonstration of having met exit criteria for testing
&
\sphinxAtStartPar
Release Notes
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Conformance Process}

\sphinxAtStartPar
Conformance and issuance of NFVI badges will be as follows:
\begin{itemize}
\item {} 
\sphinxAtStartPar
NFVI supplier utilizes, or installs a target RM/RA\sphinxhyphen{}x in a RI lab.

\item {} 
\sphinxAtStartPar
Required artifacts are submitted/supplied to the OVP, demonstrating
proper Lab Installation, Compliance, Validation, Performance, and
Release of Results \& Known Issues.

\item {} 
\sphinxAtStartPar
Artifact validations will be corroborated and confirmed by the OVP.
with direct comparison between measured results and documented
FRs/NFRs for applications, hardware and software configuration
settings, and host systems.

\item {} 
\sphinxAtStartPar
All OVP inquiries, requests for re\sphinxhyphen{}tests, or reformatting /
re\sphinxhyphen{}uploading of results data are closed.

\end{itemize}

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{NFVI_certifying_vendor_swhw_solutions}.jpg}
\caption{NFVI Badges}\label{\detokenize{chapters/chapter02:id6}}\end{figure}


\subsection{NFVI Test Cases Requirements}
\label{\detokenize{chapters/chapter02:nfvi-test-cases-requirements}}
\sphinxAtStartPar
The objective of this chapter is to describe the requirements for NFVI
test cases as derived from the reference model and architecture for the
LFN\sphinxhyphen{}based compliance program. This set of requirements eventually
determines the scope of the compliance program and the corresponding
list of test cases included in the compliance program. In particular,
this chapter extends the generic list of NFVI test case requirements
which is provided in Section Test Case Selection Requirements
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter08.html\#multi-cloud-interactions-model}{Multi\sphinxhyphen{}Cloud Interactions Model}
of the reference model.


\subsubsection{Generic Requirements on Test Cases}
\label{\detokenize{chapters/chapter02:generic-requirements-on-test-cases}}
\sphinxAtStartPar
All test cases must fulfill the generic requirements listed in Section
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter08.html\#multi-cloud-interactions-model}{Multi\sphinxhyphen{}Cloud Interactions Model}
of the reference model.

\sphinxAtStartPar
In addition, for test cases targeting the NFVI compliance program, the
following requirements must be met:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{NFVI compliance requirements}\label{\detokenize{chapters/chapter02:id7}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{30}{100}|\X{70}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
\\
\hline
\sphinxAtStartPar
x
&
\sphinxAtStartPar
All NFVI test cases \sphinxstyleemphasis{must} be automated. Once the pre\sphinxhyphen{}conditions of a
test case are met, i.e., the system under test is configured and in a
state according to the pre\sphinxhyphen{}conditions of the particular test case, no
manual steps must be required to run a test case to completion.
\\
\hline
\sphinxAtStartPar
x
&
\sphinxAtStartPar
All NFVI test cases \sphinxstyleemphasis{must} be implemented using publicly available open
source tools. This enables access to test tools and test case
implementations to all interested parties and organizations.
\\
\hline
\sphinxAtStartPar
x
&
\sphinxAtStartPar
All NFVI test cases \sphinxstyleemphasis{must} be integrated and run in the Anuket CI/CD
pipeline. This requirement ensures that test cases are functionally
correct, reliable, mature and pass on the NFVI reference
implementation.
\\
\hline
\sphinxAtStartPar
x
&
\sphinxAtStartPar
All NFVI test cases \sphinxstyleemphasis{must} treat the NFVI platform as a black box. In
particular, test cases must not perform actions on or change the state
of the system under test outside the scope of well\sphinxhyphen{}defined APIs as
listed by RA1. This requirement ensures applicability of test cases
across different implementations: reference implementations as well as
commercial implementations.
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Requirement Types}
\label{\detokenize{chapters/chapter02:requirement-types}}
\sphinxAtStartPar
The compliance and Conformance program intends to validate four
different types of requirements and system properties:
\begin{itemize}
\item {} 
\sphinxAtStartPar
API compliance: This is the most relevant type of test case,
validating the functional correctness of the system under test. API
compliance test cases exercise only the specific well\sphinxhyphen{}defined APIs
described in the reference architecture (see Interfaces and
APIs in \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}).

\item {} 
\sphinxAtStartPar
Performance: Test cases covering this type of requirement measure
specific performance characteristics of the system under test as
defined in the reference model, the corresponding reference
architectures and in sections further below in this chapter.

\item {} 
\sphinxAtStartPar
Resilience: Test cases covering this type of requirement measure
specific resilience characteristics of the system under test as
defined in the reference model, the corresponding reference
architectures and in sections further below in this chapter.

\item {} 
\sphinxAtStartPar
Hardware configuration: Validation of the bare\sphinxhyphen{}metal hardware itself
in terms of specs and configuration should be included in the scope
of the compliance test suite eventually. This validation step ensures
that the underlying hardware is correctly configured according to
Anuket hardware specification (TODO: add reference to updated “Pharos
specs”). The purpose of this validation is to act as a pre\sphinxhyphen{}flight
check before performing the extensive compliance test suite.
Moreover, by validating key hardware configuration aspects, it
ensures comparability of performance\sphinxhyphen{}related test results.

\end{itemize}

\sphinxAtStartPar
The extend to which these different types of requirements are included
in the compliance and Conformance test suite is subject to the
availability of test cases. See Section NFVI Test Cases
Requirements below.


\subsubsection{Profile Catalog}
\label{\detokenize{chapters/chapter02:profile-catalog}}
\sphinxAtStartPar
Section Infrastructure Profiles Catalogue
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter04.html\#profiles-and-workload-flavours}{Profiles and Workload Flavours}
of the reference model defines two software profiles, targeting two
different use cases:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Basic

\item {} 
\sphinxAtStartPar
Network intensive

\end{itemize}

\sphinxAtStartPar
The test cases selected for validating compliance of the two profiles
must cover the functional and non\sphinxhyphen{}functional requirements as listed in
Section Instance Capabilities Mapping
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter04.html\#virtual-network-interface-specifications}{Virtual Network Interface Specifications}
and Section Instance Performance Measurement Mapping in
\sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter04.html\#storage-extensions}{Storage Extensions}
of the reference model.

\sphinxAtStartPar
TODO: what actually needs to be done here is to reference the table from
chapter 4.2.5 and mark for which of those requirements test cases are
actually available in the set of test tools available to us.


\subsubsection{Software \& Hardware Reference}
\label{\detokenize{chapters/chapter02:software-hardware-reference}}
\sphinxAtStartPar
The LFN\sphinxhyphen{}based compliance and Conformance program comprises three
distinct types of NFVI deployment and runtime environments:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
A reference implementation deployed in the CI/CD environment,

\item {} 
\sphinxAtStartPar
A commercial NFVI product deployed in a vendor’s internal development
and testing environment, and

\item {} 
\sphinxAtStartPar
A reference implementation of a commercial NFVI product deployed in a
3rd party lab providing testing and Conformance services.

\end{enumerate}

\sphinxAtStartPar
The test tooling, harnesses and corresponding test cases which are part
of the compliance and Conformance test suite must be capable of running
across all of those environments. This results in the following list of
requirements:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Test case requirements}\label{\detokenize{chapters/chapter02:id8}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{30}{100}|\X{70}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
\\
\hline
\sphinxAtStartPar
x
&
\sphinxAtStartPar
NFVI test cases \sphinxstyleemphasis{must not} interact with remote (Internet) services
apart from downloading container or VM images. In particular, test
tools and test cases must not automatically upload test data to any
system or service run by LFN or GSMA. The purpose of this requirement
is to protect the confidentially of (intermediate) test data.
\\
\hline
\sphinxAtStartPar
x
&
\sphinxAtStartPar
NFVI test cases \sphinxstyleemphasis{must} support a means of running in an internal
enterprise lab environment. This could be achieved by either i)
natively supporting proxied Internet connectivity and non\sphinxhyphen{}public DNS
servers or ii) by providing a high\sphinxhyphen{}level description of remote
dependencies (e.g., container and VM images, network services (DNS),
etc.) such that local mirrors can be set up.
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Options \& Extensions}
\label{\detokenize{chapters/chapter02:options-extensions}}

\subsubsection{Measurement Criteria}
\label{\detokenize{chapters/chapter02:measurement-criteria}}
\sphinxAtStartPar
Test validations will be corroborated, and confirmed, with direct
comparison between measured results and documented non\sphinxhyphen{}functional
requirements (NFRs) for applications, hardware and software
configuration settings, and host systems. Throughput, latency,
concurrent connections/threads, are all examples of non\sphinxhyphen{}functional
requirements which specify criteria which can be used to judge the
operation of a system, rather than specific behavior of the application
which are defined by functional requirements.

\sphinxAtStartPar
This section attempts to summarize a categorical list of metrics used
for test validations. \sphinxstylestrong{For a complete list of metrics, and
requirements, please refer to Reference Model}


\paragraph{Storage and IOPS}
\label{\detokenize{chapters/chapter02:storage-and-iops}}
\sphinxAtStartPar
\sphinxstylestrong{IOPS} validations for Storage, and/or Storage Extensions, will be
included as part of the final NFVI verification, and validation,
process.

\sphinxAtStartPar
From a definition perspective, IOPS is the standard unit of measurement
for I/O (Input/Output) operations per second. This measurement is a
performance\sphinxhyphen{}based measurement and is usually seen written as\sphinxstylestrong{(1)}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Total IOPS}: Average number of I/O operations per second.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Read IOPS}: Average number of read I/O operations per second.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Write IOPS}: Average number of write I/O operations per second.

\end{itemize}

\sphinxAtStartPar
For example, if you have a disk that is capable of doing a 100 IOPS, it
means that it is theoretically capable of issuing a 100 read and or
write operations per second. This is in theory. In reality, additional
time is needed to actually process the 100 reads/writes. This additional
time is referred to as “latency”, which reduces the total IOPS that is
calculated, and measured. Latency needs needs to be measured, and
included in the IOPS calculation. Latency will tell us how long it takes
to process a single I/O request, and is generally in the 2 millisecond
(ms) range per IO operation for a physical disk, through 20+ ms, at
which time users will notice an impact in their experience\sphinxstylestrong{(2)}.

\sphinxAtStartPar
Additional factors to consider when measuring IOPS:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Take into consideration the percentage of Input (write) vs. Output
(reads) operations, as Writes can be more resource intensive.

\item {} 
\sphinxAtStartPar
Determine if Reads were performed from Cache, as this may (will)
result in faster performance, and faster IOPS.

\item {} 
\sphinxAtStartPar
Confirm the storage types (Physical, RAID), as storage arrays with
linear, or sequential reading/writing may (will) be slower.

\item {} 
\sphinxAtStartPar
Identify the block size used, as using large block sizes vs. small
block sizes can (will) impact IOPS performance.

\item {} 
\sphinxAtStartPar
Determine Hard Disk Speeds (HDD in RPMs) used, as the higher the
RPMS, the potential for faster IOPS performance.

\item {} 
\sphinxAtStartPar
Quantify the number of disk controllers used to process the number of
requested IO requests.

\item {} 
\sphinxAtStartPar
Determine the specific work\sphinxhyphen{}load requirements, as this will dictate
speed, controllers, disk RPM, and latency tolerances.

\end{itemize}

\sphinxAtStartPar
For additional insight, or deeper understanding and reading of IOPS,
refer to the references below.


\subsubsection{Measurement Types}
\label{\detokenize{chapters/chapter02:measurement-types}}

\paragraph{Performance Measurements}
\label{\detokenize{chapters/chapter02:performance-measurements}}
\sphinxAtStartPar
\sphinxstylestrong{Objectives}

\sphinxAtStartPar
The NFVI performance measurements aim at assessing the performance of a
given NFVI implementation on the execution plan (i.e., excluding VIM) by
providing it with a set of significant metrics to be measured.

\sphinxAtStartPar
They should allow validating the performance of any software and/or
hardware NFVI implementation as described in Reference Model.

\sphinxAtStartPar
Of course, they can also be used for other purposes, such as:
\begin{itemize}
\item {} 
\sphinxAtStartPar
fine tuning of software and/or hardware NFVI configuration (e.g., the
number of cores dedicated to the DPDK vSwitch)

\item {} 
\sphinxAtStartPar
comparing the performances of different software or hardware
technologies (e.g., DPDK vSwitch vs hardware\sphinxhyphen{}offloaded vSwitch)

\item {} 
\sphinxAtStartPar
assessing the performance impact of specific features (e.g., with or
without encapsulation)

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Metrics Baseline}

\sphinxAtStartPar
For the purpose of validation, a baseline of the performance metrics is
required for comparison with the results of their measurements on the
NFVI implementation to be validated.

\sphinxAtStartPar
That baseline is a set of threshold values which could be determined by
\sphinxstylestrong{measuring the performance metrics on Reference Implementations}.

\sphinxAtStartPar
The validation can then be based on simple pass/fail test results or on
a grade (e.g., “class” A, B or C) provided by the combination of
pass/fail results for 2 different threshold values of some (or all)
metrics.

\sphinxAtStartPar
\sphinxstylestrong{Metrics Description}

\sphinxAtStartPar
Two categories of metrics are considered depending on whether they are
related to either the VNF domain or the NFVI domain itself:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Metrics related to the VNF domain are defined from VNF perspective
(i.e., per VNFC, per vNIC, per vCPU…) and should concern VNF as well
as NFVI actors.

\item {} 
\sphinxAtStartPar
Metrics related to the NFVI domain are defined per NFVI node ; their
measurement is based on virtual workloads (i.e., VM or container) in
order to reflect the performance of a NFVI node with a given profile
; they should only concern NFVI actors.

\end{itemize}

\sphinxAtStartPar
The following table contains the list of performance metrics related to
the VNF domain.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Performance metrics related to the VNF domain}\label{\detokenize{chapters/chapter02:id9}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{100}|\X{20}{100}|\X{20}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Name
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Unit
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Definition/Notes
\\
\hline
\sphinxAtStartPar
vnf.nfvi.perf.001
&
\sphinxAtStartPar
vNIC throughput
&
\sphinxAtStartPar
bits/s
&
\sphinxAtStartPar
Throughput per vNIC
\\
\hline
\sphinxAtStartPar
vnf.nfvi.perf.002
&
\sphinxAtStartPar
vNIC latency
&
\sphinxAtStartPar
second
&
\sphinxAtStartPar
Frame transfer time to vNIC at the throughput (vnf.nfvi.perf.001)
\\
\hline
\sphinxAtStartPar
vnf.nfvi.perf.003
&
\sphinxAtStartPar
vNIC delay variation
&
\sphinxAtStartPar
second
&
\sphinxAtStartPar
Frame Delay Variation (FDV) to vNIC at the throughput
(vnf.nfvi.perf.001)
\\
\hline
\sphinxAtStartPar
vnf.nfvi.perf.004
&
\sphinxAtStartPar
vNIC simultaneous active flows
&
\sphinxAtStartPar
number
&
\sphinxAtStartPar
Simultaneous active L3/L4 flows per vNIC before a new flow is dropped
\\
\hline
\sphinxAtStartPar
vnf.nfvi.perf.005
&
\sphinxAtStartPar
vNIC new flows rate
&
\sphinxAtStartPar
flows/s
&
\sphinxAtStartPar
New L3/L4 flows rate per vNIC
\\
\hline
\sphinxAtStartPar
vnf.nfvi.perf.006
&
\sphinxAtStartPar
Storage throughput
&
\sphinxAtStartPar
bytes/s
&
\sphinxAtStartPar
Throughput per virtual storage unit
\\
\hline
\sphinxAtStartPar
vnf.nfvi.perf.007
&
\sphinxAtStartPar
vCPU capacity
&
\sphinxAtStartPar
test\sphinxhyphen{}specifics core
&
\sphinxAtStartPar
Compute capacity per vCPU
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
The following table contains the list of performance metrics related to
the NFVI domain.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Performance metrics related to the NFVI domain}\label{\detokenize{chapters/chapter02:id10}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{100}|\X{20}{100}|\X{20}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Name
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Unit
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Definition/Notes
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.001
&
\sphinxAtStartPar
Node network throughput
&
\sphinxAtStartPar
bits/s
&
\sphinxAtStartPar
Network throughput per node
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.002
&
\sphinxAtStartPar
Node simultaneous active flows
&
\sphinxAtStartPar
number
&
\sphinxAtStartPar
Simultaneous active L3/L4 flows per node before a new flow is dropped
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.003
&
\sphinxAtStartPar
Node new flows rate
&
\sphinxAtStartPar
flows/s
&
\sphinxAtStartPar
New L3/L4 flows rate per node
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.004
&
\sphinxAtStartPar
Node storage throughput
&
\sphinxAtStartPar
bytes/s
&
\sphinxAtStartPar
Storage throughput per node
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.005
&
\sphinxAtStartPar
Physical core capacity
&
\sphinxAtStartPar
test\sphinxhyphen{}specifics core
&
\sphinxAtStartPar
Compute capacity per physical core usable by VNFs
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.006
&
\sphinxAtStartPar
Energy consumption
&
\sphinxAtStartPar
W
&
\sphinxAtStartPar
Energy consumption of the node without hosting any VNFC
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.007
&
\sphinxAtStartPar
Network energy efficiency
&
\sphinxAtStartPar
W/bits/s
&
\sphinxAtStartPar
Energy consumption of the node at the network throughput,
(infra.nfvi.perf.001), normalized to the measured bit rate
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.008
&
\sphinxAtStartPar
Storage energy efficiency
&
\sphinxAtStartPar
W/bits/s
&
\sphinxAtStartPar
Energy consumption of the node at the storage throughput
(infra.nfvi.perf.004), normalized to the measured byte rate
\\
\hline
\sphinxAtStartPar
infra.nfvi.perf.009
&
\sphinxAtStartPar
Compute energy efficiency
&
\sphinxAtStartPar
W/core
&
\sphinxAtStartPar
Energy consumption of the node during compute capacity test
(vnf.nfvi.perf.007 or infra.nfvi.perf.005), normalized to the number of
physical cores usable by VNFs
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{MVP Metrics}

\sphinxAtStartPar
The following metrics should be considered as MVP:
\begin{itemize}
\item {} 
\sphinxAtStartPar
vnf.nfvi.perf.001,002,006,007

\item {} 
\sphinxAtStartPar
infra.nfvi.perf.001,005,006,007,009

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Network Metrics Measurement Test Cases}

\sphinxAtStartPar
The network performance metrics are vnf.nfvi.perf.001\sphinxhyphen{}005 and
infra.nfvi.perf.001\sphinxhyphen{}003,006.

\sphinxAtStartPar
The different possible test cases are defined by each of the 3 following
test traffic conditions.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Test traffic path across NFVI}

\sphinxAtStartPar
3 traffic path topologies should be considered:
\begin{itemize}
\item {} 
\begin{DUlineblock}{0em}
\item[] \sphinxstylestrong{North/South traffic}, between VNFCs within a node and outside
NFVI
\item[] This can be provided by PVP test setup of ETSI GS NFV\sphinxhyphen{}TST009.
\end{DUlineblock}

\item {} 
\begin{DUlineblock}{0em}
\item[] \sphinxstylestrong{East/West intra\sphinxhyphen{}node traffic}, between VNFCs within a node
\item[] This can be provided by a V2V (Virtual\sphinxhyphen{}to\sphinxhyphen{}Virtual) test setup
and, in some cases, by PVVP test setup of ETSI GS NFV\sphinxhyphen{}TST009.
\end{DUlineblock}

\item {} 
\begin{DUlineblock}{0em}
\item[] \sphinxstylestrong{East/West inter\sphinxhyphen{}node traffic}, between VNFCs in different
nodes
\item[] This can be provided by VPV (Virtual\sphinxhyphen{}Physical\sphinxhyphen{}Virtual) test
setup and, in some cases, by PVVP test setup between 2 nodes.
\end{DUlineblock}

\end{itemize}

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Test traffic processing by NFVI}

\sphinxAtStartPar
Different processing complexity applicable to the traffic crossing
the NFVI should be considered, including especially (but not
exhaustively):
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{L2 processing} (Ethernet switching), possibly including VLAN
tagging/mapping and encapsulation (e.g., VXLAN)

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{L3 processing} (IP routing), possibly including L2 processing

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{L4 stateful processing} (e.g., FW, NAT, SFC), also including L3
processing

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Encryption} (e.g., IPSec ESP tunneling)

\end{itemize}

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Test traffic profile}

\sphinxAtStartPar
Two different test traffic profiles should be considered according to
the two VNF types that must be provided with network connectivity by
the NFVI.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Forwarded traffic} for L3/L4 forwarding VNF (e.g., PGW, FW)

\sphinxAtStartPar
It is based on ETSI GS NFV\sphinxhyphen{}TST009 and it should be:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{bidirectional UDP traffic} with \sphinxstylestrong{0.001\%} frame loss ratio,
\sphinxstylestrong{300B} average frame size, \sphinxstylestrong{10k} L3/L4 flows,

\item {} 
\sphinxAtStartPar
between a \sphinxstylestrong{traffic generator} and a \sphinxstylestrong{traffic receiver}
through a \sphinxstylestrong{L3 forwarding} pseudo\sphinxhyphen{}VNF with sufficient capacity
not to be the test bottleneck.

\end{itemize}

\sphinxAtStartPar
Latency and delay variation measurement should be the 99th
percentile of measured values for one\sphinxhyphen{}way frame transfer
(i.e. from generator to receiver).

\sphinxAtStartPar
The main Anuket test tools candidates for that purpose are
NFVbench and VSPerf.
\begin{quote}

\sphinxAtStartPar
\sphinxstylestrong{Note:}\sphinxstyleemphasis{to be studied whether additional frame sizes and
flows number should be considered}
\end{quote}

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Client\sphinxhyphen{}server traffic} for L4/L7 endpoint VNF (e.g., MME, CDN)

\sphinxAtStartPar
It should be:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{bidirectional TCP traffic} with \sphinxstylestrong{1400B} maximum frame
size, \sphinxstylestrong{5k} TCP sessions,

\item {} 
\sphinxAtStartPar
between \sphinxstylestrong{2 TCP client\&server endpoints}, one or both as
pseudo\sphinxhyphen{}VNF, with sufficient capacity not to be the test
bottleneck.

\end{itemize}

\sphinxAtStartPar
\sphinxstyleemphasis{Note}: the maximum TCP frame size can be forced by configuring
TCP endpoint link MTU.

\sphinxAtStartPar
The main Anuket test tool candidate for that purpose is Functest
(VMTP and Shaker).
\begin{quote}

\sphinxAtStartPar
\sphinxstylestrong{Note:}\sphinxstyleemphasis{to be studied whether metrics related to latency
and flows for that traffic profile should be considered (how?
with UDP and/or ICMP test traffic in addition?)}
\end{quote}

\end{itemize}

\end{itemize}

\sphinxAtStartPar
The combination of each of those 3 test conditions types and the
different NFVI profiles results in a wide matrix of test cases
(potentially more than 50 cases). Furthermore, these test cases should
be combined with the different metrics resulting in a huge number of
measurements (potentially more than 400 measurements). For the
efficiency of the validation, only the most relevant combinations should
be kept.

\sphinxAtStartPar
This optimization should be based on the following principles:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
NFVI domain metrics measurement: on PVP topology only

\item {} 
\sphinxAtStartPar
Metrics measurement with forwarded traffic: with no L4 stateful
processing

\item {} 
\sphinxAtStartPar
Basic profile metrics measurement: client\sphinxhyphen{}server traffic profile only

\item {} 
\sphinxAtStartPar
Flows \& latency related metrics measurement: for PVP only

\end{enumerate}

\sphinxAtStartPar
The following table proposed a possible optimized matrix model of the
test cases against the metrics to be measured.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Optimized matrix model of the test cases against the metrics
             to be measured}\label{\detokenize{chapters/chapter02:id11}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{100}|\X{20}{100}|\X{10}{100}|\X{10}{100}|\X{10}{100}|\X{10}{100}|\X{10}{100}|\X{10}{100}|}
\hline
\sphinxstyletheadfamily &\sphinxstyletheadfamily 
\sphinxAtStartPar
NFVI Profiles
&\sphinxstyletheadfamily 
\sphinxAtStartPar
B
&\sphinxstyletheadfamily &\sphinxstyletheadfamily &\sphinxstyletheadfamily &\sphinxstyletheadfamily 
\sphinxAtStartPar
N
&\sphinxstyletheadfamily \\
\hline&
\sphinxAtStartPar
Test Cases
&
\sphinxAtStartPar
V2V \sphinxhyphen{} L2 \sphinxhyphen{} SRV
&
\sphinxAtStartPar
VPV \sphinxhyphen{} L3 \sphinxhyphen{} SRV
&
\sphinxAtStartPar
PVP \sphinxhyphen{} L2 \sphinxhyphen{} SRV
&
\sphinxAtStartPar
PVP \sphinxhyphen{} L4 \sphinxhyphen{} SRV
&
\sphinxAtStartPar
PVP \sphinxhyphen{} L2 \sphinxhyphen{} SRV
&
\sphinxAtStartPar
PVP \sphinxhyphen{} L2 \sphinxhyphen{} FWD
\\
\hline&&&&&&&\\
\hline
\sphinxAtStartPar
MVP Metrics
&
\sphinxAtStartPar
vnf.nfvi.perf.001
&
\sphinxAtStartPar
50Gbps
&
\sphinxAtStartPar
20Gbps
&
\sphinxAtStartPar
20Gbps
&
\sphinxAtStartPar
10Gbps
&
\sphinxAtStartPar
40Gbps
&
\sphinxAtStartPar
40Gbps
\\
\hline&
\sphinxAtStartPar
vnf.nfvi.perf.002
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
0.5ms
\\
\hline&
\sphinxAtStartPar
infra.nfvi.perf.001
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
40Gbps
&
\sphinxAtStartPar
20Gbps
&
\sphinxAtStartPar
60Gbps
&
\sphinxAtStartPar
80Gbps
\\
\hline&
\sphinxAtStartPar
infra.nfvi.perf.007
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
? W/Gbps
&
\sphinxAtStartPar
? W/Gbps
&
\sphinxAtStartPar
? W/Gbps
&
\sphinxAtStartPar
? W/Gbps
\\
\hline&&&&&&&\\
\hline
\sphinxAtStartPar
Non\sphinxhyphen{}MVP Metrics
&
\sphinxAtStartPar
vnf.nfvi.perf.003
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
1ms
\\
\hline&
\sphinxAtStartPar
vnf.nfvi.perf.004
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
500k
\\
\hline&
\sphinxAtStartPar
vnf.nfvi.perf.005
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
n/a (4)
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
110kfps
\\
\hline&
\sphinxAtStartPar
infra.nfvi.perf.002
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
1G
\\
\hline&
\sphinxAtStartPar
infra.nfvi.perf.003
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
n/a (1)
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
?
&
\sphinxAtStartPar
200kfps
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstyleemphasis{Table notes}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Values are only indicative (see “Metrics Baseline” below)

\item {} 
\sphinxAtStartPar
L2/L3/L4 refers to network processing layer
\begin{itemize}
\item {} 
\sphinxAtStartPar
L2 for Ethernet switching

\item {} 
\sphinxAtStartPar
L3 for IP routing

\item {} 
\sphinxAtStartPar
L4 for IP routing with L4 stateful processing (e.g. NAT)

\end{itemize}

\item {} 
\sphinxAtStartPar
SRV/FWD refers to the traffic profile (and pseudo\sphinxhyphen{}VNF type implied)
\begin{itemize}
\item {} 
\sphinxAtStartPar
SRV for client\sphinxhyphen{}server traffic (and L4/L7 endpoint pseudo\sphinxhyphen{}VNF)

\item {} 
\sphinxAtStartPar
FWD for forwarded traffic (and L3/L4 forwarding pseudo\sphinxhyphen{}VNF)

\end{itemize}

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Energy Metrics Measurement Test Cases}

\sphinxAtStartPar
Energy metrics (infra.nfvi.perf.006\sphinxhyphen{}009) should be considered carefully
for NFVI validation since energy consumption may vary a lot across
processor architectures, models and power management features.

\sphinxAtStartPar
They mainly enable to have metrics available regarding NFVI environment
footprint. They also allow energy\sphinxhyphen{}based comparison of different NFVI
software implementations running on a same physical NFVI hardware
implementation.

\sphinxAtStartPar
\sphinxstylestrong{Storage Metrics Measurement Test Cases}

\sphinxAtStartPar
Metric (MVP): vnf.nfvi.perf.006 and infra.nfvi.perf.004,008

\sphinxAtStartPar
\sphinxstylestrong{Compute Metrics Measurement Test Cases}

\sphinxAtStartPar
The compute performance metrics are vnf.nfvi.perf.007 and
infra.nfvi.perf.004,009.

\sphinxAtStartPar
For normalized results, the compute performance test requires all of the
possible vCPUs available for running workloads to execute workloads. You
need to start as many VMs as needed to force all of the possible CPUs on
the node to run a workload. In this case, the result is normalized:
\begin{itemize}
\item {} 
\sphinxAtStartPar
to the number of vCPU, for the vCPU capacity measurements
(vnf.nfvi.perf.007)

\item {} 
\sphinxAtStartPar
to the number of physical core usable by VNFs, for the physical core
capacity and compute energy efficiency measurements
infra.nfvi.perf.004,009)

\end{itemize}


\paragraph{Resiliency Measurements}
\label{\detokenize{chapters/chapter02:resiliency-measurements}}

\section{Cloud Infrastructure Test Cases and Traceability to Requirements}
\label{\detokenize{chapters/chapter03:cloud-infrastructure-test-cases-and-traceability-to-requirements}}\label{\detokenize{chapters/chapter03::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter03:introduction}}
\sphinxAtStartPar
The scope of this chapter is to identify and list down test cases based
on requirements defined in \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/index.html}{OpenStack based Reference Architecture}.
This will serve as traceability between test cases and requirements.

\sphinxAtStartPar
Note that each requirement may have one or more test cases associated
with it.

\sphinxAtStartPar
\sphinxstylestrong{must}: Test Cases that are marked as must are considered mandatory
and must pass successfully.

\sphinxAtStartPar
\sphinxstylestrong{should}: Test Cases that are marked as should are expected to be
fulfilled by NFVI but it is up to each service provider to accept an
NFVI targeting reference architecture that is not reflecting on any of
those requirements. The same applies to should not.

\sphinxAtStartPar
\sphinxstylestrong{may}: Test cases that are marked as may are considered optional. The
same applies to may not.


\subsection{Selection Criteria}
\label{\detokenize{chapters/chapter03:selection-criteria}}\begin{quote}

\sphinxAtStartPar
Test cases below are selected based on available test cases in
open\sphinxhyphen{}source tools like FuncTest etc.
\end{quote}


\subsection{Traceability Matrix}
\label{\detokenize{chapters/chapter03:traceability-matrix}}
\sphinxAtStartPar
The following is a Requirements Traceability Matrix (RTM) mapping Test
Case, and/or Test Case Coverage, to RM and RA\sphinxhyphen{}1 requirements (config and
deployment).

\sphinxAtStartPar
The RTM contains RM config (i.e. .conf) requirements listed “per
profile”, followed by RA\sphinxhyphen{}1 requirements. Requirements fall into 8
domains: general(gen), infrastructure(inf), VIM(vim), Interface \&
API(int), Tenants(tnt), LCM(lcm), Assurance(asr), Security(sec).

\sphinxAtStartPar
For detailed information on RM \& RA\sphinxhyphen{}1 NFVI and VNF requirements, please
refer to \sphinxhref{https://cntt.readthedocs.io/projects/ri1/en/latest/chapters/chapter03.html}{Cloud Infrastructure + VNF Target State \& Specification}.


\subsubsection{Architecture and OpenStack Requirements}
\label{\detokenize{chapters/chapter03:architecture-and-openstack-requirements}}

\subsubsection{Infrastructure}
\label{\detokenize{chapters/chapter03:infrastructure}}

\subsubsection{VIM}
\label{\detokenize{chapters/chapter03:vim}}

\subsubsection{Interfaces \& APIs}
\label{\detokenize{chapters/chapter03:interfaces-apis}}
\sphinxAtStartPar
The \sphinxhref{https://opendev.org/openstack/devstack-gate}{OpenStack Gates}
verify all changes proposed mostly by running thousands of Tempest tests
completed by Rally scenarios in a few cases. Skipping tests is allowed
in all OpenStack Gates and only failures rate the review \sphinxhyphen{}1 because of
the multiple capabilities and backends selected in the different Gate
jobs. The classical \sphinxhref{https://wiki.anuket.io/display/HOME/Functest+Wallaby}{Functest
containers}
conform to this model which also fits the heterogeneous user
deployments.

\sphinxAtStartPar
From a Anuket Compliance state point, the capabilities are well
described in \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
which allows tuning the test configurations and the test lists to avoid
skipping any test. It results that all tests covering optional
capabilities and all upstream skipped tests due to known bugs are not
executed. All remaining tests must be executed and must pass
successfully.

\sphinxAtStartPar
New \sphinxhref{https://lists.opnfv.org/g/opnfv-tsc/message/5717}{Functest
containers} have
been proposed for Anuket Compliance which simply override the default
test configurations and the default test lists. Any optional capability
or services (e.g. Barbican) can be still verified by the classical
Functest containers.

\sphinxAtStartPar
The next subsections only detail the Tempest tests which must not be
executed from a Compliance state point. The remaining tests have to pass
successfully. They cover all together the API testing requirements as
asked by \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}

\sphinxAtStartPar
The following software versions are considered here to verify OpenStack
Wallaby selected by Anuket:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Software versions}\label{\detokenize{chapters/chapter03:id1}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{60}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Software
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Version
\\
\hline
\sphinxAtStartPar
Functest
&
\sphinxAtStartPar
wallaby
\\
\hline
\sphinxAtStartPar
Cinder Tempest plugin
&
\sphinxAtStartPar
1.4.0
\\
\hline
\sphinxAtStartPar
Keystone Tempest plugin
&
\sphinxAtStartPar
0.7.0
\\
\hline
\sphinxAtStartPar
Heat Tempest plugin
&
\sphinxAtStartPar
1.2.0
\\
\hline
\sphinxAtStartPar
Neutron Tempest plugin
&
\sphinxAtStartPar
1.4.0
\\
\hline
\sphinxAtStartPar
Rally OpenStack
&
\sphinxAtStartPar
2.2.1.dev11
\\
\hline
\sphinxAtStartPar
Tempest
&
\sphinxAtStartPar
27.0.0
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Identity \sphinxhyphen{} Keystone API testing}
\label{\detokenize{chapters/chapter03:identity-keystone-api-testing}}
\sphinxAtStartPar
Keystone API is covered in the OpenStack Gates via
\sphinxhref{https://opendev.org/openstack/tempest}{Tempest} and
\sphinxhref{https://opendev.org/openstack/keystone-tempest-plugin}{keystone\sphinxhyphen{}tempest\sphinxhyphen{}plugin}
as integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}.

\sphinxAtStartPar
According to \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
the following test names must not be executed:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Keystone API testing}\label{\detokenize{chapters/chapter03:id2}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{60}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\sphinxAtStartPar
.*api.identity.v3.test\_oauth1\_tokens
&
\sphinxAtStartPar
oauth1
\\
\hline
\sphinxAtStartPar
.*scenario.test\_federated\_authentication
&
\sphinxAtStartPar
federation
\\
\hline
\sphinxAtStartPar
.*identity.admin.v2
&
\sphinxAtStartPar
API v2
\\
\hline
\sphinxAtStartPar
.*identity.v2
&
\sphinxAtStartPar
API v2
\\
\hline
\sphinxAtStartPar
.*identity.v3.test\_access\_rules
&
\sphinxAtStartPar
access\_rules
\\
\hline
\sphinxAtStartPar
.*identity.v3.test\_application\_credentials.\textbackslash{}

\sphinxAtStartPar
ApplicationCredentialsV3Test.\textbackslash{}

\sphinxAtStartPar
test\_create\_application\_credential\_access\_rules
&
\sphinxAtStartPar
access\_rules
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Keystone API is also covered by
\sphinxhref{https://opendev.org/openstack/rally}{Rally}.

\sphinxAtStartPar
Here are the mainline tasks integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Authenticate.keystone

\item {} 
\sphinxAtStartPar
KeystoneBasic.add\_and\_remove\_user\_role

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_add\_and\_list\_user\_roles

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_and\_list\_tenants

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_and\_delete\_role

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_and\_delete\_service

\item {} 
\sphinxAtStartPar
KeystoneBasic.get\_entities

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_update\_and\_delete\_tenant

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_user

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_tenant

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_and\_list\_users

\item {} 
\sphinxAtStartPar
KeystoneBasic.create\_tenant\_with\_users

\end{itemize}


\paragraph{Image \sphinxhyphen{} Glance API testing}
\label{\detokenize{chapters/chapter03:image-glance-api-testing}}
\sphinxAtStartPar
Glance API is covered in the OpenStack Gates via
\sphinxhref{https://opendev.org/openstack/tempest}{Tempest} as integrated in
\sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml}{Functest Smoke
CNTT}.

\sphinxAtStartPar
According to \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
the following test names must not be executed:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Glance API testing}\label{\detokenize{chapters/chapter03:id3}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{60}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\sphinxAtStartPar
.*image.v1
&
\sphinxAtStartPar
API v1
\\
\hline
\sphinxAtStartPar
.*image.v2.admin.test\_images.ImportCopyImagesTest
&
\sphinxAtStartPar
import\_image
\\
\hline
\sphinxAtStartPar
.*image.v2.test\_images\_negative.ImagesNegativeTest.\textbackslash{}

\sphinxAtStartPar
test\_create\_image\_reserved\_property
&
\sphinxAtStartPar
os\_glance\_reserved
\\
\hline
\sphinxAtStartPar
.*image.v2.test\_images\_negative.ImagesNegativeTest.\textbackslash{}

\sphinxAtStartPar
test\_update\_image\_reserved\_property
&
\sphinxAtStartPar
os\_glance\_reserved
\\
\hline
\sphinxAtStartPar
.*image.v2.test\_images\_negative.ImportImagesNegativeTest.\textbackslash{}

\sphinxAtStartPar
test\_image\_web\_download\_import\_with\_bad\_url
&
\sphinxAtStartPar
web\sphinxhyphen{}downloadimport
\\
\hline
\sphinxAtStartPar
.*image.v2.test\_images.ImportImagesTest
&
\sphinxAtStartPar
import\_image
\\
\hline
\sphinxAtStartPar
.*image.v2.test\_images.MultiStoresImportImages
&
\sphinxAtStartPar
import\_image
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Glance API is also covered by
\sphinxhref{https://opendev.org/openstack/rally}{Rally}.

\sphinxAtStartPar
Here are the mainline tasks integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Authenticate.validate\_glance

\item {} 
\sphinxAtStartPar
GlanceImages.create\_and\_delete\_image

\item {} 
\sphinxAtStartPar
GlanceImages.create\_and\_list\_image

\item {} 
\sphinxAtStartPar
GlanceImages.list\_images

\item {} 
\sphinxAtStartPar
GlanceImages.create\_image\_and\_boot\_instances

\end{itemize}


\paragraph{Block Storage \sphinxhyphen{} Cinder API testing}
\label{\detokenize{chapters/chapter03:block-storage-cinder-api-testing}}
\sphinxAtStartPar
Cinder API is covered in the OpenStack Gates via
\sphinxhref{https://opendev.org/openstack/tempest}{Tempest} and
\sphinxhref{https://opendev.org/openstack/cinder-tempest-plugin}{cinder\sphinxhyphen{}tempest\sphinxhyphen{}plugin}
as integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}.

\sphinxAtStartPar
According to \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
the following test names must not be executed:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Cinder API testing}\label{\detokenize{chapters/chapter03:id4}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{60}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\sphinxAtStartPar
.*test\_incremental\_backup
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/68881}
\\
\hline
\sphinxAtStartPar
.*test\_consistencygroups
&
\sphinxAtStartPar
consistency\_group
\\
\hline
\sphinxAtStartPar
.*test\_backup\_crossproject\_admin\_negative
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/71011}
\\
\hline
\sphinxAtStartPar
.*test\_backup\_crossproject\_user\_negative
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/71011}
\\
\hline
\sphinxAtStartPar
.*test\_volume\_encrypted.TestEncryptedCinderVolumes
&
\sphinxAtStartPar
attach\_encrypted\_volume
\\
\hline
\sphinxAtStartPar
.*test\_encrypted\_volumes\_extend
&
\sphinxAtStartPar
extend\_attached\_encrypted\_volume
\\
\hline
\sphinxAtStartPar
.*test\_group\_snapshots.GroupSnapshotsV319Test.\textbackslash{}

\sphinxAtStartPar
test\_reset\_group\_snapshot\_status
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1770179}
\\
\hline
\sphinxAtStartPar
.*test\_multi\_backend
&
\sphinxAtStartPar
multi\sphinxhyphen{}backend
\\
\hline
\sphinxAtStartPar
.*test\_volume\_retype.VolumeRetypeWithMigrationTest
&
\sphinxAtStartPar
multi\sphinxhyphen{}backend
\\
\hline
\sphinxAtStartPar
.*test\_volume\_delete\_cascade.VolumesDeleteCascade.\textbackslash{}

\sphinxAtStartPar
test\_volume\_from\_snapshot\_cascade\_delete
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1677525}
\\
\hline
\sphinxAtStartPar
.*test\_volumes\_backup.VolumesBackupsTest.\textbackslash{}

\sphinxAtStartPar
test\_volume\_backup\_create\_get\_detailed\_list\_restore\_delete
&
\sphinxAtStartPar
ceph
\\
\hline
\sphinxAtStartPar
.*test\_volumes\_extend.VolumesExtendAttachedTest.\textbackslash{}

\sphinxAtStartPar
test\_extend\_attached\_volume
&
\sphinxAtStartPar
extend\_attached\_volume
\\
\hline
\sphinxAtStartPar
.*tempest.scenario.test\_volume\_migrate\_attached
&
\sphinxAtStartPar
multi\sphinxhyphen{}backend
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Cinder API is also covered by
\sphinxhref{https://opendev.org/openstack/rally}{Rally}.

\sphinxAtStartPar
Here are the mainline tasks integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Authenticate.validate\_cinder

\item {} 
\sphinxAtStartPar
CinderVolumes.create\_and\_delete\_snapshot

\item {} 
\sphinxAtStartPar
CinderVolumes.create\_and\_delete\_volume

\item {} 
\sphinxAtStartPar
CinderVolumes.create\_and\_extend\_volume

\item {} 
\sphinxAtStartPar
CinderVolumes.create\_from\_volume\_and\_delete\_volume

\item {} 
\sphinxAtStartPar
CinderQos.create\_and\_list\_qos

\item {} 
\sphinxAtStartPar
CinderQos.create\_and\_set\_qos

\item {} 
\sphinxAtStartPar
CinderVolumeTypes.create\_and\_list\_volume\_types

\item {} 
\sphinxAtStartPar
CinderVolumeTypes.create\_volume\_type\_and\_encryption\_type

\item {} 
\sphinxAtStartPar
Quotas.cinder\_update\_and\_delete

\item {} 
\sphinxAtStartPar
Quotas.cinder\_update

\end{itemize}


\paragraph{Object Storage \sphinxhyphen{} Swift API testing}
\label{\detokenize{chapters/chapter03:object-storage-swift-api-testing}}
\sphinxAtStartPar
Swift API is covered in the OpenStack Gates via
\sphinxhref{https://opendev.org/openstack/tempest}{Tempest} as integrated in
\sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}.

\sphinxAtStartPar
According to \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
the following test names must not be executed:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Swift API testing}\label{\detokenize{chapters/chapter03:id5}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{60}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\sphinxAtStartPar
.*test\_container\_sync.ContainerSyncTest.\textbackslash{}

\sphinxAtStartPar
test\_container\_synchronization
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1317133}
\\
\hline
\sphinxAtStartPar
.*test\_container\_sync\_middleware.ContainerSyncMiddlewareTest.\textbackslash{}

\sphinxAtStartPar
test\_container\_synchronization
&
\sphinxAtStartPar
container\_sync
\\
\hline
\sphinxAtStartPar
.*test\_object\_services.ObjectTest.\textbackslash{}

\sphinxAtStartPar
test\_create\_object\_with\_transfer\_encoding
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1905432}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Swift API is also covered by
\sphinxhref{https://opendev.org/openstack/rally}{Rally}.

\sphinxAtStartPar
Here are the mainline tasks integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
SwiftObjects.create\_container\_and\_object\_then\_list\_objects

\item {} 
\sphinxAtStartPar
SwiftObjects.list\_objects\_in\_containers

\item {} 
\sphinxAtStartPar
SwiftObjects.create\_container\_and\_object\_then\_download\_object

\item {} 
\sphinxAtStartPar
SwiftObjects.create\_container\_and\_object\_then\_delete\_all

\item {} 
\sphinxAtStartPar
SwiftObjects.list\_and\_download\_objects\_in\_containers

\end{itemize}


\paragraph{Networking \sphinxhyphen{} Neutron API testing}
\label{\detokenize{chapters/chapter03:networking-neutron-api-testing}}
\sphinxAtStartPar
Neutron API is covered in the OpenStack Gates via
\sphinxhref{https://opendev.org/openstack/tempest}{Tempest} and
\sphinxhref{https://opendev.org/openstack/neutron-tempest-plugin}{neutron\sphinxhyphen{}tempest\sphinxhyphen{}plugin}
as integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}.

\sphinxAtStartPar
According to \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
the following test names must not be executed:


\begin{savenotes}\sphinxatlongtablestart\begin{longtable}[c]{|\X{60}{100}|\X{40}{100}|}
\sphinxthelongtablecaptionisattop
\caption{Neutron API testing\strut}\label{\detokenize{chapters/chapter03:id6}}\\*[\sphinxlongtablecapskipadjust]
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\endfirsthead

\multicolumn{2}{c}%
{\makebox[0pt]{\sphinxtablecontinued{\tablename\ \thetable{} \textendash{} continued from previous page}}}\\
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\endhead

\hline
\multicolumn{2}{r}{\makebox[0pt][r]{\sphinxtablecontinued{continues on next page}}}\\
\endfoot

\endlastfoot

\sphinxAtStartPar
.*admin.test\_agent\_availability\_zone
&
\sphinxAtStartPar
DHCP agent and L3 agent
\\
\hline
\sphinxAtStartPar
.*admin.test\_dhcp\_agent\_scheduler
&
\sphinxAtStartPar
dhcp\_agent\_scheduler
\\
\hline
\sphinxAtStartPar
.*admin.test\_l3\_agent\_scheduler
&
\sphinxAtStartPar
l3\_agent\_scheduler
\\
\hline
\sphinxAtStartPar
.*admin.test\_logging
&
\sphinxAtStartPar
logging
\\
\hline
\sphinxAtStartPar
.*admin.test\_logging\_negative
&
\sphinxAtStartPar
logging
\\
\hline
\sphinxAtStartPar
.*admin.test\_network\_segment\_range
&
\sphinxAtStartPar
network\sphinxhyphen{}segment\sphinxhyphen{}range
\\
\hline
\sphinxAtStartPar
.*admin.test\_ports.PortTestCasesAdmin.\textbackslash{}

\sphinxAtStartPar
test\_regenerate\_mac\_address
&
\sphinxAtStartPar
port\sphinxhyphen{}mac\sphinxhyphen{}address\sphinxhyphen{}regenerate
\\
\hline
\sphinxAtStartPar
.*admin.test\_ports.PortTestCasesResourceRequest
&
\sphinxAtStartPar
port\sphinxhyphen{}resource\sphinxhyphen{}request
\\
\hline
\sphinxAtStartPar
.*admin.test\_routers\_dvr
&
\sphinxAtStartPar
dvr
\\
\hline
\sphinxAtStartPar
.*admin.test\_routers\_flavors
&
\sphinxAtStartPar
l3\sphinxhyphen{}flavors
\\
\hline
\sphinxAtStartPar
.*admin.test\_routers\_ha
&
\sphinxAtStartPar
l3\sphinxhyphen{}ha
\\
\hline
\sphinxAtStartPar
.*test\_floating\_ips.FloatingIPPoolTestJSON
&
\sphinxAtStartPar
floatingip\sphinxhyphen{}pools
\\
\hline
\sphinxAtStartPar
.*test\_floating\_ips.FloatingIPTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_update\_floatingip\_port\_details
&
\sphinxAtStartPar
fip\sphinxhyphen{}port\sphinxhyphen{}details
\\
\hline
\sphinxAtStartPar
.*test\_metering\_extensions
&
\sphinxAtStartPar
metering
\\
\hline
\sphinxAtStartPar
.*test\_metering\_negative
&
\sphinxAtStartPar
metering
\\
\hline
\sphinxAtStartPar
.*test\_networks.NetworksSearchCriteriaTest.\textbackslash{}

\sphinxAtStartPar
test\_list\_validation\_filters
&
\sphinxAtStartPar
filter\sphinxhyphen{}validation
\\
\hline
\sphinxAtStartPar
.*test\_networks.NetworksTestAdmin.\textbackslash{}

\sphinxAtStartPar
test\_create\_tenant\_network\_vxlan
&
\sphinxAtStartPar
vxlan
\\
\hline
\sphinxAtStartPar
.*test\_networks.NetworksTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_update\_network\_dns\_domain
&
\sphinxAtStartPar
dns\sphinxhyphen{}integration
\\
\hline
\sphinxAtStartPar
.*test\_port\_forwardings
&
\sphinxAtStartPar
floating\sphinxhyphen{}ip\sphinxhyphen{}port\sphinxhyphen{}forwarding
\\
\hline
\sphinxAtStartPar
.*test\_port\_forwarding\_negative
&
\sphinxAtStartPar
floating\sphinxhyphen{}ip\sphinxhyphen{}port\sphinxhyphen{}forwarding
\\
\hline
\sphinxAtStartPar
.*test\_ports.PortsTaggingOnCreation
&
\sphinxAtStartPar
tag\sphinxhyphen{}ports\sphinxhyphen{}during\sphinxhyphen{}bulk\sphinxhyphen{}creation
\\
\hline
\sphinxAtStartPar
.*test\_ports.PortsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_port\_with\_propagate\_uplink\_status
&
\sphinxAtStartPar
uplink\sphinxhyphen{}status\sphinxhyphen{}propagation
\\
\hline
\sphinxAtStartPar
.*test\_ports.PortsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_port\_without\_propagate\_uplink\_status
&
\sphinxAtStartPar
uplink\sphinxhyphen{}status\sphinxhyphen{}propagation
\\
\hline
\sphinxAtStartPar
.*test\_ports.PortsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_update\_port\_with\_dns\_domain
&
\sphinxAtStartPar
dns\sphinxhyphen{}domain\sphinxhyphen{}ports
\\
\hline
\sphinxAtStartPar
.*test\_ports.PortsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_update\_port\_with\_dns\_name
&
\sphinxAtStartPar
dns\sphinxhyphen{}integration
\\
\hline
\sphinxAtStartPar
.*test\_ports.PortsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_update\_port\_with\_no\_dns\_name
&
\sphinxAtStartPar
dns\sphinxhyphen{}integration
\\
\hline
\sphinxAtStartPar
.*test\_revisions.TestRevisions.\textbackslash{}

\sphinxAtStartPar
test\_update\_dns\_domain\_bumps\_revision
&
\sphinxAtStartPar
dns\sphinxhyphen{}integration
\\
\hline
\sphinxAtStartPar
.*test\_revisions.TestRevisions.\textbackslash{}

\sphinxAtStartPar
test\_update\_router\_extra\_attributes\_\textbackslash{}

\sphinxAtStartPar
bumps\_revision
&
\sphinxAtStartPar
l3\sphinxhyphen{}ha
\\
\hline
\sphinxAtStartPar
.*test\_router\_interface\_fip
&
\sphinxAtStartPar
router\sphinxhyphen{}interface\sphinxhyphen{}fip
\\
\hline
\sphinxAtStartPar
.*test\_routers.DvrRoutersTest
&
\sphinxAtStartPar
dvr
\\
\hline
\sphinxAtStartPar
.*test\_routers.HaRoutersTest
&
\sphinxAtStartPar
l3\sphinxhyphen{}ha
\\
\hline
\sphinxAtStartPar
.*test\_routers.RoutersIpV6Test.\textbackslash{}

\sphinxAtStartPar
test\_extra\_routes\_atomic
&
\sphinxAtStartPar
extraroute\sphinxhyphen{}atomic
\\
\hline
\sphinxAtStartPar
.*test\_routers.RoutersTest.\textbackslash{}

\sphinxAtStartPar
test\_extra\_routes\_atomic
&
\sphinxAtStartPar
extraroute\sphinxhyphen{}atomic
\\
\hline
\sphinxAtStartPar
.*test\_routers\_negative.DvrRoutersNegativeTest
&
\sphinxAtStartPar
dvr
\\
\hline
\sphinxAtStartPar
.*test\_routers\_negative.\textbackslash{}

\sphinxAtStartPar
DvrRoutersNegativeTestExtended
&
\sphinxAtStartPar
dvr
\\
\hline
\sphinxAtStartPar
.*test\_routers\_negative.HaRoutersNegativeTest
&
\sphinxAtStartPar
l3\sphinxhyphen{}ha
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups.RbacSharedSecurityGroupTest
&
\sphinxAtStartPar
rbac\sphinxhyphen{}security\sphinxhyphen{}groups
\\
\hline
\sphinxAtStartPar
.*test\_subnetpool\_prefix\_ops
&
\sphinxAtStartPar
subnetpool\sphinxhyphen{}prefix\sphinxhyphen{}ops
\\
\hline
\sphinxAtStartPar
.*test\_subnetpools.RbacSubnetPoolTest
&
\sphinxAtStartPar
rbac\sphinxhyphen{}subnetpool
\\
\hline
\sphinxAtStartPar
.*test\_subnetpools\_negative.SubnetPoolsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_tenant\_create\_subnetpool\_associate\_shared\_address\_scope
&
\sphinxAtStartPar
rbac\sphinxhyphen{}subnetpool
\\
\hline
\sphinxAtStartPar
.*test\_subnetpools.SubnetPoolsSearchCriteriaTest.\textbackslash{}

\sphinxAtStartPar
test\_list\_validation\_filters
&
\sphinxAtStartPar
filter\sphinxhyphen{}validation
\\
\hline
\sphinxAtStartPar
.*test\_subnets.SubnetsSearchCriteriaTest.\textbackslash{}

\sphinxAtStartPar
test\_list\_validation\_filters
&
\sphinxAtStartPar
filter\sphinxhyphen{}validation
\\
\hline
\sphinxAtStartPar
.*test\_timestamp.TestTimeStamp.\textbackslash{}

\sphinxAtStartPar
test\_segment\_with\_timestamp
&
\sphinxAtStartPar
standard\sphinxhyphen{}attr\sphinxhyphen{}segment
\\
\hline
\sphinxAtStartPar
.*test\_trunk.TrunkTestInheritJSONBase.\textbackslash{}

\sphinxAtStartPar
test\_add\_subport
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1863707}
\\
\hline
\sphinxAtStartPar
.*test\_trunk.TrunkTestMtusJSON
&
\sphinxAtStartPar
vxlan
\\
\hline
\sphinxAtStartPar
.*test\_trunk\_negative.TrunkTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_create\_subport\_invalid\_inherit\_network\_\textbackslash{}

\sphinxAtStartPar
segmentation\_type
&
\sphinxAtStartPar
vxlan
\\
\hline
\sphinxAtStartPar
.*test\_trunk\_negative.TrunkTestMtusJSON
&
\sphinxAtStartPar
vxlan
\\
\hline
\sphinxAtStartPar
.*test\_qos.QosMinimumBandwidthRuleTestJSON
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/69105}
\\
\hline
\sphinxAtStartPar
.*network.test\_tags
&
\sphinxAtStartPar
tag\sphinxhyphen{}ext
\\
\hline
\sphinxAtStartPar
.*test\_routers.RoutersIpV6Test.\textbackslash{}

\sphinxAtStartPar
test\_create\_router\_set\_gateway\_with\_fixed\_ip
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1676207}
\\
\hline
\sphinxAtStartPar
.*test\_routers.RoutersTest.\textbackslash{}

\sphinxAtStartPar
test\_create\_router\_set\_gateway\_with\_fixed\_ip
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1676207}
\\
\hline
\sphinxAtStartPar
.*test\_network\_basic\_ops.\textbackslash{}

\sphinxAtStartPar
TestNetworkBasicOps.test\_router\_rescheduling
&
\sphinxAtStartPar
l3\_agent\_scheduler
\\
\hline
\sphinxAtStartPar
.*test\_network\_advanced\_server\_ops.\textbackslash{}

\sphinxAtStartPar
TestNetworkAdvancedServerOps.\textbackslash{}

\sphinxAtStartPar
test\_server\_connectivity\_cold\_migration\_revert
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1836595}
\\
\hline
\end{longtable}\sphinxatlongtableend\end{savenotes}

\sphinxAtStartPar
Neutron API is also covered by
\sphinxhref{https://opendev.org/openstack/rally}{Rally}.

\sphinxAtStartPar
Here are the mainline tasks integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Authenticate.validate\_neutron

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_networks

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_ports

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_routers

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_subnets

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_networks

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_ports

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_routers

\item {} 
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_subnets

\item {} 
\sphinxAtStartPar
NeutronSecurityGroup.create\_and\_delete\_security\_groups

\item {} 
\sphinxAtStartPar
NeutronSecurityGroup.create\_and\_delete\_security\_group\_rule

\item {} 
\sphinxAtStartPar
NeutronNetworks.set\_and\_clear\_router\_gateway

\item {} 
\sphinxAtStartPar
Quotas.neutron\_update

\end{itemize}


\paragraph{Compute \sphinxhyphen{} Nova API testing}
\label{\detokenize{chapters/chapter03:compute-nova-api-testing}}
\sphinxAtStartPar
Nova API is covered in the OpenStack Gates via
\sphinxhref{https://opendev.org/openstack/tempest}{Tempest} as integrated in
\sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}.

\sphinxAtStartPar
According to \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
the following test names must not be executed:


\begin{savenotes}\sphinxatlongtablestart\begin{longtable}[c]{|\X{60}{100}|\X{40}{100}|}
\sphinxthelongtablecaptionisattop
\caption{Nova API testing\strut}\label{\detokenize{chapters/chapter03:id7}}\\*[\sphinxlongtablecapskipadjust]
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\endfirsthead

\multicolumn{2}{c}%
{\makebox[0pt]{\sphinxtablecontinued{\tablename\ \thetable{} \textendash{} continued from previous page}}}\\
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\endhead

\hline
\multicolumn{2}{r}{\makebox[0pt][r]{\sphinxtablecontinued{continues on next page}}}\\
\endfoot

\endlastfoot

\sphinxAtStartPar
.*admin.test\_agents
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_fixed\_ips
&
\sphinxAtStartPar
neutron
\\
\hline
\sphinxAtStartPar
.*test\_fixed\_ips\_negative
&
\sphinxAtStartPar
neutron
\\
\hline
\sphinxAtStartPar
.*test\_auto\_allocate\_network
&
\sphinxAtStartPar
shared networks
\\
\hline
\sphinxAtStartPar
.*test\_flavors\_microversions.FlavorsV255TestJSON
&
\sphinxAtStartPar
max\_microversion: 2.53
\\
\hline
\sphinxAtStartPar
.*test\_flavors\_microversions.FlavorsV261TestJSON
&
\sphinxAtStartPar
max\_microversion: 2.53
\\
\hline
\sphinxAtStartPar
.*test\_floating\_ips\_bulk
&
\sphinxAtStartPar
nova\sphinxhyphen{}network
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.\textbackslash{}

\sphinxAtStartPar
LiveAutoBlockMigrationV225Test.test\_iscsi\_volume
&
\sphinxAtStartPar
block live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.\textbackslash{}

\sphinxAtStartPar
LiveAutoBlockMigrationV225Test.\textbackslash{}

\sphinxAtStartPar
test\_live\_block\_migration
&
\sphinxAtStartPar
block live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.\textbackslash{}

\sphinxAtStartPar
LiveAutoBlockMigrationV225Test.\textbackslash{}

\sphinxAtStartPar
test\_live\_block\_migration\_paused
&
\sphinxAtStartPar
block live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.\textbackslash{}

\sphinxAtStartPar
LiveAutoBlockMigrationV225Test.\textbackslash{}

\sphinxAtStartPar
test\_volume\_backed\_live\_migration
&
\sphinxAtStartPar
volume\sphinxhyphen{}backed live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.LiveMigrationTest.\textbackslash{}

\sphinxAtStartPar
test\_iscsi\_volume
&
\sphinxAtStartPar
block live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.LiveMigrationTest.\textbackslash{}

\sphinxAtStartPar
test\_live\_block\_migration
&
\sphinxAtStartPar
block live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.LiveMigrationTest.\textbackslash{}

\sphinxAtStartPar
test\_live\_block\_migration\_paused
&
\sphinxAtStartPar
block live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.LiveMigrationTest.\textbackslash{}

\sphinxAtStartPar
test\_volume\_backed\_live\_migration
&
\sphinxAtStartPar
volume\sphinxhyphen{}backed live migration
\\
\hline
\sphinxAtStartPar
.*test\_live\_migration.\textbackslash{}

\sphinxAtStartPar
LiveMigrationRemoteConsolesV26Test
&
\sphinxAtStartPar
serial\_console
\\
\hline
\sphinxAtStartPar
.*test\_quotas.QuotasAdminTestV257
&
\sphinxAtStartPar
max\_microversion: 2.53
\\
\hline
\sphinxAtStartPar
.*test\_servers.ServersAdminTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_reset\_network\_inject\_network\_info
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*certificates.test\_certificates
&
\sphinxAtStartPar
cert
\\
\hline
\sphinxAtStartPar
.*test\_quotas\_negative.\textbackslash{}

\sphinxAtStartPar
QuotasSecurityGroupAdminNegativeTest
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1186354}
\\
\hline
\sphinxAtStartPar
.*test\_novnc
&
\sphinxAtStartPar
vnc\_console
\\
\hline
\sphinxAtStartPar
.*test\_server\_personality
&
\sphinxAtStartPar
personality
\\
\hline
\sphinxAtStartPar
.*test\_servers.ServerShowV263Test.\textbackslash{}

\sphinxAtStartPar
test\_show\_update\_rebuild\_list\_server
&
\sphinxAtStartPar
certified\_image\_ref
\\
\hline
\sphinxAtStartPar
.*test\_servers\_microversions.ServerShowV254Test
&
\sphinxAtStartPar
max\_microversion: 2.53
\\
\hline
\sphinxAtStartPar
.*test\_servers\_microversions.ServerShowV257Test
&
\sphinxAtStartPar
max\_microversion: 2.53
\\
\hline
\sphinxAtStartPar
.*test\_servers\_negative.ServersNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_personality\_file\_contents\_not\_encoded
&
\sphinxAtStartPar
personality
\\
\hline
\sphinxAtStartPar
.*test\_server\_actions.ServerActionsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_change\_server\_password
&
\sphinxAtStartPar
change\_password
\\
\hline
\sphinxAtStartPar
.*test\_server\_actions.ServerActionsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_get\_vnc\_console
&
\sphinxAtStartPar
vnc\_console
\\
\hline
\sphinxAtStartPar
.*test\_server\_actions.ServerActionsTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_reboot\_server\_soft
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1014647}
\\
\hline
\sphinxAtStartPar
.*test\_server\_rescue.\textbackslash{}

\sphinxAtStartPar
ServerBootFromVolumeStableRescueTest
&
\sphinxAtStartPar
stable\_rescue
\\
\hline
\sphinxAtStartPar
.*test\_server\_rescue.ServerStableDeviceRescueTest
&
\sphinxAtStartPar
stable\_rescue
\\
\hline
\sphinxAtStartPar
.*test\_security\_group\_default\_rules
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1311500}
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups\_negative.\textbackslash{}
\begin{quote}

\sphinxAtStartPar
SecurityGroupsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_security\_group\_create\_with\_duplicate\_name
\end{quote}
&
\sphinxAtStartPar
neutron
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups\_negative.\textbackslash{}

\sphinxAtStartPar
SecurityGroupsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_security\_group\_create\_with\_\textbackslash{}

\sphinxAtStartPar
invalid\_group\_description
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1161411}
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups\_negative.\textbackslash{}

\sphinxAtStartPar
SecurityGroupsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_security\_group\_create\_with\_invalid\_group\_name
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1161411}
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups\_negative.\textbackslash{}

\sphinxAtStartPar
SecurityGroupsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_update\_security\_group\_with\_invalid\_sg\_description
&
\sphinxAtStartPar
neutron
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups\_negative.\textbackslash{}

\sphinxAtStartPar
SecurityGroupsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_update\_security\_group\_with\_invalid\_sg\_description
&
\sphinxAtStartPar
neutron
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups\_negative.\textbackslash{}

\sphinxAtStartPar
SecurityGroupsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_update\_security\_group\_with\_invalid\_sg\_id
&
\sphinxAtStartPar
neutron
\\
\hline
\sphinxAtStartPar
.*test\_security\_groups\_negative.\textbackslash{}

\sphinxAtStartPar
SecurityGroupsNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_update\_security\_group\_with\_invalid\_sg\_name
&
\sphinxAtStartPar
neutron
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata.ServerMetadataTestJSON
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_delete\_metadata\_non\_existent\_server
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_metadata\_items\_limit
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_set\_metadata\_invalid\_key
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_set\_metadata\_non\_existent\_server
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_set\_server\_metadata\_blank\_key
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_set\_server\_metadata\_missing\_metadata
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_update\_metadata\_non\_existent\_server
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_server\_metadata\_negative.\textbackslash{}

\sphinxAtStartPar
ServerMetadataNegativeTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_update\_metadata\_with\_blank\_key
&
\sphinxAtStartPar
xenapi\_apis
\\
\hline
\sphinxAtStartPar
.*test\_list\_server\_filters.\textbackslash{}

\sphinxAtStartPar
ListServerFiltersTestJSON.\textbackslash{}

\sphinxAtStartPar
test\_list\_servers\_filtered\_by\_ip\_regex
&
\sphinxAtStartPar
\sphinxurl{https://launchpad.net/bugs/1540645}
\\
\hline
\sphinxAtStartPar
.*servers.test\_virtual\_interfaces
&
\sphinxAtStartPar
nova\sphinxhyphen{}network
\\
\hline
\sphinxAtStartPar
.*compute.test\_virtual\_interfaces\_negative
&
\sphinxAtStartPar
nova\sphinxhyphen{}network
\\
\hline
\sphinxAtStartPar
.*compute.test\_networks
&
\sphinxAtStartPar
nova\sphinxhyphen{}network
\\
\hline
\sphinxAtStartPar
.*test\_attach\_volume.AttachVolumeMultiAttach
&
\sphinxAtStartPar
volume\_multiattach
\\
\hline
\sphinxAtStartPar
.*test\_volume\_boot\_pattern.\textbackslash{}

\sphinxAtStartPar
TestVolumeBootPattern.\textbackslash{}

\sphinxAtStartPar
test\_boot\_server\_from\_encrypted\_volume\_luks
&
\sphinxAtStartPar
attach\_encrypted\_volume
\\
\hline
\sphinxAtStartPar
.*test\_volume\_swap
&
\sphinxAtStartPar
swap\_volume
\\
\hline
\sphinxAtStartPar
.*test\_encrypted\_cinder\_volumes
&
\sphinxAtStartPar
attach\_encrypted\_volume
\\
\hline
\sphinxAtStartPar
.*test\_minbw\_allocation\_placement
&
\sphinxAtStartPar
microversion
\\
\hline
\sphinxAtStartPar
.*test\_volumes\_negative.\textbackslash{}

\sphinxAtStartPar
UpdateMultiattachVolumeNegativeTest.\textbackslash{}

\sphinxAtStartPar
test\_multiattach\_rw\_volume\_update\_failure
&
\sphinxAtStartPar
volume\_multiattach
\\
\hline
\sphinxAtStartPar
.*test\_shelve\_instance.TestShelveInstance.\textbackslash{}

\sphinxAtStartPar
test\_cold\_migrate\_unshelved\_instance
&
\sphinxAtStartPar
shelve\_migrate
\\
\hline
\end{longtable}\sphinxatlongtableend\end{savenotes}

\sphinxAtStartPar
Nova API is also covered by
\sphinxhref{https://opendev.org/openstack/rally}{Rally}.

\sphinxAtStartPar
Here are the mainline tasks integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Authenticate.validate\_nova

\item {} 
\sphinxAtStartPar
NovaServers.boot\_and\_live\_migrate\_server

\item {} 
\sphinxAtStartPar
NovaServers.boot\_server\_attach\_created\_volume\_and\_live\_migrate

\item {} 
\sphinxAtStartPar
NovaServers.boot\_server\_from\_volume\_and\_live\_migrate

\item {} 
\sphinxAtStartPar
NovaKeypair.boot\_and\_delete\_server\_with\_keypair

\item {} 
\sphinxAtStartPar
NovaServers.boot\_server\_from\_volume\_and\_delete

\item {} 
\sphinxAtStartPar
NovaServers.pause\_and\_unpause\_server

\item {} 
\sphinxAtStartPar
NovaServers.boot\_and\_migrate\_server

\item {} 
\sphinxAtStartPar
NovaServers.boot\_server\_and\_list\_interfaces

\item {} 
\sphinxAtStartPar
NovaServers.boot\_server\_associate\_and\_dissociate\_floating\_ip

\item {} 
\sphinxAtStartPar
NovaServerGroups.create\_and\_delete\_server\_group

\item {} 
\sphinxAtStartPar
Quotas.nova\_update

\end{itemize}


\paragraph{Orchestration \sphinxhyphen{} Heat API testing}
\label{\detokenize{chapters/chapter03:orchestration-heat-api-testing}}
\sphinxAtStartPar
Heat API is covered in the OpenStack Gates via
\sphinxhref{https://opendev.org/openstack/heat-tempest-plugin}{heat\sphinxhyphen{}tempest\sphinxhyphen{}plugin}
as integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}

\sphinxAtStartPar
According to \sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
the following test names must not be executed:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Heat API testing}\label{\detokenize{chapters/chapter03:id8}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{60}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test rejection regular expressions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reasons
\\
\hline
\sphinxAtStartPar
.*functional.test\_lbaasv2
&
\sphinxAtStartPar
lbaasv2
\\
\hline
\sphinxAtStartPar
.*functional.test\_encryption\_vol\_type
&
\sphinxAtStartPar
\sphinxurl{https://storyboard.openstack.org/\#!/story/2007804}
\\
\hline
\sphinxAtStartPar
.*RemoteStackTest.\textbackslash{}

\sphinxAtStartPar
test\_stack\_create\_with\_cloud\_credential
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/c/functest/+/69926}
\\
\hline
\sphinxAtStartPar
.*scenario.test\_aodh\_alarm
&
\sphinxAtStartPar
aodh
\\
\hline
\sphinxAtStartPar
.*tests.scenario.test\_autoscaling\_lb
&
\sphinxAtStartPar
lbaas
\\
\hline
\sphinxAtStartPar
.*scenario.test\_autoscaling\_lbv2
&
\sphinxAtStartPar
lbaasv2
\\
\hline
\sphinxAtStartPar
.*scenario.test\_server\_software\_config
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/c/functest/+/69926}
\\
\hline
\sphinxAtStartPar
.*test\_volumes.\textbackslash{}

\sphinxAtStartPar
VolumeBackupRestoreIntegrationTest
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/c/functest/+/69931}
\\
\hline
\sphinxAtStartPar
.*scenario.test\_octavia\_lbaas
&
\sphinxAtStartPar
octavia
\\
\hline
\sphinxAtStartPar
.*scenario.test\_server\_cfn\_init
&
\sphinxAtStartPar
\sphinxurl{https://gerrit.opnfv.org/gerrit/c/functest/+/70004}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Heat API is also covered by
\sphinxhref{https://opendev.org/openstack/rally}{Rally}.

\sphinxAtStartPar
Here are the mainline tasks integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Smoke
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Authenticate.validate\_heat

\item {} 
\sphinxAtStartPar
HeatStacks.create\_update\_delete\_stack

\item {} 
\sphinxAtStartPar
HeatStacks.create\_check\_delete\_stack

\item {} 
\sphinxAtStartPar
HeatStacks.create\_suspend\_resume\_delete\_stack

\item {} 
\sphinxAtStartPar
HeatStacks.list\_stacks\_and\_resources

\end{itemize}


\subsubsection{Dashboard}
\label{\detokenize{chapters/chapter03:dashboard}}
\sphinxAtStartPar
Horizon is covered in the OpenStack Gates via
\sphinxhref{https://github.com/openstack/tempest-horizon}{tempest\sphinxhyphen{}horizon} as
integrated in \sphinxhref{https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable\%2Fwallaby}{Functest
Healthcheck}.


\subsubsection{OpenStack API benchmarking}
\label{\detokenize{chapters/chapter03:openstack-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{https://opendev.org/openstack/rally}{Rally} is tool and framework
that allows to perform OpenStack API benchmarking.

\sphinxAtStartPar
Here are the Rally\sphinxhyphen{}based test cases proposed by \sphinxhref{https://git.opnfv.org/functest/tree/docker/benchmarking-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Benchmarking
CNTT}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{rally\_full}:
Functest scenarios iterating 10 times the mainline Rally scenarios

\item {} 
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_jobs\_cntt-run-5/rally\_jobs\_cntt/rally\_jobs\_cntt.html}{rally\_jobs}:
Neutron scenarios executed in the OpenStack gates

\end{itemize}

\sphinxAtStartPar
At the time of writing, no KPI is defined in
\sphinxhref{https://cntt.readthedocs.io/projects/ra1/en/latest/chapters/chapter05.html}{Interfaces and APIs}
which would have asked for an update of the default SLA (maximum failure
rate of 0\%) proposed in \sphinxhref{https://git.opnfv.org/functest/tree/docker/benchmarking-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Benchmarking
CNTT}


\paragraph{Identity \sphinxhyphen{} Keystone API benchmarking}
\label{\detokenize{chapters/chapter03:identity-keystone-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_full\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Keystone API benchmarking}\label{\detokenize{chapters/chapter03:id9}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
Authenticate.keystone
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.add\_and\_remove\_user\_role
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_add\_and\_list\_user\_roles
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_and\_list\_tenants
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_and\_delete\_role
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_and\_delete\_service
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.get\_entities
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_update\_and\_delete\_tenant
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_user
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_tenant
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_and\_list\_users
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
KeystoneBasic.create\_tenant\_with\_users
&
\sphinxAtStartPar
10
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\paragraph{Image \sphinxhyphen{} Glance API benchmarking}
\label{\detokenize{chapters/chapter03:image-glance-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_full\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Glance API benchmarking}\label{\detokenize{chapters/chapter03:id10}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
Authenticate.validate\_glance
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.create\_and\_delete\_image
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.create\_and\_list\_image
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.list\_images
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.create\_image\_and\_boot\_instances
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.create\_and\_deactivate\_image
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.create\_and\_download\_image
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.create\_and\_get\_image
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
GlanceImages.create\_and\_update\_image
&
\sphinxAtStartPar
10
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\paragraph{Block Storage \sphinxhyphen{} Cinder API benchmarking}
\label{\detokenize{chapters/chapter03:block-storage-cinder-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_full\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Cinder API benchmarking}\label{\detokenize{chapters/chapter03:id11}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
Authenticate.validate\_glance
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_and\_attach\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_and\_list\_snapshots
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_and\_list\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_and\_upload\_volume\_to\_image
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_nested\_snapshots\_and\_attach\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_snapshot\_and\_attach\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.list\_volumes
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_and\_delete\_snapshot
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_and\_delete\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_and\_extend\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumes.create\_from\_volume\_and\_delete\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderQos.create\_and\_get\_qos
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderQos.create\_and\_list\_qos
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderQos.create\_and\_set\_qos
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumeTypes.create\_and\_get\_volume\_type
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumeTypes.create\_and\_list\_volume\_types
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumeTypes.create\_and\_update\_volume\_type
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumeTypes.create\_volume\_type\_and\_encryption\_type
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
CinderVolumeTypes.create\_volume\_type\_add\_and\_list\_type\_access
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Quotas.cinder\_update\_and\_delete
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Quotas.cinder\_update
&
\sphinxAtStartPar
10
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\paragraph{Object Storage \sphinxhyphen{} Swift API benchmarking}
\label{\detokenize{chapters/chapter03:object-storage-swift-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_full\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Swift API benchmarking}\label{\detokenize{chapters/chapter03:id12}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
SwiftObjects.create\_container\_and\_object\_then\_list\_objects
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
SwiftObjects.list\_objects\_in\_containers
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
SwiftObjects.create\_container\_and\_object\_then\_download\_object
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
SwiftObjects.create\_container\_and\_object\_then\_delete\_all
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
SwiftObjects.list\_and\_download\_objects\_in\_containers
&
\sphinxAtStartPar
10
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\paragraph{Networking \sphinxhyphen{} Neutron API benchmarking}
\label{\detokenize{chapters/chapter03:networking-neutron-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_full\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Neutron API benchmarking}\label{\detokenize{chapters/chapter03:id13}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
Authenticate.validate\_neutron
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_networks
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_ports
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_routers
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_subnets
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_networks
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_ports
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_routers
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_subnets
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_networks
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_ports
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_routers
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_subnets
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronSecurityGroup.create\_and\_delete\_security\_groups
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronSecurityGroup.create\_and\_delete\_security\_group\_rule
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronSecurityGroup.create\_and\_list\_security\_group\_rules
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronSecurityGroup.create\_and\_show\_security\_group
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.set\_and\_clear\_router\_gateway
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_show\_ports
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_show\_routers
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_show\_subnets
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Quotas.neutron\_update
&
\sphinxAtStartPar
10
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_jobs\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Neutron API benchmarking}\label{\detokenize{chapters/chapter03:id14}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_networks
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_ports
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_routers
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_delete\_subnets
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_networks
&
\sphinxAtStartPar
100
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_ports
&
\sphinxAtStartPar
8
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_routers
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_list\_subnets
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_networks
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_ports
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_routers
&
\sphinxAtStartPar
40
\\
\hline
\sphinxAtStartPar
NeutronNetworks.create\_and\_update\_subnets
&
\sphinxAtStartPar
100
\\
\hline
\sphinxAtStartPar
NeutronTrunks.create\_and\_list\_trunks
&
\sphinxAtStartPar
4
\\
\hline
\sphinxAtStartPar
Quotas.neutron\_update
&
\sphinxAtStartPar
40
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\paragraph{Compute \sphinxhyphen{} Nova API benchmarking}
\label{\detokenize{chapters/chapter03:compute-nova-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_full\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Nova API benchmarking}\label{\detokenize{chapters/chapter03:id15}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{70}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
Authenticate.validate\_nova
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaKeypair.create\_and\_delete\_keypair
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaKeypair.create\_and\_list\_keypairs
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_bounce\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_delete\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_list\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_rebuild\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.snapshot\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_from\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.list\_servers
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.resize\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_live\_migrate\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_attach\_created\_volume\_and\_live\_migrate
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_from\_volume\_and\_live\_migrate
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaKeypair.boot\_and\_delete\_server\_with\_keypair
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_from\_volume\_and\_delete
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.pause\_and\_unpause\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_migrate\_server
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_and\_list\_interfaces
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_get\_console\_url
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_and\_attach\_interface
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_attach\_volume\_and\_list\_attachments
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_server\_associate\_and\_dissociate\_floating\_ip
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServers.boot\_and\_associate\_floating\_ip
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServerGroups.create\_and\_delete\_server\_group
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServerGroups.create\_and\_get\_server\_group
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
NovaServerGroups.create\_and\_list\_server\_groups
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Quotas.nova\_update
&
\sphinxAtStartPar
10
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Orchestration \sphinxhyphen{} Heat API benchmarking}
\label{\detokenize{chapters/chapter03:orchestration-heat-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally\_full\_cntt-run-5/rally\_full\_cntt/rally\_full\_cntt.html}{Functest
rally\_full\_cntt}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Heat API benchmarking}\label{\detokenize{chapters/chapter03:id16}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
Authenticate.validate\_heat
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
HeatStacks.create\_and\_delete\_stack
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
HeatStacks.create\_and\_list\_stack
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
HeatStacks.create\_update\_delete\_stack
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
HeatStacks.create\_check\_delete\_stack
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
HeatStacks.create\_suspend\_resume\_delete\_stack
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
HeatStacks.list\_stacks\_and\_resources
&
\sphinxAtStartPar
10
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Dataplane benchmarking}
\label{\detokenize{chapters/chapter03:dataplane-benchmarking}}
\sphinxAtStartPar
\sphinxhref{https://git.opnfv.org/functest/tree/docker/benchmarking-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Benchmarking
CNTT}
offers two benchmarking dataplane test cases leveraging on:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{http://vmtp.readthedocs.io/en/latest}{VMTP}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://pyshaker.readthedocs.io/en/latest/}{Shaker}

\end{itemize}

\sphinxAtStartPar
\sphinxhref{https://vmtp.readthedocs.io/en/latest/}{VMTP} is a small python
application that will automatically perform ping connectivity, round
trip time measurement (latency) and TCP/UDP throughput measurement on
any OpenStack deployment.

\sphinxAtStartPar
\sphinxhref{https://pyshaker.readthedocs.io/en/latest/}{Shaker} wraps around
popular system network testing tools like iperf, iperf3 and netperf
(with help of flent).
\sphinxhref{https://pyshaker.readthedocs.io/en/latest/}{Shaker} is able to deploy
OpenStack instances and networks in different topologies.
\sphinxhref{https://pyshaker.readthedocs.io/en/latest/}{Shaker} scenario
specifies the deployment and list of tests to execute.

\sphinxAtStartPar
At the time of writing, no KPIs are defined in Anuket specifications
which would have asked for an update of the default SLA proposed in
\sphinxhref{https://git.opnfv.org/functest/tree/docker/benchmarking-cntt/testcases.yaml?h=stable\%2Fwallaby}{Functest Benchmarking
CNTT}

\sphinxAtStartPar
On top of this dataplane benchmarking described in VMTP \& Shaker, we
need to integrate testing as described in \sphinxhref{https://www.etsi.org/deliver/etsi\_gs/NFV-TST/001\_099/009/03.01.01\_60/gs\_NFV-TST009v030101p.pdf}{ETSI GS NFV\sphinxhyphen{}TST 009:
Specification of Networking Benchmarks and Measurement Methods for
NFVI}.
This type of testing is better suited to measure the networking
capabilities of a compute node. The \sphinxhref{https://git.opnfv.org/samplevnf}{rapid
scripts} in
conjunction with the \sphinxhref{https://docs.anuket.io/projects/samplevnf/en/latest/testing/user/userguide/index.html}{PROX
tool}
offers an open source implementation for this type of testing.


\paragraph{VMTP}
\label{\detokenize{chapters/chapter03:vmtp}}
\sphinxAtStartPar
Here are the
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-vmtp-run-8/vmtp/vmtp.json}{scenarios}
executed by \sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-vmtp-run-8/vmtp/vmtp.html}{Functest
vmtp}:
\sphinxhyphen{} VM to VM same network fixed IP (intra\sphinxhyphen{}node) \sphinxhyphen{} VM to VM different
network fixed IP (intra\sphinxhyphen{}node) \sphinxhyphen{} VM to VM different network floating IP
(intra\sphinxhyphen{}node) \sphinxhyphen{} VM to VM same network fixed IP (inter\sphinxhyphen{}node) \sphinxhyphen{} VM to VM
different network fixed IP (inter\sphinxhyphen{}node) \sphinxhyphen{} VM to VM different network
floating IP (inter\sphinxhyphen{}node)

\sphinxAtStartPar
Here are all results per scenario:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{All results per scenario}\label{\detokenize{chapters/chapter03:id17}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
protocol
&\sphinxstyletheadfamily 
\sphinxAtStartPar
pkt\_size
&\sphinxstyletheadfamily 
\sphinxAtStartPar
results
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
64
&
\sphinxAtStartPar
rtt\_avg\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
64
&
\sphinxAtStartPar
rtt\_max\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
64
&
\sphinxAtStartPar
rtt\_min\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
64
&
\sphinxAtStartPar
rtt\_stddev
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
391
&
\sphinxAtStartPar
rtt\_avg\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
391
&
\sphinxAtStartPar
rtt\_max\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
391
&
\sphinxAtStartPar
rtt\_min\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
391
&
\sphinxAtStartPar
rtt\_stddev
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
1500
&
\sphinxAtStartPar
rtt\_avg\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
1500
&
\sphinxAtStartPar
rtt\_max\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
1500
&
\sphinxAtStartPar
rtt\_min\_ms
\\
\hline
\sphinxAtStartPar
ICMP
&
\sphinxAtStartPar
1500
&
\sphinxAtStartPar
rtt\_stddev
\\
\hline
\sphinxAtStartPar
UDP
&
\sphinxAtStartPar
128
&
\sphinxAtStartPar
loss\_rate
\\
\hline
\sphinxAtStartPar
UDP
&
\sphinxAtStartPar
128
&
\sphinxAtStartPar
throughput\_kbps
\\
\hline
\sphinxAtStartPar
UDP
&
\sphinxAtStartPar
1024
&
\sphinxAtStartPar
loss\_rate
\\
\hline
\sphinxAtStartPar
UDP
&
\sphinxAtStartPar
1024
&
\sphinxAtStartPar
throughput\_kbps
\\
\hline
\sphinxAtStartPar
UDP
&
\sphinxAtStartPar
8192
&
\sphinxAtStartPar
loss\_rate
\\
\hline
\sphinxAtStartPar
UDP
&
\sphinxAtStartPar
8192
&
\sphinxAtStartPar
throughput\_kbps
\\
\hline
\sphinxAtStartPar
TCP
&
\sphinxAtStartPar
65536
&
\sphinxAtStartPar
rtt\_ms
\\
\hline
\sphinxAtStartPar
TCP
&
\sphinxAtStartPar
65536
&
\sphinxAtStartPar
throughput\_kbps
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\paragraph{Shaker}
\label{\detokenize{chapters/chapter03:shaker}}
\sphinxAtStartPar
Here are the
\sphinxhref{http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-shaker-run-8/shaker/report.json}{scenarios}
executed by Shaker:
\begin{itemize}
\item {} 
\sphinxAtStartPar
OpenStack L2

\item {} 
\sphinxAtStartPar
OpenStack L3 East\sphinxhyphen{}West

\item {} 
\sphinxAtStartPar
OpenStack L3 North\sphinxhyphen{}South

\item {} 
\sphinxAtStartPar
OpenStack L3 North\sphinxhyphen{}South Performance

\end{itemize}

\sphinxAtStartPar
Here are all samples:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{All samples}\label{\detokenize{chapters/chapter03:id18}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
test
&\sphinxstyletheadfamily 
\sphinxAtStartPar
samples
\\
\hline
\sphinxAtStartPar
Bi\sphinxhyphen{}directional
&
\sphinxAtStartPar
ping\_icmp (ms)
\\
\hline
\sphinxAtStartPar
Bi\sphinxhyphen{}directional
&
\sphinxAtStartPar
tcp\_download (Mbits/s)
\\
\hline
\sphinxAtStartPar
Bi\sphinxhyphen{}directional
&
\sphinxAtStartPar
tcp\_upload (Mbits/s)
\\
\hline
\sphinxAtStartPar
Download
&
\sphinxAtStartPar
ping\_icmp (ms)
\\
\hline
\sphinxAtStartPar
Download
&
\sphinxAtStartPar
tcp\_download (Mbits/s)
\\
\hline
\sphinxAtStartPar
Upload
&
\sphinxAtStartPar
ping\_icmp (ms)
\\
\hline
\sphinxAtStartPar
Upload
&
\sphinxAtStartPar
tcp\_upload (Mbits/s)
\\
\hline
\sphinxAtStartPar
Ping
&
\sphinxAtStartPar
ping\_icmp (ms)
\\
\hline
\sphinxAtStartPar
Ping
&
\sphinxAtStartPar
ping\_udp (ms)
\\
\hline
\sphinxAtStartPar
TCP
&
\sphinxAtStartPar
bandwidth (bit/s)
\\
\hline
\sphinxAtStartPar
TCP
&
\sphinxAtStartPar
retransmits
\\
\hline
\sphinxAtStartPar
UDP
&
\sphinxAtStartPar
packets (pps)
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\paragraph{PROX}
\label{\detokenize{chapters/chapter03:prox}}
\sphinxAtStartPar
The generator used with the rapid scripts is PROX with a specific
generator configuration file. When multiple flows are requested, the
generator starts randomizing bits in the source and destination UDP
ports. The number of flows to be generated during each run of the test
is specified in the test files (e.g. TST009\_Throughput.test). Packet
size used during the test is also defined in the test file. IMIX is not
supported yet, but you could take the average packet size of the IMIX
for now. When defining n packet sizes with m different flow sizes, the
test will run n x m times and will produce the results for these n x m
combinations. All throughput benchmarking is done by a generator sending
packets to a reflector. This results in bidirectional traffic which
should be identical (src and dest IP and ports swapped) if all traffic
goes through. The VMs or containers use only 1 vNIC for incoming and
outgoing traffic. Multiple queues can be used. Multiple VMs or
containers can be deployed prior to running any tests. This allows to
use generator\sphinxhyphen{}reflector pairs on the same or different compute nodes, on
the same or different NUMA nodes.


\subsubsection{Opensource VNF onboarding and testing}
\label{\detokenize{chapters/chapter03:opensource-vnf-onboarding-and-testing}}
\sphinxAtStartPar
Running opensource VNFs is a key technical solution to ensure that the
platforms meet Network Functions Virtualization requirements. \sphinxhref{https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable\%2Fwallaby}{Functest
VNF}
offers 5 test cases which automatically onboard and test the following 3
opensource VNFs:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://clearwater.readthedocs.io/en/stable/}{Clearwater IMS}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://www.vyos.io/}{VyOS vRouter}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://www.openairinterface.org/}{OpenAirInterface vEPC}

\end{itemize}

\sphinxAtStartPar
Here are the full list of orchestrators used for all these deployments:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://cloudify.co/}{Cloudify}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://wiki.openstack.org/wiki/Heat}{Heat}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://jaas.ai/}{Juju}

\end{itemize}

\sphinxAtStartPar
The VNF are covered by upstream tests when possible (see
\sphinxhref{https://github.com/Metaswitch/clearwater-live-test}{clearwater\sphinxhyphen{}live\sphinxhyphen{}test})
and by Functest VNF tests in the other cases.


\subsubsection{Tenants}
\label{\detokenize{chapters/chapter03:tenants}}

\subsubsection{LCM}
\label{\detokenize{chapters/chapter03:lcm}}

\subsubsection{Assurance}
\label{\detokenize{chapters/chapter03:assurance}}

\subsubsection{Security}
\label{\detokenize{chapters/chapter03:security}}

\subsubsection{Resilience}
\label{\detokenize{chapters/chapter03:resilience}}

\subsubsection{Bare\sphinxhyphen{}metal validations}
\label{\detokenize{chapters/chapter03:bare-metal-validations}}

\subsection{Test Cases Traceability to Requirements}
\label{\detokenize{chapters/chapter03:test-cases-traceability-to-requirements}}

\subsubsection{RM/RA\sphinxhyphen{}1 Requirements}
\label{\detokenize{chapters/chapter03:rm-ra-1-requirements}}
\sphinxAtStartPar
According to
{\hyperref[\detokenize{chapters/chapter04:openstack-based-cloud-infrastructure-testing-cookbook}]{\sphinxcrossref{\DUrole{std,std-ref}{OpenStack\sphinxhyphen{}based cloud infrastructure Testing Cookbook}}}}
the following test cases must pass as they are for Anuket NFVI Conformance:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Anuket NFVI Conformance}\label{\detokenize{chapters/chapter03:id19}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
container
&\sphinxstyletheadfamily 
\sphinxAtStartPar
test case
&\sphinxstyletheadfamily 
\sphinxAtStartPar
criteria
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}healthcheck:wallaby
&
\sphinxAtStartPar
tempest\_horizon
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}smoke\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
tempest\_neutron\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}smoke\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
tempest\_cinder\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}smoke\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
tempest\_keystone\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}smoke\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
rally\_sanity\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}smoke\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
tempest\_full\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}smoke\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
tempest\_scenario\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}smoke\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
tempest\_slow\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}benchmarking\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
rally\_full\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}benchmarking\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
rally\_jobs\_cntt
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}benchmarking\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
vmtp
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}benchmarking\sphinxhyphen{}cntt:wallaby
&
\sphinxAtStartPar
shaker
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}vnf:wallaby
&
\sphinxAtStartPar
cloudify
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}vnf:wallaby
&
\sphinxAtStartPar
cloudify\_ims
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}vnf:wallaby
&
\sphinxAtStartPar
heat\_ims
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}vnf:wallaby
&
\sphinxAtStartPar
vyos\_vrouter
&
\sphinxAtStartPar
PASS
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}vnf:wallaby
&
\sphinxAtStartPar
juju\_epc
&
\sphinxAtStartPar
PASS
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\subsubsection{TC Mapping to Requirements}
\label{\detokenize{chapters/chapter03:tc-mapping-to-requirements}}

\begin{savenotes}\sphinxattablestart
\centering
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
test case
&\sphinxstyletheadfamily 
\sphinxAtStartPar
requirements
\\
\hline
\sphinxAtStartPar
tempest\_horizon
&
\sphinxAtStartPar
Horizon testing
\\
\hline
\sphinxAtStartPar
tempest\_neutron\_cntt
&
\sphinxAtStartPar
Neutron API testing
\\
\hline
\sphinxAtStartPar
tempest\_cinder\_cntt
&
\sphinxAtStartPar
Cinder API testing
\\
\hline
\sphinxAtStartPar
tempest\_keystone\_cntt
&
\sphinxAtStartPar
Keystone API testing
\\
\hline
\sphinxAtStartPar
rally\_sanity\_cntt
&
\sphinxAtStartPar
Keystone, Glance, Cinder, Swift, Neutron, Nova and
Heat API testing
\\
\hline
\sphinxAtStartPar
tempest\_full\_cntt
&
\sphinxAtStartPar
Keystone, Glance, Cinder, Swift, Neutron and Nova
API testing
\\
\hline
\sphinxAtStartPar
tempest\_scenario\_cntt
&
\sphinxAtStartPar
Keystone, Glance, Cinder, Swift, Neutron and Nova
API testing
\\
\hline
\sphinxAtStartPar
tempest\_slow\_cntt
&
\sphinxAtStartPar
Keystone, Glance, Cinder, Swift, Neutron and Nova
API testing
\\
\hline
\sphinxAtStartPar
rally\_full\_cntt
&
\sphinxAtStartPar
Keystone, Glance, Cinder, Swift, Neutron, Nova and
Heat API benchmarking
\\
\hline
\sphinxAtStartPar
rally\_jobs\_cntt
&
\sphinxAtStartPar
Neutron API benchmarking
\\
\hline
\sphinxAtStartPar
vmtp
&
\sphinxAtStartPar
Dataplane benchmarking
\\
\hline
\sphinxAtStartPar
shaker
&
\sphinxAtStartPar
Dataplane benchmarking
\\
\hline
\sphinxAtStartPar
cloudify
&
\sphinxAtStartPar
opensource VNF onboarding and testing
\\
\hline
\sphinxAtStartPar
cloudify\_ims
&
\sphinxAtStartPar
opensource VNF onboarding and testing
\\
\hline
\sphinxAtStartPar
heat\_ims
&
\sphinxAtStartPar
opensource VNF onboarding and testing
\\
\hline
\sphinxAtStartPar
vyos\_vrouter
&
\sphinxAtStartPar
opensource VNF onboarding and testing
\\
\hline
\sphinxAtStartPar
juju\_epc
&
\sphinxAtStartPar
opensource VNF onboarding and testing
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\section{OpenStack\sphinxhyphen{}based cloud infrastructure Testing Cookbook}
\label{\detokenize{chapters/chapter04:openstack-based-cloud-infrastructure-testing-cookbook}}\label{\detokenize{chapters/chapter04::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter04:introduction}}
\sphinxAtStartPar
Define the purpose of the chapter which is to:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Identify Framework Needs, Goals, and Dependencies

\item {} 
\sphinxAtStartPar
Define Opensource Integration (OVP, Functest, CVC, others)

\item {} 
\sphinxAtStartPar
Provide Automation Toolchain (list, topology, flow)

\end{itemize}


\subsection{Relevant Community Projects and Initiatives}
\label{\detokenize{chapters/chapter04:relevant-community-projects-and-initiatives}}

\subsubsection{Functest}
\label{\detokenize{chapters/chapter04:functest}}
\sphinxAtStartPar
\sphinxhref{https://functest.readthedocs.io/en/stable-xena/}{Functest} was
initially created to verify OPNFV Installers and Scenarios and then to
publish fair, trustable and public results regarding the status of the
different opensource technologies, especially for Neutron backends
(e.g. Neutron agents, OpenDaylight, OVN, etc.). It has been continuously
updated to offer the best testing coverage for any kind of OpenStack and
Kubernetes deployments including production environments. It also
ensures that the platforms meet Network Functions Virtualization
requirements by running and testing VNFs amongst all tests available.

\sphinxAtStartPar
Functest is driven by a true verification of the platform under test as
opposed to the interoperability programs such as
\sphinxhref{https://refstack.openstack.org/}{RefStack} or \sphinxhref{https://www.opnfv.org/verification}{OPNFV Verification
Program} which select a small
subset of Functional tests passing in many different opensource software
combinations:
\begin{itemize}
\item {} 
\sphinxAtStartPar
tests are skipped if an optional support is missing (e.g.
\sphinxhref{https://docs.openstack.org/barbican/latest/}{Barbican} or
networking features such as \sphinxhref{https://docs.openstack.org/networking-bgpvpn/latest/}{BGPVPN
interconnection}
or \sphinxhref{https://docs.openstack.org/networking-sfc/latest/}{Service Function
Chaining})

\item {} 
\sphinxAtStartPar
tests are parameterized (e.g. shared vs non\sphinxhyphen{}shared live migration)

\item {} 
\sphinxAtStartPar
blacklist mechanisms are available if needed

\end{itemize}

\sphinxAtStartPar
It should be noted that \sphinxhref{https://refstack.openstack.org/}{the RefStack
lists} are included as
they are in Functest in the next 3 dedicated testcases:
\begin{itemize}
\item {} 
\sphinxAtStartPar
refstack\_compute (OpenStack Powered Compute)

\item {} 
\sphinxAtStartPar
refstack\_object (OpenStack Powered Object Storage)

\item {} 
\sphinxAtStartPar
refstack\_platform (OpenStack Powered Platform)

\end{itemize}

\sphinxAtStartPar
Functest also integrates \sphinxhref{https://kubernetes.io/blog/2019/03/22/kubernetes-end-to-end-testing-for-everyone/}{Kubernetes End\sphinxhyphen{}to\sphinxhyphen{}end
tests}
and allows verifying Kubernetes Conformance (see
\sphinxhref{https://build.opnfv.org/ci/job/functest-kubernetes-opnfv-functest-kubernetes-smoke-v1.22-k8s\_conformance-run/25/console}{k8s\sphinxhyphen{}conformance}).

\sphinxAtStartPar
Dovetail (OVP) mostly leverages on Functest but only runs a small part
of Functest (\textasciitilde{}15\% of all functional tests, no benchmarking tests, no VNF
deployment and testing). It’s worth mentioning that Functest is patched
to \sphinxhref{https://github.com/opnfv/dovetail/tree/master/etc/patches/functest/disable-api-validation}{disable API
verification}
which has differed from OpenStack rules for years.

\sphinxAtStartPar
Then Functest conforms with the upstream rules (versions, code quality,
etc.) and especially their
\sphinxhref{https://docs.opendev.org/opendev/system-config/latest/devstack-gate.html}{gates}
(a.k.a. the automatic verification prior to any code review) to preserve
the quality between code and deployment. In that case, Functest can be
considered as a smooth and lightweight integration of tests developed
upstream (and the Functest team directly contributes in these projects:
\sphinxhref{https://github.com/openstack/rally-openstack}{Rally},
\sphinxhref{https://github.com/openstack/tempest}{Tempest}, etc.). It’s worth
mentioning that, as opposed to the OpenStack Gates leveraging on
\sphinxhref{https://docs.openstack.org/devstack/latest/}{DevStack}, it can check
the same already deployed SUT over and over even from a \sphinxhref{https://www.raspberrypi.org/}{Raspberry
PI}. Here the testcases can be executed
in parallel vs the same deployment instead of being executed vs
different pools of virtual machines.

\sphinxAtStartPar
Here are the functional tests (\textgreater{}2000) running in OpenStack gates
integrated in Functest Smoke (see \sphinxhref{https://build.opnfv.org/ci/view/functest/job/functest-wallaby-daily/17/}{Functest daily
jobs}
for more details):


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Functional tests}\label{\detokenize{chapters/chapter04:id1}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Testcases
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Gates
\\
\hline
\sphinxAtStartPar
tempest\_neutron
&
\sphinxAtStartPar
Neutron
\\
\hline
\sphinxAtStartPar
tempest\_cinder
&
\sphinxAtStartPar
Cinder
\\
\hline
\sphinxAtStartPar
tempest\_keystone
&
\sphinxAtStartPar
Keystone
\\
\hline
\sphinxAtStartPar
rally\_sanity
&
\sphinxAtStartPar
General
\\
\hline
\sphinxAtStartPar
refstack\_defcore
&
\sphinxAtStartPar
General
\\
\hline
\sphinxAtStartPar
tempest\_full
&
\sphinxAtStartPar
General
\\
\hline
\sphinxAtStartPar
tempest\_slow
&
\sphinxAtStartPar
General
\\
\hline
\sphinxAtStartPar
tempest\_scenario
&
\sphinxAtStartPar
General
\\
\hline
\sphinxAtStartPar
patrole
&
\sphinxAtStartPar
Patrole
\\
\hline
\sphinxAtStartPar
tempest\_barbican
&
\sphinxAtStartPar
Barbican
\\
\hline
\sphinxAtStartPar
networking\sphinxhyphen{}bgpvpn
&
\sphinxAtStartPar
Networking BGP VPN
\\
\hline
\sphinxAtStartPar
networking\sphinxhyphen{}sfc
&
\sphinxAtStartPar
Networking SFC
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
To complete functional testing, Functest also integrates a few
\sphinxhref{https://docs.openstack.org/developer/performance-docs/methodologies/tools.html}{performance
tools}
(2\sphinxhyphen{}3 hours) as proposed by OpenStack:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Performance tools}\label{\detokenize{chapters/chapter04:id2}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Testcases
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Benchmarking
\\
\hline
\sphinxAtStartPar
rally\_full
&
\sphinxAtStartPar
Control Plane (API) testing
\\
\hline
\sphinxAtStartPar
rally\_jobs
&
\sphinxAtStartPar
Control Plane (API) testing
\\
\hline
\sphinxAtStartPar
vmtp
&
\sphinxAtStartPar
Data Plane testing
\\
\hline
\sphinxAtStartPar
shaker
&
\sphinxAtStartPar
Data Plane testing
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
And VNFs automatically deployed and tested :


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{VNFs}\label{\detokenize{chapters/chapter04:id3}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Testcases
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Benchmarking
\\
\hline
\sphinxAtStartPar
cloudify
&
\sphinxAtStartPar
Cloudify deployment
\\
\hline
\sphinxAtStartPar
cloudify\_ims
&
\sphinxAtStartPar
Clearwater IMS deployed via Coudify
\\
\hline
\sphinxAtStartPar
heat\_ims
&
\sphinxAtStartPar
Clearwater IMS deployed via Heat
\\
\hline
\sphinxAtStartPar
vyos\_vrouter
&
\sphinxAtStartPar
VyOS deployed via Cloudify
\\
\hline
\sphinxAtStartPar
juju\_epc
&
\sphinxAtStartPar
OAI deployed via Juju
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Functest should be considered as a whole as it meets multiple objectives
about the reference implementation:
\begin{itemize}
\item {} 
\sphinxAtStartPar
verify all APIs (services, advances, features, etc.) exposed by the
reference implementation

\item {} 
\sphinxAtStartPar
compare the reference implementation and local deployments from a
functional standpoint and from OpenStack control plane and dataplane
capabilities

\end{itemize}

\sphinxAtStartPar
It’s worth mentioning that Functest already takes into account the first
Anuket
\sphinxhref{https://git.opnfv.org/functest/tree/functest/ci/config\_patch.yaml\#n2}{profiles}.
Anuket should simply add the next Functest inputs according the
reference implementation:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/opnfv/functest/blob/stable/iruya/functest/utils/env.py\#L17}{Functest
inputs}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/opnfv/functest/blob/stable/iruya/functest/opnfv\_tests/openstack/tempest/custom\_tests/tempest\_conf.yaml}{tempest specific
configuration}

\end{itemize}

\sphinxAtStartPar
Additional links:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://functest.readthedocs.io/en/stable-iruya/}{Homepage}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://wiki.anuket.io/display/HOME/Functest+Wallaby}{Run Alpine Functest containers
(Wallaby)}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/collivier/ansible-role-xtesting\#readme}{Deploy your own Functest CI/CD
toolchains}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://build.opnfv.org/ci/view/functest/}{Functest gates}

\end{itemize}


\subsubsection{Yardstick}
\label{\detokenize{chapters/chapter04:yardstick}}

\subsubsection{Bottlenecks}
\label{\detokenize{chapters/chapter04:bottlenecks}}

\subsubsection{Test Tools}
\label{\detokenize{chapters/chapter04:test-tools}}\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Shaker: \sphinxurl{https://pyshaker.readthedocs.io/en/latest/} (The distributed
data\sphinxhyphen{}plane testing tool built for OpenStack)

\item {} 
\sphinxAtStartPar
Sonubuoy: \sphinxurl{https://sonobuoy.io/} It is a diagnostic tool that makes it
easier to understand the state of a Kubernetes cluster by running a
set of plugins (including Kubernetes conformance tests) in an
accessible and non\sphinxhyphen{}destructive manner.

\end{enumerate}


\subsubsection{Scenario Descriptor File (SDF)}
\label{\detokenize{chapters/chapter04:scenario-descriptor-file-sdf}}
\sphinxAtStartPar
As defined by Anuket, Scenario Descriptor File’s (SDF) will be utilized
to relay information from the Scenario Designer (or Test Manager), to
Release Managers, CI Pipeline Owners, and Installer Agents, to define
test scenario content, and specifications.

\sphinxAtStartPar
SDF’s will contain, but not limited to, the following Metadata,
Components, Deployment Options, Deployment Tools, and Hardware
prerequistes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Metadata}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Name

\item {} 
\sphinxAtStartPar
History

\item {} 
\sphinxAtStartPar
Purpose

\item {} 
\sphinxAtStartPar
Owner

\end{itemize}

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Components}
\begin{itemize}
\item {} 
\sphinxAtStartPar
e.g. SDN controllers

\item {} 
\sphinxAtStartPar
Versions

\item {} 
\sphinxAtStartPar
Optional features, e.g. NFV features

\end{itemize}

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Deployment Options}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Hardware types

\item {} 
\sphinxAtStartPar
Virtual deploy

\item {} 
\sphinxAtStartPar
HA, NUMA

\end{itemize}

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Deployment Tools}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Supporting installers.

\item {} 
\sphinxAtStartPar
Valid options per installer.

\end{itemize}

\end{itemize}


\subsection{OpenStack Testing Cookbook}
\label{\detokenize{chapters/chapter04:openstack-testing-cookbook}}
\sphinxAtStartPar
At the time of writing, the CI description file is hosted in Functest
and only runs the containers listed in RM/RA\sphinxhyphen{}1 Requirements. It will be
completed by the next Anuket mandatory test cases and then a new CI
description file will be proposed in CIRV tree.

\sphinxAtStartPar
Please note the next two points depending on the GNU/Linux distributions
and the network settings:
\begin{itemize}
\item {} 
\sphinxAtStartPar
SELinux: you may have to add \textendash{}system\sphinxhyphen{}site\sphinxhyphen{}packages when creating the
virtualenv (“Aborting, target uses selinux but python bindings
(libselinux\sphinxhyphen{}python) aren’t installed!”)

\item {} 
\sphinxAtStartPar
Proxy: you may set your proxy in env for Ansible and in systemd for
Docker \sphinxurl{https://docs.docker.com/config/daemon/systemd/\#httphttps-proxy}

\end{itemize}

\sphinxAtStartPar
To deploy your own CI toolchain running Anuket Compliance:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
virtualenv functest \PYGZhy{}\PYGZhy{}system\PYGZhy{}site\PYGZhy{}packages
. functest/bin/activate
pip install ansible
ansible\PYGZhy{}galaxy install collivier.xtesting
ansible\PYGZhy{}galaxy collection install ansible.posix community.general community.grafana kubernetes.core community.docker community.postgresql
git clone https://gerrit.opnfv.org/gerrit/functest functest\PYGZhy{}src
\PYG{o}{(}\PYG{n+nb}{cd} functest\PYGZhy{}src \PYG{o}{\PYGZam{}\PYGZam{}} git checkout \PYGZhy{}b stable/wallaby origin/stable/wallaby\PYG{o}{)}
ansible\PYGZhy{}playbook functest\PYGZhy{}src/ansible/site.cntt.yml
\end{sphinxVerbatim}


\subsubsection{OpenStack API testing configuration}
\label{\detokenize{chapters/chapter04:openstack-api-testing-configuration}}
\sphinxAtStartPar
Here is the default Functest tree as proposed in \sphinxhref{https://wiki.anuket.io/display/HOME/Functest+Wallaby}{Functest
Wallaby}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
/home/opnfv/functest/openstack.creds

\item {} 
\sphinxAtStartPar
/home/opnfv/functest/images

\end{itemize}

\sphinxAtStartPar
Download the images and fill /home/opnfv/functest/openstack.creds as
proposed in \sphinxhref{https://wiki.anuket.io/display/HOME/Functest+Wallaby}{Functest
Wallaby}

\sphinxAtStartPar
You may have to modify a few Functest env vars according to the SUT (see
env in \sphinxhref{https://wiki.anuket.io/display/HOME/Functest+Wallaby}{Functest
Wallaby}). Be
free to modify functest\sphinxhyphen{}src/ansible/host\_vars/127.0.0.1 at your
convenience and then to reconfigure the toolchain:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
ansible\PYGZhy{}playbook functest\PYGZhy{}src/ansible/site.cntt.yml
\end{sphinxVerbatim}


\subsubsection{Run Anuket OpenStack Testing}
\label{\detokenize{chapters/chapter04:run-anuket-openstack-testing}}
\sphinxAtStartPar
Open \sphinxurl{http://127.0.0.1:8080/job/functest-wallaby-daily/} in a web browser,
login as admin/admin and click on “Build with Parameters” (keep the
default build\_tag value).

\sphinxAtStartPar
If the System under test (SUT) is Anuket compliant, a link to the full
archive containing all test results and artifacts will be printed in
functest\sphinxhyphen{}wallaby\sphinxhyphen{}zip’s console. Be free to download it and then to send
it to any reviewer committee.

\sphinxAtStartPar
To clean your working dir:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
deactivate
rm \PYGZhy{}rf functest\PYGZhy{}src functest
\end{sphinxVerbatim}


\section{VNF Testing Framework Requirements}
\label{\detokenize{chapters/chapter05:vnf-testing-framework-requirements}}\label{\detokenize{chapters/chapter05::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter05:introduction}}
\sphinxAtStartPar
This chapter covers comprehensive VNF Conformance requirements for
enabling required process and steps to provide VNF badging based on
define scope of compliance and validation. This includes end to end test
framework requirements, badging entry and exit criteria, profiles to
reference, different stake holders and Conformance Methodologies by
using certified NFVi under NFVi badging program.


\subsection{Conformance Methodology}
\label{\detokenize{chapters/chapter05:conformance-methodology}}
\sphinxAtStartPar
It defines the end\sphinxhyphen{}end framework and process required for certifying the
given VNF.

\sphinxAtStartPar
\sphinxstylestrong{End\sphinxhyphen{}End framework}:

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{f3b0c214bc58c44406fd5b801d3dfc91}.png}
\caption{End\sphinxhyphen{}End framework}\label{\detokenize{chapters/chapter05:id1}}\end{figure}

\sphinxAtStartPar
Here, the steps 1\sphinxhyphen{}4 are NFVI related steps are covered in detail in
{\hyperref[\detokenize{chapters/chapter02:nfvi-conformance-requirements}]{\sphinxcrossref{\DUrole{std,std-ref}{NFVI Conformance Requirements}}}}.

\sphinxAtStartPar
Step\sphinxhyphen{}5. Interoperability validations for VNF functional testing defined.

\sphinxAtStartPar
Step\sphinxhyphen{}6. Interoperability validations for VNF performance testing defined
(IOPS, connection, threading, resource consumption).

\sphinxAtStartPar
Step\sphinxhyphen{}7. Sending requirements to the VNF requirements projects in terms
of t\sphinxhyphen{}shirt sizes, config settings, required for VNF/orchestration
validation.

\sphinxAtStartPar
\sphinxstylestrong{Conformance flow}:

\sphinxAtStartPar
The entry and exit criteria defined in below section are pre\sphinxhyphen{}requisities
for this flow.
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{VNF Vendors} submit the VNF into OVP Lab for Conformance (Fulfilling
the entry criteria is pre\sphinxhyphen{}requisities for this step.)

\item {} 
\sphinxAtStartPar
As part of OVP lab, already required test cases, test tools,
eco\sphinxhyphen{}system like MANO and appropriate certified NFVi to be setup as
defined part of entry criteria. This lab could either \sphinxstyleemphasis{OVP 3rd party
lab} or \sphinxstyleemphasis{VNF vendors}.

\end{enumerate}

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{f3b0c214bc58c44406fd5b801d3dfc89}.png}
\caption{OVP 3rd party lab}\label{\detokenize{chapters/chapter05:id2}}\end{figure}
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\setcounter{enumi}{2}
\item {} 
\sphinxAtStartPar
Once testing is completed done, test results will be submitted to the
OVP portal for community review along with additional information
such as product name, documentation links, primary company contact
information, etc.

\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{LFN CVC} community team reviewers will review the results submitted
and will approve or reject it based the details provided.

\item {} 
\sphinxAtStartPar
If reviewer rejected it, then step 2 and 3 will be ran again to
address the review comments. Otherwise once reviewer approved it,
corresponding VNF will be published into OVP VNF Portal with OVP
badge.

\item {} 
\begin{DUlineblock}{0em}
\item[] LFN staff will provide the certificate badge graphics and graphical
usage guidelines.
\item[] The OVP portal will reflect LFN’s disposition and assignment of the
certified VNF badge.
\end{DUlineblock}

\end{enumerate}

\sphinxAtStartPar
Now VNF is ready and \sphinxstyleemphasis{Telco Operators} can start consume it.


\subsubsection{Profiles Reference}
\label{\detokenize{chapters/chapter05:profiles-reference}}
\sphinxAtStartPar
The NFV Infrastructure (NFVI) is the totality of all hardware and
software components which build up the environment in which VNFs are
deployed, managed and executed. It is, therefore, inevitable that
different VNFs would require different capabilities and have different
expectations from it. So one of the main targets of Anuket is to define
an agnostic NFVI and removes any dependencies between VNFs and deployed
Infrastructure (NFVI) and offer NFVI to VNFs in an abstracted way with
defined capabilities and metrics. This would help operators to host
their Telco Workload (VNF) with different traffic types, behaviour and
from any vendor on a unified consistent Infrastructure. so as part of
VNF Conformance, its important to certify the VNF based on profiled
defined in :doc:ref\_model/chapters/chapter02\textasciigrave{}.

\sphinxAtStartPar
In \sphinxhref{https://cntt.readthedocs.io/projects/rm/en/latest/chapters/chapter02.html}{Workload Requirements \& Analysis},
following NFVi profiles are proposed as reference:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Basic}: for VNF that can tolerate resource over\sphinxhyphen{}subscription and
variable latency.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Network Intensive}: for VNF that require predictable computing
performance, high network throughput and low network latency.

\end{itemize}


\subsubsection{Protoype VNFs}
\label{\detokenize{chapters/chapter05:protoype-vnfs}}
\sphinxAtStartPar
A portion of the NFVI badging methodology includes Empirical Validation
with Reference Golden VNFs (aka CVC Validation) which will ensure the
NFVI runs with a set of VNF Families, or Classes, to mimic
production\sphinxhyphen{}like VNF connectivity. These tests are to 1) ensure
interoperability checks pass, and 2) there is an established baseline of
VNF behaviors and characters before vendor supplied VNFs are tested and
certified. In other words, empirical validations will confirm
performance and stability between Platform and VNF, such as validating
packet loss is within acceptable tolerances.


\subsection{Badging Requirements}
\label{\detokenize{chapters/chapter05:badging-requirements}}
\sphinxAtStartPar
\sphinxstylestrong{Defined}. \sphinxstyleemphasis{Badging} refers to the granting of a Conformance badge by
the OVP to Suppliers/Testers of Anuket NFVI+VNF upon demonstration the
testing performed confirms:
\begin{itemize}
\item {} 
\sphinxAtStartPar
NFVI adheres to Anuket RA/RM requirements.

\item {} 
\sphinxAtStartPar
Anuket certified VNFs functionally perform as expected (i.e. test
cases pass) on NFVI with acceptable levels of stability and
performance.

\end{itemize}

\sphinxAtStartPar
Following table shows the badging requirements with scope of mandatory
(must) or optional.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{The badging requirements}\label{\detokenize{chapters/chapter05:id3}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{100}|\X{10}{100}|\X{70}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement id
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Scope
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Details
\\
\hline
\sphinxAtStartPar
CVreq.VNF.001
&
\sphinxAtStartPar
must
&
\sphinxAtStartPar
Receive NFVi badge in lab setup per RI\sphinxhyphen{}1 standards, performing h/w
validations, performing s/w manifest validations, running nfvi
compliance, validation, and performance checks
\\
\hline
\sphinxAtStartPar
CVreq.VNF.002
&
\sphinxAtStartPar
must
&
\sphinxAtStartPar
met all entry and exit criteria
\\
\hline
\sphinxAtStartPar
CVreq.VNF.003
&
\sphinxAtStartPar
must
&
\sphinxAtStartPar
run interoperability validations, including instantiation,
communication / health, and removal
\\
\hline
\sphinxAtStartPar
CVreq.VNF.004
&
\sphinxAtStartPar
shall
&
\sphinxAtStartPar
utilize automation frameworks to run all required tests. Conformance
process would improve, if test framework satisfy the required defined
in this chapter under
\sphinxstyleemphasis{VNF Test Conformance platform requirements section}
\\
\hline
\sphinxAtStartPar
CVreq.VNF.005
&
\sphinxAtStartPar
must
&
\sphinxAtStartPar
pass all required tests
\\
\hline
\sphinxAtStartPar
CVreq.VNF.006
&
\sphinxAtStartPar
must
&
\sphinxAtStartPar
prepare release notes, with issues known, their severity and magnitude,
mitigation plan
\\
\hline
\sphinxAtStartPar
CVreq.VNF.007
&
\sphinxAtStartPar
must
&
\sphinxAtStartPar
publish results in defined normalized output
\\
\hline
\sphinxAtStartPar
CVreq.VNF.008
&
\sphinxAtStartPar
must
&
\sphinxAtStartPar
respond /closed badging inquiries
\\
\hline
\sphinxAtStartPar
CVreq.VNF.010
&
\sphinxAtStartPar
optional
&
\sphinxAtStartPar
for bading VNF supplier can choose to run their own test
harnesses/suites to validate VNF functional and performance behaviors
and performance
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Badging Scope}
\label{\detokenize{chapters/chapter05:badging-scope}}
\sphinxAtStartPar
The VNF badging includes:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
NFVi Verifications (Compliance): Manifest Verifications will ensure
the NFVI is compliant, and delivered for testing, with hardware and
software profile specifications defined by the Ref Model and Ref
Architecture.

\item {} 
\sphinxAtStartPar
Empirical Validation with Reference VNF (Validation): Empirical
Validation with Reference Golden VNFs will ensure the NFVI runs with
a set of VNF Families, or Classes, to mimic production\sphinxhyphen{}like VNFs to
baseline infrastructure conformance.

\item {} 
\sphinxAtStartPar
Candidate VNF Validation (Validation \& Performance): Candidate VNF
Validation will ensure complete interoperability of VNF behavior on
the NFVI leveraging VVP/VNFSDK test suites to ensure VNF can be spun
up, modified, or removed, on the target NFVI (aka Interoperability).

\end{enumerate}

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{f3b0c214bc58c44406fd5b801d3dfc90}.png}
\caption{Candidate VNF Validation}\label{\detokenize{chapters/chapter05:id4}}\end{figure}


\subsubsection{Entry criteria}
\label{\detokenize{chapters/chapter05:entry-criteria}}
\sphinxAtStartPar
Before entering into the VNF badging process, VNF needs to satisfy the
following requirements as entry criteria:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Environment Requirements} : Published details providing evidence
that a RAx compliant lab has been implemented, meeting requirements
set forth in respective RM and RAx documentation for features,
options, and capabilities needed for VNF test validations. Expected
information includes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Lab Flavor

\item {} 
\sphinxAtStartPar
Component software rev levels

\item {} 
\sphinxAtStartPar
Confirmation of compatibility with external systems

\item {} 
\sphinxAtStartPar
Tenant needs identified

\item {} 
\sphinxAtStartPar
All connectivity, network, image, VMs, delivered with successful
pairwise tests

\item {} 
\sphinxAtStartPar
Lab instrumented for proper monitoring

\end{itemize}

\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{VNF artifact} : VNF cloud (native) image, VNF configurations and
guidelines, automation scripts, etc

\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{NFVi profiles}: List of supporting OVP Certified Anuket compliant
NFVi

\item {} 
\sphinxAtStartPar
Completed Security review report

\item {} 
\sphinxAtStartPar
Vendor specific test cases and its deployment and usage guidelines

\end{itemize}


\subsubsection{Exit criteria}
\label{\detokenize{chapters/chapter05:exit-criteria}}
\sphinxAtStartPar
VNF Conformance testing should be completed with following exit
criteria:
\begin{itemize}
\item {} 
\sphinxAtStartPar
All required test cases should be passed

\item {} 
\sphinxAtStartPar
No outstanding high severity issues and other known issues to be
documented

\item {} 
\sphinxAtStartPar
Release notes

\item {} 
\sphinxAtStartPar
Provided with required installation guide, configuration guide, etc.

\item {} 
\sphinxAtStartPar
Test results collated, centralized, and normalized, with a final
report generated showing status of the test scenario/case (e.g. Pass,
Fail, Skip, Measurement Success/Fail, etc), along with traceability
to a functional, or non\sphinxhyphen{}functional, requirement

\end{itemize}


\subsection{VNF Test Conformance platform Requirements}
\label{\detokenize{chapters/chapter05:vnf-test-conformance-platform-requirements}}
\sphinxAtStartPar
Test platform requirements are provided to address test case design,
distribution, execution and result reporting along with required
artifacts and environments in place and are defined based on below
scope.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{c665a3d13461f67ea8729042cf8d975d}.png}
\caption{Test platform requirements}\label{\detokenize{chapters/chapter05:id5}}\end{figure}


\subsubsection{Standards/Profiles}
\label{\detokenize{chapters/chapter05:standards-profiles}}\begin{itemize}
\item {} 
\sphinxAtStartPar
ETSI (TOSCA)

\item {} 
\sphinxAtStartPar
GSMA

\item {} 
\sphinxAtStartPar
ONAP VNFREQS (HOT)

\end{itemize}


\subsubsection{Test cases}
\label{\detokenize{chapters/chapter05:test-cases}}
\sphinxAtStartPar
Refer \sphinxstyleemphasis{chapter RC\sphinxhyphen{}06} for more details on test case requirements defined
for VNF under Anuket. Platform should support to managed and execute
these test cases.

\begin{sphinxadmonition}{note}{Note:}
\sphinxAtStartPar
For Conformance, only compliance and verification test cases will
be considered, but in future, it could be extent to validation and
Performance related testing.
\end{sphinxadmonition}


\paragraph{Compliance}
\label{\detokenize{chapters/chapter05:compliance}}
\sphinxAtStartPar
Perform compliance check based on
\begin{itemize}
\item {} 
\sphinxAtStartPar
TOSCA using ETSI SOL004 \& SOL001

\item {} 
\sphinxAtStartPar
OpenStack HOT using ONAP VNFREQS

\item {} 
\sphinxAtStartPar
GSMA profile as defined in \sphinxstyleemphasis{chapter RM\sphinxhyphen{}04}.

\end{itemize}


\paragraph{Verification}
\label{\detokenize{chapters/chapter05:verification}}
\sphinxAtStartPar
Perform on\sphinxhyphen{}boarding/ verification life cycle operation (from
instantiation, configuration, update, termination) using MANO supporting
Anuket compliant NFVI.


\paragraph{Validation}
\label{\detokenize{chapters/chapter05:validation}}
\sphinxAtStartPar
Perform various VNF type specific functionality operations on Anuket RA
\& RM compliant NFVI


\paragraph{Performance}
\label{\detokenize{chapters/chapter05:performance}}
\sphinxAtStartPar
Perform various performance related testing and facilitate for
benchmarking the VNF performance on different profile and scenarios.


\subsubsection{Eco\sphinxhyphen{}system MANO/NFVI}
\label{\detokenize{chapters/chapter05:eco-system-mano-nfvi}}
\sphinxAtStartPar
Platform would support to execute various test cases on Anuket RA \& RM
compliant NFVi along with required MANO system supporting these NFVi.


\subsubsection{VNF}
\label{\detokenize{chapters/chapter05:vnf}}
\sphinxAtStartPar
Suppliers of VNFs/CNFs seeking to receive VNF Conformance badges must
first ensure their testing is performed against a compliant RM/RA
architecture supporting all capabilities, features, and services defined
by the respective \sphinxstyleemphasis{RM/RA requirements}. More specifically, the VNF
Supplier must ensure their implementation of the RM/RA receives the NFVI
Conformance badge prior to starting VNF testing. Finally, to receive VNF
Conformance, the test platform will need to support TOSCA and HOT based
VNF distros.

\sphinxAtStartPar
In addition, Platform should be able to perform the required test case
management and executions and produce the result the CVC OVP portal for
Conformance process along with required testing foot print details. So
overall scoped example architecture could be as below:

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{2269537e91994b5b49858734fe73bbb1}.png}
\caption{VNF Test Certification Platform}\label{\detokenize{chapters/chapter05:id6}}\end{figure}


\subsubsection{Test Case Model}
\label{\detokenize{chapters/chapter05:test-case-model}}
\sphinxAtStartPar
As there are more number of VNF at different levels of networking such
as access, transport and core level as well as OSI level L0\sphinxhyphen{}L7. Every
network function provides set of pre\sphinxhyphen{}defined features and
functionalities. So its important to model test cases for every
functionality to identify it uniquely and use it as part of test flow
design.

\sphinxAtStartPar
As part of modeling its very important to capture the following details
\begin{itemize}
\item {} 
\sphinxAtStartPar
Test case Name

\item {} 
\sphinxAtStartPar
Test case description

\item {} 
\sphinxAtStartPar
Virtual Network function Name

\item {} 
\sphinxAtStartPar
Network function Feature/functionality name

\item {} 
\sphinxAtStartPar
Test case input parameters

\item {} 
\sphinxAtStartPar
Test case result attributes

\item {} 
\sphinxAtStartPar
Test case version

\end{itemize}

\sphinxAtStartPar
while implementing the test cases, this model would act as specification
and as it captures the input and output, it would help while designing
the test flow which will help to execute set of test cases in
pre\sphinxhyphen{}defined flow.


\subsubsection{Test case management}
\label{\detokenize{chapters/chapter05:test-case-management}}\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Test case} : On\sphinxhyphen{}board/discover, update, disable/enable, delete

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Test suite} : On\sphinxhyphen{}board/discover, update, disable/enable, delete

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Test flow} : design/discover, update, disable/enable, delete

\end{itemize}


\subsubsection{Test Execution management}
\label{\detokenize{chapters/chapter05:test-execution-management}}\begin{itemize}
\item {} 
\begin{DUlineblock}{0em}
\item[] \sphinxstylestrong{Run\sphinxhyphen{}time}: One of the common nature of the test environment is
heterogeneous and multiple vendors and open communities would
provide various test tool and environment to support execution of
test cases developed under different run\sphinxhyphen{}times
\item[] (JVM, Python, Shell, Container, Cloud VM, etc)
\end{DUlineblock}

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{RPC}: In order to enable the scaling/remote execution, it should
be enabled with required RPC support.

\end{itemize}

\sphinxAtStartPar
When VNF test platform execute the test cases, it captures the
footprints of test case execution along with results, which are made
available to user and integrated system for consuming.


\subsubsection{Test Result management}
\label{\detokenize{chapters/chapter05:test-result-management}}
\sphinxAtStartPar
\sphinxstylestrong{Categorization}. Test suites will be categorized as
Functional/Platform or Performance based.

\sphinxAtStartPar
\sphinxstylestrong{Results.} Test results reporting will be communicated as a boolean
(pass/fail), or Measurements Only.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Functional Pass/Fail} signals the assertions set in a test script
verify the Functional Requirements (FR) has met its stated objective
as delivered by the developer. This will consist of both positive
validation of expected behavior, as well as negative based testing
when to confirm error handling is working as expected.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Performance\sphinxhyphen{}based Pass/Fail} determination will be made by
comparing Non\sphinxhyphen{}Functional (NFR) KPIs (obtained after testing) with the
Golden KPIs. Some of the examples of performance KPIs include, but
not limited to: TCP bandwidth, UDP throughput, Memory latency,
Jitter, IOPS etc.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Measurement Results}. Baseline Measurements will be performed when
there are no benchmark standards to compare results, or established
FRs/NFRs for which to gauge application / platform behavior in an
integrated environment, or under load conditions. In these cases,
test results will be executed to measure the application, platform,
then prepare FRs/NFRs for subsequent enhancements and test runs.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Formats}. As part of execution management, system produces the result
in JSON format which can be represented in various form like YAML, CSV,
Table, etc.

\sphinxAtStartPar
\sphinxstylestrong{Search \& Reporting}. Search would help to query the test results
based on various fact such as test case, VNF, date of execution,
environment, etc. and produce the report in various format like
pie\sphinxhyphen{}chart, success rates, etc

\sphinxAtStartPar
\sphinxstylestrong{Collation | Portal}. The following criteria will be applied to the
collation and presentation of test\sphinxhyphen{}runs seeking Conformance:
\begin{itemize}
\item {} 
\sphinxAtStartPar
RA number and name (e.g. RA\sphinxhyphen{}1 OpenStack)

\item {} 
\sphinxAtStartPar
Version of software tested (e.g. OpenStack Ocata)

\item {} 
\sphinxAtStartPar
Normalized results will be collated across all test runs
(i.e. centralized database)

\item {} 
\sphinxAtStartPar
Clear time stamps of test runs will be provided.

\item {} 
\sphinxAtStartPar
Identification of test engineer / executor.

\item {} 
\sphinxAtStartPar
Traceability to requirements.

\item {} 
\sphinxAtStartPar
Summarized conclusion if conditions warrant test Conformance (see
Badging Section).

\item {} 
\sphinxAtStartPar
Portal contains links to Conformance badge(s) received.

\end{itemize}


\subsubsection{Test Artifact management}
\label{\detokenize{chapters/chapter05:test-artifact-management}}
\sphinxAtStartPar
As part of testing various binaries, configurations, images, scripts
,etc would be used during test cases building or execution and Version
artifact supports such as VNF CSAR.


\subsubsection{Test Scenario management}
\label{\detokenize{chapters/chapter05:test-scenario-management}}
\sphinxAtStartPar
Allow to create repeatable scenario includes test cases, artifacts and
profiles.

\sphinxAtStartPar
It helps to create dynamic testing scenario development and testing from
the existing test cases and flows along with required artifacts and
profiles. It allows to run repeated testing with one or different
profiles.


\subsubsection{Test Profile management}
\label{\detokenize{chapters/chapter05:test-profile-management}}
\sphinxAtStartPar
For every test case execution needs to be configured with required
environments and predefined test input parameter values. This is
provided by means of profile

\sphinxAtStartPar
Profile should be having option to include other profiles to manage the
hierarchy of them.

\sphinxAtStartPar
As part of profile, testing environment URL, credentials and related
security keys are captured and while running the test cases, user would
be able to inputs the required profile in place of actual inputs and
artifacts.

\sphinxAtStartPar
Also helps in Managing System under test configuration and multiple MANO
/ NFVI and related eco system management elements.


\subsubsection{Tenant \& User management}
\label{\detokenize{chapters/chapter05:tenant-user-management}}
\sphinxAtStartPar
Testing involves design, distribution by different user roles and
executed across multiple tenant’s environments.


\subsubsection{Conformance management \& integration}
\label{\detokenize{chapters/chapter05:conformance-management-integration}}
\sphinxAtStartPar
Platform should have integration with OVP Conformance portal for
submitting results with OVP defined format.

\sphinxAtStartPar
It should enable repository of certified VNFs which can be used for
testing validation and performance.


\subsubsection{User \& System interfaces}
\label{\detokenize{chapters/chapter05:user-system-interfaces}}
\sphinxAtStartPar
\sphinxstylestrong{User interface}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
CLI

\item {} 
\sphinxAtStartPar
Web portal

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Programming interface}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
REST API

\item {} 
\sphinxAtStartPar
gRPC

\end{itemize}


\subsubsection{Deliverables}
\label{\detokenize{chapters/chapter05:deliverables}}
\sphinxAtStartPar
Platform should be able to get deployed in both container and cloud
environments. so following model deliverables would enable it:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Docker image based installation

\item {} 
\sphinxAtStartPar
Standalone installation scripts and zip artifact

\end{itemize}


\subsection{VNF Test Cases Requirements}
\label{\detokenize{chapters/chapter05:vnf-test-cases-requirements}}

\subsubsection{Rationale}
\label{\detokenize{chapters/chapter05:rationale}}
\sphinxAtStartPar
Network functions virtualization (NFV) and softwaredefined networking
(SDN) offer service providers increased service agility, OpEx
improvements, and back\sphinxhyphen{}office automation. Disaggregation, the approach
of decoupling the various layers of the stack, from hardware, to
NFVI/VIM software, to dataplane acceleration, SDN controllers, MANO
components, and VNFs, enables multi\sphinxhyphen{}vendor deployments with
best\sphinxhyphen{}of\sphinxhyphen{}breed options at each layer.

\sphinxAtStartPar
The Anuket specifications define the required architecture and model for
NFVI which will help to decouple the various commercial product layers
and it is important to define and certify the VNF and NFVI. Therefore,
in addition to verify general NFVI capabilities based on Anuket
RM/RA/RI, it is also necessary to verify that VNFs can provide
virtualization functions normally based on the Anuket\sphinxhyphen{}compliant NFVI. So
the VNF testing should at least include:
Compliance, verification, validation, Performance. With the improvement
of specifications, the types of tests may continue to add in the future.

\sphinxAtStartPar
In this chapter, the scope and requirements of VNF test cases are
defined as reference for VNF Conformance, which helps to perform the
various compliance and verification (C\&V) testing and submit results to
LFN OVP Conformance portal.


\subsubsection{Assumptions}
\label{\detokenize{chapters/chapter05:assumptions}}
\sphinxAtStartPar
Here lists the assumptions for VNF Conformance: \sphinxhyphen{} NFVI is ready and it
should be an Anuket\sphinxhyphen{}compliant NFVI \sphinxhyphen{} VNF template is ready to deploy and
certificate \sphinxhyphen{} VNF Test environment is ready, the test environment
contains test functions and entities(NFVI, MANO, VNF Test Platform, VNF
Test Tools) to enable controlling the test execution and collecting the
test measurements. \sphinxhyphen{} VNF Test Platform has been integrated with CICD
chain \sphinxhyphen{} VNF test result can be generated with OVP defined format


\subsubsection{Developer Deliverables}
\label{\detokenize{chapters/chapter05:developer-deliverables}}
\sphinxAtStartPar
This section define the developer Deliverables (artifacts),the following
list the expectations and deliverables we expect from developers in
order to achieve the VNF Conformance: \sphinxhyphen{} VNF test cases
model/scripts/programs \sphinxhyphen{} VNF test cases configuration/profile \sphinxhyphen{} VNF test
tools


\subsubsection{Requirement Type}
\label{\detokenize{chapters/chapter05:requirement-type}}
\sphinxAtStartPar
VNF test cases are used to verify whether the virtualization network
functions can be deployed on the Anuket\sphinxhyphen{}compliant NFVI and provide
normal functions and meet performance, security and other requirements.

\sphinxAtStartPar
By running these VNF test cases and analysis the test results, can be
used for VNF compliance, verfication,validation and performance
Conformance and help on Anuket\sphinxhyphen{}compliant NFVI validation and performance
Conformance.

\sphinxAtStartPar
All the VNF test cases should be supported and run by VNF E2E
Conformance and verification Framework and generate outputs, logs to
identify whether the test passed or failed.

\sphinxAtStartPar
Anuket defines the following four category testing which should be
consistent with the VNF test category defined by OVP.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{VNF Test Case categories}\label{\detokenize{chapters/chapter05:id7}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{100}|\X{20}{100}|\X{20}{100}|\X{40}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
VNF Test Case Category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement Number
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Type (Measurement/Boolean)
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Definition/Description
\\
\hline
\sphinxAtStartPar
Compliance
&
\sphinxAtStartPar
VNF.COMPreq.001
&
\sphinxAtStartPar
Boolean (i.e. Pass/Fail)
&
\sphinxAtStartPar
Test case “must” perform a platform check against the OpenStack
requirements and VNF package structure and syntax requirements
\\
\hline
\sphinxAtStartPar
Verification
&
\sphinxAtStartPar
VNF.VERIFYreq.001
&
\sphinxAtStartPar
Boolean (i.e. Pass/Fail)
&
\sphinxAtStartPar
Test case “must” perform on\sphinxhyphen{}boarding/ verification life cycle operation
validation
\\
\hline
\sphinxAtStartPar
Validation
&
\sphinxAtStartPar
VNF.VALIDreq.001
&
\sphinxAtStartPar
Boolean (i.e. Pass/Fail)
&
\sphinxAtStartPar
Test case “must” perform API validation tests to verify operability
\\
\hline
\sphinxAtStartPar
Performance
&
\sphinxAtStartPar
VNF.PERFreq.001
&
\sphinxAtStartPar
Measurement
&
\sphinxAtStartPar
Test case “must” execute various performance related testing and
facilitate for benchmarking the VNF performance on different profile
and scenarios
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\begin{sphinxadmonition}{note}{Note:}
\sphinxAtStartPar
The four category testing can be gradually supported and in the
future, will also cover secutiry and other test category.
\end{sphinxadmonition}


\subsubsection{Interaction Type}
\label{\detokenize{chapters/chapter05:interaction-type}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Describe the types of Interactions: Extended Topology, Complex
(Akraino), Functional, HA, Fault, Interoperability

\end{itemize}


\subsubsection{Performance Profiles}
\label{\detokenize{chapters/chapter05:performance-profiles}}
\sphinxAtStartPar
Performance profiles are not in the scope of current release, and in
future it would need to align with \sphinxstyleemphasis{chapter RM\sphinxhyphen{}4} defined measurements.


\subsubsection{VNF Class/Family and Characteristics}
\label{\detokenize{chapters/chapter05:vnf-class-family-and-characteristics}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Describe and provide a Table of VNF Class/Family \& Characteristics of
Each

\end{itemize}

\sphinxAtStartPar
The communication network usually consists of three parts: access
network, transmission network/bearer network and core network. Following
are some examples of network elements for each type of network


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Communication network}\label{\detokenize{chapters/chapter05:id8}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{40}{100}|\X{60}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Network Type
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Network Elements
\\
\hline
\sphinxAtStartPar
Access Network
&
\sphinxAtStartPar
Including mobile access network, wireless access network, wired access
network
\\
\hline
\sphinxAtStartPar
Transport network \& Bearer network
&
\sphinxAtStartPar
Including Trunk Optical Transport Network, Metro transport network, IP
backbone network, etc.
\\
\hline
\sphinxAtStartPar
Core Network
&
\sphinxAtStartPar
Circuit domain, including MSC/VLR, GMSC, MGW, NPMSC, HLR/AUC,
NPHLR, HSS, etc, Packet domain devices, including MME, SAE GW, EPC CG,
EPC DNS, PCC, etc; Core network equipment for IoT private network,
including PGW/GGSN, PCRF, HSS/HLR, etc, 5G core network element,
including AMF, SMF, UPF, UDM/UDR/AUS F, PCF, NSSF, NRF, SMSF, etc
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
In addition to the above network elements, there are some other data
communication network element, including FW, DNS, Router, GW, etc

\sphinxAtStartPar
According to the current level of the entire network virtualization, the
core network already has many VNFs, and also includes some
datacom\sphinxhyphen{}type(data communication) VNFs.

\sphinxAtStartPar
We can also classify VNFs based on the level of VNF operation:
\begin{enumerate}
\sphinxsetlistlabels{\alph}{enumi}{enumii}{}{)}%
\item {} 
\sphinxAtStartPar
VNFs that operate at Layer 2 or Layer 3 are primarily involved in
switching or routing packets at these layers. Examples include
vRouter, vBNG, vCE device, or vSwitch.

\item {} 
\sphinxAtStartPar
VNFs that operate at Layer 4 through Layer 7 and are involved in
forwarding, dropping, filtering or redirecting packets at Layer 4
through 7. Examples include vFirewall, vADC, vIDS/vIPS, or vWAN
Accelerator.

\item {} 
\sphinxAtStartPar
VNFs that are involved in the dataplane forwarding through the
evolved packet core.

\end{enumerate}


\subsubsection{Measurement}
\label{\detokenize{chapters/chapter05:measurement}}
\sphinxAtStartPar
As part of Conformance testing, following measurement would help for
evaluating the badging:
\begin{itemize}
\item {} 
\sphinxAtStartPar
VNF type defined as part of \sphinxstyleemphasis{Chapter RM\sphinxhyphen{}02} and its profile used for
testing.

\item {} 
\sphinxAtStartPar
Test cases and their test results including the test case outputs,
logs

\item {} 
\sphinxAtStartPar
VNF model type (TOSCA/HOT)

\item {} 
\sphinxAtStartPar
Test case pass/failed

\item {} 
\sphinxAtStartPar
Different NFVi profiles used and LAB reference identifier

\item {} 
\sphinxAtStartPar
Test owner (point of contact)

\end{itemize}


\subsubsection{VNF Test Cases}
\label{\detokenize{chapters/chapter05:vnf-test-cases}}

\paragraph{Compliance test cases}
\label{\detokenize{chapters/chapter05:compliance-test-cases}}
\sphinxAtStartPar
Currently, there VNFs can be packaged as HEAT templates or in a CSAR
file using TOSCA and OVP has supported the VNF compliance test
cases(compliance check based on TOSCA using ETSI SOL004 \&
SOL001; OpenStack HOT using ONAP VNFREQS; GSMA profile), all the OVP
supported test case can be found in the following two link:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{OVP supported test case}\label{\detokenize{chapters/chapter05:id9}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{40}{100}|\X{60}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Test Cases
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Link
\\
\hline
\sphinxAtStartPar
Heat Test Cases
&
\sphinxAtStartPar
\sphinxurl{https://docs.onap.org/projects/onap-vnfrqts-testcases/en/istanbul/Appendix.html\#list-of-requirements-with-associated-tests}
\\
\hline
\sphinxAtStartPar
Tosca Test Cases
&
\sphinxAtStartPar
\sphinxurl{https://docs.onap.org/projects/onap-vnfsdk-model/en/istanbul/files/csar-validation.html}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Above compliance test cases defination can be found
\sphinxurl{https://github.com/onap/vnfsdk-validation/tree/master/csarvalidation/src/main/resources/open-cli-schema}

\sphinxAtStartPar
In order to adapt Anuket specification, more compliance test case will
be added here.


\paragraph{Verification test cases}
\label{\detokenize{chapters/chapter05:verification-test-cases}}
\sphinxAtStartPar
In general, the VNF Manager, in collaboration with the NFV
Orchestrator, the VIM and the EM, is responsible for managing a VNF’s
lifecycle. The lifecycle phases are listed below:
\begin{itemize}
\item {} 
\sphinxAtStartPar
VNF on\sphinxhyphen{}boarding, it refers to VNF package onboarding to
service/resouce Orchestrator

\item {} 
\sphinxAtStartPar
VNF instantiation, once the VNF is instantiated, its associated VNFCs
have been successfully instantiated and have been allocated necessary
NFVI resources\sphinxhyphen{}

\item {} 
\sphinxAtStartPar
VNF scaling/updating, it means the VNF can scale or update by
allocating more or less NFVI resources

\item {} 
\sphinxAtStartPar
VNF termination, any NFVI resources consumed by the VNF can be
cleaned up and released.

\end{itemize}

\sphinxAtStartPar
OVP has also supported the lifecycle test
case:https://wiki.lfnetworking.org/display/LN/VNF+Validation+Minimum+Viable+Product?src=contextnavpagetreemode


\paragraph{Validation Test cases}
\label{\detokenize{chapters/chapter05:validation-test-cases}}
\sphinxAtStartPar
From the current situation of operators, there are usually corresponding
functional test specifications for each types of VNFs. Therefore,
different types of VNFs have different functional test cases. Normally,
functional tests for VNFs require the cooperation of surrounding VNFs.
Or use the instruments to simulate the functions of surrounding VNFs for
testing. Therefore, different test cases need to be defined according to
different types of VNFs


\paragraph{Performance Test cases}
\label{\detokenize{chapters/chapter05:performance-test-cases}}
\sphinxAtStartPar
This is the same as what described in validation test cases, the
performance test cases need to be defined according to different types
of VNFs. Combined with the classification of VNF, according to the
protocol level that VNF operates, it can include:
\begin{itemize}
\item {} 
\sphinxAtStartPar
VNF data plane benchmarking, like forwarding Performance
Benchmarking,Long duration traffic testing, low misrouting and so on.

\item {} 
\sphinxAtStartPar
VNF control plane benchmarking, like throughput

\item {} 
\sphinxAtStartPar
VNF user plane benchmarking, like Packet Loss,Latency, Packet Delay

\end{itemize}

\sphinxAtStartPar
ETSI spec has also defined the testing method
\sphinxurl{https://www.etsi.org/deliver/etsi\_gs/NFV-TST/001\_099/001/01.01.01\_60/gs\_nfv-tst001v010101p.pdf}


\section{VNF Test Cases and Traceability to Requirements}
\label{\detokenize{chapters/chapter06:vnf-test-cases-and-traceability-to-requirements}}\label{\detokenize{chapters/chapter06::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter06:introduction}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Provide an overview of the purpose for the VNF TC Traceability to RM
Requirements chapter

\item {} 
\sphinxAtStartPar
Start with defining requirements

\item {} 
\sphinxAtStartPar
Finish with tracing test cases to requirements

\end{itemize}


\subsection{RM/RA1 Requirements}
\label{\detokenize{chapters/chapter06:rm-ra1-requirements}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Define RM/RA\sphinxhyphen{}1 Openstack requirements

\end{itemize}


\subsection{Test Case Traceability}
\label{\detokenize{chapters/chapter06:test-case-traceability}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Define and provide a table mapping Test Cases to Requirements

\end{itemize}


\section{VNF Testing Cookbook}
\label{\detokenize{chapters/chapter07:vnf-testing-cookbook}}\label{\detokenize{chapters/chapter07::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter07:introduction}}
\sphinxAtStartPar
Define the purpose of the chapter which is to:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Identify Framework Needs, Goals, and Dependencies

\item {} 
\sphinxAtStartPar
Define Opensource Integration (OVP, Functest, CVC, others)

\item {} 
\sphinxAtStartPar
Provide Automation Toolchain (list, topology, flow)

\end{itemize}


\section{Gap analysis and Development}
\label{\detokenize{chapters/chapter08:gap-analysis-and-development}}\label{\detokenize{chapters/chapter08::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter08:introduction}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Describe the purpose of this chapter, which includes, but not limited
to:

\item {} 
\sphinxAtStartPar
Test Case Gaps (analysis)

\item {} 
\sphinxAtStartPar
Automation Gaps

\item {} 
\sphinxAtStartPar
OpenStack release Comparisons

\end{itemize}


\subsection{Openstack Release Comparisons}
\label{\detokenize{chapters/chapter08:openstack-release-comparisons}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Provide details, preferably in table format, comparing OpenStack
releases based on Wallaby baseline for RI\sphinxhyphen{}1

\end{itemize}


\subsection{Test Case Gaps}
\label{\detokenize{chapters/chapter08:test-case-gaps}}
\sphinxAtStartPar
Anuket has developed many test cases in the different \sphinxhref{https://wiki.opnfv.org/display/testing/TestPerf}{test
projects} which can
quickly improve RC. As listed in
{\hyperref[\detokenize{chapters/chapter02:nfvi-conformance-requirements}]{\sphinxcrossref{\DUrole{std,std-ref}{NFVI Conformance Requirements}}}},
porting all the existing testcases to Xtesting will unify the test case
execution and simplify the test integration as required by RC. Here are all the
related issues:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/cntt-n/CNTT/issues/511}{port VinePerf to
Xtesting}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://github.com/cntt-n/CNTT/issues/865}{port NFVbench testcases to
Xtesting}

\end{itemize}

\sphinxAtStartPar
Here are the possible new test cases which could be integrated in the
existing Anuket projects to improve RC:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Possible new test cases}\label{\detokenize{chapters/chapter08:id1}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline

\sphinxAtStartPar
Issues
&
\sphinxAtStartPar
Requirements
\\
\hline
\sphinxAtStartPar
\sphinxhref{https://github.com/cntt-n/CNTT/issues/508}{integrate KloudBuster in Functest}
&
\sphinxAtStartPar
disk benchmarking
\\
\hline
\sphinxAtStartPar
\sphinxhref{https://github.com/cntt-n/CNTT/issues/916}{add tempest\sphinxhyphen{}stress in Functest}
&
\sphinxAtStartPar
stress testing
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\subsection{Framework Gaps}
\label{\detokenize{chapters/chapter08:framework-gaps}}
\sphinxAtStartPar
As proposed in \sphinxhref{https://github.com/cntt-n/CNTT/issues/917}{port VTP test cases to
Xtesting}, VTP selected in
{\hyperref[\detokenize{chapters/chapter05:vnf-testing-framework-requirements}]{\sphinxcrossref{\DUrole{std,std-ref}{VNF Testing Framework Requirements}}}}
requires small adaptations to fully fulfill the current
{\hyperref[\detokenize{chapters/chapter02:nfvi-conformance-requirements}]{\sphinxcrossref{\DUrole{std,std-ref}{NFVI Conformance Requirements}}}}.
It seems trivial changes as VTP proposed a REST API but will ensure that both
NFVI and VNF testing can be executed in the same CI toolchain very easily.



\renewcommand{\indexname}{Index}
\printindex
\end{document}