%% Generated by Sphinx.
\def\sphinxdocclass{article}
\documentclass[letterpaper,10pt,english]{sphinxhowto}
\ifdefined\pdfpxdimen
   \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
\fi \sphinxpxdimen=.75bp\relax
\ifdefined\pdfimageresolution
    \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax
\fi
%% let collapsable pdf bookmarks panel have high depth per default
\PassOptionsToPackage{bookmarksdepth=5}{hyperref}

\PassOptionsToPackage{warn}{textcomp}
\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
% support both utf8 and utf8x syntaxes
  \ifdefined\DeclareUnicodeCharacterAsOptional
    \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}}
  \else
    \let\sphinxDUC\DeclareUnicodeCharacter
  \fi
  \sphinxDUC{00A0}{\nobreakspace}
  \sphinxDUC{2500}{\sphinxunichar{2500}}
  \sphinxDUC{2502}{\sphinxunichar{2502}}
  \sphinxDUC{2514}{\sphinxunichar{2514}}
  \sphinxDUC{251C}{\sphinxunichar{251C}}
  \sphinxDUC{2572}{\textbackslash}
\fi
\usepackage{cmap}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{babel}



\usepackage{tgtermes}
\usepackage{tgheros}
\renewcommand{\ttdefault}{txtt}



\usepackage[Bjarne]{fncychap}
\usepackage[,numfigreset=2,mathnumfig]{sphinx}

\fvset{fontsize=auto}
\usepackage{geometry}


% Include hyperref last.
\usepackage{hyperref}
% Fix anchor placement for figures with captions.
\usepackage{hypcap}% it must be loaded after hyperref.
% Set up styles of URL: it should be placed after hyperref.
\urlstyle{same}


\usepackage{sphinxmessages}
\setcounter{tocdepth}{1}



\title{Anuket Reference Conformance for Kubernetes (RC2)}
\date{Aug 01, 2022}
\release{}
\author{Anuket}
\newcommand{\sphinxlogo}{\vbox{}}
\renewcommand{\releasename}{}
\makeindex
\begin{document}

\pagestyle{empty}
\sphinxmaketitle
\pagestyle{plain}
\sphinxtableofcontents
\pagestyle{normal}
\phantomsection\label{\detokenize{index::doc}}



\section{Introduction}
\label{\detokenize{chapters/chapter01:introduction}}\label{\detokenize{chapters/chapter01::doc}}

\subsection{Executive Summary}
\label{\detokenize{chapters/chapter01:executive-summary}}
\sphinxAtStartPar
The Reference Conformance for the Kubernetes\sphinxhyphen{}based workstream (RC2) was
established to ensure implementations of the Anuket Reference
Architecture 2 (RA2), such as the Reference Implementation 2 (RI2), meet
functional and performance requirements specified in RA2 and the Anuket
Reference Model (RM). Cloud infrastructure and workload verification and
validation will be utilised to evaluate \sphinxstylestrong{Conformance} (i.e. adherence)
to the RA2 and RM requirements. Conformance scope includes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Test cases, with traceability to requirements, to validate that the
cloud infrastructure implementation meets the expected capabilities
specified in RA\sphinxhyphen{}2 and that the workloads consume compliant cloud
infrastructure resources

\item {} 
\sphinxAtStartPar
Verify, with requirements traceability, that the installation
cookbooks (manifests) of RI\sphinxhyphen{}2 are in conformance with the RA\sphinxhyphen{}2
specifications (for example, software versions, plugins, and
configurations)

\item {} 
\sphinxAtStartPar
Guidelines for processes, environments, and tools for enabling
conformance testing

\end{itemize}

\sphinxAtStartPar
In summary, \sphinxstylestrong{Conformance} testing will be performed as part of cloud
infrastructure and workload lifecycle testing which includes
\sphinxstylestrong{Verification} and \sphinxstylestrong{Validation}, defined further as:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Verification} is performed using reviews (e.g., manifests) to
ensure that the cloud infrastructure or workload is delivered as per
implementation specifications.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Validation} confirms the infrastructure or workload meet the
expected or desired behaviour by means of automated testing.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{All Terms utilized throughout this chapter are intended to align with
LFN Compliance and Verification Committee (CVC) definitions, and their
use through CVC documentation, guidelines, and standards.}


\subsection{Overview}
\label{\detokenize{chapters/chapter01:overview}}
\sphinxAtStartPar
\sphinxstylestrong{Document Purpose}

\sphinxAtStartPar
This chapter includes traceability from requirements to test cases and a
test case execution framework to ensure Kubernetes infrastructure meets
the design, feature, and capability expectations of the RM and RA2.
Ultimately, RC2 will reduce the amount of time and cost it takes each
operator to on\sphinxhyphen{}board and maintain cloud infrastructure and CNFs.


\subsection{RC2 End User Requirements}
\label{\detokenize{chapters/chapter01:rc2-end-user-requirements}}
\sphinxAtStartPar
Telecommunication service providers / operators are the primary intended
audience for RC2 results. Specifically those selecting infrastructure
and network function technologies to use in their network. The RC2
result artifact(s) should be clear and provide confidence to the service
provider that the test results meet the requirements \sphinxstyleemphasis{they have.}

\sphinxAtStartPar
Vendors/Developers are a secondary audience. They should be able to
clearly see the mapping from a specific test result (pass or fail) to
the requirement.


\subsubsection{Operator/SP Test Result Requirements}
\label{\detokenize{chapters/chapter01:operator-sp-test-result-requirements}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Clickable links from test cases to requirements

\item {} 
\sphinxAtStartPar
Pass, Fail, or Skipped for every test

\item {} 
\sphinxAtStartPar
Reason for failures

\item {} 
\sphinxAtStartPar
Reason for skipping

\item {} 
\sphinxAtStartPar
Format supporting clickable links (e.g., HTML)

\item {} 
\sphinxAtStartPar
Provide a stable set of point\sphinxhyphen{}in\sphinxhyphen{}time requirements and tests to
achieve conformance

\item {} 
\sphinxAtStartPar
Testing tools allow users to select between validation of mandatory
and optional requirements

\item {} 
\sphinxAtStartPar
Enable clear tracability with versioning to know which requirements
have and haven’t been covered and track changes over time

\item {} 
\sphinxAtStartPar
Tests must be available to run locally

\item {} 
\sphinxAtStartPar
Testing tools must produce machine\sphinxhyphen{}readable result formats

\end{itemize}


\subsubsection{Vendor Test Result Requirements}
\label{\detokenize{chapters/chapter01:vendor-test-result-requirements}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Clear mapping between requirements and test results

\item {} 
\sphinxAtStartPar
Enable clear tracability to know which requirements have and haven’t
been covered and track changes over time

\item {} 
\sphinxAtStartPar
Failures should provide additional content to inform the user where
or how the requirement was violated without having to read the test

\item {} 
\sphinxAtStartPar
Expected preconditions and environment requirements for any test
tooling must be defined

\end{itemize}


\subsection{Scope}
\label{\detokenize{chapters/chapter01:scope}}
\sphinxAtStartPar
This document covers aspects of conformance for both Kubernetes based
cloud infrastructure and workloads. The document will cover the
following topics:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Identify in detail the requirements of test\sphinxhyphen{}cases (mapped from RA2
and RM)

\item {} 
\sphinxAtStartPar
Test criteria that shows a certain capability or feature of the
system\sphinxhyphen{}under\sphinxhyphen{}test exists and behaves as expected

\item {} 
\sphinxAtStartPar
An E2E framework for conformance of Kubernetes infrastructures and
workloads, including specification for conformance test
infrastructure (lab environment and tools)

\item {} 
\sphinxAtStartPar
Analysis to identify where the gaps are in the industry for
implementing conformance test objectives (tooling, methods, process,
etc)

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Not in Scope}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Functional testing / validation of the application provided by the
workload is outside the scope of this work

\item {} 
\sphinxAtStartPar
Testing to confirm anything not in RM or RA2 requirements

\item {} 
\sphinxAtStartPar
VNFM/NFVO, like ONAP, is not used in the process flow for
infrastructure verifications or validations

\item {} 
\sphinxAtStartPar
Upgrades to workloads, and the respective processes of verifying
upgrade procedures and validating (testing) the success and
compatibility of upgrades is not in scope

\end{itemize}


\subsection{Guidelines}
\label{\detokenize{chapters/chapter01:guidelines}}
\sphinxAtStartPar
The objectives of the Reference Conformance for cloud infrastructure is
to verify implementations against the reference architecture which
satisfies infrastructure needs for workloads. The objectives of the
Reference Conformance for workloads is to verify workload
implementations consume resources and behave as expected against the
reference architecture.

\sphinxAtStartPar
These guidelines will drive RC2 deliverables:
\begin{itemize}
\item {} 
\sphinxAtStartPar
RC2 requirements are completely derived from RM and RA2 which specify
infrastructure capabilities including compute, memory, storage,
resource capabilities, performance optimization capabilities, and
monitoring capabilities.

\item {} 
\sphinxAtStartPar
Requirements in the RM and RAs that are performance related may not
have minimum performance criteria identified but where feasible will
have tests with metrics to show relevant capabilities are present and
working as expected.

\item {} 
\sphinxAtStartPar
Must/shall conformance criteria are testable as pass/fail and/or
reporting of quantitative test results. This will ensure
infrastructures and workloads meet minimum thresholds of functional
operability and/or performance behavior. This is the focus of RC2
since it is what will drive a commercially significant badging
program.

\item {} 
\sphinxAtStartPar
Should/may conformance criteria, which may or may not be testable,
provide recommendations or best\sphinxhyphen{}practices for functional operability
and/or performance behavior. These criteria and associated tests can
be very useful for developing, evaluating or deploying a cloud
infrastructure but are not critical to a commercially significant
badging program.

\end{itemize}


\subsection{Conformance Methodologies}
\label{\detokenize{chapters/chapter01:conformance-methodologies}}
\sphinxAtStartPar
The RC2 test suite will provide validation to ensure workloads can
interoperate with the RA2 conformant infrastructure. Upstream projects
will define features/capabilities, test scenarios, and test cases to be
executed. 3rd Party test platforms may also be leveraged if desired.

\sphinxAtStartPar
\sphinxstylestrong{Dependencies} infrastructure and workload validation will rely upon
test harnesses, test tools, and test suites provided by upstream
projects, including Anuket and CNF conformance. These upstream projects
will be reviewed semi\sphinxhyphen{}annually to verify they are still healthy and
active projects. Over time, the projects representing the conformance
process may change, but test parity is required if new test suites are
added in place of older, stale projects.


\subsection{Reading Guide and Usage}
\label{\detokenize{chapters/chapter01:reading-guide-and-usage}}
\sphinxAtStartPar
RC2 focuses on testing of Kubernetes based cloud infrastructure thus the
chapter structure is designed to facilitate this by matching test cases
to requirements and building test cookbooks. If you are looking for
requirements or the reasons behind them, please refer to the RA2.
Chapters 2 and 3 cover Kubernetes infrastructure conformance while 4 and
5 cover CNF conformance.

\sphinxAtStartPar
Chapter 2 takes the requirements from the RA2 and matches them to
upstream test cases. This will cover how specific test cases map to
requirements and the overall coverage of requirements with test cases.
Chapter 3 outlines how these test cases can be integrated together into
an automated toolchain to test conformance of the Kubernetes
infrastructure.

\sphinxAtStartPar
Similarly, Chapter 4 maps test cases map to requirements for CNFs and
Chapter 5 builds a testing cookbook. Chapter 6 encompasses any gaps in
the Reference Conformance 2.


\section{Kubernetes Test Cases and Requirements Traceability}
\label{\detokenize{chapters/chapter02:kubernetes-test-cases-and-requirements-traceability}}\label{\detokenize{chapters/chapter02::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter02:introduction}}
\sphinxAtStartPar
All of the requirements for RC2 have been defined in the Reference Model
(RM) and Reference Architecture (RA2). The scope of this chapter is to
identify and list down test cases based on these requirements. Users of
this chapter will be able to use it to determine which test cases they
must run in order to test compliance with the requirements. This will
enable traceability between the test cases and requirements. They should
be able to clearly see which requirements are covered by which tests and
the mapping from a specific test result (pass or fail) to a requirement.
Each requirement may have one or more test case associated with it.


\subsubsection{Goals}
\label{\detokenize{chapters/chapter02:goals}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Clear mapping between requirements and test cases

\item {} 
\sphinxAtStartPar
Provide a stable set of point\sphinxhyphen{}in\sphinxhyphen{}time requirements and tests to
achieve conformance

\item {} 
\sphinxAtStartPar
Enable clear traceability of the coverage of requirements across
consecutive releases of this document

\item {} 
\sphinxAtStartPar
Clickable links from test cases to requirements

\item {} 
\sphinxAtStartPar
One or more tests for every MUST requirement

\item {} 
\sphinxAtStartPar
A set of test cases to serve as a template for Anuket Assured

\end{itemize}


\subsubsection{Non\sphinxhyphen{}Goals}
\label{\detokenize{chapters/chapter02:non-goals}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Defining any requirements

\item {} 
\sphinxAtStartPar
Providing coverage for non\sphinxhyphen{}testable requirements

\end{itemize}


\subsubsection{Definitions}
\label{\detokenize{chapters/chapter02:definitions}}
\sphinxAtStartPar
\sphinxstylestrong{must}: Test Cases that are marked as must are considered mandatory and
must pass successfully

\sphinxAtStartPar
\sphinxstylestrong{should}: Test Cases that are marked as should are expected to be
fulfilled by the cloud infrastructure but it is up to each service
provider whether to accept a cloud infrastructure that is not fulfilling
any of these requirements. The same applies to should not.

\sphinxAtStartPar
\sphinxstylestrong{may}: Test cases that are marked as may are considered optional. The
same applies to may not.


\subsection{Traceability Matrix}
\label{\detokenize{chapters/chapter02:traceability-matrix}}

\subsubsection{Kubernetes API testing}
\label{\detokenize{chapters/chapter02:kubernetes-api-testing}}
\sphinxAtStartPar
The primary objectives of the \sphinxhref{https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md}{e2e
tests}
are to ensure a consistent and reliable behavior of the Kubernetes code
base, and to catch hard\sphinxhyphen{}to\sphinxhyphen{}test bugs before users do, when unit and
integration tests are insufficient. They are partially selected for the
\sphinxhref{https://github.com/cncf/k8s-conformance}{Software Conformance Certification
program} run by the
Kubernetes community (under the aegis of the CNCF).

\sphinxAtStartPar
Anuket shares the same goal to give end users the confidence that when
they use a certified product they can rely on a high level of common
functionality. Then Anuket RC2 starts with the test list defined by \sphinxhref{https://github.com/cncf/k8s-conformance}{K8s
Conformance} which is
expected to grow according to the ongoing requirement traceability.

\sphinxAtStartPar
\sphinxhref{https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md}{End\sphinxhyphen{}to\sphinxhyphen{}End
Testing}
basically asks for focus and skip regexes to select or to exclude
single tests:
\begin{itemize}
\item {} 
\sphinxAtStartPar
focus basically matches Conformance or \sphinxhref{https://github.com/kubernetes/community/blob/master/sig-testing/charter.md}{Testing Special Interest
Groups}
in sub\sphinxhyphen{}sections below

\item {} 
\sphinxAtStartPar
skip excludes the SIG labels listed as optional in
\sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements}.

\end{itemize}

\sphinxAtStartPar
The Reference Conformance suites must be stable and executed on real
deployments. Then all the following labels are defacto skipped in
\sphinxhref{https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md}{End\sphinxhyphen{}to\sphinxhyphen{}End
Testing}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
alpha

\item {} 
\sphinxAtStartPar
Disruptive

\item {} 
\sphinxAtStartPar
Flaky

\end{itemize}

\sphinxAtStartPar
It’s worth mentioning that no alpha or Flaky test can be included in
Conformance as per the rules.


\paragraph{Conformance}
\label{\detokenize{chapters/chapter02:conformance}}
\sphinxAtStartPar
It must be noted that the default \sphinxhref{https://github.com/cncf/k8s-conformance}{K8s
Conformance} testing is
disruptive thus Anuket RC2 rather picks
\sphinxhref{https://sonobuoy.io/docs/main/e2eplugin/}{non\sphinxhyphen{}disruptive\sphinxhyphen{}conformance}
testing as defined by \sphinxhref{https://sonobuoy.io/}{Sonobuoy}.

\sphinxAtStartPar
focus: {\hyperref[\detokenize{chapters/chapter02:conformance}]{\emph{Conformance}}}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
NoExecuteTaintManager

\end{itemize}


\paragraph{API Machinery Testing}
\label{\detokenize{chapters/chapter02:api-machinery-testing}}
\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}api\sphinxhyphen{}machinery{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Feature:CrossNamespacePodAffinity{]}

\item {} 
\sphinxAtStartPar
{[}Feature:CustomResourceValidationExpressions{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StorageVersionAPI{]}

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-api-machinery}{API Machinery Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements} for more details.


\paragraph{Apps Testing}
\label{\detokenize{chapters/chapter02:apps-testing}}
\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}apps{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Feature:DaemonSetUpdateSurge{]}

\item {} 
\sphinxAtStartPar
{[}Feature:IndexedJob{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StatefulSet{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StatefulSetAutoDeletePVC{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StatefulUpgrade{]}

\item {} 
\sphinxAtStartPar
{[}Feature:SuspendJob{]}

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-apps}{Apps Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements} for more details.


\paragraph{Auth Testing}
\label{\detokenize{chapters/chapter02:auth-testing}}
\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}auth{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Feature:BoundServiceAccountTokenVolume{]}

\item {} 
\sphinxAtStartPar
{[}Feature:PodSecurityPolicy{]}

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-auth}{Auth Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements} for more details.


\paragraph{Cluster Lifecycle Testing}
\label{\detokenize{chapters/chapter02:cluster-lifecycle-testing}}
\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}cluster\sphinxhyphen{}lifecycle{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle}{Cluster Lifecycle Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements} for more details.


\paragraph{Instrumentation Testing}
\label{\detokenize{chapters/chapter02:instrumentation-testing}}
\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}instrumentation{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Elasticsearch{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StackdriverAcceleratorMonitoring{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StackdriverCustomMetrics{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StackdriverExternalMetrics{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StackdriverMetadataAgent{]}

\item {} 
\sphinxAtStartPar
{[}Feature:StackdriverMonitoring{]}

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-instrumentation}{Instrumentation Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements} for more details.


\paragraph{Network Testing}
\label{\detokenize{chapters/chapter02:network-testing}}
\sphinxAtStartPar
The regexes load.balancer, LoadBalancer and
Network.should.set.TCP.CLOSE\_WAIT.timeout are currently skipped because
they haven’t been covered successfully neither by
\sphinxhref{https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-release/release-branch-jobs/1.23.yaml}{sig\sphinxhyphen{}release\sphinxhyphen{}1.23\sphinxhyphen{}blocking}
nor by \sphinxhref{https://build.opnfv.org/ci/view/functest-kubernetes/job/functest-kubernetes-v1.23-daily/7/}{Anuket RC2
verification}

\sphinxAtStartPar
Please note that a couple of tests must be skipped by name below as they
are no appropriate labels.

\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}network{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Example{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Ingress{]}

\item {} 
\sphinxAtStartPar
{[}Feature:IPv6DualStack{]}

\item {} 
\sphinxAtStartPar
{[}Feature:kubemci{]}

\item {} 
\sphinxAtStartPar
{[}Feature:KubeProxyDaemonSetMigration{]}

\item {} 
\sphinxAtStartPar
{[}Feature:KubeProxyDaemonSetUpgrade{]}

\item {} 
\sphinxAtStartPar
{[}Feature:NEG{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Networking\sphinxhyphen{}IPv6{]}

\item {} 
\sphinxAtStartPar
{[}Feature:NetworkPolicy{]}

\item {} 
\sphinxAtStartPar
{[}Feature:PerformanceDNS{]}

\item {} 
\sphinxAtStartPar
{[}Feature:SCTP{]}

\item {} 
\sphinxAtStartPar
{[}Feature:SCTPConnectivity{]}

\item {} 
\sphinxAtStartPar
DNS configMap nameserver

\item {} 
\sphinxAtStartPar
load.balancer

\item {} 
\sphinxAtStartPar
LoadBalancer

\item {} 
\sphinxAtStartPar
Network.should.set.TCP.CLOSE\_WAIT.timeout

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-network}{Network Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements}.


\paragraph{Node Testing}
\label{\detokenize{chapters/chapter02:node-testing}}
\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}node{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Feature:ExperimentalResourceUsageTracking{]}

\item {} 
\sphinxAtStartPar
{[}Feature:GRPCContainerProbe{]}

\item {} 
\sphinxAtStartPar
{[}Feature:GPUUpgrade{]}

\item {} 
\sphinxAtStartPar
{[}Feature:PodGarbageCollector{]}

\item {} 
\sphinxAtStartPar
{[}Feature:RegularResourceUsageTracking{]}

\item {} 
\sphinxAtStartPar
{[}NodeFeature:DownwardAPIHugePages{]}

\item {} 
\sphinxAtStartPar
{[}NodeFeature:RuntimeHandler{]}

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-node}{Node Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements}.


\paragraph{Scheduling Testing}
\label{\detokenize{chapters/chapter02:scheduling-testing}}
\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}scheduling{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Feature:GPUDevicePlugin{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Recreate{]}

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-scheduling}{Scheduling Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements}.


\paragraph{Storage Testing}
\label{\detokenize{chapters/chapter02:storage-testing}}
\sphinxAtStartPar
It should be noted that all in\sphinxhyphen{}tree driver testing, {[}Driver:+{]}, is
skipped. Conforming to \sphinxhref{https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-release/release-branch-jobs/1.22.yaml}{the upstream
gate},
all PersistentVolumes NFS testing is also skipped. The following
exclusions are about \sphinxhref{https://github.com/kubernetes-sigs/kind/issues/2356}{the deprecated in\sphinxhyphen{}tree GitRepo volume
type}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
should provision storage with different parameters

\item {} 
\sphinxAtStartPar
should not cause race condition when used for git\_repo

\end{itemize}

\sphinxAtStartPar
Please note that a couple of tests must be skipped by name below as they
are no appropriate labels.

\sphinxAtStartPar
focus: {[}sig\sphinxhyphen{}storage{]}

\sphinxAtStartPar
skip:
\begin{itemize}
\item {} 
\sphinxAtStartPar
{[}alpha{]}

\item {} 
\sphinxAtStartPar
{[}Disruptive{]}

\item {} 
\sphinxAtStartPar
{[}Flaky{]}

\item {} 
\sphinxAtStartPar
{[}Driver:+{]}

\item {} 
\sphinxAtStartPar
{[}Feature:ExpandInUsePersistentVolumes{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Flexvolumes{]}

\item {} 
\sphinxAtStartPar
{[}Feature:GKELocalSSD{]}

\item {} 
\sphinxAtStartPar
{[}Feature:VolumeSnapshotDataSource{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Flexvolumes{]}

\item {} 
\sphinxAtStartPar
{[}Feature:vsphere{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Volumes{]}

\item {} 
\sphinxAtStartPar
{[}Feature:Windows{]}

\item {} 
\sphinxAtStartPar
{[}NodeFeature:EphemeralStorage{]}

\item {} 
\sphinxAtStartPar
PersistentVolumes.NFS

\item {} 
\sphinxAtStartPar
should provision storage with different parameters

\item {} 
\sphinxAtStartPar
should not cause race condition when used for git\_repo

\end{itemize}

\sphinxAtStartPar
See \sphinxhref{https://github.com/kubernetes/community/tree/master/sig-storage}{Storage Special Interest
Group}
and \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter06.html}{API and Feature Testing requirements}.


\subsubsection{Kubernetes API benchmarking}
\label{\detokenize{chapters/chapter02:kubernetes-api-benchmarking}}
\sphinxAtStartPar
\sphinxhref{https://github.com/openstack/rally}{Rally} is a tool and framework
that performs Kubernetes API benchmarking.

\sphinxAtStartPar
\sphinxhref{https://git.opnfv.org/functest-kubernetes/tree/docker/benchmarking/testcases.yaml?h=stable\%2Fv1.23}{Functest Kubernetes
Benchmarking}
proposed a Rally\sphinxhyphen{}based test case,
\sphinxhref{https://artifacts.opnfv.org/functest-kubernetes/GFAB1XPJBRQT/functest-kubernetes-opnfv-functest-kubernetes-benchmarking-v1.23-xrally\_kubernetes\_full-run-2/xrally\_kubernetes\_full/xrally\_kubernetes\_full.html}{xrally\_kubernetes\_full},
which iterates 10 times the mainline
\sphinxhref{https://github.com/xrally/xrally-kubernetes}{xrally\sphinxhyphen{}kubernetes}
scenarios.

\sphinxAtStartPar
At the time of writing, no KPI is defined in \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/index.html}{Kubernetes based Reference Architecture}
which would have asked for an update of the default SLA (maximum failure
rate of 0\%) proposed in \sphinxhref{https://git.opnfv.org/functest-kubernetes/tree/docker/benchmarking/testcases.yaml?h=stable\%2Fv1.23}{Functest Kubernetes
Benchmarking}

\sphinxAtStartPar
\sphinxhref{https://artifacts.opnfv.org/functest-kubernetes/GFAB1XPJBRQT/functest-kubernetes-opnfv-functest-kubernetes-benchmarking-v1.23-xrally\_kubernetes\_full-run-2/xrally\_kubernetes\_full/xrally\_kubernetes\_full.html}{Functest
xrally\_kubernetes\_full}:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Kubernetes API benchmarking}\label{\detokenize{chapters/chapter02:id1}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{80}{100}|\X{20}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Scenarios
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Iterations
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_deployment
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_job
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_namespace
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod\_with\_configmap\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod\_with\_configmap\_volume {[}2{]}
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod\_with\_emptydir\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod\_with\_emptydir\_volume {[}2{]}
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod\_with\_hostpath\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod\_with\_secret\_volume
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_pod\_with\_secret\_volume {[}2{]}
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_replicaset
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_replication\_controller
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_and\_delete\_statefulset
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_check\_and\_delete\_pod\_with\_cluster\_ip\_service
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_check\_and\_delete\_pod\_with\_cluster\_ip\_service {[}2{]}
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_check\_and\_delete\_pod\_with\_node\_port\_service
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_rollout\_and\_delete\_deployment
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_scale\_and\_delete\_replicaset
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_scale\_and\_delete\_replication\_controller
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.create\_scale\_and\_delete\_statefulset
&
\sphinxAtStartPar
10
\\
\hline
\sphinxAtStartPar
Kubernetes.list\_namespaces
&
\sphinxAtStartPar
10
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
The following software versions are considered to benchmark Kubernetes
v1.23 (latest stable release) selected by Anuket:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Software versions}\label{\detokenize{chapters/chapter02:id2}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{50}{100}|\X{50}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Software
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Version
\\
\hline
\sphinxAtStartPar
Functest
&
\sphinxAtStartPar
v1.23
\\
\hline
\sphinxAtStartPar
xrally\sphinxhyphen{}kubernetes
&
\sphinxAtStartPar
1.1.1.dev12
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Dataplane benchmarking}
\label{\detokenize{chapters/chapter02:dataplane-benchmarking}}
\sphinxAtStartPar
\sphinxhref{https://github.com/kubernetes/perf-tests}{Kubernetes perf\sphinxhyphen{}tests
repository} hosts various
Kubernetes\sphinxhyphen{}related performance test related tools especially
\sphinxhref{https://github.com/kubernetes/perf-tests/tree/master/network/benchmarks/netperf}{netperf}
which benchmarks Kubernetes networking performance.

\sphinxAtStartPar
As listed in \sphinxhref{https://github.com/kubernetes/perf-tests/tree/master/network/benchmarks/netperf\#readme}{netperf’s
README},
the 5 major network traffic paths are combination of pod IP vs virtual
IP and whether the pods are co\sphinxhyphen{}located on the same node versus a
remotely located pod:
\begin{itemize}
\item {} 
\sphinxAtStartPar
same node using pod IP

\item {} 
\sphinxAtStartPar
same node using cluster/virtual IP

\item {} 
\sphinxAtStartPar
remote node using pod IP

\item {} 
\sphinxAtStartPar
remote node using cluster/virtual IP

\item {} 
\sphinxAtStartPar
same node pod hairpin to itself using cluster/virtual IP

\end{itemize}

\sphinxAtStartPar
It should be noted that
\sphinxhref{https://github.com/kubernetes/perf-tests/tree/master/network/benchmarks/netperf}{netperf}
leverages \sphinxhref{https://github.com/esnet/iperf}{iperf} (both TCP and UDP
modes) and \sphinxhref{https://github.com/HewlettPackard/netperf/}{Netperf}.

\sphinxAtStartPar
At the time of writing, no KPI is defined in Anuket chapters which would
have asked for an update of the default SLA proposed in \sphinxhref{https://git.opnfv.org/functest-kubernetes/tree/docker/benchmarking?h=stable/v1.23}{Functest
Kubernetes
Benchmarking}.


\subsubsection{Security testing}
\label{\detokenize{chapters/chapter02:security-testing}}
\sphinxAtStartPar
There are a couple of opensource tools that help securing the Kubernetes
stack. Amongst them, \sphinxhref{https://git.opnfv.org/functest-kubernetes/tree/docker/security/testcases.yaml?h=stable\%2Fv1.23}{Functest Kubernetes
Security}
offers two test cases based on
\sphinxhref{https://github.com/aquasecurity/kube-hunter}{kube\sphinxhyphen{}hunter} and
\sphinxhref{https://github.com/aquasecurity/kube-bench}{kube\sphinxhyphen{}bench}.

\sphinxAtStartPar
\sphinxhref{https://github.com/aquasecurity/kube-hunter}{kube\sphinxhyphen{}hunter} hunts for
security weaknesses in Kubernetes clusters and
\sphinxhref{https://github.com/aquasecurity/kube-bench}{kube\sphinxhyphen{}bench} checks
whether Kubernetes is deployed securely by running the checks documented
in the \sphinxhref{https://www.cisecurity.org/benchmark/kubernetes/}{CIS Kubernetes
Benchmark}.

\sphinxAtStartPar
\sphinxhref{https://github.com/aquasecurity/kube-hunter}{kube\sphinxhyphen{}hunter} classifies
all vulnerabilities as low, medium, and high. In context of this
conformance suite, only the high vulnerabilities lead to a test case
failure. Then all low and medium vulnerabilities are only printed for
information.

\sphinxAtStartPar
Here are the \sphinxhref{https://github.com/aquasecurity/kube-hunter/blob/v0.3.1/kube\_hunter/core/events/types.py}{vulnerability
categories}
tagged as high by
\sphinxhref{https://github.com/aquasecurity/kube-hunter}{kube\sphinxhyphen{}hunter}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
RemoteCodeExec

\item {} 
\sphinxAtStartPar
IdentityTheft

\item {} 
\sphinxAtStartPar
PrivilegeEscalation

\end{itemize}

\sphinxAtStartPar
At the time of writing, none of the Center for Internet Security (CIS)
rules are defined as mandatory (e.g. sec.std.001: The Cloud Operator
\sphinxstylestrong{should} comply with Center for Internet Security CIS Controls) else
it would have required an update of the default kube\sphinxhyphen{}bench behavior (all
failures and warnings are only printed) as integrated in \sphinxhref{https://git.opnfv.org/functest-kubernetes/tree/docker/security/testcases.yaml?h=stable\%2Fv1.23}{Functest
Kubernetes
Security}.

\sphinxAtStartPar
The following software versions are considered to verify Kubernetes
v1.23 (latest stable release) selected by Anuket:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Software versions}\label{\detokenize{chapters/chapter02:id3}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{50}{100}|\X{50}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Software
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Version
\\
\hline
\sphinxAtStartPar
Functest
&
\sphinxAtStartPar
v1.23
\\
\hline
\sphinxAtStartPar
kube\sphinxhyphen{}hunter
&
\sphinxAtStartPar
0.3.1
\\
\hline
\sphinxAtStartPar
kube\sphinxhyphen{}bench
&
\sphinxAtStartPar
0.3.1
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Opensource CNF onboarding and testing}
\label{\detokenize{chapters/chapter02:opensource-cnf-onboarding-and-testing}}
\sphinxAtStartPar
Running opensource containerized network functions (CNF) is a key
technical solution to ensure that the platforms meet Network Functions
Virtualization requirements.

\sphinxAtStartPar
Functest CNF offers 2 test cases which automatically onboard and test
\sphinxhref{https://github.com/Metaswitch/clearwater-docker}{Clearwater IMS} via
kubectl and Helm. It’s worth mentioning that this CNF is covered by the
upstream tests (see
\sphinxhref{https://github.com/Metaswitch/clearwater-live-test}{clearwater\sphinxhyphen{}live\sphinxhyphen{}test}).

\sphinxAtStartPar
The following software versions are considered to verify Kubernetes
v1.23 (latest stable release) selected by Anuket:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Software versions}\label{\detokenize{chapters/chapter02:id4}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{50}{100}|\X{50}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Software
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Version
\\
\hline
\sphinxAtStartPar
Functest
&
\sphinxAtStartPar
v1.23
\\
\hline
\sphinxAtStartPar
clearwater
&
\sphinxAtStartPar
release\sphinxhyphen{}130
\\
\hline
\sphinxAtStartPar
Helm
&
\sphinxAtStartPar
v3.3.1
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsection{Test Cases Traceability to Requirements}
\label{\detokenize{chapters/chapter02:test-cases-traceability-to-requirements}}
\sphinxAtStartPar
The following test case must pass as they are for Reference Conformance:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Mandory test cases}\label{\detokenize{chapters/chapter02:id5}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{40}{100}|\X{25}{100}|\X{10}{100}|\X{25}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Container
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Test suite
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Criteria
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirements
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
xrally\_kubernetes
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
k8s\_conformance
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
k8s\_conformance\_serial
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_api\_machinery
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_api\_machinery\_serial
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_apps
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_apps\_serial
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_auth
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_cluster\_lifecycle
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_instrumentation
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_network
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_node
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_scheduling\_serial
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_storage
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}smoke:v1.23
&
\sphinxAtStartPar
sig\_storage\_serial
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}security:v1.23
&
\sphinxAtStartPar
kube\_hunter
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Security testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}security:v1.23
&
\sphinxAtStartPar
kube\_bench\_master
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Security testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}security:v1.23
&
\sphinxAtStartPar
kube\_bench\_node
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Security testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}benchmarking:v1.23
&
\sphinxAtStartPar
xrally\_kubernetes\_full
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Kubernetes API benchmarking
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}benchmarking:v1.23
&
\sphinxAtStartPar
netperf
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Dataplane benchmarking
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}cnf:v1.23
&
\sphinxAtStartPar
k8s\_vims
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Opensource CNF onboarding and testing
\\
\hline
\sphinxAtStartPar
opnfv/functest\sphinxhyphen{}kubernetes\sphinxhyphen{}cnf:v1.23
&
\sphinxAtStartPar
helm\_vims
&
\sphinxAtStartPar
PASS
&
\sphinxAtStartPar
Opensource CNF onboarding and testing
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\section{Kubernetes Testing Cookbook}
\label{\detokenize{chapters/chapter03:kubernetes-testing-cookbook}}\label{\detokenize{chapters/chapter03::doc}}

\subsection{Deploy your own conformance toolchain}
\label{\detokenize{chapters/chapter03:deploy-your-own-conformance-toolchain}}
\sphinxAtStartPar
At the time of writing, the CI description file is hosted in Functest
and only runs the containers selected by Anuket RC2. It will be
completed by the next Anuket mandatory test cases and then a new CI
description file will be proposed in a shared tree.

\sphinxAtStartPar
\sphinxhref{https://galaxy.ansible.com/collivier/xtesting}{Xtesting CI} only
requires internet access, GNU/Linux as Operating System and asks for a
few dependencies as described in \sphinxhref{https://github.com/collivier/ansible-role-xtesting\#readme}{Deploy your own Xtesting CI/CD
toolchains}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
python\sphinxhyphen{}virtualenv

\item {} 
\sphinxAtStartPar
git

\end{itemize}

\sphinxAtStartPar
Please note the next two points depending on the GNU/Linux distributions
and the network settings:
\begin{itemize}
\item {} 
\sphinxAtStartPar
SELinux: you may have to add \textendash{}system\sphinxhyphen{}site\sphinxhyphen{}packages when creating the
virtualenv (“Aborting, target uses selinux but python bindings
(libselinux\sphinxhyphen{}python) aren’t installed!”)

\item {} 
\sphinxAtStartPar
Proxy: you may set your proxy in env for Ansible and in systemd for
Docker \sphinxurl{https://docs.docker.com/config/daemon/systemd/\#httphttps-proxy}

\end{itemize}

\sphinxAtStartPar
To deploy your own CI toolchain running Anuket Compliance:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
virtualenv functest\PYGZhy{}kubernetes \PYGZhy{}\PYGZhy{}system\PYGZhy{}site\PYGZhy{}packages
. functest\PYGZhy{}kubernetes/bin/activate
pip install ansible
ansible\PYGZhy{}galaxy install collivier.xtesting
ansible\PYGZhy{}galaxy collection install ansible.posix community.general community.grafana kubernetes.core community.docker community.postgresql
git clone https://gerrit.opnfv.org/gerrit/functest\PYGZhy{}kubernetes functest\PYGZhy{}kubernetes\PYGZhy{}src
\PYG{o}{(}\PYG{n+nb}{cd} functest\PYGZhy{}kubernetes\PYGZhy{}src \PYG{o}{\PYGZam{}\PYGZam{}} git checkout \PYGZhy{}b stable/v1.23 origin/stable/v1.23\PYG{o}{)}
ansible\PYGZhy{}playbook functest\PYGZhy{}kubernetes\PYGZhy{}src/ansible/site.cntt.yml
\end{sphinxVerbatim}


\subsubsection{Configure Kubernetes API testing}
\label{\detokenize{chapters/chapter03:configure-kubernetes-api-testing}}
\sphinxAtStartPar
Place the kubeconfig configuration file corresponding to the Kubernetes
cluster under test in the following location on the machine running the
cookbook:

\sphinxAtStartPar
\sphinxcode{\sphinxupquote{/home/opnfv/functest\sphinxhyphen{}kubernetes/config}}


\subsubsection{Run Kubernetes conformance suite}
\label{\detokenize{chapters/chapter03:run-kubernetes-conformance-suite}}
\sphinxAtStartPar
Open \sphinxurl{http://127.0.0.1:8080/job/functest-kubernetes-v1.23-daily/} in a web
browser, login as admin/admin and click on “Build with Parameters” (keep
the default values).

\sphinxAtStartPar
If the System under test (SUT) is Anuket compliant, a link to the full
archive containing all test results and artifacts will be printed in
functest\sphinxhyphen{}kubernetes\sphinxhyphen{}v1.23\sphinxhyphen{}zip’s console. Be free to download it and then
to send it to any reviewer committee.

\sphinxAtStartPar
To clean your working dir:

\begin{sphinxVerbatim}[commandchars=\\\{\}]
deactivate
rm \PYGZhy{}rf functest\PYGZhy{}kubernetes\PYGZhy{}src functest\PYGZhy{}kubernetes
\end{sphinxVerbatim}


\section{CNF Test Cases and Requirements Traceability}
\label{\detokenize{chapters/chapter04:cnf-test-cases-and-requirements-traceability}}\label{\detokenize{chapters/chapter04::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter04:introduction}}
\sphinxAtStartPar
The scope of this chapter is to identify and list test cases based on
requirements defined in \sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/index.html}{Kubernetes based Reference Architecture}.
This will serve as traceability between test cases and requirements for
Kubernetes platform interoperability.

\sphinxAtStartPar
Note that each requirement may have one or more test cases associated
with it.


\subsection{Selection Criteria}
\label{\detokenize{chapters/chapter04:selection-criteria}}
\sphinxAtStartPar
Test cases, tools and their dependencies must be open source. The test
cases (or test suite with the test case) as well as the environment
needed to run the test should be reproducible by any party following
publicly available documentation.

\sphinxAtStartPar
Examples of initiatives (having testing tools, test suites, etc) with
test cases which could be used include K8s Conformance, K8s e2e,
Sonobuoy, Anuket Functest, CNF Conformance.


\subsection{Traceability Matrix}
\label{\detokenize{chapters/chapter04:traceability-matrix}}
\sphinxAtStartPar
The following is a Requirements Traceability Matrix (RTM) mapping Test
Case, and/or Test Case Coverage, to RM and RA requirements \textendash{}
configuration, deployment, runtime.


\subsubsection{Test Case Traceability to RA2 Requirements}
\label{\detokenize{chapters/chapter04:test-case-traceability-to-ra2-requirements}}
\sphinxAtStartPar
This section focuses on the test cases covering the requirements in
\sphinxhref{https://cntt.readthedocs.io/projects/ra2/en/latest/chapters/chapter04.html\#kubernetes-workloads}{Kubernetes workloads}
for Kubernetes workloads.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Traceability to RA2 Requirements}\label{\detokenize{chapters/chapter04:id1}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{35}{130}|\X{35}{130}|\X{30}{130}|\X{30}{130}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
RM/RA Ref
&\sphinxstyletheadfamily 
\sphinxAtStartPar
High\sphinxhyphen{}level test definition
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Test name and project
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Priority
\\
\hline
\sphinxAtStartPar
ra2.app.001
&&&
\sphinxAtStartPar
Must
\\
\hline
\sphinxAtStartPar
ra2.app.002
&&&
\sphinxAtStartPar
Must
\\
\hline
\sphinxAtStartPar
ra2.app.003
&&&
\sphinxAtStartPar
Must
\\
\hline
\sphinxAtStartPar
ra2.app.004
&&&
\sphinxAtStartPar
Must
\\
\hline
\sphinxAtStartPar
ra2.app.005
&&&
\sphinxAtStartPar
Must
\\
\hline
\sphinxAtStartPar
ra2.app.006
&&&
\sphinxAtStartPar
Must
\\
\hline
\sphinxAtStartPar
ra2.app.007
&&&
\sphinxAtStartPar
Must
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\section{CNF Testing Cookbook}
\label{\detokenize{chapters/chapter05:cnf-testing-cookbook}}\label{\detokenize{chapters/chapter05::doc}}

\section{Gap Analysis and Development}
\label{\detokenize{chapters/chapter06:gap-analysis-and-development}}\label{\detokenize{chapters/chapter06::doc}}


\renewcommand{\indexname}{Index}
\printindex
\end{document}