%% Generated by Sphinx.
\def\sphinxdocclass{article}
\documentclass[letterpaper,10pt,english]{sphinxhowto}
\ifdefined\pdfpxdimen
   \let\sphinxpxdimen\pdfpxdimen\else\newdimen\sphinxpxdimen
\fi \sphinxpxdimen=.75bp\relax
\ifdefined\pdfimageresolution
    \pdfimageresolution= \numexpr \dimexpr1in\relax/\sphinxpxdimen\relax
\fi
%% let collapsable pdf bookmarks panel have high depth per default
\PassOptionsToPackage{bookmarksdepth=5}{hyperref}

\PassOptionsToPackage{warn}{textcomp}
\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
% support both utf8 and utf8x syntaxes
  \ifdefined\DeclareUnicodeCharacterAsOptional
    \def\sphinxDUC#1{\DeclareUnicodeCharacter{"#1}}
  \else
    \let\sphinxDUC\DeclareUnicodeCharacter
  \fi
  \sphinxDUC{00A0}{\nobreakspace}
  \sphinxDUC{2500}{\sphinxunichar{2500}}
  \sphinxDUC{2502}{\sphinxunichar{2502}}
  \sphinxDUC{2514}{\sphinxunichar{2514}}
  \sphinxDUC{251C}{\sphinxunichar{251C}}
  \sphinxDUC{2572}{\textbackslash}
\fi
\usepackage{cmap}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{babel}



\usepackage{tgtermes}
\usepackage{tgheros}
\renewcommand{\ttdefault}{txtt}



\usepackage[Bjarne]{fncychap}
\usepackage[,numfigreset=2,mathnumfig]{sphinx}

\fvset{fontsize=auto}
\usepackage{geometry}


% Include hyperref last.
\usepackage{hyperref}
% Fix anchor placement for figures with captions.
\usepackage{hypcap}% it must be loaded after hyperref.
% Set up styles of URL: it should be placed after hyperref.
\urlstyle{same}


\usepackage{sphinxmessages}
\setcounter{tocdepth}{1}



\title{Anuket OpenStack based Reference Architecture}
\date{Jun 06, 2022}
\release{}
\author{Anuket}
\newcommand{\sphinxlogo}{\vbox{}}
\renewcommand{\releasename}{}
\makeindex
\begin{document}

\pagestyle{empty}
\sphinxmaketitle
\pagestyle{plain}
\sphinxtableofcontents
\pagestyle{normal}
\phantomsection\label{\detokenize{index::doc}}



\section{Introduction}
\label{\detokenize{chapters/chapter01:introduction}}\label{\detokenize{chapters/chapter01::doc}}

\subsection{Overview}
\label{\detokenize{chapters/chapter01:overview}}
\sphinxAtStartPar
This Reference Architecture is focussed on OpenStack as the Virtualised
Infrastructure Manager (VIM) chosen based on the criteria laid out in
the \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter01.html}{Introduction}.
\sphinxhref{https://docs.openstack.org}{OpenStack} has the advantage of being a
mature and widely accepted open\sphinxhyphen{}source technology; a strong ecosystem of
vendors that support it, the OpenInfra Foundation for managing the
community, and, most importantly, it is widely deployed by the global
operator community for both internal infrastructure and external facing
products and services. This means that the operators have existing staff
with the right skill sets to support a Cloud Infrastructure
(\sphinxhref{https://www.etsi.org/deliver/etsi\_gs/NFV-INF/001\_099/001/01.01.01\_60/gs\_NFV-INF001v010101p.pdf}{NFVI})
deployment into development, test and production. Another reason to
choose OpenStack is that it has a large active community of vendors and
operators, which means that any code or component changes needed to
support the Common Telco Cloud Infrastructure requirements can be
managed through the existing project communities’ processes to add and
validate the required features through well\sphinxhyphen{}established mechanisms.


\subsubsection{Vision}
\label{\detokenize{chapters/chapter01:vision}}
\sphinxAtStartPar
The OpenStack\sphinxhyphen{}based Anuket Reference Architecture will host NFV
workloads, primarily VNFs, of interest to the Anuket community. The
Reference Architecture document can be used by operators to deploy
Anuket conformant infrastructure; hereafter, “conformant” denotes that
the resource can satisfy tests conducted to verify conformance with this
reference architecture.


\subsection{Use Cases}
\label{\detokenize{chapters/chapter01:use-cases}}
\sphinxAtStartPar
Several NFV use cases are documented in OpenStack. For more examples and
details refer to the OpenStack
\sphinxhref{https://docs.openstack.org/arch-design/use-cases.html}{docs}.
Examples include:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Overlay networks}: The overlay functionality design includes
OpenStack Networking in \sphinxhref{https://www.openvswitch.org}{Open
vSwitch} GRE tunnel mode. In this
case, the layer\sphinxhyphen{}3 external routers pair with VRRP, and switches pair
with an implementation of MLAG to ensure that you do not lose
connectivity with the upstream routing infrastructure.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Performance tuning}: Network level tuning for this workload is
minimal. Quality of Service (QoS) applies to these workloads for a
middle ground Class Selector depending on existing policies. It is
higher than a best effort queue but lower than an Expedited
Forwarding or Assured Forwarding queue. Since this type of
application generates larger packets with longer\sphinxhyphen{}lived connections,
you can optimize bandwidth utilization for long duration TCP. Normal
bandwidth planning applies here with regards to benchmarking a
session’s usage multiplied by the expected number of concurrent
sessions with overhead.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Network functions}: Network functions is a broad category but
encompasses workloads that support the exchange of information (data,
voice, multi\sphinxhyphen{}media) over a system’s network. Some of these workloads
tend to consist of a large number of small\sphinxhyphen{}sized packets that are
short lived, such as DNS queries or SNMP traps. These messages need
to arrive quickly and, thus, do not handle packet loss. Network
function workloads have requirements that may affect configurations
including at the hypervisor level. For an application that generates
10 TCP sessions per user with an average bandwidth of 512 kilobytes
per second per flow and expected user count of ten thousand (10,000)
concurrent users, the expected bandwidth plan is approximately 4.88
gigabits per second. The supporting network for this type of
configuration needs to have a low latency and evenly distributed load
across the topology. These types of workload benefit from having
services local to the consumers of the service. Thus, use a
multi\sphinxhyphen{}site approach, as well as, deploying many copies of the
application to handle load as close as possible to consumers. Since
these applications function independently, they do not warrant
running overlays to interconnect tenant networks. Overlays also have
the drawback of performing poorly with rapid flow setup and may incur
too much overhead with large quantities of small packets and
therefore we do not recommend them. QoS is desirable for some
workloads to ensure delivery. DNS has a major impact on the load
times of other services and needs to be reliable and provide rapid
responses. Configure rules in upstream devices to apply a
higher\sphinxhyphen{}Class Selector to DNS to ensure faster delivery or a better
spot in queuing algorithms.

\end{itemize}


\subsection{Anuket OpenStack Reference Release}
\label{\detokenize{chapters/chapter01:anuket-openstack-reference-release}}
\sphinxAtStartPar
This Reference Architecture document conforms to the \sphinxhref{https://docs.openstack.org/wallaby/projects.html}{OpenStack
Wallaby} release.
While many features and capabilities are conformant with many OpenStack
releases, this document will refer to features, capabilities and APIs
that are part of the OpenStack Wallaby release. For ease, this Anuket
Reference Architecture document version can be referred to as “RA\sphinxhyphen{}1 OSTK
Wallaby.”


\subsection{Principles}
\label{\detokenize{chapters/chapter01:principles}}
\sphinxAtStartPar
OpenStack Reference Architecture must obey to the following set of
principles:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://cntt.readthedocs.io/en/latest/common/chapter00.html\#anuket-general-principles}{Anuket General Principles}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://cntt.readthedocs.io/en/latest/common/chapter00.html\#architectural-principles}{Architectural Principles}

\end{itemize}


\subsubsection{OpenStack specific principles}
\label{\detokenize{chapters/chapter01:openstack-specific-principles}}
\sphinxAtStartPar
OpenStack considers the following Four Opens essential for success:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Open Source

\item {} 
\sphinxAtStartPar
Open Design

\item {} 
\sphinxAtStartPar
Open Development

\item {} 
\sphinxAtStartPar
Open Community

\end{itemize}

\sphinxAtStartPar
This OpenStack Reference Architecture is organised around the three
major Cloud Infrastructure resource types as core services of compute,
storage and networking, and a set of shared services of identity
management, image management, graphical user interface, orchestration
engine, etc.


\subsection{Document Organisation}
\label{\detokenize{chapters/chapter01:document-organisation}}
\sphinxAtStartPar
Chapter 2 defines the Reference Architecture requirements and, when
appropriate, provides references to where these requirements are
addressed in this document. The intent of this document is to address
all of the mandatory (“must”) requirements and the most useful of the
other optional (“should”) requirements. Chapter 3 and 4 cover the Cloud
Infrastructure resources and the core OpenStack services, while the APIs
are covered in Chapter 5. Chapter 6 covers the implementation and
enforcement of security capabilities and controls. Life Cycle Management
of the Cloud Infrastructure and VIM are covered in Chapter 7 with stress
on Logging, Monitoring and Analytics (LMA), configuration management and
some other operational items. Please note that Chapter 7 is not a
replacement for the implementation, configuration and operational
documentation that accompanies the different OpenStack distributions.
Chapter 8 identifies certain Gaps that currently exist and plans on how
to address them. For example, Service Function Chaining support needs to
be addressed to realise the full potential and value of SDN and NFV.


\subsection{Terminology}
\label{\detokenize{chapters/chapter01:terminology}}
\sphinxAtStartPar
General terminology definitions can be found in
\sphinxhref{https://cntt.readthedocs.io/en/latest/common/glossary.html}{Glossary} and specific terms relating
to this reference architecture are to be found in \sphinxtitleref{OpenStack Related
Terminology :ref:\textasciigrave{}common/glossary:openstack related terminology}.


\section{Architecture Requirements}
\label{\detokenize{chapters/chapter02:architecture-requirements}}\label{\detokenize{chapters/chapter02::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter02:introduction}}
\sphinxAtStartPar
\sphinxstylestrong{must}: Requirements that are marked as \sphinxstyleemphasis{must} are considered
mandatory and must exist in the reference architecture and reflected in
any implementation targeting this reference architecture. The same
applies to \sphinxstyleemphasis{must not}.

\sphinxAtStartPar
\sphinxstylestrong{should}: Requirements that are marked as \sphinxstyleemphasis{should} are expected to be
fulfilled by the reference architecture but it is up to each service
provider to accept an implementation targeting this reference
architecture that is not reflecting on any of those requirements. The
same applies to \sphinxstyleemphasis{should not}. \textgreater{} RFC2119

\sphinxAtStartPar
\sphinxstylestrong{may}: Requirements that are marked as \sphinxstyleemphasis{may} are considered optional.
The same applies to \sphinxstyleemphasis{may not}.

\sphinxAtStartPar
This chapter includes both “Requirements” that must be satisifed in an
RA\sphinxhyphen{}1 conformant implementation and “Recommendations” that are optional
for implementation.


\subsection{Reference Model Requirements}
\label{\detokenize{chapters/chapter02:reference-model-requirements}}
\sphinxAtStartPar
The tables below contain the requirements from the Reference Model to
cover the Basic and High\sphinxhyphen{}Performance profiles.

\sphinxAtStartPar
To ensure alignment with the infrastructure profile catalogue, the
following requirements are referenced through:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Those relating to Cloud Infrastructure Software Profiles

\item {} 
\sphinxAtStartPar
Those relating to Cloud Infrastructure Hardware Profiles

\item {} 
\sphinxAtStartPar
Those relating to Cloud Infrastructure Management

\item {} 
\sphinxAtStartPar
Those relating to Cloud Infrastructure Security

\end{itemize}


\subsubsection{Cloud Infrastructure Software Profile Requirements for Compute}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-software-profile-requirements-for-compute}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-software-profiles-features-and-requirements}{Cloud Infrastructure Software Profiles features and requirements})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: Cloud Infrastructure Software Profile Capabilities}\label{\detokenize{chapters/chapter02:id1}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for Basic Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for High\sphinxhyphen{}Performance Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
e.cap.001
&
\sphinxAtStartPar
Max number of vCPU that can be assigned to a single instance by the Cloud Infrastructure
&
\sphinxAtStartPar
At least 16
&
\sphinxAtStartPar
At least 16
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute-nodes}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute Nodes}}}}
\\
\hline
\sphinxAtStartPar
e.cap.002
&
\sphinxAtStartPar
Max memory that can be assigned to a single instance by the Cloud Infrastructure
&
\sphinxAtStartPar
at least 32 GB
&
\sphinxAtStartPar
at least 32 GB
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:virtual-storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtual Storage}}}}
\\
\hline
\sphinxAtStartPar
e.cap.003
&
\sphinxAtStartPar
Max storage that can be assigned to a single instance by the Cloud Infrastructure
&
\sphinxAtStartPar
at least 320 GB
&
\sphinxAtStartPar
at least 320 GB
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:virtual-storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtual Storage}}}} and
{\hyperref[\detokenize{chapters/chapter04:storage-backend}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage Backend}}}}
\\
\hline
\sphinxAtStartPar
e.cap.004
&
\sphinxAtStartPar
Max number of connection points that can be assigned to a single instance by the Cloud Infrastructure
&
\sphinxAtStartPar
6
&
\sphinxAtStartPar
6
&
\sphinxAtStartPar
Not Detailed
\\
\hline
\sphinxAtStartPar
e.cap.005
&
\sphinxAtStartPar
Max storage that can be attached / mounted to an instance by the Cloud Infrastructure
&
\sphinxAtStartPar
Up to 16TB:sup:\sphinxtitleref{1}
&
\sphinxAtStartPar
Up to 16TB:sup:\sphinxtitleref{1}
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:storage-backend}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage Backend}}}}
\\
\hline
\sphinxAtStartPar
e.cap.006/infra.com.cfg.003
&
\sphinxAtStartPar
CPU pinning support
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
e.cap.007/infra.com.cfg.002
&
\sphinxAtStartPar
NUMA support
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
e.cap.018/infra.com.cfg.005
&
\sphinxAtStartPar
Simultaneous Multithreading (SMT) enabled
&
\sphinxAtStartPar
Must
&
\sphinxAtStartPar
Optional support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
i.cap.018/infra.com.cfg.004
&
\sphinxAtStartPar
Huge pages configured
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{:sup:\textasciigrave{}1\textasciigrave{}} Defined in the \sphinxcode{\sphinxupquote{.bronze}} configuration in
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter04.html\#storage-extensions}{Storage Extensions}


\subsubsection{Cloud Infrastructure Software Profile Extensions Requirements for Compute}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-software-profile-extensions-requirements-for-compute}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Cloud Infrastructure Software Profile Extensions Requirements for Compute}\label{\detokenize{chapters/chapter02:id2}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Profile Extensions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Profile Extra\sphinxhyphen{}Specs
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
e.cap.008/infra.com.acc.cfg.001
&
\sphinxAtStartPar
IPSec Acceleration using the virtio\sphinxhyphen{}ipsec interface
&
\sphinxAtStartPar
Compute Intensive GPU
&&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.010/infra.com.acc.cfg.002
&
\sphinxAtStartPar
Transcoding Acceleration
&
\sphinxAtStartPar
Compute Intensive GPU
&
\sphinxAtStartPar
Video Transcoding
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.011/infra.com.acc.cfg.003
&
\sphinxAtStartPar
Programmable Acceleration
&
\sphinxAtStartPar
Firmware\sphinxhyphen{}programmable adapter
&
\sphinxAtStartPar
Accelerator
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.012
&
\sphinxAtStartPar
Enhanced Cache Management: L=Lean; E=Equal; X=eXpanded
&
\sphinxAtStartPar
E
&
\sphinxAtStartPar
E
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
e.cap.014/infra.com.acc.cfg.004
&
\sphinxAtStartPar
Hardware coprocessor support (GPU/NPU)
&
\sphinxAtStartPar
Compute Intensive GPU
&&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.016/infra.com.acc.cfg.005
&
\sphinxAtStartPar
FPGA/other Acceleration H/W
&
\sphinxAtStartPar
Firmware\sphinxhyphen{}programmable adapter
&&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Cloud Infrastructure Software Profile Requirements for Networking}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-software-profile-requirements-for-networking}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#virtual-networking}{Virtual Networking})

\sphinxAtStartPar
The features and configuration requirements related to virtual
networking for the two (2) types of Cloud Infrastructure Profiles are
specified below followed by networking bandwidth requirements.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Virtual Networking}\label{\detokenize{chapters/chapter02:id3}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{30}{80}|\X{10}{80}|\X{10}{80}|\X{10}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for Basic Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for High\sphinxhyphen{}Performance Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
infra.net.cfg.001
&
\sphinxAtStartPar
IO virtualisation using virtio1.1
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:virtualisation}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtualisation}}}}
\\
\hline
\sphinxAtStartPar
infra.net.cfg.002
&
\sphinxAtStartPar
The overlay network encapsulation protocol needs to enable ECMP in the
underlay to take advantage of the scale\sphinxhyphen{}out features of the network fabric
&
\sphinxAtStartPar
Must support VXLAN, MPLSoUDP, GENEVE, other
&
\sphinxAtStartPar
No requirement specified
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:network-fabric}]{\sphinxcrossref{\DUrole{std,std-ref}{Network Fabric}}}}
\\
\hline
\sphinxAtStartPar
infra.net.cfg.003
&
\sphinxAtStartPar
Network Address Translation
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:network-fabric}]{\sphinxcrossref{\DUrole{std,std-ref}{Network Fabric}}}}
\\
\hline
\sphinxAtStartPar
infra.net.cfg.004
&
\sphinxAtStartPar
Security Groups
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
infra.net.cfg.005
&
\sphinxAtStartPar
SFC support
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:virtual-networking-3rd-party-sdn-solution}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtual Networking \textendash{} 3rd party SDN solution}}}}
\\
\hline
\sphinxAtStartPar
infra.net.cfg.006
&
\sphinxAtStartPar
Traffic patterns symmetry
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
The required number of connection points to an instance is described in
\sphinxcode{\sphinxupquote{e.cap.004}} {\hyperref[\detokenize{chapters/chapter02:2.2.1}]{\emph{above}}}. The table below specifies the required
bandwidth of those connection points.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Network Interface Specifications}\label{\detokenize{chapters/chapter02:id4}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for Basic Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for High Performance Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
n1, n2, n3, n4, n5, n6
&
\sphinxAtStartPar
1, 2, 3, 4, 5, 6 Gbps
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
n10, n20, n30, n40, n50, n60
&
\sphinxAtStartPar
10, 20, 30, 40, 50, 60 Gbps
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
n25, n50, n75, n100, n125, n150
&
\sphinxAtStartPar
25, 50, 75, 100, 125, 150 Gbps
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
n50, n100, n150, n200, n250, n300
&
\sphinxAtStartPar
50, 100, 150, 200, 250, 300 Gbps
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
n100, n200, n300, n400, n500, n600
&
\sphinxAtStartPar
100, 200, 300, 400, 500, 600 Gbps
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Cloud Infrastructure Software Profile Extensions Requirements for Networking}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-software-profile-extensions-requirements-for-networking}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Cloud Infrastructure Software Profile Extensions Requirements
             for Networking}\label{\detokenize{chapters/chapter02:id5}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for Basic Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for High\sphinxhyphen{}Performance Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
e.cap.013/infra.hw.nac.cfg.004
&
\sphinxAtStartPar
SR\sphinxhyphen{}IOV over PCI\sphinxhyphen{}PT
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute-nodes}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute Nodes}}}}
\\
\hline
\sphinxAtStartPar
e.cap.019/infra.net.acc.cfg.001
&
\sphinxAtStartPar
vSwitch optimisation (DPDK)
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute-nodes}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute Nodes}}}} and
{\hyperref[\detokenize{chapters/chapter04:network-quality-of-service}]{\sphinxcrossref{\DUrole{std,std-ref}{Network quality of service}}}}
\\
\hline
\sphinxAtStartPar
e.cap.015/infra.net.acc.cfg.002
&
\sphinxAtStartPar
SmartNIC (for HW Offload)
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.009/infra.net.acc.cfg.003
&
\sphinxAtStartPar
Crypto acceleration
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
infra.net.acc.cfg.004
&
\sphinxAtStartPar
Crypto Acceleration Interface
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Cloud Infrastructure Software Profile Requirements for Storage}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-software-profile-requirements-for-storage}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-software-profiles-features-and-requirements}{Cloud Infrastructure Software Profiles features and requirements})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Cloud Infrastructure Software
             Profile Requirements for Storage}\label{\detokenize{chapters/chapter02:id6}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for Basic Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for High\sphinxhyphen{}Performance Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
infra.stg.cfg.002
&
\sphinxAtStartPar
Storage Block
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage}}}} and
{\hyperref[\detokenize{chapters/chapter04:cinder}]{\sphinxcrossref{\DUrole{std,std-ref}{Cinder}}}}
\\
\hline
\sphinxAtStartPar
infra.stg.cfg.003
&
\sphinxAtStartPar
Storage with replication
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage}}}} and
{\hyperref[\detokenize{chapters/chapter04:transaction-volume-considerations}]{\sphinxcrossref{\DUrole{std,std-ref}{Transaction Volume Considerations}}}}
\\
\hline
\sphinxAtStartPar
infra.stg.cfg.004
&
\sphinxAtStartPar
Storage with encryption
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage}}}}
\\
\hline
\sphinxAtStartPar
infra.stg.acc.cfg.001
&
\sphinxAtStartPar
Storage IOPS oriented
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage}}}}
\\
\hline
\sphinxAtStartPar
infra.stg.acc.cfg.002
&
\sphinxAtStartPar
Storage capacity oriented
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Cloud Infrastructure Software Profile Extensions Requirements for Storage}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-software-profile-extensions-requirements-for-storage}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Cloud Infrastructure Software
Profile Extensions Requirements for Storage
for Networking}\label{\detokenize{chapters/chapter02:id7}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{20}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Profile Extensions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Profile Extra\sphinxhyphen{}Specs
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
infra.stg.acc.cfg.001
&
\sphinxAtStartPar
Storage IOPS oriented
&
\sphinxAtStartPar
Storage Intensive High\sphinxhyphen{}performance storage
&&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
infra.stg.acc.cfg.002
&
\sphinxAtStartPar
Storage capacity oriented
&
\sphinxAtStartPar
High Capacity
&&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Cloud Infrastructure Hardware Profile Requirements}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-hardware-profile-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-hardware-profiles-features-and-requirements}{Cloud Infrastructure Hardware Profiles features and requirements.})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Cloud Infrastructure Hardware
Profile Requirements}\label{\detokenize{chapters/chapter02:id8}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for Basic Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for High\sphinxhyphen{}Performance Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
infra.hw.001
&
\sphinxAtStartPar
CPU Architecture (Values such as x64, ARM, etc.)
&&&\\
\hline
\sphinxAtStartPar
infra.hw.cpu.cfg.001
&
\sphinxAtStartPar
Minimum number of CPU (Sockets)
&
\sphinxAtStartPar
2
&
\sphinxAtStartPar
2
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.cpu.cfg.002
&
\sphinxAtStartPar
Minimum number of Cores per CPU
&
\sphinxAtStartPar
20
&
\sphinxAtStartPar
20
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.cpu.cfg.003
&
\sphinxAtStartPar
NUMA
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.cpu.cfg.004
&
\sphinxAtStartPar
Simultaneous Multithreading/Symmetric Multiprocessing (SMT/SMP)
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.stg.hdd.cfg.001
&
\sphinxAtStartPar
Local Storage HDD
&
\sphinxAtStartPar
No requirement specified
&
\sphinxAtStartPar
No requirement specified
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.stg.ssd.cfg.002
&
\sphinxAtStartPar
Local Storage SSD
&
\sphinxAtStartPar
Should support
&
\sphinxAtStartPar
Should support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.nic.cfg.001
&
\sphinxAtStartPar
Total Number of NIC Ports available in the host
&
\sphinxAtStartPar
4
&
\sphinxAtStartPar
4
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.nic.cfg.002
&
\sphinxAtStartPar
Port speed specified in Gbps (minimum values)
&
\sphinxAtStartPar
10
&
\sphinxAtStartPar
25
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.pci.cfg.001
&
\sphinxAtStartPar
Number of PCIe slots available in the host
&
\sphinxAtStartPar
8
&
\sphinxAtStartPar
8
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
infra.hw.pci.cfg.002
&
\sphinxAtStartPar
PCIe speed
&
\sphinxAtStartPar
Gen 3
&
\sphinxAtStartPar
Gen 3
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
infra.hw.pci.cfg.003
&
\sphinxAtStartPar
PCIe Lanes
&
\sphinxAtStartPar
8
&
\sphinxAtStartPar
8
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
infra.hw.nac.cfg.003
&
\sphinxAtStartPar
Compression
&
\sphinxAtStartPar
No requirement specified
&
\sphinxAtStartPar
No requirement specified
&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Cloud Infrastructure Hardware Profile\sphinxhyphen{}Extensions Requirements}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-hardware-profile-extensions-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-hardware-profiles-features-and-requirements}{Cloud Infrastructure Hardware Profiles features and requirements.})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Cloud Infrastructure Hardware
Profile Extensions Requirements}\label{\detokenize{chapters/chapter02:id9}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for Basic Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement for High\sphinxhyphen{}Performance Profile
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
e.cap.014/infra.hw.cac.cfg.001
&
\sphinxAtStartPar
GPU
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.016/infra.hw.cac.cfg.002
&
\sphinxAtStartPar
FPGA/other Acceleration H/W
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.009/infra.hw.nac.cfg.001
&
\sphinxAtStartPar
Crypto Acceleration
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.015/infra.hw.nac.cfg.002
&
\sphinxAtStartPar
SmartNIC
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
infra.hw.nac.cfg.003
&
\sphinxAtStartPar
Compression
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
e.cap.013/infra.hw.nac.cfg.004
&
\sphinxAtStartPar
SR\sphinxhyphen{}IOV over PCI\sphinxhyphen{}PT
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
Yes
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:compute-node-configurations-for-profiles-and-openstack-flavors}]{\sphinxcrossref{\DUrole{std,std-ref}{Compute node configurations for Profiles and OpenStack Flavors}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Cloud Infrastructure Management Requirements}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-management-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter04.html\#cloud-infrastructure-management-capabilities}{Cloud Infrastructure Management Capabilities})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Cloud Infrastructure
Management Requirements}\label{\detokenize{chapters/chapter02:id10}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{30}{80}|\X{10}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Requirement (common to all Profiles)
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
e.man.001
&
\sphinxAtStartPar
Capability to allocate virtual compute resources to a workload
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
e.man.002
&
\sphinxAtStartPar
Capability to allocate virtual storage resources to a workload
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
e.man.003
&
\sphinxAtStartPar
Capability to allocate virtual networking resources to a workload
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
e.man.004
&
\sphinxAtStartPar
Capability to isolate resources between tenants
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:tenant-isolation}]{\sphinxcrossref{\DUrole{std,std-ref}{Tenant Isolation}}}}
\\
\hline
\sphinxAtStartPar
e.man.005
&
\sphinxAtStartPar
Capability to manage workload software images
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:glance}]{\sphinxcrossref{\DUrole{std,std-ref}{Glance}}}}
\\
\hline
\sphinxAtStartPar
e.man.006
&
\sphinxAtStartPar
Capability to provide information related to allocated virtualised
resources per tenant
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter07:logging-monitoring-and-analytics}]{\sphinxcrossref{\DUrole{std,std-ref}{Logging, Monitoring and Analytics}}}}
\\
\hline
\sphinxAtStartPar
e.man.007
&
\sphinxAtStartPar
Capability to notify state changes of allocated resources
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter07:logging-monitoring-and-analytics}]{\sphinxcrossref{\DUrole{std,std-ref}{Logging, Monitoring and Analytics}}}}
\\
\hline
\sphinxAtStartPar
e.man.008
&
\sphinxAtStartPar
Capability to collect and expose performance information on virtualised
resources allocated
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter07:logging-monitoring-and-analytics}]{\sphinxcrossref{\DUrole{std,std-ref}{Logging, Monitoring and Analytics}}}}
\\
\hline
\sphinxAtStartPar
e.man.009
&
\sphinxAtStartPar
Capability to collect and notify fault information on virtualised
resources
&
\sphinxAtStartPar
Must support
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter07:logging-monitoring-and-analytics}]{\sphinxcrossref{\DUrole{std,std-ref}{Logging, Monitoring and Analytics}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Cloud Infrastructure Security Requirements}
\label{\detokenize{chapters/chapter02:cloud-infrastructure-security-requirements}}

\paragraph{System Hardening Requirements}
\label{\detokenize{chapters/chapter02:system-hardening-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#system-hardening}{System Hardening})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} System Hardening Requirements}\label{\detokenize{chapters/chapter02:id11}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.gen.001
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} maintain the specified configuration
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-lcm}]{\sphinxcrossref{\DUrole{std,std-ref}{Security LCM}}}} and
{\hyperref[\detokenize{chapters/chapter07:cloud-infrastructure-provisioning-and-configuration-management}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud Infrastructure provisioning and configuration management}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.002
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
All systems part of Cloud Infrastructure \sphinxstylestrong{must} support hardening as
defined in \sphinxhref{https://www.cisecurity.org/white-papers/cis-password-policy-guide/}{CIS Password Policy Guide}
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:password-policy}]{\sphinxcrossref{\DUrole{std,std-ref}{Password policy}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.003
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
All servers part of Cloud Infrastructure \sphinxstylestrong{must} support a root of
trust and secure boot
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:server-boot-hardening}]{\sphinxcrossref{\DUrole{std,std-ref}{Server boot hardening}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.004
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Operating Systems of all the servers part of Cloud Infrastructure
\sphinxstylestrong{must} be hardened by removing or disabling unnecessary services,
applications and network protocols, configuring operating system user
authentication, configuring resource controls, installing and
configuring additional security controls where needed, and testing the
security of the Operating System (NIST SP 800\sphinxhyphen{}123)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:function-and-software}]{\sphinxcrossref{\DUrole{std,std-ref}{Function and Software}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.005
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Operating System level access control
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:system-access}]{\sphinxcrossref{\DUrole{std,std-ref}{System Access}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.006
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Secure logging. Logging with root account
must be prohibited when root privileges are not required
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:system-access}]{\sphinxcrossref{\DUrole{std,std-ref}{System Access}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.007
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
All servers part of Cloud Infrastructure \sphinxstylestrong{must} be Time synchronised
with authenticated Time service
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-logs-time-synchronisation}]{\sphinxcrossref{\DUrole{std,std-ref}{Security Logs Time Synchronisation}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.008
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
All servers part of Cloud Infrastructure \sphinxstylestrong{must} be regularly updated
to address security vulnerabilities
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-lcm}]{\sphinxcrossref{\DUrole{std,std-ref}{Security LCM}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.009
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support software integrity protection and
verification
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:integrity-of-openstack-components-configuration}]{\sphinxcrossref{\DUrole{std,std-ref}{Integrity of OpenStack components configuration}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.010
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Cloud Infrastructure \sphinxstylestrong{must} support encrypted storage, for
example, block, object and file storage, with access to encryption
keys restricted based on a need to know
(\sphinxhref{https://www.cisecurity.org/controls/controlled-access-based-on-the-need-to-know/}{Controlled Access Based on the Need to Know})
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.012
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Operator \sphinxstylestrong{must} ensure that only authorised actors have physical
access to the underlying infrastructure
&
\sphinxAtStartPar
This requirement’s verification must be part of the organisation security process
\\
\hline
\sphinxAtStartPar
sec.gen.013
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} ensure that only authorised actors have logical
access to the underlying infrastructure
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:system-access}]{\sphinxcrossref{\DUrole{std,std-ref}{System Access}}}}
\\
\hline
\sphinxAtStartPar
sec.gen.015
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
Any change to the Platform \sphinxstylestrong{must} be logged as a security event, and
the logged event must include the identity of the entity making the
change, the change, the date and the time of the change
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-lcm}]{\sphinxcrossref{\DUrole{std,std-ref}{Security LCM}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Platform and Access Requirements}
\label{\detokenize{chapters/chapter02:platform-and-access-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#platform-and-access}{Platform and Access})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Platform and Access
Requirements}\label{\detokenize{chapters/chapter02:id12}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{10}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.sys.001
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support authenticated and secure access to API, GUI
and command line interfaces
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:rbac}]{\sphinxcrossref{\DUrole{std,std-ref}{RBAC}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.002
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Traffic Filtering for workloads
(for example, Firewall)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.003
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Secure and encrypted communications, and
confidentiality and integrity of network
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.004
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Cloud Infrastructure \sphinxstylestrong{must} support authentication, integrity and
confidentiality on all network channels
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.005
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Cloud Infrastructure \sphinxstylestrong{must} segregate the underlay and overlay
networks
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.006
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Cloud Infrastructure \sphinxstylestrong{must} be able to utilise the Cloud
Infrastructure Manager identity lifecycle management capabilities
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:identity-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Identity Security}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.007
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} implement controls enforcing separation of duties
and privileges, least privilege use and least common mechanism
(Role\sphinxhyphen{}Based Access Control)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:rbac}]{\sphinxcrossref{\DUrole{std,std-ref}{RBAC}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.008
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} be able to assign the Entities that comprise the
tenant networks to different trust domains. Communication between
different trust domains is not allowed, by default
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.009
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support creation of Trust Relationships between
trust domains. These maybe uni\sphinxhyphen{}directional relationships where the
trusting domain trusts another domain (the “trusted domain”) to
authenticate users for them them or to allow access to its resources
from the trusted domain. In a bidirectional relationship both domain
are “trusting” and “trusted”
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:logical-segregation-and-high-availability}]{\sphinxcrossref{\DUrole{std,std-ref}{Logical segregation and high availability}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.010
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
For two or more domains without existing trust relationships, the Platform
\sphinxstylestrong{must not} allow the effect of an attack on one domain to impact the other
domains either directly or indirectly
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:logical-segregation-and-high-availability}]{\sphinxcrossref{\DUrole{std,std-ref}{Logical segregation and high availability}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.011
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must not} reuse the same authentication credentials
(e.g., key pairs) on different Platform components (e.g., different
hosts, or different services)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:system-access}]{\sphinxcrossref{\DUrole{std,std-ref}{System Access}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.012
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} protect all secrets by using strong encryption
techniques and storing the protected secrets externally from the
component (e.g., in OpenStack Barbican)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:barbican}]{\sphinxcrossref{\DUrole{std,std-ref}{Barbican}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.013
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} generate secrets dynamically as and when needed
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:barbican}]{\sphinxcrossref{\DUrole{std,std-ref}{Barbican}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.015
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must not} contain back door entries (unpublished access
points, APIs, etc.)
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
sec.sys.016
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
Login access to the Platform’s components \sphinxstylestrong{must} be through encrypted
protocols such as SSH v2 or TLS v1.2 or higher. Note: Hardened jump
servers isolated from external networks are recommended
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-lcm}]{\sphinxcrossref{\DUrole{std,std-ref}{Security LCM}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.017
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} provide the capability of using digital certificates
that comply with X.509 standards issued by a trusted Certification Authority
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.018
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} provide the capability of allowing certificate renewal
and revocation
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.sys.019
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} provide the capability of testing the validity
of a digital certificate (CA signature, validity period, non revocation
identity)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Confidentiality and Integrity Requirements}
\label{\detokenize{chapters/chapter02:confidentiality-and-integrity-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#confidentiality-and-integrity}{Confidentiality and Integrity})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Confidentiality and Integrity
Requirements}\label{\detokenize{chapters/chapter02:id13}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.ci.001
&
\sphinxAtStartPar
Confidentiality/Integrity
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Confidentiality and Integrity of data
at rest and in transit
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.ci.003
&
\sphinxAtStartPar
Confidentiality/Integrity
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Confidentiality and Integrity of data
related metadata
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.ci.004
&
\sphinxAtStartPar
Confidentiality
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Confidentiality of processes and
restrict information sharing with only the process owner (e.g.,
tenant)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.ci.005
&
\sphinxAtStartPar
Confidentiality/Integrity
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Confidentiality and Integrity of process\sphinxhyphen{}
related metadata and restrict information sharing with only the
process owner (e.g., tenant)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.ci.006
&
\sphinxAtStartPar
Confidentiality/Integrity
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Confidentiality and Integrity of
workload resource utilisation (RAM, CPU, Storage, Network I/O, cache,
hardware offload) and restrict information sharing with only the
workload owner (e.g., tenant)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:platform-access}]{\sphinxcrossref{\DUrole{std,std-ref}{Platform Access}}}}
\\
\hline
\sphinxAtStartPar
sec.ci.007
&
\sphinxAtStartPar
Confidentiality/Integrity
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must not} allow Memory Inspection by any actor
other than the authorised actors for the Entity to which Memory is
assigned (e.g., tenants owning the workload), for Lawful
Inspection, and for secure monitoring services. Administrative
access must be managed using Platform Identity Lifecycle
Management
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:platform-access}]{\sphinxcrossref{\DUrole{std,std-ref}{Platform Access}}}}
\\
\hline
\sphinxAtStartPar
sec.ci.008
&
\sphinxAtStartPar
Confidentiality
&
\sphinxAtStartPar
The Cloud Infrastructure \sphinxstylestrong{must} support tenant networks segregation
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Workload Security Requirements}
\label{\detokenize{chapters/chapter02:workload-security-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#workload-security-requirements}{Workload security requirements})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Workload Security
Requirements}\label{\detokenize{chapters/chapter02:id14}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.wl.001
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Workload placement policy
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
sec.wl.002
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Cloud Infrastructure \sphinxstylestrong{must} provide methods to ensure the
platform’s trust status and integrity (e.g., remote attestation,
Trusted Platform Module)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:cloud-infrastructure-and-vim-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud Infrastructure and VIM Security}}}}
\\
\hline
\sphinxAtStartPar
sec.wl.003
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support secure provisioning of Workloads
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
sec.wl.004
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Location assertion (for mandated in\sphinxhyphen{}
country or location requirements)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
sec.wl.005
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support the separation of production and non\sphinxhyphen{}
production Workloads
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
sec.wl.006
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support the separation of Workloads based on
their categorisation (for example, payment card information,
healthcare, etc.)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\sphinxAtStartPar
sec.wl.007
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Operator \sphinxstylestrong{must} implement processes and tools to verify
NF authenticity and integrity
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Image Security Requirements}
\label{\detokenize{chapters/chapter02:image-security-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#image-security}{Image Security})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Image Security
Requirements}\label{\detokenize{chapters/chapter02:id15}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.img.001
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Images from untrusted sources \sphinxstylestrong{must not} be used
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\sphinxAtStartPar
sec.img.002
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Images \sphinxstylestrong{must} be scanned to be maintained free from known
vulnerabilities
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\sphinxAtStartPar
sec.img.003
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Images \sphinxstylestrong{must not} be configured to run with privileges higher
than the privileges of the actor authorised to run them
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\sphinxAtStartPar
sec.img.004
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Images \sphinxstylestrong{must} only be accessible to authorised actors
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:integrity-of-openstack-components-configuration}]{\sphinxcrossref{\DUrole{std,std-ref}{Integrity of OpenStack components configuration}}}}
\\
\hline
\sphinxAtStartPar
sec.img.005
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Image Registries \sphinxstylestrong{must} only be accessible to authorised actors
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:integrity-of-openstack-components-configuration}]{\sphinxcrossref{\DUrole{std,std-ref}{Integrity of OpenStack components configuration}}}}
\\
\hline
\sphinxAtStartPar
sec.img.006
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Image Registries \sphinxstylestrong{must} only be accessible over networks that
enforce authentication, integrity and confidentiality
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:integrity-of-openstack-components-configuration}]{\sphinxcrossref{\DUrole{std,std-ref}{Integrity of OpenStack components configuration}}}}
\\
\hline
\sphinxAtStartPar
sec.img.007
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Image registries \sphinxstylestrong{must} be clear of vulnerable and out of date versions
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\sphinxAtStartPar
sec.img.008
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Images \sphinxstylestrong{must not} include any secrets. Secrets include passwords,
cloud provider credentials, SSH keys, TLS certificate keys, etc.
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Security LCM Requirements}
\label{\detokenize{chapters/chapter02:security-lcm-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#security-lcm}{Security LCM})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Security LCM
Requirements}\label{\detokenize{chapters/chapter02:id16}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.lcm.001
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} support Secure Provisioning, Availability, and
Deprovisioning (Secure Clean\sphinxhyphen{}Up) of workload resources where Secure
Clean\sphinxhyphen{}Up includes tear\sphinxhyphen{}down, defense against virus or other attacks
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:monitoring-and-security-audit}]{\sphinxcrossref{\DUrole{std,std-ref}{Monitoring and Security Audit}}}}
\\
\hline
\sphinxAtStartPar
sec.lcm.002
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Cloud Operator \sphinxstylestrong{must} use management protocols limiting security
risk such as SNMPv3, SSH v2, ICMP, NTP, syslog and TLS v1.2 or higher
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-lcm}]{\sphinxcrossref{\DUrole{std,std-ref}{Security LCM}}}}
\\
\hline
\sphinxAtStartPar
sec.lcm.003
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Cloud Operator \sphinxstylestrong{must} implement and strictly follow change
management processes for Cloud Infrastructure, Infrastructure
Manager and othercomponents of the cloud, and Platform change control
on hardware
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:monitoring-and-security-audit}]{\sphinxcrossref{\DUrole{std,std-ref}{Monitoring and Security Audit}}}}
\\
\hline
\sphinxAtStartPar
sec.lcm.005
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
Platform \sphinxstylestrong{must} provide logs and these logs must be monitored for
anomalous behaviour
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:monitoring-and-security-audit}]{\sphinxcrossref{\DUrole{std,std-ref}{Monitoring and Security Audit}}}}
\\
\hline
\sphinxAtStartPar
sec.lcm.006
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} verify the integrity of all Resource management
requests
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity-of-tenant-data-sec-ci-001}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity of tenant data (sec.ci.001)}}}}
\\
\hline
\sphinxAtStartPar
sec.lcm.007
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} be able to update newly instantiated, suspended,
hibernated, migrated and restarted images with current time information
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
sec.lcm.008
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} be able to update newly instantiated, suspended,
hibernated, migrated and restarted images with relevant DNS information
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
sec.lcm.009
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} be able to update the tag of newly instantiated,
suspended, hibernated, migrated and restarted images with relevant
geolocation (geographical) information
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
sec.lcm.010
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} log all changes to geolocation along with the
mechanisms and sources of location information (i.e. GPS, IP block,
and timing)
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
sec.lcm.011
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} implement Security life cycle management
processes including the proactive update and patching of all
deployed Cloud Infrastructure software
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:patches}]{\sphinxcrossref{\DUrole{std,std-ref}{Patches}}}}
\\
\hline
\sphinxAtStartPar
sec.lcm.012
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} log any access privilege escalation
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:what-to-log-what-not-to-log}]{\sphinxcrossref{\DUrole{std,std-ref}{What to Log / What NOT to Log}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Monitoring and Security Audit Requirements}
\label{\detokenize{chapters/chapter02:monitoring-and-security-audit-requirements}}
\sphinxAtStartPar
(source
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#monitoring-and-security-audit}{Monitoring and Security Audit})

\sphinxAtStartPar
The Platform is assumed to provide configurable alerting and
notification capability and the operator is assumed to have automated
systems, policies and procedures to act on alerts and notifications in a
timely fashion. In the following the monitoring and logging capabilities
can trigger alerts and notifications for appropriate action.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Monitoring and Security Audit
Requirements}\label{\detokenize{chapters/chapter02:id17}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.mon.001
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
Platform \sphinxstylestrong{must} provide logs and these logs must be regularly
monitored for events of interest. The logs must contain the following
fields: event type, date/time, protocol, service or program used for
access, success/failure, login ID or process ID, IP address and ports
(source and destination) involved
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:required-fields}]{\sphinxcrossref{\DUrole{std,std-ref}{Required Fields}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.002
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
Security logs \sphinxstylestrong{must} be time synchronised
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-logs-time-synchronisation}]{\sphinxcrossref{\DUrole{std,std-ref}{Security Logs Time Synchronisation}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.003
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} log all changes to time server source, time,
date and time zones
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-logs-time-synchronisation}]{\sphinxcrossref{\DUrole{std,std-ref}{Security Logs Time Synchronisation}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.004
&
\sphinxAtStartPar
Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} secure and protect Audit logs (containing
sensitive information) both in\sphinxhyphen{}transit and at rest
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:security-lcm}]{\sphinxcrossref{\DUrole{std,std-ref}{Security LCM}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.005
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit various behaviours of
connection and login attempts to detect access attacks and potential
access attempts and take corrective accordingly actions
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:what-to-log-what-not-to-log}]{\sphinxcrossref{\DUrole{std,std-ref}{What to Log / What NOT to Log}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.006
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit operations by authorised
account access after login to detect malicious operational activity
and take corrective actions
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:monitoring-and-security-audit}]{\sphinxcrossref{\DUrole{std,std-ref}{Monitoring and Security Audit}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.007
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit security parameter
configurations for compliance with defined security policies
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:integrity-of-openstack-components-configuration}]{\sphinxcrossref{\DUrole{std,std-ref}{Integrity of OpenStack components configuration}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.008
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit externally exposed interfaces
for illegal access (attacks) and take corrective security hardening
measures
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity-of-communications-sec-ci-001}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity of communications (sec.ci.001)}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.009
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit service for various attacks
(malformed messages, signalling flooding and replaying, etc.) and take
corrective actions accordingly
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:monitoring-and-security-audit}]{\sphinxcrossref{\DUrole{std,std-ref}{Monitoring and Security Audit}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.010
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit running processes to detect
unexpected or unauthorised processes and take corrective actions
accordingly
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:monitoring-and-security-audit}]{\sphinxcrossref{\DUrole{std,std-ref}{Monitoring and Security Audit}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.011
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit logs from infrastructure elements
and workloads to detected anomalies in the system components and take
corrective actions accordingly
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:creating-logs}]{\sphinxcrossref{\DUrole{std,std-ref}{Creating Logs}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.012
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} Monitor and Audit Traffic patterns and volumes to
prevent malware download attempts
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:confidentiality-and-integrity}]{\sphinxcrossref{\DUrole{std,std-ref}{Confidentiality and Integrity}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.013
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The monitoring system \sphinxstylestrong{must not} affect the security (integrity and
confidentiality) of the infrastructure, workloads, or the user data
(through back door entries)
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
sec.mon.015
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} ensure that the Monitoring systems are never
starved of resources and must activate alarms when resource utilisation
exceeds a configurable threshold
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:monitoring-and-security-audit}]{\sphinxcrossref{\DUrole{std,std-ref}{Monitoring and Security Audit}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.017
&
\sphinxAtStartPar
Audit
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} audit systems for any missing security patches
and take appropriate actions
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:patches}]{\sphinxcrossref{\DUrole{std,std-ref}{Patches}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.018
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Platform, starting from initialisation, \sphinxstylestrong{must} collect and
analyse logs to identify security events, and store these events
in an external system
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:where-to-log}]{\sphinxcrossref{\DUrole{std,std-ref}{Where to Log}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.019
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Platform’s components \sphinxstylestrong{must not} include an authentication
credential, e.g., password, in any logs, even if encrypted
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:what-to-log-what-not-to-log}]{\sphinxcrossref{\DUrole{std,std-ref}{What to Log / What NOT to Log}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.020
&
\sphinxAtStartPar
Monitoring/Audit
&
\sphinxAtStartPar
The Platform’s logging system \sphinxstylestrong{must} support the storage of security
audit logs for a configurable period of time
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:data-retention}]{\sphinxcrossref{\DUrole{std,std-ref}{Data Retention}}}}
\\
\hline
\sphinxAtStartPar
sec.mon.021
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{must} store security events locally if the external
logging system is unavailable and shall periodically attempt to send
these to the external logging system until successful
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:where-to-log}]{\sphinxcrossref{\DUrole{std,std-ref}{Where to Log}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Open\sphinxhyphen{}Source Software Security Requirements}
\label{\detokenize{chapters/chapter02:open-source-software-security-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#open-source-software-security}{Open\sphinxhyphen{}Source Software Security})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} Open\sphinxhyphen{}Source Software Security
Requirements}\label{\detokenize{chapters/chapter02:id18}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.oss.001
&
\sphinxAtStartPar
Software
&
\sphinxAtStartPar
Open\sphinxhyphen{}source code \sphinxstylestrong{must} be inspected by tools with various capabilities
for static and dynamic code analysis
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\sphinxAtStartPar
sec.oss.002
&
\sphinxAtStartPar
Software
&
\sphinxAtStartPar
The CVE (Common Vulnerabilities and Exposures) \sphinxstylestrong{must} be used to
identify vulnerabilities and their severity rating for open\sphinxhyphen{}source
code part of Cloud Infrastructure and workloads software
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:patches}]{\sphinxcrossref{\DUrole{std,std-ref}{Patches}}}}
\\
\hline
\sphinxAtStartPar
sec.oss.003
&
\sphinxAtStartPar
Software
&
\sphinxAtStartPar
Critical and high severity rated vulnerabilities \sphinxstylestrong{must} be
fixed in a timely manner. Refer to the CVSS (Common Vulnerability
Scoring System) to know a vulnerability score and its associated rate
(low, medium, high, or critical)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:patches}]{\sphinxcrossref{\DUrole{std,std-ref}{Patches}}}}
\\
\hline
\sphinxAtStartPar
sec.oss.004
&
\sphinxAtStartPar
Software
&
\sphinxAtStartPar
A dedicated internal isolated repository separated from the production
environment \sphinxstylestrong{must} be used to store vetted open\sphinxhyphen{}source content
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{IaaC security Requirements}
\label{\detokenize{chapters/chapter02:iaac-security-requirements}}
\sphinxAtStartPar
(source
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#iaac-secure-design-and-architecture-stage-requirements}{IaaC \sphinxhyphen{} Secure Design and Architecture Stage Requirements})

\sphinxAtStartPar
\sphinxstylestrong{Secure Code Stage Requirements}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: IaaC Security Requirements,
Secure Code Stage}\label{\detokenize{chapters/chapter02:id19}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.code.001
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
SAST \sphinxhyphen{}Static Application Security Testing \sphinxstylestrong{must} be applied during
Secure Coding stage triggered by Pull, Clone or Comment trigger.
Security testing that analyses application source code for software
vulnerabilities and gaps against bestpractices. Example: open source
OWASP range of tools
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:workload-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Workload Security}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Continuous Build, Integration and Testing Stage Requirements}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} IaaC Security Requirements,
Continuous Build, Integration and Testing Stage}\label{\detokenize{chapters/chapter02:id20}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.bld.003
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Image Scan \sphinxstylestrong{must} be applied during the Continuous Build,
Integration and Testing stage triggered by Package trigger,
example: A push of a container image to a containerregistry may
trigger a vulnerability scan before the image becomes available in
the registry
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Continuous Delivery and Deployment Stage Requirements}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} IaaC Security Requirements,
Continuous Delivery and Deployment Stage}\label{\detokenize{chapters/chapter02:id21}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.del.001
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Image Scan \sphinxstylestrong{must} be applied during the Continuous Delivery and
Deployment stage triggered by Publish to Artifact and Image
Repository trigger. Example: GitLab uses the open source Clair
engine for container image scanning
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\sphinxAtStartPar
sec.del.002
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Code Signing \sphinxstylestrong{must} be applied during the Continuous Deliveryand
Deployment stage and Image Repository trigger. Code Signing provides
authentication to assure that downloaded files are form the publisher
named on the certificate
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\sphinxAtStartPar
sec.del.004
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Component Vulnerability Scan \sphinxstylestrong{must} be applied during the Continuous
Delivery and Deployment stage triggered by Instantiate Infrastructure
trigger. The vulnerability scanning system is deployed on the cloud
platform to detect security vulnerabilities of specified components
through scanning and to provide timely security protection. Example:
OWASP Zed Attack Proxy (ZAP)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter06:image-security}]{\sphinxcrossref{\DUrole{std,std-ref}{Image Security}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Runtime Defence and Monitoring Requirements}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements \sphinxhyphen{} IaaC Security Requirements,
Runtime Defence and Monitoring Stage}\label{\detokenize{chapters/chapter02:id22}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.run.001
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Component Vulnerability Monitoring \sphinxstylestrong{must} be continuously applied
during the Runtime Defence and monitoring stage. Security technology that
monitors components like virtual servers and assesses data, applications,
and infrastructure forsecurity risks
&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Compliance with Standards Requirements}
\label{\detokenize{chapters/chapter02:compliance-with-standards-requirements}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#compliance-with-standards}{Compliance with Standards})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: Compliance with Standards}\label{\detokenize{chapters/chapter02:id23}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
sec.std.012
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Public Cloud Operator \sphinxstylestrong{must}, and the Private Cloud Operator
\sphinxstylestrong{may} be certified to be compliant with the International Standard
on Awareness Engagements (ISAE) 3402 (in the US:SSAE 16); International
Standard on Awareness Engagements (ISAE) 3402. US Equivalent: SSAE16
&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsection{Architecture and OpenStack Requirements}
\label{\detokenize{chapters/chapter02:architecture-and-openstack-requirements}}
\sphinxAtStartPar
“Architecture” in this chapter refers to Cloud Infrastructure (referred
to as NFVI by ETSI) + VIM (as specified in Reference Model Chapter 3).


\subsubsection{General Requirements}
\label{\detokenize{chapters/chapter02:general-requirements}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{General Requirements}\label{\detokenize{chapters/chapter02:id24}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
gen.ost.01
&
\sphinxAtStartPar
Open source
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} use OpenStack APIs
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:consolidated-set-of-apis}]{\sphinxcrossref{\DUrole{std,std-ref}{Consolidated Set of APIs}}}}
\\
\hline
\sphinxAtStartPar
gen.ost.02
&
\sphinxAtStartPar
Open source
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support dynamic request and configuration of
virtual resources (compute, network, storage) through OpenStack APIs
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:consolidated-set-of-apis}]{\sphinxcrossref{\DUrole{std,std-ref}{Consolidated Set of APIs}}}}
\\
\hline
\sphinxAtStartPar
gen.rsl.01
&
\sphinxAtStartPar
Resiliency
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support resilient OpenStack components that are
required for the continued availability of running workloads
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:containerised-openstack-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Containerised OpenStack Services}}}}
\\
\hline
\sphinxAtStartPar
gen.avl.01
&
\sphinxAtStartPar
Availability
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide High Availability for OpenStack
components
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:underlying-resources}]{\sphinxcrossref{\DUrole{std,std-ref}{Underlying Resources}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Infrastructure Requirements}
\label{\detokenize{chapters/chapter02:infrastructure-requirements}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Infrastructure Requirements}\label{\detokenize{chapters/chapter02:id25}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
inf.com.01
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide compute resources for instances
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:cloud-workload-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud Workload Services}}}}
\\
\hline
\sphinxAtStartPar
inf.com.04
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} be able to support multiple CPU type options
to support various infrastructure profiles (Basic and High
Performance)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:support-for-cloud-infrastructure-profiles-and-flavors}]{\sphinxcrossref{\DUrole{std,std-ref}{Support for Cloud Infrastructure Profiles and flavors}}}}
\\
\hline
\sphinxAtStartPar
inf.com.05
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support Hardware Platforms with NUMA
capabilities
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:support-for-cloud-infrastructure-profiles-and-flavors}]{\sphinxcrossref{\DUrole{std,std-ref}{Support for Cloud Infrastructure Profiles and flavors}}}}
\\
\hline
\sphinxAtStartPar
inf.com.06
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support CPU Pinning of the vCPUs of an
instance
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:support-for-cloud-infrastructure-profiles-and-flavors}]{\sphinxcrossref{\DUrole{std,std-ref}{Support for Cloud Infrastructure Profiles and flavors}}}}
\\
\hline
\sphinxAtStartPar
inf.com.07
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support different hardware configurations
to support various infrastructure profiles (Basic and High
Performance)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:cloud-partitioning-host-aggregates-availability-zones}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud partitioning: Host Aggregates, Availability Zones}}}}
\\
\hline
\sphinxAtStartPar
inf.com.08
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support allocating certain number of host
cores for all non\sphinxhyphen{}tenant workloads such as for OpenStack services.
SMT threads can be allocated to individual OpenStack services or their
components. \sphinxhref{https://docs.openstack.org/nova/latest/configuration/config.html\#compute.cpu\_dedicated\_set}{Dedicating host cores to certain workloads
(e.g., OpenStack services)}.
Please see example, \sphinxhref{https://docs.openstack.org/nova/latest/admin/cpu-topologies.html}{Configuring libvirt compute nodes for CPU pinning}
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:cloud-partitioning-host-aggregates-availability-zones}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud partitioning: Host Aggregates, Availability Zones}}}}
\\
\hline
\sphinxAtStartPar
inf.com.09
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} ensure that the host cores assigned to
non\sphinxhyphen{}tenant and tenant workloads are SMT aware: that is, a host core and
its associated SMT threads are either all assigned to non\sphinxhyphen{}tenant
workloads or all assigned to tenant workloads
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:pinned-and-unpinned-cpus}]{\sphinxcrossref{\DUrole{std,std-ref}{Pinned and Unpinned CPUs}}}}
\\
\hline
\sphinxAtStartPar
inf.stg.01
&
\sphinxAtStartPar
Storage
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide remote (not directly attached to the
host) Block storage for Instances
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage}}}}
\\
\hline
\sphinxAtStartPar
inf.stg.02
&
\sphinxAtStartPar
Storage
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide Object storage for Instances.
Operators \sphinxstylestrong{may} choose not to implement Object Storage but must be
cognizant of the the risk of “Compliant VNFs” failing in their
environment
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:swift}]{\sphinxcrossref{\DUrole{std,std-ref}{Swift}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.01
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide virtual network interfaces to
instances
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:neutron}]{\sphinxcrossref{\DUrole{std,std-ref}{Neutron}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.02
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} include capabilities for integrating SDN
controllers to support provisioning of network services, from the SDN
OpenStack Neutron service, such as networking of VTEPs to the Border
Edge based VRFs
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:virtual-networking-3rd-party-sdn-solution}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtual Networking \textendash{} 3rd party SDN solution}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.03
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support low latency and high throughput
traffic needs
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:network-fabric}]{\sphinxcrossref{\DUrole{std,std-ref}{Network Fabric}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.05
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} allow for East/West tenant traffic within the
cloud (via tunnelled encapsulation overlay such as VXLAN or Geneve)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:network-fabric}]{\sphinxcrossref{\DUrole{std,std-ref}{Network Fabric}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.07
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture must support network \sphinxhref{https://cntt.readthedocs.io/en/latest/common/glossary.html\#cloud-platform-abstraction-related-terminology}{\DUrole{xref,std,std-ref}{resiliency}}
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:network}]{\sphinxcrossref{\DUrole{std,std-ref}{Network}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.10
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Cloud Infrastructure Network Fabric \sphinxstylestrong{must} be capable of enabling
highly available (Five 9’s or better) Cloud Infrastructure
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:network}]{\sphinxcrossref{\DUrole{std,std-ref}{Network}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.15
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support multiple networking options for Cloud
Infrastructure to support various infrastructure profiles (Basic and
High Performance)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:neutron-extensions}]{\sphinxcrossref{\DUrole{std,std-ref}{Neutron Extensions}}}} and \sphinxhref{https://wiki.openstack.org/wiki/Neutron\_Plugins\_and\_Drivers}{OpenStack Neutron Plugins}
\\
\hline
\sphinxAtStartPar
inf.nw.16
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support dual stack IPv4 and IPv6 for tenant
networks and workloads
&
\sphinxAtStartPar
Not detailed
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{VIM Requirements}
\label{\detokenize{chapters/chapter02:vim-requirements}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{VIM Requirements}\label{\detokenize{chapters/chapter02:id26}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
vim.01
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} allow infrastructure resource sharing
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:consumable-infrastructure-resources-and-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Consumable Infrastructure Resources and Services}}}}
\\
\hline
\sphinxAtStartPar
vim.03
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} allow VIM to discover and manage Cloud
Infrastructure resources
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:placement}]{\sphinxcrossref{\DUrole{std,std-ref}{Placement}}}}
\\
\hline
\sphinxAtStartPar
vim.05
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} include image repository management
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:glance}]{\sphinxcrossref{\DUrole{std,std-ref}{Glance}}}}
\\
\hline
\sphinxAtStartPar
vim.07
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support multi\sphinxhyphen{}tenancy
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:multi-tenancy-execution-environment}]{\sphinxcrossref{\DUrole{std,std-ref}{Multi\sphinxhyphen{}Tenancy (execution environment)}}}}
\\
\hline
\sphinxAtStartPar
vim.08
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support resource tagging
&
\sphinxAtStartPar
\sphinxhref{https://specs.openstack.org/openstack/api-wg/guidelines/tags.html}{OpenStack Resource Tags}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Interfaces \& APIs Requirements}
\label{\detokenize{chapters/chapter02:interfaces-apis-requirements}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Interfaces and APIs Requirements}\label{\detokenize{chapters/chapter02:id27}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
int.api.01
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs to access the authentication service
and the associated mandatory features detailed in chapter 5
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:keystone}]{\sphinxcrossref{\DUrole{std,std-ref}{Keystone}}}}
\\
\hline
\sphinxAtStartPar
int.api.02
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs to access the image management
service and the associated mandatory features detailed in chapter 5
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:glance}]{\sphinxcrossref{\DUrole{std,std-ref}{Glance}}}}
\\
\hline
\sphinxAtStartPar
int.api.03
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs to access the block storage
management service and the associated mandatory features detailed in chapter 5
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:cinder}]{\sphinxcrossref{\DUrole{std,std-ref}{Cinder}}}}
\\
\hline
\sphinxAtStartPar
int.api.04
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs to access the object storage
management service and the associated mandatory features detailed in chapter 5
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:swift}]{\sphinxcrossref{\DUrole{std,std-ref}{Swift}}}}
\\
\hline
\sphinxAtStartPar
int.api.05
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs to access the network management
service and the associated mandatory features detailed in chapter 5
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:neutron}]{\sphinxcrossref{\DUrole{std,std-ref}{Neutron}}}}
\\
\hline
\sphinxAtStartPar
int.api.06
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs to access the compute resources
management service and the associated mandatory features detailed in chapter 5
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:nova}]{\sphinxcrossref{\DUrole{std,std-ref}{Nova}}}}
\\
\hline
\sphinxAtStartPar
int.api.07
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide GUI access to tenant facing cloud
platform core services except at Edge/Far Edge clouds
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:horizon}]{\sphinxcrossref{\DUrole{std,std-ref}{Horizon}}}}
\\
\hline
\sphinxAtStartPar
int.api.08
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs needed to discover and manage
Cloud Infrastructure resources
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:placement}]{\sphinxcrossref{\DUrole{std,std-ref}{Placement}}}}
\\
\hline
\sphinxAtStartPar
int.api.09
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} provide APIs to access the orchestration service
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:heat}]{\sphinxcrossref{\DUrole{std,std-ref}{Heat}}}}
\\
\hline
\sphinxAtStartPar
int.api.10
&
\sphinxAtStartPar
API
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} expose the latest version and microversion of the
APIs for the given Anuket OpenStack release for each of the OpenStack core
services
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter05:core-openstack-services-apis}]{\sphinxcrossref{\DUrole{std,std-ref}{Core OpenStack Services APIs}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Tenant Requirements}
\label{\detokenize{chapters/chapter02:tenant-requirements}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Tenant Requirements}\label{\detokenize{chapters/chapter02:id28}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
tnt.gen.01
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support self\sphinxhyphen{}service dashboard (GUI) and
APIs for users to deploy, configure and manage their workloads
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:horizon}]{\sphinxcrossref{\DUrole{std,std-ref}{Horizon}}}} and
{\hyperref[\detokenize{chapters/chapter03:cloud-workload-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud Workload Services}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Operations and LCM}
\label{\detokenize{chapters/chapter02:operations-and-lcm}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{LCM Requirements}\label{\detokenize{chapters/chapter02:id29}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
lcm.gen.01
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support zero downtime of running workloads when
the number of compute hosts and/or the storage capacity is being
expanded or unused capacity is being removed
&
\sphinxAtStartPar
Not detailed
\\
\hline
\sphinxAtStartPar
lcm.adp.02
&
\sphinxAtStartPar
Automated deployment
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} support upgrades of software, provided by the
cloud provider, so that the running workloads are not impacted
(viz., hitless upgrades). Please note that this means that the existing
data plane services should not fail (go down)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:containerised-openstack-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Containerised OpenStack Services}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Assurance Requirements}
\label{\detokenize{chapters/chapter02:assurance-requirements}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Assurance Requirements}\label{\detokenize{chapters/chapter02:id30}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{15}{80}|\X{30}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Specification Reference
\\
\hline
\sphinxAtStartPar
asr.mon.01
&
\sphinxAtStartPar
Integration
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} include integration with various infrastructure
components to support collection of telemetry for assurance monitoring
and network intelligence
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter07:logging-monitoring-and-analytics}]{\sphinxcrossref{\DUrole{std,std-ref}{Logging, Monitoring and Analytics}}}}
\\
\hline
\sphinxAtStartPar
asr.mon.03
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{must} allow for the collection and dissemination of
performance and fault information
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter07:logging-monitoring-and-analytics}]{\sphinxcrossref{\DUrole{std,std-ref}{Logging, Monitoring and Analytics}}}}
\\
\hline
\sphinxAtStartPar
asr.mon.04
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Cloud Infrastructure Network Fabric and Network Operating System
\sphinxstylestrong{must} provide network operational visibility through alarming and
streaming telemetry services for operational management, engineering
planning, troubleshooting, and network performance optimisation
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter07:logging-monitoring-and-analytics}]{\sphinxcrossref{\DUrole{std,std-ref}{Logging, Monitoring and Analytics}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Architecture and OpenStack Recommendations}
\label{\detokenize{chapters/chapter02:architecture-and-openstack-recommendations}}
\sphinxAtStartPar
The requirements listed in this section are optional, and are not
required in order to be deemed a conformant implementation.


\subsubsection{General Recommendations}
\label{\detokenize{chapters/chapter02:general-recommendations}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{General Recommendations}\label{\detokenize{chapters/chapter02:id31}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
gen.cnt.01
&
\sphinxAtStartPar
Cloud nativeness
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} consist of stateless service components.
However, where state is required it must be kept external to the
component
&
\sphinxAtStartPar
OpenStack consists of both stateless and stateful services where the
stateful services utilise a database. For latter see \sphinxhref{https://docs.openstack.org/ha-guide/control-plane-stateful.html}{Configuring the
stateful services}
\\
\hline
\sphinxAtStartPar
gen.cnt.02
&
\sphinxAtStartPar
Cloud nativeness
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} consist of service components implemented
as microservices that are individually dynamically scalable
&\\
\hline
\sphinxAtStartPar
gen.scl.01
&
\sphinxAtStartPar
Scalability
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support policy driven auto\sphinxhyphen{}scaling.
&
\sphinxAtStartPar
This requirement is currently not addressed but will likely be
supported through
\sphinxhref{https://docs.openstack.org/senlin/wallaby/}{Senlin}, cluster
management service
\\
\hline
\sphinxAtStartPar
gen.rsl.02
&
\sphinxAtStartPar
Resiliency
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support resilient OpenStack service
components that are not subject to gen.rsl.01
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Infrastructure Recommendations}
\label{\detokenize{chapters/chapter02:infrastructure-recommendations}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Infrastructure Recommendations}\label{\detokenize{chapters/chapter02:id32}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
inf.com.02
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} include industry standard hardware
management systems at both HW device level (embedded) and HW platform
level (external to device)
&\\
\hline
\sphinxAtStartPar
inf.com.03
&
\sphinxAtStartPar
Compute
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support Symmetric Multiprocessing with
shared memory access as well as Simultaneous Multithreading
&\\
\hline
\sphinxAtStartPar
inf.stg.08
&
\sphinxAtStartPar
Storage
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} allow use of externally provided large
archival storage for its Backup / Restore / Archival needs
&\\
\hline
\sphinxAtStartPar
inf.stg.09
&
\sphinxAtStartPar
Storage
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} make available all non\sphinxhyphen{}host OS / Hypervisor
/ Host systems storage as network\sphinxhyphen{}based Block, File or Object Storage
for tenant/management consumption
&\\
\hline
\sphinxAtStartPar
inf.stg.10
&
\sphinxAtStartPar
Storage
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} provide local Block storage for Instances
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:virtual-storage}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtual Storage}}}}
\\
\hline
\sphinxAtStartPar
inf.nw.04
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support service function chaining
&\\
\hline
\sphinxAtStartPar
inf.nw.06
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support Distributed Virtual Routing (DVR)
to allow compute nodes to route traffic efficiently
&\\
\hline
\sphinxAtStartPar
inf.nw.08
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Cloud Infrastructure Network Fabric \sphinxstylestrong{should} embrace the concepts
of open networking and disaggregation using commodity networking
hardware and disaggregated Network Operating Systems
&\\
\hline
\sphinxAtStartPar
inf.nw.09
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Cloud Infrastructure Network Fabric \sphinxstylestrong{should} embrace open\sphinxhyphen{}based
standards and technologies
&\\
\hline
\sphinxAtStartPar
inf.nw.11
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Cloud Infrastructure Network Fabric \sphinxstylestrong{should} be architected to
provide a standardised, scalable, and repeatable deployment model
across all applicable Cloud Infrastructure sites
&\\
\hline
\sphinxAtStartPar
inf.nw.17
&
\sphinxAtStartPar
Network
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} use dual stack IPv4 and IPv6 for Cloud
Infrastructure internal networks
&\\
\hline
\sphinxAtStartPar
inf.acc.01
&
\sphinxAtStartPar
Acceleration
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support Application Specific Acceleration
(exposed to VNFs)
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:acceleration}]{\sphinxcrossref{\DUrole{std,std-ref}{Acceleration}}}}
\\
\hline
\sphinxAtStartPar
inf.acc.02
&
\sphinxAtStartPar
Acceleration
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support Cloud Infrastructure Acceleration
(such as SmartNICs)
&
\sphinxAtStartPar
\sphinxhref{https://specs.openstack.org/openstack/neutron-specs/specs/stein/neutron-ovs-agent-support-baremetal-with-smart-nic.html}{OpenStack Future \sphinxhyphen{} Specs defined}
\\
\hline
\sphinxAtStartPar
inf.acc.03
&
\sphinxAtStartPar
Acceleration
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{may} rely on on SR\sphinxhyphen{}IOV PCI\sphinxhyphen{}Pass through to provide
acceleration to VNFs
&\\
\hline
\sphinxAtStartPar
inf.img.01
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} make the immutable images available via
location independent means
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:glance}]{\sphinxcrossref{\DUrole{std,std-ref}{Glance}}}}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{VIM Recommendations}
\label{\detokenize{chapters/chapter02:vim-recommendations}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{VIM Recommendations}\label{\detokenize{chapters/chapter02:id33}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
vim.02
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support deployment of OpenStack components
in containers
&
\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter04:containerised-openstack-services}]{\sphinxcrossref{\DUrole{std,std-ref}{Containerised OpenStack Services}}}}
\\
\hline
\sphinxAtStartPar
vim.04
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support Enhanced Platform Awareness (EPA)
only for discovery of infrastructure resource capabilities
&\\
\hline
\sphinxAtStartPar
vim.06
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} allow orchestration solutions to be integrated
with VIM
&\\
\hline
\sphinxAtStartPar
vim.09
&
\sphinxAtStartPar
General
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support horizontal scaling of OpenStack core
services
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Interfaces and APIs Recommendations}
\label{\detokenize{chapters/chapter02:interfaces-and-apis-recommendations}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Interfaces and APIs Recommendations}\label{\detokenize{chapters/chapter02:id34}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
int.acc.01
&
\sphinxAtStartPar
Acceleration
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} provide an open and standard acceleration
interface to VNFs
&\\
\hline
\sphinxAtStartPar
int.acc.02
&
\sphinxAtStartPar
Acceleration
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should not} rely on SR\sphinxhyphen{}IOV PCI\sphinxhyphen{}Pass through for
acceleration interface exposed to VNFs
&
\sphinxAtStartPar
duplicate of inf.acc.03 under “Infrastructure Recommendation”
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Tenant Recommendations}
\label{\detokenize{chapters/chapter02:tenant-recommendations}}
\sphinxAtStartPar
This section is left blank for future use.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Tenant Recommendations}\label{\detokenize{chapters/chapter02:id35}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline&&&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Operations and LCM Recommendations}
\label{\detokenize{chapters/chapter02:operations-and-lcm-recommendations}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{LCM Recommendations}\label{\detokenize{chapters/chapter02:id36}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
lcm.adp.01
&
\sphinxAtStartPar
Automated deployment
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} allow for cookie cutter automated
deployment, configuration, provisioning and management of multiple
Cloud Infrastructure sites
&\\
\hline
\sphinxAtStartPar
lcm.adp.03
&
\sphinxAtStartPar
Automated deployment
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support hitless upgrade of all software
provided by the cloud provider that are not covered by lcm.adp.02.
Whenever hitless upgrades are not feasible, attempt should be made
to minimise the duration and nature of impact
&\\
\hline
\sphinxAtStartPar
lcm.adp.04
&
\sphinxAtStartPar
Automated deployment
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support declarative specifications of
hardware and software assets for automated deployment, configuration,
maintenance and management
&\\
\hline
\sphinxAtStartPar
lcm.adp.05
&
\sphinxAtStartPar
Automated deployment
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support automated process for Deployment
and life\sphinxhyphen{}cycle management of VIM Instances
&\\
\hline
\sphinxAtStartPar
lcm.cid.02
&
\sphinxAtStartPar
CI/CD
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support integrating with CI/CD Toolchain
for Cloud Infrastructure and VIM components Automation
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Assurance Recommendations}
\label{\detokenize{chapters/chapter02:assurance-recommendations}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Assurance Recommendations}\label{\detokenize{chapters/chapter02:id37}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
asr.mon.02
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Architecture \sphinxstylestrong{should} support Network Intelligence capabilities
that allow richer diagnostic capabilities which take as input broader
set of data across the network and from VNF workloads
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Security Recommendations}
\label{\detokenize{chapters/chapter02:security-recommendations}}

\paragraph{System Hardening Recommendations}
\label{\detokenize{chapters/chapter02:system-hardening-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#system-hardening}{System Hardening})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{System Hardening Recommendations}\label{\detokenize{chapters/chapter02:id38}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.gen.011
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
The Cloud Infrastructure \sphinxstylestrong{should} support Read and Write only storage
partitions (write only permission to one or more authorised actors)
&\\
\hline
\sphinxAtStartPar
sec.gen.014
&
\sphinxAtStartPar
Hardening
&
\sphinxAtStartPar
All servers part of Cloud Infrastructure \sphinxstylestrong{should} support measured
boot and an attestation server that monitors the measurements of the
servers
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Platform and Access Recommendations}
\label{\detokenize{chapters/chapter02:platform-and-access-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#platform-and-access}{Platform and Access})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Platform and Access Recommendations}\label{\detokenize{chapters/chapter02:id39}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.sys.014
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{should} use Linux Security Modules such as SELinux to
control access to resources
&\\
\hline
\sphinxAtStartPar
sec.sys.020
&
\sphinxAtStartPar
Access
&
\sphinxAtStartPar
The Cloud Infrastructure architecture \sphinxstylestrong{should} rely on Zero Trust
principles to build a secure by design environment
&
\sphinxAtStartPar
Zero Trust Architecture (ZTA) described in NIST SP 800\sphinxhyphen{}207
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Confidentiality and Integrity Recommendations}
\label{\detokenize{chapters/chapter02:confidentiality-and-integrity-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#confidentiality-and-integrity}{Confidentiality and Integrity})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Confidentiality and Integrity Recommendations}\label{\detokenize{chapters/chapter02:id40}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.ci.002
&
\sphinxAtStartPar
Confidentiality/Integrity
&
\sphinxAtStartPar
The Platform \sphinxstylestrong{should} support self\sphinxhyphen{}encrypting storage devices
&\\
\hline
\sphinxAtStartPar
sec.ci.009
&
\sphinxAtStartPar
Confidentiality/Integrity
&
\sphinxAtStartPar
For sensitive data encryption, the key management service \sphinxstylestrong{should}
leverage a Hardware Security Module to manage and protect cryptographic
keys
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Workload Security Recommendations}
\label{\detokenize{chapters/chapter02:workload-security-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#workload-security}{Workload Security})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Workload Security Recommendations}\label{\detokenize{chapters/chapter02:id41}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.wl.007
&
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
The Operator \sphinxstylestrong{should} implement processes and tools to verify VNF
authenticity and integrity
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Image Security Recommendations}
\label{\detokenize{chapters/chapter02:image-security-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#image-security}{Image Security})

\sphinxAtStartPar
This section is left blank for future use.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Image Security Recommendations}\label{\detokenize{chapters/chapter02:id42}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.img.009
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
CIS Hardened Images \sphinxstylestrong{should} be used whenever possible
&\\
\hline
\sphinxAtStartPar
sec.img.010
&
\sphinxAtStartPar
Image
&
\sphinxAtStartPar
Minimalist base images \sphinxstylestrong{should} be used whenever possible
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Security LCM Recommendations}
\label{\detokenize{chapters/chapter02:security-lcm-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#security-lcm}{Security LCM})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{LCM Security Recommendations}\label{\detokenize{chapters/chapter02:id43}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.lcm.004
&
\sphinxAtStartPar
LCM
&
\sphinxAtStartPar
The Cloud Operator \sphinxstylestrong{should} support automated templated approved
changes; Templated approved changes for automation where available
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Monitoring and Security Audit Recommendations}
\label{\detokenize{chapters/chapter02:monitoring-and-security-audit-recommendations}}
\sphinxAtStartPar
(source
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#monitoring-and-security-audit}{Monitoring and Security Audit})

\sphinxAtStartPar
The Platform is assumed to provide configurable alerting and
notification capability and the operator is assumed to have automated
systems, policies and procedures to act on alerts and notifications in a
timely fashion. In the following the monitoring and logging capabilities
can trigger alerts and notifications for appropriate action.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Monitoring and Security Audit Recommendations}\label{\detokenize{chapters/chapter02:id44}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.mon.014
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Monitoring systems \sphinxstylestrong{should} not impact IaaS, PaaS, and SaaS SLAs
including availability SLAs
&\\
\hline
\sphinxAtStartPar
sec.mon.016
&
\sphinxAtStartPar
Monitoring
&
\sphinxAtStartPar
The Platform Monitoring components \sphinxstylestrong{should} follow security best
practices for auditing, including secure logging and tracing
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Open\sphinxhyphen{}Source Software Security Recommendations}
\label{\detokenize{chapters/chapter02:open-source-software-security-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#open-source-software}{Open Source Software})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Open\sphinxhyphen{}Source Software Security Recommendations}\label{\detokenize{chapters/chapter02:id45}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.oss.005
&
\sphinxAtStartPar
Software
&
\sphinxAtStartPar
A Software Bill of Materials (SBOM) \sphinxstylestrong{should} be provided or build,
and maintained to identify the software components and their origins.
Inventory of software components
&
\sphinxAtStartPar
\sphinxhref{https://www.ntia.gov/SBOM}{NTIA SBOM}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{IaaC security Recommendations}
\label{\detokenize{chapters/chapter02:iaac-security-recommendations}}
\sphinxAtStartPar
(source
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#iaac-secure-design-and-architecture-stage-requirements}{IaaC \sphinxhyphen{} Secure Design and Architecture Stage Requirements})

\sphinxAtStartPar
\sphinxstylestrong{Secure Design and Architecture Stage}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: IaaC Security,
             Design and Architecture Stage}\label{\detokenize{chapters/chapter02:id46}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.arch.001
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Threat Modelling methodologies and tools \sphinxstylestrong{should} be used during the
Secure Design and Architecture stage triggered by Software Feature
Design trigger. Methodology to identify and understand threats
impacting a resource or set of resources
&
\sphinxAtStartPar
It may be done manually or using tools like open source OWASP Threat
Dragon
\\
\hline
\sphinxAtStartPar
sec.arch.002
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Security Control Baseline Assessment \sphinxstylestrong{should} be performed during the
Secure Design and Architecture stage triggered by Software Feature
Design trigger
&
\sphinxAtStartPar
Typically done manually by internal or independent assessors
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Secure Code Stage Recommendations}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: IaaC Security, Secure Code Stage}\label{\detokenize{chapters/chapter02:id47}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.code.002
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
SCA \textendash{} Software Composition Analysis \sphinxstylestrong{should} be applied during
Secure Coding stage triggered by Pull, Clone or Comment trigger.
Security testing that analyses application source code or compiled code
for software components with known vulnerabilities
&
\sphinxAtStartPar
Example: open source OWASP range of tools
\\
\hline
\sphinxAtStartPar
sec.code.003
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Source Code Review \sphinxstylestrong{should} be performed continuously during Secure
Coding stage.
&
\sphinxAtStartPar
Typically done manually.
\\
\hline
\sphinxAtStartPar
sec.code.004
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Integrated SAST via IDE Plugins should be used during Secure Coding
stage triggered by Developer Code trigger. On the local machine:
through the IDE or integrated test suites; triggered on completion of
coding by developer
&\\
\hline
\sphinxAtStartPar
sec.code.005
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
SAST of Source Code Repo \sphinxstylestrong{should} be performed during Secure Coding
stage triggered by Developer Code trigger. Continuous delivery
pre \sphinxhyphen{}deployment: scanning prior to deployment
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Continuous Build, Integration and Testing Stage Recommendations}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: IaaC Security, Continuous Build,
             Integration and Testing Stage}\label{\detokenize{chapters/chapter02:id48}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.bld.001
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
SAST \sphinxhyphen{}Static Application Security Testing \sphinxstylestrong{should} be applied during
the Continuous Build, Integration and Testing stage triggered by Build
and Integrate trigger
&
\sphinxAtStartPar
Example: open source OWASP range of tools.
\\
\hline
\sphinxAtStartPar
sec.bld.002
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
SCA \textendash{} Software Composition Analysis \sphinxstylestrong{should} be applied during the
Continuous Build, Integration and Testing stage triggered by Build and
Integrate trigger
&
\sphinxAtStartPar
Example: open source OWASP range of tools
\\
\hline
\sphinxAtStartPar
sec.bld.004
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
SDAST \textendash{} Dynamic Application Security Testing \sphinxstylestrong{should} be applied
during the Continuous Build, Integration and Testing stage triggered
by Stage \& Test trigger. Security testing that analyses a running
application by exercising application functionality and detecting
vulnerabilities based on application behaviour and response
&
\sphinxAtStartPar
Example: OWASP ZAP
\\
\hline
\sphinxAtStartPar
sec.bld.005
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Fuzzing \sphinxstylestrong{should} be applied during the Continuous Build, Integration
and testing stage triggered by Stage \& Test trigger. Fuzzing or fuzz
testing is an automated software testing technique that involves
providing invalid, unexpected, or random data as inputs to a computer
program
&
\sphinxAtStartPar
Example: GitLab Open Sources Protocol Fuzzer Community Edition
\\
\hline
\sphinxAtStartPar
sec.bld.006
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
IAST \textendash{} Interactive Application Security Testing \sphinxstylestrong{should} be applied
during the Continuous Build, Integration and Testing stage triggered by
Stage \& Test trigger. Software component deployed with an application
that assesses application behaviour and detects presence of
vulnerabilities on an application being exercised in realistic testing
scenarios
&
\sphinxAtStartPar
Example: Contrast Community Edition
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Continuous Delivery and Deployment Stage Recommendations}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: IaaC Security, Continuous
             Delivery and Deployment Stage}\label{\detokenize{chapters/chapter02:id49}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.del.003
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Artifact and Image Repository Scan \sphinxstylestrong{should} be continuously applied
during the Continuous Delivery and Deployment stage
&
\sphinxAtStartPar
Example: GitLab uses the open source Clair engine for container
scanning
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Runtime Defence and Monitoring Recommendations}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Reference Model Requirements: Iaac Security, Runtime Defence
             and Monitoring Stage}\label{\detokenize{chapters/chapter02:id50}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.run.002
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
RASP \textendash{} Runtime Application Self\sphinxhyphen{}Protection \sphinxstylestrong{should} be continuously
applied during the Runtime Defence and Monitoring stage. Security
technology deployed within the target application in production for
detecting, alerting, and blocking attacks
&\\
\hline
\sphinxAtStartPar
sec.run.003
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Application testing and Fuzzing \sphinxstylestrong{should} be continuously applied
during the Runtime Defence and Monitoring stage. Fuzzing or fuzz
testing is an automated software testing technique that involves
providing invalid, unexpected, or random data as inputs to a computer
program
&
\sphinxAtStartPar
Example: GitLab Open Sources Protocol Fuzzer Community Edition
\\
\hline
\sphinxAtStartPar
sec.run.004
&
\sphinxAtStartPar
IaaC
&
\sphinxAtStartPar
Penetration Testing \sphinxstylestrong{should} be continuously applied during the
Runtime Defence and Monitoring stage
&
\sphinxAtStartPar
Typically done manually
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\paragraph{Compliance with Standards Recommendations}
\label{\detokenize{chapters/chapter02:compliance-with-standards-recommendations}}
\sphinxAtStartPar
(source \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter07.html\#compliance-with-standards}{Compliance with Standards})


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Compliance with Security Recommendations}\label{\detokenize{chapters/chapter02:id51}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{100}|\X{15}{100}|\X{40}{100}|\X{30}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference
&\sphinxstyletheadfamily 
\sphinxAtStartPar
sub\sphinxhyphen{}category
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
sec.std.001
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator \sphinxstylestrong{should} comply with \sphinxhref{https://www.cisecurity.org/}{Center for Internet Security
CIS Controls}
&\\
\hline
\sphinxAtStartPar
sec.std.002
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator, Platform and Workloads \sphinxstylestrong{should} follow the
guidance in the CSA Security Guidance for Critical Areas of Focus in
Cloud Computing (latest version)\sphinxhyphen{} CSA, \sphinxhref{https://cloudsecurityalliance.org/}{Cloud Security Alliance}
&\\
\hline
\sphinxAtStartPar
sec.std.003
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Platform and Workloads \sphinxstylestrong{should} follow the guidance in the
\sphinxhref{https://github.com/OWASP/CheatSheetSeries}{OWASP Cheat Sheet Series (OCSS)} \sphinxhyphen{} OWASP, \sphinxhref{https://www.owasp.org}{Open Web
Application Security Project}
&\\
\hline
\sphinxAtStartPar
sec.std.004
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator, Platform and Workloads \sphinxstylestrong{should} ensure that their
code is not vulnerable to the \sphinxhref{https://owasp.org/www-project-top-ten/}{OWASP Top Ten Security Risks}
&\\
\hline
\sphinxAtStartPar
sec.std.005
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator, Platform and Workloads \sphinxstylestrong{should} strive to improve
their maturity on the \sphinxhref{https://owaspsamm.org/blog/2019/12/20/version2-community-release/}{OWASP Software Maturity Model (SAMM)}
&\\
\hline
\sphinxAtStartPar
sec.std.006
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator, Platform and Workloads should utilise the
\sphinxhref{https://github.com/OWASP/wstg/tree/master/document}{OWASP Web Security Testing Guide}
&\\
\hline
\sphinxAtStartPar
sec.std.007
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator, and Platform \sphinxstylestrong{should} satisfy the requirements
for Information Management Systems specified in \sphinxhref{https://www.iso.org/obp/ui/\#iso:std:iso-iec:27001:ed-2:v1:en}{ISO/IEC 27001};
ISO/IEC 27001 is the international Standard for best\sphinxhyphen{}practice
information security management systems (ISMSs)
&\\
\hline
\sphinxAtStartPar
sec.std.008
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator, and Platform \sphinxstylestrong{should} implement the Code of
practice for Security Controls specified
\sphinxhref{https://www.iso.org/obp/ui/\#iso:std:iso-iec:27002:ed-2:v1:en}{ISO/IEC 27002:2013 (or latest)}
&\\
\hline
\sphinxAtStartPar
sec.std.009
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator, and Platform \sphinxstylestrong{should} implement the
\sphinxhref{https://www.iso.org/obp/ui/\#iso:std:iso-iec:27032:ed-1:v1:en}{ISO/IEC 27032:2012 (or latest) Guidelines for Cybersecurity techniques};
ISO/IEC 27032 is the international Standard focusing explicitly on
cybersecurity
&\\
\hline
\sphinxAtStartPar
sec.std.010
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator \sphinxstylestrong{should} conform to the ISO/IEC 27035 standard for
incidence management; ISO/IEC 27035 is the international Standard for
incident management
&\\
\hline
\sphinxAtStartPar
sec.std.011
&
\sphinxAtStartPar
Standards
&
\sphinxAtStartPar
The Cloud Operator \sphinxstylestrong{should} conform to the ISO/IEC 27031 standard for
business continuity; ISO/IEC 27031 \sphinxhyphen{} ISO/IEC 27031 is the international
Standard for ICT readiness for business continuity
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\section{Cloud Infrastructure Architecture \sphinxhyphen{} OpenStack}
\label{\detokenize{chapters/chapter03:cloud-infrastructure-architecture-openstack}}\label{\detokenize{chapters/chapter03::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter03:introduction}}
\sphinxAtStartPar
This Reference Architecture (RA\sphinxhyphen{}1) aims to provide an OpenStack
distribution agnostic reference architecture that includes the Network
Function Virtualisation Infrastructure (NFVI) and Virtual Infrastructure
Manager (VIM). The different OpenStack distributions, without the not
up\sphinxhyphen{}streamed vendor specific enhancements, are assumed to be Anuket
conformant. This Reference Architecture allows operators to provide a
common OpenStack\sphinxhyphen{}based architecture for any Anuket compliant VNF to be
deployed and operated as expected. The purpose of this chapter is to
outline all the components required to provide the Cloud Infrastructure
(NFVI and the VIM) in a consistent and reliable way.

\sphinxAtStartPar
\sphinxhref{http://docs.openstack.org}{OpenStack} is already very well
documented and, hence, this document will describe the specific
OpenStack services and features, Cloud Infrastructure features and how
we expect them to be implemented.

\sphinxAtStartPar
This reference architecture provides optionality in terms of pluggable
components such as SDN, hardware acceleration and support tools.

\sphinxAtStartPar
The Cloud Infrastructure layer includes the physical infrastructure
which is then offered as virtual resources via a hypervisor. The VIM is
the OpenStack Wallaby release.

\sphinxAtStartPar
This chapter is organised as follows:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Consumable Infrastructure Resources and Services: these are
infrastructure services and resources being exposed northbound for
consumption
\begin{itemize}
\item {} 
\sphinxAtStartPar
Multi\sphinxhyphen{}tenancy with quotas
\begin{itemize}
\item {} 
\sphinxAtStartPar
Virtual compute: vCPU / vRAM

\item {} 
\sphinxAtStartPar
Virtual storage: Ephemeral, Persistent and Image

\item {} 
\sphinxAtStartPar
Virtual networking \textendash{} neutron standalone: network plugin,
virtual switch, accelerator features

\item {} 
\sphinxAtStartPar
Virtual networking \textendash{} 3rd party SDN solution

\item {} 
\sphinxAtStartPar
Additional network services: Firewall, DC Gateway

\end{itemize}

\end{itemize}

\item {} 
\sphinxAtStartPar
Cloud Infrastructure Management Software (VIM): is how we manage the
Consumable Infrastructure Resources and Services
\begin{itemize}
\item {} 
\sphinxAtStartPar
VIM Core services (keystone, cinder, nova, neutron etc.)
\begin{itemize}
\item {} 
\sphinxAtStartPar
Tenant Separation

\item {} 
\sphinxAtStartPar
Host aggregates providing resource pooling

\item {} 
\sphinxAtStartPar
Flavor* management

\end{itemize}

\end{itemize}

\item {} 
\sphinxAtStartPar
Underlying Resources: are what provides the resources that allow the
Consumable Infrastructure Resources and Services to be created and
managed by the Cloud Infrastructure Management Software (VIM).
\begin{itemize}
\item {} 
\sphinxAtStartPar
Virtualisation

\item {} 
\sphinxAtStartPar
Physical infrastructure
\begin{itemize}
\item {} 
\sphinxAtStartPar
Compute

\item {} 
\sphinxAtStartPar
Network: Spine/Leaf; East/West and North/South traffic

\item {} 
\sphinxAtStartPar
Storage

\end{itemize}

\end{itemize}

\end{itemize}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Please note “flavours” is used in the Reference Model and shall
continue to be used in the context of specifying the geometry of
the virtual resources. The term “flavor” is used in this document
in the OpenStack context including when specifying configurations;
the OpenStack term flavor includes the profile configuration
information as “extra specs”.

\end{itemize}


\subsection{Consumable Infrastructure Resources and Services}
\label{\detokenize{chapters/chapter03:consumable-infrastructure-resources-and-services}}
\sphinxAtStartPar
This section will describe the different services that are exposed for
the VNF consumption within the execution zone:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Tenants: to provide isolated environments

\item {} 
\sphinxAtStartPar
Virtual Compute: to provide computing resources

\item {} 
\sphinxAtStartPar
Virtual Storage: to provide storage capacity and performance

\item {} 
\sphinxAtStartPar
Virtual networking: to provide connectivity within the Cloud
Infrastructure and with external networks

\end{itemize}


\subsubsection{Multi\sphinxhyphen{}Tenancy (execution environment)}
\label{\detokenize{chapters/chapter03:multi-tenancy-execution-environment}}
\sphinxAtStartPar
The multi tenancy service permits hosting of several VNF projects with
the assurance of isolated environments for each project. Tenants or
confusingly “Projects” in OpenStack are isolated environments that
enable workloads to be logically separated from each other with:
\begin{itemize}
\item {} 
\sphinxAtStartPar
differentiated set of associated users

\item {} 
\sphinxAtStartPar
role\sphinxhyphen{}based access of two levels \textendash{} admin or member (see {\hyperref[\detokenize{chapters/chapter06:rbac}]{\sphinxcrossref{\DUrole{std,std-ref}{RBAC}}}}).

\item {} 
\sphinxAtStartPar
quota system to provide maximum resources that can be consumed.

\end{itemize}

\sphinxAtStartPar
This RA does not intend to restrict how workloads are distributed across
tenants.


\subsubsection{Virtual Compute (vCPU and vRAM)}
\label{\detokenize{chapters/chapter03:virtual-compute-vcpu-and-vram}}
\sphinxAtStartPar
The virtual compute resources (vCPU and vRAM) used by the VNFs behave
like their physical counterparts. A physical core is an actual processor
and can support multiple vCPUs through Simultaneous Multithreading (SMT)
and CPU overbooking. With no overbooking and SMT of 2 (2 threads per
core), each core can support 2 vCPUs. With the same SMT of 2 and
overbooking factor of 4, each core can support 8 vCPUs. The performance
of a vCPU can be affected by various configurations such as CPU pinning,
NUMA alignment, and SMT.

\sphinxAtStartPar
The configuration of the virtual resources will depend on the software
and hardware profiles and the flavour (resource sizing) needed to host
VNF components. Profiles are defined in the
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter02.html\#profiles-profile-extensions-flavours}{Profiles, Profile Extensions \& Flavours}.


\subsubsection{Virtual Storage}
\label{\detokenize{chapters/chapter03:virtual-storage}}
\sphinxAtStartPar
The Reference Model
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter03.html\#storage-for-tenant-consumption}{Storage for Tenant Consumption}
details consumption models for tenants: Platform native,
object storage, shared file storage and archival.
The choice of a solution will depend on the storage use case needs.

\sphinxAtStartPar
The two storage services offered by Cloud Infrastructure are:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Persistent storage

\item {} 
\sphinxAtStartPar
Ephemeral storage

\end{itemize}

\sphinxAtStartPar
The OpenStack services, Cinder for block storage and Swift for Object
Storage, are discussed below in Section 3.3 “Cloud Infrastructure
Management Software (VIM)”.

\sphinxAtStartPar
Ephemeral data is typically stored on the compute host’s local disks,
in the form of a file system as part of the provisioning.
This storage is volatile, it is deleted when instances are stopped.
In environments that support live instance migration between
compute hosts, the ephemeral data would need to be
stored in a storage system shared between the compute hosts such as on
persistent block or object storage.

\sphinxAtStartPar
Three types of persistent data storage are supported in OpenStack:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Block storage

\item {} 
\sphinxAtStartPar
Object storage

\item {} 
\sphinxAtStartPar
Shared file systems storage

\end{itemize}

\sphinxAtStartPar
The \sphinxhref{https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html\#table-openstack-storage}{OpenStack Storage Table}
explains the differences between the storage types and typical use
cases.

\sphinxAtStartPar
Block storage is dedicated to persistent data. Data is stored
in the form of volumes. Block storage is managed by OpenStack
Cinder service and storage Backends. \sphinxhref{https://docs.openstack.org/cinder/latest/reference/support-matrix.html}{OpenStack compatible
storage backend drivers table}
lists the storage backends compatible with Cinder and their capabilities.

\sphinxAtStartPar
The Object storage is a persistent data storage, not attached to an instance.
Data is accessed via API. Object storage is managed by OpenStack Swift.

\sphinxAtStartPar
Images are persistent data, stored using the OpenStack Glance service.

\sphinxAtStartPar
Cinder, Swift, and Glance services are discussed in the section
{\hyperref[\detokenize{chapters/chapter04:virtualised-infrastructure-manager-vim}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtualised Infrastructure Manager (VIM)}}}}.


\subsubsection{Virtual Networking Neutron standalone}
\label{\detokenize{chapters/chapter03:virtual-networking-neutron-standalone}}
\sphinxAtStartPar
Neutron is an OpenStack project that provides “network connectivity as a
service” between interface devices (e.g., vNICs) managed by other
OpenStack services (e.g., Nova). Neutron allows users to create
networks, subnets, ports, routers etc. Neutron also facilitates traffic
isolation between different subnets \sphinxhyphen{} within as well as across
project(s) by using different type drivers/mechanism drivers that use
VLANs, VxLANs, GRE (Generic Routing Encapsulation) tunnels etc. For
Neutron API consumer, this is abstracted and provided by Neutron.
Multiple network segments are supported by Neutron via ML2 plugins to
simultaneously utilise variety of layer 2 networking technologies like
VLAN, VxLAN, GRE etc. Neutron also allows to create routers to connect
layer 2 networks via “neutron\sphinxhyphen{}l3\sphinxhyphen{}agent”. In addition, floating IP
support is also provided that allows a project VM to be accessed using a
public IP.


\subsubsection{Virtual Networking \textendash{} 3rd party SDN solution}
\label{\detokenize{chapters/chapter03:virtual-networking-3rd-party-sdn-solution}}
\sphinxAtStartPar
SDN (Software Defined Networking) controllers separate control and data
(user) plane functions where the control plane programmatically
configures and controls all network data path elements via open APIs.
Open Networking Forum (ONF) defines SDN as “Software\sphinxhyphen{}Defined Networking
(SDN) is an emerging architecture that is dynamic, manageable,
cost\sphinxhyphen{}effective, and adaptable, making it ideal for the high\sphinxhyphen{}bandwidth,
dynamic nature of today’s applications. This architecture decouples the
network control and forwarding functions enabling the network control to
become directly programmable and the underlying infrastructure to be
abstracted for applications and network services.”

\sphinxAtStartPar
The key messages of the SDN definition are:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Decoupling of control and forwarding functions into control plane and
data plane

\item {} 
\sphinxAtStartPar
Networking capabilities that can be instantiated, deployed,
configured and managed like software. Network control is programmable
and supports dynamic, manageable and adaptable networking.

\item {} 
\sphinxAtStartPar
Support for both overlay and underlay networking

\end{itemize}

\sphinxAtStartPar
OpenStack Neutron supports open APIs and a pluggable backend where
different plugins can be incorporated in the neutron\sphinxhyphen{}server.

\sphinxAtStartPar
Plugins for various SDN controllers include either the standard ML\sphinxhyphen{}2
plugin or specific monolithic plugins. Neutron supports both core
plugins that deal with L2 connectivity and IP address management, and
service plugins that support services such as L3 routing, Load
Balancers, Firewalls, etc.

\sphinxAtStartPar
Below we will explore an example of an SDN controller from LFN projects,
that can be integrated with a Neutron plugin, to help overcome a number
of shortcomings of the vanilla Neutron and provide many needed features
that can be consumed by VNF/CNF.


\paragraph{Tungsten Fabric (SDN Controller)}
\label{\detokenize{chapters/chapter03:tungsten-fabric-sdn-controller}}
\sphinxAtStartPar
\sphinxhref{https://tungsten.io/}{Tungsten Fabric}, an open source SDN in Linux
Foundation Networking, offers neutron networking through ML2 based
plugin, additionally it supports advanced networking features beyond
basic neutron networking via monolithic plugin. It also supports the
same advanced networking features via CNI plugin in Kubernetes. Hence,
it works as a multi\sphinxhyphen{}stack SDN to support VMs, containers, and baremetal
workloads. It provides separation of control plane functions and data
plane functions with its two components:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Tungsten Fabric Controller\textendash{} a set of software services that maintains
a model of networks and network policies, typically running on
several servers for high availability

\item {} 
\sphinxAtStartPar
Tungsten Fabric vRouter\textendash{} installed in each host that runs workloads
(virtual machines or containers), the vRouter performs packet
forwarding and enforces network and security policies

\end{itemize}

\sphinxAtStartPar
It is based on proven, standards\sphinxhyphen{}based networking technologies but
repurposed to work with virtualised workloads and cloud automation in
data centres that can range from large scale enterprise data centres to
much smaller telco DC (aka POPs) . It provides many enhanced features
over the native networking implementations of orchestrators, including:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Highly scalable, multi\sphinxhyphen{}tenant networking

\item {} 
\sphinxAtStartPar
Multi\sphinxhyphen{}tenant IP address management

\item {} 
\sphinxAtStartPar
DHCP, ARP proxies to avoid flooding into networks

\item {} 
\sphinxAtStartPar
Efficient edge replication for broadcast and multicast traffic

\item {} 
\sphinxAtStartPar
Local, per\sphinxhyphen{}tenant DNS resolution

\item {} 
\sphinxAtStartPar
Distributed firewall with access control lists

\item {} 
\sphinxAtStartPar
Application\sphinxhyphen{}based security policies

\item {} 
\sphinxAtStartPar
Distributed load balancing across hosts

\item {} 
\sphinxAtStartPar
Network address translation (1:1 floating IPs and distributed SNAT)

\item {} 
\sphinxAtStartPar
Service chaining with virtual network functions

\item {} 
\sphinxAtStartPar
Dual stack IPv4 and IPv6

\item {} 
\sphinxAtStartPar
BGP peering with gateway routers

\item {} 
\sphinxAtStartPar
BGP as a Service (BGPaaS) for distribution of routes between
privately managed customer networks and service provider networks

\end{itemize}

\sphinxAtStartPar
Based on the network layering concepts introduced in the Reference
Model Section \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter03.html\#network}{Network}, the
Tungsten Fabric Controller performs functions of both the SDN underlay
(SDNu) and overlay (SDNo) controllers.

\sphinxAtStartPar
The SDN controller exposes a NB API that can be consumed by ETSI MANO
for VNF/CNF onboarding, network service onboarding and dynamic service
function chaining.


\subsubsection{Acceleration}
\label{\detokenize{chapters/chapter03:acceleration}}
\sphinxAtStartPar
Acceleration deals with both hardware and software accelerations.
Hardware acceleration is the use of specialised hardware to perform some
function faster than is possible by executing the same function on a
general\sphinxhyphen{}purpose CPU or on a traditional networking (or other I/O) device
(e.g., NIC, switch, storage controller, etc.). The hardware accelerator
covers the options for ASICs, SmartNIC, FPGAs, GPU etc. to offload the
main CPU, and to accelerate workload performance. Cloud Infrastructure
should manage the accelerators by plugins and provide the acceleration
capabilities to VNFs.

\sphinxAtStartPar
With the acceleration abstraction layer defined, hardware accelerators
as well as software accelerators can be abstracted as a set of
acceleration functions (or acceleration capabilities) which exposes a
common API to either the VNF or the host.


\subsection{Virtualised Infrastructure Manager (VIM)}
\label{\detokenize{chapters/chapter03:virtualised-infrastructure-manager-vim}}
\sphinxAtStartPar
The Cloud Infrastructure Management Software (VIM) provides the services
for the management of Consumable Resources/Services.


\subsubsection{VIM Core services}
\label{\detokenize{chapters/chapter03:vim-core-services}}
\sphinxAtStartPar
OpenStack is a complex, multi\sphinxhyphen{}project framework, and so we will
initially focus on the core services required to provide
Infrastructure\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}Service (IaaS) as this is generally all that is
required for Cloud Infrastructure/VIM use cases. Other components are
optional and provide functionality above and beyond Cloud
Infrastructure/VIM requirements.

\sphinxAtStartPar
The architecture consists of the core services shown in the
\hyperref[\detokenize{chapters/chapter03:openstack-core-services}]{Fig.\@ \ref{\detokenize{chapters/chapter03:openstack-core-services}}};
Ironic is an optional OpenStack service needed only for bare\sphinxhyphen{}metal
containers. The rest of this document will address the specific Anuket
conformant implementation requirements and recommendations for the core
services.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch03-Core-Cloud-Infra-Services}.png}
\caption{OpenStack Core Services}\label{\detokenize{chapters/chapter03:openstack-core-services}}\end{figure}

\sphinxAtStartPar
We will refer to the functions above as falling into the following
categories to avoid any confusion with other terminology that may be
used:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Foundation node

\item {} 
\sphinxAtStartPar
Control nodes

\item {} 
\sphinxAtStartPar
Compute nodes

\item {} 
\sphinxAtStartPar
Other supporting service nodes e.g. network, shared storage, logging,
monitoring and alerting.

\end{itemize}

\sphinxAtStartPar
Each deployment of OpenStack should be a unique cloud with its own API
endpoint. Sharing underlying cloud resources across OpenStack clouds is
not recommended.


\paragraph{OpenStack Services Topology}
\label{\detokenize{chapters/chapter03:openstack-services-topology}}
\sphinxAtStartPar
OpenStack software services are distributed over 2 planes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Control Plane that hosts all Control and Management services

\item {} 
\sphinxAtStartPar
Data Plane (a.k.a. User plane) that provides physical and virtual
resources (compute, storage and networking) for the actual virtual
workloads to run.

\end{itemize}

\sphinxAtStartPar
The architecture based on OpenStack technology relies on different types
of nodes associated with specific roles:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Controller node types with control and management services, which
include VIM functionalities

\item {} 
\sphinxAtStartPar
Compute node types running workloads

\item {} 
\sphinxAtStartPar
Network node types offering L3 connectivity

\item {} 
\sphinxAtStartPar
Storage node types offering external attached storage (block, object,
flat files)

\end{itemize}

\sphinxAtStartPar
The data plane consists of the compute nodes. It is typical to consider
the other node types to be part of the control plane.
\hyperref[\detokenize{chapters/chapter03:id1}]{Fig.\@ \ref{\detokenize{chapters/chapter03:id1}}} depicts
the 4 types of nodes constitutive of the Infrastructure: control,
compute, network and storage nodes.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch03-OpenStack-Services-Topology}.png}
\caption{OpenStack Services Topology}\label{\detokenize{chapters/chapter03:id1}}\end{figure}

\sphinxAtStartPar
Deployments can be structured using the distribution of services amongst
the 4 node types as depicted in \hyperref[\detokenize{chapters/chapter03:id1}]{Fig.\@ \ref{\detokenize{chapters/chapter03:id1}}},
but depending on workloads requirements, OpenStack services can also be
hosted on the same nodes. For instance, services related to Controller,
network and storage roles can be hosted on controller nodes.


\paragraph{Foundation Services}
\label{\detokenize{chapters/chapter03:foundation-services}}
\sphinxAtStartPar
To build and lifecycle manage an OpenStack cloud, it is typically
necessary to deploy a server or virtual machine as a deployment node or
foundation node.

\sphinxAtStartPar
This function must be able to manage the bare\sphinxhyphen{}metal provisioning of the
hardware resources but since this does not affect cloud execution it can
be detached from the OpenStack cloud and an operator can select their
own tooling as they wish. Functional requirements of this node include:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Build the cloud (control, compute, storage, network hardware
resources)

\item {} 
\sphinxAtStartPar
Patch management / upgrades / change management

\item {} 
\sphinxAtStartPar
Grow / Shrink resources

\end{itemize}


\paragraph{Cloud Controller Services}
\label{\detokenize{chapters/chapter03:cloud-controller-services}}
\sphinxAtStartPar
The following OpenStack components are deployed on the Infrastructure.
Some of them will be only deployed on control hosts and some of them
will be deployed within both control and compute hosts. The table below
also maps the OpenStack core services to the Reference Model (RM)
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter03.html\#virtual-infrastructure-manager}{Virtual Infrastructure Manager}.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{OpenStack components deployment}\label{\detokenize{chapters/chapter03:id2}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{80}|\X{10}{80}|\X{20}{80}|\X{10}{80}|\X{10}{80}|\X{10}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
RM Management Software
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Service
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Required / Optional
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Deployed on Controller Nodes
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Deployed on Compute Nodes
\\
\hline
\sphinxAtStartPar
Identity Management (Additional Management Functions) + Catalogue
&
\sphinxAtStartPar
Keystone
&
\sphinxAtStartPar
the authentication service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
Storage Resources Manager
&
\sphinxAtStartPar
Glance
&
\sphinxAtStartPar
the image management service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
Storage Resources Manager
&
\sphinxAtStartPar
Cinder
&
\sphinxAtStartPar
the block storage management service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
Storage Resources Manager
&
\sphinxAtStartPar
Swift
&
\sphinxAtStartPar
the Object storage management service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
Network Resources Manager
&
\sphinxAtStartPar
Neutron
&
\sphinxAtStartPar
the network management service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
Compute Resources Inventory
&
\sphinxAtStartPar
Placement
&
\sphinxAtStartPar
resource provider inventory service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
Compute Resources Manager + Scheduler
&
\sphinxAtStartPar
Nova
&
\sphinxAtStartPar
the compute resources management service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
Compute Resources Manager
&
\sphinxAtStartPar
Ironic
&
\sphinxAtStartPar
the Bare Metal Provisioning service
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
X
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
(Tool that utilises APIs)
&
\sphinxAtStartPar
Heat
&
\sphinxAtStartPar
the orchestration service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
UI
&
\sphinxAtStartPar
Horizon
&
\sphinxAtStartPar
the WEB UI service
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
Key Manager
&
\sphinxAtStartPar
Barbican
&
\sphinxAtStartPar
the secret data management service
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
X
&\\
\hline
\sphinxAtStartPar
Acceleration Resources Manager
&
\sphinxAtStartPar
Cyborg
&
\sphinxAtStartPar
the acceleration resources and their life cycle management
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
X
&
\sphinxAtStartPar
X
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
All components must be deployed within a high available architecture
that can withstand at least a single node failure and respects the
anti\sphinxhyphen{}affinity rules for the location of the services (i.e. instances of
a same service must run on different nodes).

\sphinxAtStartPar
The services can be containerised or VM hosted as long as they provide
the high availability principles described above.

\sphinxAtStartPar
The APIs for these OpenStack services are listed in
{\hyperref[\detokenize{chapters/chapter05:interfaces-and-apis}]{\sphinxcrossref{\DUrole{std,std-ref}{Interfaces and APIs}}}}.


\paragraph{Cloud Workload Services}
\label{\detokenize{chapters/chapter03:cloud-workload-services}}
\sphinxAtStartPar
This section describes the core set of services and service components
needed to run workloads; instances (such as VMs), their networks and
storage are referred to as the “Compute Node Services” (a.k.a. user or
data plane services). Contrast this with the Controller nodes which host
OpenStack services used for cloud administration and management. The
Compute Node Services include virtualisation, hypervisor instance
creation/deletion, networking and storage services; some of these
activities include RabbitMQ queues in the control plane including the
scheduling, networking and cinder volume creation/attachment.
\begin{itemize}
\item {} 
\sphinxAtStartPar
Compute, Storage, Network services:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Nova Compute service: nova\sphinxhyphen{}compute (creating/deleting servers
(a.k.a. instances))

\item {} 
\sphinxAtStartPar
Neutron Networking service: neutron\sphinxhyphen{}l2\sphinxhyphen{}agent (manage local Open
vSwitch (OVS) configuration), VXLAN

\item {} 
\sphinxAtStartPar
Local Storage (Ephemeral, Root, etc.)

\item {} 
\sphinxAtStartPar
Attached Storage (using Local drivers)

\end{itemize}

\end{itemize}


\subsubsection{Tenant Isolation}
\label{\detokenize{chapters/chapter03:tenant-isolation}}
\sphinxAtStartPar
In Keystone v1 and v2 (both deprecated), the term “tenant” was used in
OpenStack. With Keystone v3, the term “project” got adopted and both the
terms became interchangeable. According to \sphinxhref{https://docs.openstack.org/doc-contrib-guide/common/glossary.html}{OpenStack
glossary},
Projects represent the base unit of resources (compute, storage and
network) in OpenStack, in that all assigned resources in OpenStack are
owned by a specific project. OpenStack offers multi\sphinxhyphen{}tenancy by means of
resource (compute, network and storage)separation via projects.
OpenStack offers ways to share virtual resources between projects while
maintaining logical separation. As an example, traffic separation is
provided by creating different VLAN ids for neutron networks of
different projects. As another example, if host separation is needed,
nova scheduler offers AggregateMultiTenancyIsolation scheduler filter to
separate projects in host aggregates. Thus, if a host in an aggregate is
configured for a particular project, only the instances from that
project are placed on the host. Overall, tenant isolation ensures that
the resources of a project are not affected by resources of another
project.

\sphinxAtStartPar
This document uses the term “project” when referring to OpenStack
services and “tenant” (RM Section
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter03.html\#virtual-resources}{Virtual Resources})
to represent an independently manageable logical pool of resources.


\subsubsection{Cloud partitioning: Host Aggregates, Availability Zones}
\label{\detokenize{chapters/chapter03:cloud-partitioning-host-aggregates-availability-zones}}
\sphinxAtStartPar
Cloud administrators can partition the hosts within an OpenStack cloud
using Host Aggregates and Availability Zones.

\sphinxAtStartPar
A Host Aggregate is a group of hosts (compute nodes) with specific
characteristics and with the same specifications, software and/or
hardware properties. Example would be a Host Aggregate created for
specific hardware or performance characteristics. The administrator
assigns key\sphinxhyphen{}value pairs to Host Aggregates, these are then used when
scheduling VMs. A host can belong to multiple Host Aggregates. Host
Aggregates are not explicitly exposed to tenants.

\sphinxAtStartPar
Availability Zones (AZs) rely on Host Aggregates and make the
partitioning visible to tenants. They are defined by attaching specific
metadata information to an aggregate, making the aggregate visible for
tenants. Hosts can only be in a single Availability Zone. By default a
host is part of a default Availability Zone, even if it doesn’t belong
to an aggregate. Availability Zones can be used to provide resiliency
and fault tolerance for workloads deployments, for example by means of
physical hosting distribution of Compute Nodes in separate racks with
separate power supply and eventually in different rooms. They permit
rolling upgrades \textendash{} an AZ at a time upgrade with enough time between AZ
upgrades to allow recovery of tenant workloads on the upgraded AZ. AZs
can also be used to seggregate workloads.

\sphinxAtStartPar
An over use of Host Aggregates and Availability Zones can result in a
granular partition of the cloud and, hence, operational complexities and
inefficiencies.


\subsubsection{Flavor management}
\label{\detokenize{chapters/chapter03:flavor-management}}
\sphinxAtStartPar
In OpenStack a flavor defines the compute, memory, and storage capacity
of nova instances. When instances are spawned, they are mapped to
flavors which define the available hardware configuration for them. For
simplicity, operators may create named flavors specifying both the
sizing and the
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html}{\DUrole{xref,std,std-doc}{software and hardware profile configurations}}.


\subsection{Underlying Resources}
\label{\detokenize{chapters/chapter03:underlying-resources}}
\sphinxAtStartPar
The number of Compute nodes (for workloads) determines the load on the
controller nodes and networking traffic and, hence, the number of
controller nodes needed in the OpenStack cloud; the number of controller
nodes required is determined on the load placed on these controller
nodes and the need for High availability and quorum requires at least 3
instances of many of the services on these controller nodes.


\subsubsection{Virtualisation}
\label{\detokenize{chapters/chapter03:virtualisation}}
\sphinxAtStartPar
Virtualisation is a technology that enables a guest Operating System
(OS) to be abstracted from the underlying hardware and software. This
allows to run multiple Virtual Machines(VMs) on the same hardware. Each
such VMs have their own OS and are isolated from each other
i.e. application running on one VM does not have the access to resources
of another VM. Such virtualisation is supported by various hypervisors
available as open\sphinxhyphen{}source (KVM, Xen, etc.) as well as commercial
(Hyper\sphinxhyphen{}V, Citrix XenServer, etc.). Selecting a hypervisor depends on the
workload needs and the features provided by various hypervisors as
illustrated in Hypervisor \sphinxhref{https://docs.openstack.org/nova/latest/user/support-matrix.html}{Feature Support
Matrix}.
OpenStack (Nova) allows the use of various hypervisors within a single
installation by means of scheduler filters like ComputeFilter,
ImagePropertiesFilter etc.

\sphinxAtStartPar
Virtualisation Services: The OpenStack nova\sphinxhyphen{}compute service supports
multiple hypervisors natively or through libvirt. The preferred
supported hypervisor in this Reference Architecture is KVM.

\sphinxAtStartPar
\sphinxstyleemphasis{Note}: Other hypervisors (such as ESXi) can also be supported as long
as they can interoperate with other OpenStack components (e.g., those
listed in this Reference Architecture) using standard interfaces and
APIs as specified in Chapter 5.


\subsubsection{Physical Infrastructure}
\label{\detokenize{chapters/chapter03:physical-infrastructure}}
\sphinxAtStartPar
The aim is to specify the requirements on deploying the VIM, from ground
up (in a shipping container), and what resources are required of the DC
(Data Centre).
\begin{itemize}
\item {} 
\sphinxAtStartPar
Servers
\begin{itemize}
\item {} 
\sphinxAtStartPar
Compute

\item {} 
\sphinxAtStartPar
Storage

\item {} 
\sphinxAtStartPar
Control (min 3 for Core DC)

\end{itemize}

\item {} 
\sphinxAtStartPar
Network considerations
\begin{itemize}
\item {} 
\sphinxAtStartPar
Data centre gateway

\item {} 
\sphinxAtStartPar
Firewall (around the control plane, storage, etc.)

\item {} 
\sphinxAtStartPar
Data centre network fabric / Clos (spine/leaf) \textendash{} Horizontal scale

\item {} 
\sphinxAtStartPar
Storage networking, control plane and data plane

\item {} 
\sphinxAtStartPar
Raw packet \textendash{} tenant networking allowing “wild west” connection

\end{itemize}

\item {} 
\sphinxAtStartPar
Storage
\begin{itemize}
\item {} 
\sphinxAtStartPar
Storage technologies are multiple, they are extensively
described in
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter03.html\#storage-implementation-stereotypes}{Storage Implementation Stereotypes}.
Storage backends are discussed in
{\hyperref[\detokenize{chapters/chapter04:storage-backend}]{\sphinxcrossref{\DUrole{std,std-ref}{Storage Backend}}}}.

\end{itemize}

\item {} 
\sphinxAtStartPar
Acceleration
\begin{itemize}
\item {} 
\sphinxAtStartPar
SmartNIC

\item {} 
\sphinxAtStartPar
GPU

\item {} 
\sphinxAtStartPar
FPGA

\end{itemize}

\end{itemize}


\paragraph{Compute}
\label{\detokenize{chapters/chapter03:compute}}
\sphinxAtStartPar
Cloud Infrastructure physical Nodes

\sphinxAtStartPar
The physical resources required for the Cloud Infrastructure are mainly
based on COTS x86 hardware for control and data plane nodes. HW profiles
are defined in Reference Model chapters
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-hardware-profile-description}{Cloud Infrastructure Hardware Profile description}
and
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-hardware-profiles-features-and-requirements}{Cloud Infrastructure Hardware Profiles features and requirements.}.


\paragraph{Network}
\label{\detokenize{chapters/chapter03:network}}
\sphinxAtStartPar
The recommended network architecture is spine and leaf topology.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch03-Network-Fabric}.png}
\caption{Network Fabric \textendash{} Physical}\label{\detokenize{chapters/chapter03:network-fabric-physical}}\end{figure}

\sphinxAtStartPar
\hyperref[\detokenize{chapters/chapter03:network-fabric-physical}]{Fig.\@ \ref{\detokenize{chapters/chapter03:network-fabric-physical}}} shows a physical network layout where each
physical server is dual homed to TOR (Leaf/Access) switches with redundant
(2x) connections. The Leaf switches are dual homed with redundant connections
to spines.


\paragraph{Storage}
\label{\detokenize{chapters/chapter03:storage}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/arch-design/design-storage.html}{OpenStack}
supports many different storage architectures and backends. The choice
of a particular backend storage is driven by a number of factors
including: scalability, resiliency, availability, data durability,
capacity and performance.

\sphinxAtStartPar
Most cloud storage architectures incorporate a number of clustered
storage nodes that provide high bandwidth access to physical storage
backends connected by high speed networks. The architecture consists of
multiple storage controller units, each a generic server (CPU, Cache,
storage), managing a number of high\sphinxhyphen{}performance hard drives. The
distributed block storage software creates an abstract single pool of
storage by aggregating all of the controller units. Advanced and
high\sphinxhyphen{}speed networking (data routing) and global load balancing
techniques ensure high\sphinxhyphen{}performance, high availability storage system.


\subsection{Cloud Topology}
\label{\detokenize{chapters/chapter03:cloud-topology}}
\sphinxAtStartPar
A telco cloud will typically be deployed in multiple locations (“sites”)
of varying size and capabilities (HVAC, for example); or looking at this
in the context of OpenStack, multiple clouds (i.e. OpenStack end\sphinxhyphen{}points)
will be deployed that do not rely on each other, by design; each cloud
consists of a set of resources isolated from resources of the other
clouds. The application layer must span such end\sphinxhyphen{}points in order to
provide the required service SLA. Irrespective of the nature of the
deployment characteristics (e.g., number of racks, number of hosts), the
intent of the architecture would be to allow VNFs to be deployed in
these sites without major changes.

\sphinxAtStartPar
Some examples of such topologies include:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Large data centre capable of hosting potentially thousands of servers
and the networking to support them

\item {} 
\sphinxAtStartPar
Intermediate data centre (such as a central office) capable of
hosting up to a hundred servers

\item {} 
\sphinxAtStartPar
Edge (not customer premise) capable of hosting ten to fifty servers

\end{itemize}

\sphinxAtStartPar
In order to provide the expected availability for any given service, a
number of different OpenStack deployment topologies can be considered.
This section explores the main options and highlights the
characteristics of each. Ultimately the decision rests with the operator
to achieve specific availability target taking into account use case,
data centre capabilities, economics and risks.

\sphinxAtStartPar
\sphinxstylestrong{Topology Overview}

\sphinxAtStartPar
Availability of any single OpenStack cloud is dependent on a number of
factors including:
\begin{itemize}
\item {} 
\sphinxAtStartPar
environmental \textendash{} dual connected power and PDUs, redundant cooling,
rack distribution, etc.

\item {} 
\sphinxAtStartPar
resilient network fabric \textendash{} ToR (leaf), spine, overlay networking,
underlay networking, etc. It is assumed that all network components
are designed to be fault tolerant and all OpenStack controllers,
computes and storage are dual\sphinxhyphen{}homed to alternate leaf switches.

\item {} 
\sphinxAtStartPar
controller nodes setup in\sphinxhyphen{}line with the vendor recommendation (e.g.,
min 3 physical nodes)

\item {} 
\sphinxAtStartPar
network nodes (where applicable)

\item {} 
\sphinxAtStartPar
backend storage nodes setup for highly availablility based on quorum
(aligned with vendor implementation)

\item {} 
\sphinxAtStartPar
compute nodes sized to handle the entire workload following local
failure scenario

\end{itemize}

\sphinxAtStartPar
Assumptions and conventions:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Region is represented by a single OpenStack control plane.

\item {} 
\sphinxAtStartPar
Resource Failure Domain is effectively the “blast radius” of any
major infrastructure failure such as loss of PDU or network leafs.

\item {} 
\sphinxAtStartPar
Control plane includes redundant network nodes where OVS\sphinxhyphen{}kernel is
used.

\item {} 
\sphinxAtStartPar
Controller nodes should be setup for high availability based on
quorum (aligned with vendor implementation).

\item {} 
\sphinxAtStartPar
Shared storage is optional, but it is important to ensure shared
assets are distributed across serving clouds such as boot images.
Storage needs, per deployment and use cases, can be found in
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter03.html\#storage-scenarios-and-architecture-fit}{Storage Scenarios and Architecture Fit}.

\end{itemize}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Cloud Topology: Redundancy Models}\label{\detokenize{chapters/chapter03:id3}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{8}{80}|\X{15}{80}|\X{8}{80}|\X{8}{80}|\X{8}{80}|\X{8}{80}|\X{8}{80}|\X{17}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Topology Ref
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Type
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Control Planes
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Shared Storage (optional)
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Compute AZs
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Achievable Service Availability \%
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Service Multi \sphinxhyphen{}region awareness
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Notes
\\
\hline
\sphinxAtStartPar
1
&
\sphinxAtStartPar
Local Redundancy \sphinxhyphen{} workload spread across servers
&
\sphinxAtStartPar
1
&
\sphinxAtStartPar
1
&
\sphinxAtStartPar
1
&
\sphinxAtStartPar
Variable
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Suitable where only limited local application availability is required
e.g. nova anti\sphinxhyphen{}affinity
\\
\hline
\sphinxAtStartPar
2
&
\sphinxAtStartPar
Regional Redundancy \sphinxhyphen{} workload spread across AZs
&
\sphinxAtStartPar
1
&
\sphinxAtStartPar
\textgreater{}=2
&
\sphinxAtStartPar
\textgreater{}=2
&
\sphinxAtStartPar
\textgreater{}99.n
&
\sphinxAtStartPar
Not required
&
\sphinxAtStartPar
Suitable where local application HA is required. Control plane should be
distributed across DC failure domains (assuming layer 2 connectivity) but
may be unavailable during up grades
\\
\hline
\sphinxAtStartPar
3
&
\sphinxAtStartPar
Global Redundancy \sphinxhyphen{} workload spread across multiple Regions
&
\sphinxAtStartPar
\textgreater{}=2
&
\sphinxAtStartPar
\textgreater{}=2
&
\sphinxAtStartPar
\textgreater{}=2
&
\sphinxAtStartPar
\textgreater{}99.nn
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
Suitable where local and region application HA is required Control plane
could be kept available in one site during upgrades
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{Topology Overview}

\sphinxAtStartPar
\sphinxstylestrong{Topology 1 \sphinxhyphen{} Local Redundancy}

\sphinxAtStartPar
Under normal operation this deployment can handle a single failure of a
controller node or storage node without any impact to the service. If a
compute node fails the application layer (often the VNFM) would need to
restart workloads on a spare compute node of similar capability i.e.,
cloud may need to be provided with n+1 capacity. In the case of an
active/active application deployed to separate compute nodes (with
hypervisor anti\sphinxhyphen{}affinity) there would be no service impact.

\sphinxAtStartPar
\sphinxstyleemphasis{Important to consider:}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Where possible servers should be distributed and cabled to reduce the
impact of any failure e.g., PDU, rack failure. Because each operator
has individual site constraints this document will not propose a
standard rack layout.

\item {} 
\sphinxAtStartPar
During maintenance of the control plane, whilst the data (forwarding)
plane remains unaffected, the control plane API may not be available
and some applications may be relying on it during normal application
operation for example for scaling. Additionally if the upgrade
involves updating OpenStack services on the compute nodes care needs
to be taken. OVS\sphinxhyphen{}kernel networking operations may also be impacted
during this time.

\item {} 
\sphinxAtStartPar
During maintenance of storage (e.g., ceph) there is an increased risk
of a service\sphinxhyphen{}impacting failure, so it is generally recommended to
deploy at least one more server than the minimum required for
redundancy.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Topology 2 \sphinxhyphen{} Regional Redundancy}

\sphinxAtStartPar
Under normal operation this topology can handle a single failure of a
controller node but provides additional protection to the compute plane
and storage. If the application is deployed across 2 or more AZs a major
failure impacting the nodes in one AZ can be tolerated assuming the
application deployment allows for this. There is a risk with split\sphinxhyphen{}brain
so a means of deciding application quorum is recommended or by using a
third AZ or arbitrator.

\sphinxAtStartPar
\sphinxstyleemphasis{Important to consider:}
\begin{itemize}
\item {} 
\sphinxAtStartPar
All those points listed for Topology 1 above.

\item {} 
\sphinxAtStartPar
When using 3 controller nodes and distributing these physically
across the same locations as the computes, if you lose the location
with 2 controllers the OpenStack services would be impacted as quorum
cannot be gained with a single controller node. It is also possible
to use more than 3 controller nodes and co\sphinxhyphen{}locate one with each
compute AZ allowing lower\sphinxhyphen{}risk maintenance but care must be taken to
avoid split brain.

\item {} 
\sphinxAtStartPar
The distributed network fabric must support L2 for the OpenStack
control plane VIPs.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Topology 3 \sphinxhyphen{} Global Redundancy}

\sphinxAtStartPar
Following the example set by public cloud providers who provide Regions
and Availability Zones this is effectively a multi\sphinxhyphen{}region OpenStack.
Assuming the application can make use of this model this provides the
highest level of availability but would mean IP level failure controlled
outside of OpenStack by global service load balancing (GSLB) i.e., DNS
with minimum TTL configured, or client applications that are capable of
failing over themselves. This has the added advantage that no resources
are shared between different Regions so any fault is isolated to a
single cloud and also allows maintenance to take place without service
impact.


\section{Cloud Infrastructure \& VIM Component Level Architecture}
\label{\detokenize{chapters/chapter04:cloud-infrastructure-vim-component-level-architecture}}\label{\detokenize{chapters/chapter04::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter04:introduction}}
\sphinxAtStartPar
Chapter 3 introduced the components of an OpenStack\sphinxhyphen{}based IaaS
\begin{itemize}
\item {} 
\sphinxAtStartPar
Consumable Infrastructure Resources and Services

\item {} 
\sphinxAtStartPar
Cloud Infrastructure Management Software (VIM: OpenStack) core
services and architectural constructs needed to consume and manage
the consumable resources

\item {} 
\sphinxAtStartPar
Underlying physical compute, storage and networking resources

\end{itemize}

\sphinxAtStartPar
This chapter delves deeper into the capabilities of these different
resources and their needed configurations to create and operate an
OpenStack\sphinxhyphen{}based IaaS cloud. This chapter specifies details on the
structure of control and user planes, operating systems, hypervisors and
BIOS configurations, and architectural details of underlay and overlay
networking, and storage, and the distribution of OpenStack service
components among nodes. The chapter also covers implementation support
for the \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter02.html\#profiles-profile-extensions-flavours}{Profiles, Profile Extensions \& Flavours};
the OpenStack flavor types capture both the sizing and the profile
configuration (of the host).


\subsection{Underlying Resources}
\label{\detokenize{chapters/chapter04:underlying-resources}}

\subsubsection{Virtualisation}
\label{\detokenize{chapters/chapter04:virtualisation}}
\sphinxAtStartPar
In OpenStack, KVM is configured as the default hypervisor for compute
nodes.
\begin{itemize}
\item {} 
\sphinxAtStartPar
Configuration:
\sphinxhref{https://docs.openstack.org/nova/wallaby/admin/configuration/hypervisor-kvm.html}{OpenStack}
specifies the steps/instructions to configure KVM:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Enable KVM based hardware virtualisation in BIOS. OpenStack
provides instructions on how to enable hardware virtualisation for
different hardware platforms (x86, Power)
\begin{itemize}
\item {} 
\sphinxAtStartPar
QEMU is similar to KVM in that both are libvirt controlled,
have the same feature set and utilise compatible virtual
machine images

\end{itemize}

\item {} 
\sphinxAtStartPar
Configure Compute backing storage

\item {} 
\sphinxAtStartPar
Specify the CPU Model for KVM guests (VMs)

\item {} 
\sphinxAtStartPar
KVM Performance Tweaks

\end{itemize}

\item {} 
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html}{Hardening the virtualisation
layers}
\begin{itemize}
\item {} 
\sphinxAtStartPar
OpenStack recommends minimizing the code base by removing unused
components

\item {} 
\sphinxAtStartPar
sVirt (Secure Virtualisation) provides isolation between VM
processes, devices, data files and system processes

\end{itemize}

\end{itemize}


\subsubsection{Compute}
\label{\detokenize{chapters/chapter04:compute}}

\paragraph{Cloud Deployment (Foundation/management) Node}
\label{\detokenize{chapters/chapter04:cloud-deployment-foundation-management-node}}
\sphinxAtStartPar
Minimal configuration: 1 node


\paragraph{OpenStack Control Plane Servers (Control Nodes)}
\label{\detokenize{chapters/chapter04:openstack-control-plane-servers-control-nodes}}\begin{itemize}
\item {} 
\sphinxAtStartPar
BIOS Requirements

\end{itemize}

\sphinxAtStartPar
For OpenStack control nodes we use the BIOS parameters for the basic
profile defined in \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-hardware-profiles-features-and-requirements}{Cloud Infrastructure Hardware Profiles features and requirements.}.
Additionally, for OpenStack we need to set the following boot parameters:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Boot parameters}\label{\detokenize{chapters/chapter04:id1}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
BIOS/boot Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Value
\\
\hline
\sphinxAtStartPar
Boot disks
&
\sphinxAtStartPar
RAID 1
\\
\hline
\sphinxAtStartPar
CPU reservation for host (kernel)
&
\sphinxAtStartPar
1 core per NUMA
\\
\hline
\sphinxAtStartPar
CPU allocation ratio
&
\sphinxAtStartPar
2:1
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}
\begin{itemize}
\item {} 
\sphinxAtStartPar
How many nodes to meet SLA
\begin{itemize}
\item {} 
\sphinxAtStartPar
Minimum 3 nodes for high availability

\end{itemize}

\item {} 
\sphinxAtStartPar
HW specifications
\begin{itemize}
\item {} 
\sphinxAtStartPar
Boot disks are dedicated with Flash technology disks

\end{itemize}

\item {} 
\sphinxAtStartPar
Sizing rules
\begin{itemize}
\item {} 
\sphinxAtStartPar
It is easy to horizontally scale the number of control nodes

\item {} 
\sphinxAtStartPar
The number of control nodes is determined by a minimum number
needed for high availability (viz., 3 nodes) and the extra nodes
needed to handle the transaction volumes, in particular, for
Messaging service (e.g., RabbitMQ) and Database (e.g., MySQL) to
track state.

\item {} 
\sphinxAtStartPar
The number of control nodes only needs to be increased in
environments with a lot of changes, such as a testing lab, or a
very large cloud footprint (rule of thumb: number of control nodes
= 3 + quotient(number of compute nodes/1000)).

\item {} 
\sphinxAtStartPar
The \sphinxhref{https://fuel-ccp.readthedocs.io/en/latest/design/ref\_arch\_100\_nodes.html}{Services Placement Summary
table}
specifies the number of instances that are required based upon the
cloud size (number of nodes).

\end{itemize}

\end{itemize}


\paragraph{Network nodes}
\label{\detokenize{chapters/chapter04:network-nodes}}
\sphinxAtStartPar
Networks nodes are mainly used for L3 traffic management for overlay
tenant network (see more detail in section Neutron).
\begin{itemize}
\item {} 
\sphinxAtStartPar
BIOS requirements

\end{itemize}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{BIOS requirements}\label{\detokenize{chapters/chapter04:id2}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
BIOS/boot Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Value
\\
\hline
\sphinxAtStartPar
Boot disks
&
\sphinxAtStartPar
RAID 1
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}
\begin{itemize}
\item {} 
\sphinxAtStartPar
How many nodes to meet SLA
\begin{itemize}
\item {} 
\sphinxAtStartPar
Minimum 2 nodes for high availibility using VRRP.

\end{itemize}

\item {} 
\sphinxAtStartPar
HW specifications
\begin{itemize}
\item {} 
\sphinxAtStartPar
3 NICs card are needed if we want to isolate the different flows:
\begin{itemize}
\item {} 
\sphinxAtStartPar
1 NIC for Tenant Network

\item {} 
\sphinxAtStartPar
1 NIC for External Network

\item {} 
\sphinxAtStartPar
1 NIC for Other Networks (PXE, Mngt …)

\end{itemize}

\end{itemize}

\item {} 
\sphinxAtStartPar
Sizing rules
\begin{itemize}
\item {} 
\sphinxAtStartPar
Scale out of network node is not easy

\item {} 
\sphinxAtStartPar
DVR can be an option for large deployment (see more detail in
section Neutron)

\end{itemize}

\end{itemize}


\paragraph{Storage nodes}
\label{\detokenize{chapters/chapter04:storage-nodes}}\begin{itemize}
\item {} 
\sphinxAtStartPar
BIOS requirements

\end{itemize}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{BIOS requirements}\label{\detokenize{chapters/chapter04:id3}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
BIOS/boot Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Value
\\
\hline
\sphinxAtStartPar
Boot disks
&
\sphinxAtStartPar
RAID 1
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}
\begin{itemize}
\item {} 
\sphinxAtStartPar
HW specifications: please see \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter03.html\#storage}{Storage}

\item {} 
\sphinxAtStartPar
How many nodes to meet SLA: Active\sphinxhyphen{}Passive is the default and
recently OpenStack started to support Active\sphinxhyphen{}Active

\item {} 
\sphinxAtStartPar
Sizing rules: minimum 2 x 1 TB; recommended 2 x 10 TB

\end{itemize}


\paragraph{Compute Nodes}
\label{\detokenize{chapters/chapter04:compute-nodes}}
\sphinxAtStartPar
This section specifies the compute node configurations to support the
Basic and High\sphinxhyphen{}Performance profiles; in OpenStack this would be
accomplished by specifying the configurations when creating “flavors”.
The cloud operator may choose to implement certain profile\sphinxhyphen{}extensions
(\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter02.html\#profile-extensions-specialisations}{Profile Extensions (specialisations)})
as a set of standard configurations, of a given profile, capturing some
of the variability through different values or extra specifications.
\begin{itemize}
\item {} 
\sphinxAtStartPar
The software and hardware configurations are as specified in the
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-hardware-profiles-features-and-requirements}{Cloud Infrastructure Hardware Profiles features and requirements.}

\item {} 
\sphinxAtStartPar
BIOS requirement
\begin{itemize}
\item {} 
\sphinxAtStartPar
The general BIOS requirements are described in the
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-hardware-profiles-features-and-requirements}{Cloud Infrastructure Hardware Profiles features and requirements.}

\end{itemize}

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Example Profiles and their Extensions}

\sphinxAtStartPar
The Reference Model specifies the Basic (B) and High\sphinxhyphen{}Performance (H)
profile types. The Reference Model also provides a choice of network
acceleration capabilities utilising, for example, DPDK and SR\sphinxhyphen{}IOV
technologies. The Table :numref:\textasciigrave{} Profile Extensions and Capabilities\textasciigrave{}
(below) lists a few simple examples of profile
extensions and some of their capabilities.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Profile Extensions and Capabilities}\label{\detokenize{chapters/chapter04:id4}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{8}{100}|\X{30}{100}|\X{10}{100}|\X{10}{100}|\X{10}{100}|\X{10}{100}|\X{10}{100}|\X{12}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Profile Extensions
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
CPU Allocation Ratio
&\sphinxstyletheadfamily 
\sphinxAtStartPar
SMT
&\sphinxstyletheadfamily 
\sphinxAtStartPar
CPU Pinning
&\sphinxstyletheadfamily 
\sphinxAtStartPar
NUMA
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Huge pages
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Data Traffic
\\
\hline
\sphinxAtStartPar
B1
&
\sphinxAtStartPar
Basic Profile NoCPU over\sphinxhyphen{} subscription profile extension
&
\sphinxAtStartPar
1:1
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
N
&
\sphinxAtStartPar
OVS\sphinxhyphen{} kernel
\\
\hline
\sphinxAtStartPar
HV
&
\sphinxAtStartPar
High Performance Profile
&
\sphinxAtStartPar
1:1
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
OVS\sphinxhyphen{} kernel
\\
\hline
\sphinxAtStartPar
HD
&
\sphinxAtStartPar
High Performance Profile with DPDK profile extension
&
\sphinxAtStartPar
1:1
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
OVS\sphinxhyphen{}DPDK
\\
\hline
\sphinxAtStartPar
HS
&
\sphinxAtStartPar
High Performance Profile with SR\sphinxhyphen{}IOV profile extension
&
\sphinxAtStartPar
1:1
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
Y
&
\sphinxAtStartPar
SR\sphinxhyphen{}IOV
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{BIOS Settings}

\sphinxAtStartPar
A number of capabilities need to be enabled in the BIOS (such as NUMA
and SMT); the Reference Model section on
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter05.html\#cloud-infrastructure-software-profile-description}{Cloud Infrastructure Software profile description}
specifies the capabilities required to be configured. Please note that
capabilities may need to be configured in multiple systems. For
OpenStack, we also need to set the following boot parameters:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{BIOS requirements}\label{\detokenize{chapters/chapter04:id5}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
BIOS/boot Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Basic
&\sphinxstyletheadfamily 
\sphinxAtStartPar
High Performance
\\
\hline
\sphinxAtStartPar
Boot disks
&
\sphinxAtStartPar
RAID 1
&
\sphinxAtStartPar
RAID 1
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}
\begin{itemize}
\item {} 
\sphinxAtStartPar
How many nodes to meet SLA
\begin{itemize}
\item {} 
\sphinxAtStartPar
minimum: two nodes per profile

\end{itemize}

\item {} 
\sphinxAtStartPar
HW specifications
\begin{itemize}
\item {} 
\sphinxAtStartPar
Boot disks are dedicated with Flash technology disks

\end{itemize}

\item {} 
\sphinxAtStartPar
In case of DPDK usage:

\end{itemize}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{DPDK usage}\label{\detokenize{chapters/chapter04:id6}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{30}{100}|\X{70}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Layer
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
\\
\hline
\sphinxAtStartPar
Cloud infrastructure
&
\sphinxAtStartPar
Important is placement of NICs to get NUMA\sphinxhyphen{}balanced system (balancing
the I/O, memory, and storage across both sockets), and configuration of
NIC features. Server BIOS and Host OS kernel command line settings are
described in
\sphinxhref{http://doc.dpdk.org/guides/rel\_notes/}{DPDK release notes} and
\sphinxhref{http://core.dpdk.org/perf-reports/}{DPDK performance reports}.
Disabling power settings (like Intel Turbo Boost Technology) brings
stable performance results, although understanding if and when they
benefit workloads and enabling them can achieve better performance
results.
\\
\hline
\sphinxAtStartPar
Workload
&
\sphinxAtStartPar
DPDK uses core affinity along with 1G or 2M huge pages, NUMA settings
(to avoid crossing inteconnect between CPUs), and DPDK Poll Mode
Drivers (PMD, on reserved cores) to get the best performance. DPDK
versions xx.11 are Long\sphinxhyphen{}Term Support maintained stable release with
back\sphinxhyphen{}ported bug fixes for a two\sphinxhyphen{}year period.
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Sizing rules

\end{itemize}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Mnemonic}\label{\detokenize{chapters/chapter04:id7}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Mnemonic
\\
\hline
\sphinxAtStartPar
Number of CPU sockets
&
\sphinxAtStartPar
s
\\
\hline
\sphinxAtStartPar
Number of cores
&
\sphinxAtStartPar
c
\\
\hline
\sphinxAtStartPar
SMT
&
\sphinxAtStartPar
t
\\
\hline
\sphinxAtStartPar
RAM
&
\sphinxAtStartPar
rt
\\
\hline
\sphinxAtStartPar
Storage
&
\sphinxAtStartPar
d
\\
\hline
\sphinxAtStartPar
Overcommit
&
\sphinxAtStartPar
o
\\
\hline
\sphinxAtStartPar
Average vCPU per instance
&
\sphinxAtStartPar
v
\\
\hline
\sphinxAtStartPar
Average RAM per instance
&
\sphinxAtStartPar
ri
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Sizing rules}\label{\detokenize{chapters/chapter04:id8}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|T|}
\hline
\sphinxstyletheadfamily &\sphinxstyletheadfamily &\sphinxstyletheadfamily 
\sphinxAtStartPar
Basic
&\sphinxstyletheadfamily 
\sphinxAtStartPar
High\sphinxhyphen{}Performance
\\
\hline
\sphinxAtStartPar
\# of VMs per node (vCPU)
&
\sphinxAtStartPar
(s*c*t*o)/v
&
\sphinxAtStartPar
4*(sct)/v
&
\sphinxAtStartPar
(s*c*t)/v
\\
\hline
\sphinxAtStartPar
\# of VMs per node (RAM)
&
\sphinxAtStartPar
rt/ri
&
\sphinxAtStartPar
rt/ri
&
\sphinxAtStartPar
rt/ri
\\
\hline
\sphinxAtStartPar
Max \# of VMs per node
&&
\sphinxAtStartPar
min(4*(sct)/v,rt/ri)
&
\sphinxAtStartPar
min((s*c*t)/v,rt/ri)
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Caveats:
\begin{itemize}
\item {} 
\sphinxAtStartPar
These are theoretical limits

\item {} 
\sphinxAtStartPar
Affinity and anti\sphinxhyphen{}affinity rules, among other factors, affect the
sizing

\end{itemize}


\paragraph{Compute Resource Pooling Considerations}
\label{\detokenize{chapters/chapter04:compute-resource-pooling-considerations}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Multiple pools of hardware resources where each resource pool caters
for workloads of a specific profile (for example, High\sphinxhyphen{}Performance)
leads to inefficient use of the hardware as the server resources are
configured specifically for a profile. If not properly sized or when
demand changes, this can lead to oversupply/starvation scenarios;
reconfiguration may not be possible because of the underlying
hardware or inability to vacate servers for reconfiguration to
support another profile type.

\item {} 
\sphinxAtStartPar
Single pool of hardware resources including for controllers have the
same CPU configuration. This is operationally efficient as any server
can be utilised to support any profile or controller. The single pool
is valuable with unpredictable workloads or when the demand of
certain profiles is insufficient to justify individual hardware
selection.

\end{itemize}


\paragraph{Reservation of Compute Node Cores}
\label{\detokenize{chapters/chapter04:reservation-of-compute-node-cores}}
\sphinxAtStartPar
The {\hyperref[\detokenize{chapters/chapter02:infrastructure-requirements}]{\sphinxcrossref{\DUrole{std,std-ref}{Infrastructure Requirements}}}}
\sphinxcode{\sphinxupquote{inf.com.08}} requires the allocation of “certain number of host
cores/threads to non\sphinxhyphen{}tenant workloads such as for OpenStack services.” A
number (“n”) of random cores can be reserved for host services
(including OpenStack services) by specifying the following in nova.conf:

\sphinxAtStartPar
         reserved\_host\_cpus = n

\sphinxAtStartPar
where n is any positive integer.

\sphinxAtStartPar
If we wish to dedicate specific cores for host processing we need to
consider two different usage scenarios:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Require dedicated cores for Guest resources

\item {} 
\sphinxAtStartPar
No dedicated cores are required for Guest resources

\end{enumerate}

\sphinxAtStartPar
Scenario \#1, results in compute nodes that host both pinned and unpinned
workloads. In the OpenStack Wallaby release, scenario \#1 is not
supported; it may also be something that operators may not allow.
Scenario \#2 is supported through the specification of the cpu\_shared\_set
configuration. The cores and their sibling threads dedicated to the host
services are those that do not exist in the cpu\_shared\_set
configuration.

\sphinxAtStartPar
Let us consider a compute host with 20 cores with SMT enabled (let us
disregard NUMA) and the following parameters specified. The physical
cores are numbered ‘0’ to ‘19’ while the sibling threads are numbered
‘20’ to ‘39’ where the vCPUs numbered ‘0’ and ‘20’, ‘1’ and ‘21’, etc.
are siblings:

\sphinxAtStartPar
         cpu\_shared\_set = 1\sphinxhyphen{}7,9\sphinxhyphen{}19,21\sphinxhyphen{}27,29\sphinxhyphen{}39          (can also be
specified as cpu\_shared\_set = 1\sphinxhyphen{}19,\&8,21\sphinxhyphen{}39,\&28)

\sphinxAtStartPar
This implies that the two physical cores ‘0’ and ‘8’ and their sibling
threads ‘20’ and ‘28’ are dedicated to the host services, and 19 cores
and their sibling threads are available for Guest instances and can be
over allocated as per the specified cpu\_allocation\_ratio in nova.conf.


\paragraph{Pinned and Unpinned CPUs}
\label{\detokenize{chapters/chapter04:pinned-and-unpinned-cpus}}
\sphinxAtStartPar
When a server (viz., an instance) is created the vCPUs are, by default,
not assigned to a particular host CPU. Certain workloads require
real\sphinxhyphen{}time or near real\sphinxhyphen{}time behavior viz., uninterrupted access to their
cores. For such workloads, CPU pinning allows us to bind an instance’s
vCPUs to particular host cores or SMT threads. To configure a flavor to
use pinned vCPUs, we use a dedicated CPU policy.

\sphinxAtStartPar
         openstack flavor set .xlarge \textendash{}property hw:cpu\_policy=dedicated

\sphinxAtStartPar
While an instance with pinned CPUs cannot use CPUs of another pinned
instance, this does not apply to unpinned instances; an unpinned
instance can utilise the pinned CPUs of another instance. To prevent
unpinned instances from disrupting pinned instances, the hosts with CPU
pinning enabled are pooled in their own host aggregate and hosts with
CPU pinning disabled are pooled in another non\sphinxhyphen{}overlapping host
aggregate.


\paragraph{Compute node configurations for Profiles and OpenStack Flavors}
\label{\detokenize{chapters/chapter04:compute-node-configurations-for-profiles-and-openstack-flavors}}
\sphinxAtStartPar
This section specifies the compute node configurations to support
profiles and flavors.


\subparagraph{Cloud Infrastructure Hardware Profile}
\label{\detokenize{chapters/chapter04:cloud-infrastructure-hardware-profile}}
\sphinxAtStartPar
The Cloud Infrastructure Hardware (or simply “host”) profile and
configuration parameters are utilised in the reference architecture to
define different hardware profiles; these are used to configure the BIOS
settings on a physical server and configure utility software (such as
Operating System and Hypervisor).

\sphinxAtStartPar
An OpenStack flavor defines the characteristics (“capabilities”) of a
server (viz., VMs or instances) that will be deployed on hosts assigned
a host\sphinxhyphen{}profile. A many to many relationship exists between flavors and
host profiles. Multiple flavors can be defined with overlapping
capability specifications with only slight variations that servers of
these flavor types can be hosted on similary configured (host profile)
compute hosts. Similarly, a server can be specified with a flavor that
allows it to be hosted on, say, a host configured as per the Basic
profile or a host configured as per the High\sphinxhyphen{}Performance profile. Please
note that workloads that specify a server flavor so as to be hosted on a
host configured as per the High\sphinxhyphen{}Performance profile, may not be able to
run (adequately with expected performance) on a host configured as per
the Basic profile.

\sphinxAtStartPar
A given host can only be assigned a single host profile; a host profile
can be assigned to multiple hosts. Host profiles are immutable and hence
when a configuration needs to be changed, a new host profile is created.


\subparagraph{CPU Allocation Ratio and CPU Pinning}
\label{\detokenize{chapters/chapter04:cpu-allocation-ratio-and-cpu-pinning}}
\sphinxAtStartPar
A given host (compute node) can only support a single CPU Allocation
Ratio. Thus, to support the B1 and B4 Basic profile extensions (Section
4.2.2.5) with CPU Allocation Ratios of 1.0 and 4.0 we will need to
create 2 different host profiles and separate host aggregates for each
of the host profiles. The CPU Allocation Ratio is set in the hypervisor
on the host.
\begin{quote}

\sphinxAtStartPar
When the CPU Allocation Ratio exceeds 1.0 then CPU Pinning also needs
to be disabled.
\end{quote}


\subparagraph{Server Configurations}
\label{\detokenize{chapters/chapter04:server-configurations}}
\sphinxAtStartPar
The different networking choices \textendash{} OVS\sphinxhyphen{}Kernel, OVS\sphinxhyphen{}DPDK, SR\sphinxhyphen{}IOV \textendash{} result
in different NIC port, LAG (Link Aggregation Group), and other
configurations. Some of these are shown diagrammatically in section
4.2.9.5.


\subparagraph{Leaf and Compute Ports for Server Flavors must align}
\label{\detokenize{chapters/chapter04:leaf-and-compute-ports-for-server-flavors-must-align}}
\sphinxAtStartPar
Compute hosts have varying numbers of Ports/Bonds/LAGs/Trunks/VLANs
connected with Leaf ports. Each Leaf port (in A/B pair) must be
configured to align with the interfaces required for the compute flavor.

\sphinxAtStartPar
Physical Connections/Cables are generally the same within a zone,
regardless of these specific L2/L3/SR\sphinxhyphen{}IOV configurations for the
compute.

\sphinxAtStartPar
\sphinxstylestrong{Compute Bond Port:} TOR port maps VLANs directly with IRBs on the TOR
pair for tunnel packets and Control Plane Control and Storage packets.
These packets are then routed on the underlay network GRT.

\sphinxAtStartPar
Server Flavors: B1, B4, HV, HD

\sphinxAtStartPar
\sphinxstylestrong{Compute SR\sphinxhyphen{}IOV Port:} TOR port maps VLANs with bridge domains that
extend to IRBs, using VXLAN VNI. The TOR port associates each packet’s
outer VLAN tag with a bridge domain to support VNF interface adjacencies
over the local EVPN/MAC bridge domain. This model also applies to direct
physical connections with transport elements.

\sphinxAtStartPar
Server Flavors: HS

\sphinxAtStartPar
\sphinxstylestrong{Notes on SR\sphinxhyphen{}IOV}

\sphinxAtStartPar
SR\sphinxhyphen{}IOV, at the compute server, routes Guest traffic directly with a
partitioned NIC card, bypassing the hypervisor and vSwitch software,
which provides higher bps/pps throughput for the Guest server. OpenStack
and MANO manage SR\sphinxhyphen{}IOV configurations for Tenant server interfaces.
\begin{itemize}
\item {} 
\sphinxAtStartPar
Server, Linux, and NIC card hardware standards include SR\sphinxhyphen{}IOV and VF
requirements

\item {} 
\sphinxAtStartPar
High Performance profile for SR\sphinxhyphen{}IOV (hs series) with specific
NIC/Leaf port configurations

\item {} 
\sphinxAtStartPar
OpenStack supports SR\sphinxhyphen{}IOV provisioning

\item {} 
\sphinxAtStartPar
Implement Security Policy, Tap/Mirror, QoS, etc. functions in the
NIC, Leaf, and other places

\end{itemize}

\sphinxAtStartPar
Because SR\sphinxhyphen{}IOV involves Guest VLANs between the compute server and the
ToR/Leafs, Guest automation and server placement necessarily involves
the Leaf switches (e.g., access VLAN outer tag mapping with VXLAN EVPN).
\begin{itemize}
\item {} 
\sphinxAtStartPar
Local VXLAN tunneling over IP\sphinxhyphen{}switched fabric implemented between
VTEPs on Leaf switches.

\item {} 
\sphinxAtStartPar
Leaf configuration controlled by SDN\sphinxhyphen{}Fabric/Global Controller.

\item {} 
\sphinxAtStartPar
Underlay uses VXLAN\sphinxhyphen{}enabled switches for EVPN support

\end{itemize}

\sphinxAtStartPar
SR\sphinxhyphen{}IOV\sphinxhyphen{}based networking for Tenant Use Cases is required where
vSwitch\sphinxhyphen{}based networking throughput is inadequate.


\subparagraph{Example Host Configurations}
\label{\detokenize{chapters/chapter04:example-host-configurations}}
\sphinxAtStartPar
\sphinxstyleemphasis{Host configurations for B1, B4 Profile Extensions}

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch04-Basic-host-config}.png}
\caption{Basic Profile Host Configuration (example and simplified)}\label{\detokenize{chapters/chapter04:basic-profile-host-configuration}}\end{figure}

\sphinxAtStartPar
Let us refer to the data traffic networking configuration of
\hyperref[\detokenize{chapters/chapter04:basic-profile-host-configuration}]{Fig.\@ \ref{\detokenize{chapters/chapter04:basic-profile-host-configuration}}} to be part of the hp\sphinxhyphen{}B1\sphinxhyphen{}a and
hp\sphinxhyphen{}B4\sphinxhyphen{}a host profiles and this requires the configurations as Table
{\hyperref[\detokenize{chapters/chapter04:configuration-of-basic-flavor-capabilities}]{\sphinxcrossref{\DUrole{std,std-ref}{Configuration of Basic Flavor Capabilities}}}}.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Configuration of Basic Flavor Capabilities}\label{\detokenize{chapters/chapter04:id9}}\label{\detokenize{chapters/chapter04:configuration-of-basic-flavor-capabilities}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{50}|\X{10}{50}|\X{10}{50}|\X{10}{50}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Capability
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Configured in
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Host profile: hp\sphinxhyphen{}B1\sphinxhyphen{}a
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Host profile: hp\sphinxhyphen{}B4\sphinxhyphen{}a
\\
\hline
\sphinxAtStartPar
CPU Allocation Ratio
&
\sphinxAtStartPar
Hypervisor
&
\sphinxAtStartPar
1:1
&
\sphinxAtStartPar
4:1
\\
\hline
\sphinxAtStartPar
CPU Pinning
&
\sphinxAtStartPar
BIOS
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Disable
\\
\hline
\sphinxAtStartPar
SMT
&
\sphinxAtStartPar
BIOS
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Enable
\\
\hline
\sphinxAtStartPar
NUMA
&
\sphinxAtStartPar
BIOS
&
\sphinxAtStartPar
Disable
&
\sphinxAtStartPar
Disable
\\
\hline
\sphinxAtStartPar
Huge pages
&
\sphinxAtStartPar
BIOS
&
\sphinxAtStartPar
No
&
\sphinxAtStartPar
No
\\
\hline
\sphinxAtStartPar
Profile Extensions
&&
\sphinxAtStartPar
B1
&
\sphinxAtStartPar
B4
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\hyperref[\detokenize{chapters/chapter04:basic-profile-host-config-with-shared-storage-and-oam-networking}]{Fig.\@ \ref{\detokenize{chapters/chapter04:basic-profile-host-config-with-shared-storage-and-oam-networking}}}
shows the networking configuration where the storage and OAM share networking
but are independent of the PXE network.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch04-Basic-host-config-w-Storage-Network}.png}
\caption{Basic Profile Host Configuration with shared Storage and OAM
networking (example and simplified)}\label{\detokenize{chapters/chapter04:basic-profile-host-config-with-shared-storage-and-oam-networking}}\end{figure}

\sphinxAtStartPar
Let us refer to the above networking set up to be part of the hp\sphinxhyphen{}B1\sphinxhyphen{}b
and hp\sphinxhyphen{}B4\sphinxhyphen{}b host profiles but the basic configurations as specified in
Table {\hyperref[\detokenize{chapters/chapter04:configuration-of-basic-flavor-capabilities}]{\sphinxcrossref{\DUrole{std,std-ref}{Configuration of Basic Flavor Capabilities}}}}.

\sphinxAtStartPar
In our example, the Profile Extensions B1 and B4, are each mapped to two
different host profiles hp\sphinxhyphen{}B1\sphinxhyphen{}a and hp\sphinxhyphen{}B1\sphinxhyphen{}b, and hp\sphinxhyphen{}B4\sphinxhyphen{}a and hp\sphinxhyphen{}B4\sphinxhyphen{}b
respectively. Different network configurations, reservation of CPU
cores, Lag values, etc. result in different host profiles.

\sphinxAtStartPar
To ensure Tenant CPU isolation from the host services (Operating System
(OS), hypervisor and OpenStack agents), the following needs to be
configured:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{GRUB Configuration of Basic Profile with shared Storage}\label{\detokenize{chapters/chapter04:id10}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{70}|\X{30}{70}|\X{20}{70}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
GRUB Bootloader Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Values
\\
\hline
\sphinxAtStartPar
isolcpus (Applicable only on Compute Servers)
&
\sphinxAtStartPar
A set of cores isolated from the host processes. Contains vCPUs reserved for Tenants and DPDK
&
\sphinxAtStartPar
isolcpus=1\sphinxhyphen{}19, 21\sphinxhyphen{}39, 41\sphinxhyphen{}59, 61\sphinxhyphen{}79
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstyleemphasis{Host configuration for HV Profile Extensions}

\sphinxAtStartPar
The above examples of host networking configurations for the B1 and B4
Profile Extensions are also suitable for the HV Profile Extensions;
however, the hypervisor and BIOS settings will be different (see table
below) and hence there will be a need for different host profiles. Table
{\hyperref[\detokenize{chapters/chapter04:configuration-of-high-performance-flavor-capabilities}]{\sphinxcrossref{\DUrole{std,std-ref}{Configuration of High Performance Flavor Capabilities}}}} gives examples of
three different host profiles; one each for HV, HD and HS Profile Extensions.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Configuration of High Performance Flavor Capabilities}\label{\detokenize{chapters/chapter04:id11}}\label{\detokenize{chapters/chapter04:configuration-of-high-performance-flavor-capabilities}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{29}{80}|\X{12}{80}|\X{12}{80}|\X{12}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Capability
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Configured in
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Host profile: hp\sphinxhyphen{}hv\sphinxhyphen{}a
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Host profile: hp\sphinxhyphen{}hd\sphinxhyphen{}a
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Host profile: hp\sphinxhyphen{}hs\sphinxhyphen{}a
\\
\hline\sphinxstyletheadfamily 
\sphinxAtStartPar
Profile Extensions
&\sphinxstyletheadfamily &\sphinxstyletheadfamily 
\sphinxAtStartPar
HV
&\sphinxstyletheadfamily 
\sphinxAtStartPar
HD
&\sphinxstyletheadfamily 
\sphinxAtStartPar
HS
\\
\hline
\sphinxAtStartPar
CPU Allocation Ratio
&
\sphinxAtStartPar
Hypervisor
&
\sphinxAtStartPar
1:1
&
\sphinxAtStartPar
1:1
&
\sphinxAtStartPar
1:1
\\
\hline
\sphinxAtStartPar
NUMA
&
\sphinxAtStartPar
BIOS, Operating System, Hypervisor and OpenStack Nova Scheduler
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Enable
\\
\hline
\sphinxAtStartPar
CPU Pinning (requires NUMA)
&
\sphinxAtStartPar
OpenStack Nova Scheduler
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Enable
\\
\hline
\sphinxAtStartPar
SMT
&
\sphinxAtStartPar
BIOS
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Enable
&
\sphinxAtStartPar
Enable
\\
\hline
\sphinxAtStartPar
Huge pages
&
\sphinxAtStartPar
BIOS
&
\sphinxAtStartPar
Yes
&
\sphinxAtStartPar
Yes
&
\sphinxAtStartPar
Yes
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstyleemphasis{Host Networking configuration for HD Profile Extensions}

\sphinxAtStartPar
An example of the data traffic configuration for the HD (OVS\sphinxhyphen{}DPDK)
Profile Extensions is shown in
\hyperref[\detokenize{chapters/chapter04:high-performance-profile-host-conf-with-dpdk}]{Fig.\@ \ref{\detokenize{chapters/chapter04:high-performance-profile-host-conf-with-dpdk}}}.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch04-Network-Intensive-DPDK}.png}
\caption{High Performance Profile Host Configuration with DPDK acceleration
(example and simplified)}\label{\detokenize{chapters/chapter04:high-performance-profile-host-conf-with-dpdk}}\end{figure}

\sphinxAtStartPar
To ensure Tenant and DPDK CPU isolation from the host services
(Operating System (OS), hypervisor and OpenStack agents), the following
needs to be configured:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{GRUB Configuration of High Performance Flavor with DPDK}\label{\detokenize{chapters/chapter04:id12}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{70}|\X{30}{70}|\X{20}{70}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
GRUB Bootloader Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Values
\\
\hline
\sphinxAtStartPar
isolcpus (Applicable only on Compute Servers)
&
\sphinxAtStartPar
A set of cores isolated from the host processes. Contains vCPUs
reserved for Tenants and DPDK
&
\sphinxAtStartPar
isolcpus=3\sphinxhyphen{}19, 23\sphinxhyphen{}39, 43\sphinxhyphen{}59, 63\sphinxhyphen{}79
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstyleemphasis{Host Networking configuration for HS Profile Extensions}

\sphinxAtStartPar
An example of the data traffic configuration for the HS (SR\sphinxhyphen{}IOV) Profile
Extensions is shown in
\hyperref[\detokenize{chapters/chapter04:high-performance-profile-host-configuration-with-sr-iov}]{Fig.\@ \ref{\detokenize{chapters/chapter04:high-performance-profile-host-configuration-with-sr-iov}}}.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch04-Network-Intensive-SRIOV}.png}
\caption{High Performance Profile Host Configuration with SR\sphinxhyphen{}IOV
(example and simplified)}\label{\detokenize{chapters/chapter04:high-performance-profile-host-configuration-with-sr-iov}}\end{figure}

\sphinxAtStartPar
To ensure Tenant CPU isolation from the host services (Operating System
(OS), hypervisor and OpenStack agents), the following needs to be
configured:


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{GRUB Configuration of High Performance Flavor with SR\sphinxhyphen{}IOV}\label{\detokenize{chapters/chapter04:id13}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{70}|\X{30}{70}|\X{20}{70}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
GRUB Bootloader Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Values
\\
\hline
\sphinxAtStartPar
isolcpus (Applicable only on Compute Servers)
&
\sphinxAtStartPar
A set of cores isolated from the host processes. Contains vCPUs reserved for Tenants
&
\sphinxAtStartPar
isolcpus=1\sphinxhyphen{}19, 21\sphinxhyphen{}39, 41\sphinxhyphen{}59, 61\sphinxhyphen{}79
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subparagraph{Using Hosts of a Host Profile type}
\label{\detokenize{chapters/chapter04:using-hosts-of-a-host-profile-type}}
\sphinxAtStartPar
As we have seen Profile Extensions are supported by configuring hosts in
accordance with the Profile Extensions specifications. For example, an
instance of flavor type B1 can be hosted on a compute node that is
configured as an hp\sphinxhyphen{}B1\sphinxhyphen{}a or hp\sphinxhyphen{}B1\sphinxhyphen{}b host profile. All compute nodes
configured with hp\sphinxhyphen{}B1\sphinxhyphen{}a or hp\sphinxhyphen{}B1\sphinxhyphen{}b host profile are made part of a host
aggregate, say, ha\sphinxhyphen{}B1 and, thus, during server instantiation of B1
flavor hosts from the ha\sphinxhyphen{}B1 host aggregate will be selected.


\subsubsection{Network Fabric}
\label{\detokenize{chapters/chapter04:network-fabric}}
\sphinxAtStartPar
Networking Fabric consists of:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Physical switches, routers…

\item {} 
\sphinxAtStartPar
Switch OS

\item {} 
\sphinxAtStartPar
Minimum number of switches

\item {} 
\sphinxAtStartPar
Dimensioning for East/West and North/South

\item {} 
\sphinxAtStartPar
Spine / Leaf topology \textendash{} east \textendash{} west

\item {} 
\sphinxAtStartPar
Global Network parameters

\item {} 
\sphinxAtStartPar
OpenStack control plane VLAN / VXLAN layout

\item {} 
\sphinxAtStartPar
Provider VLANs

\end{itemize}


\paragraph{Physical Network Topology}
\label{\detokenize{chapters/chapter04:physical-network-topology}}

\paragraph{High Level Logical Network Layout}
\label{\detokenize{chapters/chapter04:high-level-logical-network-layout}}
\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch04-Indicative-OpenStack-Network}.png}
\caption{Indicative OpenStack Network Layout}\label{\detokenize{chapters/chapter04:id14}}\end{figure}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{OpenStack Network Characteristics}\label{\detokenize{chapters/chapter04:id15}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{35}{80}|\X{30}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Network
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Characteristics
\\
\hline
\sphinxAtStartPar
Provisioning \& Management
&
\sphinxAtStartPar
Initial OS bootstrapping of the servers via PXE, deployment of software
and thereafter for access from within the control plane.
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Management

\item {} 
\sphinxAtStartPar
Externally Routable: No

\item {} 
\sphinxAtStartPar
Connected to: All nodes

\end{itemize}
\\
\hline
\sphinxAtStartPar
Internal API
&
\sphinxAtStartPar
Intra\sphinxhyphen{}OpenStack service API communications, messaging, and database replication
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Management

\item {} 
\sphinxAtStartPar
Externally Routable: No

\item {} 
\sphinxAtStartPar
Connected to: All nodes except foundation

\end{itemize}
\\
\hline
\sphinxAtStartPar
Storage Management
&
\sphinxAtStartPar
Backend connectivity between storage nodes for heartbeats, data object replication and synchronisation
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Storage

\item {} 
\sphinxAtStartPar
Externally Routable: No

\item {} 
\sphinxAtStartPar
Connected to: All nodes except foundation

\end{itemize}
\\
\hline
\sphinxAtStartPar
Storage Front\sphinxhyphen{}end
&
\sphinxAtStartPar
Block/Object storage access via cinder/swift
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Storage

\item {} 
\sphinxAtStartPar
Externally Routable: No

\item {} 
\sphinxAtStartPar
Connected to: All nodes except foundation

\end{itemize}
\\
\hline
\sphinxAtStartPar
Tenant
&
\sphinxAtStartPar
VXLAN / Geneve project overlay networks (OVS kernel mode) \textendash{} i.e., RFC1918 re\sphinxhyphen{}usable private networks as controlled
by cloud administrator
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Underlay

\item {} 
\sphinxAtStartPar
Externally Routable: No

\item {} 
\sphinxAtStartPar
Connected to: controllers and computes

\end{itemize}
\\
\hline
\sphinxAtStartPar
External API
&
\sphinxAtStartPar
Hosts the public OpenStack API endpoints including the dashboard (Horizon)
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Public

\item {} 
\sphinxAtStartPar
Externally routable: Yes

\item {} 
\sphinxAtStartPar
Connected to: controllers

\end{itemize}
\\
\hline
\sphinxAtStartPar
External Provider (FIP)
&
\sphinxAtStartPar
Network with a pool of externally routable IP addresses used by neutron routers to NAT to/from the tenant RFC1918
private networks
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Data Centre

\item {} 
\sphinxAtStartPar
Externally routable: Yes

\item {} 
\sphinxAtStartPar
Connected to: controllers, OVS computes

\end{itemize}
\\
\hline
\sphinxAtStartPar
External Provider (VLAN)
&\begin{description}
\item[{External Data Centre L2 networks (VLANs) that are directly accessible to the project.}] \leavevmode
\sphinxAtStartPar
Note: External IP address management is required

\end{description}
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Data Centre

\item {} 
\sphinxAtStartPar
Externally routable: Yes

\item {} 
\sphinxAtStartPar
Connected to: OVS DPDK computes

\end{itemize}
\\
\hline
\sphinxAtStartPar
IPMI / Out of Band
&
\sphinxAtStartPar
The remote “lights\sphinxhyphen{}out” management port of the servers e.g., iLO, IDRAC / IPMI / Redfish
&\begin{itemize}
\item {} 
\sphinxAtStartPar
Security Domain: Management

\item {} 
\sphinxAtStartPar
Externally routable: No

\item {} 
\sphinxAtStartPar
Connected to: IPMI port on all servers

\end{itemize}
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
A VNF application network topology is expressed in terms of servers,
vNIC interfaces with vNet access networks, and WAN Networks while the
VNF Application Servers require multiple vNICs, VLANs, and host routes
configured within the server’s Kernel.


\paragraph{Octavia v2 API conformant Load Balancing}
\label{\detokenize{chapters/chapter04:octavia-v2-api-conformant-load-balancing}}
\sphinxAtStartPar
Load balancing is needed for automatic scaling, managing availability
and changes.
\sphinxhref{https://docs.openstack.org/octavia/latest/reference/introduction.html}{Octavia}
is an open\sphinxhyphen{}source load balancer for OpenStack, based on HAProxy, and
replaces the deprecated (as of OpenStack Queens release) Neutron LBaaS.
The Octavia v2 API is a superset of the deprecated Neutron LBaaS v2 API
and has a similar CLI for seamless transition.

\sphinxAtStartPar
As a default Octavia utilises Amphorae Load Balancer. Amphorae consists
of a fleet of servers (VMs, containers or bare metal servers) and
delivers horizontal scaling by managing and spinning these resources on
demand. The reference implementation of the Amphorae image is an Ubuntu
virtual machine running HAProxy.

\sphinxAtStartPar
Octavia depends upon a number of OpenStack services including Nova for
spinning up compute resources on demand and their life cycle management;
Neutron for connectivity between the compute resources, project
environment and external networks; Keystone for authentication; and
Glance for storing of the compute resource images.

\sphinxAtStartPar
Octavia supports provider drivers which allows third\sphinxhyphen{}party load
balancing drivers (such as F5, AVI, etc.) to be utilised instead of the
default Amphorae load balancer. When creating a third\sphinxhyphen{}party load
balancer, the \sphinxstylestrong{provider} attribute is used to specify the backend to
be used to create the load balancer. The \sphinxstylestrong{list providers} lists all
enabled provider drivers. Instead of using the provider parameter, an
alternate is to specify the flavor\_id in the create call where
provider\sphinxhyphen{}specific Octavia flavors have been created.


\paragraph{Neutron Extensions}
\label{\detokenize{chapters/chapter04:neutron-extensions}}
\sphinxAtStartPar
OpenStack Neutron is an extensible framework that allows incorporation
through plugins and API Extensions. API Extensions provide a method for
introducing new functionality and vendor specific capabilities. Neutron
plugins support new or vendor\sphinxhyphen{}specific functionality. Extensions also
allow specifying new resources or extensions to existing resources and
the actions on these resources. Plugins implement these resources and
actions.

\sphinxAtStartPar
This Reference Architecture supports the ML2 plugin (see below) as well
as the service plugins including for \sphinxhref{https://governance.openstack.org/tc/reference/projects/octavia.html}{LBaaS (Load Balancer as a
Service)},
and \sphinxhref{https://opendev.org/openstack/neutron-vpnaas/}{VPNaaS (VPN as a
Service)}. The
OpenStack wiki provides a list of \sphinxhref{https://wiki.openstack.org/wiki/Neutron\#Plugins}{Neutron
plugins}.

\sphinxAtStartPar
Every Neutron plugin needs to implement a minimum set of common \sphinxhref{https://docs.openstack.org/neutron/latest/contributor/internals/api\_extensions.html}{methods
(actions for Wallaby
release)}.
Resources can inherit Standard Attributes and thereby have the
extensions for these standard attributes automatically incorporated.
Additions to resources, such as additional attributes, must be
accompanied by an extension.

\sphinxAtStartPar
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_arch/openstack/chapters/chapter05.html}{Interfaces and APIs} of this Reference
Architecture provides a list of \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_arch/openstack/chapters/chapter05.html\#neutron}{\DUrole{xref,std,std-ref}{Neutron Extensions}}.
The current available
extensions can be obtained using the \sphinxhref{https://docs.openstack.org/api-ref/network/v2/\#list-extensions}{List Extensions
API}
and details about an extension using the \sphinxhref{https://docs.openstack.org/api-ref/network/v2/\#show-extension-details}{Show extension details
API}.

\sphinxAtStartPar
\sphinxstylestrong{Neutron ML2 integration} The OpenStack Modular Layer 2 (ML2) plugin
simplifies adding networking technologies by utilising drivers that
implement these network types and methods for accessing them. Each
network type is managed by an ML2 type driver and the mechanism driver
exposes interfaces to support the actions that can be performed on the
network type resources. The \sphinxhref{https://wiki.openstack.org/wiki/Neutron/ML2}{OpenStack ML2
documentation} lists
example mechanism drivers.


\paragraph{Network quality of service}
\label{\detokenize{chapters/chapter04:network-quality-of-service}}
\sphinxAtStartPar
For VNF workloads, the resource bottlenecks are not only the CPU and the
memory but also the I/O bandwidth and the forwarding capacity of virtual
and non\sphinxhyphen{}virtual switches and routers within the infrastructure. Several
techniques (all complementary) can be used to improve QoS and try to
avoid any issue due to a network bottleneck (mentioned per order of
importance):
\begin{itemize}
\item {} 
\sphinxAtStartPar
Nodes interfaces segmentation: Have separated NIC ports for Storage
and Tenant networks. Actually, the storage traffic is bursty, and
especially in case of service restoration after some failure or new
service implementation, upgrades, etc. Control and management
networks should rely on a separate interface from the interface used
to handle tenant networks.

\item {} 
\sphinxAtStartPar
Capacity planning: FW, physical links, switches, routers, NIC
interfaces and DCGW dimensioning (+ load monitoring: each link within
a LAG or a bond shouldn’t be loaded over 50\% of its maximum capacity
to guaranty service continuity in case of individual failure).

\item {} 
\sphinxAtStartPar
Hardware choice: e.g., ToR/fabric switches, DCGW and NIC cards should
have appropriate buffering and queuing capacity.

\item {} 
\sphinxAtStartPar
High Performance compute node tuning (including OVS\sphinxhyphen{}DPDK).

\end{itemize}


\paragraph{Integration Interfaces}
\label{\detokenize{chapters/chapter04:integration-interfaces}}\begin{itemize}
\item {} 
\sphinxAtStartPar
DHCP:

\end{itemize}

\sphinxAtStartPar
When the Neutron\sphinxhyphen{}DHCP agent is hosted in controller nodes, then for the
servers, on a Tenant network, that need to acquire an IPv4 and/or IPv6
address, the VLAN for the Tenant must be extended to the control plane
servers so that the Neutron agent can receive the DHCP requests from the
server and send the response to the server with the IPv4 and/or IPv6
addresses and the lease time. Please see OpenStack provider Network.
\begin{itemize}
\item {} 
\sphinxAtStartPar
DNS

\item {} 
\sphinxAtStartPar
LDAP

\item {} 
\sphinxAtStartPar
IPAM

\end{itemize}


\subsubsection{Storage Backend}
\label{\detokenize{chapters/chapter04:storage-backend}}
\sphinxAtStartPar
Storage systems are available from multiple vendors and can also utilise
commodity hardware from any number of open\sphinxhyphen{}source based storage packages
(such as LVM, Ceph, NFS, etc.). The proprietary and open\sphinxhyphen{}source storage
systems are supported in Cinder through specific plugin drivers. The
OpenStack \sphinxhref{https://docs.openstack.org/cinder/latest/reference/support-matrix.html}{Cinder
documentation}
specifies the minimum functionality that all storage drivers must
support. The functions include:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Volume: create, delete, attach, detach, extend, clone (volume from
volume), migrate

\item {} 
\sphinxAtStartPar
Snapshot: create, delete and create volume from snapshot

\item {} 
\sphinxAtStartPar
Image: create from volume

\end{itemize}

\sphinxAtStartPar
The document also includes a matrix for a number of proprietary drivers
and some of the optional functions that these drivers support. This
matrix is a handy tool to select storage backends that have the optional
storage functions needed by the cloud operator. The cloud workload
storage requirements helps determine the backends that should be
deployed by the cloud operator. The common storage backend attachment
methods include iSCSI, NFS, local disk, etc. and the matrix lists the
supported methods for each of the vendor drivers. The OpenStack Cinder
\sphinxhref{https://docs.openstack.org/cinder/latest/drivers.html}{Available
Drivers}
documentation provides a list of all OpenStack compatible drivers and
their configuration options.

\sphinxAtStartPar
The \sphinxhref{https://docs.openstack.org/cinder/latest/configuration/index.html}{Cinder
Configuration}
document provides information on how to configure Cinder including
Anuket required capabilities for volume encryption, Policy
configuration, quotas, etc. The \sphinxhref{https://docs.openstack.org/cinder/latest/admin/index.html}{Cinder
Administration}
document provides information on the capabilities required by Anuket
including managing volumes, snapshots, multi\sphinxhyphen{}storage backends, migrate
volumes, etc.

\sphinxAtStartPar
\sphinxhref{https://ceph.io/en/}{Ceph} is the default Anuket Reference Architecture
storage backend and is discussed below.


\paragraph{Ceph Storage Cluster}
\label{\detokenize{chapters/chapter04:ceph-storage-cluster}}
\sphinxAtStartPar
The Ceph storage cluster is deployed on bare metal hardware. The minimal
configuration is a cluster of three bare metal servers to ensure High
availability. The Ceph Storage cluster consists of the following
components:
\begin{itemize}
\item {} 
\sphinxAtStartPar
CEPH\sphinxhyphen{}MON (Ceph Monitor)

\item {} 
\sphinxAtStartPar
OSD (object storage daemon)

\item {} 
\sphinxAtStartPar
RadosGW (Rados Gateway)

\item {} 
\sphinxAtStartPar
Journal

\item {} 
\sphinxAtStartPar
Manager

\end{itemize}

\sphinxAtStartPar
Ceph monitors maintain a master copy of the maps of the cluster state
required by Ceph daemons to coordinate with each other. Ceph OSD handles
the data storage (read/write data on the physical disks), data
replication, recovery, rebalancing, and provides some monitoring
information to Ceph Monitors. The RadosGW provides Object Storage
RESTful gateway with a Swift\sphinxhyphen{}compatible API for Object Storage.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch04-Ceph}.png}
\caption{Ceph Storage System}\label{\detokenize{chapters/chapter04:id16}}\end{figure}

\sphinxAtStartPar
\sphinxstylestrong{BIOS Requirement for Ceph servers}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{BIOS Requirement for Ceph servers}\label{\detokenize{chapters/chapter04:id17}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
BIOS/boot Parameter
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Value
\\
\hline
\sphinxAtStartPar
Boot disks
&
\sphinxAtStartPar
RAID 1
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
How many nodes to meet SLA :
\begin{itemize}
\item {} 
\sphinxAtStartPar
minimum: three bare metal servers where Monitors are collocated with
OSD. Note: at least 3 Monitors and 3 OSDs are required for High
Availability.

\end{itemize}

\sphinxAtStartPar
HW specifications :
\begin{itemize}
\item {} 
\sphinxAtStartPar
Boot disks are dedicated with Flash technology disks

\item {} 
\sphinxAtStartPar
For an IOPS oriented cluster (Flash technology ), the journal can be
hosted on OSD disks

\item {} 
\sphinxAtStartPar
For a capacity\sphinxhyphen{}oriented cluster (HDD), the journal must be hosted on
dedicated Flash technology disks

\end{itemize}

\sphinxAtStartPar
Sizing rules :
\begin{itemize}
\item {} 
\sphinxAtStartPar
Minimum of 6 disks per server

\item {} 
\sphinxAtStartPar
Replication factor : 3

\item {} 
\sphinxAtStartPar
1 Core\sphinxhyphen{}GHz per OSD

\item {} 
\sphinxAtStartPar
16GB RAM baseline + 2\sphinxhyphen{}3 GB per OSD

\end{itemize}


\subsection{Virtualised Infrastructure Manager (VIM)}
\label{\detokenize{chapters/chapter04:virtualised-infrastructure-manager-vim}}
\sphinxAtStartPar
This section covers:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Detailed breakdown of OpenStack core services

\item {} 
\sphinxAtStartPar
Specific build\sphinxhyphen{}time parameters

\end{itemize}


\subsubsection{VIM Services}
\label{\detokenize{chapters/chapter04:vim-services}}
\sphinxAtStartPar
A high\sphinxhyphen{}level overview of the core OpenStack Services was provided in
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_arch/openstack/chapters/chapter03.html}{Cloud Infrastructure Architecture \sphinxhyphen{} OpenStack}. In this section we describe the core and
other needed services in more detail.


\paragraph{Keystone}
\label{\detokenize{chapters/chapter04:keystone}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/keystone/wallaby/}{Keystone} is the
authentication service, the foundation of identity management in
OpenStack. Keystone needs to be the first deployed service. Keystone has
services running on the control nodes and no services running on the
compute nodes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Keystone admin API

\item {} 
\sphinxAtStartPar
Keystone public API \textendash{} in Keystone V3 this is the same as the admin
API

\end{itemize}


\paragraph{Glance}
\label{\detokenize{chapters/chapter04:glance}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/glance/wallaby/}{Glance} is the image
management service. Glance has only a dependency on the Keystone service
therefore it is the second one deployed. Glance has services running on
the control nodes and no services running on the compute nodes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Glance API

\item {} 
\sphinxAtStartPar
Glance Registry

\end{itemize}

\sphinxAtStartPar
\sphinxstyleemphasis{The Glance backends include Swift, Ceph RBD and NFS.}


\paragraph{Cinder}
\label{\detokenize{chapters/chapter04:cinder}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/cinder/wallaby/}{Cinder} is the block
device management service, depends on Keystone and possibly Glance to be
able to create volumes from images. Cinder has services running on the
control nodes and no services running on the compute nodes: \sphinxhyphen{} Cinder API
\sphinxhyphen{} Cinder Scheduler \sphinxhyphen{} Cinder Volume \textendash{} the Cinder volume process needs to
talk to its backends

\sphinxAtStartPar
\sphinxstyleemphasis{The Cinder backends include SAN/NAS storage, iSCSI drives, Ceph RBD and
NFS.}


\paragraph{Swift}
\label{\detokenize{chapters/chapter04:swift}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/swift/wallaby/}{Swift} is the object
storage management service, Swift depends on Keystone and possibly
Glance to be able to create volumes from images. Swift has services
running on the control nodes and the compute nodes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Proxy Services

\item {} 
\sphinxAtStartPar
Object Services

\item {} 
\sphinxAtStartPar
Container Services

\item {} 
\sphinxAtStartPar
Account Services

\end{itemize}

\sphinxAtStartPar
\sphinxstyleemphasis{The Swift backends include iSCSI drives, Ceph RBD and NFS.}


\paragraph{Neutron}
\label{\detokenize{chapters/chapter04:neutron}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/neutron/wallaby/}{Neutron} is the
networking service, depends on Keystone and has services running on the
control nodes and the compute nodes. Depending upon the workloads to be
hosted by the Infrastructure, and the expected load on the controller
node, some of the Neutron services can run on separate network node(s).
Factors affecting controller node load include number of compute nodes
and the number of API calls being served for the various OpenStack
services (nova, neutron, cinder, glance etc.). To reduce controller node
load, network nodes are widely added to manage L3 traffic for overlay
tenant networks and interconnection with external networks. The Table
below lists the networking service components and their placement.
Please note that while network nodes are listed in the table below,
network nodes only deal with tenant networks and not provider networks.
Also, network nodes are not required when SDN is utilised for
networking.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Neutron Services Placement}\label{\detokenize{chapters/chapter04:id18}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{15}{80}|\X{30}{80}|\X{15}{80}|\X{20}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Networking Service component
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Description
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Required or Optional Service
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Placement
\\
\hline
\sphinxAtStartPar
neutron server (neutron\sphinxhyphen{}server and neutron\sphinxhyphen{}*\sphinxhyphen{}plugin)
&
\sphinxAtStartPar
Manages user requests and exposes the Neutron APIs
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
Controller node
\\
\hline
\sphinxAtStartPar
DHCP agent (neutron\sphinxhyphen{}dhcp\sphinxhyphen{}agent)
&
\sphinxAtStartPar
Provides DHCP services to tenant networks and is responsible for
maintaining DHCP configuration. For High availability, multiple DHCP
agents can be assigned.
&
\sphinxAtStartPar
Optional depending upon plug\sphinxhyphen{}in
&
\sphinxAtStartPar
Network node (Controller node if no network node present)
\\
\hline
\sphinxAtStartPar
L3 agent (neutron\sphinxhyphen{}l3\sphinxhyphen{}agent)
&
\sphinxAtStartPar
Provides L3/NAT forwarding for external network access of servers on
tenant networks and supports services such as Firewall\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}service
(FWaaS) and Load Balancer\sphinxhyphen{}as\sphinxhyphen{}a\sphinxhyphen{}service (LBaaS)
&
\sphinxAtStartPar
Optional depending upon plug\sphinxhyphen{}in
&
\sphinxAtStartPar
Network node (Controller node if no network node present) NB in DVR
based OpenStack Networking, also in all Compute nodes.
\\
\hline
\sphinxAtStartPar
neutron metadata agent (neutron\sphinxhyphen{}metadata\sphinxhyphen{}agent)
&
\sphinxAtStartPar
The metadata service provides a way for instances to retrieve
instance\sphinxhyphen{}specific data. The networking service, neutron, is responsible
for intercepting these requests and adding HTTP headers which uniquely
identify the source of the request before forwarding it to the metadata
API server. These functions are performed by the neutron metadata
agent.
&
\sphinxAtStartPar
Optional
&
\sphinxAtStartPar
Network node (Controller node if no network node present)
\\
\hline
\sphinxAtStartPar
neutron plugin agent (neutron\sphinxhyphen{}*\sphinxhyphen{}agent)
&
\sphinxAtStartPar
Runs on each compute node to control and manage the local virtual
network driver (such as the Open vSwitch or Linux Bridge)
configuration and local networking configuration for servers hosted on that node.
&
\sphinxAtStartPar
Required
&
\sphinxAtStartPar
Every Compute Node
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subparagraph{Issues with the standard networking (centralised routing) approach}
\label{\detokenize{chapters/chapter04:issues-with-the-standard-networking-centralised-routing-approach}}
\sphinxAtStartPar
The network node performs both routing and NAT functions and represents
both a scaling bottleneck and a single point of failure.

\sphinxAtStartPar
Consider two servers on different compute nodes and using different
project networks (a.k.a. tenant networks) where the both of the project
networks are connected by a project router. For communication between
the two servers (instances with a fixed or floating IP address), the
network node routes East\sphinxhyphen{}West network traffic among project networks
using the same project router. Even though the instances are connected
by a router, all routed traffic must flow through the network node, and
this becomes a bottleneck for the whole network.

\sphinxAtStartPar
While the separation of the routing function from the controller node to
the network node provides a degree of scaling it is not a truly scalable
solution. We can either add additional cores/compute\sphinxhyphen{}power or network
node to the network node cluster, but, eventually, it runs out of
processing power especially with high throughput requirement. Therefore,
for scaled deployments, there are multiple options including use of
Dynamic Virtual Routing (DVR) and Software Defined Networking (SDN).


\subparagraph{Distributed Virtual Routing (DVR)}
\label{\detokenize{chapters/chapter04:distributed-virtual-routing-dvr}}
\sphinxAtStartPar
With DVR, each compute node also hosts the L3\sphinxhyphen{}agent (providing the
distributed router capability) and this then allows direct instance to
instance (East\sphinxhyphen{}West) communications.

\sphinxAtStartPar
The OpenStack “\sphinxhref{https://docs.openstack.org/liberty/networking-guide/scenario-dvr-ovs.html}{High Availability Using Distributed Virtual Routing
(DVR)}”
provides an in\sphinxhyphen{}depth view into how DVR works and the traffic flow
between the various nodes and interfaces for three different use cases.
Please note that DVR was introduced in the OpenStack Juno release and,
thus, its detailed analysis in the Liberty release documentation is not
out of character for OpenStack documentation.

\sphinxAtStartPar
DVR addresses both scalability and high availability for some L3
functions but is not fully fault tolerant. For example, North/South SNAT
traffic is vulnerable to single node (network node) failures. \sphinxhref{https://docs.openstack.org/neutron/wallaby/admin/config-dvr-ha-snat.html}{DVR with
VRRP}
addresses this vulnerability.


\subparagraph{Software Defined Networking (SDN)}
\label{\detokenize{chapters/chapter04:software-defined-networking-sdn}}
\sphinxAtStartPar
For the most reliable solution that addresses all the above issues and
Telco workload requirements requires SDN to offload Neutron calls.

\sphinxAtStartPar
SDN provides a truly scalable and preferred solution to suport dynamic,
very large\sphinxhyphen{}scale, high\sphinxhyphen{}density, telco cloud environments. OpenStack
Neutron, with its plugin architecture, provides the ability to integrate
SDN controllers ({\hyperref[\detokenize{chapters/chapter03:virtual-networking-3rd-party-sdn-solution}]{\sphinxcrossref{\DUrole{std,std-ref}{Virtual Networking \textendash{} 3rd party SDN solution}}}}).
With SDN incorporated in OpenStack, changes to the network is triggered
by workloads (and users), translated into Neutron APIs and then handled
through neutron plugins by the corresponding SDN agents.


\paragraph{Nova}
\label{\detokenize{chapters/chapter04:nova}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/nova/wallaby/}{Nova} is the compute
management service, depends on all above components and is deployed
after their deployment. Nova has services running on the control nodes
and the compute nodes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
nova\sphinxhyphen{}metadata\sphinxhyphen{}api

\item {} 
\sphinxAtStartPar
nova\sphinxhyphen{}compute api

\item {} 
\sphinxAtStartPar
nova\sphinxhyphen{}consoleauth

\item {} 
\sphinxAtStartPar
nova\sphinxhyphen{}scheduler

\item {} 
\sphinxAtStartPar
nova\sphinxhyphen{}conductor

\item {} 
\sphinxAtStartPar
nova\sphinxhyphen{}novncproxy

\item {} 
\sphinxAtStartPar
nova\sphinxhyphen{}compute\sphinxhyphen{}agent which runs on Compute node

\end{itemize}

\sphinxAtStartPar
Please note that the Placement\sphinxhyphen{}API must have been installed and
configured prior to nova compute starts.


\paragraph{Ironic}
\label{\detokenize{chapters/chapter04:ironic}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/ironic/wallaby/}{Ironic} is the bare
metal provisioning service. Ironic depends on all above components and
is deployed after them. Ironic has services running on the control nodes
and the compute nodes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Ironic API

\item {} 
\sphinxAtStartPar
ironic\sphinxhyphen{}conductor which executes operation on bare metal nodes

\end{itemize}

\sphinxAtStartPar
Note: This is an optional service. The \sphinxhref{https://docs.openstack.org/api-ref/baremetal/}{Ironic
APIs} are still under
development.


\paragraph{Heat}
\label{\detokenize{chapters/chapter04:heat}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/heat/wallaby/}{Heat} is the orchestration
service using templates to provision cloud resources, Heat integrates
with all OpenStack services. Heat has services running on the control
nodes and no services running on the compute nodes:
\begin{itemize}
\item {} 
\sphinxAtStartPar
heat\sphinxhyphen{}api

\item {} 
\sphinxAtStartPar
heat\sphinxhyphen{}cfn\sphinxhyphen{}api

\item {} 
\sphinxAtStartPar
heat\sphinxhyphen{}engine

\end{itemize}


\paragraph{Horizon}
\label{\detokenize{chapters/chapter04:horizon}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/horizon/wallaby/}{Horizon} is the Web
User Interface to all OpenStack services. Horizon has services running
on the control nodes and no services running on the compute nodes.


\paragraph{Placement}
\label{\detokenize{chapters/chapter04:placement}}
\sphinxAtStartPar
The OpenStack \sphinxhref{https://docs.openstack.org/placement/wallaby/index.html}{Placement
service}
enables tracking (or accounting) and scheduling of resources. It
provides a RESTful API and a data model for the managing of resource
provider inventories and usage for different classes of resources. In
addition to standard resource classes, such as vCPU, MEMORY\_MB and
DISK\_GB, the Placement service supports custom resource classes
(prefixed with “CUSTOM\_”) provided by some external resource pools such
as a shared storage pool provided by, say, Ceph. The placement service
is primarily utilised by nova\sphinxhyphen{}compute and nova\sphinxhyphen{}scheduler. Other
OpenStack services such as Neutron or Cyborg can also utilise placement
and do so by creating \sphinxhref{https://docs.openstack.org/placement/latest/user/provider-tree.html}{Provider
Trees}.
The following data objects are utilised in the \sphinxhref{https://docs.openstack.org/placement/latest/user/index.html}{placement
service}:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Resource Providers provide consumable inventory of one or more
classes of resources (CPU, memory or disk). A resource provider can
be a compute host, for example.

\item {} 
\sphinxAtStartPar
Resource Classes specify the type of resources (vCPU, MEMORY\_MB and
DISK\_GB or CUSTOM\_*)

\item {} 
\sphinxAtStartPar
Inventory: Each resource provider maintains the total and reserved
quantity of one or more classes of resources. For example, RP\_1 has
available inventory of 16 vCPU, 16384 MEMORY\_MB and 1024 DISK\_GB.

\item {} 
\sphinxAtStartPar
Traits are qualitative characteristics of the resources from a
resource provider. For example, the trait for RPA\_1 “is\_SSD” to
indicate that the DISK\_GB provided by RP\_1 are solid state drives.

\item {} 
\sphinxAtStartPar
Allocations represent resources that have been assigned/used by some
consumer of that resource.

\item {} 
\sphinxAtStartPar
Allocation candidates is the collection of resource providers that
can satisfy an allocation request.

\end{itemize}

\sphinxAtStartPar
The Placement API is stateless and, thus, resiliency, availability and
scaling, it is possible to deploy as many servers as needed. On start,
the nova\sphinxhyphen{}compute service will attempt to make a connection to the
Placement API and keep attempting to connect to the Placement API,
logging and warning periodically until successful. Thus, the Placement
API must be installed and enabled prior to Nova compute.

\sphinxAtStartPar
Placement has services running on the control node: \sphinxhyphen{} nova\sphinxhyphen{}placement\sphinxhyphen{}api


\paragraph{Barbican}
\label{\detokenize{chapters/chapter04:barbican}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/barbican/wallaby/}{Barbican} is the
OpenStack Key Manager service. It is an optional service hosted on
controller nodes. It provides secure storage, provisioning, and
management of secrets as passwords, encryption keys and X.509
Certificates. Barbican API is used to centrally manage secrets used by
OpenStack services, e.g., symmetric encryption keys used for Block
storage encryption or Object Storage encryption or asymmetric keys and
certificates used for Glance image signing and verification.

\sphinxAtStartPar
Barbican usage provides a means to fulfill security requirements such as
sec.sys.012 “The Platform \sphinxstylestrong{must} protect all secrets by using strong
encryption techniques and storing the protected secrets externally from
the component” and sec.ci.001 “The Platform \sphinxstylestrong{must} support
Confidentiality and Integrity of data at rest and in transit.”.


\paragraph{Cyborg}
\label{\detokenize{chapters/chapter04:cyborg}}
\sphinxAtStartPar
\sphinxhref{https://docs.openstack.org/cyborg/wallaby/}{Cyborg} is the OpenStack
project for the general purpose management framework for accelerators
(including GPUs, FPGAs, ASIC\sphinxhyphen{}based devices, etc.), and their lifecycle
management.

\sphinxAtStartPar
Cyborg will support only a subset of the \sphinxhref{https://docs.openstack.org/api-guide/compute/server\_concepts.html}{Nova
operations};
the set of Nova operations supported in Cyborg depends upon the merge of
a set of Nova patches in Cyborg. In Wallaby, not all the required Nova
patches have been merged. The list of Cyborg operations with Nova
dependencies supported in Wallaby is listed
\sphinxhref{https://docs.openstack.org/cyborg/wallaby/reference/support-matrix.html}{here};
the Nova operations supported in Cyborg at any given time is also
\sphinxhref{https://docs.openstack.org/cyborg/latest/reference/support-matrix.html}{available}.

\sphinxAtStartPar
Cyborg supports:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Acceleration Resource Discovery

\item {} 
\sphinxAtStartPar
Accelerator Life Cycle Management

\end{itemize}

\sphinxAtStartPar
Accelerators can be of type:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Software: dpdk/spdk, pmem, …

\item {} 
\sphinxAtStartPar
Hardware (device types): FPGA, GPU, ARM SoC, NVMe SSD, CCIX based
Caches, …

\end{itemize}

\sphinxAtStartPar
The \sphinxhref{https://docs.openstack.org/cyborg/latest/user/architecture.html}{Cyborg
architecture}
consists of the cyborg\sphinxhyphen{}api, cyborg\sphinxhyphen{}conductor, cyborg\sphinxhyphen{}db, cyborg\sphinxhyphen{}agent,
and generic device type drivers. cyborg\sphinxhyphen{}api, cyborg\sphinxhyphen{}conductor and
cyborg\sphinxhyphen{}db are hosted on control nodes. cyborg\sphinxhyphen{}agent, which runs on
compute nodes, interacts with generic device type drivers on those
nodes. These generic device type drivers are an abstraction of the
vendor specific drivers; there is a generic device type driver for each
device type (see above for list of some of the device types). The
current list of the supported vendor drivers is listed under “\sphinxhref{https://docs.openstack.org/cyborg/latest/reference/support-matrix.html}{Driver
Support}”.


\subsubsection{Containerised OpenStack Services}
\label{\detokenize{chapters/chapter04:containerised-openstack-services}}
\sphinxAtStartPar
Containers are lightweight compared to Virtual Machines and leads to
efficient resource utilisation. Kubernetes auto manages scaling,
recovery from failures, etc. Thus, it is recommended that the OpenStack
services be containerised for resiliency and resource efficiency.

\sphinxAtStartPar
In Chapter 3, \hyperref[\detokenize{chapters/chapter03:id1}]{Fig.\@ \ref{\detokenize{chapters/chapter03:id1}}} shows a
high level Virtualised OpenStack services topology. The containerised
OpenStack services topology version is shown in
\hyperref[\detokenize{chapters/chapter04:containerised-openstack-services-topology}]{Fig.\@ \ref{\detokenize{chapters/chapter04:containerised-openstack-services-topology}}}.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch04-Containerised-OpenStack-Services-Stack}.png}
\caption{Containerised OpenStack Services Topology}\label{\detokenize{chapters/chapter04:containerised-openstack-services-topology}}\end{figure}


\subsection{Consumable Infrastructure Resources and Services}
\label{\detokenize{chapters/chapter04:consumable-infrastructure-resources-and-services}}

\subsubsection{Support for Cloud Infrastructure Profiles and flavors}
\label{\detokenize{chapters/chapter04:support-for-cloud-infrastructure-profiles-and-flavors}}
\sphinxAtStartPar
Reference Model Chapter 4 and 5 provide information about the Cloud
Infrastructure Profiles and their size information. OpenStack flavors
with their set of properties describe the server capabilities and size
required to determine the compute host which will run this server. The
set of properties must match compute profiles available in the
infrastructure. To implement these profiles and sizes, it is required to
set up the flavors as specified in the tables below.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Neutron Services Placement}\label{\detokenize{chapters/chapter04:id19}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{10}{80}|\X{15}{80}|\X{20}{80}|\X{35}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Flavor Capabilities
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Reference RM Chapter 4 and 5
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Basic
&\sphinxstyletheadfamily 
\sphinxAtStartPar
High\sphinxhyphen{}Performance
\\
\hline
\sphinxAtStartPar
CPU allocation ratio (custom extra\_specs)
&
\sphinxAtStartPar
infra.com.cfg.001
&
\sphinxAtStartPar
In flavor create or flavor set \textendash{}property cpu\_all ocation\_ratio=4.0
&
\sphinxAtStartPar
In flavor create or flavor set \textendash{}property cpu\_allocation\_ratio=1.0
\\
\hline
\sphinxAtStartPar
NUMA Awareness
&
\sphinxAtStartPar
infra.com.cfg.002
&&\begin{itemize}
\item {} 
\sphinxAtStartPar
In flavor create or flavor set specify \textendash{}property hw:numa\_nodes=\textless{}integer
range of 0 to \#numa\_nodes \textendash{} 1\textgreater{}.

\item {} 
\sphinxAtStartPar
To restrict an instance’s vCPUs to a
single host NUMA node, specify: \textendash{}property hw:numa\_nodes=1.

\item {} 
\sphinxAtStartPar
Some compute intensive* workloads with highly sensitive memory latency
or bandwidth requirements, the instance may benefit from spreading
across multiple NUMA nodes: \textendash{}property hw:numa\_nodes=2

\end{itemize}
\\
\hline
\sphinxAtStartPar
CPU Pinning
&
\sphinxAtStartPar
infra.com.cfg.003
&
\sphinxAtStartPar
In flavor create or flavor set specify \textendash{}property hw: cpu\_policy=shared
(default)
&\begin{itemize}
\item {} 
\sphinxAtStartPar
In flavor create or flavor set specify \textendash{}property
hw:cpu\_policy=dedicated and \textendash{}property hw:cpu\_thread\_policy=\textless{}prefer,
require, isolate\textgreater{}.

\item {} 
\sphinxAtStartPar
Use “isolate” thread policy for very high
compute intensive workloads that require that each vCPU be placed on a
different physical core

\end{itemize}
\\
\hline
\sphinxAtStartPar
Huge pages
&
\sphinxAtStartPar
infra.com.cfg.004
&&
\sphinxAtStartPar
\textendash{}property hw:mem\_page\_size=\textless{}small |large | size\textgreater{}
\\
\hline
\sphinxAtStartPar
SMT
&
\sphinxAtStartPar
infra.com.cfg.005
&&
\sphinxAtStartPar
In flavor create or flavor set specify \textendash{}property
hw:cpu\_threads=\textless{}integer\#threads (usually 1 or 2)\textgreater{}
\\
\hline
\sphinxAtStartPar
OVS\sphinxhyphen{}DPDK
&
\sphinxAtStartPar
infra.net.acc.cfg.001
&&
\sphinxAtStartPar
ml2.conf.ini configured to support {[}OVS{]} datapath\_type=netdev Note:
huge pages should be configured to large
\\
\hline
\sphinxAtStartPar
Local Storage SSD
&
\sphinxAtStartPar
infra.hw.stg.ssd.cfg.002
&
\sphinxAtStartPar
trait:STORAGEDISK\_SSD=required
&
\sphinxAtStartPar
trait:STORAGE\_DISK\_SSD=required
\\
\hline
\sphinxAtStartPar
Port speed
&
\sphinxAtStartPar
infra.hw.nic.cfg.002
&
\sphinxAtStartPar
\textendash{}property quota vif\_inbound\_average=1310720 and
vif\_outbound\_average=1310720.
Note:10 Gbps = 1250000 kilobytes per second
&
\sphinxAtStartPar
\textendash{}property quota vif\_inboundaverage=3125000 and
vif\_outbound\_average=3125000 Note: 25 Gbps = 3125000 kilobytes per second
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}
\begin{itemize}
\item {} 
\sphinxAtStartPar
To configure profile\sphinxhyphen{}extensions, for example, the “Storage
Intensive High Performance” profile, as defined in
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter02.html\#profile-extensions-specialisations}{Profile Extensions (specialisations)},
in addition to the above, need to configure the storage IOPS: the
following two parameters need to be specified in the flavor
create: \textendash{}property quota:disk\_write\_iops\_sec=\textless{}IOPS\#\textgreater{} and \textendash{}property
quota:disk\_read\_iops\_sec=\textless{}IOPS\#\textgreater{}.

\end{itemize}

\sphinxAtStartPar
The flavor create command and the mandatory and optional configuration
parameters is documented in
\sphinxurl{https://docs.openstack.org/nova/latest/user/flavors.html}.


\subsubsection{Logical segregation and high availability}
\label{\detokenize{chapters/chapter04:logical-segregation-and-high-availability}}
\sphinxAtStartPar
To ensure logical segregation and high availability, the architecture
will rely on the following principles:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Availability zone: provide resiliency and fault tolerance for VNF
deployments, by means of physical hosting distribution of compute
nodes in separate racks with separate power supply, in the same or
different DC room

\item {} 
\sphinxAtStartPar
Affinity\sphinxhyphen{}groups: allow tenants to make sure that VNFC instances are
on the same compute node or are on different compute nodes.

\end{itemize}

\sphinxAtStartPar
Note: The Cloud Infrastructure doesn’t provide any resiliency mechanisms
at the service level. Any server restart shall be triggered by the VNF
Manager instead of OpenStack:
\begin{itemize}
\item {} 
\sphinxAtStartPar
It doesn’t implement Instance High Availability which could allow
OpenStack Platform to automatically re\sphinxhyphen{}spawn instances on a different
compute node when their host compute node breaks.

\item {} 
\sphinxAtStartPar
Physical host reboot does not trigger automatic server recovery.

\item {} 
\sphinxAtStartPar
Physical host reboot does not trigger the automatic start of a
server.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Limitations and constraints}
\begin{itemize}
\item {} 
\sphinxAtStartPar
NUMA Overhead: isolated core will be used for overhead tasks from the
hypervisor.

\end{itemize}


\subsubsection{Transaction Volume Considerations}
\label{\detokenize{chapters/chapter04:transaction-volume-considerations}}
\sphinxAtStartPar
Storage transaction volumes impose a requirement on North\sphinxhyphen{}South network
traffic in and out of the storage backend. Data availability requires
that the data be replicated on multiple storage nodes and each new write
imposes East\sphinxhyphen{}West network traffic requirements.


\subsection{Cloud Topology and Control Plane Scenarios}
\label{\detokenize{chapters/chapter04:cloud-topology-and-control-plane-scenarios}}
\sphinxAtStartPar
Typically, Clouds have been implemented in large (central) data centres
with hundreds to tens of thousands of servers. Telco Operators have also
been creating intermediate data centres in central office locations,
colocation centres, and now edge centres at the physical edge of their
networks because of the demand for low latency and high throughput for
5G, IoT and connected devices (including autonomous driverless vehicles
and connected vehicles). Chapter 3.5 of this document, discusses
{\hyperref[\detokenize{chapters/chapter03:cloud-topology}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud Topology}}}}
and lists 3 types of data centres: Large, Intermediate and Edge.

\sphinxAtStartPar
For ease of convenience, unless specifically required, in this section
we will use Central Cloud Centre, Edge Cloud Centre and Intermediate
Cloud Centre as representative terms for cloud services hosted at
centralised large data centres, Telco edge locations and for locations
with capacity somewhere in between the large data centres and edge
locations, respectively. The mapping of various terms, including the
Reference Model terminology specified in Table \sphinxtitleref{8\sphinxhyphen{}5
:ref:\textasciigrave{}ref\_model/chapters/chapter08:comparison of deployment topologies and edge terms}
and \sphinxhref{https://github.com/State-of-the-Edge/glossary/blob/master/edge-glossary.md}{Open Glossary of Edge Computing}
is as follows:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Central Cloud Centre: Large Centralised Data Centre, Regional Data
Centre

\item {} 
\sphinxAtStartPar
Intermediate Cloud Centre: Metro Data Centre, Regional Edge,
Aggregation Edge

\item {} 
\sphinxAtStartPar
Edge Cloud Centre: Edge, Mini\sphinxhyphen{}/Micro\sphinxhyphen{}Edge, Micro Modular Data Centre,
Service Provider Edge, Access Edge, Aggregation Edge

\end{itemize}

\sphinxAtStartPar
In the Intermediate and Edge cloud centres, there may be limitations on
the resource capacity, as in the number of servers, and the capacity of
these servers in terms of \# of cores, RAM, etc. restricting the set of
services that can be deployed and, thus, creating a dependency between
other data centres. In \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter08.html\#telco-edge-cloud}{Telco Edge Cloud},
Table 8\sphinxhyphen{}5
specifies the physical and environmental characteristics, infrastructure
capabilities and deployment scenarios of different locations.

\sphinxAtStartPar
{\hyperref[\detokenize{chapters/chapter03:openstack-services-topology}]{\sphinxcrossref{\DUrole{std,std-ref}{OpenStack Services Topology}}}}
of this document, specifies the differences between the Control Plane and Data
Plane, and specifies which of the control nodes, compute nodes, storage
nodes (optional) and network nodes (optional) are components of these
planes. The previous sections of this Chapter 4 include a description of
the OpenStack services and their deployment in control nodes, compute
nodes, and optionally storage nodes and network nodes (rarely). The
Control Plane deployment scenarios determine the distribution of
OpenStack and other needed services among the different node types. This
section considers the Centralised Control Plane (CCP) and Distributed
Control Plane (DCP) scenarios. The choice of control plane and the cloud
centre resource capacity and capabilities determine the deployment of
OpenStack services in the different node types.

\sphinxAtStartPar
The Central Cloud Centres are organised around a Centralised Control
Plane. With the introduction of Intermediate and Edge Cloud Centres, the
Distributed Control Plane deployment becomes a possibility. A number of
independent control planes (sometimes referred to as Local Control
Planes (LCP)) exist in the Distributed Control Plane scenario, compared
with a single control plane in the Centralised Control Plane scenario.
Thus, in addition to the control plane and controller services deployed
at the Central Cloud Centre, Local Control Planes hosting a full\sphinxhyphen{}set or
subset of the controller services are also deployed on the Intermediate
and Edge Cloud Centres. Table 4\sphinxhyphen{}5 presents examples of such deployment
choices.


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Distribution of OpenStack services on different nodes
             depending upon Control Plane Scenario}\label{\detokenize{chapters/chapter04:id20}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{10}{80}|\X{10}{80}|\X{10}{80}|\X{10}{80}|\X{10}{80}|\X{10}{80}|\X{10}{80}|\X{10}{80}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Control Plane
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Deployed in
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Orchestration
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Identity Management
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Image Management
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Compute
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Network Management
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Storage Management
\\
\hline
\sphinxAtStartPar
CCP
&
\sphinxAtStartPar
Centralised DC \textendash{} control nodes
&
\sphinxAtStartPar
heat\sphinxhyphen{}api, heat\sphinxhyphen{}engine, nova\sphinxhyphen{}placement\sphinxhyphen{}api
&
\sphinxAtStartPar
Identity Provider (IdP), Keystone API
&
\sphinxAtStartPar
Glance API, Glance Registry
&
\sphinxAtStartPar
nova\sphinxhyphen{}compute api, nova\sphinxhyphen{}scheduler, nova\sphinxhyphen{}conductor
&
\sphinxAtStartPar
neutron\sphinxhyphen{}server, neutron\sphinxhyphen{}dhcp\sphinxhyphen{}agent, neutron\sphinxhyphen{}L2\sphinxhyphen{}agent,
neutron\sphinxhyphen{}L3\sphinxhyphen{}agent (optional), neutron\sphinxhyphen{}metadata \sphinxhyphen{}agent
&
\sphinxAtStartPar
Cinder API, Cinder Scheduler, Cinder Volume
\\
\hline
\sphinxAtStartPar
DCP: combination of services depending upon Center size
&
\sphinxAtStartPar
Any DC \sphinxhyphen{} Control nodes Option 1
&
\sphinxAtStartPar
heat\sphinxhyphen{}api, heat\sphinxhyphen{}engine, nova\sphinxhyphen{}placement\sphinxhyphen{}api
&
\sphinxAtStartPar
Identity Provider (IdP), Keystone API
&
\sphinxAtStartPar
Glance API, Glance Registry
&
\sphinxAtStartPar
nova\sphinxhyphen{}compute api, nova\sphinxhyphen{}scheduler, nova\sphinxhyphen{}conductor
&
\sphinxAtStartPar
neutron\sphinxhyphen{}server, neutron\sphinxhyphen{}dhcp\sphinxhyphen{}agent, neutron\sphinxhyphen{}L2\sphinxhyphen{}agent, neutron\sphinxhyphen{}L3\sphinxhyphen{}agent
(optional), neutron\sphinxhyphen{}metadata \sphinxhyphen{}agent
&
\sphinxAtStartPar
Cinder API, Cinder Scheduler, Cinder Volume
\\
\hline&
\sphinxAtStartPar
Any DC \sphinxhyphen{} Control nodes Option 2: split services between two or more DCs
&
\sphinxAtStartPar
in one of the DC
&
\sphinxAtStartPar
in the Large DC
&
\sphinxAtStartPar
in the Large DC
&
\sphinxAtStartPar
in one of the DC
&
\sphinxAtStartPar
in one of ther DC
&
\sphinxAtStartPar
in one of the DC
\\
\hline
\sphinxAtStartPar
CCP or DCP
&
\sphinxAtStartPar
Compute nodes
&&&&
\sphinxAtStartPar
nova\sphinxhyphen{}compute \sphinxhyphen{}agent
&
\sphinxAtStartPar
neutron\sphinxhyphen{}L2\sphinxhyphen{} agent, neutron\sphinxhyphen{}L3\sphinxhyphen{}agent (optional)
&\\
\hline
\sphinxAtStartPar
CCP
&
\sphinxAtStartPar
Compute nodes
&
\sphinxAtStartPar
nova\sphinxhyphen{}placement\sphinxhyphen{}api
&&&
\sphinxAtStartPar
nova\sphinxhyphen{}compute\sphinxhyphen{}agent, nova\sphinxhyphen{}conductor
&
\sphinxAtStartPar
neutron \sphinxhyphen{}server, neutron\sphinxhyphen{}dhcp\sphinxhyphen{}agent, neutron\sphinxhyphen{}L2\sphinxhyphen{}agent, neutron\sphinxhyphen{}L3\sphinxhyphen{}agent (optional)
&\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Edge Cloud Topology}
\label{\detokenize{chapters/chapter04:edge-cloud-topology}}
\sphinxAtStartPar
The Reference Model Chapter
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter08.html\#telco-edge-cloud}{Telco Edge Cloud},
presents the deployment environment characteristics, infrastructure
characteristics and new values for the Infrastructure Profiles at the Edge.

\sphinxAtStartPar
The \sphinxhref{https://www.openstack.org/use-cases/edge-computing/edge-computing-next-steps-in-architecture-design-and-testing/}{Edge computing
whitepaper}
includes information such as the services that run on various nodes. The
information from the whitepaper coupled with that from the \sphinxhref{https://fuel-ccp.readthedocs.io/en/latest/design/ref\_arch\_100\_nodes.html\#services-placement-summary}{OpenStack
Reference
Architecture}
for 100, 300 and 500 nodes will help in deciding which OpenStack and
other services (such as database, messaging) run on which nodes in what
Cloud Centre and the number of copies that should be deployed. These
references also present the pros and cons of DCP and CCP and designs to
address some of the challenges of each of the models.

\sphinxAtStartPar
Table 8\sphinxhyphen{}4 in the Reference Model
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter08.html\#telco-edge-cloud-platform-services-deployment}{Telco Edge Cloud: Platform Services Deployment}
lists the Platform Services that may be placed in the different node types
(control, compute and storage). Depending upon the capacity and
resources available only the compute nodes may exist at the Edge thereby
impacting operations.

\sphinxAtStartPar
Table 8\sphinxhyphen{}3 in the Reference Model Chapter
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter08.html\#telco-edge-cloud-infrastructure-profiles}{Telco Edge Cloud: Infrastructure Profiles}
lists a number of Infrastructure Profile characteristics and the changes that
may need to be made for certain Edge clouds depending upon their
resource capabilities. It should be noted that none of these changes
affect the definition of OpenStack flavors.

\sphinxAtStartPar
The previous section listed the OpenStack services deployed on the controller
nodes depending upon the control plane distribution. As specified earlier in
this chapter, at least 3 controller nodes should be deployed for HA.  Compute
nodes may also exist at the sites where controller nodes are deployed.

\sphinxAtStartPar
Control plane services are not hosted at edge sites. Each edge site can be
treated as its own OpenStack AZ. The compute nodes, will host \sphinxtitleref{nova\sphinxhyphen{}compute},
a component of the the Compute Service (Nova), and \sphinxtitleref{neutron\sphinxhyphen{}L2\sphinxhyphen{}agent},
a component of the Network Service (Neutron).

\sphinxAtStartPar
The Edge sites may or may not contain local storage. If the edge sites contain
storage, then the Block Storage  service (Cinder) is usually deployed to run
in an active/active mode with the centrally deployed Block Storage service.
Instance images are downloaded and stored locally; they can be downloaded even
prior to use.

\sphinxAtStartPar
If the edge site doesn’t contain storage then the images would need to be
cached from the central site. Two ptions exist:
\begin{itemize}
\item {} 
\sphinxAtStartPar
The instance images would be downloaded and
cached in the Nova cache on first use; they will then be available for
subsequent use.

\item {} 
\sphinxAtStartPar
Pre\sphinxhyphen{}caching the instance images for low time\sphinxhyphen{}to\sphinxhyphen{}boot latency. This has been supported
in Nova since the OpenStack Ussuri release.

\end{itemize}

\sphinxAtStartPar
Image caching and considerations for its use are discussed in the OpenStack document
\sphinxhref{https://docs.openstack.org/nova/latest/admin/image-caching.html\#image-pre-caching}{Image Caching}.


\paragraph{Edge Cloud Deployment Tools}
\label{\detokenize{chapters/chapter04:edge-cloud-deployment-tools}}
\sphinxAtStartPar
Deployment at the Edge requires support for large scale deployment. A
number of open\sphinxhyphen{}source tools are available for the purpose including:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{https://docs.airshipit.org/}{Airship}: declaratively configure,
deploy and maintain an integrated virtualisation and containerisation
platform

\item {} 
\sphinxAtStartPar
\sphinxhref{https://www.starlingx.io/}{Starling\sphinxhyphen{}X}: cloud infrastructure
software stack for the edge

\item {} 
\sphinxAtStartPar
\sphinxhref{https://wiki.openstack.org/wiki/TripleO}{Triple\sphinxhyphen{}O}: for
installing, upgrading and operating OpenStack clouds

\end{itemize}

\sphinxAtStartPar
These installers are described in more details in
{\hyperref[\detokenize{chapters/chapter07:operations-and-life-cycle-management}]{\sphinxcrossref{\DUrole{std,std-ref}{Operations and Life Cycle Management}}}}.


\section{Interfaces and APIs}
\label{\detokenize{chapters/chapter05:interfaces-and-apis}}\label{\detokenize{chapters/chapter05::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter05:introduction}}
\sphinxAtStartPar
This chapter presents a consolidated set of OpenStack Service APIs
corresponding to the ETSI NFV Nf\sphinxhyphen{}Vi, Vi\sphinxhyphen{}Vnfm and Or\sphinxhyphen{}Vi interfaces. The
OpenStack Wallaby version is used as the baseline for these APIs and
CLIs in this Reference Architecture (RA\sphinxhyphen{}1) version. Any Cloud
Infrastructure + VIM reference implementations that \sphinxstylestrong{get certified by
RC} can be considered as Anuket RA Conformant.

\sphinxAtStartPar
The Chapter presents the APIs for the core OpenStack services defined in
Chapter 3 and a consolidated view of these and other APIs that are of
interest.

\sphinxAtStartPar
OpenStack is a multi\sphinxhyphen{}project framework composed of independently
evolving services. It is not enough to rely only on the OpenStack
release to characterise the capabilities supported by these services.
Regarding OpenStack services APIs, an “API version” is associated with
each OpenStack service. In addition to major API versions, some
OpenStack services (Nova, Glance, Keystone, Cinder…) support
microversions. The microversions allow new features to be introduced
over time. In this chapter, the \sphinxstylestrong{major version} and \sphinxstylestrong{microversion}
are specified per service. The specified microversion is the minimal
microversion that supports the features requested for this RA. For the
purpose of conformance tests, this chapter also identifies the set of
features, offered by a service, that are mandatory for Anuket compliant
implementation.


\subsection{Core OpenStack Services APIs}
\label{\detokenize{chapters/chapter05:core-openstack-services-apis}}
\sphinxAtStartPar
Please note that OpenStack provides a maximum microversion to be used
with an OpenStack release. In the following sections the “Maximal API
Version” refers to this maximum microversion specified for the OpenStack
Wallaby release. Please note that in Reference Conformance (RC\sphinxhyphen{}1)
testing, the System Under Test (SUT) can utilise newer microversions
because of the OpenStack microversion policies. As per multiple
OpenStack services documentation, for example the \sphinxhref{https://docs.openstack.org/api-guide/compute/microversions.html}{Compute
Service},
“A cloud that is upgraded to support newer microversions will still
support all older microversions to maintain the backward compatibility
for those users who depend on older microversions.”


\subsubsection{Keystone}
\label{\detokenize{chapters/chapter05:keystone}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Keystone}\label{\detokenize{chapters/chapter05:id1}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Maximal API Microversion}
\\
\hline
\sphinxAtStartPar
Identity: Keystone
&
\sphinxAtStartPar
v3
&
\sphinxAtStartPar
3.14
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Keystone Features}\label{\detokenize{chapters/chapter05:id2}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Keystone Features}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\sphinxAtStartPar
access\_rules
&\\
\hline
\sphinxAtStartPar
application\_credentials
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
external\_idp
&\\
\hline
\sphinxAtStartPar
federation
&\\
\hline
\sphinxAtStartPar
oauth1
&\\
\hline
\sphinxAtStartPar
project\_tags
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
security\_compliance
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
trust
&
\sphinxAtStartPar
X
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Identity API v3:
\sphinxurl{https://docs.openstack.org/api-ref/identity/v3/index.html}

\sphinxAtStartPar
Identity API v3 extensions:
\sphinxurl{https://docs.openstack.org/api-ref/identity/v3-ext/}

\sphinxAtStartPar
Security compliance and PCI\sphinxhyphen{}DSS:
\sphinxurl{https://docs.openstack.org/keystone/train/admin/configuration.html\#security-compliance-and-pci-dss}


\subsubsection{Glance}
\label{\detokenize{chapters/chapter05:glance}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Glance}\label{\detokenize{chapters/chapter05:id3}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Maximal API Microversion}
\\
\hline
\sphinxAtStartPar
Image: Glance
&
\sphinxAtStartPar
v2
&
\sphinxAtStartPar
2.9
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Glance Features}\label{\detokenize{chapters/chapter05:id4}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Glance Features}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\sphinxAtStartPar
import\_image
&\\
\hline
\sphinxAtStartPar
os\_glance\_reserved
&\\
\hline
\sphinxAtStartPar
web\sphinxhyphen{}download import
&\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Image Service Versions:
\sphinxurl{https://docs.openstack.org/api-ref/image/versions/index.html\#version-history}


\subsubsection{Cinder}
\label{\detokenize{chapters/chapter05:cinder}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Cinder}\label{\detokenize{chapters/chapter05:id5}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Maximal API Microversion}
\\
\hline
\sphinxAtStartPar
Block Storage: Cinder
&
\sphinxAtStartPar
v3
&
\sphinxAtStartPar
3.64
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Cinder Features}\label{\detokenize{chapters/chapter05:id6}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Cinder Features}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\sphinxAtStartPar
backup
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
clone
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
consistency\_group
&\\
\hline
\sphinxAtStartPar
extend\_attached\_volume
&\\
\hline
\sphinxAtStartPar
extend\_attached\_encrypted\_volume
&\\
\hline
\sphinxAtStartPar
manage\_snapshot
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
manage\_volume
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
multi\_backend
&\\
\hline
\sphinxAtStartPar
snapshot
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
volume\_revert
&
\sphinxAtStartPar
X
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Block Storage API: \sphinxurl{https://docs.openstack.org/api-ref/block-storage/}

\sphinxAtStartPar
REST API Version History:
\sphinxurl{https://docs.openstack.org/cinder/latest/contributor/api\_microversion\_history.html}


\subsubsection{Swift}
\label{\detokenize{chapters/chapter05:swift}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Swift}\label{\detokenize{chapters/chapter05:id7}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
\\
\hline
\sphinxAtStartPar
Object Storage: Swift
&
\sphinxAtStartPar
v1
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Swift Features}\label{\detokenize{chapters/chapter05:id8}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Swift Features}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\sphinxAtStartPar
account\_quotas
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
bulk\_delete
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
bulk\_upload
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
container\_quotas
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
container\_sync
&\\
\hline
\sphinxAtStartPar
crossdomain
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
discoverability
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
form\_post
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
ratelimit
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
s3api
&\\
\hline
\sphinxAtStartPar
slo
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
staticweb
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
symlink
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
temp\_url
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
tempauth
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
versioned\_writes
&
\sphinxAtStartPar
X
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Object Storage API:
\sphinxurl{https://docs.openstack.org/api-ref/object-store/index.html}

\sphinxAtStartPar
Discoverability:
\sphinxurl{https://docs.openstack.org/swift/latest/api/discoverability.html}


\subsubsection{Neutron}
\label{\detokenize{chapters/chapter05:neutron}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Neutron}\label{\detokenize{chapters/chapter05:id9}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
\\
\hline
\sphinxAtStartPar
Networking: Neutron
&
\sphinxAtStartPar
v2.0
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\begin{savenotes}\sphinxatlongtablestart\begin{longtable}[c]{|l|l|}
\sphinxthelongtablecaptionisattop
\caption{Neutron Extensions\strut}\label{\detokenize{chapters/chapter05:id10}}\\*[\sphinxlongtablecapskipadjust]
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Neutron Extensions}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\endfirsthead

\multicolumn{2}{c}%
{\makebox[0pt]{\sphinxtablecontinued{\tablename\ \thetable{} \textendash{} continued from previous page}}}\\
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Neutron Extensions}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\endhead

\hline
\multicolumn{2}{r}{\makebox[0pt][r]{\sphinxtablecontinued{continues on next page}}}\\
\endfoot

\endlastfoot

\sphinxAtStartPar
address\sphinxhyphen{}scope
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
agent
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
allowed\sphinxhyphen{}address\sphinxhyphen{}pairs
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
auto\sphinxhyphen{}allocated\sphinxhyphen{}topology
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
availability\_zone
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
availability\_zone\_filter
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
binding
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
binding\sphinxhyphen{}extended
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
default\sphinxhyphen{}subnetpools
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
dhcp\_agent\_scheduler
&\\
\hline
\sphinxAtStartPar
dns\sphinxhyphen{}domain\sphinxhyphen{}ports
&\\
\hline
\sphinxAtStartPar
dns\sphinxhyphen{}integration
&\\
\hline
\sphinxAtStartPar
dvr
&\\
\hline
\sphinxAtStartPar
empty\sphinxhyphen{}string\sphinxhyphen{}filtering
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
ext\sphinxhyphen{}gw\sphinxhyphen{}mode
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
external\sphinxhyphen{}net
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
extra\_dhcp\_opt
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
extraroute
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
extraroute\sphinxhyphen{}atomic
&\\
\hline
\sphinxAtStartPar
flavors
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
filter\sphinxhyphen{}validation
&\\
\hline
\sphinxAtStartPar
fip\sphinxhyphen{}port\sphinxhyphen{}details
&\\
\hline
\sphinxAtStartPar
floating\sphinxhyphen{}ip\sphinxhyphen{}port\sphinxhyphen{}forwarding
&\\
\hline
\sphinxAtStartPar
floatingip\sphinxhyphen{}pools
&\\
\hline
\sphinxAtStartPar
ip\sphinxhyphen{}substring\sphinxhyphen{}filtering
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
l3\_agent\_scheduler
&\\
\hline
\sphinxAtStartPar
l3\sphinxhyphen{}flavors
&\\
\hline
\sphinxAtStartPar
l3\sphinxhyphen{}ha
&\\
\hline
\sphinxAtStartPar
logging
&\\
\hline
\sphinxAtStartPar
metering
&\\
\hline
\sphinxAtStartPar
multi\sphinxhyphen{}provider
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
net\sphinxhyphen{}mtu
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
net\sphinxhyphen{}mtu\sphinxhyphen{}writable
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
network\_availability\_zone
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
network\sphinxhyphen{}ip\sphinxhyphen{}availability
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
network\sphinxhyphen{}segment\sphinxhyphen{}range
&\\
\hline
\sphinxAtStartPar
pagination
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
port\sphinxhyphen{}mac\sphinxhyphen{}address\sphinxhyphen{}regenerate
&\\
\hline
\sphinxAtStartPar
port\sphinxhyphen{}resource\sphinxhyphen{}request
&\\
\hline
\sphinxAtStartPar
port\sphinxhyphen{}security
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
port\sphinxhyphen{}security\sphinxhyphen{}groups\sphinxhyphen{}filtering
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
project\sphinxhyphen{}id
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
provider
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
rbac\sphinxhyphen{}policies
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
router
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
router\_availability\_zone
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos\sphinxhyphen{}bw\sphinxhyphen{}limit\sphinxhyphen{}direction
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos\sphinxhyphen{}bw\sphinxhyphen{}minimum\sphinxhyphen{}ingress
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos\sphinxhyphen{}default
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos\sphinxhyphen{}fip
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos\sphinxhyphen{}gateway\sphinxhyphen{}ip
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos\sphinxhyphen{}rule\sphinxhyphen{}type\sphinxhyphen{}details
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
qos\sphinxhyphen{}rules\sphinxhyphen{}alias
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
quotas
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
quota\_details
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
revision\sphinxhyphen{}if\sphinxhyphen{}match
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
rbac\sphinxhyphen{}address\sphinxhyphen{}scope
&\\
\hline
\sphinxAtStartPar
rbac\sphinxhyphen{}security\sphinxhyphen{}groups
&\\
\hline
\sphinxAtStartPar
rbac\sphinxhyphen{}subnetpool
&\\
\hline
\sphinxAtStartPar
router\sphinxhyphen{}interface\sphinxhyphen{}fip
&\\
\hline
\sphinxAtStartPar
security\sphinxhyphen{}group
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
service\sphinxhyphen{}type
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
sorting
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
standard\sphinxhyphen{}attr\sphinxhyphen{}description
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
standard\sphinxhyphen{}attr\sphinxhyphen{}revisions
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
standard\sphinxhyphen{}attr\sphinxhyphen{}tag
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
standard\sphinxhyphen{}attr\sphinxhyphen{}timestamp
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
subnet\_allocation
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
subnet\sphinxhyphen{}service\sphinxhyphen{}types
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
subnetpool\sphinxhyphen{}prefix\sphinxhyphen{}ops
&\\
\hline
\sphinxAtStartPar
tag\sphinxhyphen{}ext
&\\
\hline
\sphinxAtStartPar
tag\sphinxhyphen{}ports\sphinxhyphen{}during\sphinxhyphen{}bulk\sphinxhyphen{}creation
&\\
\hline
\sphinxAtStartPar
trunk
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
trunk\sphinxhyphen{}details
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
uplink\sphinxhyphen{}status\sphinxhyphen{}propagation
&\\
\hline
\end{longtable}\sphinxatlongtableend\end{savenotes}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Neutron Type Drivers}\label{\detokenize{chapters/chapter05:id11}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Neutron Type Drivers}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\sphinxAtStartPar
geneve
&\\
\hline
\sphinxAtStartPar
gre
&\\
\hline
\sphinxAtStartPar
vlan
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
vxlan
&\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Networking Service APIs: \sphinxurl{https://docs.openstack.org/api-ref/network/}

\sphinxAtStartPar
The exhaustive list of extensions is available at
\sphinxurl{https://docs.openstack.org/api-ref/network/v2/}


\subsubsection{Nova}
\label{\detokenize{chapters/chapter05:nova}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Nova}\label{\detokenize{chapters/chapter05:id12}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Maximal API Microversion}
\\
\hline
\sphinxAtStartPar
Compute: Nova
&
\sphinxAtStartPar
v2.1
&
\sphinxAtStartPar
2.88
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Nova Features}\label{\detokenize{chapters/chapter05:id13}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Nova Features}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Mandatory}
\\
\hline
\sphinxAtStartPar
attach\_encrypted\_volume
&\\
\hline
\sphinxAtStartPar
cert
&\\
\hline
\sphinxAtStartPar
change\_password
&\\
\hline
\sphinxAtStartPar
cold\_migration
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
console\_output
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
disk\_config
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
instance\_password
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
interface\_attach
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
live\_migration
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
metadata\_service
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
pause
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
personality
&\\
\hline
\sphinxAtStartPar
rdp\_console
&\\
\hline
\sphinxAtStartPar
rescue
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
resize
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
serial\_console
&\\
\hline
\sphinxAtStartPar
shelve
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
shelve\_migrate
&\\
\hline
\sphinxAtStartPar
snapshot
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
stable\_rescue
&\\
\hline
\sphinxAtStartPar
spice\_console
&\\
\hline
\sphinxAtStartPar
suspend
&
\sphinxAtStartPar
X
\\
\hline
\sphinxAtStartPar
swap\_volume
&\\
\hline
\sphinxAtStartPar
vnc\_console
&\\
\hline
\sphinxAtStartPar
volume\_multiattach
&\\
\hline
\sphinxAtStartPar
xenapi\_apis
&\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Compute API: \sphinxurl{https://docs.openstack.org/api-ref/compute/}

\sphinxAtStartPar
REST API Version History:
\sphinxurl{https://docs.openstack.org/nova/latest/reference/api-microversion-history.html}


\subsubsection{Placement}
\label{\detokenize{chapters/chapter05:placement}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Placement}\label{\detokenize{chapters/chapter05:id14}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Maximal API Microversion}
\\
\hline
\sphinxAtStartPar
Placement
&
\sphinxAtStartPar
v1
&
\sphinxAtStartPar
1.36
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Placement API: \sphinxurl{https://docs.openstack.org/api-ref/placement/}

\sphinxAtStartPar
REST API Version History:
\sphinxurl{https://docs.openstack.org/placement/latest/placement-api-microversion-history.html}


\subsubsection{Heat}
\label{\detokenize{chapters/chapter05:heat}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Heat}\label{\detokenize{chapters/chapter05:id15}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{Maximal Template Version}
\\
\hline
\sphinxAtStartPar
Orchestration: Heat
&
\sphinxAtStartPar
v1
&
\sphinxAtStartPar
2021\sphinxhyphen{}04\sphinxhyphen{}16
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Orchestration Service API:
\sphinxurl{https://docs.openstack.org/api-ref/orchestration/}

\sphinxAtStartPar
Template version history:
\sphinxurl{https://docs.openstack.org/heat/latest/template\_guide/hot\_spec.html}

\sphinxAtStartPar
Heat Orchestration Template (HOT) specification:
\sphinxurl{https://docs.openstack.org/heat/latest/template\_guide/hot\_spec.html\#rocky}


\subsection{Consolidated Set of APIs}
\label{\detokenize{chapters/chapter05:consolidated-set-of-apis}}

\subsubsection{OpenStack Interfaces}
\label{\detokenize{chapters/chapter05:openstack-interfaces}}
\sphinxAtStartPar
This section illustrates some of the Interfaces provided by OpenStack;
the exhaustive list of APIs is available at
\sphinxurl{https://docs.openstack.org/api-ref/}.

\sphinxAtStartPar
OpenStack REST APIs are simple to interact with using either of two
options. Clients can either call the APIs directly using the HTTP or
REST library, or they can use one of the many cloud specific programming
language libraries.

\sphinxAtStartPar
\sphinxstylestrong{APIs}


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{APIs}\label{\detokenize{chapters/chapter05:id16}}
\sphinxaftertopcaption
\begin{tabular}[t]{|\X{20}{100}|\X{50}{100}|\X{15}{100}|\X{15}{100}|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
OpenStack Service
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Link for API list
&\sphinxstyletheadfamily 
\sphinxAtStartPar
API Version
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Maximal API Microversion
\\
\hline
\sphinxAtStartPar
Identity: Keystone
&
\sphinxAtStartPar
\sphinxurl{https://docs.openstack.org/api-ref/identity/v3/}
&
\sphinxAtStartPar
v3
&
\sphinxAtStartPar
3.14
\\
\hline
\sphinxAtStartPar
Compute: Nova
&
\sphinxAtStartPar
\sphinxurl{https://docs.openstack.org/api-ref/compute/}
&
\sphinxAtStartPar
v2.1
&
\sphinxAtStartPar
2.88
\\
\hline
\sphinxAtStartPar
Networking: Neutron
&
\sphinxAtStartPar
\sphinxurl{https://docs.openstack.org/api-ref/network/v2/}
&
\sphinxAtStartPar
v2.0
&\\
\hline
\sphinxAtStartPar
Image: Glance
&
\sphinxAtStartPar
\sphinxurl{https://docs.openstack.org/api-ref/image/v2/}
&
\sphinxAtStartPar
v2
&
\sphinxAtStartPar
2.9
\\
\hline
\sphinxAtStartPar
Block Storage: Cinder
&
\sphinxAtStartPar
\sphinxurl{https://docs.openstack.org/api-ref/block-storage/v3/}
&
\sphinxAtStartPar
v3
&
\sphinxAtStartPar
3.64
\\
\hline
\sphinxAtStartPar
Placement
&
\sphinxAtStartPar
\sphinxurl{https://docs.openstack.org/api-ref/placement/}
&
\sphinxAtStartPar
v1
&
\sphinxAtStartPar
1.36
\\
\hline
\sphinxAtStartPar
Orchestration: Heat
&
\sphinxAtStartPar
\sphinxurl{https://docs.openstack.org/api-ref/orchestration/v1/}
&
\sphinxAtStartPar
v1
&
\sphinxAtStartPar
2021\sphinxhyphen{}04\sphinxhyphen{}06 (template)
\\
\hline
\end{tabular}
\par
\sphinxattableend\end{savenotes}


\subsubsection{Kubernetes Interfaces}
\label{\detokenize{chapters/chapter05:kubernetes-interfaces}}
\sphinxAtStartPar
The Kubernetes APIs are available at
\sphinxurl{https://kubernetes.io/docs/concepts/overview/kubernetes-api/}.


\subsubsection{KVM Interfaces}
\label{\detokenize{chapters/chapter05:kvm-interfaces}}
\sphinxAtStartPar
The KVM APIs are documented in Section 4 of the document
\sphinxurl{https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt}.


\paragraph{Libvirt Interfaces}
\label{\detokenize{chapters/chapter05:libvirt-interfaces}}
\sphinxAtStartPar
The Libvirt APIs are documented in \sphinxurl{https://libvirt.org/html/index.html}.


\subsubsection{Barbican}
\label{\detokenize{chapters/chapter05:barbican}}

\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Barbican}\label{\detokenize{chapters/chapter05:id17}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{OpenStack Service}
&\sphinxstyletheadfamily 
\sphinxAtStartPar
\sphinxstylestrong{API Version}
\\
\hline
\sphinxAtStartPar
Key Manager: Barbican
&
\sphinxAtStartPar
v1
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
Barbican API Documentation:
\sphinxurl{https://docs.openstack.org/barbican/latest/api/}


\section{Security}
\label{\detokenize{chapters/chapter06:security}}\label{\detokenize{chapters/chapter06::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter06:introduction}}
\sphinxAtStartPar
This guide is intended to provide basic security requirements to
architects who are implementing Cloud Infrastructure using
\sphinxhref{https://www.openstack.org/}{OpenStack} technology. This is a minimal
set of high\sphinxhyphen{}level general security practices, not intended to cover all
implementation scenarios. Please ensure to also reference your
enterprise security and compliance requirements in addition to this
guide.


\subsection{Security Requirements}
\label{\detokenize{chapters/chapter06:security-requirements}}
\sphinxAtStartPar
Chapter 2 ({\hyperref[\detokenize{chapters/chapter02:cloud-infrastructure-security-requirements}]{\sphinxcrossref{\DUrole{std,std-ref}{Cloud Infrastructure Security Requirements}}}}
and {\hyperref[\detokenize{chapters/chapter02:security-recommendations}]{\sphinxcrossref{\DUrole{std,std-ref}{Security Recommendations}}}}) gathers
all requirements and recommendations regarding security topics developed
in this chapter.


\subsection{Cloud Infrastructure and VIM Security}
\label{\detokenize{chapters/chapter06:cloud-infrastructure-and-vim-security}}
\sphinxAtStartPar
In the “\sphinxhref{https://docs.openstack.org/security-guide/introduction/security-boundaries-and-threats.html}{Security boundaries and
threats}”
section of the \sphinxhref{https://docs.openstack.org/security-guide/introduction/introduction-to-openstack.html}{OpenStack security
guide},
there is extensive description on security domains, threat
classifications, and attack vectors. The following only touches on some
of the topics and at a high level.


\subsubsection{System Hardening}
\label{\detokenize{chapters/chapter06:system-hardening}}
\sphinxAtStartPar
All infrastructure components should undergo system hardening, establish
processes to govern the hardening, and documents to cover at a minimal
for the following areas.


\paragraph{Server boot hardening}
\label{\detokenize{chapters/chapter06:server-boot-hardening}}
\sphinxAtStartPar
Server boot process must be trusted. For this purpose, the integrity and
authenticity of all BIOS firmware components must be verified at boot.
Per sec.gen.003 requirement, Secure Boot based on UEFI must be used. By
verifying the signatures of all BIOS components, Secure Boot will ensure
that servers start with the firmware expected and without malware
insertion into the system. Secure Boot checks the digital signatures
locally. To implement a chain of trust, Secure Boot must be complemented
by the use of a hardware based Root of Trust provided by a TPM (Trusted
Platform Module).


\paragraph{System Access}
\label{\detokenize{chapters/chapter06:system-access}}
\sphinxAtStartPar
Access to all the platform’s components must be restricted (sec.gen.013)
applying the following rules:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Remove, or at a minimal, disable all unnecessary user accounts

\item {} 
\sphinxAtStartPar
Change all default user accounts where technically feasible

\item {} 
\sphinxAtStartPar
Change all default credentials

\item {} 
\sphinxAtStartPar
Prohibit logging with root account when root privileges are not
required (sec.gen.006)

\item {} 
\sphinxAtStartPar
Restrict access according to only those protocols/service/address
adhering to the Principle of Least Privilege

\item {} 
\sphinxAtStartPar
The same authentication credentials must not be reused on different
components (sec.sys.011)

\item {} 
\sphinxAtStartPar
Restrict access to Operating System (sec.gen.005)

\end{itemize}


\paragraph{Password policy}
\label{\detokenize{chapters/chapter06:password-policy}}
\sphinxAtStartPar
For all infrastructure components, passwords must be hardened, and a
strict password policy must be applied (sec.gen.002).

\sphinxAtStartPar
Passwords must be strengthened:
\begin{itemize}
\item {} 
\sphinxAtStartPar
All vendors default passwords must be changed

\item {} 
\sphinxAtStartPar
Passwords must contain at least 8 characters as a minimal value, 14
characters length passwords are recommended

\item {} 
\sphinxAtStartPar
Passwords must contain at least one upper case letter, one lower case
letter and one non\sphinxhyphen{}alphabetic character

\item {} 
\sphinxAtStartPar
For administration privileges accounts, passwords must contain at
least one upper case letter, one lower case letter, one numeral and
one special (non\sphinxhyphen{}alphanumeric) character

\end{itemize}

\sphinxAtStartPar
For passwords updates, the identity of users must be verified before
permitting a password change.

\sphinxAtStartPar
Passwords must be encrypted at rest and in\sphinxhyphen{}transit. Password files must
be stored separately from application system data.

\sphinxAtStartPar
Password’s composition, complexity and policy should follow the
recommendations consolidated within the \sphinxhref{https://www.cisecurity.org/white-papers/cis-password-policy-guide/}{CIS Password Policy
guide}
such as:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Check the password for known bad passwords (repetitive or sequential
characters, dictionary words, context\sphinxhyphen{}specific words, previously used
passwords, etc.)

\item {} 
\sphinxAtStartPar
Limit number of failed login attempts

\item {} 
\sphinxAtStartPar
Implement Multi\sphinxhyphen{}factor Authentication

\item {} 
\sphinxAtStartPar
Periodic (for example, Yearly, Quarterly, etc.) password change or on
key events such as indication of compromise, change of user roles, a
defined period of inactivity, when a user leaves the organisation,
etc.

\end{itemize}


\paragraph{Function and Software}
\label{\detokenize{chapters/chapter06:function-and-software}}
\sphinxAtStartPar
Infrastructure must be implemented to perform the minimal functions
needed to operate the Cloud Infrastructure.

\sphinxAtStartPar
Regarding software (sec.gen.004):
\begin{itemize}
\item {} 
\sphinxAtStartPar
Install only software which is required to support the functions

\item {} 
\sphinxAtStartPar
Remove any unnecessary software or packages

\item {} 
\sphinxAtStartPar
Where software cannot be removed, disable all services to it

\end{itemize}


\paragraph{Patches}
\label{\detokenize{chapters/chapter06:patches}}
\sphinxAtStartPar
All deployed Cloud Infrastructure software must be audited and must be
implemented to allow installation of the latest patches to address
security vulnerabilities in the following timescale from discovery
(sec.gen.008, sec.lcm.011):


\begin{savenotes}\sphinxattablestart
\centering
\sphinxcapstartof{table}
\sphinxthecaptionisattop
\sphinxcaption{Time to Remediate}\label{\detokenize{chapters/chapter06:id1}}
\sphinxaftertopcaption
\begin{tabulary}{\linewidth}[t]{|T|T|}
\hline
\sphinxstyletheadfamily 
\sphinxAtStartPar
Severity
&\sphinxstyletheadfamily 
\sphinxAtStartPar
Time to Remediate
\\
\hline
\sphinxAtStartPar
Zero\sphinxhyphen{}Day
&
\sphinxAtStartPar
Immediately or as soon
as practically possible
\\
\hline
\sphinxAtStartPar
Critical
&
\sphinxAtStartPar
30 days
\\
\hline
\sphinxAtStartPar
High
&
\sphinxAtStartPar
60 days
\\
\hline
\sphinxAtStartPar
Medium
&
\sphinxAtStartPar
90 days
\\
\hline
\sphinxAtStartPar
Low
&
\sphinxAtStartPar
180 days
\\
\hline
\end{tabulary}
\par
\sphinxattableend\end{savenotes}

\sphinxAtStartPar
\sphinxstylestrong{See} \sphinxhref{https://cve.mitre.org/}{Common Vulnerability Scoring System}
and \sphinxhref{https://nvd.nist.gov/vuln-metrics/cvss}{NIST Vulnerability
Metrics}.


\paragraph{Network Protocols}
\label{\detokenize{chapters/chapter06:network-protocols}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Only allow protocols that are required by the system functions
(sec.sys.002)

\item {} 
\sphinxAtStartPar
Tighten all required TCP/IP (Transmission Control Protocol/Internet
Protocol) services

\end{itemize}


\paragraph{Anti\sphinxhyphen{}Virus and Firewall}
\label{\detokenize{chapters/chapter06:anti-virus-and-firewall}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Install and run your Enterprise approved anti\sphinxhyphen{}virus software/
intrusion protection/ malware/ spyware endpoint security software
with up\sphinxhyphen{}to\sphinxhyphen{}date profiles; minimal daily refresh

\item {} 
\sphinxAtStartPar
Install and run firewall software where applicable

\end{itemize}


\paragraph{Vulnerability Detection and Prevention}
\label{\detokenize{chapters/chapter06:vulnerability-detection-and-prevention}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Implement DoS (Denial of Service) protection where applicable

\item {} 
\sphinxAtStartPar
Ensure logging and alerting is actively running

\item {} 
\sphinxAtStartPar
Run host\sphinxhyphen{}based scanning and fix all findings per vulnerability
severity

\item {} 
\sphinxAtStartPar
Run network\sphinxhyphen{}based scanning and fix all findings per vulnerability
severity

\end{itemize}


\subsubsection{Platform Access}
\label{\detokenize{chapters/chapter06:platform-access}}

\paragraph{Identity Security}
\label{\detokenize{chapters/chapter06:identity-security}}
\sphinxAtStartPar
The \sphinxhref{https://docs.openstack.org/security-guide/identity.html}{OpenStack Identity service
(Keystone)}
provides identity, token, catalog, and policy services for use
specifically by services in the OpenStack family. Identity service is
organised as a group of internal services exposed on one or many
endpoints. Many of these services are used in a combined fashion by the
front end (sec.sys.006).

\sphinxAtStartPar
OpenStack Keystone can work with an Identity service that your
enterprise may already have, such as LDAP with Active Directory. In
those cases, the recommendation is to integrate Keystone with the cloud
provider’s Identity Services.


\paragraph{Authentication}
\label{\detokenize{chapters/chapter06:authentication}}
\sphinxAtStartPar
Authentication is the first line of defence for any real\sphinxhyphen{}world
implementation of OpenStack. At its core, authentication is the process
of confirming the user logging in is who they claim to be. OpenStack
Keystone supports multiple methods of authentication, such as
username/password, LDAP, and others. For more details, please refer to
\sphinxhref{https://docs.openstack.org/security-guide/identity/authentication-methods.html}{OpenStack Authentication
Methods}.

\sphinxAtStartPar
Limiting the number of repeated failed login attempts (configurable)
reduces the risk of unauthorised access via password guessing (Bruce
force attack) \sphinxhyphen{} sec.mon.006. The restriction on the number of
consecutive failed login attempts (“lockout\_failure\_attempts”) and any
actions post such access attempts (such as locking the account where the
“lockout\_duration” is left unspecified) should abide by the operator’s
policies. For example, an operator may restrict the number of
consecutive failed login attempts to 3 (“lockout\_failure\_attempts = 3”)
and lock the account preventing any further access and where the account
is unlocked by getting necessary approvals.


\subparagraph{Keystone Tokens}
\label{\detokenize{chapters/chapter06:keystone-tokens}}
\sphinxAtStartPar
Once a user is authenticated, a token is generated for authorisation and
access to an OpenStack environment and resources. By default, the token
is set to expire in one hour. This setting can be changed based on the
business and operational needs, but it’s highly recommended to set the
expiration to the shortest possible value without dramatically impacting
your operations.

\sphinxAtStartPar
\sphinxstylestrong{Special Note on Logging Tokens:} since the token would allow access
to the OpenStack services, it \sphinxstyleemphasis{MUST} be masked before outputting to any
logs.


\paragraph{Authorisation}
\label{\detokenize{chapters/chapter06:authorisation}}
\sphinxAtStartPar
Authorisation serves as the next level of defence. At its core, it
checks if the authenticated users have the permission to execute an
action. Most Identity Services support the notion of groups and roles. A
user belongs to groups and each group has a list of roles that permits
certain actions on certain resources. OpenStack services reference the
roles of the user attempting to access the service. OpenStack policy
enforcer middleware takes into consideration the policy rules associated
with each resource and the user’s group/roles and association to
determine if access will be permitted for the requested resource. For
more details on policies, please refer to the \sphinxhref{https://docs.openstack.org/security-guide/identity/policies.html\#policy-section}{OpenStack
Policies}.


\paragraph{RBAC}
\label{\detokenize{chapters/chapter06:rbac}}
\sphinxAtStartPar
In order to properly manage user access to OpenStack services, service
providers must utilise the Role Based Access Control (RBAC) system
(sec.sys.001, sec.sys.007). Based on the OpenStack Identify Service
(Keystone v3) Group and Domain component, the RBAC system implements a
set of access roles that accommodate most use cases. Operations staff
can create users and assign them to roles using standard OpenStack
commands for users, groups, and roles.

\sphinxAtStartPar
Keystone provides three \sphinxhref{https://docs.openstack.org/keystone/latest/admin/service-api-protection.html}{default
roles}:
admin, member, and reader. As of Train release, Keystone applies the
following personas consistently across its API.
\begin{itemize}
\item {} 
\sphinxAtStartPar
The reader role provides read\sphinxhyphen{}only access to resources within the
system, a domain, or a project.

\item {} 
\sphinxAtStartPar
The member role is the same as reader in Keystone, but allows to
introduce granularity between admin and reader to other OpenStack
services.

\item {} 
\sphinxAtStartPar
The admin role is reserved for the most privileged operations within
a given scope for managing resources.

\end{itemize}

\sphinxAtStartPar
For specific use\sphinxhyphen{}case, policies can be overridden, and new roles can be
created for each OpenStack service by editing the policy.json file.


\subparagraph{Rules}
\label{\detokenize{chapters/chapter06:rules}}
\sphinxAtStartPar
The following rules govern create, read, update, and delete (CRUD) level
access.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{member} can create, read, update, and delete the resources defined
at the tenant level.

\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{support\_member} can create and read the resources defined at the
tenant level.

\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{viewer} can read the resources defined at the tenant level.

\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{admin} can create, read, update, and delete all resources.

\end{itemize}


\subparagraph{Recommended Default Roles to Start}
\label{\detokenize{chapters/chapter06:recommended-default-roles-to-start}}
\sphinxAtStartPar
\sphinxstylestrong{site\_admin} (HIGHLY RESTRICTED)
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Site Level Super Admin} \sphinxhyphen{} usually assign to Operation Staffs who
already have root level access to hosts

\item {} 
\sphinxAtStartPar
Permission to create/read/update/delete all tenants and resources at
the site, including creating snapshot and upload public images

\item {} 
\sphinxAtStartPar
Limited ability to create/read/update/delete tenant projects

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{site\_admin\_support}
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Site Level Admin} \sphinxhyphen{} usually assign to Operation Staffs who need to
manage resource except delete

\item {} 
\sphinxAtStartPar
Permission to create/read/update all tenants and resources at the
site

\item {} 
\sphinxAtStartPar
Cannot create snapshots

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{site\_admin\_viewer}
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Site Level Admin Read Only} \sphinxhyphen{} usually assign to groups who need to
view all resources, such as Capacity Planners

\item {} 
\sphinxAtStartPar
Permission to read all tenants and resources at the site

\item {} 
\sphinxAtStartPar
Cannot create/update/delete

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{site\_image\_manager}
\begin{itemize}
\item {} 
\sphinxAtStartPar
Site wide admin level privileges to Glance API (via CLI)

\item {} 
\sphinxAtStartPar
Restricted to Image team

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{tenant\_member}
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Tenant Level Admin} \sphinxhyphen{} typically assign to majority of tenant users
to manage their resources

\item {} 
\sphinxAtStartPar
Permission to create/read/update/delete to all resources at the
tenant project level

\item {} 
\sphinxAtStartPar
Cannot upload image or create snapshot

\item {} 
\sphinxAtStartPar
Cannot touch any other tenant except the one the role is located

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{tenant\_snapshot\_member}
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Tenant Level Admin with Snapshot} \sphinxhyphen{} typically assign to tenant users
who need to create snapshot via special request to Operations Staff

\item {} 
\sphinxAtStartPar
Permission is same as tenant\_member except the user can also create
snapshots

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{tenant\_support\_member}
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Tenant Level Support} \sphinxhyphen{} typically assign to tenant users who need to
create resource in the project space

\item {} 
\sphinxAtStartPar
Permission to create/read all resources at the tenant project level

\item {} 
\sphinxAtStartPar
Cannot update/delete or create snapshots

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{tenant\_viewer}
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstyleemphasis{Tenant Level Read Only} \sphinxhyphen{} typically assign to tenant users who need
to read all resources in the project space

\item {} 
\sphinxAtStartPar
Permission to read all resources at the tenant level

\item {} 
\sphinxAtStartPar
Cannot create/update/delete

\end{itemize}


\subsubsection{Confidentiality and Integrity}
\label{\detokenize{chapters/chapter06:confidentiality-and-integrity}}
\sphinxAtStartPar
Confidentiality implies that data and resources must be protected
against unauthorised introspection/exfiltration. Integrity implies that
the data must be protected from unauthorised modifications or deletions.

\sphinxAtStartPar
Regarding confidentiality and integrity in Cloud Infrastructure, 2 main
concerns are raised:
\begin{itemize}
\item {} 
\sphinxAtStartPar
confidentiality and integrity of the Cloud Infrastructure components
(networks, hypervisor, OpenStack services)

\item {} 
\sphinxAtStartPar
confidentiality and integrity of the tenant’s data

\end{itemize}

\sphinxAtStartPar
The Cloud Infrastructure must also provide the mechanism to identify
corrupted data.


\paragraph{Confidentiality and Integrity of communications (sec.ci.001)}
\label{\detokenize{chapters/chapter06:confidentiality-and-integrity-of-communications-sec-ci-001}}
\sphinxAtStartPar
It is essential to secure the infrastructure from external attacks. To
counter this threat, API endpoints exposed to external networks must be
protected by either a rate\sphinxhyphen{}limiting proxy or web application firewall
(WAF), and must be placed behind a reverse HTTPS proxy (sec.mon.008).
Attacks can also be generated by corrupted internal components, and for
this reason, it is security best practice to ensure integrity and
confidentiality of all network communications (internal and external) by
using Transport Layer Security (TLS) protocol (sec.sys.003,
sec.sys.004). When using TLS, according to the \sphinxhref{https://docs.openstack.org/security-guide/secure-communication/introduction-to-ssl-and-tls.html}{OpenStack security
guide}
recommendation, the minimum version to be used is TLS 1.2.

\sphinxAtStartPar
3 categories of traffic will be protected using TLS:
\begin{itemize}
\item {} 
\sphinxAtStartPar
traffic from and to external domains

\item {} 
\sphinxAtStartPar
communications between OpenStack components (OpenStack services, Bus
message, Data Base)

\item {} 
\sphinxAtStartPar
management traffic

\end{itemize}

\sphinxAtStartPar
Certificates used for TLS encryption must be compliant with X.509
standards and be signed by a trusted authority (sec.sys.017). To issue
certificates for internal OpenStack users or services, the cloud
provider can use a Public Key Infrastructure (PKI) with its own internal
Certification Authority (CA), certificate policies, and management.


\paragraph{Integrity of OpenStack components configuration}
\label{\detokenize{chapters/chapter06:integrity-of-openstack-components-configuration}}
\sphinxAtStartPar
The cloud deployment components/tools store all the information required
to install the infrastructure including sensitive information such as
credentials. It is recommended to turn off deployment components after
deployment to minimise the attack surface area, limit the risk of
compromise, and to deploy and provision the infrastructure through a
dedicated network (VLAN).

\sphinxAtStartPar
Configuration files contain sensitive information. These files must be
protected from malicious or accidental modifications or deletions by
configuring strict access permissions for such files. All access, failed
attempts to change and all changes (pre\sphinxhyphen{}change, post\sphinxhyphen{}change and by who)
must be securely logged, and all failed access and failed changes must
be alerted on (sec.mon.005).

\sphinxAtStartPar
The Cloud Infrastructure must provide the mechanisms to identify
corrupted data (sec.gen.009):
\begin{itemize}
\item {} 
\sphinxAtStartPar
the integrity of configuration files and binaries must be checked by
using cryptographic hash

\item {} 
\sphinxAtStartPar
it is recommended to run scripts (such as checksec.sh) to verify the
properties of the QEMU/KVM

\item {} 
\sphinxAtStartPar
it is recommended to use tools such as CIS\sphinxhyphen{}CAT (\sphinxhref{https://www.cisecurity.org/cybersecurity-tools/cis-cat-pro/}{Center for Internet
security\sphinxhyphen{} Configuration Assessment
Tool})
to check the compliance of systems configuration against respective
\sphinxhref{https://www.cisecurity.org/cis-benchmarks/}{CIS benchmarks}.

\end{itemize}

\sphinxAtStartPar
It is strongly recommend to protect all repositories, such as Linux
repositories and Docker registries, against the corruption of their data
and unauthorised access, by adopting protection measures such as hosting
a local repository/registry with restricted and controlled access, and
using TLS (sec.img.004, sec.img.005, sec.img.006). This
repository/registry must contain only signed images or packages.


\paragraph{Confidentiality and Integrity of tenant data (sec.ci.001)}
\label{\detokenize{chapters/chapter06:confidentiality-and-integrity-of-tenant-data-sec-ci-001}}
\sphinxAtStartPar
Tenant data are forwarded unencrypted over the network. Since the VNF is
responsible for its security, it is up to the VMs to establish secure
data plane, e.g., using IPsec over its tenant network.

\sphinxAtStartPar
A Cloud actor must not be able to retrieve secrets used by VNF managers.
All communications between the VNFM or orchestrator, and the
infrastructure must be protected in integrity and confidentiality (e.g.,
by using TLS) and controlled via appropriate IP filtering rules
(sec.lcm.006).

\sphinxAtStartPar
The Cloud Infrastructure must onboard only trusted and verified VM
images, implying that VNF vendors provide signed images (sec.img.001);
images from non\sphinxhyphen{}trusted sources may contain security breaches or
unsolicited malicious code (spoofing, information disclosure). It is
recommended to scan all VM images with a vulnerability
scanner(sec.img.002). The scan is mandatory for images from unknown or
untrusted sources.

\sphinxAtStartPar
To mitigate tampering attacks, it is recommended to use the \sphinxhref{https://docs.openstack.org/glance/wallaby/user/signature.html}{Glance
image signing
feature}
to validate an image when uploading. In this case, Barbican service must
be installed.

\sphinxAtStartPar
In order to protect data, VNFs must encrypt the volumes they use. In
this case, the encryption key must not be stored on the infrastructure.
When a key management service is provided by the infrastructure,
OpenStack can encrypt data on behalf of tenants (sec.gen.010). It is
recommended to rely on Barbican, as the key manager service of
OpenStack.


\subsubsection{Workload Security}
\label{\detokenize{chapters/chapter06:workload-security}}
\sphinxAtStartPar
OpenStack segregates its infrastructure (sec.ci.008) (for example,
hosts) by Regions, Host Aggregates and Availability Zones (AZ).
Workloads can also be segregated by server groups (affinity and
non\sphinxhyphen{}affinity groups) (sec.sys.008). These options support the workloads
placement requirement (sec.wl.001, sec.wl.004).

\sphinxAtStartPar
Separation of non\sphinxhyphen{}production and production workloads, or by workload
category (for example, payment card information, healthcare, etc.)
requires separation through server groups (for example, Regions, AZs),
but also requires network and storage segregation as in Regions. Thus,
the separation of these workloads is handled through placement of
workloads in separate AZs and/or Regions (sec.wl.005 and sec.wl.006).

\sphinxAtStartPar
Regions also support the sec.wl.004 requirement for separation by
Location (for example, country).

\sphinxAtStartPar
Operational security is handled through a combination of mechanisms
including the above and security groups (sec.sys.002). Security groups
limit the types of traffic that have access to instances. One or more
security groups can be automatically assigned to an instance at launch.
The rules associated with a security group control the incoming traffic.
Any incoming traffic not matched by a rule is denied access. The
security group rules govern access through the setting of different
parameters: traffic source, protocols and destination port on a VM.
Errors in provisioning/managing OpenStack Security Groups can lead to
non\sphinxhyphen{}functioning applications, and it can take a long time to identify
faults and correct them. Thus, the use of tools for auto provisioning
and continued inspection of security groups and network policies is
required.

\sphinxAtStartPar
Given the rate of change in the workload development and deployment, and
the cloud environment itself, sec.wl.003 requires that the workloads
must be assessed during the CI/CD process as the images are created and
then whenever they are deployed. In addition, the infrastructure must be
configured for security as discussed elsewhere in this chapter including
secure boot.


\subsubsection{SR\sphinxhyphen{}IOV and DPDK Considerations}
\label{\detokenize{chapters/chapter06:sr-iov-and-dpdk-considerations}}
\sphinxAtStartPar
The SR\sphinxhyphen{}IOV agent only works with NoopFirewallDriver when Security Groups
are enabled, but can still use other firewall\_driver for other Agents by
updating their conf with the requested firewall driver. Please see
\sphinxhref{https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking}{SR\sphinxhyphen{}IOV Passthrough for
Networking}.

\sphinxAtStartPar
Operators typically do not implement Security Groups when using SR\sphinxhyphen{}IOV
or DPDK networking technologies.


\subsubsection{Image Security}
\label{\detokenize{chapters/chapter06:image-security}}
\sphinxAtStartPar
Images from untrusted sources must not be used (sec.img.001). Valuable
guidance on trusted image creation process and image signature
verification is provided in the “Trusted Images” section of the
\sphinxhref{https://docs.openstack.org/security-guide/instance-management/security-services-for-instances.html\#trusted-images}{OpenStack Security
Guide}.
The OpenStack Security Guide includes reference to the “\sphinxhref{https://docs.openstack.org/image-guide/}{OpenStack
Virtual Machine Image
Guide}” that describes how
to obtain, create, and modify OpenStack compatible virtual machine
images.

\sphinxAtStartPar
Images to be ingested, including signed images from trusted sources,
need to be verified prior to ingestion into the Image Service (Glance)
(sec.gen.009). The operator will need toolsets for scanning images,
including for virus and malware detection (sec.img.002). Adding Signed
Images to the Image Service (Glance) is specified in \sphinxhref{https://docs.openstack.org/operations-guide/ops-user-facing-operations.html\#adding-signed-images}{OpenStack
Operations
Guide}.
Image signing and verification protects image integrity and authenticity
by enabling deployers to sign images and save the signatures and public
key certificates as image properties. The creation of signature per
individual artifact in the VNF package is required by \sphinxhref{http://www.etsi.org/deliver/etsi\_gs/NFV-SOL/001\_099/004/02.03.01\_60/gs\_nfv-sol004v020301p.pdf}{ETSI NFV
SOL004}.

\sphinxAtStartPar
The chain of trust requires that all images are verified again in the
Compute service (Nova) prior to use. Integrity verification at the time
of instantiation is required by \sphinxhref{https://www.etsi.org/deliver/etsi\_gs/NFV-SEC/001\_099/021/02.06.01\_60/gs\_nfv-sec021v020601p.pdf}{ETSI NFV
SEC021}.

\sphinxAtStartPar
Images must be also updated to benefit from the latest security patches
(sec.gen.008, sec.img.007).


\subsubsection{Security LCM}
\label{\detokenize{chapters/chapter06:security-lcm}}
\sphinxAtStartPar
Cloud Infrastructure LCM encompasses provisioning, deployment,
configuration and management (resources scaling, services upgrades,
etc.) as described in \sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_arch/openstack/chapters/chapter07.html}{Operations and Life Cycle Management}. These operations
must be securely performed in order to keep the infrastructure safe and
operational (sec.lcm.003).

\sphinxAtStartPar
\sphinxstylestrong{Provisioning/Deployment}

\sphinxAtStartPar
Regarding the provisioning of servers, switches, routers and networking,
tools must be used to automate the provisioning eliminating human error.
For Infrastructure hardware resources, a set of recommendations is
detailed in {\hyperref[\detokenize{chapters/chapter07:underlying-resources-provisioning}]{\sphinxcrossref{\DUrole{std,std-ref}{Underlying resources provisioning}}}} to automate and secure their provisioning (sec.lcm.001).

\sphinxAtStartPar
For OpenStack services and software components, deployment tools or
components must be used to automate the deployment and avoid errors. The
deployment tool is a sensitive component storing critical information
(deployment scripts, credentials, etc.). The following rules must be
applied:
\begin{itemize}
\item {} 
\sphinxAtStartPar
The boot of the server or the VM hosting the deployment tool must be
protected

\item {} 
\sphinxAtStartPar
Integrity of the deployment images must be checked, before starting
deployment

\item {} 
\sphinxAtStartPar
Deployment must be done through dedicated network (e.g. VLAN)

\item {} 
\sphinxAtStartPar
When the deployment is finished, the deployment tool must be
turned\sphinxhyphen{}off, if the tool is only dedicated to deployment. Otherwise,
any access to the deployment tool must be restricted.

\item {} 
\sphinxAtStartPar
Strict access permissions must be set on OpenStack configuration
files.

\end{itemize}

\sphinxAtStartPar
\sphinxstylestrong{Configuration and management}

\sphinxAtStartPar
Configuration operations must be tracked (sec.gen.015, sec.mon.006,
sec.mon.007). Events such as system access attempts, actions with high
privileges, modification of configuration, must be logged and exported
on the fly to a non\sphinxhyphen{}local storage. The communication channel used for
log collection must be protected for integrity and confidentiality, and
the logs protected against unauthorised modification (sec.mon.004).

\sphinxAtStartPar
Per sec.sys.0016 and sec.lcm.002 requirements, management protocols
limiting security risks must be used such as SNMPv3, SSH v2, ICMP, NTP,
syslog and TLS. How to secure logging is described in the following
section.

\sphinxAtStartPar
\sphinxstylestrong{Platform backup}

\sphinxAtStartPar
The storage for backup must be independent of storage offered to
tenants.

\sphinxAtStartPar
\sphinxstylestrong{Security upgrades}

\sphinxAtStartPar
To defend against virus or other attacks, security patches must be
installed for firmware, OS, Hypervisor and OpenStack services according
to their criticality.


\subsubsection{Monitoring and Security Audit}
\label{\detokenize{chapters/chapter06:monitoring-and-security-audit}}
\sphinxAtStartPar
The intent of this section is to provide a key baseline and minimum
requirements to implement logging that can meet the basic monitoring and
security auditing needs. This should provide sufficient preliminary
guidance, but is not intended to provide a comprehensive solution.
Regular review of security logs that record user access, as well as
session (sec.mon.010) and network activity (sec.mon.012), is critical in
preventing and detecting intrusions that could disrupt business
operations. This monitoring process also allows administrators to
retrace an intruder’s activity and may help correct any damage caused by
the intrusion (sec.mon.011).

\sphinxAtStartPar
The logs have to be continuously monitored and analysed with alerts
created for anomalies (sec.lcm.005). The resources for logging,
monitoring and alerting also need to be logged and monitored, and
corrective actions taken so that they are never short of the needed
resources (sec.mon.015).


\paragraph{Creating Logs}
\label{\detokenize{chapters/chapter06:creating-logs}}\begin{itemize}
\item {} 
\sphinxAtStartPar
All resources to which access is controlled, including but not
limited to applications and operating systems, must have the
capability of generating security audit logs (sec.mon.001).

\item {} 
\sphinxAtStartPar
Logs must be generated for all components (e.g., Nova in OpenStack)
that form the Cloud Infrastructure (sec.mon.001).

\item {} 
\sphinxAtStartPar
All security logging mechanisms must be active from system
initialisation (sec.mon.018):
\begin{itemize}
\item {} 
\sphinxAtStartPar
These mechanisms include any automatic routines necessary to
maintain the activity records and clean\sphinxhyphen{}up programs to ensure the
integrity of the security audit/logging systems.

\end{itemize}

\item {} 
\sphinxAtStartPar
Logs must be time synchronised (sec.mon.002).

\end{itemize}


\paragraph{What to Log / What NOT to Log}
\label{\detokenize{chapters/chapter06:what-to-log-what-not-to-log}}

\subparagraph{What to log}
\label{\detokenize{chapters/chapter06:what-to-log}}
\sphinxAtStartPar
Where technically feasible the following system events must be recorded
(sec.mon.005):
\begin{itemize}
\item {} 
\sphinxAtStartPar
Successful and unsuccessful login attempts including:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Command line authentication (i.e., when initially getting token
from keystone)

\item {} 
\sphinxAtStartPar
Horizon authentication

\item {} 
\sphinxAtStartPar
SSH authentication and sudo on the computes, controllers, network
and storage nodes

\end{itemize}

\item {} 
\sphinxAtStartPar
Logoffs

\item {} 
\sphinxAtStartPar
Successful and unsuccessful changes to a privilege level
(sec.lcm.012)

\item {} 
\sphinxAtStartPar
Successful and unsuccessful configuration changes

\item {} 
\sphinxAtStartPar
Successful and unsuccessful security policy changes

\item {} 
\sphinxAtStartPar
Starting and stopping of security logging

\item {} 
\sphinxAtStartPar
Creating, removing, or changing the inherent privilege level of users
(sec.lcm.012)

\item {} 
\sphinxAtStartPar
Connections to a network listener of the resource

\item {} 
\sphinxAtStartPar
Starting and stopping of processes including attempts to start
unauthorised processes

\item {} 
\sphinxAtStartPar
All command line activity performed by the following innate OS
programs known to otherwise leave no evidence upon command completion
including PowerShell on Windows systems (e.g., Servers, Desktops, and
Laptops)

\item {} 
\sphinxAtStartPar
Where technically feasible, any other security events should be
recorded

\end{itemize}


\subparagraph{What NOT to log}
\label{\detokenize{chapters/chapter06:what-not-to-log}}
\sphinxAtStartPar
Security audit logs must NOT contain:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Authentication credentials, even if encrypted (e.g., password)
(sec.mon.019);

\item {} 
\sphinxAtStartPar
Keystone Token;

\item {} 
\sphinxAtStartPar
Proprietary or Sensitive Personal Information.

\end{itemize}


\paragraph{Where to Log}
\label{\detokenize{chapters/chapter06:where-to-log}}\begin{itemize}
\item {} 
\sphinxAtStartPar
The logs must be stored in an external system (sec.mon.018), in a
manner where the event can be linked to the resource on which it
occurred.

\item {} 
\sphinxAtStartPar
Where technically feasible, events must be recorded on the device
(e.g. VM, physical node, etc.) where the event occurs, if the
external logging system is not available (sec.mon.021).

\item {} 
\sphinxAtStartPar
Security audit logs must be protected in transit and at rest
(sec.mon.004).

\end{itemize}


\paragraph{Required Fields}
\label{\detokenize{chapters/chapter06:required-fields}}
\sphinxAtStartPar
The security audit log must contain at minimum the following fields
(sec.mon.001) where applicable and technically feasible:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Event type

\item {} 
\sphinxAtStartPar
Date/time

\item {} 
\sphinxAtStartPar
Protocol

\item {} 
\sphinxAtStartPar
Service or program used for access

\item {} 
\sphinxAtStartPar
Success/failure

\item {} 
\sphinxAtStartPar
Login ID — Where the Login ID is defined on the
system/application/authentication server; otherwise, the field should
contain ‘unknown’, in order to protect authentication credentials
accidentally entered at the Login ID prompt from appearing in the
security audit log.

\item {} 
\sphinxAtStartPar
Source and destination IP Addresses and ports

\end{itemize}


\paragraph{Data Retention}
\label{\detokenize{chapters/chapter06:data-retention}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Log files must be retained for 180 days, or the relevant regulator
mandate, or your customer mandate, whichever is higher (sec.mon.020).

\item {} 
\sphinxAtStartPar
Implementation and monitoring: after 180 days or your mandated
retention period, security audit logs must be destroyed.

\end{itemize}


\paragraph{Security Logs Time Synchronisation}
\label{\detokenize{chapters/chapter06:security-logs-time-synchronisation}}
\sphinxAtStartPar
The host and various system clocks must be synchronised with an
authenticated time service/NTP server (sec.gen.007).

\sphinxAtStartPar
In any time synchronisation, we need to specify the synchronisation
interval and the tolerance where the latter specifies the permissible
difference the local time can be out of synchronisation. Whenever the
time synchronisation forces the local time to change or the use of
another NTP server, the change details must be logged including time
server source, time, date and time zones (sec.mon.003).


\section{Operations and Life Cycle Management}
\label{\detokenize{chapters/chapter07:operations-and-life-cycle-management}}\label{\detokenize{chapters/chapter07::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter07:introduction}}
\sphinxAtStartPar
To create an Infrastructure as a Service (IaaS) cloud requires the
provisioning and deployment of the underlying infrastructure (compute,
networking and storage) and deployment, configuration and management of
the necessary software on the infrastructure; in the process of
deploying the software, configuration of the infrastructure may also
need to be performed.

\sphinxAtStartPar
Instead of deploying the infrastructure components and services
manually, the current best practice is to write \sphinxstyleemphasis{code} (Infrastructure
as Code, IaC) to define, provision, deploy, configure and manage the
IaaS cloud infrastructure and services. IaC tools allows the entire
provisioning, configuration and management processes to be automated.
The desired state of the infrastructure and services is represented in a
set of human readable, machine executable, and version\sphinxhyphen{}controlled files.
With version control, it is easy to roll back to an older version and
have access to the history of all committed changes.

\sphinxAtStartPar
The provisioning of the infrastructure is typically performed by
provisioning tools while the deployment of the software and the
configuration of the software, and where needed the infrastructure,
falls in the domain of configuration management tools. A single tool may
support both provisioning and configuration management.

\sphinxAtStartPar
Operators may choose certain paradigms with respect to how they
provision and configure their IaaS cloud. These paradigms will drive the
selection of the provisioning and configuration tools. In this chapter
we will discuss the capabilities of provisioning and configuration
management systems; some Open Source tools may be mentioned but their
capabilities are beyond the scope of this chapter.


\subsubsection{Procedural versus Declarative code}
\label{\detokenize{chapters/chapter07:procedural-versus-declarative-code}}
\sphinxAtStartPar
The procedural style IaC tools require code that specifies how to
achieve the desired state. Whilst the declarative style IaC tools
require code that specifies the desired state (what not how). The major
difference between the two styles emerges when changes to the desired
state are required. In the procedural style, the change is coded in
terms of the difference between the desired and current states while in
the declarative style the new desired state is specified. In the
procedural style since the state difference has to be coded, a new code
file has to be created for each change; in the declarative style the
existing code file is updated with the new state information. In the
declarative style knowledge of the current state is not required. In the
procedural style, knowledge of the current state has to be manually
figured by tracing the created code files and the order in which they
were applied.


\subsubsection{Mutable versus Immutable infrastructure}
\label{\detokenize{chapters/chapter07:mutable-versus-immutable-infrastructure}}
\sphinxAtStartPar
In the mutable infrastructure paradigm, software updates are made in
place. Over time this can lead to configuration drift where each server
becomes slightly different from all other servers. In the immutable
infrastructure paradigm, new servers are deployed with the new software
version and then the old servers are undeployed.


\subsection{Cloud Infrastructure provisioning and configuration management}
\label{\detokenize{chapters/chapter07:cloud-infrastructure-provisioning-and-configuration-management}}
\sphinxAtStartPar
In the Reference Model,
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter09.html\#configuration-and-lifecycle-management}{Configuration and Lifecycle Management}
defines the functions of Configuration and Life Cycle Management (LCM).
To operate and manage a scalable cloud, that minimizes operational
costs, requires tools that incorporates systems for automated
provisioning and deployment, and managing configurations that ensures
the correctness and integrity of the deployed and configured systems.


\subsubsection{Underlying resources provisioning}
\label{\detokenize{chapters/chapter07:underlying-resources-provisioning}}
\sphinxAtStartPar
This section deals with automated provisioning of the Cloud
Infrastructure; for example, provisioning the servers, switches,
routers, networking (e.g., subnets, routing tables, load balancers,
etc.), databases and all required operating systems (Servers, switches,
etc.).

\sphinxAtStartPar
The following are the minimum tasks that need to be performed by
automation:
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Pre\sphinxhyphen{}boot configuration} such as BIOS/RAID/IPMI settings: Hardware
manufacturers typically have their proprietary interface for these
tasks but standards such as Redfish are being increasingly utilised.
Consider using tooling to ensure consistency across all
infrastructure components.

\item {} 
\sphinxAtStartPar
\sphinxstylestrong{Bootloader installation} of base Network Operating System (NOS) on
networking equipment or the Operating System (OS) should be performed
using PXE; again consider tooling to ensure consistency across all
infrastructure components.

\end{itemize}

\sphinxAtStartPar
To ensure operational efficiency and save cost and time, the lifecycle
management for physical and virtual servers must be automated using
tools which will handle the repetitive tasks like provisioning,
configuration, and monitoring.
\sphinxhref{https://www.theforeman.org/}{Foreman} is commonly used
to automate the provisoning and management of bare metal infrastructure.
Foreman is an open\sphinxhyphen{}source project, base of several commercial products.
Foreman provides the full management of PXE configuration and the
installation for many Operating Systems (CentOS, Fedora, Ubuntu, Debian,
Red Hat Entreprise Linux, OpenSUSE, etc.).
Foreman service can be installed by Ansible \sphinxurl{https://docs.ansible.com/}
playbooks. Ansible playbooks are basic tools for the automation of the
infrastructure virtualization layer deployments.


\subsubsection{VIM deployment}
\label{\detokenize{chapters/chapter07:vim-deployment}}
\sphinxAtStartPar
When the underlying resources are installed and configured, the VIM
software is deployed. An automated deployment is highly recommended
for the same reasons of efficiency. Open\sphinxhyphen{}source installers are
available to perform the deployments of the OpenStack services.
A subset of these tools is described below.
\begin{itemize}
\item {} 
\sphinxAtStartPar
\sphinxhref{http://opendev.org/openstack/tripleo-common}{OpenStack TripleO},
“OpenStack on OpenStack”

\sphinxAtStartPar
TripleO is an official OpenStack project which allows to deploy and
manage a production cloud onto bare metal hardware using a subset
of existing OpenStack components. The first step of deployment is
the creation of an “undercloud” or deployment cloud. The
undercloud contains the necessary OpenStack components to deploy
and manage an “overcloud”, representing the deployed cloud.
The \sphinxhref{https://docs.openstack.org/tripleo-docs/latest/install/introduction/architecture.html\#project-architecture}{architecture document}
describes the solution. Nova and Ironic are used in the undercloud to manage
the servers in bare metal environment. TripleO leverages
on Heat tempates.

\item {} 
\sphinxAtStartPar
\sphinxhref{https://www.airshipit.org/}{Airship v2}

\sphinxAtStartPar
Airship is supported by the OpenStack Foundation.
It is a collection of interopable open\sphinxhyphen{}source components allowing
to declaratively automate cloud provisioning. The
configurations are defined by YAML documents. All services are
running on containers. Airship v2 is aligned with maturing CNCF
projects such as Kubernetes, Kubectl, Kubeadmin, Argo, Cluster API,
Kustomize, and Metal3. Airship v2.1, released in November 2021,
leverages on Kubernetes 1.21. It includes cloud provisioning at edge
and for 3rd party cloud. The use of the OpenStack\sphinxhyphen{}Helm project allows
the deployment of OpenStack on top of Kubernetes.Airship is not only
a provisioning tool, but also a also a configuration management
system.

\item {} 
\sphinxAtStartPar
\sphinxhref{https://www.starlingx.io/}{StarlingX}

\sphinxAtStartPar
StarlingX is dedicated to cloud infrastructure deployment at
the edge, taking into account the specific edge use cases requirements
for low latency and precision clock synchronisation. It aims to install
a containerised version of OpenStack services, leveraging on Kubernetes,
Docker registry, Airship Armada, and Helm.

\sphinxAtStartPar
OpenStack\sphinxhyphen{}Helm is used as a starting point. OpenStack is installed and
managed as an Armada application. Armada Applications are a set of one or
more interdependent Application Helm charts.   In the case of StarlingX,
there is generally a Helm chart for every OpenStack service.

\end{itemize}


\subsubsection{Configuration Management}
\label{\detokenize{chapters/chapter07:configuration-management}}
\sphinxAtStartPar
The configuration management system ensures the correctness and
integrity of the deployed and configured systems. The tools provide the
assurance that the expected software is running with the expected
configurations on correctly configured nodes that continue to be
configured correctly.

\sphinxAtStartPar
Configuration Management is composed of the following activities:
\begin{itemize}
\item {} 
\sphinxAtStartPar
Desired (Target) State: a version of the software and hardware and
their configurations. Depending upon the configuration management
system these configurations are specified in cookbooks, playbooks,
manifests, etc. The configuration specifications in these artefacts
is used to configure the different types of nodes, BIOS, operating
systems, hypervisor and OpenStack services (through settings within
their config files such as nova.conf, etc.).

\item {} 
\sphinxAtStartPar
Current State: the current configuration of software and hardware as
provided by monitoring systems

\item {} 
\sphinxAtStartPar
State variance mitigation: The CM system, on discovering a variance
between the desired and current states, acts to drive the state to
the desired state. Each CM system accomplishes the task in different
ways.

\end{itemize}


\subsection{Cloud Infrastructure and VIM Maintenance}
\label{\detokenize{chapters/chapter07:cloud-infrastructure-and-vim-maintenance}}
\sphinxAtStartPar
Cloud Infrastructure and VIM Maintenance activities can be classified as
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Deployment of additional infrastructure components (or removal of
infrastructure components)

\item {} 
\sphinxAtStartPar
Cloud Infrastructure Configuration changes

\item {} 
\sphinxAtStartPar
VIM Configuration changes

\item {} 
\sphinxAtStartPar
Version changes (upgrade) of Cloud Infrastructure software (for
example, Host Operating System, Hypervisor, etc.)

\item {} 
\sphinxAtStartPar
Version changes of VIM Software (or component services)

\end{enumerate}

\sphinxAtStartPar
\sphinxstylestrong{Deployment (or removal) of infrastructure components}

\sphinxAtStartPar
In declarative tools, the code with the specified desired state (for
example, number of compute servers) is modified to the new desired
state. The IaC tool then ensures that the desired state is achieved. In
procedural tools, the step\sphinxhyphen{}by\sphinxhyphen{}step code to deploy (remove)
infrastructure components needs to be specified. Existing code can be
cloned, and appropriate changes made to get to the desired state.

\sphinxAtStartPar
\sphinxstylestrong{Configuration and Version Changes}

\sphinxAtStartPar
Configuration and Version Changes are made in a similar fashion to the
“Deployment of infrastructure components” except that the IaC tools used
maybe different.


\subsection{Logging, Monitoring and Analytics}
\label{\detokenize{chapters/chapter07:logging-monitoring-and-analytics}}\begin{itemize}
\item {} 
\sphinxAtStartPar
Logging

\item {} 
\sphinxAtStartPar
Monitoring

\item {} 
\sphinxAtStartPar
Alerting

\item {} 
\sphinxAtStartPar
Logging, Monitoring, and Analytics (LMA) Framework

\end{itemize}


\subsubsection{Logging}
\label{\detokenize{chapters/chapter07:logging}}
\sphinxAtStartPar
A log, in the context of computing, is the automatically produced and
time\sphinxhyphen{}stamped documentation of events relevant to a particular system.
All software, including operating systems, middleware and applications
produce log files. Enterprises and vendors may have custom monitoring
and logging solutions. The intent of the logging and monitoring is to
capture events and data of interest to the Cloud Infrastructure and
workloads so that appropriate actions can be taken. For example,
\begin{itemize}
\item {} 
\sphinxAtStartPar
Operating systems and web servers maintain an access log of all
access requests, session details and file access.

\item {} 
\sphinxAtStartPar
Databases maintain a transaction log of all transaction executed
including an added, changed and deleted data.

\item {} 
\sphinxAtStartPar
Audit logs record chronological documentation of any activities that
could have affected a particular operation or event. Data typically
includes resources accessed, destination and source addresses, and a
timestamp and login information for the person who accessed the
resources.

\end{itemize}

\sphinxAtStartPar
Some of the data is to support the metrics collection specified in the
\sphinxhref{https://cntt.readthedocs.io/en/latest/ref\_model/chapters/chapter04.html}{Infrastructure Capabilities, Measurements and Catalogue}.

\sphinxAtStartPar
Logs have multiple operational uses including for:
\begin{enumerate}
\sphinxsetlistlabels{\arabic}{enumi}{enumii}{}{.}%
\item {} 
\sphinxAtStartPar
Regulatory Compliance and Security Information and Event Management
(SIEM) featuring the automated gathering, analysis and correlation of
log data across all systems and devices across an operator to provide
real\sphinxhyphen{}time analysis, event prioritization, reporting, notification and
alerting.

\item {} 
\sphinxAtStartPar
Monitoring across systems in real\sphinxhyphen{}time to detect particular log
events, patterns, anomalies or inactivity to gauge system and
application health

\item {} 
\sphinxAtStartPar
Identify system and application performance and configuration issues

\item {} 
\sphinxAtStartPar
Root cause analysis for system and application failures and errors

\item {} 
\sphinxAtStartPar
Ensuring that operational objectives and SLAs are met

\end{enumerate}


\subsubsection{Monitoring}
\label{\detokenize{chapters/chapter07:monitoring}}
\sphinxAtStartPar
Monitoring is the process of collecting, aggregating, and analysing
values that improve awareness of the components’ characteristics and
behavior. The data from various parts of the environment are collected
into a monitoring system that is responsible for storage, aggregation,
visualisation, and initiating automated responses when the values meet
specific threshold.

\sphinxAtStartPar
Monitoring systems fulfill many related functions. Their first
responsibility is to accept and store incoming and historical data.
While values representing the current point in time are useful, it is
almost always more helpful to view those numbers in relation to past
values to provide context around changes and trends.


\subsubsection{Alerting}
\label{\detokenize{chapters/chapter07:alerting}}
\sphinxAtStartPar
Alerting is the responsive component of a monitoring system that
performs actions based on changes in metric values. Alert definitions
are composed of two components: a metrics\sphinxhyphen{}based condition or threshold,
and an action to perform when the values fall outside of the acceptable
conditions.

\sphinxAtStartPar
While monitoring systems are incredibly useful for active interpretation
and investigation, one of the primary benefits of a complete monitoring
system is letting administrators disengage from the system. Alerts allow
the specification of situations that make sense to actively manage,
while relying on the passive monitoring of the software to watch for
changing conditions.


\subsubsection{Logging, Monitoring, and Analytics (LMA) Framework}
\label{\detokenize{chapters/chapter07:logging-monitoring-and-analytics-lma-framework}}
\sphinxAtStartPar
In this section, a possible framework utilising Prometheus, Fluentd,
Elasticsearch and Kibana is given as an example only.

\begin{figure}[htbp]
\centering
\capstart

\noindent\sphinxincludegraphics{{RA1-Ch07-Monitoring-Logging-Framework}.png}
\caption{Monitoring and Logging Framework}\label{\detokenize{chapters/chapter07:monitoring-and-logging-framework}}\end{figure}

\sphinxAtStartPar
The monitoring and logging framework
(\hyperref[\detokenize{chapters/chapter07:monitoring-and-logging-framework}]{Fig.\@ \ref{\detokenize{chapters/chapter07:monitoring-and-logging-framework}}}) leverages
Prometheus as the monitoring engine and Fluentd for logging. In
addition, the framework uses Elasticsearch to store and organise logs
for easy access. Prometheus agents pull information from individual
components on every host. Fluentd, an Open Source data collector,
unifies data collection and consumption for better use and understanding
of data. Fluentd captures the access, application and system logs.


\section{Gaps, Innovation, and Development}
\label{\detokenize{chapters/chapter08:gaps-innovation-and-development}}\label{\detokenize{chapters/chapter08::doc}}

\subsection{Introduction}
\label{\detokenize{chapters/chapter08:introduction}}
\sphinxAtStartPar
The purpose of this chapter is to identify the gaps between what is
required for automated deployment of VNFs on Cloud Infrastructure
frameworks and the framework offered by OpenStack. Once gaps are
identified, the next step will be to propose a plan to address these
gaps. The most obvious way to address the gaps will be to propose a set
of APIs in the upstream OpenStack community


\subsection{The Gap}
\label{\detokenize{chapters/chapter08:the-gap}}

\subsubsection{Autoscaling}
\label{\detokenize{chapters/chapter08:autoscaling}}
\sphinxAtStartPar
With regards to resource autoscaling
(gen.scl.01
{\hyperref[\detokenize{chapters/chapter02:general-recommendations}]{\sphinxcrossref{\DUrole{std,std-ref}{General Recommendations}}}}) it is
recommended that the NFVO/VNFM manages the policy and triggers a
scale\sphinxhyphen{}up or scale\sphinxhyphen{}down action based on application telemetry, event, AI,
or ML etc. While the use of telemetry and alarming system can trigger a
scaling operation based on resource utilisation, without application
context this may not provide the granularity or reaction time required
by the application. It is therefore suggested that an OpenStack scaling
operation is called using an appropriate autoscaling web\sphinxhyphen{}hook by the
NFVO/VNFM.

\sphinxAtStartPar
For more information on auto\sphinxhyphen{}scaling with Heat please see:
\sphinxurl{https://docs.openstack.org/senlin/latest/scenarios/autoscaling\_heat.html}.
Please note that the OpenStack Senlin service is still under development
with major architectural changes made in the OpenStack Ussuri release.

\sphinxAtStartPar
Please note: physical compute node autoscaling is out of scope.



\renewcommand{\indexname}{Index}
\printindex
\end{document}