1. Introduction

1.1. Overview

This Reference Architecture is focussed on OpenStack as the Virtualised Infrastructure Manager (VIM) chosen based on the criteria laid out in the Reference Model Introduction. OpenStack [1] has the advantage of being a mature and widely accepted open-source technology; a strong ecosystem of vendors that support it, the OpenInfra Foundation for managing the community, and, most importantly, it is widely deployed by the global operator community for both internal infrastructure and external facing products and services. This means that the operators have existing staff with the right skill sets to support a Cloud Infrastructure (or Network Function Virtualisation Infrastructure, NFVI [2]) deployment into development, test and production. Another reason to choose OpenStack is that it has a large active community of vendors and operators, which means that any code or component changes needed to support the Common Telco Cloud Infrastructure requirements can be managed through the existing project communities’ processes to add and validate the required features through well-established mechanisms.

1.1.1. Vision

The OpenStack-based Reference Architecture will host NFV workloads, primarily VNFs (Virtual Network Functions), of interest to the Anuket community. The Reference Architecture document can be used by operators to deploy Anuket conformant infrastructure; hereafter, “conformant” denotes that the resource can satisfy tests conducted to verify conformance with this reference architecture.

1.2. Use Cases

Several NFV use cases are documented in OpenStack. For more examples and details refer to the OpenStack Use cases [3].

Examples include:

• Overlay networks: The overlay functionality design includes OpenStack Networking in Open vSwitch [4] GRE tunnel mode. In this case, the layer-3 external routers pair with VRRP, and switches pair with an implementation of MLAG to ensure that you do not lose connectivity with the upstream routing infrastructure.

• Performance tuning: Network level tuning for this workload is minimal. Quality of Service (QoS) applies to these workloads for a middle ground Class Selector depending on existing policies. It is higher than a best effort queue but lower than an Expedited Forwarding or Assured Forwarding queue. Since this type of application generates larger packets with longer-lived connections, you can optimize bandwidth utilization for long duration TCP. Normal bandwidth planning applies here with regards to benchmarking a session’s usage multiplied by the expected number of concurrent sessions with overhead.

• Network functions: Network functions is a broad category but encompasses workloads that support the exchange of information (data, voice, multi-media) over a system’s network. Some of these workloads tend to consist of a large number of small-sized packets that are short lived, such as DNS queries or SNMP traps. These messages need to arrive quickly and, thus, do not handle packet loss. Network function workloads have requirements that may affect configurations including at the hypervisor level. For an application that generates 10 TCP sessions per user with an average bandwidth of 512 kilobytes per second per flow and expected user count of ten thousand (10,000) concurrent users, the expected bandwidth plan is approximately 4.88 gigabits per second. The supporting network for this type of configuration needs to have a low latency and evenly distributed load across the topology. These types of workload benefit from having services local to the consumers of the service. Thus, use a multi-site approach, as well as, deploying many copies of the application to handle load as close as possible to consumers. Since these applications function independently, they do not warrant running overlays to interconnect tenant networks. Overlays also have the drawback of performing poorly with rapid flow setup and may incur too much overhead with large quantities of small packets and therefore we do not recommend them. QoS is desirable for some workloads to ensure delivery. DNS has a major impact on the load times of other services and needs to be reliable and provide rapid responses. Configure rules in upstream devices to apply a higher-Class Selector to DNS to ensure faster delivery or a better spot in queuing algorithms.

1.3. OpenStack Reference Release

This Reference Architecture document conforms to the OpenStack Wallaby [5] release. While many features and capabilities are conformant with many OpenStack releases, this document will refer to features, capabilities and APIs that are part of the OpenStack Wallaby release. For ease, this Reference Architecture document version can be referred to as “RA-1 OSTK Wallaby.”

1.4. Principles

OpenStack Reference Architecture must obey to the following set of principles described in:

• Anuket General Principles

• Architectural Principles

1.4.1. OpenStack specific principles

OpenStack considers the following Four Opens essential for success:

• Open Source

• Open Design

• Open Development

• Open Community

This OpenStack Reference Architecture is organised around the three major Cloud Infrastructure resource types as core services of compute, storage and networking, and a set of shared services of identity management, image management, graphical user interface, orchestration engine, etc.

1.5. Document Organisation

Chapter 2 defines the Reference Architecture requirements and, when appropriate, provides references to where these requirements are addressed in this document. The intent of this document is to address all of the mandatory (“must”) requirements and the most useful of the other optional (“should”) requirements. Chapter 3 and 4 cover the Cloud Infrastructure resources and the core OpenStack services, while the APIs are covered in Chapter 5. Chapter 6 covers the implementation and enforcement of security capabilities and controls. Life Cycle Management of the Cloud Infrastructure and VIM are covered in Chapter 7 with stress on Logging, Monitoring and Analytics (LMA), configuration management and some other operational items. Please note that Chapter 7 is not a replacement for the implementation, configuration and operational documentation that accompanies the different OpenStack distributions. Chapter 8 identifies certain Gaps that currently exist and plans on how to address them (for example, resources autoscaling).

1.6. Terminology

Cloud Infrastructure

A generic term covering NFVI, IaaS and CaaS capabilities - essentially the infrastructure on which a Workload can be executed.

Note

The official OpenStack Glossary is an extensive list of OpenStack-related concepts. Some additional terms used in the Reference Architecture RA-1 or used to relate RA-1 terms with terms defined elsewhere.

Core (physical)

An independent computer processing unit that can independently execute CPU instructions and is integrated with other cores on a multiprocessor (chip, integrated circuit die). Please note that the multiprocessor chip is also referred to as a CPU that is placed in a socket of a computer motherboard.

Flavor Capability

The capability of the Cloud Infrastructure Profile, such as CPU Pinning, NUMA or huge pages.

Flavor Geometry

Flavor sizing such as number of vCPUs, RAM, disk, etc.

Huge pages

Physical memory is partitioned and accessed using the basic page unit (in Linux default size of 4 KB). Huge pages, typically 2 MB and 1GB size, allows large amounts of memory to be utilised with reduced overhead. In an NFV environment, huge pages are critical to support large memory pool allocation for data packet buffers. This results in fewer Translation Lookaside Buffers (TLB) lookups, which reduces the virtual to physical pages address translations. Without huge pages enabled high TLB miss rates would occur thereby degrading performance.

Server

For the OpenStack Compute API, a server is a virtual machine (VM), a physical machine (bare metal) or a container.

1.7. Conventions

The key words “must”, “must not”, “required”, “shall”, “shall not”, “should”, “should not”, “recommended”, “may”, and “optional” in this document are to be interpreted as described in RFC 2119 [6].

1.8. References

1

OpenStack. OpenStack Documentation. Accessed: 2022-07-25. URL: https://docs.openstack.org/.

2

Network Functions Virtualisation (NFV); Infrastructure Overview. ETSI GS NFV-INF 001 V1.1.1, January 2015. URL: https://www.etsi.org/deliver/etsi_gs/NFV-INF/001_099/001/01.01.01_60/gs_NFV-INF001v010101p.pdf.

3

OpenStack. OpenStack Use cases. Accessed: 2022-07-25. URL: https://docs.openstack.org/arch-design/use-cases.html.

4

OpenvSwitch. Open vSwitch. Accessed: 2022-07-25. URL: https://www.openvswitch.org/.

5

OpenStack. OpenStack Wallaby projects. Accessed: 2022-07-25. URL: https://docs.openstack.org/wallaby/projects.html.

6

Scott O. Bradner. Key words for use in RFCs to Indicate Requirement Levels. RFC 2119, March 1997. URL: https://www.rfc-editor.org/info/rfc2119, doi:10.17487/RFC2119.

7

H. Philip White. CIS Password Policy Guide. Technical Report, 2020. URL: https://www.cisecurity.org/insights/white-papers/cis-password-policy-guide.

8

cisecurity. CIS Controls V7.1. Technical Report. URL: https://www.cisecurity.org/controls/cis-controls-list.

9

OpenStack. CPU Dedicated Set. Accessed: 2022-07-25. URL: https://docs.openstack.org/nova/latest/configuration/config.html#compute.cpu_dedicated_set.

10

OpenStack. CPU Topologies. Accessed: 2022-07-25. URL: https://docs.openstack.org/nova/latest/admin/cpu-topologies.html.

11

OpenStack. Neutron Plugins and Drivers. Accessed: 2022-07-25. URL: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers.

12

OpenStack. Tags. Accessed: 2022-07-25. URL: https://specs.openstack.org/openstack/api-wg/guidelines/tags.html.

13

OpenStack. Configuring the stateful services. Accessed: 2022-07-25. URL: https://docs.openstack.org/ha-guide/control-plane-stateful.html.

14

OpenStack. Senlin documentation. Accessed: 2022-07-25. URL: https://docs.openstack.org/senlin/wallaby/.

15

OpenStack. Neutron OVS Agent Support for Baremetal with Smart NIC. Accessed: 2022-07-25. URL: https://specs.openstack.org/openstack/neutron-specs/specs/stein/neutron-ovs-agent-support-baremetal-with-smart-nic.html.

16

National Telecommunications and Information Administration. Software Bill Of Materials. Accessed: 2022-07-25. URL: https://www.ntia.gov/SBOM.

17

Center for Internet Security. Accessed: 2022-07-25. URL: https://www.cisecurity.org/.

18

Cloud Security Alliance. Accessed: 2022-07-25. URL: https://cloudsecurityalliance.org/.

19

OWASP Cheat Sheet Series. Accessed: 2022-07-25. URL: https://github.com/OWASP/CheatSheetSeries.

20

Open Web Application Security Project. Accessed: 2022-07-25. URL: https://www.owasp.org.

21

OWASP Top Ten Security Risks. Accessed: 2022-07-25. URL: https://owasp.org/www-project-top-ten/.

22

OWASP Software Maturity Model (SAMM). Accessed: 2022-07-25. URL: https://owaspsamm.org/blog/2019/12/20/version2-community-release/.

23

OWASP Web Security Testing Guide. Accessed: 2022-07-25. URL: https://github.com/OWASP/wstg/tree/master/document.

24

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission). ISO/IEC 27001:2013. 2013. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27001:ed-2:v1:en.

25

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission). ISO/IEC 27002:2013. 2013. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27002:ed-2:v1:en.

26

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission). ISO/IEC 7032:2012. 2012. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27032:ed-1:v1:en.

27

OpenStack. OpenStack Storage. Accessed: 2022-07-25. URL: https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html#table-openstack-storage.

28

OpenStack. Cinder Driver Support Matrix. Accessed: 2022-07-25. URL: https://docs.openstack.org/cinder/latest/reference/support-matrix.html.

29

Tungsten Fabric. Multicloud Multistack SDN. Accessed: 2022-07-25. URL: https://tungsten.io.

30

OpenStack. OpenStack Glossary. Accessed: 2022-07-25. URL: https://docs.openstack.org/doc-contrib-guide/common/glossary.html.

31

OpenStack. Feature Support Matrix. Accessed: 2022-07-25. URL: https://docs.openstack.org/nova/latest/user/support-matrix.html.

32

OpenStack. Storage Architecture Design. Accessed: 2022-07-25. URL: https://docs.openstack.org/arch-design/design-storage.html.

33

OpenStack. Nova: KVM. Accessed: 2022-07-29. URL: https://docs.openstack.org/nova/wallaby/admin/configuration/hypervisor-kvm.html.

34

OpenStack. Hardening the virtualization layers. Accessed: 2022-07-29. URL: https://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html.

35

OpenStack. OpenStack Reference Architecture For 100, 300 and 500 Nodes. Accessed: 2022-07-29. URL: https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html.

36

DPDK. (DPDK) Release Notes. Accessed: 2022-07-29. URL: http://doc.dpdk.org/guides/rel_notes.

37

DPDK. (DPDK) Performance Reports. Accessed: 2022-07-29. URL: http://core.dpdk.org/perf-reports/.

38

Robert Moskowitz, Daniel Karrenberg, Yakov Rekhter, Eliot Lear, and Geert Jan de Groot. Address Allocation for Private Internets. RFC 1918, February 1996. URL: https://www.rfc-editor.org/info/rfc1918, doi:10.17487/RFC1918.

39

OpenStack. Introducing Octavia. Accessed: 2022-07-29. URL: https://docs.openstack.org/octavia/latest/reference/introduction.html.

40

OpenStack. Octavia (Load-balancer service). Accessed: 2022-07-29. URL: https://governance.openstack.org/tc/reference/projects/octavia.html.

41

OpenStack. openstack/neutron-vpnaas. Accessed: 2022-07-29. URL: https://opendev.org/openstack/neutron-vpnaas.

42

OpenStack. Neutron: Plugins. Accessed: 2022-07-29. URL: https://wiki.openstack.org/wiki/Neutron#Plugins.

43

OpenStack. Neutron: API Extensions. Accessed: 2022-07-29. URL: https://docs.openstack.org/neutron/latest/contributor/internals/api_extensions.html.

44

OpenStack. Networking API v2.0: List extensions. Accessed: 2022-07-29. URL: https://docs.openstack.org/api-ref/network/v2/#list-extensions.

45

OpenStack. Networking API v2.0: Show extension details. Accessed: 2022-07-29. URL: https://docs.openstack.org/api-ref/network/v2/#show-extension-details.

46

OpenStack. Neutron/ML2. Accessed: 2022-07-29. URL: https://wiki.openstack.org/wiki/Neutron/ML2.

47

OpenStack. Cinder Driver Support Matrix. Accessed: 2022-07-29. URL: https://docs.openstack.org/cinder/latest/reference/support-matrix.html.

48

OpenStack. (Cinder) Available Drivers. Accessed: 2022-07-29. URL: https://docs.openstack.org/cinder/latest/drivers.html.

49

OpenStack. Cinder Service Configuration. Accessed: 2022-07-29. URL: https://docs.openstack.org/cinder/latest/configuration/index.html.

50

OpenStack. Cinder Administration. Accessed: 2022-07-29. URL: https://docs.openstack.org/cinder/latest/admin/index.html.

51

Ceph. The Future of Storage. Accessed: 2022-07-29. URL: https://ceph.io/en.

52

OpenStack. Keystone, the OpenStack Identity Service. Accessed: 2022-07-29. URL: https://docs.openstack.org/keystone/wallaby/.

53

OpenStack. Welcome to Glance’s documentation! Accessed: 2022-07-29. URL: https://docs.openstack.org/glance/wallaby/.

54

OpenStack. OpenStack Block Storage (Cinder) documentation. Accessed: 2022-07-29. URL: https://docs.openstack.org/cinder/wallaby/.

55

OpenStack. Welcome to Swift’s documentation! Accessed: 2022-07-29. URL: https://docs.openstack.org/swift/wallaby/.

56

OpenStack. Welcome to Neutron’s documentation! Accessed: 2022-07-29. URL: https://docs.openstack.org/neutron/wallaby/.

57

OpenStack. Scenario: High Availability using Distributed Virtual Routing (DVR). Accessed: 2022-07-29. URL: https://docs.openstack.org/liberty/networking-guide/scenario-dvr-ovs.html.

58

OpenStack. Neureton: Distributed Virtual Routing with VRRP. Accessed: 2022-07-29. URL: https://docs.openstack.org/neutron/wallaby/admin/config-dvr-ha-snat.html.

59

OpenStack. OpenStack Compute (nova). Accessed: 2022-07-29. URL: https://docs.openstack.org/nova/wallaby/.

60

OpenStack. Welcome to Ironic’s documentation! Accessed: 2022-07-29. URL: https://docs.openstack.org/ironic/wallaby/.

61

OpenStack. Ironic API Reference: Bare Metal API. Accessed: 2022-07-29. URL: https://docs.openstack.org/api-ref/baremetal/.

62

OpenStack. Welcome to the Heat documentation! Accessed: 2022-07-29. URL: https://docs.openstack.org/heat/wallaby/.

63

OpenStack. Horizon: The OpenStack Dashboard Project. Accessed: 2022-07-29. URL: https://docs.openstack.org/horizon/wallaby/.

64

OpenStack. Placement. Accessed: 2022-07-29. URL: https://docs.openstack.org/placement/wallaby/index.html.

65

OpenStack. Placement: Modeling with Provider Trees. Accessed: 2022-07-29. URL: https://docs.openstack.org/placement/latest/user/provider-tree.html.

66

OpenStack. Placement Usage. Accessed: 2022-07-29. URL: https://docs.openstack.org/placement/latest/user/index.html.

67

OpenStack. OpenStack Key Manager (barbican). Accessed: 2022-07-29. URL: https://docs.openstack.org/barbican/wallaby/.

68

OpenStack. OpenStack Accelerator (Cyborg). Accessed: 2022-07-29. URL: https://docs.openstack.org/cyborg/wallaby/.

69

OpenStack. Compute API Guide 2.1.0: Server concepts. Accessed: 2022-07-29. URL: https://docs.openstack.org/api-guide/compute/server_concepts.html.

70

OpenStack. Cyborg Support Matrix (Wallaby). Accessed: 2022-07-29. URL: https://docs.openstack.org/cyborg/wallaby/reference/support-matrix.html.

71

OpenStack. Cyborg Support Matrix. Accessed: 2022-07-29. URL: https://docs.openstack.org/cyborg/latest/reference/support-matrix.html.

72

OpenStack. Cyborg architecture. Accessed: 2022-07-29. URL: https://docs.openstack.org/cyborg/latest/user/architecture.html.

73

OpenStack. Nova: Flavors. Accessed: 2022-07-29. URL: https://docs.openstack.org/nova/latest/user/flavors.html.

74

State of the Edge The Linux Foundation. Open Glossary of Edge Computing. Accessed: 2022-07-29. URL: https://github.com/State-of-the-Edge/glossary/blob/master/edge-glossary.md.

75

OpenStack. Edge Computing: Next Steps in Architecture, Design and Testing. Accessed: 2022-07-29. URL: https://www.openstack.org/use-cases/edge-computing/edge-computing-next-steps-in-architecture-design-and-testing.

76

OpenStack. OpenStack Reference Architecture For 100, 300 and 500 Nodes: Services Placement Summary. Accessed: 2022-07-29. URL: https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html#services-placement-summary.

77

OpenStack. Nova: Image pre-caching. Accessed: 2022-07-29. URL: https://docs.openstack.org/nova/latest/admin/image-caching.html#image-pre-caching.

78

Airship Community. Airship v2. Accessed: 2022-07-25. URL: https://www.airshipit.org/.

79

StarlingX Community. Deploy Your Edge Cloud Now. Accessed: 2022-07-25. URL: https://www.starlingx.io/.

80

OpenStack. TripleO. Accessed: 2022-07-25. URL: http://opendev.org/openstack/tripleo-common.

81

OpenStack. Security Boundaries and Threats. Accessed: 2022-07-25. URL: https://docs.openstack.org/security-guide/introduction/security-boundaries-and-threats.html.

82

OpenStack. OpenStack Security Guide. Accessed: 2022-07-25. URL: https://docs.openstack.org/security-guide/introduction/introduction-to-openstack.html.

83

Mitre. Common Vulnerabilities and Exposures. Accessed: 2022-07-25. URL: https://cve.mitre.org/.

84

National Institute of Standards and Technology. NIST Vulnerabilities Metrics. Accessed: 2022-07-25. URL: https://nvd.nist.gov/vuln-metrics/cvss.

85

OpenStack. OpenStack Security Guide - Identity. Accessed: 2022-07-25. URL: https://docs.openstack.org/security-guide/identity.html.

86

OpenStack. OpenStack Security Guide - Authentication Methods. Accessed: 2022-07-25. URL: https://docs.openstack.org/security-guide/identity/authentication-methods.html.

87

OpenStack. OpenStack Security Guide - Policies. Accessed: 2022-07-25. URL: https://docs.openstack.org/security-guide/identity/policies.html#policy-section.

88

OpenStack. KeyStone Default Roles. Accessed: 2022-07-25. URL: https://docs.openstack.org/keystone/latest/admin/service-api-protection.html.

89

OpenStack. Introduction to TLS and SSL. Accessed: 2022-07-25. URL: https://docs.openstack.org/security-guide/secure-communication/introduction-to-ssl-and-tls.html.

90

Center for Internet Security. CIS-CAT Pro. Accessed: 2022-07-25. URL: https://www.cisecurity.org/cybersecurity-tools/cis-cat-pro/.

91

Center for Internet Security. CIS Benchmarks. Accessed: 2022-07-25. URL: https://www.cisecurity.org/cis-benchmarks/.

92

OpenStack. Image Signature Verification. Accessed: 2022-07-25. URL: https://docs.openstack.org/glance/wallaby/user/signature.html.

93

OpenStack. SR-IOV Passthrough For Networking. Accessed: 2022-07-25. URL: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking.

94

OpenStack. Trusted Images. Accessed: 2022-07-25. URL: https://docs.openstack.org/security-guide/instance-management/security-services-for-instances.html#trusted-images.

95

OpenStack. Adding Signed Images. Accessed: 2022-07-25. URL: https://docs.openstack.org/operations-guide/ops-user-facing-operations.html#adding-signed-images.

96

Network Functions Virtualisation (NFV) Release 4; Protocols and Data Models; VNF Package and PNFD Archive specification. ETSI GS NFV-SOL 004 V4.3.1, July 2022. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/004/04.03.01_60/gs_NFV-SOL004v040301p.pdf.

97

Network Functions Virtualisation (NFV) Release 2; Security; VNF Package Security Specification. ETSI GS NFV-SEC 021 V2.6.1, June 2019. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SEC/001_099/021/02.06.01_60/gs_nfv-sec021v020601p.pdf.

98

Foreman. Foreman. Accessed: 2022-07-25. URL: https://www.theforeman.org/.

99

Ansible Community. Ansible Documentation. Accessed: 2022-07-25. URL: https://docs.ansible.com/.

100

OpenStack. TripleO Architecture. Accessed: 2022-07-25. URL: https://docs.openstack.org/tripleo-docs/latest/install/introduction/architecture.html#project-architecture.

101

OpenStack. Autoscaling with Heat. Accessed: 2022-07-25. URL: https://docs.openstack.org/senlin/latest/scenarios/autoscaling_heat.html.