Doctor: Fault Management and Maintenance

Project:

Doctor, https://wiki.opnfv.org/doctor

Editors:

Ashiq Khan (NTT DOCOMO), Gerald Kunzmann (NTT DOCOMO)

Authors:

Ryota Mibu (NEC), Carlos Goncalves (NEC), Tomi Juvonen (Nokia), Tommy Lindgren (Ericsson), Bertrand Souville (NTT DOCOMO), Balazs Gibizer (Ericsson), Ildiko Vancsa (Ericsson) and others.

Abstract:

Doctor is an OPNFV requirement project [DOCT]. Its scope is NFVI fault management, and maintenance and it aims at developing and realizing the consequent implementation for the OPNFV reference platform.

This deliverable is introducing the use cases and operational scenarios for Fault Management considered in the Doctor project. From the general features, a high level architecture describing logical building blocks and interfaces is derived. Finally, a detailed implementation is introduced, based on available open source components, and a related gap analysis is done as part of this project. The implementation plan finally discusses an initial realization for a NFVI fault management and maintenance solution in open source software.

History:

Date	Description
02.12.2014	Project creation
14.04.2015	Initial version of the deliverable uploaded to Gerrit
18.05.2015	Stable version of the Doctor deliverable

Definition of terms

Different SDOs and communities use different terminology related to NFV/Cloud/SDN. This list tries to define an OPNFV terminology, mapping/translating the OPNFV terms to terminology used in other contexts.

ACT-STBY configuration

Failover configuration common in Telco deployments. It enables the operator to use a standby (STBY) instance to take over the functionality of a failed active (ACT) instance.

Administrator

Administrator of the system, e.g. OAM in Telco context.

Consumer

User-side Manager; consumer of the interfaces produced by the VIM; VNFM, NFVO, or Orchestrator in ETSI NFV [ENFV] terminology.

EPC

Evolved Packet Core, the main component of the core network architecture of 3GPP’s LTE communication standard.

MME

Mobility Management Entity, an entity in the EPC dedicated to mobility management.

NFV

Network Function Virtualization

NFVI

Network Function Virtualization Infrastructure; totality of all hardware and software components which build up the environment in which VNFs are deployed.

S/P-GW

Serving/PDN-Gateway, two entities in the EPC dedicated to routing user data packets and providing connectivity from the UE to external packet data networks (PDN), respectively.

Physical resource

Actual resources in NFVI; not visible to Consumer.

VNFM

Virtualized Network Function Manager; functional block that is responsible for the lifecycle management of VNF.

NFVO

Network Functions Virtualization Orchestrator; functional block that manages the Network Service (NS) lifecycle and coordinates the management of NS lifecycle, VNF lifecycle (supported by the VNFM) and NFVI resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity.

VIM

Virtualized Infrastructure Manager; functional block that is responsible for controlling and managing the NFVI compute, storage and network resources, usually within one operator’s Infrastructure Domain, e.g. NFVI Point of Presence (NFVI-PoP).

Virtual Machine (VM)

Virtualized computation environment that behaves very much like a physical computer/server.

Virtual network

Virtual network routes information among the network interfaces of VM instances and physical network interfaces, providing the necessary connectivity.

Virtual resource

A Virtual Machine (VM), a virtual network, or virtualized storage; Offered resources to “Consumer” as result of infrastructure virtualization; visible to Consumer.

Virtual Storage

Virtualized non-volatile storage allocated to a VM.

VNF

Virtualized Network Function. Implementation of an Network Function that can be deployed on a Network Function Virtualization Infrastructure (NFVI).

• 1. Introduction
  • 1.1. Problem description
• 2. Use cases and scenarios
  • 2.1. Faults
    • 2.1.1. Fault management using ACT-STBY configuration
    • 2.1.2. Preventive actions based on fault prediction
  • 2.2. NVFI Maintenance
    • 2.2.1. VM Retirement
• 3. High level architecture and general features
  • 3.1. Functional overview
  • 3.2. Architecture Overview
  • 3.3. General Features and Requirements
    • 3.3.1. Monitoring
    • 3.3.2. Detection
    • 3.3.3. Correlation and Cognition
    • 3.3.4. Notification
    • 3.3.5. Fencing
    • 3.3.6. Recovery Action
  • 3.4. High level northbound interface specification
    • 3.4.1. Fault management
    • 3.4.2. NFVI Maintenance
• 4. Gap analysis in upstream projects
  • 4.1. VIM Northbound Interface
    • 4.1.1. Immediate Notification
    • 4.1.2. Maintenance Notification
  • 4.2. VIM Southbound interface
    • 4.2.1. Normalization of data collection models
  • 4.3. OpenStack
    • 4.3.1. Ceilometer
      • 4.3.1.1. Scalability of fault aggregation
      • 4.3.1.2. Monitoring of hardware and software
    • 4.3.2. Nova
      • 4.3.2.1. Correct states when compute host is down
      • 4.3.2.2. Evacuate VMs in Maintenance mode
    • 4.3.3. Monasca
      • 4.3.3.1. Anomaly detection
      • 4.3.3.2. Sensor monitoring
  • 4.4. Hardware monitoring tools
    • 4.4.1. Zabbix
      • 4.4.1.1. Delay in execution of actions
• 5. Detailed architecture and interface specification
  • 5.1. Functional Blocks
    • 5.1.1. Monitor
    • 5.1.2. Inspector
    • 5.1.3. Controller
    • 5.1.4. Notifier
  • 5.2. Sequence
    • 5.2.1. Fault Management
    • 5.2.2. NFVI Maintenance
  • 5.3. Implementation plan for OPNFV Release 1
    • 5.3.1. Fault management
    • 5.3.2. NFVI Maintenance
  • 5.4. Information elements
  • 5.5. Detailed northbound interface specification
    • 5.5.1. Fault management interface
      • 5.5.1.1. SubscribeRequest (Consumer -> VIM)
      • 5.5.1.2. SubscribeResponse (VIM -> Consumer)
      • 5.5.1.3. FaultNotification (VIM -> Consumer)
      • 5.5.1.4. FaultQueryRequest (Consumer -> VIM)
      • 5.5.1.5. FaultQueryResponse (VIM -> Consumer)
    • 5.5.2. NFVI maintenance
      • 5.5.2.1. SubscribeRequest (Consumer -> VIM)
      • 5.5.2.2. SubscribeResponse (VIM -> Consumer)
      • 5.5.2.3. MaintenanceNotification (VIM -> Consumer)
      • 5.5.2.4. StateChangeRequest (Administrator -> VIM)
      • 5.5.2.5. StateChangeResponse (VIM -> Administrator)
      • 5.5.2.6. StateQueryRequest (Administrator -> VIM)
      • 5.5.2.7. StateQueryResponse (VIM -> Administrator)
  • 5.6. Detailed southbound interface specification
    • 5.6.1. Fault event interface
      • 5.6.1.1. EventNotification
    • 5.6.2. NFV IFA, OPNFV Doctor and AODH alarms
  • 5.7. Blueprints
    • 5.7.1. Instance State Notification (Ceilometer)
    • 5.7.2. Event Publisher for Alarm (Ceilometer)
    • 5.7.3. Notification-driven alarm evaluator (Ceilometer)
    • 5.7.4. Report host fault to update server state immediately (Nova)
    • 5.7.5. Other related BPs
      • 5.7.5.1. pacemaker-servicegroup-driver
• 6. Summary and conclusion
• 7. Annex: NFVI Faults

References and bibliography

[DOCT]

OPNFV, “Doctor” requirements project, [Online]. Available at https://wiki.opnfv.org/doctor

[PRED]

OPNFV, “Data Collection for Failure Prediction” requirements project [Online]. Available at https://wiki.opnfv.org/prediction

[OPSK]

OpenStack, [Online]. Available at https://www.openstack.org/

[CEIL]

OpenStack Telemetry (Ceilometer), [Online]. Available at https://wiki.openstack.org/wiki/Ceilometer

[NOVA]

OpenStack Nova, [Online]. Available at https://wiki.openstack.org/wiki/Nova

[NEUT]

OpenStack Neutron, [Online]. Available at https://wiki.openstack.org/wiki/Neutron

[CIND]

OpenStack Cinder, [Online]. Available at https://wiki.openstack.org/wiki/Cinder

[MONA]

OpenStack Monasca, [Online], Available at https://wiki.openstack.org/wiki/Monasca

[OSAG]

OpenStack Cloud Administrator Guide, [Online]. Available at http://docs.openstack.org/admin-guide-cloud/content/

[ZABB]

ZABBIX, the Enterprise-class Monitoring Solution for Everyone, [Online]. Available at http://www.zabbix.com/

[ENFV]

ETSI NFV, [Online]. Available at http://www.etsi.org/technologies-clusters/technologies/nfv