OPNFV Apex Release Notes


1. OPNFV Release Notes for the Colorado 3.0 release of OPNFV Apex deployment tool

1.1. Abstract

This document provides the release notes for Colorado 3.0 release with the Apex deployment toolchain.

1.2. License

All Apex and “common” entities are protected by the Apache License ( http://www.apache.org/licenses/ )

1.3. Version History

Date Ver. Authors Comment
2016-12-05 2.3.0 Tim Rozet More updates for Colorado 3.0
2016-10-26 2.2.0 Tim Rozet More updates for Colorado 2.0
2016-09-20 2.1.0 Tim Rozet More updates for Colorado
2016-08-11 2.0.0 Dan Radez Updates for Colorado
2015-09-17 1.0.0 Dan Radez Rewritten for RDO Manager update

1.4. Important Notes

This is the OPNFV Colorado 3.0 release that implements the deploy stage of the OPNFV CI pipeline via Apex.

Apex is based on RDO’s Triple-O installation tool chain. More information at http://rdoproject.org

Carefully follow the installation-instructions which guide a user on how to deploy OPNFV using Apex installer.

1.5. Summary

Colorado release with the Apex deployment toolchain will establish an OPNFV target system on a Pharos compliant lab infrastructure. The current definition of an OPNFV target system is OpenStack Mitaka combined with an SDN controller, such as OpenDaylight. The system is deployed with OpenStack High Availability (HA) for most OpenStack services. SDN controllers are deployed only on the first controller (see HAIssues for known HA SDN issues). Ceph storage is used as Cinder backend, and is the only supported storage for Colorado. Ceph is setup as 3 OSDs and 3 Monitors, one OSD+Mon per Controller node in an HA setup. Apex also supports non-HA deployments, which deploys a single controller and n number of compute nodes. Furthermore, Apex is capable of deploying scenarios in a bare metal or virtual fashion. Virtual deployments use multiple VMs on the jump host and internal networking to simulate the a bare metal deployment.

  • Documentation is built by Jenkins
  • .iso image is built by Jenkins
  • .rpm packages are built by Jenkins
  • Jenkins deploys a Colorado release with the Apex deployment toolchain bare metal, which includes 3 control+network nodes, and 2 compute nodes.

1.6. Release Data

Project apex
Repo/tag apex/colorado.3.0
Release designation colorado.3.0
Release date 2016-12-5
Purpose of the delivery OPNFV Colorado 3.0 release

1.6.1. Version change Module version changes

This is the second tracked version of the Colorado release with the Apex deployment toolchain. It is based on following upstream versions:

  • OpenStack (Mitaka release)
  • OpenDaylight (Beryllium/Boron releases)
  • CentOS 7 Document Version Changes

The following documentation is provided with this release:

  • OPNFV Installation instructions for the Colorado release with the Apex deployment toolchain - ver. 3.0
  • OPNFV Release Notes for the Colorado release with the Apex deployment toolchain - ver. 2.3.0 (this document) Feature Additions
JIRA: APEX-107 OpenDaylight HA - OVSDB Clustering
JIRA: APEX-108 Migrate to OpenStack Mitaka
JIRA: APEX-30 Support VLAN tagged deployments
JIRA: APEX-105 Enable Huge Page Configuration Options
JIRA: APEX-111 Allow RAM to be specified for Control/Compute in Virtual Deployments
JIRA: APEX-119 Enable OVS DPDK as a deployment Scenario in Apex
JIRA: APEX-126 Tacker Service deployed by Apex
JIRA: APEX-135 Congress Service deployed by Apex
JIRA: APEX-127 Nova Instance CPU Pinning
JIRA: APEX-130 IPv6 Underlay Deployment
JIRA: APEX-133 FDIO with Honeycomb Agent
JIRA: APEX-141 Integrate VSPERF into Apex
JIRA: APEX-172 Enable ONOS SFC Bug Corrections


JIRA: APEX-86 Need ability to specify number of compute nodes
JIRA: APEX-180 Baremetal deployment error: Failed to mount root partition /dev/sda on /mnt/rootfs
JIRA: APEX-161 Heat autoscaling stack creation fails for non-admin users
JIRA: APEX-198 Missing NAT iptables rule for public network in instack VM
JIRA: APEX-147 Installer doesn’t generate/distribute SSH keys between compute nodes
JIRA: APEX-109 ONOS routes local subnet traffic to GW
JIRA: APEX-146 Swift service present in available endpoints
JIRA: APEX-160 Enable force_metadata to support subnets with VM as the router
JIRA: APEX-114 OpenDaylight GUI is not available
JIRA: APEX-100 DNS1 and DNS2 should be handled in nic bridging
JIRA: APEX-100 DNS1 and DNS2 should be handled in nic bridging
JIRA: APEX-155 NIC Metric value not used when bridging NICs
JIRA: APEX-136 2 network deployment fails
JIRA: APEX-89 Deploy Ceph OSDs on compute nodes
JIRA: APEX-137 added arping ass dependency for ONOS deployments
JIRA: APEX-121 VM Storage deletion intermittently fails
JIRA: APEX-182 Nova services not correctly deployed
JIRA: APEX-153 brbm bridge not created in jumphost
JIRA: APEX-274 OVS DPDK scenario does not create vhost user ports
JIRA: APEX-277 ONOS fails to launch due to OVS and Kernel version mismatch
JIRA: APEX-179 DNS settings not working properly for baremetal deployments
JIRA: APEX-203 Swift proxy enabled and fails in noha deployment
JIRA: APEX-230 Tacker heat_uri setting is wrong
JIRA: APEX-241 Glance services are not running after reboot
JIRA: APEX-265 Metadata problems with os-odl_l2-sfc scenario
JIRA: APEX-271 L3 agent interface_driver set incorrectly for nosdn fdio noha
JIRA: APEX-275 Metadata fails in ODL Boron
JIRA: APEX-278 Duplicate neutron config class declaration for SDNVPN
JIRA: APEX-281 ONOS sometimes fails to provide addresses to instances
JIRA: APEX-283 Change networking-vpp ML2 driver to etcd
JIRA: APEX-284 vpp and hc versions for os-odl_l2 -fdio-noha scenario
JIRA: APEX-285 Remove vlan-strip-offload off config from os-odl_l2-fdio-noha scenario
JIRA: APEX-286 Make fdio repo an option in puppet- fdio
JIRA: APEX-289 Set undercloud VM to autostart for reboot
JIRA: APEX-291 link to apex installer missing on OPNFV download site
JIRA: APEX-293 tenant_nic is assumed to be a virtual name for nosdn fdio scenario
JIRA: APEX-294 dns server settings is ignored from network_settings.yaml
JIRA: APEX-297 Add details in “11.2 Creating a Node Inventory File”
JIRA: APEX-326 Consolidate SFC rpm into regular Apex rpm
JIRA: APEX-331 Conditionally install OVS NSH in ONOS Deployment
JIRA: APEX-332 IPv6 Scenario fails to deploy
JIRA: APEX-333 OVS DPDK does not work with UCS
JIRA: APEX-334 ONOS SFC not using SFC version of OVS
JIRA: APEX-335 Performance settings missing in os-odl_l2-fdio-ha.yaml
JIRA: APEX-163 Prefix to mask calculation doesn’t work
JIRA: APEX-266 ntp server shoudl be in network settings
JIRA: APEX-292 unclear that ip ranges in network settings may not overlap
JIRA: APEX-349 puppet-keystone failures in Colorado errors deplyoment
JIRA: APEX-344 Update vpp/hc from 16.09 to 17.01
JIRA: APEX-346 Update ODL to Carbon for FDIO scenario
JIRA: APEX-347 Add deploy setting for ceph_device
JIRA: APEX-341 FDIO scenarios: not supplying main-core and core-list workers causes deploy failure
JIRA: APEX-343 Update qemu for fdio scenarios
JIRA: APEX-344 nosdn fdio scenario missing iommu kernel params
JIRA: APEX-337 enable isolcpu on kernel and pin vpp to the unused CPU for FDIO scenarios
JIRA: APEX-338 Install Procedure for baremetal updated

1.6.2. Deliverables Software Deliverables
  • Apex .iso file
  • Apex release .rpm (opnfv-apex-release)
  • Apex overcloud .rpm (opnfv-apex) - For nosdn and OpenDaylight Scenarios
  • Apex overcloud onos .rpm (opnfv-apex-onos) - ONOS Scenarios
  • Apex undercloud .rpm (opnfv-apex-undercloud)
  • Apex common .rpm (opnfv-apex-common)
  • build.sh - Builds the above artifacts
  • opnfv-deploy - Automatically deploys Target OPNFV System
  • opnfv-clean - Automatically resets a Target OPNFV Deployment
  • opnfv-util - Utility to connect to or debug Overcloud nodes + OpenDaylight Documentation Deliverables
  • OPNFV Installation instructions for the Colorado release with the Apex deployment toolchain - ver. 3.0
  • OPNFV Release Notes for the Colorado release with the Apex deployment toolchain - ver. 2.2.0 (this document)

1.7. Known Limitations, Issues and Workarounds

1.7.1. System Limitations

Max number of blades: 1 Apex undercloud, 3 Controllers, 20 Compute blades

Min number of blades: 1 Apex undercloud, 1 Controller, 1 Compute blade

Storage: Ceph is the only supported storage configuration.

Min master requirements: At least 16GB of RAM for baremetal jumphost, 24GB for virtual deployments (noHA).

1.7.2. Known Issues


JIRA: APEX-112 ODL route local subnet traffic to GW
JIRA: APEX-149 OpenFlow rules are populated slowly
JIRA: APEX-268 VMs with multiple floating IPs can only access first NIC
JIRA: APEX-270 Glance v2 api does not support url images
JIRA: APEX-272 Enable VLAN tenant networking and ability to configure range
JIRA: APEX-280 Deleted network not cleaned up on controller
JIRA: APEX-290 Instances cannot connect to metadata service after cluster reboot
JIRA: APEX-295 VLAN tenant networking does not work in OVS DPDK scenario
JIRA: APEX-339 OVS DPDK PMD is not CPU pinned
JIRA: APEX-355 Undercloud CPU, RAM, Disk cannot be adjusted
JIRA: APEX-345 Dashboard links time-out due to running out of file descriptors for mariadb
JIRA: APEX-215 Keystone services not configured and the error is silently ignored (VLAN Deployments)
JIRA: APEX-208 Need ability to specify which NIC to place VLAN on
JIRA: APEX-254 Add dynamic hugepages configuration
JIRA: APEX-138 Unclear error message when interface set to dhcp

1.7.3. Workarounds


1.8. Scenario specific release notes

1.8.1. Scenario os-odl_l3-nofeature known issues

  • APEX-112:

    ODL routes local subnet traffic to GW

1.8.2. Scenario os-odl_l2-nofeature known issues

  • APEX-149:

    Openflow rules are populated very slowly

1.8.3. Scenario os-odl_l2-bgpvpn known issues


1.8.4. Scenario os-onos-nofeature known issues


1.8.5. Scenario os-onos-sfc known issues

  • APEX-281:

    ONOS sometimes fails to provide addresses to instances

1.8.6. Scenario os-odl_l2-sfc-noha known issues


1.8.7. Scenario os-nosdn-ovs known issues


1.8.8. Scenario os-odl_l2-fdio-noha known issues

  • FDS-121:

    qemu doesn’t receive connection on socket, vhost-user reconnect problem

  • FDS-62:

    APEX - Increase number of files MariaDB can open

  • FDS-79:

    Sometimes (especially in bulk crete/delete operations when multiple networks/ports are created within short time) OpenDaylight doesn’t accept creation requests

  • FDS-81:

    After functest finishes there are two bds on computes and none on controller

  • APEX-217:

    qemu not configured with correct group:user

  • APEX-337:

    enable isolcpu on kernel and pin vpp to unused cpu for fdio scenarios

1.8.9. Scenario os-nosdn-fdio-noha known issues

Note that a set of manual configration steps need to be performed post an automated deployment for the scenario to be fully functional. Please refer to FDS-159 and FDS-160 for details.

  • FDS-155:

    os-nosdn-fdio-noha scenario: tempest_smoke_serial causes mariadb/mysqld process to hang

  • FDS-156:

    os-nosdn-fdio-noha scenario: Race conditions for network-vif-plugged notification

  • FDS-157:

    os-nosdn-fdio-noha scenario: Intermittently VMs would get assigned 2 IPs instead of 1

  • FDS-158:

    os-nosdn-fdio-noha scenario: VM start/launch fails with “no more IP addresses” in neutron logs

  • FDS-159:

    os-nosdn-fdio-noha scenario: Security groups not yet supported

  • FDS-160:

    os-nosdn-fdio-noha scenario: Vlan fix on controller

  • FDS-161:

    os-nosdn-fdio-noha scenario: VPP fails with certain UCS B-series blades

1.8.10. General HA scenario known issues

  • COPPER-22:

    Congress service HA deployment is not yet supported/verified.

  • APEX-276:

    ODL HA unstable and crashes frequently

1.9. Test Result

The Colorado release with the Apex deployment toolchain has undergone QA test runs with the OPNFV FuncTest project. The latest results per scenario can be found here for Colorado release.

1.10. References

For more information on the OPNFV Colorado release, please see:


Authors:Tim Rozet (trozet@redhat.com)
Authors:Dan Radez (dradez@redhat.com)