Dovetail - user guide¶
Dovetail Introduction¶
Welcome to Dovetail’s documentation !
Dovetail is an OPNFV Project.
This project intends to define and provide a set of OPNFV related validation criteria that will provide input for the evaluation of the use of OPNFV trademarks. The dovetail project will be executed with the guidance and oversight of the Compliance and Certification committee and work to secure the goals of the C&C committee for each release.
This project intends to incrementally define qualification criteria that establish the foundations of how we are able to measure the ability to utilize the OPNFV platform, how the platform itself should behave, and how applications may be deployed on the platform.
Contact Dovetail¶
Feedback? Contact us
Revision: 6fc0897edb21e13718f5254b5bb28597191a2efb Build date: 2016-10-25
Revision: 6fc0897edb21e13718f5254b5bb28597191a2efb Build date: 2016-10-25
Config Template Syntax¶
Dovetail uses Functest/Yardstick Docker container to run its testcases. So you need to give
configurations for building the container and the commands it needs to do. In dovetail,
all of these are defined in config yaml files dovetail/dovetail/conf/functest_config.yml
and dovetail/dovetail/conf/yardstick_config.yml
.
Functest template syntax¶
For example, you can define your functest_config.yml
as:
functest:
image_name: opnfv/functest
docker_tag: latest
envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
-e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
-e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
opts: '-id --privileged=true'
pre_condition:
cmds:
- 'echo test for precondition'
testcase:
cmds:
- 'python /home/opnfv/repos/functest/ci/prepare_env.py start'
- 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r'
post_condition:
cmds:
- ''
result:
dir: '/home/opnfv/functest/results'
store_type: 'file'
file_path: 'tempest/tempest.log'
db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'
First, you need to give the image that you want to use for building functest/yardstick container.
Besides, there also need some envirnment parameters such as INSTALLER_TYPE
and INSTALLER_IP
and the options for you container. Then the functest/yardstick container can be build with your
settings.
Second, there need three kinds of commands, pre_condition
, testcase
and post_condition
.
If you want to do some cleanups or preparations, the commands can be put into pre_condition
section orderly. All commands in this section will just be executed once in the begining.
The testcase
section does the main jobs of the testing. All functest testcases will use the
container to execute these commands one by one. After finishing that, the test is accomplished
and the results are stored in files or uploaded to database. The post_condition
section
does some work such as clean Docker images or something else after all testcases finished.
All commands in this section will just execute once.
Besides, there need a result
section and it gives the directory of the functest/yardstick
results. The store_type
should be the same with the cmds in testcase
. That means if the
test results are stored in files, then store_type need to be file and the file_path is also
needed. If the test results are uploaded to database, then a db_url is needed for acquiring the results.
Yardstick template syntax¶
The framework of yardstick_config.yml
is almost the same as functest_config.yml
.
yardstick:
image_name: opnfv/yardstick
docker_tag: latest
envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
-e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
-e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal
-e EXTERNAL_NETWORK=ext-net'
opts: '-id --privileged=true'
pre_condition:
cmds:
- 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup'
- 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh'
testcase:
cmds:
- 'mkdir -p /home/opnfv/yardstick/results/'
- 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh &&
yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml
--output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &>
/home/opnfv/yardstick/results/yardstick.log'
post_condition:
cmds:
- ''
result:
dir: '/home/opnfv/yardstick/results'
store_type: 'file'
file_path: 'yardstick.log'
db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'
The main differences between yardstick_config.yml
and functest_config.yml
are the commands.
Jinja2 template syntax¶
Note that you can use jinja2 template for your parameters such as {{script_testcase}}
. The
parameters are defined in dovetail/dovetail/conf/dovetail_config.yml
:
parameters:
- name: testcase
path: '("name",)'
- name: script_testcase
path: '("scripts", "testcase")'
Here path
is the path in testcase config files that you can find the value of parameters. Take
script_testcase
as the example. For testcase dovetail.ipv6.tc001:
dovetail.ipv6.tc001:
name: dovetail.ipv6.tc001
objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation
scripts:
type: functest
testcase: tempest_smoke_serial
sub_testcase_list:
- tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
- tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
- tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
The path (“scripts”, “testcase”) means ‘testcase’ subsection of ‘scripts’ section. So follow
the path (“scripts”, “testcase”) we can get the value of {{script_testcase}}
that is
‘tempest_smoke_serial’.
Command Line Interface¶
Dovetail supports modifying some parameters at the run-time by using the command line interface. The commands can be defined through a config file by developers easily and be used when running certification manually. It now fits three kinds of running, directly running the python script, running after setup and running with docker container.
Define CLI with config file¶
For easy to be modified, dovetail provides dovetail/dovetail/conf/cmd_config.yml
to define CLI dynamically.
cli:
scenario:
flags:
- '--scenario'
default: 'basic'
help: 'certification scenario.'
SUT_TYPE:
flags:
- '--SUT_TYPE'
- '-TYPE'
help: 'Give SUT_TYPE.'
SUT_IP:
flags:
- '--SUT_IP'
- '-IP'
help: 'Give SUT_IP.'
The basic yaml file shown above contains three commands named scenario,
SUT_TYPE and SUT_IP, and all of them are options. Dovetail uses click
module in python to parse these commands, so for each command its keys should
be consistent with click’s keys with the exception of flags. There is a list
in flags such as ['--SUT_TYPE', '-TYPE']
defining the full name and
short name of the command. The full name is necessary but the short name is
optional. As you can see, the full name --SUT_TYPE
should be the same with
this block’s name SUT_TYPE. Other keys such as default and help are
the same as defined in python module click.
The first command scenario is used for choosing the scenario to be run, and
the cmd config file needs to include this command at least. All other commands
are the environment paramenters that will be used to run functest/yardstick
containers. For example, the default envs for functest container are given in
functest’s config file dovetail/dovetail/conf/functest_config.yml
:
envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
-e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
-e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
If running with the command python run.py --SUT_TYPE fuel
, then the envs
will be changed into
envs: '-e INSTALLER_TYPE=fuel -e INSTALLER_IP=192.168.200.2
-e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
-e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
The envs commands can be added or deleted just by modifying cmd_config.yml
rather than changing the source code. Now it only can dynamically modify envs
commands. If there is a need for adding non-envs command, besides adding it into
cmd_config.yml
, some other operations about the source code are also needed.
Run certification with CLI¶
For users, they can use CLI to input their own envs at the run-time instead of
modifying the config files of functest and yardstick. So dovetail can supports
different environments more flexible with CLI. Dovetail now can be run with three
methods, directly running run.py
script, running after setup and running
in docker container. The uses of CLI are almost the same for these three methods
and here takes the first one as the example.
All commands offered by dovetail can be listed by using help command --help
.
root@90256c4efd05:~/dovetail/dovetail$ python run.py --help
Usage: run.py [OPTIONS]
Dovetail certification test entry!
Options:
-TYPE, --SUT_TYPE TEXT Give SUT_TYPE.
--DEPLOY_SCENARIO TEXT Give DEPLOY_SCENARIO.
--DEPLOY_TYPE TEXT Give DEPLOY_TYPE.
-IP, --SUT_IP TEXT Give SUT_IP.
--CI_DEBUG TEXT Give CI_DEBUG.
--scenario TEXT certification scenario.
--help Show this message and exit.
All options listed can be used to input special environment values at the run-time. For example:
python run.py --SUT_TYPE compass --SUT_IP 192.168.200.2
There is no need to give all these commands. If it is not given by CLI, it will be set with the system’s environment value. If it is not included in system’s environment variables, it will be set with the default value in functest/yardstick config file.