2018-02-15 18:28:38,298 main.py 92 INFO doctor test starting....... 2018-02-15 18:28:38,298 apex.py 38 INFO Setup Apex installer start...... 2018-02-15 18:28:38,298 apex.py 51 INFO Get SSH keys from Apex installer...... 2018-02-15 18:28:38,367 apex.py 68 INFO Get controller ips from Apex installer...... 2018-02-15 18:28:41,947 apex.py 77 INFO Get controller_ips:['192.30.9.8', '192.30.9.5', '192.30.9.10'] from Apex installer 2018-02-15 18:28:41,947 apex.py 104 INFO Set apply patches start...... 2018-02-15 18:29:05,952 apex.py 94 INFO Setup ssh stunnel in controller nodes in Apex installer...... 2018-02-15 18:29:05,965 image.py 48 INFO image create start...... 2018-02-15 18:29:11,976 image.py 66 INFO image create end...... 2018-02-15 18:29:11,976 user.py 63 INFO user create start...... 2018-02-15 18:29:13,267 user.py 71 INFO user create end...... 2018-02-15 18:29:13,267 user.py 146 INFO user quota update start...... 2018-02-15 18:29:13,587 user.py 161 INFO user quota update end...... 2018-02-15 18:29:13,587 network.py 41 INFO network create start....... 2018-02-15 18:29:14,713 network.py 47 INFO network create end....... 2018-02-15 18:29:14,713 network.py 49 INFO subnet create start....... 2018-02-15 18:29:15,663 network.py 57 INFO subnet create end....... 2018-02-15 18:29:15,663 instance.py 51 INFO instance create start...... 2018-02-15 18:29:18,207 instance.py 72 INFO instance create end...... 2018-02-15 18:29:18,207 instance.py 91 INFO wait for vm launch start...... 2018-02-15 18:29:23,313 instance.py 109 INFO wait for vm launch end...... 2018-02-15 18:29:23,313 alarm.py 45 INFO alarm create start...... 2018-02-15 18:29:25,028 alarm.py 81 INFO alarm create end...... 2018-02-15 18:29:25,028 sample.py 69 INFO sample inspector start...... * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit) 2018-02-15 18:29:27,243 apex.py 81 INFO Get host ip from host name in Apex installer...... 2018-02-15 18:29:30,396 apex.py 90 INFO Get host_ip:['192.30.9.6'] from host_name:overcloud-novacompute-1.opnfvlf.org in Apex installer 2018-02-15 18:29:30,397 main.py 143 INFO Get host info(name:overcloud-novacompute-1.opnfvlf.org, ip:192.30.9.6) which vm(doctor_vm0) launched at 2018-02-15 18:29:30,397 sample.py 29 INFO sample monitor start...... 2018-02-15 18:29:30,398 sample.py 84 INFO Starting Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.6) 2018-02-15 18:29:30,398 sample.py 26 INFO sample consumer start...... * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit) 2018-02-15 18:30:30,400 network_failure.py 39 INFO fault inject start...... 2018-02-15 18:30:30,402 apex.py 51 INFO Get SSH keys from Apex installer...... 2018-02-15 18:30:30,402 apex.py 54 INFO Already have SSH keys from Apex installer...... 2018-02-15 18:30:30,485 utils.py 91 INFO Copy /src/doctor-tests/doctor_tests/disable_network.sh -> disable_network.sh 2018-02-15 18:30:30,613 utils.py 73 INFO Executing: bash disable_network.sh > disable_network.log 2>&1 & 2018-02-15 18:30:30,677 utils.py 86 INFO *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 & 2018-02-15 18:30:30,678 network_failure.py 42 INFO fault inject end...... 2018-02-15 18:30:31,881 sample.py 96 INFO doctor monitor detected at 1518719431.8818376 2018-02-15 18:30:31,882 sample.py 40 INFO sample monitor report error...... 2018-02-15 18:30:31,890 sample.py 156 INFO event posted in sample inspector at 1518719431.8904173 2018-02-15 18:30:31,890 sample.py 157 INFO sample inspector = 2018-02-15 18:30:31,891 sample.py 158 INFO sample inspector received data = b'[{"id": "monitor_sample_id1", "time": "2018-02-15T18:30:31.882373", "type": "compute.host.down", "details": {"hostname": "overcloud-novacompute-1.opnfvlf.org", "status": "down", "monitor": "monitor_sample", "monitor_event_id": "monitor_sample_event1"}}]' 2018-02-15 18:30:31,944 sample.py 108 INFO doctor mark host(overcloud-novacompute-1.opnfvlf.org) down at 1518719431.9440413 2018-02-15 18:30:32,032 sample.py 116 INFO doctor mark vm() error at 1518719432.0326905 127.0.0.1 - - [15/Feb/2018 18:30:32] "PUT /events HTTP/1.1" 200 - 2018-02-15 18:30:32,035 sample.py 98 INFO ping timeout, quit monitoring... 2018-02-15 18:30:32,177 sample.py 57 INFO doctor consumer notified at 1518719432.177645 2018-02-15 18:30:32,178 sample.py 58 INFO sample consumer received data = b'{"severity": "moderate", "alarm_name": "doctor_alarm0", "current": "alarm", "alarm_id": "720a576a-ace4-4782-8e3a-b37302738a82", "reason": "Event hits the query .", "reason_data": {"type": "event", "event": {"event_type": "compute.instance.update", "traits": [["resource_id", 1, "2b513aa2-f10d-4736-babd-ed922de6497e"], ["ephemeral_gb", 2, 0], ["instance_type_id", 2, 3], ["user_id", 1, "d997eca4035849bc99c6d1926f4771a2"], ["service", 1, "compute"], ["state", 1, "error"], ["old_state", 1, "active"], ["project_id", 1, "4be3f8fb9af04d2eb58e6765a1cacd2c"], ["launched_at", 4, "2018-02-15T18:23:40"], ["disk_gb", 2, 1], ["instance_id", 1, "2b513aa2-f10d-4736-babd-ed922de6497e"], ["host", 1, "overcloud-controller-1.opnfvlf.org"], ["root_gb", 2, 1], ["tenant_id", 1, "4be3f8fb9af04d2eb58e6765a1cacd2c"], ["memory_mb", 2, 512], ["instance_type", 1, "m1.tiny"], ["vcpus", 2, 1], ["request_id", 1, "req-0ccc7470-7484-43b2-9df8-57a44d03489d"]], "message_signature": "247d220c76b387d0f1bfa63a9d27fca0b9fbedd5c08b9f3fc47a4954b5c69371", "raw": {}, "generated": "2018-02-15T18:30:32.026949", "message_id": "73821899-e52c-486e-8ce5-e2246323c489"}}, "previous": "insufficient data"}' 127.0.0.1 - - [15/Feb/2018 18:30:32] "POST /failure HTTP/1.1" 200 - 2018-02-15 18:30:40,722 main.py 113 INFO doctor test successfully, notification_time=0.2958073616027832 2018-02-15 18:30:40,723 main.py 119 INFO doctor test begin to run profile....... 2018-02-15 18:30:40,723 apex.py 51 INFO Get SSH keys from Apex installer...... 2018-02-15 18:30:40,724 apex.py 54 INFO Already have SSH keys from Apex installer...... 2018-02-15 18:30:43,813 utils.py 91 INFO Copy disable_network.log -> /src/doctor-tests/doctor_tests/disable_network.log 2018-02-15 18:30:43,918 network_failure.py 62 INFO Get the disable_netork.log from down_host(host_name:overcloud-novacompute-1.opnfvlf.org, host_ip:192.30.9.6) 2018-02-15 18:30:43,919 network_failure.py 50 INFO Already get the disable_netork.log from down_host...... 2018-02-15 18:30:43,928 profiler_poc.py 96 INFO Total time cost: 342745(ms) ==============================================================================> |Monitor|Inspector |Controller|Notifier|Evaluator | |342449 |62 |? |? |? | | | | | | | | | | | link down:0 | | | | | | | | | raw failure:342449| | | | | | | | found affected:? | | | | | | | set VM error:342600| | | | | | marked host down:342511| | | | | notified VM error:? | | | | transformed event:? | | | evaluated event:? | | fired alarm:? | received alarm:342745 2018-02-15 18:30:46,006 sample.py 75 INFO sample inspector stop...... 2018-02-15 18:30:46,036 sample.py 165 INFO shutdown inspector app server at 1518719446.0361729 127.0.0.1 - - [15/Feb/2018 18:30:46] "POST /shutdown HTTP/1.1" 200 - 2018-02-15 18:30:46,038 sample.py 34 INFO sample monitor stop...... 2018-02-15 18:30:46,038 sample.py 105 INFO Stopping Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.6) 2018-02-15 18:30:46,039 sample.py 31 INFO sample consumer stop...... 2018-02-15 18:30:46,043 sample.py 64 INFO shutdown consumer app server at 1518719446.0432997 127.0.0.1 - - [15/Feb/2018 18:30:46] "POST /shutdown HTTP/1.1" 200 - 2018-02-15 18:30:46,045 apex.py 112 INFO restore apply patches start...... 2018-02-15 18:31:29,672 alarm.py 84 INFO alarm delete start....... 2018-02-15 18:31:30,047 alarm.py 93 INFO alarm delete end....... 2018-02-15 18:31:30,047 instance.py 75 INFO instance delete start....... 2018-02-15 18:31:32,568 instance.py 88 INFO instance delete end....... 2018-02-15 18:31:32,568 network.py 60 INFO subnet delete start....... 2018-02-15 18:31:33,335 network.py 63 INFO subnet delete end....... 2018-02-15 18:31:33,335 network.py 65 INFO network delete start....... 2018-02-15 18:31:33,969 network.py 68 INFO network delete end....... 2018-02-15 18:31:33,969 image.py 69 INFO image delete start....... 2018-02-15 18:31:35,104 image.py 74 INFO image delete end....... 2018-02-15 18:31:35,104 network_failure.py 45 INFO fault inject cleanup...... 2018-02-15 18:31:35,104 network_failure.py 50 INFO Already get the disable_netork.log from down_host...... 2018-02-15 18:31:35,104 user.py 122 INFO user delete start...... 2018-02-15 18:31:35,987 user.py 143 INFO user delete end...... Total time cost: 342745(ms) ==============================================================================> |Monitor|Inspector |Controller|Notifier|Evaluator | |342449 |62 |? |? |? | | | | | | | | | | | link down:0 | | | | | | | | | raw failure:342449| | | | | | | | found affected:? | | | | | | | set VM error:342600| | | | | | marked host down:342511| | | | | notified VM error:? | | | | transformed event:? | | | evaluated event:? | | fired alarm:? | received alarm:342745 2018-02-15 18:31:36,114 - functest.core.feature - INFO - Test result is stored in '/home/opnfv/functest/results/doctor-notification.log' 2018-02-15 18:34:54,653 - functest.core.feature - INFO - Test result is stored in '/home/opnfv/functest/results/functest-odl-sfc.log' 2018-02-15 18:35:14,858 - functest.core.feature - INFO - Test result is stored in '/home/opnfv/functest/results/domino-multinode.log'