2018-02-08 00:20:03,480 main.py 92 INFO doctor test starting....... 2018-02-08 00:20:03,480 apex.py 38 INFO Setup Apex installer start...... 2018-02-08 00:20:03,480 apex.py 51 INFO Get SSH keys from Apex installer...... 2018-02-08 00:20:03,550 apex.py 68 INFO Get controller ips from Apex installer...... 2018-02-08 00:20:07,032 apex.py 77 INFO Get controller_ips:['192.30.9.8', '192.30.9.9', '192.30.9.10'] from Apex installer 2018-02-08 00:20:07,032 apex.py 104 INFO Set apply patches start...... 2018-02-08 00:20:35,773 apex.py 94 INFO Setup ssh stunnel in controller nodes in Apex installer...... 2018-02-08 00:20:35,787 image.py 48 INFO image create start...... 2018-02-08 00:20:40,380 image.py 66 INFO image create end...... 2018-02-08 00:20:40,380 user.py 63 INFO user create start...... 2018-02-08 00:20:41,336 user.py 71 INFO user create end...... 2018-02-08 00:20:41,336 user.py 146 INFO user quota update start...... 2018-02-08 00:20:41,649 user.py 161 INFO user quota update end...... 2018-02-08 00:20:41,650 network.py 41 INFO network create start....... 2018-02-08 00:20:42,534 network.py 47 INFO network create end....... 2018-02-08 00:20:42,534 network.py 49 INFO subnet create start....... 2018-02-08 00:20:43,517 network.py 57 INFO subnet create end....... 2018-02-08 00:20:43,518 instance.py 51 INFO instance create start...... 2018-02-08 00:20:45,912 instance.py 72 INFO instance create end...... 2018-02-08 00:20:45,912 instance.py 91 INFO wait for vm launch start...... 2018-02-08 00:20:55,759 instance.py 109 INFO wait for vm launch end...... 2018-02-08 00:20:55,759 alarm.py 45 INFO alarm create start...... 2018-02-08 00:20:57,201 alarm.py 81 INFO alarm create end...... 2018-02-08 00:20:57,201 sample.py 69 INFO sample inspector start...... * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit) 2018-02-08 00:20:59,376 apex.py 81 INFO Get host ip from host name in Apex installer...... 2018-02-08 00:21:02,099 apex.py 90 INFO Get host_ip:['192.30.9.3'] from host_name:overcloud-novacompute-1.opnfvlf.org in Apex installer 2018-02-08 00:21:02,099 main.py 143 INFO Get host info(name:overcloud-novacompute-1.opnfvlf.org, ip:192.30.9.3) which vm(doctor_vm0) launched at 2018-02-08 00:21:02,099 sample.py 29 INFO sample monitor start...... 2018-02-08 00:21:02,100 sample.py 84 INFO Starting Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.3) 2018-02-08 00:21:02,100 sample.py 26 INFO sample consumer start...... * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit) 2018-02-08 00:22:02,161 network_failure.py 39 INFO fault inject start...... 2018-02-08 00:22:02,162 apex.py 51 INFO Get SSH keys from Apex installer...... 2018-02-08 00:22:02,163 apex.py 54 INFO Already have SSH keys from Apex installer...... 2018-02-08 00:22:02,243 utils.py 91 INFO Copy /src/doctor-tests/doctor_tests/disable_network.sh -> disable_network.sh 2018-02-08 00:22:02,353 utils.py 73 INFO Executing: bash disable_network.sh > disable_network.log 2>&1 & 2018-02-08 00:22:02,416 utils.py 86 INFO *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 & 2018-02-08 00:22:02,417 network_failure.py 42 INFO fault inject end...... 2018-02-08 00:22:03,572 sample.py 96 INFO doctor monitor detected at 1518049323.5720024 2018-02-08 00:22:03,572 sample.py 40 INFO sample monitor report error...... 2018-02-08 00:22:03,580 sample.py 156 INFO event posted in sample inspector at 1518049323.5800045 2018-02-08 00:22:03,580 sample.py 157 INFO sample inspector = 2018-02-08 00:22:03,581 sample.py 158 INFO sample inspector received data = b'[{"id": "monitor_sample_id1", "time": "2018-02-08T00:22:03.572522", "type": "compute.host.down", "details": {"hostname": "overcloud-novacompute-1.opnfvlf.org", "status": "down", "monitor": "monitor_sample", "monitor_event_id": "monitor_sample_event1"}}]' 2018-02-08 00:22:03,621 sample.py 108 INFO doctor mark host(overcloud-novacompute-1.opnfvlf.org) down at 1518049323.6216583 2018-02-08 00:22:03,734 sample.py 116 INFO doctor mark vm() error at 1518049323.7343845 127.0.0.1 - - [08/Feb/2018 00:22:03] "PUT /events HTTP/1.1" 200 - 2018-02-08 00:22:03,737 sample.py 98 INFO ping timeout, quit monitoring... 2018-02-08 00:22:03,924 sample.py 57 INFO doctor consumer notified at 1518049323.9248426 2018-02-08 00:22:03,925 sample.py 58 INFO sample consumer received data = b'{"severity": "moderate", "alarm_name": "doctor_alarm0", "current": "alarm", "alarm_id": "b90555ee-c40e-4a2c-96f5-bea2b17cb901", "reason": "Event hits the query .", "reason_data": {"type": "event", "event": {"event_type": "compute.instance.update", "traits": [["resource_id", 1, "0b76fe9a-7865-4c8c-8a84-cc5480b49345"], ["ephemeral_gb", 2, 0], ["instance_type_id", 2, 2], ["user_id", 1, "9479f6d6a3f54c04bc1ab92e7d93ede4"], ["service", 1, "compute"], ["state", 1, "error"], ["old_state", 1, "active"], ["project_id", 1, "c6a8e22f79b04534ac0f42d5e39a13e5"], ["launched_at", 4, "2018-02-08T00:15:17"], ["disk_gb", 2, 1], ["instance_id", 1, "0b76fe9a-7865-4c8c-8a84-cc5480b49345"], ["host", 1, "overcloud-controller-2.opnfvlf.org"], ["root_gb", 2, 1], ["tenant_id", 1, "c6a8e22f79b04534ac0f42d5e39a13e5"], ["memory_mb", 2, 512], ["instance_type", 1, "m1.tiny"], ["vcpus", 2, 1], ["request_id", 1, "req-50ac5ca4-9639-4834-82c3-8403f00b3fbb"]], "message_signature": "ccd86c5992003c06bafe5c12d1b96101ae7c6a4ca70cf94af5920b924c88d15b", "raw": {}, "generated": "2018-02-08T00:22:03.729497", "message_id": "afe7cc54-f92e-43b8-87ea-5e39a3f88a6a"}}, "previous": "insufficient data"}' 127.0.0.1 - - [08/Feb/2018 00:22:03] "POST /failure HTTP/1.1" 200 - 2018-02-08 00:22:12,462 main.py 113 INFO doctor test successfully, notification_time=0.3528401851654053 2018-02-08 00:22:12,462 main.py 119 INFO doctor test begin to run profile....... 2018-02-08 00:22:12,462 apex.py 51 INFO Get SSH keys from Apex installer...... 2018-02-08 00:22:12,462 apex.py 54 INFO Already have SSH keys from Apex installer...... 2018-02-08 00:22:15,551 utils.py 91 INFO Copy disable_network.log -> /src/doctor-tests/doctor_tests/disable_network.log 2018-02-08 00:22:15,651 network_failure.py 62 INFO Get the disable_netork.log from down_host(host_name:overcloud-novacompute-1.opnfvlf.org, host_ip:192.30.9.3) 2018-02-08 00:22:15,652 network_failure.py 50 INFO Already get the disable_netork.log from down_host...... 2018-02-08 00:22:15,660 profiler_poc.py 96 INFO Total time cost: 333193(ms) ==============================================================================> |Monitor|Inspector |Controller|Notifier|Evaluator | |332841 |49 |? |? |? | | | | | | | | | | | link down:0 | | | | | | | | | raw failure:332841| | | | | | | | found affected:? | | | | | | | set VM error:333003| | | | | | marked host down:332890| | | | | notified VM error:? | | | | transformed event:? | | | evaluated event:? | | fired alarm:? | received alarm:333193 2018-02-08 00:22:17,734 sample.py 75 INFO sample inspector stop...... 2018-02-08 00:22:17,762 sample.py 165 INFO shutdown inspector app server at 1518049337.7625782 127.0.0.1 - - [08/Feb/2018 00:22:17] "POST /shutdown HTTP/1.1" 200 - 2018-02-08 00:22:17,765 sample.py 34 INFO sample monitor stop...... 2018-02-08 00:22:17,765 sample.py 105 INFO Stopping Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.3) 2018-02-08 00:22:17,765 sample.py 31 INFO sample consumer stop...... 2018-02-08 00:22:17,770 sample.py 64 INFO shutdown consumer app server at 1518049337.7700226 127.0.0.1 - - [08/Feb/2018 00:22:17] "POST /shutdown HTTP/1.1" 200 - 2018-02-08 00:22:17,772 apex.py 112 INFO restore apply patches start...... 2018-02-08 00:22:29,469 alarm.py 84 INFO alarm delete start....... 2018-02-08 00:22:29,811 alarm.py 93 INFO alarm delete end....... 2018-02-08 00:22:29,811 instance.py 75 INFO instance delete start....... 2018-02-08 00:22:31,959 instance.py 88 INFO instance delete end....... 2018-02-08 00:22:31,959 network.py 60 INFO subnet delete start....... 2018-02-08 00:22:32,584 network.py 63 INFO subnet delete end....... 2018-02-08 00:22:32,584 network.py 65 INFO network delete start....... 2018-02-08 00:22:33,221 network.py 68 INFO network delete end....... 2018-02-08 00:22:33,221 image.py 69 INFO image delete start....... 2018-02-08 00:22:34,333 image.py 74 INFO image delete end....... 2018-02-08 00:22:34,333 network_failure.py 45 INFO fault inject cleanup...... 2018-02-08 00:22:34,334 network_failure.py 50 INFO Already get the disable_netork.log from down_host...... 2018-02-08 00:22:34,334 user.py 122 INFO user delete start...... 2018-02-08 00:22:35,417 user.py 143 INFO user delete end...... Total time cost: 333193(ms) ==============================================================================> |Monitor|Inspector |Controller|Notifier|Evaluator | |332841 |49 |? |? |? | | | | | | | | | | | link down:0 | | | | | | | | | raw failure:332841| | | | | | | | found affected:? | | | | | | | set VM error:333003| | | | | | marked host down:332890| | | | | notified VM error:? | | | | transformed event:? | | | evaluated event:? | | fired alarm:? | received alarm:333193 2018-02-08 00:22:35,560 - functest.core.feature - INFO - Test result is stored in '/home/opnfv/functest/results/doctor-notification.log' 2018-02-08 00:25:52,818 - functest.core.feature - INFO - Test result is stored in '/home/opnfv/functest/results/functest-odl-sfc.log' 2018-02-08 00:26:12,748 - functest.core.feature - INFO - Test result is stored in '/home/opnfv/functest/results/domino-multinode.log'