/usr/lib/python3.6/site-packages/paramiko/client.py:779: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.46: b'396005631626ac370e20e370d4f423aa' key.get_name(), hostname, hexlify(key.get_fingerprint()), 2018-12-03 01:44:04,156 main.py 128 INFO doctor test starting....... 2018-12-03 01:44:04,156 apex.py 43 INFO Setup Apex installer start...... 2018-12-03 01:44:04,156 base.py 112 INFO Get SSH keys from apex installer...... 2018-12-03 01:44:04,515 apex.py 60 INFO Get overcloud config details from Apex installer...... 2018-12-03 01:44:04,515 base.py 131 INFO Run command=source stackrc; nova list | grep ' overcloud-' in apex installer...... 2018-12-03 01:44:07,844 base.py 140 INFO Output=['| 8fa1f58a-96bd-4229-a02b-8f415f9fb3a9 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.30.9.3 |', '| 935623c4-990b-4ddc-aef8-5c11fde0d741 | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.30.9.6 |', '| 41fc9f87-7532-4af1-a653-39a9c069b31b | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.30.9.4 |'] command=source stackrc; nova list | grep ' overcloud-' in apex installer 2018-12-03 01:44:07,845 base.py 145 INFO Check command=grep docker /home/stack/deploy_command return in apex installer...... 2018-12-03 01:44:07,917 base.py 148 INFO return 0 2018-12-03 01:44:07,917 apex.py 73 INFO controller_ips:['192.30.9.3'] 2018-12-03 01:44:07,918 apex.py 74 INFO compute_ips:['192.30.9.6', '192.30.9.4'] 2018-12-03 01:44:07,918 apex.py 75 INFO use_containers:True 2018-12-03 01:44:08,663 apex.py 115 INFO Set apply patches start...... /usr/lib/python3.6/site-packages/paramiko/client.py:779: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.3: b'112a54baedadb616f1fcbfec6839b524' key.get_name(), hostname, hexlify(key.get_fingerprint()), 2018-12-03 01:44:09,188 base.py 62 INFO Setup ssh stunnel in apex installer...... 2018-12-03 01:44:09,189 base.py 75 INFO tunnel for port 12346 2018-12-03 01:44:09,194 image.py 48 INFO image create start...... 2018-12-03 01:44:15,382 image.py 68 INFO image create end...... 2018-12-03 01:44:15,382 user.py 70 INFO user create start...... 2018-12-03 01:44:15,589 user.py 86 INFO create project...... 2018-12-03 01:44:15,842 user.py 95 INFO test project 2018-12-03 01:44:16,084 user.py 103 INFO create user...... 2018-12-03 01:44:16,661 user.py 113 INFO test user 2018-12-03 01:44:16,859 user.py 127 INFO role _member_ already created...... 2018-12-03 01:44:16,860 user.py 128 INFO test role 2018-12-03 01:44:17,963 user.py 78 INFO user create end...... 2018-12-03 01:44:17,964 main.py 55 INFO doctor fault management test starting....... 2018-12-03 01:44:18,472 fault_management.py 65 INFO fault management setup...... 2018-12-03 01:44:18,473 user.py 190 INFO quota update start...... 2018-12-03 01:44:18,473 user.py 206 INFO default quota update start...... 2018-12-03 01:44:19,191 user.py 217 INFO user quota update start...... 2018-12-03 01:44:19,508 user.py 230 INFO quota update end...... 2018-12-03 01:44:19,509 network.py 41 INFO network create start....... 2018-12-03 01:44:20,826 network.py 47 INFO network create end....... 2018-12-03 01:44:20,827 network.py 49 INFO subnet create start....... 2018-12-03 01:44:21,660 network.py 58 INFO subnet create end....... 2018-12-03 01:44:21,660 instance.py 51 INFO instance create start...... 2018-12-03 01:44:25,313 instance.py 73 INFO instance create end...... 2018-12-03 01:44:25,314 instance.py 92 INFO wait for vm launch start...... 2018-12-03 01:44:35,080 instance.py 110 INFO wait for vm launch end...... 2018-12-03 01:44:35,080 alarm.py 45 INFO alarm create start...... 2018-12-03 01:44:38,230 alarm.py 81 INFO alarm create end...... 2018-12-03 01:44:38,230 sample.py 73 INFO sample inspector start...... 2018-12-03 01:44:39,989 sample.py 26 INFO sample consumer start...... * Serving Flask app "consumer" (lazy loading) * Environment: production WARNING: Do not use the development server in a production environment. Use a production WSGI server instead. * Debug mode: off * Serving Flask app "inspector" (lazy loading) * Environment: production WARNING: Do not use the development server in a production environment. Use a production WSGI server instead. * Debug mode: off * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit) * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit) 2018-12-03 01:44:41,241 apex.py 79 INFO Get host ip by hostname=overcloud-novacompute-1.opnfvlf.org from Apex installer...... 2018-12-03 01:44:41,241 base.py 131 INFO Run command=source stackrc; nova show overcloud-novacompute-1 | awk '/ ctlplane network /{print $5}' in apex installer...... 2018-12-03 01:44:43,919 base.py 140 INFO Output=['192.30.9.4'] command=source stackrc; nova show overcloud-novacompute-1 | awk '/ ctlplane network /{print $5}' in apex installer 2018-12-03 01:44:43,920 fault_management.py 118 INFO Get host info(name:overcloud-novacompute-1.opnfvlf.org, ip:192.30.9.4) which vm(doctor_vm0) launched at 2018-12-03 01:44:43,920 sample.py 30 INFO sample monitor start...... 2018-12-03 01:44:43,921 sample.py 85 INFO Starting Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.4) 2018-12-03 01:46:43,978 fault_management.py 89 INFO fault management start...... 2018-12-03 01:46:43,979 base.py 112 INFO Get SSH keys from apex installer...... 2018-12-03 01:46:43,979 base.py 116 INFO Already have SSH keys from apex installer...... /usr/lib/python3.6/site-packages/paramiko/client.py:779: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.4: b'd5704f60e0acc7680c96c400a82bdca2' key.get_name(), hostname, hexlify(key.get_fingerprint()), 2018-12-03 01:46:44,061 utils.py 91 INFO Copy /src/doctor-tests/doctor_tests/disable_network.sh -> disable_network.sh 2018-12-03 01:46:44,278 utils.py 72 INFO Executing: bash disable_network.sh > disable_network.log 2>&1 & 2018-12-03 01:46:44,328 utils.py 86 INFO *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 & 2018-12-03 01:46:44,329 fault_management.py 91 INFO fault management end...... 2018-12-03 01:46:45,494 sample.py 98 INFO doctor monitor detected at 1543801605.4946978 2018-12-03 01:46:45,495 sample.py 41 INFO sample monitor report error...... 2018-12-03 01:46:45,504 sample.py 202 INFO event posted in sample inspector at 1543801605.5041494 2018-12-03 01:46:45,504 sample.py 203 INFO sample inspector = 2018-12-03 01:46:45,505 sample.py 205 INFO sample inspector received data = b'[{"time": "2018-12-03T01:46:45.495237", "type": "compute.host.down", "details": {"hostname": "overcloud-novacompute-1.opnfvlf.org", "status": "down", "monitor": "monitor_sample", "monitor_event_id": "monitor_sample_event1"}}]' 2018-12-03 01:46:45,559 sample.py 149 INFO doctor mark host(overcloud-novacompute-1.opnfvlf.org) down at 1543801605.5596414 2018-12-03 01:46:45,643 sample.py 160 INFO doctor mark vm() error at 1543801605.6432326 127.0.0.1 - - [03/Dec/2018 01:46:45] "PUT /events HTTP/1.1" 200 - 2018-12-03 01:46:45,647 sample.py 101 INFO ping timeout, quit monitoring... 2018-12-03 01:46:45,811 sample.py 58 INFO doctor consumer notified at 1543801605.8117154 2018-12-03 01:46:45,812 sample.py 61 INFO sample consumer received data = {'severity': 'moderate', 'alarm_name': 'doctor_alarm0', 'current': 'alarm', 'alarm_id': '4ad08ef4-4b7b-43a1-8248-eef468734f0b', 'reason': 'Event hits the query .', 'reason_data': {'type': 'event', 'event': {'event_type': 'compute.instance.update', 'traits': [['resource_id', 1, 'f7f1c3d4-e924-46dd-bd4b-3792ebf05506'], ['ephemeral_gb', 2, 0], ['instance_type_id', 2, 29], ['user_id', 1, '22993ef28533463bb0edc7f5bb0c4331'], ['service', 1, 'compute'], ['state', 1, 'error'], ['old_state', 1, 'active'], ['project_id', 1, '0932506096f145008aae0a12cee6dbeb'], ['launched_at', 4, '2018-12-03T01:44:33'], ['disk_gb', 2, 1], ['instance_id', 1, 'f7f1c3d4-e924-46dd-bd4b-3792ebf05506'], ['host', 1, 'overcloud-controller-0.opnfvlf.org'], ['display_name', 1, 'doctor_vm0'], ['root_gb', 2, 1], ['tenant_id', 1, '0932506096f145008aae0a12cee6dbeb'], ['memory_mb', 2, 512], ['instance_type', 1, 'm1.tiny'], ['vcpus', 2, 1], ['request_id', 1, 'req-3ea71ee3-fc62-4ddd-8c8d-087dc8db3a1e']], 'message_signature': '2b3886f7fbd792660010d0891ac431569812936a59dc3c1cbd9d08331f66c95c', 'raw': {}, 'generated': '2018-12-03T01:46:45.631177', 'message_id': '5950969b-fe28-4bea-b29d-829a39ddc8d7'}}, 'previous': 'insufficient data'} 127.0.0.1 - - [03/Dec/2018 01:46:45] "POST /failure HTTP/1.1" 200 - 2018-12-03 01:47:14,373 fault_management.py 185 INFO doctor fault management notification_time=0.3170175552368164 2018-12-03 01:47:14,373 fault_management.py 188 INFO doctor fault management test successfully 2018-12-03 01:47:14,374 fault_management.py 198 INFO run doctor fault management profile....... 2018-12-03 01:47:14,374 base.py 112 INFO Get SSH keys from apex installer...... 2018-12-03 01:47:14,374 base.py 116 INFO Already have SSH keys from apex installer...... 2018-12-03 01:47:15,469 utils.py 91 INFO Copy disable_network.log -> /src/doctor-tests/doctor_tests/disable_network.log 2018-12-03 01:47:15,665 fault_management.py 155 INFO Get the disable_netork.log fromdown_host(host_name:overcloud-novacompute-1.opnfvlf.org, host_ip:192.30.9.4) 2018-12-03 01:47:15,667 profiler_poc.py 97 INFO Total time cost: 467(ms) ==============================================================================> |Monitor|Inspector |Controller|Notifier|Evaluator | |150 |65 |? |? |? | | | | | | | | | | | link down:0 | | | | | | | | | raw failure:150 | | | | | | | | found affected:? | | | | | | | set VM error:298 | | | | | | marked host down:215 | | | | | notified VM error:? | | | | transformed event:? | | | evaluated event:? | | fired alarm:? | received alarm:467 2018-12-03 01:47:15,667 fault_management.py 94 INFO fault management cleanup...... 2018-12-03 01:47:15,668 fault_management.py 136 INFO Already get the disable_netork.log from down_host...... 2018-12-03 01:47:17,760 sample.py 79 INFO sample inspector stop...... 2018-12-03 01:47:17,792 sample.py 217 INFO shutdown inspector app server at 1543801637.791968 127.0.0.1 - - [03/Dec/2018 01:47:17] "POST /events/shutdown HTTP/1.1" 200 - 2018-12-03 01:47:17,794 sample.py 35 INFO sample monitor stop...... 2018-12-03 01:47:17,795 sample.py 108 INFO Stopping Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.4) 2018-12-03 01:47:17,795 sample.py 31 INFO sample consumer stop...... 2018-12-03 01:47:17,800 sample.py 66 INFO shutdown consumer app server at 1543801637.8003285 127.0.0.1 - - [03/Dec/2018 01:47:17] "POST /shutdown HTTP/1.1" 200 - 2018-12-03 01:47:17,802 alarm.py 84 INFO alarm delete start....... 2018-12-03 01:47:17,856 alarm.py 93 INFO alarm delete end....... 2018-12-03 01:47:17,856 instance.py 76 INFO instance delete start....... 2018-12-03 01:47:46,818 instance.py 89 INFO instance delete end....... 2018-12-03 01:47:46,818 network.py 61 INFO subnet delete start....... 2018-12-03 01:47:47,900 network.py 64 INFO subnet delete end....... 2018-12-03 01:47:47,900 network.py 66 INFO network delete start....... 2018-12-03 01:47:48,569 network.py 69 INFO network delete end....... 2018-12-03 01:47:48,569 apex.py 167 INFO restore apply patches start...... 2018-12-03 01:47:48,570 image.py 71 INFO image delete start....... 2018-12-03 01:47:49,314 image.py 76 INFO image delete end....... 2018-12-03 01:47:49,315 user.py 163 INFO user delete start...... 2018-12-03 01:47:50,354 user.py 187 INFO user delete end...... Total time cost: 467(ms) ==============================================================================> |Monitor|Inspector |Controller|Notifier|Evaluator | |150 |65 |? |? |? | | | | | | | | | | | link down:0 | | | | | | | | | raw failure:150 | | | | | | | | found affected:? | | | | | | | set VM error:298 | | | | | | marked host down:215 | | | | | notified VM error:? | | | | transformed event:? | | | evaluated event:? | | fired alarm:? | received alarm:467