2019-08-22 10:44:37,484 mcp.py 42 INFO Get SSH keys from MCP...... /usr/lib/python3.6/site-packages/paramiko/client.py:779: UserWarning: Unknown ssh-ed25519 host key for 10.20.0.2: b'f994c3bf92fe181f41ecadf298bc35e0' key.get_name(), hostname, hexlify(key.get_fingerprint()), 2019-08-22 10:44:37,561 main.py 130 INFO doctor test starting....... 2019-08-22 10:44:37,561 mcp.py 29 INFO Setup MCP installer start...... 2019-08-22 10:44:37,561 mcp.py 52 INFO Get controller ips from Mcp installer...... 2019-08-22 10:44:37,562 base.py 174 INFO Run command=sudo salt --out yaml 'ctl*' pillar.get _param:openstack_control_address |awk '{print $2}' in fuel installer...... 2019-08-22 10:44:38,343 base.py 183 INFO Output=['172.16.10.36'] command=sudo salt --out yaml 'ctl*' pillar.get _param:openstack_control_address |awk '{print $2}' in fuel installer 2019-08-22 10:44:38,344 mcp.py 59 INFO Get controller_ips:['172.16.10.36'] from Mcp installer 2019-08-22 10:44:39,027 mcp.py 70 INFO Set apply patches start...... 2019-08-22 10:44:39,027 base.py 63 INFO Setup ssh stunnel in fuel installer...... 2019-08-22 10:44:39,027 base.py 76 INFO tunnel for port 12346 2019-08-22 10:44:39,030 image.py 48 INFO image create start...... 2019-08-22 10:44:41,317 image.py 68 INFO image create end...... 2019-08-22 10:44:41,317 user.py 70 INFO user create start...... 2019-08-22 10:44:41,391 user.py 86 INFO create project...... 2019-08-22 10:44:41,482 user.py 95 INFO test project 2019-08-22 10:44:41,693 user.py 103 INFO create user...... 2019-08-22 10:44:42,106 user.py 113 INFO test user 2019-08-22 10:44:42,153 user.py 120 INFO create role...... 2019-08-22 10:44:42,208 user.py 128 INFO test role 2019-08-22 10:44:42,630 user.py 78 INFO user create end...... 2019-08-22 10:44:42,630 main.py 55 INFO doctor fault management test starting....... /usr/lib/python3.6/site-packages/paramiko/client.py:779: UserWarning: Unknown ssh-ed25519 host key for 172.16.10.36: b'fb32b6599c078454fb54017916e0535f' key.get_name(), hostname, hexlify(key.get_fingerprint()), 2019-08-22 10:44:43,279 fault_management.py 65 INFO fault management setup...... 2019-08-22 10:44:43,279 user.py 190 INFO quota update start...... 2019-08-22 10:44:43,280 user.py 206 INFO default quota update start...... 2019-08-22 10:44:44,423 user.py 217 INFO user quota update start...... 2019-08-22 10:44:44,528 user.py 230 INFO quota update end...... 2019-08-22 10:44:44,528 network.py 41 INFO network create start....... 2019-08-22 10:44:46,260 network.py 47 INFO network create end....... 2019-08-22 10:44:46,261 network.py 49 INFO subnet create start....... 2019-08-22 10:44:47,781 network.py 58 INFO subnet create end....... 2019-08-22 10:44:47,781 instance.py 51 INFO instance create start...... 2019-08-22 10:44:52,480 instance.py 73 INFO instance create end...... 2019-08-22 10:44:52,481 instance.py 92 INFO wait for vm launch start...... 2019-08-22 10:45:05,779 instance.py 110 INFO wait for vm launch end...... 2019-08-22 10:45:05,779 alarm.py 45 INFO alarm create start...... 2019-08-22 10:45:08,711 alarm.py 81 INFO alarm create end...... 2019-08-22 10:45:08,712 sample.py 85 INFO sample inspector start...... 2019-08-22 10:45:09,579 sample.py 26 INFO sample consumer start...... * Serving Flask app "inspector" (lazy loading) * Environment: production WARNING: Do not use the development server in a production environment. * Serving Flask app "consumer" (lazy loading) Use a production WSGI server instead. * Environment: production * Debug mode: off WARNING: Do not use the development server in a production environment. Use a production WSGI server instead. * Debug mode: off * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit) * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit) 2019-08-22 10:45:10,842 base.py 174 INFO Run command=sudo salt --out yaml 'cmp001*' pillar.get _param:single_address |awk '{print $2}' in fuel installer...... 2019-08-22 10:45:11,380 base.py 183 INFO Output=['172.16.10.55'] command=sudo salt --out yaml 'cmp001*' pillar.get _param:single_address |awk '{print $2}' in fuel installer 2019-08-22 10:45:11,381 fault_management.py 118 INFO Get host info(name:cmp001, ip:172.16.10.55) which vm(doctor_vm0) launched at 2019-08-22 10:45:11,381 sample.py 30 INFO sample monitor start...... 2019-08-22 10:45:11,382 sample.py 85 INFO Starting Pinger host_name(cmp001), host_ip(172.16.10.55) 2019-08-22 10:47:11,446 fault_management.py 89 INFO fault management start...... 2019-08-22 10:47:11,447 mcp.py 42 INFO Get SSH keys from MCP...... /usr/lib/python3.6/site-packages/paramiko/client.py:779: UserWarning: Unknown ssh-ed25519 host key for 172.16.10.55: b'b718a6acbc5e462559019980138ce83c' key.get_name(), hostname, hexlify(key.get_fingerprint()), 2019-08-22 10:47:11,551 utils.py 91 INFO Copy /src/doctor-tests/doctor_tests/disable_network.sh -> disable_network.sh 2019-08-22 10:47:11,855 utils.py 72 INFO Executing: bash disable_network.sh > disable_network.log 2>&1 & 2019-08-22 10:47:11,903 utils.py 86 INFO *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 & 2019-08-22 10:47:11,904 fault_management.py 91 INFO fault management end...... 2019-08-22 10:47:13,069 sample.py 98 INFO doctor monitor detected at 1566470833.0695248 2019-08-22 10:47:13,070 sample.py 41 INFO sample monitor report error...... 2019-08-22 10:47:13,076 sample.py 238 INFO event posted in sample inspector at 1566470833.0765946 2019-08-22 10:47:13,077 sample.py 239 INFO sample inspector = 2019-08-22 10:47:13,077 sample.py 241 INFO sample inspector received data = b'[{"time": "2019-08-22T10:47:13.070127", "type": "compute.host.down", "details": {"hostname": "cmp001", "status": "down", "monitor": "monitor_sample", "monitor_event_id": "monitor_sample_event1"}}]' 2019-08-22 10:47:13,139 sample.py 165 INFO doctor mark host(cmp001) down at 1566470833.139273 2019-08-22 10:47:13,325 sample.py 176 INFO doctor mark vm() error at 1566470833.3253937 127.0.0.1 - - [22/Aug/2019 10:47:13] "PUT /events HTTP/1.1" 200 - 2019-08-22 10:47:13,327 sample.py 101 INFO ping timeout, quit monitoring... 2019-08-22 10:47:41,957 main.py 82 ERROR doctor fault management test failed, Exception=doctor fault management test failed, detected_time=1566470833.0695248, notified_time=None 2019-08-22 10:47:41,958 main.py 83 ERROR Traceback (most recent call last): File "/src/doctor-tests/doctor_tests/main.py", line 78, in test_fault_management self.fault_management.check_notification_time() File "/src/doctor-tests/doctor_tests/scenario/fault_management.py", line 179, in check_notification_time self.consumer.notified_time)) Exception: doctor fault management test failed, detected_time=1566470833.0695248, notified_time=None 2019-08-22 10:47:41,958 fault_management.py 94 INFO fault management cleanup...... 2019-08-22 10:47:41,958 mcp.py 42 INFO Get SSH keys from MCP...... 2019-08-22 10:47:43,087 utils.py 91 INFO Copy disable_network.log -> /src/doctor-tests/doctor_tests/disable_network.log 2019-08-22 10:47:43,310 fault_management.py 155 INFO Get the disable_netork.log fromdown_host(host_name:cmp001, host_ip:172.16.10.55) 2019-08-22 10:47:45,416 sample.py 91 INFO sample inspector stop...... 2019-08-22 10:47:45,455 sample.py 253 INFO shutdown inspector app server at 1566470865.4551713 127.0.0.1 - - [22/Aug/2019 10:47:45] "POST /events/shutdown HTTP/1.1" 200 - 2019-08-22 10:47:45,456 sample.py 35 INFO sample monitor stop...... 2019-08-22 10:47:45,456 sample.py 108 INFO Stopping Pinger host_name(cmp001), host_ip(172.16.10.55) 2019-08-22 10:47:45,456 sample.py 31 INFO sample consumer stop...... 2019-08-22 10:47:45,460 sample.py 66 INFO shutdown consumer app server at 1566470865.4601436 127.0.0.1 - - [22/Aug/2019 10:47:45] "POST /shutdown HTTP/1.1" 200 - 2019-08-22 10:47:45,461 alarm.py 84 INFO alarm delete start....... 2019-08-22 10:47:45,519 alarm.py 93 INFO alarm delete end....... 2019-08-22 10:47:45,519 instance.py 76 INFO instance delete start....... 2019-08-22 10:48:22,806 instance.py 89 INFO instance delete end....... 2019-08-22 10:48:22,806 network.py 61 INFO subnet delete start....... 2019-08-22 10:48:23,584 network.py 64 INFO subnet delete end....... 2019-08-22 10:48:23,584 network.py 66 INFO network delete start....... 2019-08-22 10:48:24,575 network.py 69 INFO network delete end....... 2019-08-22 10:48:24,575 mcp.py 73 INFO restore apply patches start...... 2019-08-22 10:48:24,575 image.py 71 INFO image delete start....... 2019-08-22 10:48:24,697 image.py 76 INFO image delete end....... 2019-08-22 10:48:24,698 user.py 163 INFO user delete start...... 2019-08-22 10:48:25,466 user.py 187 INFO user delete end......