2018-03-18 23:08:49,965 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-18 23:08:49,966 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-18 23:08:54,520 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-18 23:08:54,521 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.5', 'overcloud-controller-1: 192.30.9.7', 'overcloud-controller-2: 192.30.9.10'] 2018-03-18 23:08:54,521 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.8', 'overcloud-novacompute-1: 192.30.9.4'] 2018-03-18 23:08:54,521 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-18 23:09:08,363 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-18 23:09:08,407 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-18 23:09:21,695 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-18 23:09:21,950 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-18 23:09:21,950 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-18 23:11:14,101 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-18 23:11:27,007 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-18 23:11:27,007 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-18 23:12:48,744 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-18 23:13:14,137 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:13:51,788 - baro_tests.barometer - INFO - PASS 2018-03-18 23:13:51,789 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Hugepages. 2018-03-18 23:14:56,139 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-18 23:15:22,956 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:15:25,499 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-18 23:15:28,180 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-18 23:15:30,924 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-18 23:16:36,368 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-18 23:17:02,528 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:17:05,243 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-18 23:17:07,791 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-18 23:17:10,412 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-18 23:18:16,785 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-18 23:18:42,214 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:19:20,669 - baro_tests.barometer - INFO - PASS 2018-03-18 23:19:20,669 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on OVS events. 2018-03-18 23:20:28,109 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-18 23:20:53,249 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:21:30,468 - baro_tests.barometer - INFO - PASS 2018-03-18 23:21:30,468 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on OVS stats. 2018-03-18 23:22:38,427 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-18 23:23:03,421 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-18 23:23:22,185 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-18 23:23:22,185 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-18 23:24:25,868 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-18 23:24:50,406 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-18 23:24:52,731 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-18 23:24:55,055 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-18 23:24:57,410 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-18 23:24:57,411 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-18 23:24:57,411 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-18 23:25:09,801 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-18 23:25:20,947 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-18 23:25:31,249 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-18 23:26:43,739 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-18 23:27:09,326 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node ff5f26c1-4a53-4805-820d-6d3aafa29cdd... 2018-03-18 23:27:09,326 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-18 23:27:09,327 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-18 23:27:09,327 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-18 23:27:22,335 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:27:22,336 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521415251.42 ... 2018-03-18 23:27:22,336 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:22,336 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521415251.42 ... 2018-03-18 23:27:22,336 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:22,337 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521415251.42 ... 2018-03-18 23:27:22,337 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:22,337 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521415251.42 ... 2018-03-18 23:27:22,337 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:22,337 - baro_tests.barometer - INFO - OK 2018-03-18 23:27:22,337 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-18 23:27:34,348 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-18 23:27:47,951 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-18 23:27:47,952 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521415251.42, 1521415261.42), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521415251.42, 1521415261.42), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521415251.42, 1521415261.42), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521415251.42, 1521415261.42)] 2018-03-18 23:27:47,952 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521415271.42, 1521415281.42), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521415271.42, 1521415281.42), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521415271.42, 1521415281.42), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521415271.42, 1521415281.42)] 2018-03-18 23:27:47,953 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521415251.42 - hugepages-mm-2048Kb vmpage_number-free 1521415271.42 ... 2018-03-18 23:27:47,953 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:47,953 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521415251.42 - hugepages-mm-2048Kb vmpage_number-used 1521415271.42 ... 2018-03-18 23:27:47,953 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:47,954 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521415251.42 - hugepages-node0-2048Kb vmpage_number-free 1521415271.42 ... 2018-03-18 23:27:47,954 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:47,954 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521415251.42 - hugepages-node0-2048Kb vmpage_number-used 1521415271.42 ... 2018-03-18 23:27:47,954 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:27:47,954 - baro_tests.barometer - INFO - OK 2018-03-18 23:27:47,955 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-18 23:28:52,043 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-18 23:29:16,958 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node ff5f26c1-4a53-4805-820d-6d3aafa29cdd... 2018-03-18 23:29:16,959 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-18 23:29:16,959 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-18 23:29:16,960 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-18 23:29:30,838 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:29:30,839 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521415371.42 ... 2018-03-18 23:29:30,839 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:29:30,839 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521415371.42 ... 2018-03-18 23:29:30,839 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:29:30,840 - baro_tests.barometer - INFO - OK 2018-03-18 23:29:30,840 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-18 23:29:42,852 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-18 23:29:55,119 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-18 23:29:55,119 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521415371.42, 1521415381.42), ('intel_rdt-0-2', 'ipc', 1521415371.42, 1521415381.42)] 2018-03-18 23:29:55,120 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521415401.42, 1521415411.42), ('intel_rdt-0-2', 'ipc', 1521415401.42, 1521415411.42)] 2018-03-18 23:29:55,120 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521415371.42 - intel_rdt-0-2 bytes-llc 1521415401.42 ... 2018-03-18 23:29:55,120 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:29:55,120 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521415371.42 - intel_rdt-0-2 ipc 1521415401.42 ... 2018-03-18 23:29:55,121 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:29:55,121 - baro_tests.barometer - INFO - OK 2018-03-18 23:29:55,121 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-18 23:30:57,575 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-18 23:31:23,854 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node ff5f26c1-4a53-4805-820d-6d3aafa29cdd... 2018-03-18 23:31:23,854 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-18 23:31:23,854 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-18 23:31:23,855 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-18 23:31:36,763 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-18 23:31:36,900 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-18 23:31:37,033 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-18 23:31:37,136 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-18 23:31:37,137 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-18 23:31:37,140 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-18 23:32:41,268 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-18 23:33:05,684 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node ff5f26c1-4a53-4805-820d-6d3aafa29cdd... 2018-03-18 23:33:05,685 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-18 23:33:05,685 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-18 23:33:05,686 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-18 23:33:18,562 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:33:18,563 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521415617.42 ... 2018-03-18 23:33:18,563 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:33:18,563 - baro_tests.barometer - INFO - OK 2018-03-18 23:33:18,563 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-18 23:33:21,567 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-18 23:33:33,569 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-18 23:33:33,570 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521415617.421, 1521415618.421)] 2018-03-18 23:33:33,570 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521415632.421, 1521415633.421)] 2018-03-18 23:33:33,570 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521415617.42 - ovs_events-br0 gauge-link_status 1521415632.42 ... 2018-03-18 23:33:33,570 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:33:33,571 - baro_tests.barometer - INFO - OK 2018-03-18 23:33:33,571 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-18 23:34:37,041 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-18 23:35:03,364 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node ff5f26c1-4a53-4805-820d-6d3aafa29cdd... 2018-03-18 23:35:03,364 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-18 23:35:03,366 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-18 23:35:03,366 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-18 23:35:15,722 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:35:15,724 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521415734.42 ... 2018-03-18 23:35:15,724 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:35:15,724 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521415734.42 ... 2018-03-18 23:35:15,725 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:35:15,725 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521415734.42 ... 2018-03-18 23:35:15,725 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:35:15,725 - baro_tests.barometer - INFO - OK 2018-03-18 23:35:15,725 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-18 23:35:18,727 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-18 23:35:31,319 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-18 23:35:31,319 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521415734.421, 1521415735.421), ('ovs_stats-br0.br0', 'if_errors', 1521415734.421, 1521415735.421), ('ovs_stats-br0.br0', 'if_packets', 1521415734.421, 1521415735.421)] 2018-03-18 23:35:31,320 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521415749.421, 1521415750.421), ('ovs_stats-br0.br0', 'if_errors', 1521415749.421, 1521415750.421), ('ovs_stats-br0.br0', 'if_packets', 1521415749.421, 1521415750.421)] 2018-03-18 23:35:31,320 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521415734.42 - ovs_stats-br0.br0 if_dropped 1521415749.42 ... 2018-03-18 23:35:31,320 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:35:31,320 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521415734.42 - ovs_stats-br0.br0 if_errors 1521415749.42 ... 2018-03-18 23:35:31,321 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:35:31,321 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521415734.42 - ovs_stats-br0.br0 if_packets 1521415749.42 ... 2018-03-18 23:35:31,321 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:35:31,321 - baro_tests.barometer - INFO - OK 2018-03-18 23:35:31,321 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-18 23:35:44,714 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-18 23:35:57,588 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-18 23:35:57,590 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-18 23:37:17,771 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-18 23:37:43,092 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:38:20,375 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-18 23:38:20,376 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-18 23:38:20,376 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-18 23:39:25,441 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-18 23:39:50,098 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:40:27,293 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-18 23:40:27,294 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-18 23:40:27,294 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-03-18 23:41:32,003 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-18 23:41:56,989 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:41:59,561 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-18 23:42:02,257 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-18 23:42:04,857 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-18 23:43:08,229 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-18 23:43:33,050 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:44:10,766 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-18 23:44:10,766 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-18 23:44:10,766 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-18 23:45:16,037 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-18 23:45:42,920 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-18 23:46:20,434 - baro_tests.barometer - INFO - PASS 2018-03-18 23:46:20,435 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS stats. 2018-03-18 23:47:22,621 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-18 23:47:48,309 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-18 23:48:06,977 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-18 23:48:06,977 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-18 23:49:11,468 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-18 23:49:37,000 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-18 23:49:39,236 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-18 23:49:41,507 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-18 23:49:43,740 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-18 23:49:43,741 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-18 23:49:43,741 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-18 23:49:56,324 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-18 23:50:06,848 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-18 23:50:17,155 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-18 23:51:30,256 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-18 23:51:54,653 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node afc239f5-ca68-4a1b-b644-e8ee51e4a054... 2018-03-18 23:51:54,654 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-18 23:51:54,656 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-18 23:51:54,657 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-18 23:52:07,035 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:52:07,036 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521416743.23 ... 2018-03-18 23:52:07,036 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:07,037 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521416743.23 ... 2018-03-18 23:52:07,037 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:07,037 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521416743.23 ... 2018-03-18 23:52:07,037 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:07,037 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521416743.23 ... 2018-03-18 23:52:07,037 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:07,038 - baro_tests.barometer - INFO - OK 2018-03-18 23:52:07,038 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-18 23:52:19,048 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-18 23:52:31,954 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-18 23:52:31,955 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521416743.234, 1521416753.234), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521416743.234, 1521416753.234), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521416743.234, 1521416753.234), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521416743.234, 1521416753.234)] 2018-03-18 23:52:31,955 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521416773.234, 1521416783.234), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521416773.234, 1521416783.234), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521416773.234, 1521416783.234), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521416773.234, 1521416783.234)] 2018-03-18 23:52:31,955 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521416743.23 - hugepages-mm-2048Kb vmpage_number-free 1521416773.23 ... 2018-03-18 23:52:31,956 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:31,957 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521416743.23 - hugepages-mm-2048Kb vmpage_number-used 1521416773.23 ... 2018-03-18 23:52:31,958 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:31,960 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521416743.23 - hugepages-node0-2048Kb vmpage_number-free 1521416773.23 ... 2018-03-18 23:52:31,961 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:31,961 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521416743.23 - hugepages-node0-2048Kb vmpage_number-used 1521416773.23 ... 2018-03-18 23:52:31,961 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:52:31,961 - baro_tests.barometer - INFO - OK 2018-03-18 23:52:31,961 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-18 23:53:35,176 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-18 23:53:59,772 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node afc239f5-ca68-4a1b-b644-e8ee51e4a054... 2018-03-18 23:53:59,772 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-18 23:53:59,773 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-18 23:53:59,773 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-18 23:54:13,029 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:54:13,030 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521416873.23 ... 2018-03-18 23:54:13,030 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:54:13,030 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521416873.23 ... 2018-03-18 23:54:13,031 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:54:13,031 - baro_tests.barometer - INFO - OK 2018-03-18 23:54:13,032 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-18 23:54:25,044 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-18 23:54:37,691 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-18 23:54:37,692 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521416873.234, 1521416883.234), ('intel_rdt-0-2', 'ipc', 1521416873.234, 1521416883.234)] 2018-03-18 23:54:37,692 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521416903.234, 1521416913.234), ('intel_rdt-0-2', 'ipc', 1521416903.234, 1521416913.234)] 2018-03-18 23:54:37,693 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521416873.23 - intel_rdt-0-2 bytes-llc 1521416903.23 ... 2018-03-18 23:54:37,693 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:54:37,693 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521416873.23 - intel_rdt-0-2 ipc 1521416903.23 ... 2018-03-18 23:54:37,693 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:54:37,693 - baro_tests.barometer - INFO - OK 2018-03-18 23:54:37,693 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-18 23:55:38,281 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-18 23:56:03,534 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node afc239f5-ca68-4a1b-b644-e8ee51e4a054... 2018-03-18 23:56:03,534 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-18 23:56:03,535 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-18 23:56:03,536 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-18 23:56:16,108 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-18 23:56:16,240 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-18 23:56:16,354 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-18 23:56:16,460 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-18 23:56:16,460 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-18 23:56:16,461 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-18 23:57:19,549 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-18 23:57:43,623 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node afc239f5-ca68-4a1b-b644-e8ee51e4a054... 2018-03-18 23:57:43,623 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-18 23:57:43,623 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-18 23:57:43,624 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-18 23:57:55,992 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:57:55,993 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521417110.23 ... 2018-03-18 23:57:55,993 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:57:55,993 - baro_tests.barometer - INFO - OK 2018-03-18 23:57:55,994 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-18 23:57:58,996 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-18 23:58:11,518 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-18 23:58:11,519 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521417110.235, 1521417111.235)] 2018-03-18 23:58:11,519 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521417126.235, 1521417127.235)] 2018-03-18 23:58:11,519 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521417110.23 - ovs_events-br0 gauge-link_status 1521417126.23 ... 2018-03-18 23:58:11,519 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:58:11,520 - baro_tests.barometer - INFO - OK 2018-03-18 23:58:11,520 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-18 23:59:14,255 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-18 23:59:39,764 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node afc239f5-ca68-4a1b-b644-e8ee51e4a054... 2018-03-18 23:59:39,765 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-18 23:59:39,765 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-18 23:59:39,765 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-18 23:59:53,038 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-18 23:59:53,038 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521417227.23 ... 2018-03-18 23:59:53,038 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:59:53,038 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521417227.23 ... 2018-03-18 23:59:53,039 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:59:53,040 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521417227.23 ... 2018-03-18 23:59:53,041 - baro_tests.barometer - DEBUG - OK 2018-03-18 23:59:53,042 - baro_tests.barometer - INFO - OK 2018-03-18 23:59:53,042 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-18 23:59:56,045 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-19 00:00:08,217 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-19 00:00:08,217 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521417227.235, 1521417228.235), ('ovs_stats-br0.br0', 'if_errors', 1521417227.235, 1521417228.235), ('ovs_stats-br0.br0', 'if_packets', 1521417227.235, 1521417228.235)] 2018-03-19 00:00:08,217 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521417242.235, 1521417243.235), ('ovs_stats-br0.br0', 'if_errors', 1521417242.235, 1521417244.235), ('ovs_stats-br0.br0', 'if_packets', 1521417242.235, 1521417243.235)] 2018-03-19 00:00:08,218 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521417227.23 - ovs_stats-br0.br0 if_dropped 1521417242.23 ... 2018-03-19 00:00:08,218 - baro_tests.barometer - DEBUG - OK 2018-03-19 00:00:08,219 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521417227.23 - ovs_stats-br0.br0 if_errors 1521417242.23 ... 2018-03-19 00:00:08,221 - baro_tests.barometer - DEBUG - OK 2018-03-19 00:00:08,222 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521417227.23 - ovs_stats-br0.br0 if_packets 1521417242.23 ... 2018-03-19 00:00:08,223 - baro_tests.barometer - DEBUG - OK 2018-03-19 00:00:08,223 - baro_tests.barometer - INFO - OK 2018-03-19 00:00:08,225 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-19 00:00:20,224 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-19 00:00:20,225 - baro_tests.barometer - INFO - ====================================================================== 2018-03-19 00:00:20,227 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-19 00:00:20,227 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-19 00:00:20,228 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-19 00:00:20,229 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-19 00:00:20,230 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-19 00:00:20,231 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-19 00:00:20,232 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | FAIL | 2018-03-19 00:00:20,233 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-03-19 00:00:20,233 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-19 00:00:20,233 - baro_tests.barometer - INFO - | IN:OVS events | PASS | FAIL | 2018-03-19 00:00:20,233 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-19 00:00:20,234 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-19 00:00:20,234 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-19 00:00:20,235 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-19 00:00:20,235 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-19 00:00:20,236 - baro_tests.barometer - INFO - ====================================================================== 2018-03-19 00:00:20,236 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-19 00:00:20,236 - baro_tests.barometer - ERROR - Overall Result is Fail