2018-03-12 13:17:33,569 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-12 13:17:33,570 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-12 13:17:37,258 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-12 13:17:37,259 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.9'] 2018-03-12 13:17:37,259 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.3', 'overcloud-novacompute-1: 192.30.9.6'] 2018-03-12 13:17:37,259 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-12 13:17:50,011 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-12 13:17:50,053 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-12 13:18:01,904 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-12 13:18:02,176 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-12 13:18:02,177 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-12 13:18:43,135 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-12 13:18:55,212 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-12 13:18:55,212 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-12 13:20:05,517 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-12 13:20:28,908 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:20:47,812 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-12 13:20:47,812 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-12 13:20:47,812 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-03-12 13:21:36,134 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-12 13:21:58,424 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:22:17,300 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-12 13:22:17,300 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-12 13:22:17,301 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-12 13:23:03,714 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-12 13:23:26,720 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:23:29,399 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-12 13:24:16,733 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-12 13:24:40,631 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:24:43,214 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-12 13:25:30,370 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-12 13:25:53,854 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:26:03,833 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-12 13:26:03,834 - baro_tests.barometer - INFO - Data not updated after 3 seconds 2018-03-12 13:26:03,834 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-12 13:26:51,441 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-12 13:27:13,959 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-12 13:27:33,297 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-12 13:27:33,297 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-12 13:28:20,476 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-12 13:28:43,104 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-12 13:28:45,218 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-12 13:28:45,219 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-12 13:28:45,219 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-12 13:28:56,694 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-12 13:29:08,654 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-12 13:29:19,681 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-12 13:30:15,352 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-12 13:30:38,005 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node fd953b58-5757-41c0-b72a-278fd97f7837... 2018-03-12 13:30:38,005 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-12 13:30:38,005 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb', 'hugepages-node1-2048Kb'] 2018-03-12 13:30:38,005 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-12 13:30:49,632 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:30:49,632 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1520861059.47 ... 2018-03-12 13:30:49,633 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:30:49,633 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1520861059.47 ... 2018-03-12 13:30:49,633 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:30:49,633 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1520861059.47 ... 2018-03-12 13:30:49,634 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:30:49,634 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1520861059.47 ... 2018-03-12 13:30:49,634 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:30:49,634 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-free 1520861059.47 ... 2018-03-12 13:30:49,635 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:30:49,635 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-used 1520861059.47 ... 2018-03-12 13:30:49,635 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:30:49,635 - baro_tests.barometer - INFO - OK 2018-03-12 13:30:49,635 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-12 13:31:01,646 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-12 13:31:14,124 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:31:14,124 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1520861059.472, 1520861069.471), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1520861059.472, 1520861069.471), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1520861059.472, 1520861069.472), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1520861059.472, 1520861069.472), ('hugepages-node1-2048Kb', 'vmpage_number-free', 1520861059.472, 1520861069.472), ('hugepages-node1-2048Kb', 'vmpage_number-used', 1520861059.472, 1520861069.472)] 2018-03-12 13:31:14,124 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1520861089.472, 1520861099.472), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1520861089.472, 1520861099.472), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1520861089.472, 1520861099.472), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1520861089.472, 1520861099.472), ('hugepages-node1-2048Kb', 'vmpage_number-free', 1520861089.472, 1520861099.472), ('hugepages-node1-2048Kb', 'vmpage_number-used', 1520861089.472, 1520861099.472)] 2018-03-12 13:31:14,125 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1520861059.47 - hugepages-mm-2048Kb vmpage_number-free 1520861089.47 ... 2018-03-12 13:31:14,125 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:31:14,125 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1520861059.47 - hugepages-mm-2048Kb vmpage_number-used 1520861089.47 ... 2018-03-12 13:31:14,126 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:31:14,126 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1520861059.47 - hugepages-node0-2048Kb vmpage_number-free 1520861089.47 ... 2018-03-12 13:31:14,126 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:31:14,126 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1520861059.47 - hugepages-node0-2048Kb vmpage_number-used 1520861089.47 ... 2018-03-12 13:31:14,126 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:31:14,126 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-free 1520861059.47 - hugepages-node1-2048Kb vmpage_number-free 1520861089.47 ... 2018-03-12 13:31:14,126 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:31:14,127 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-used 1520861059.47 - hugepages-node1-2048Kb vmpage_number-used 1520861089.47 ... 2018-03-12 13:31:14,127 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:31:14,127 - baro_tests.barometer - INFO - OK 2018-03-12 13:31:14,127 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-12 13:32:00,201 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-12 13:32:22,596 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node fd953b58-5757-41c0-b72a-278fd97f7837... 2018-03-12 13:32:22,596 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-12 13:32:22,597 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-12 13:32:22,597 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-12 13:32:34,166 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:32:34,166 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1520861169.47 ... 2018-03-12 13:32:34,167 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:32:34,167 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1520861169.47 ... 2018-03-12 13:32:34,167 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:32:34,168 - baro_tests.barometer - INFO - OK 2018-03-12 13:32:34,168 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-12 13:32:46,179 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-12 13:32:57,222 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:32:57,223 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1520861169.472, 1520861179.472), ('intel_rdt-0-2', 'ipc', 1520861169.472, 1520861179.472)] 2018-03-12 13:32:57,223 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1520861189.472, 1520861199.472), ('intel_rdt-0-2', 'ipc', 1520861189.472, 1520861199.472)] 2018-03-12 13:32:57,224 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1520861169.47 - intel_rdt-0-2 bytes-llc 1520861189.47 ... 2018-03-12 13:32:57,224 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:32:57,224 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1520861169.47 - intel_rdt-0-2 ipc 1520861189.47 ... 2018-03-12 13:32:57,224 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:32:57,225 - baro_tests.barometer - INFO - OK 2018-03-12 13:32:57,225 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-12 13:33:43,503 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-12 13:34:05,788 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node fd953b58-5757-41c0-b72a-278fd97f7837... 2018-03-12 13:34:05,789 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-12 13:34:05,789 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-12 13:34:05,789 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-12 13:34:17,641 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-12 13:34:17,791 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-12 13:34:17,898 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-12 13:34:18,006 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-12 13:34:18,006 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-12 13:34:18,007 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-12 13:35:04,161 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-12 13:35:27,440 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node fd953b58-5757-41c0-b72a-278fd97f7837... 2018-03-12 13:35:27,440 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-12 13:35:27,440 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-12 13:35:27,441 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-12 13:35:40,076 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:35:40,077 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1520861368.47 ... 2018-03-12 13:35:40,077 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:35:40,077 - baro_tests.barometer - INFO - OK 2018-03-12 13:35:40,078 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-12 13:35:43,081 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-12 13:35:55,170 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:35:55,170 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1520861368.473, 1520861369.473)] 2018-03-12 13:35:55,170 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1520861383.473, 1520861384.473)] 2018-03-12 13:35:55,171 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1520861368.47 - ovs_events-br0 gauge-link_status 1520861383.47 ... 2018-03-12 13:35:55,171 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:35:55,172 - baro_tests.barometer - INFO - OK 2018-03-12 13:35:55,175 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-12 13:36:42,568 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-12 13:37:04,693 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node fd953b58-5757-41c0-b72a-278fd97f7837... 2018-03-12 13:37:04,694 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-12 13:37:04,695 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-12 13:37:04,696 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-12 13:37:16,233 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:37:16,233 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1520861464.47 ... 2018-03-12 13:37:16,234 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:37:16,234 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1520861464.47 ... 2018-03-12 13:37:16,234 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:37:16,234 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1520861464.47 ... 2018-03-12 13:37:16,235 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:37:16,235 - baro_tests.barometer - INFO - OK 2018-03-12 13:37:16,235 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-12 13:37:19,238 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-12 13:37:30,203 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:37:30,203 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1520861464.473, 1520861465.473), ('ovs_stats-br0.br0', 'if_errors', 1520861464.473, 1520861465.473), ('ovs_stats-br0.br0', 'if_packets', 1520861464.473, 1520861465.473)] 2018-03-12 13:37:30,203 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1520861478.473, 1520861479.473), ('ovs_stats-br0.br0', 'if_errors', 1520861478.473, 1520861479.473), ('ovs_stats-br0.br0', 'if_packets', 1520861478.473, 1520861479.473)] 2018-03-12 13:37:30,204 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1520861464.47 - ovs_stats-br0.br0 if_dropped 1520861478.47 ... 2018-03-12 13:37:30,204 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:37:30,205 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1520861464.47 - ovs_stats-br0.br0 if_errors 1520861478.47 ... 2018-03-12 13:37:30,205 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:37:30,206 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1520861464.47 - ovs_stats-br0.br0 if_packets 1520861478.47 ... 2018-03-12 13:37:30,207 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:37:30,207 - baro_tests.barometer - INFO - OK 2018-03-12 13:37:30,207 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-12 13:37:40,108 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-12 13:37:51,360 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-12 13:37:51,360 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-12 13:38:56,191 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-12 13:39:19,584 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:39:38,290 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-12 13:39:38,290 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-12 13:39:38,291 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-12 13:40:24,657 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-12 13:40:48,153 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:41:07,147 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-12 13:41:07,147 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-12 13:41:07,147 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-03-12 13:41:52,910 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-12 13:42:15,747 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:42:18,361 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-12 13:43:03,665 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-12 13:43:25,857 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:43:28,400 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-12 13:44:14,672 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-12 13:44:36,675 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-12 13:44:46,596 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-12 13:44:46,597 - baro_tests.barometer - INFO - Data not updated after 3 seconds 2018-03-12 13:44:46,597 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-03-12 13:45:31,388 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-12 13:45:53,428 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-12 13:46:11,904 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-12 13:46:11,905 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-12 13:46:59,523 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-12 13:47:22,394 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-12 13:47:24,505 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-12 13:47:24,505 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-12 13:47:24,506 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-12 13:47:34,912 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-12 13:47:46,521 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-12 13:47:57,068 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-12 13:48:53,647 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-12 13:49:15,676 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node fcbd1b9f-5837-4a5b-ac25-b85a68cf7c82... 2018-03-12 13:49:15,677 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-12 13:49:15,677 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb', 'hugepages-node1-2048Kb'] 2018-03-12 13:49:15,678 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-12 13:49:28,206 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:49:28,207 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1520862200.56 ... 2018-03-12 13:49:28,207 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:28,207 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1520862200.56 ... 2018-03-12 13:49:28,208 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:28,208 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1520862200.56 ... 2018-03-12 13:49:28,208 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:28,208 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1520862200.56 ... 2018-03-12 13:49:28,208 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:28,208 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-free 1520862200.56 ... 2018-03-12 13:49:28,209 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:28,209 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-used 1520862200.56 ... 2018-03-12 13:49:28,209 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:28,209 - baro_tests.barometer - INFO - OK 2018-03-12 13:49:28,209 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-12 13:49:40,221 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-12 13:49:52,085 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:49:52,086 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1520862200.558, 1520862210.558), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1520862200.558, 1520862210.558), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1520862200.558, 1520862210.558), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1520862200.558, 1520862210.558), ('hugepages-node1-2048Kb', 'vmpage_number-free', 1520862200.558, 1520862210.558), ('hugepages-node1-2048Kb', 'vmpage_number-used', 1520862200.558, 1520862210.558)] 2018-03-12 13:49:52,086 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1520862220.558, 1520862230.557), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1520862220.558, 1520862230.557), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1520862220.558, 1520862230.558), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1520862220.558, 1520862230.558), ('hugepages-node1-2048Kb', 'vmpage_number-free', 1520862220.558, 1520862230.558), ('hugepages-node1-2048Kb', 'vmpage_number-used', 1520862220.558, 1520862230.558)] 2018-03-12 13:49:52,086 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1520862200.56 - hugepages-mm-2048Kb vmpage_number-free 1520862220.56 ... 2018-03-12 13:49:52,087 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:52,087 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1520862200.56 - hugepages-mm-2048Kb vmpage_number-used 1520862220.56 ... 2018-03-12 13:49:52,087 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:52,087 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1520862200.56 - hugepages-node0-2048Kb vmpage_number-free 1520862220.56 ... 2018-03-12 13:49:52,087 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:52,088 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1520862200.56 - hugepages-node0-2048Kb vmpage_number-used 1520862220.56 ... 2018-03-12 13:49:52,088 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:52,088 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-free 1520862200.56 - hugepages-node1-2048Kb vmpage_number-free 1520862220.56 ... 2018-03-12 13:49:52,088 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:52,088 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-used 1520862200.56 - hugepages-node1-2048Kb vmpage_number-used 1520862220.56 ... 2018-03-12 13:49:52,088 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:49:52,088 - baro_tests.barometer - INFO - OK 2018-03-12 13:49:52,088 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-12 13:50:37,411 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-12 13:51:00,093 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node fcbd1b9f-5837-4a5b-ac25-b85a68cf7c82... 2018-03-12 13:51:00,094 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-12 13:51:00,094 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-12 13:51:00,094 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-12 13:51:11,171 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:51:11,171 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1520862300.56 ... 2018-03-12 13:51:11,172 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:51:11,172 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1520862300.56 ... 2018-03-12 13:51:11,172 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:51:11,172 - baro_tests.barometer - INFO - OK 2018-03-12 13:51:11,172 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-12 13:51:23,184 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-12 13:51:34,773 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:51:34,774 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1520862300.558, 1520862310.558), ('intel_rdt-0-2', 'ipc', 1520862300.558, 1520862310.558)] 2018-03-12 13:51:34,774 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1520862320.558, 1520862330.558), ('intel_rdt-0-2', 'ipc', 1520862320.558, 1520862330.558)] 2018-03-12 13:51:34,774 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1520862300.56 - intel_rdt-0-2 bytes-llc 1520862320.56 ... 2018-03-12 13:51:34,775 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:51:34,775 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1520862300.56 - intel_rdt-0-2 ipc 1520862320.56 ... 2018-03-12 13:51:34,776 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:51:34,776 - baro_tests.barometer - INFO - OK 2018-03-12 13:51:34,777 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-12 13:52:20,467 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-12 13:52:41,418 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node fcbd1b9f-5837-4a5b-ac25-b85a68cf7c82... 2018-03-12 13:52:41,418 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-12 13:52:41,419 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-12 13:52:41,419 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-12 13:52:52,849 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-12 13:52:53,008 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-12 13:52:53,112 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-12 13:52:53,218 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-12 13:52:53,219 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-12 13:52:53,219 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-12 13:53:39,099 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-12 13:54:02,389 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node fcbd1b9f-5837-4a5b-ac25-b85a68cf7c82... 2018-03-12 13:54:02,389 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-12 13:54:02,390 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-12 13:54:02,390 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-12 13:54:14,434 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:54:14,434 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1520862496.56 ... 2018-03-12 13:54:14,435 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:54:14,435 - baro_tests.barometer - INFO - OK 2018-03-12 13:54:14,436 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-12 13:54:17,439 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-12 13:54:29,114 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:54:29,114 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1520862496.559, 1520862497.559)] 2018-03-12 13:54:29,114 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1520862511.559, 1520862512.558)] 2018-03-12 13:54:29,115 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1520862496.56 - ovs_events-br0 gauge-link_status 1520862511.56 ... 2018-03-12 13:54:29,115 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:54:29,115 - baro_tests.barometer - INFO - OK 2018-03-12 13:54:29,115 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-12 13:55:13,510 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-12 13:55:37,237 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node fcbd1b9f-5837-4a5b-ac25-b85a68cf7c82... 2018-03-12 13:55:37,238 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-12 13:55:37,238 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-12 13:55:37,238 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-12 13:55:48,405 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-12 13:55:48,405 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1520862590.56 ... 2018-03-12 13:55:48,405 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:55:48,406 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1520862590.56 ... 2018-03-12 13:55:48,406 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:55:48,406 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1520862590.56 ... 2018-03-12 13:55:48,406 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:55:48,406 - baro_tests.barometer - INFO - OK 2018-03-12 13:55:48,407 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-12 13:55:51,409 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-12 13:56:02,232 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-12 13:56:02,232 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1520862590.558, 1520862591.558), ('ovs_stats-br0.br0', 'if_errors', 1520862590.558, 1520862591.558), ('ovs_stats-br0.br0', 'if_packets', 1520862590.558, 1520862591.558)] 2018-03-12 13:56:02,233 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1520862603.558, 1520862604.558), ('ovs_stats-br0.br0', 'if_errors', 1520862604.558, 1520862605.558), ('ovs_stats-br0.br0', 'if_packets', 1520862604.558, 1520862605.558)] 2018-03-12 13:56:02,233 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1520862590.56 - ovs_stats-br0.br0 if_dropped 1520862603.56 ... 2018-03-12 13:56:02,233 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:56:02,234 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1520862590.56 - ovs_stats-br0.br0 if_errors 1520862604.56 ... 2018-03-12 13:56:02,234 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:56:02,234 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1520862590.56 - ovs_stats-br0.br0 if_packets 1520862604.56 ... 2018-03-12 13:56:02,234 - baro_tests.barometer - DEBUG - OK 2018-03-12 13:56:02,234 - baro_tests.barometer - INFO - OK 2018-03-12 13:56:02,234 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-12 13:56:13,072 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-12 13:56:13,073 - baro_tests.barometer - INFO - ====================================================================== 2018-03-12 13:56:13,073 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-12 13:56:13,074 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-12 13:56:13,074 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-12 13:56:13,075 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-12 13:56:13,075 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-12 13:56:13,075 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-12 13:56:13,076 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | FAIL | 2018-03-12 13:56:13,079 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-03-12 13:56:13,079 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-12 13:56:13,079 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-12 13:56:13,079 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-03-12 13:56:13,079 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-12 13:56:13,079 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-12 13:56:13,080 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-12 13:56:13,080 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-12 13:56:13,080 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-12 13:56:13,084 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-12 13:56:13,086 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-12 13:56:13,086 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-12 13:56:13,087 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-12 13:56:13,087 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-12 13:56:13,087 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-12 13:56:13,087 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-12 13:56:13,087 - baro_tests.barometer - INFO - ====================================================================== 2018-03-12 13:56:13,087 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-12 13:56:13,087 - baro_tests.barometer - ERROR - Overall Result is Fail