2018-03-20 18:41:25,035 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-20 18:41:25,035 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-20 18:41:29,273 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-20 18:41:29,273 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.7', 'overcloud-controller-1: 192.30.9.5', 'overcloud-controller-2: 192.30.9.3'] 2018-03-20 18:41:29,274 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.9', 'overcloud-novacompute-1: 192.30.9.10'] 2018-03-20 18:41:29,274 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-20 18:41:43,255 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-20 18:41:43,296 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-20 18:41:56,704 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-20 18:41:56,980 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-20 18:41:56,980 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-20 18:43:47,760 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-20 18:44:00,944 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-20 18:44:00,944 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-20 18:45:27,147 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-20 18:45:54,634 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 18:46:32,763 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-20 18:46:32,763 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-20 18:46:32,764 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-03-20 18:47:38,118 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-20 18:48:05,300 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 18:48:43,319 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-20 18:48:43,320 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-20 18:48:43,320 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-20 18:49:48,491 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-20 18:50:14,342 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 18:50:16,917 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-20 18:50:19,713 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-20 18:50:22,487 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-20 18:51:28,130 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-20 18:51:53,867 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 18:52:31,082 - baro_tests.barometer - INFO - PASS 2018-03-20 18:52:31,082 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on OVS events. 2018-03-20 18:53:35,622 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-20 18:53:59,127 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 18:54:36,248 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-20 18:54:36,248 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-20 18:54:36,249 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-20 18:55:42,442 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-20 18:56:07,312 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-20 18:56:27,227 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-20 18:56:27,227 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-20 18:57:29,763 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-20 18:57:53,487 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-20 18:57:55,725 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-20 18:57:58,323 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-20 18:58:00,676 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-20 18:58:00,677 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-20 18:58:00,677 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-20 18:58:12,860 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-20 18:58:23,894 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-20 18:58:34,196 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-20 18:59:50,077 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-20 19:00:17,116 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node f0715908-42f3-42b0-b3a4-c53df27b8106... 2018-03-20 19:00:17,116 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-20 19:00:17,117 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-20 19:00:17,118 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-20 19:00:30,699 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:00:30,699 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521572047.71 ... 2018-03-20 19:00:30,699 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:30,699 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521572047.71 ... 2018-03-20 19:00:30,700 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:30,700 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521572047.71 ... 2018-03-20 19:00:30,700 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:30,700 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521572047.71 ... 2018-03-20 19:00:30,700 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:30,700 - baro_tests.barometer - INFO - OK 2018-03-20 19:00:30,701 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-20 19:00:42,712 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-20 19:00:56,807 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:00:56,808 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521572047.714, 1521572057.714), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521572047.714, 1521572057.714), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521572047.714, 1521572057.714), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521572047.714, 1521572057.714)] 2018-03-20 19:00:56,808 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521572077.713, 1521572087.714), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521572077.713, 1521572087.714), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521572077.714, 1521572087.714), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521572077.714, 1521572087.714)] 2018-03-20 19:00:56,808 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521572047.71 - hugepages-mm-2048Kb vmpage_number-free 1521572077.71 ... 2018-03-20 19:00:56,809 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:56,809 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521572047.71 - hugepages-mm-2048Kb vmpage_number-used 1521572077.71 ... 2018-03-20 19:00:56,809 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:56,809 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521572047.71 - hugepages-node0-2048Kb vmpage_number-free 1521572077.71 ... 2018-03-20 19:00:56,809 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:56,809 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521572047.71 - hugepages-node0-2048Kb vmpage_number-used 1521572077.71 ... 2018-03-20 19:00:56,810 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:00:56,810 - baro_tests.barometer - INFO - OK 2018-03-20 19:00:56,810 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-20 19:02:02,658 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-20 19:02:28,817 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node f0715908-42f3-42b0-b3a4-c53df27b8106... 2018-03-20 19:02:28,817 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-20 19:02:28,817 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-20 19:02:28,818 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-20 19:02:42,442 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:02:42,442 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521572177.71 ... 2018-03-20 19:02:42,442 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:02:42,442 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521572177.71 ... 2018-03-20 19:02:42,443 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:02:42,443 - baro_tests.barometer - INFO - OK 2018-03-20 19:02:42,443 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-20 19:02:54,456 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-20 19:03:07,849 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:03:07,850 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521572177.714, 1521572187.714), ('intel_rdt-0-2', 'ipc', 1521572177.714, 1521572187.714)] 2018-03-20 19:03:07,850 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521572207.713, 1521572217.714), ('intel_rdt-0-2', 'ipc', 1521572207.713, 1521572217.714)] 2018-03-20 19:03:07,850 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521572177.71 - intel_rdt-0-2 bytes-llc 1521572207.71 ... 2018-03-20 19:03:07,850 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:03:07,851 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521572177.71 - intel_rdt-0-2 ipc 1521572207.71 ... 2018-03-20 19:03:07,851 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:03:07,851 - baro_tests.barometer - INFO - OK 2018-03-20 19:03:07,851 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-20 19:04:14,158 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-20 19:04:40,151 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node f0715908-42f3-42b0-b3a4-c53df27b8106... 2018-03-20 19:04:40,152 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-20 19:04:40,152 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-20 19:04:40,152 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-20 19:04:52,401 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-20 19:04:52,556 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-20 19:04:52,684 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-20 19:04:52,837 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-20 19:04:52,837 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-20 19:04:52,838 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-20 19:05:59,146 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-20 19:06:25,819 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node f0715908-42f3-42b0-b3a4-c53df27b8106... 2018-03-20 19:06:25,819 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-20 19:06:25,820 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-20 19:06:25,820 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-20 19:06:38,810 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:06:38,812 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521572430.71 ... 2018-03-20 19:06:38,813 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:06:38,813 - baro_tests.barometer - INFO - OK 2018-03-20 19:06:38,813 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-20 19:06:41,814 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-20 19:06:54,525 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:06:54,528 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521572430.715, 1521572431.715)] 2018-03-20 19:06:54,529 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521572445.715, 1521572446.715)] 2018-03-20 19:06:54,531 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521572430.71 - ovs_events-br0 gauge-link_status 1521572445.71 ... 2018-03-20 19:06:54,533 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:06:54,533 - baro_tests.barometer - INFO - OK 2018-03-20 19:06:54,533 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-20 19:07:58,777 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-20 19:08:23,681 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node f0715908-42f3-42b0-b3a4-c53df27b8106... 2018-03-20 19:08:23,683 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-20 19:08:23,685 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-20 19:08:23,686 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-20 19:08:37,654 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:08:37,655 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521572549.71 ... 2018-03-20 19:08:37,655 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:08:37,656 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521572549.71 ... 2018-03-20 19:08:37,656 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:08:37,656 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521572549.71 ... 2018-03-20 19:08:37,656 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:08:37,657 - baro_tests.barometer - INFO - OK 2018-03-20 19:08:37,657 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-20 19:08:40,660 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-20 19:08:54,137 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:08:54,137 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521572549.715, 1521572550.715), ('ovs_stats-br0.br0', 'if_errors', 1521572549.715, 1521572550.715), ('ovs_stats-br0.br0', 'if_packets', 1521572549.715, 1521572550.715)] 2018-03-20 19:08:54,138 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521572565.715, 1521572566.715), ('ovs_stats-br0.br0', 'if_errors', 1521572565.715, 1521572566.715), ('ovs_stats-br0.br0', 'if_packets', 1521572565.715, 1521572566.715)] 2018-03-20 19:08:54,138 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521572549.71 - ovs_stats-br0.br0 if_dropped 1521572565.71 ... 2018-03-20 19:08:54,139 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:08:54,139 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521572549.71 - ovs_stats-br0.br0 if_errors 1521572565.71 ... 2018-03-20 19:08:54,139 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:08:54,139 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521572549.71 - ovs_stats-br0.br0 if_packets 1521572565.71 ... 2018-03-20 19:08:54,139 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:08:54,139 - baro_tests.barometer - INFO - OK 2018-03-20 19:08:54,139 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-20 19:09:07,309 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-20 19:09:19,757 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-20 19:09:19,758 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-20 19:10:41,628 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-20 19:11:06,260 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 19:11:43,461 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-20 19:11:43,462 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-20 19:11:43,462 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-20 19:12:46,598 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-20 19:13:11,273 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 19:13:13,936 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-20 19:13:16,638 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-20 19:13:19,345 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-03-20 19:14:22,424 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-20 19:14:47,580 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 19:14:50,250 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-20 19:14:52,876 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-20 19:14:55,599 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-20 19:15:59,081 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-20 19:16:24,209 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 19:17:01,439 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-20 19:17:01,440 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-20 19:17:01,440 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-20 19:18:05,282 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-20 19:18:29,977 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-20 19:18:32,537 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-20 19:18:35,126 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-20 19:18:37,859 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-03-20 19:19:42,952 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-20 19:20:08,304 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-20 19:20:26,891 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-20 19:20:26,891 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-20 19:21:32,450 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-20 19:21:58,002 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-20 19:22:00,228 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-20 19:22:02,521 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-20 19:22:04,753 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-20 19:22:04,755 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-20 19:22:04,756 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-20 19:22:17,809 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-20 19:22:29,411 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-20 19:22:39,722 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-20 19:23:53,211 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-20 19:24:17,973 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 71291c56-a05f-44e2-8124-62c8cb8b4f71... 2018-03-20 19:24:17,974 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-20 19:24:17,974 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-20 19:24:17,975 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-20 19:24:30,278 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:24:30,278 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521573468.62 ... 2018-03-20 19:24:30,279 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:30,280 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521573468.62 ... 2018-03-20 19:24:30,281 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:30,281 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521573478.62 ... 2018-03-20 19:24:30,281 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:30,283 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521573478.62 ... 2018-03-20 19:24:30,283 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:30,284 - baro_tests.barometer - INFO - OK 2018-03-20 19:24:30,285 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-20 19:24:42,298 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-20 19:24:55,550 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:24:55,550 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521573468.619, 1521573478.619), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521573468.619, 1521573478.619), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521573478.619, 1521573488.619), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521573478.619, 1521573488.619)] 2018-03-20 19:24:55,551 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521573498.619, 1521573508.619), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521573498.619, 1521573508.619), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521573498.619, 1521573508.619), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521573498.619, 1521573508.619)] 2018-03-20 19:24:55,551 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521573468.62 - hugepages-mm-2048Kb vmpage_number-free 1521573498.62 ... 2018-03-20 19:24:55,551 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:55,552 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521573468.62 - hugepages-mm-2048Kb vmpage_number-used 1521573498.62 ... 2018-03-20 19:24:55,552 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:55,552 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521573478.62 - hugepages-node0-2048Kb vmpage_number-free 1521573498.62 ... 2018-03-20 19:24:55,552 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:55,552 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521573478.62 - hugepages-node0-2048Kb vmpage_number-used 1521573498.62 ... 2018-03-20 19:24:55,552 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:24:55,552 - baro_tests.barometer - INFO - OK 2018-03-20 19:24:55,552 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-20 19:25:59,518 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-20 19:26:24,423 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 71291c56-a05f-44e2-8124-62c8cb8b4f71... 2018-03-20 19:26:24,423 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-20 19:26:24,424 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-20 19:26:24,424 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-20 19:26:37,081 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:26:37,082 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521573598.62 ... 2018-03-20 19:26:37,082 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:26:37,082 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521573598.62 ... 2018-03-20 19:26:37,083 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:26:37,083 - baro_tests.barometer - INFO - OK 2018-03-20 19:26:37,083 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-20 19:26:49,093 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-20 19:27:01,582 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:27:01,583 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521573598.619, 1521573608.619), ('intel_rdt-0-2', 'ipc', 1521573598.619, 1521573608.619)] 2018-03-20 19:27:01,583 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521573628.619, 1521573638.619), ('intel_rdt-0-2', 'ipc', 1521573628.619, 1521573638.619)] 2018-03-20 19:27:01,583 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521573598.62 - intel_rdt-0-2 bytes-llc 1521573628.62 ... 2018-03-20 19:27:01,583 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:27:01,584 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521573598.62 - intel_rdt-0-2 ipc 1521573628.62 ... 2018-03-20 19:27:01,584 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:27:01,584 - baro_tests.barometer - INFO - OK 2018-03-20 19:27:01,584 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-20 19:28:02,777 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-20 19:28:28,865 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 71291c56-a05f-44e2-8124-62c8cb8b4f71... 2018-03-20 19:28:28,865 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-20 19:28:28,866 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-20 19:28:28,866 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-20 19:28:40,938 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-20 19:28:41,050 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-20 19:28:41,167 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-20 19:28:41,283 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-20 19:28:41,284 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-20 19:28:41,284 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-20 19:29:44,100 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-20 19:30:07,844 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 71291c56-a05f-44e2-8124-62c8cb8b4f71... 2018-03-20 19:30:07,844 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-20 19:30:07,845 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-20 19:30:07,847 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-20 19:30:20,532 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:30:20,532 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521573837.62 ... 2018-03-20 19:30:20,532 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:30:20,533 - baro_tests.barometer - INFO - OK 2018-03-20 19:30:20,533 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-20 19:30:23,536 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-20 19:30:36,082 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:30:36,083 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521573837.62, 1521573838.62)] 2018-03-20 19:30:36,084 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521573853.62, 1521573854.62)] 2018-03-20 19:30:36,086 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521573837.62 - ovs_events-br0 gauge-link_status 1521573853.62 ... 2018-03-20 19:30:36,091 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:30:36,094 - baro_tests.barometer - INFO - OK 2018-03-20 19:30:36,095 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-20 19:31:37,286 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-20 19:32:02,421 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 71291c56-a05f-44e2-8124-62c8cb8b4f71... 2018-03-20 19:32:02,421 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-20 19:32:02,422 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-20 19:32:02,422 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-20 19:32:15,956 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-20 19:32:15,956 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521573953.62 ... 2018-03-20 19:32:15,957 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:32:15,957 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521573953.62 ... 2018-03-20 19:32:15,957 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:32:15,957 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521573953.62 ... 2018-03-20 19:32:15,957 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:32:15,957 - baro_tests.barometer - INFO - OK 2018-03-20 19:32:15,958 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-20 19:32:18,959 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-20 19:32:32,274 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-20 19:32:32,274 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521573953.62, 1521573954.62), ('ovs_stats-br0.br0', 'if_errors', 1521573953.62, 1521573954.62), ('ovs_stats-br0.br0', 'if_packets', 1521573953.62, 1521573954.62)] 2018-03-20 19:32:32,275 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521573969.62, 1521573970.62), ('ovs_stats-br0.br0', 'if_errors', 1521573969.62, 1521573970.62), ('ovs_stats-br0.br0', 'if_packets', 1521573969.62, 1521573970.62)] 2018-03-20 19:32:32,275 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521573953.62 - ovs_stats-br0.br0 if_dropped 1521573969.62 ... 2018-03-20 19:32:32,275 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:32:32,275 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521573953.62 - ovs_stats-br0.br0 if_errors 1521573969.62 ... 2018-03-20 19:32:32,275 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:32:32,275 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521573953.62 - ovs_stats-br0.br0 if_packets 1521573969.62 ... 2018-03-20 19:32:32,275 - baro_tests.barometer - DEBUG - OK 2018-03-20 19:32:32,275 - baro_tests.barometer - INFO - OK 2018-03-20 19:32:32,275 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-20 19:32:44,591 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-20 19:32:44,592 - baro_tests.barometer - INFO - ====================================================================== 2018-03-20 19:32:44,593 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-20 19:32:44,594 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-20 19:32:44,594 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-20 19:32:44,595 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-20 19:32:44,595 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-20 19:32:44,595 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-20 19:32:44,595 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | FAIL | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | IN:OVS events | PASS | FAIL | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-20 19:32:44,596 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-20 19:32:44,597 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-20 19:32:44,597 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-20 19:32:44,597 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-20 19:32:44,597 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-20 19:32:44,597 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-20 19:32:44,597 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-20 19:32:44,597 - baro_tests.barometer - INFO - ====================================================================== 2018-03-20 19:32:44,597 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-20 19:32:44,597 - baro_tests.barometer - ERROR - Overall Result is Fail