2018-04-15 02:50:30,262 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-04-15 02:50:30,263 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-04-15 02:50:33,878 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-04-15 02:50:33,878 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.3'] 2018-04-15 02:50:33,878 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.5', 'overcloud-novacompute-1: 192.30.9.10'] 2018-04-15 02:50:33,878 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-04-15 02:50:46,487 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-0... 2018-04-15 02:50:46,538 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-0... 2018-04-15 02:50:46,933 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-0. 2018-04-15 02:50:46,946 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-1... 2018-04-15 02:50:46,997 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-1... 2018-04-15 02:50:47,360 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-1. 2018-04-15 02:50:59,354 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-04-15 02:50:59,607 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-04-15 02:50:59,607 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-04-15 02:51:39,616 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-04-15 02:51:51,115 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-04-15 02:51:51,116 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-04-15 02:53:01,170 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-04-15 02:53:25,459 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 02:54:02,615 - baro_tests.barometer - INFO - PASS 2018-04-15 02:54:02,616 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Hugepages. 2018-04-15 02:54:50,762 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-04-15 02:55:26,116 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 02:56:03,197 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-15 02:56:03,197 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-15 02:56:03,197 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-04-15 02:56:50,488 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-15 02:57:14,064 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 02:57:51,217 - baro_tests.barometer - INFO - PASS 2018-04-15 02:57:51,218 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Mcelog. 2018-04-15 02:58:36,338 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-15 02:58:58,287 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 02:59:00,916 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-04-15 02:59:47,040 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-04-15 03:00:09,329 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 03:00:46,517 - baro_tests.barometer - INFO - PASS 2018-04-15 03:00:46,517 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on OVS stats. 2018-04-15 03:01:31,838 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-15 03:01:55,017 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-15 03:02:14,704 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-04-15 03:02:14,705 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-04-15 03:03:00,127 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-15 03:03:23,885 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-15 03:03:26,208 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-04-15 03:03:26,209 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-04-15 03:03:26,209 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-04-15 03:03:37,845 - baro_tests.barometer - INFO - Stopping collectd service... 2018-04-15 03:03:48,686 - baro_tests.barometer - INFO - Starting collectd service... 2018-04-15 03:03:58,979 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-04-15 03:04:53,517 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-04-15 03:05:16,126 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 0be3e3d5-4cda-48b1-82b6-f10564fce1df... 2018-04-15 03:05:16,127 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-15 03:05:16,127 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-04-15 03:05:16,127 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-04-15 03:05:27,683 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:05:27,684 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523761103.75 ... 2018-04-15 03:05:27,684 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:27,684 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523761103.75 ... 2018-04-15 03:05:27,684 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:27,685 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523761103.75 ... 2018-04-15 03:05:27,685 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:27,685 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523761103.75 ... 2018-04-15 03:05:27,685 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:27,686 - baro_tests.barometer - INFO - OK 2018-04-15 03:05:27,686 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-15 03:05:39,694 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-15 03:05:53,290 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:05:53,291 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523761103.75, 1523761113.751), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523761103.75, 1523761113.751), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523761103.751, 1523761113.751), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523761103.751, 1523761113.751)] 2018-04-15 03:05:53,291 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523761133.75, 1523761143.751), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523761133.75, 1523761143.751), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523761133.75, 1523761143.751), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523761133.75, 1523761143.751)] 2018-04-15 03:05:53,291 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523761103.75 - hugepages-mm-2048Kb vmpage_number-free 1523761133.75 ... 2018-04-15 03:05:53,292 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:53,293 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523761103.75 - hugepages-mm-2048Kb vmpage_number-used 1523761133.75 ... 2018-04-15 03:05:53,293 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:53,293 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523761103.75 - hugepages-node0-2048Kb vmpage_number-free 1523761133.75 ... 2018-04-15 03:05:53,293 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:53,293 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523761103.75 - hugepages-node0-2048Kb vmpage_number-used 1523761133.75 ... 2018-04-15 03:05:53,293 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:05:53,293 - baro_tests.barometer - INFO - OK 2018-04-15 03:05:53,294 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-04-15 03:06:39,681 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-04-15 03:07:14,062 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 0be3e3d5-4cda-48b1-82b6-f10564fce1df... 2018-04-15 03:07:14,062 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-15 03:07:14,062 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-04-15 03:07:14,063 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-04-15 03:07:25,852 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:07:25,853 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523761223.75 ... 2018-04-15 03:07:25,853 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:07:25,854 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523761223.75 ... 2018-04-15 03:07:25,854 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:07:25,854 - baro_tests.barometer - INFO - OK 2018-04-15 03:07:25,854 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-15 03:07:37,865 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-15 03:07:49,170 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:07:49,170 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523761223.75, 1523761233.75), ('intel_rdt-0-2', 'ipc', 1523761223.75, 1523761233.75)] 2018-04-15 03:07:49,171 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523761243.75, 1523761253.75), ('intel_rdt-0-2', 'ipc', 1523761243.75, 1523761253.75)] 2018-04-15 03:07:49,171 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523761223.75 - intel_rdt-0-2 bytes-llc 1523761243.75 ... 2018-04-15 03:07:49,171 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:07:49,172 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523761223.75 - intel_rdt-0-2 ipc 1523761243.75 ... 2018-04-15 03:07:49,172 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:07:49,172 - baro_tests.barometer - INFO - OK 2018-04-15 03:07:49,172 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-04-15 03:08:34,749 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-15 03:08:59,175 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 0be3e3d5-4cda-48b1-82b6-f10564fce1df... 2018-04-15 03:08:59,176 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-15 03:08:59,176 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-04-15 03:08:59,176 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-04-15 03:09:10,949 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:09:10,950 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523761343.75 ... 2018-04-15 03:09:10,950 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:10,950 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523761343.75 ... 2018-04-15 03:09:10,951 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:10,951 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523761343.75 ... 2018-04-15 03:09:10,951 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:10,951 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523761343.75 ... 2018-04-15 03:09:10,951 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:10,951 - baro_tests.barometer - INFO - OK 2018-04-15 03:09:10,952 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-15 03:09:13,953 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-15 03:09:26,574 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:09:26,574 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523761343.752, 1523761344.752), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523761343.752, 1523761344.752), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523761343.752, 1523761344.752), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523761343.752, 1523761345.752)] 2018-04-15 03:09:26,575 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523761359.752, 1523761360.752), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523761359.752, 1523761360.752), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523761359.752, 1523761360.752), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523761359.752, 1523761360.752)] 2018-04-15 03:09:26,575 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523761343.75 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523761359.75 ... 2018-04-15 03:09:26,575 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:26,575 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523761343.75 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523761359.75 ... 2018-04-15 03:09:26,576 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:26,576 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523761343.75 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523761359.75 ... 2018-04-15 03:09:26,576 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:26,576 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523761343.75 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523761359.75 ... 2018-04-15 03:09:26,576 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:09:26,576 - baro_tests.barometer - INFO - OK 2018-04-15 03:09:26,576 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Mcelog. 2018-04-15 03:10:10,940 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-15 03:10:33,536 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 0be3e3d5-4cda-48b1-82b6-f10564fce1df... 2018-04-15 03:10:33,536 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-15 03:10:33,537 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-04-15 03:10:33,537 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-04-15 03:10:44,743 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:10:44,744 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523761437.75 ... 2018-04-15 03:10:44,744 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:10:44,744 - baro_tests.barometer - INFO - OK 2018-04-15 03:10:44,744 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-15 03:10:47,748 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-15 03:10:58,479 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:10:58,479 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523761437.752, 1523761438.752)] 2018-04-15 03:10:58,479 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523761451.752, 1523761452.752)] 2018-04-15 03:10:58,480 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523761437.75 - ovs_events-br0 gauge-link_status 1523761451.75 ... 2018-04-15 03:10:58,480 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:10:58,480 - baro_tests.barometer - INFO - OK 2018-04-15 03:10:58,480 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-04-15 03:11:46,042 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-04-15 03:12:08,476 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 0be3e3d5-4cda-48b1-82b6-f10564fce1df... 2018-04-15 03:12:08,477 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-15 03:12:08,481 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-04-15 03:12:08,482 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-04-15 03:12:19,665 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:12:19,666 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523761531.75 ... 2018-04-15 03:12:19,666 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:12:19,667 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523761532.75 ... 2018-04-15 03:12:19,667 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:12:19,667 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523761532.75 ... 2018-04-15 03:12:19,667 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:12:19,667 - baro_tests.barometer - INFO - OK 2018-04-15 03:12:19,667 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-15 03:12:22,671 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-15 03:12:34,795 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:12:34,795 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523761531.752, 1523761533.752), ('ovs_stats-br0.br0', 'if_errors', 1523761532.752, 1523761533.752), ('ovs_stats-br0.br0', 'if_packets', 1523761532.752, 1523761533.752)] 2018-04-15 03:12:34,796 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523761547.752, 1523761548.752), ('ovs_stats-br0.br0', 'if_errors', 1523761547.752, 1523761548.752), ('ovs_stats-br0.br0', 'if_packets', 1523761547.752, 1523761548.752)] 2018-04-15 03:12:34,796 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523761531.75 - ovs_stats-br0.br0 if_dropped 1523761547.75 ... 2018-04-15 03:12:34,797 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:12:34,797 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523761532.75 - ovs_stats-br0.br0 if_errors 1523761547.75 ... 2018-04-15 03:12:34,797 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:12:34,797 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523761532.75 - ovs_stats-br0.br0 if_packets 1523761547.75 ... 2018-04-15 03:12:34,797 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:12:34,797 - baro_tests.barometer - INFO - OK 2018-04-15 03:12:34,797 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-04-15 03:12:46,078 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-04-15 03:12:57,574 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-04-15 03:12:57,574 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-04-15 03:14:08,506 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-04-15 03:14:32,513 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 03:15:09,662 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-15 03:15:09,663 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-15 03:15:09,664 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-04-15 03:15:53,228 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-04-15 03:16:26,926 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 03:17:04,101 - baro_tests.barometer - INFO - PASS 2018-04-15 03:17:04,101 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Intel RDT. 2018-04-15 03:17:49,913 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-15 03:18:11,017 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 03:18:48,292 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-15 03:18:48,292 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-15 03:18:48,292 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-04-15 03:19:34,219 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-15 03:19:58,884 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 03:20:36,082 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-15 03:20:36,089 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-15 03:20:36,093 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-04-15 03:21:21,444 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-04-15 03:21:42,746 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-15 03:22:20,301 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-15 03:22:20,302 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-15 03:22:20,302 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-04-15 03:23:06,167 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-15 03:23:29,215 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-15 03:23:47,941 - baro_tests.barometer - INFO - PASS 2018-04-15 03:23:47,945 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with AODH PASSED on Mcelog. 2018-04-15 03:24:32,771 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-15 03:24:55,016 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-15 03:25:13,733 - baro_tests.barometer - INFO - PASS 2018-04-15 03:25:13,735 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with AODH PASSED on OVS events. 2018-04-15 03:25:13,735 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-04-15 03:25:13,735 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-04-15 03:25:24,955 - baro_tests.barometer - INFO - Stopping collectd service... 2018-04-15 03:25:36,828 - baro_tests.barometer - INFO - Starting collectd service... 2018-04-15 03:25:47,752 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-04-15 03:26:41,797 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-04-15 03:27:04,411 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node d1e39a18-0461-4725-b593-4df33bb7fdf1... 2018-04-15 03:27:04,412 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-15 03:27:04,412 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-04-15 03:27:04,413 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-04-15 03:27:16,128 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:27:16,129 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523762424.97 ... 2018-04-15 03:27:16,129 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:16,129 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523762424.97 ... 2018-04-15 03:27:16,129 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:16,130 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523762424.97 ... 2018-04-15 03:27:16,130 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:16,130 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523762424.97 ... 2018-04-15 03:27:16,130 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:16,130 - baro_tests.barometer - INFO - OK 2018-04-15 03:27:16,130 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-15 03:27:28,142 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-15 03:27:39,549 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:27:39,549 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523762424.966, 1523762434.966), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523762424.966, 1523762434.966), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523762424.966, 1523762434.966), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523762424.966, 1523762434.966)] 2018-04-15 03:27:39,549 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523762454.966, 1523762464.966), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523762454.966, 1523762464.966), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523762454.966, 1523762464.966), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523762454.966, 1523762464.966)] 2018-04-15 03:27:39,549 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523762424.97 - hugepages-mm-2048Kb vmpage_number-free 1523762454.97 ... 2018-04-15 03:27:39,550 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:39,550 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523762424.97 - hugepages-mm-2048Kb vmpage_number-used 1523762454.97 ... 2018-04-15 03:27:39,550 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:39,550 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523762424.97 - hugepages-node0-2048Kb vmpage_number-free 1523762454.97 ... 2018-04-15 03:27:39,550 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:39,550 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523762424.97 - hugepages-node0-2048Kb vmpage_number-used 1523762454.97 ... 2018-04-15 03:27:39,551 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:27:39,551 - baro_tests.barometer - INFO - OK 2018-04-15 03:27:39,551 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-04-15 03:28:28,105 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-04-15 03:29:02,732 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node d1e39a18-0461-4725-b593-4df33bb7fdf1... 2018-04-15 03:29:02,732 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-15 03:29:02,733 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-04-15 03:29:02,733 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-04-15 03:29:13,529 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:29:13,529 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523762544.97 ... 2018-04-15 03:29:13,529 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:29:13,529 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523762544.97 ... 2018-04-15 03:29:13,530 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:29:13,530 - baro_tests.barometer - INFO - OK 2018-04-15 03:29:13,530 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-15 03:29:25,542 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-15 03:29:37,024 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:29:37,025 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523762544.966, 1523762554.966), ('intel_rdt-0-2', 'ipc', 1523762544.966, 1523762554.966)] 2018-04-15 03:29:37,025 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523762564.966, 1523762574.966), ('intel_rdt-0-2', 'ipc', 1523762564.966, 1523762574.966)] 2018-04-15 03:29:37,026 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523762544.97 - intel_rdt-0-2 bytes-llc 1523762564.97 ... 2018-04-15 03:29:37,026 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:29:37,026 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523762544.97 - intel_rdt-0-2 ipc 1523762564.97 ... 2018-04-15 03:29:37,026 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:29:37,026 - baro_tests.barometer - INFO - OK 2018-04-15 03:29:37,027 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-04-15 03:30:25,264 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-15 03:30:48,163 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node d1e39a18-0461-4725-b593-4df33bb7fdf1... 2018-04-15 03:30:48,164 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-15 03:30:48,164 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-04-15 03:30:48,164 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-04-15 03:30:59,391 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:30:59,391 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523762664.97 ... 2018-04-15 03:30:59,391 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:30:59,391 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523762664.97 ... 2018-04-15 03:30:59,392 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:30:59,392 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523762664.97 ... 2018-04-15 03:30:59,392 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:30:59,392 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523762664.97 ... 2018-04-15 03:30:59,392 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:30:59,393 - baro_tests.barometer - INFO - OK 2018-04-15 03:30:59,393 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-15 03:31:02,396 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-15 03:31:14,313 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:31:14,313 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523762664.967, 1523762665.967), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523762664.967, 1523762665.967), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523762664.967, 1523762665.967), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523762664.967, 1523762665.967)] 2018-04-15 03:31:14,313 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523762679.967, 1523762680.967), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523762679.967, 1523762680.967), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523762679.967, 1523762680.967), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523762679.967, 1523762680.967)] 2018-04-15 03:31:14,314 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523762664.97 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523762679.97 ... 2018-04-15 03:31:14,314 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:31:14,314 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523762664.97 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523762679.97 ... 2018-04-15 03:31:14,314 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:31:14,314 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523762664.97 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523762679.97 ... 2018-04-15 03:31:14,314 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:31:14,315 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523762664.97 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523762679.97 ... 2018-04-15 03:31:14,315 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:31:14,315 - baro_tests.barometer - INFO - OK 2018-04-15 03:31:14,315 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Mcelog. 2018-04-15 03:31:58,865 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-15 03:32:21,857 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node d1e39a18-0461-4725-b593-4df33bb7fdf1... 2018-04-15 03:32:21,857 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-15 03:32:21,858 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-04-15 03:32:21,858 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-04-15 03:32:33,137 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:32:33,137 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523762758.97 ... 2018-04-15 03:32:33,137 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:32:33,138 - baro_tests.barometer - INFO - OK 2018-04-15 03:32:33,138 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-15 03:32:36,141 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-15 03:32:47,597 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:32:47,598 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523762758.967, 1523762759.968)] 2018-04-15 03:32:47,598 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523762772.967, 1523762773.967)] 2018-04-15 03:32:47,598 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523762758.97 - ovs_events-br0 gauge-link_status 1523762772.97 ... 2018-04-15 03:32:47,599 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:32:47,599 - baro_tests.barometer - INFO - OK 2018-04-15 03:32:47,600 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-04-15 03:33:31,251 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-04-15 03:33:53,896 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node d1e39a18-0461-4725-b593-4df33bb7fdf1... 2018-04-15 03:33:53,897 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-15 03:33:53,897 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-04-15 03:33:53,897 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-04-15 03:34:05,395 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-15 03:34:05,395 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523762850.97 ... 2018-04-15 03:34:05,396 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:34:05,396 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523762850.97 ... 2018-04-15 03:34:05,396 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:34:05,397 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523762850.97 ... 2018-04-15 03:34:05,397 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:34:05,397 - baro_tests.barometer - INFO - OK 2018-04-15 03:34:05,397 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-15 03:34:08,400 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-15 03:34:19,118 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-15 03:34:19,119 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523762850.967, 1523762851.967), ('ovs_stats-br0.br0', 'if_errors', 1523762850.967, 1523762851.967), ('ovs_stats-br0.br0', 'if_packets', 1523762850.967, 1523762851.967)] 2018-04-15 03:34:19,119 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523762864.967, 1523762865.967), ('ovs_stats-br0.br0', 'if_errors', 1523762864.967, 1523762865.967), ('ovs_stats-br0.br0', 'if_packets', 1523762864.967, 1523762865.967)] 2018-04-15 03:34:19,120 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523762850.97 - ovs_stats-br0.br0 if_dropped 1523762864.97 ... 2018-04-15 03:34:19,120 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:34:19,120 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523762850.97 - ovs_stats-br0.br0 if_errors 1523762864.97 ... 2018-04-15 03:34:19,120 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:34:19,120 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523762850.97 - ovs_stats-br0.br0 if_packets 1523762864.97 ... 2018-04-15 03:34:19,120 - baro_tests.barometer - DEBUG - OK 2018-04-15 03:34:19,120 - baro_tests.barometer - INFO - OK 2018-04-15 03:34:19,121 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-04-15 03:34:31,370 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-04-15 03:34:31,371 - baro_tests.barometer - INFO - ====================================================================== 2018-04-15 03:34:31,371 - baro_tests.barometer - INFO - +----------------------------------+ 2018-04-15 03:34:31,372 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-04-15 03:34:31,372 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-15 03:34:31,372 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-04-15 03:34:31,373 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-15 03:34:31,373 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-04-15 03:34:31,373 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | FAIL | 2018-04-15 03:34:31,373 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | PASS | 2018-04-15 03:34:31,373 - baro_tests.barometer - INFO - | IN:Mcelog | PASS | FAIL | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | FAIL | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:Mcelog | PASS | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-15 03:34:31,374 - baro_tests.barometer - INFO - ====================================================================== 2018-04-15 03:34:31,374 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-04-15 03:34:31,375 - baro_tests.barometer - ERROR - Overall Result is Fail