2018-04-12 06:09:53,670 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-04-12 06:09:53,670 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-04-12 06:09:58,378 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-04-12 06:09:58,379 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.10'] 2018-04-12 06:09:58,379 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.8', 'overcloud-novacompute-1: 192.30.9.3'] 2018-04-12 06:09:58,379 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-04-12 06:10:10,194 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-0... 2018-04-12 06:10:10,268 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-0... 2018-04-12 06:10:10,638 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-0. 2018-04-12 06:10:10,650 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-1... 2018-04-12 06:10:10,699 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-1... 2018-04-12 06:10:11,074 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-1. 2018-04-12 06:10:22,459 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-04-12 06:10:22,712 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-04-12 06:10:22,713 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-04-12 06:11:02,203 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-04-12 06:11:14,915 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-04-12 06:11:14,916 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-04-12 06:12:30,561 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-04-12 06:12:53,639 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:13:30,919 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-12 06:13:30,919 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-12 06:13:30,920 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-04-12 06:14:17,341 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-04-12 06:14:52,123 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:15:29,334 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-12 06:15:29,334 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-12 06:15:29,334 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-04-12 06:16:16,709 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-12 06:16:40,321 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:17:17,344 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-12 06:17:17,344 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-12 06:17:17,344 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-04-12 06:18:04,039 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-12 06:18:26,789 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:19:03,819 - baro_tests.barometer - INFO - PASS 2018-04-12 06:19:03,820 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on OVS events. 2018-04-12 06:19:49,204 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-04-12 06:20:13,176 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:20:50,205 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-12 06:20:50,206 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-12 06:20:50,206 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-04-12 06:21:37,864 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-12 06:22:00,672 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-12 06:22:19,474 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-04-12 06:22:19,475 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-04-12 06:23:07,262 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-12 06:23:31,181 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-12 06:23:33,532 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-04-12 06:23:33,532 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-04-12 06:23:33,533 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-04-12 06:23:45,497 - baro_tests.barometer - INFO - Stopping collectd service... 2018-04-12 06:23:56,039 - baro_tests.barometer - INFO - Starting collectd service... 2018-04-12 06:24:06,331 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-04-12 06:25:03,493 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-04-12 06:25:26,321 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node c397fbfa-f854-49b5-ac87-a9518e0a8cc9... 2018-04-12 06:25:26,321 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-12 06:25:26,321 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-04-12 06:25:26,322 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-04-12 06:25:37,643 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:25:37,644 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523513936.46 ... 2018-04-12 06:25:37,644 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:25:37,645 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523513936.46 ... 2018-04-12 06:25:37,645 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:25:37,645 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523513936.46 ... 2018-04-12 06:25:37,645 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:25:37,646 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523513936.46 ... 2018-04-12 06:25:37,646 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:25:37,647 - baro_tests.barometer - INFO - OK 2018-04-12 06:25:37,647 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-12 06:25:49,653 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-12 06:26:02,236 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:26:02,236 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523513936.463, 1523513946.463), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523513936.463, 1523513946.463), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523513936.464, 1523513946.464), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523513936.464, 1523513946.464)] 2018-04-12 06:26:02,237 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523513956.463, 1523513966.463), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523513956.463, 1523513966.463), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523513956.464, 1523513966.464), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523513956.464, 1523513966.464)] 2018-04-12 06:26:02,237 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523513936.46 - hugepages-mm-2048Kb vmpage_number-free 1523513956.46 ... 2018-04-12 06:26:02,237 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:26:02,237 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523513936.46 - hugepages-mm-2048Kb vmpage_number-used 1523513956.46 ... 2018-04-12 06:26:02,237 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:26:02,237 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523513936.46 - hugepages-node0-2048Kb vmpage_number-free 1523513956.46 ... 2018-04-12 06:26:02,238 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:26:02,238 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523513936.46 - hugepages-node0-2048Kb vmpage_number-used 1523513956.46 ... 2018-04-12 06:26:02,238 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:26:02,238 - baro_tests.barometer - INFO - OK 2018-04-12 06:26:02,238 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-04-12 06:26:46,842 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-04-12 06:27:20,039 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node c397fbfa-f854-49b5-ac87-a9518e0a8cc9... 2018-04-12 06:27:20,039 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-12 06:27:20,039 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-04-12 06:27:20,040 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-04-12 06:27:32,076 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:27:32,077 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523514046.46 ... 2018-04-12 06:27:32,077 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:27:32,078 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523514046.46 ... 2018-04-12 06:27:32,078 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:27:32,078 - baro_tests.barometer - INFO - OK 2018-04-12 06:27:32,078 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-12 06:27:44,086 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-12 06:27:55,137 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:27:55,137 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523514046.463, 1523514056.463), ('intel_rdt-0-2', 'ipc', 1523514046.463, 1523514056.463)] 2018-04-12 06:27:55,137 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523514066.463, 1523514076.463), ('intel_rdt-0-2', 'ipc', 1523514066.463, 1523514076.463)] 2018-04-12 06:27:55,138 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523514046.46 - intel_rdt-0-2 bytes-llc 1523514066.46 ... 2018-04-12 06:27:55,138 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:27:55,138 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523514046.46 - intel_rdt-0-2 ipc 1523514066.46 ... 2018-04-12 06:27:55,139 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:27:55,139 - baro_tests.barometer - INFO - OK 2018-04-12 06:27:55,139 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-04-12 06:28:39,303 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-12 06:29:03,057 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node c397fbfa-f854-49b5-ac87-a9518e0a8cc9... 2018-04-12 06:29:03,058 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-12 06:29:03,058 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-04-12 06:29:03,058 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-04-12 06:29:14,205 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:29:14,205 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523514162.46 ... 2018-04-12 06:29:14,205 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:14,205 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523514162.46 ... 2018-04-12 06:29:14,206 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:14,206 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523514162.46 ... 2018-04-12 06:29:14,206 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:14,206 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523514162.46 ... 2018-04-12 06:29:14,207 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:14,207 - baro_tests.barometer - INFO - OK 2018-04-12 06:29:14,207 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-12 06:29:17,210 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-12 06:29:29,913 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:29:29,914 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523514162.464, 1523514163.464), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523514162.464, 1523514163.464), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523514162.464, 1523514163.464), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523514162.464, 1523514163.464)] 2018-04-12 06:29:29,914 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523514178.464, 1523514179.465), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523514178.464, 1523514179.465), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523514178.464, 1523514179.465), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523514178.464, 1523514179.465)] 2018-04-12 06:29:29,915 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523514162.46 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523514178.46 ... 2018-04-12 06:29:29,915 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:29,915 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523514162.46 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523514178.46 ... 2018-04-12 06:29:29,915 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:29,915 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523514162.46 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523514178.46 ... 2018-04-12 06:29:29,915 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:29,916 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523514162.46 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523514178.46 ... 2018-04-12 06:29:29,916 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:29:29,916 - baro_tests.barometer - INFO - OK 2018-04-12 06:29:29,916 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Mcelog. 2018-04-12 06:30:14,731 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-12 06:30:38,047 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node c397fbfa-f854-49b5-ac87-a9518e0a8cc9... 2018-04-12 06:30:38,048 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-12 06:30:38,048 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-04-12 06:30:38,048 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-04-12 06:30:49,602 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:30:49,602 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523514258.46 ... 2018-04-12 06:30:49,603 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:30:49,603 - baro_tests.barometer - INFO - OK 2018-04-12 06:30:49,603 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-12 06:30:52,606 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-12 06:31:04,341 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:31:04,341 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523514258.464, 1523514259.464)] 2018-04-12 06:31:04,341 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523514272.464, 1523514273.464)] 2018-04-12 06:31:04,342 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523514258.46 - ovs_events-br0 gauge-link_status 1523514272.46 ... 2018-04-12 06:31:04,342 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:31:04,342 - baro_tests.barometer - INFO - OK 2018-04-12 06:31:04,344 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-04-12 06:31:48,348 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-04-12 06:32:09,573 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node c397fbfa-f854-49b5-ac87-a9518e0a8cc9... 2018-04-12 06:32:09,574 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-12 06:32:09,574 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-04-12 06:32:09,575 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-04-12 06:32:20,748 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:32:20,748 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523514349.46 ... 2018-04-12 06:32:20,748 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:32:20,749 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523514349.46 ... 2018-04-12 06:32:20,749 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:32:20,749 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523514349.46 ... 2018-04-12 06:32:20,749 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:32:20,750 - baro_tests.barometer - INFO - OK 2018-04-12 06:32:20,750 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-12 06:32:23,753 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-12 06:32:35,673 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:32:35,674 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523514349.464, 1523514350.464), ('ovs_stats-br0.br0', 'if_errors', 1523514349.464, 1523514350.464), ('ovs_stats-br0.br0', 'if_packets', 1523514349.464, 1523514350.464)] 2018-04-12 06:32:35,674 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523514364.464, 1523514365.464), ('ovs_stats-br0.br0', 'if_errors', 1523514364.464, 1523514365.464), ('ovs_stats-br0.br0', 'if_packets', 1523514364.464, 1523514365.464)] 2018-04-12 06:32:35,675 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523514349.46 - ovs_stats-br0.br0 if_dropped 1523514364.46 ... 2018-04-12 06:32:35,675 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:32:35,676 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523514349.46 - ovs_stats-br0.br0 if_errors 1523514364.46 ... 2018-04-12 06:32:35,676 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:32:35,676 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523514349.46 - ovs_stats-br0.br0 if_packets 1523514364.46 ... 2018-04-12 06:32:35,676 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:32:35,676 - baro_tests.barometer - INFO - OK 2018-04-12 06:32:35,677 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-04-12 06:32:47,056 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-04-12 06:32:57,655 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-04-12 06:32:57,655 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-04-12 06:34:08,182 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-04-12 06:34:30,254 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:35:07,338 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-12 06:35:07,338 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-12 06:35:07,338 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-04-12 06:35:52,442 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-04-12 06:36:25,592 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:37:02,675 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-12 06:37:02,676 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-12 06:37:02,676 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-04-12 06:37:49,696 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-12 06:38:12,264 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:38:14,813 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-04-12 06:39:00,386 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-12 06:39:22,689 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:39:25,164 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-04-12 06:40:11,767 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-04-12 06:40:33,565 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-12 06:41:10,927 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-12 06:41:10,928 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-12 06:41:10,928 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-04-12 06:41:55,170 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-12 06:42:17,734 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-12 06:42:36,666 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-04-12 06:42:36,668 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-04-12 06:43:21,961 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-12 06:43:43,182 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-12 06:43:45,477 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-04-12 06:43:45,477 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-04-12 06:43:45,477 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-04-12 06:43:56,387 - baro_tests.barometer - INFO - Stopping collectd service... 2018-04-12 06:44:07,533 - baro_tests.barometer - INFO - Starting collectd service... 2018-04-12 06:44:17,849 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-04-12 06:45:13,251 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-04-12 06:45:34,660 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node dfbaaf7a-1eec-4fc1-a438-7b1f85542d6d... 2018-04-12 06:45:34,660 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-12 06:45:34,660 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-04-12 06:45:34,660 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-04-12 06:45:46,431 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:45:46,431 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523515125.63 ... 2018-04-12 06:45:46,431 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:45:46,431 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523515125.63 ... 2018-04-12 06:45:46,432 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:45:46,432 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523515125.63 ... 2018-04-12 06:45:46,432 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:45:46,432 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523515125.63 ... 2018-04-12 06:45:46,432 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:45:46,432 - baro_tests.barometer - INFO - OK 2018-04-12 06:45:46,432 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-12 06:45:58,439 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-12 06:46:10,522 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:46:10,522 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523515125.633, 1523515135.633), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523515125.633, 1523515135.633), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523515125.633, 1523515135.633), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523515125.633, 1523515135.633)] 2018-04-12 06:46:10,523 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523515155.633, 1523515165.633), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523515155.633, 1523515165.633), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523515155.633, 1523515165.633), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523515155.633, 1523515165.633)] 2018-04-12 06:46:10,523 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523515125.63 - hugepages-mm-2048Kb vmpage_number-free 1523515155.63 ... 2018-04-12 06:46:10,523 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:46:10,524 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523515125.63 - hugepages-mm-2048Kb vmpage_number-used 1523515155.63 ... 2018-04-12 06:46:10,524 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:46:10,524 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523515125.63 - hugepages-node0-2048Kb vmpage_number-free 1523515155.63 ... 2018-04-12 06:46:10,524 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:46:10,524 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523515125.63 - hugepages-node0-2048Kb vmpage_number-used 1523515155.63 ... 2018-04-12 06:46:10,524 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:46:10,524 - baro_tests.barometer - INFO - OK 2018-04-12 06:46:10,524 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-04-12 06:46:55,853 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-04-12 06:47:29,235 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node dfbaaf7a-1eec-4fc1-a438-7b1f85542d6d... 2018-04-12 06:47:29,236 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-12 06:47:29,237 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-04-12 06:47:29,237 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-04-12 06:47:39,922 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:47:39,922 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523515245.63 ... 2018-04-12 06:47:39,923 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:47:39,923 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523515245.63 ... 2018-04-12 06:47:39,923 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:47:39,924 - baro_tests.barometer - INFO - OK 2018-04-12 06:47:39,924 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-12 06:47:51,936 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-12 06:48:02,834 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:48:02,834 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523515245.633, 1523515255.633), ('intel_rdt-0-2', 'ipc', 1523515245.633, 1523515255.633)] 2018-04-12 06:48:02,834 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523515265.633, 1523515275.632), ('intel_rdt-0-2', 'ipc', 1523515265.633, 1523515275.632)] 2018-04-12 06:48:02,835 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523515245.63 - intel_rdt-0-2 bytes-llc 1523515265.63 ... 2018-04-12 06:48:02,835 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:48:02,835 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523515245.63 - intel_rdt-0-2 ipc 1523515265.63 ... 2018-04-12 06:48:02,835 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:48:02,836 - baro_tests.barometer - INFO - OK 2018-04-12 06:48:02,836 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-04-12 06:48:46,362 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-12 06:49:09,212 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node dfbaaf7a-1eec-4fc1-a438-7b1f85542d6d... 2018-04-12 06:49:09,213 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-12 06:49:09,213 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-04-12 06:49:09,213 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-04-12 06:49:21,685 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:49:21,686 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523515357.63 ... 2018-04-12 06:49:21,686 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:21,686 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523515357.63 ... 2018-04-12 06:49:21,687 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:21,687 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523515357.63 ... 2018-04-12 06:49:21,687 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:21,687 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523515357.63 ... 2018-04-12 06:49:21,688 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:21,688 - baro_tests.barometer - INFO - OK 2018-04-12 06:49:21,688 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-12 06:49:24,688 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-12 06:49:35,911 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:49:35,912 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523515357.634, 1523515358.634), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523515357.634, 1523515358.634), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523515357.634, 1523515358.634), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523515357.634, 1523515358.634)] 2018-04-12 06:49:35,912 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523515371.634, 1523515372.634), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523515371.634, 1523515372.634), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523515371.634, 1523515373.634), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523515372.634, 1523515373.634)] 2018-04-12 06:49:35,913 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523515357.63 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523515371.63 ... 2018-04-12 06:49:35,913 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:35,913 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523515357.63 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523515371.63 ... 2018-04-12 06:49:35,913 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:35,913 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523515357.63 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523515371.63 ... 2018-04-12 06:49:35,914 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:35,914 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523515357.63 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523515372.63 ... 2018-04-12 06:49:35,914 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:49:35,914 - baro_tests.barometer - INFO - OK 2018-04-12 06:49:35,914 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Mcelog. 2018-04-12 06:50:21,816 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-12 06:50:43,719 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node dfbaaf7a-1eec-4fc1-a438-7b1f85542d6d... 2018-04-12 06:50:43,721 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-12 06:50:43,726 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-04-12 06:50:43,729 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-04-12 06:50:55,068 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:50:55,069 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523515451.63 ... 2018-04-12 06:50:55,069 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:50:55,069 - baro_tests.barometer - INFO - OK 2018-04-12 06:50:55,070 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-12 06:50:58,073 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-12 06:51:09,549 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:51:09,550 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523515451.634, 1523515452.634)] 2018-04-12 06:51:09,550 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523515465.634, 1523515466.634)] 2018-04-12 06:51:09,550 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523515451.63 - ovs_events-br0 gauge-link_status 1523515465.63 ... 2018-04-12 06:51:09,551 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:51:09,551 - baro_tests.barometer - INFO - OK 2018-04-12 06:51:09,555 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-04-12 06:51:56,110 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-04-12 06:52:17,395 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node dfbaaf7a-1eec-4fc1-a438-7b1f85542d6d... 2018-04-12 06:52:17,395 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-12 06:52:17,395 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-04-12 06:52:17,395 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-04-12 06:52:29,661 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-12 06:52:29,661 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523515545.63 ... 2018-04-12 06:52:29,661 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:52:29,662 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523515545.63 ... 2018-04-12 06:52:29,662 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:52:29,662 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523515545.63 ... 2018-04-12 06:52:29,662 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:52:29,663 - baro_tests.barometer - INFO - OK 2018-04-12 06:52:29,663 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-12 06:52:32,666 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-12 06:52:43,339 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-12 06:52:43,339 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523515545.634, 1523515546.634), ('ovs_stats-br0.br0', 'if_errors', 1523515545.634, 1523515546.634), ('ovs_stats-br0.br0', 'if_packets', 1523515545.634, 1523515546.634)] 2018-04-12 06:52:43,340 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523515559.634, 1523515560.634), ('ovs_stats-br0.br0', 'if_errors', 1523515559.634, 1523515560.634), ('ovs_stats-br0.br0', 'if_packets', 1523515559.634, 1523515560.634)] 2018-04-12 06:52:43,340 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523515545.63 - ovs_stats-br0.br0 if_dropped 1523515559.63 ... 2018-04-12 06:52:43,340 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:52:43,341 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523515545.63 - ovs_stats-br0.br0 if_errors 1523515559.63 ... 2018-04-12 06:52:43,341 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:52:43,341 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523515545.63 - ovs_stats-br0.br0 if_packets 1523515559.63 ... 2018-04-12 06:52:43,341 - baro_tests.barometer - DEBUG - OK 2018-04-12 06:52:43,341 - baro_tests.barometer - INFO - OK 2018-04-12 06:52:43,341 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-04-12 06:52:55,647 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-04-12 06:52:55,648 - baro_tests.barometer - INFO - ====================================================================== 2018-04-12 06:52:55,650 - baro_tests.barometer - INFO - +----------------------------------+ 2018-04-12 06:52:55,652 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-04-12 06:52:55,653 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-12 06:52:55,654 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-04-12 06:52:55,656 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-12 06:52:55,656 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-04-12 06:52:55,657 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | FAIL | 2018-04-12 06:52:55,658 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-04-12 06:52:55,658 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-04-12 06:52:55,658 - baro_tests.barometer - INFO - | IN:OVS events | PASS | FAIL | 2018-04-12 06:52:55,658 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-04-12 06:52:55,658 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | IN:Mcelog | PASS | PASS | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-12 06:52:55,659 - baro_tests.barometer - INFO - ====================================================================== 2018-04-12 06:52:55,659 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-04-12 06:52:55,660 - baro_tests.barometer - ERROR - Overall Result is Fail