2018-04-08 14:28:21,096 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-04-08 14:28:21,096 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-04-08 14:28:25,217 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-04-08 14:28:25,217 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.6', 'overcloud-controller-1: 192.30.9.7', 'overcloud-controller-2: 192.30.9.8'] 2018-04-08 14:28:25,217 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.4', 'overcloud-novacompute-1: 192.30.9.10'] 2018-04-08 14:28:25,218 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-04-08 14:28:38,904 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-0... 2018-04-08 14:28:38,978 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-0... 2018-04-08 14:28:39,345 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-0. 2018-04-08 14:28:39,357 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-1... 2018-04-08 14:28:39,408 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-1... 2018-04-08 14:28:39,786 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-1. 2018-04-08 14:28:53,118 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-04-08 14:28:53,368 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-04-08 14:28:53,368 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-04-08 14:30:43,153 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-04-08 14:30:56,554 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-04-08 14:30:56,554 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-04-08 14:32:20,203 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-04-08 14:32:47,858 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 14:33:24,955 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-08 14:33:24,956 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-08 14:33:24,956 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-04-08 14:34:17,597 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-04-08 14:34:57,341 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 14:34:59,891 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-08 14:35:02,587 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-04-08 14:35:05,198 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-04-08 14:35:58,131 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-08 14:36:23,815 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 14:37:00,891 - baro_tests.barometer - INFO - PASS 2018-04-08 14:37:00,892 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Mcelog. 2018-04-08 14:37:54,786 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-08 14:38:20,957 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 14:38:23,647 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-04-08 14:38:26,228 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-04-08 14:38:28,802 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-04-08 14:39:22,418 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-04-08 14:39:49,794 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 14:40:26,882 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-08 14:40:26,883 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-08 14:40:26,883 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-04-08 14:41:18,880 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-08 14:41:44,233 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-04-08 14:42:03,258 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-04-08 14:42:03,259 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-04-08 14:42:55,385 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-08 14:43:20,167 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-04-08 14:43:22,456 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-08 14:43:25,311 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-04-08 14:43:27,752 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-04-08 14:43:27,753 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-04-08 14:43:27,753 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-04-08 14:43:40,229 - baro_tests.barometer - INFO - Stopping collectd service... 2018-04-08 14:43:50,735 - baro_tests.barometer - INFO - Starting collectd service... 2018-04-08 14:44:01,035 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-04-08 14:45:05,023 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-04-08 14:45:30,724 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 79a396a6-e500-4d19-ada2-9724b240a264... 2018-04-08 14:45:30,724 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-08 14:45:30,724 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-04-08 14:45:30,725 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-04-08 14:45:44,722 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 14:45:44,723 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523198346.83 ... 2018-04-08 14:45:44,723 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:45:44,723 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523198346.83 ... 2018-04-08 14:45:44,724 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:45:44,724 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523198346.83 ... 2018-04-08 14:45:44,724 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:45:44,724 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523198346.83 ... 2018-04-08 14:45:44,725 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:45:44,725 - baro_tests.barometer - INFO - OK 2018-04-08 14:45:44,725 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-08 14:45:56,736 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-08 14:46:09,424 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 14:46:09,425 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523198346.83, 1523198356.83), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523198346.83, 1523198356.83), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523198346.83, 1523198356.83), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523198346.83, 1523198356.83)] 2018-04-08 14:46:09,425 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523198366.831, 1523198376.83), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523198366.831, 1523198376.83), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523198366.831, 1523198376.831), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523198366.831, 1523198376.831)] 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523198346.83 - hugepages-mm-2048Kb vmpage_number-free 1523198366.83 ... 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523198346.83 - hugepages-mm-2048Kb vmpage_number-used 1523198366.83 ... 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523198346.83 - hugepages-node0-2048Kb vmpage_number-free 1523198366.83 ... 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523198346.83 - hugepages-node0-2048Kb vmpage_number-used 1523198366.83 ... 2018-04-08 14:46:09,426 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:46:09,427 - baro_tests.barometer - INFO - OK 2018-04-08 14:46:09,427 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-04-08 14:46:59,488 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-04-08 14:47:36,643 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 79a396a6-e500-4d19-ada2-9724b240a264... 2018-04-08 14:47:36,643 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-08 14:47:36,643 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-04-08 14:47:36,644 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-04-08 14:47:49,335 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 14:47:49,335 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523198466.83 ... 2018-04-08 14:47:49,336 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:47:49,336 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523198466.83 ... 2018-04-08 14:47:49,336 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:47:49,336 - baro_tests.barometer - INFO - OK 2018-04-08 14:47:49,336 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-08 14:48:01,348 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-08 14:48:14,623 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 14:48:14,623 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523198466.83, 1523198476.83), ('intel_rdt-0-2', 'ipc', 1523198466.83, 1523198476.83)] 2018-04-08 14:48:14,623 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523198496.83, 1523198506.83), ('intel_rdt-0-2', 'ipc', 1523198496.83, 1523198506.83)] 2018-04-08 14:48:14,623 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523198466.83 - intel_rdt-0-2 bytes-llc 1523198496.83 ... 2018-04-08 14:48:14,624 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:48:14,624 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523198466.83 - intel_rdt-0-2 ipc 1523198496.83 ... 2018-04-08 14:48:14,624 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:48:14,624 - baro_tests.barometer - INFO - OK 2018-04-08 14:48:14,624 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-04-08 14:49:07,085 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-04-08 14:49:32,373 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 79a396a6-e500-4d19-ada2-9724b240a264... 2018-04-08 14:49:32,374 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-08 14:49:32,374 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-04-08 14:49:32,374 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-04-08 14:49:45,539 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 14:49:45,540 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523198599.83 ... 2018-04-08 14:49:45,541 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:49:45,541 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523198599.83 ... 2018-04-08 14:49:45,542 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:49:45,545 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523198599.83 ... 2018-04-08 14:49:45,547 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:49:45,548 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523198599.83 ... 2018-04-08 14:49:45,550 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:49:45,551 - baro_tests.barometer - INFO - OK 2018-04-08 14:49:45,551 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-08 14:49:48,554 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-08 14:50:02,784 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 14:50:02,784 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523198599.831, 1523198600.831), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523198599.831, 1523198600.831), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523198599.831, 1523198600.831), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523198599.831, 1523198600.831)] 2018-04-08 14:50:02,785 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523198616.831, 1523198617.831), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523198616.831, 1523198617.831), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523198616.831, 1523198617.831), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523198616.831, 1523198617.831)] 2018-04-08 14:50:02,785 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523198599.83 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523198616.83 ... 2018-04-08 14:50:02,785 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:50:02,786 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523198599.83 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523198616.83 ... 2018-04-08 14:50:02,786 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:50:02,786 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523198599.83 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523198616.83 ... 2018-04-08 14:50:02,786 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:50:02,787 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523198599.83 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523198616.83 ... 2018-04-08 14:50:02,787 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:50:02,787 - baro_tests.barometer - INFO - OK 2018-04-08 14:50:02,787 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Mcelog. 2018-04-08 14:50:56,388 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-04-08 14:51:21,877 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 79a396a6-e500-4d19-ada2-9724b240a264... 2018-04-08 14:51:21,877 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-08 14:51:21,877 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-04-08 14:51:21,878 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-04-08 14:51:35,017 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 14:51:35,018 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523198708.83 ... 2018-04-08 14:51:35,018 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:51:35,019 - baro_tests.barometer - INFO - OK 2018-04-08 14:51:35,019 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-08 14:51:38,020 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-08 14:51:50,871 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 14:51:50,872 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523198708.831, 1523198709.831)] 2018-04-08 14:51:50,874 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523198724.831, 1523198725.831)] 2018-04-08 14:51:50,875 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523198708.83 - ovs_events-br0 gauge-link_status 1523198724.83 ... 2018-04-08 14:51:50,876 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:51:50,876 - baro_tests.barometer - INFO - OK 2018-04-08 14:51:50,878 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-04-08 14:52:40,748 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-04-08 14:53:05,182 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 79a396a6-e500-4d19-ada2-9724b240a264... 2018-04-08 14:53:05,182 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-08 14:53:05,184 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-04-08 14:53:05,184 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-04-08 14:53:17,429 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 14:53:17,429 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523198811.83 ... 2018-04-08 14:53:17,429 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:53:17,430 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523198811.83 ... 2018-04-08 14:53:17,431 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:53:17,433 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523198811.83 ... 2018-04-08 14:53:17,434 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:53:17,434 - baro_tests.barometer - INFO - OK 2018-04-08 14:53:17,434 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-08 14:53:20,435 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-04-08 14:53:33,010 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 14:53:33,010 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523198811.831, 1523198812.831), ('ovs_stats-br0.br0', 'if_errors', 1523198811.831, 1523198812.831), ('ovs_stats-br0.br0', 'if_packets', 1523198811.831, 1523198812.831)] 2018-04-08 14:53:33,011 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523198826.831, 1523198827.831), ('ovs_stats-br0.br0', 'if_errors', 1523198826.831, 1523198827.831), ('ovs_stats-br0.br0', 'if_packets', 1523198826.831, 1523198827.831)] 2018-04-08 14:53:33,011 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523198811.83 - ovs_stats-br0.br0 if_dropped 1523198826.83 ... 2018-04-08 14:53:33,011 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:53:33,012 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523198811.83 - ovs_stats-br0.br0 if_errors 1523198826.83 ... 2018-04-08 14:53:33,012 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:53:33,012 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523198811.83 - ovs_stats-br0.br0 if_packets 1523198826.83 ... 2018-04-08 14:53:33,012 - baro_tests.barometer - DEBUG - OK 2018-04-08 14:53:33,012 - baro_tests.barometer - INFO - OK 2018-04-08 14:53:33,012 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-04-08 14:53:45,301 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-04-08 14:53:57,836 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-04-08 14:53:57,836 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-04-08 14:55:14,512 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-04-08 14:55:40,065 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 14:56:17,258 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-04-08 14:56:17,258 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-08 14:56:17,258 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-04-08 14:57:08,150 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-04-08 14:57:45,840 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 14:58:23,172 - baro_tests.barometer - INFO - PASS 2018-04-08 14:58:23,173 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Intel RDT. 2018-04-08 14:59:13,383 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-08 14:59:38,177 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 15:00:15,805 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-08 15:00:15,806 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-08 15:00:15,806 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-04-08 15:01:09,634 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-08 15:01:35,344 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 15:02:12,683 - baro_tests.barometer - INFO - PASS 2018-04-08 15:02:12,683 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS events. 2018-04-08 15:03:04,800 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-04-08 15:03:29,764 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-04-08 15:04:07,372 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-04-08 15:04:07,373 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-04-08 15:04:07,373 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-04-08 15:04:57,882 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-08 15:05:23,375 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-04-08 15:05:42,338 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-04-08 15:05:42,341 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-04-08 15:06:33,743 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-08 15:07:00,336 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-04-08 15:07:02,687 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-04-08 15:07:05,067 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-04-08 15:07:07,434 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-04-08 15:07:07,436 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-04-08 15:07:07,436 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-04-08 15:07:20,211 - baro_tests.barometer - INFO - Stopping collectd service... 2018-04-08 15:07:32,253 - baro_tests.barometer - INFO - Starting collectd service... 2018-04-08 15:07:42,994 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-04-08 15:08:42,559 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-04-08 15:09:08,416 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node e18c8287-ca81-48c9-80b6-0a958f704b67... 2018-04-08 15:09:08,416 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-08 15:09:08,416 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-04-08 15:09:08,417 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-04-08 15:09:21,586 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 15:09:21,587 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523199745.57 ... 2018-04-08 15:09:21,587 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:21,587 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523199745.57 ... 2018-04-08 15:09:21,588 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:21,588 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523199745.57 ... 2018-04-08 15:09:21,588 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:21,588 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523199745.57 ... 2018-04-08 15:09:21,588 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:21,588 - baro_tests.barometer - INFO - OK 2018-04-08 15:09:21,589 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-08 15:09:33,601 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-08 15:09:47,435 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 15:09:47,435 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523199745.566, 1523199755.566), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523199745.566, 1523199755.566), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523199745.567, 1523199755.567), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523199745.567, 1523199755.567)] 2018-04-08 15:09:47,436 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1523199775.566, 1523199785.566), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1523199775.566, 1523199785.566), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1523199775.566, 1523199785.567), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1523199775.566, 1523199785.567)] 2018-04-08 15:09:47,436 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1523199745.57 - hugepages-mm-2048Kb vmpage_number-free 1523199775.57 ... 2018-04-08 15:09:47,436 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:47,436 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1523199745.57 - hugepages-mm-2048Kb vmpage_number-used 1523199775.57 ... 2018-04-08 15:09:47,437 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:47,437 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1523199745.57 - hugepages-node0-2048Kb vmpage_number-free 1523199775.57 ... 2018-04-08 15:09:47,437 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:47,437 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1523199745.57 - hugepages-node0-2048Kb vmpage_number-used 1523199775.57 ... 2018-04-08 15:09:47,437 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:09:47,437 - baro_tests.barometer - INFO - OK 2018-04-08 15:09:47,438 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-04-08 15:10:35,448 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-04-08 15:11:13,137 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node e18c8287-ca81-48c9-80b6-0a958f704b67... 2018-04-08 15:11:13,138 - baro_tests.barometer - DEBUG - Interval: 10 2018-04-08 15:11:13,139 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-04-08 15:11:13,140 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-04-08 15:11:25,623 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 15:11:25,623 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523199875.57 ... 2018-04-08 15:11:25,628 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:11:25,629 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523199875.57 ... 2018-04-08 15:11:25,629 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:11:25,630 - baro_tests.barometer - INFO - OK 2018-04-08 15:11:25,630 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-04-08 15:11:37,638 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-08 15:11:49,933 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 15:11:49,934 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523199875.566, 1523199885.566), ('intel_rdt-0-2', 'ipc', 1523199875.566, 1523199885.566)] 2018-04-08 15:11:49,934 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1523199895.566, 1523199905.566), ('intel_rdt-0-2', 'ipc', 1523199895.566, 1523199905.566)] 2018-04-08 15:11:49,934 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1523199875.57 - intel_rdt-0-2 bytes-llc 1523199895.57 ... 2018-04-08 15:11:49,935 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:11:49,935 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1523199875.57 - intel_rdt-0-2 ipc 1523199895.57 ... 2018-04-08 15:11:49,935 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:11:49,935 - baro_tests.barometer - INFO - OK 2018-04-08 15:11:49,935 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-04-08 15:12:39,666 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-04-08 15:13:03,646 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node e18c8287-ca81-48c9-80b6-0a958f704b67... 2018-04-08 15:13:03,647 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-08 15:13:03,647 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-04-08 15:13:03,647 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-04-08 15:13:15,847 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 15:13:15,848 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523199996.57 ... 2018-04-08 15:13:15,848 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:15,848 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523199996.57 ... 2018-04-08 15:13:15,848 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:15,849 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523199996.57 ... 2018-04-08 15:13:15,849 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:15,849 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523199996.57 ... 2018-04-08 15:13:15,849 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:15,849 - baro_tests.barometer - INFO - OK 2018-04-08 15:13:15,849 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-08 15:13:18,852 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-08 15:13:32,264 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 15:13:32,265 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523199996.567, 1523199997.568), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523199996.567, 1523199997.568), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523199996.567, 1523199997.568), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523199996.567, 1523199997.568)] 2018-04-08 15:13:32,265 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1523200012.567, 1523200014.567), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1523200013.567, 1523200014.567), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1523200013.567, 1523200014.567), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1523200013.567, 1523200014.567)] 2018-04-08 15:13:32,266 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523199996.57 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1523200012.57 ... 2018-04-08 15:13:32,266 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:32,266 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523199996.57 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1523200013.57 ... 2018-04-08 15:13:32,267 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:32,267 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523199996.57 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1523200013.57 ... 2018-04-08 15:13:32,267 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:32,267 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523199996.57 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1523200013.57 ... 2018-04-08 15:13:32,267 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:13:32,267 - baro_tests.barometer - INFO - OK 2018-04-08 15:13:32,268 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Mcelog. 2018-04-08 15:14:22,381 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-04-08 15:14:47,183 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node e18c8287-ca81-48c9-80b6-0a958f704b67... 2018-04-08 15:14:47,183 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-08 15:14:47,184 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-04-08 15:14:47,184 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-04-08 15:14:59,640 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 15:14:59,640 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523200100.57 ... 2018-04-08 15:14:59,640 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:14:59,641 - baro_tests.barometer - INFO - OK 2018-04-08 15:14:59,641 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-08 15:15:02,644 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-08 15:15:15,578 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 15:15:15,578 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523200100.568, 1523200101.568)] 2018-04-08 15:15:15,579 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1523200116.568, 1523200117.568)] 2018-04-08 15:15:15,579 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1523200100.57 - ovs_events-br0 gauge-link_status 1523200116.57 ... 2018-04-08 15:15:15,579 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:15:15,579 - baro_tests.barometer - INFO - OK 2018-04-08 15:15:15,580 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-04-08 15:16:05,664 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-04-08 15:16:31,344 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node e18c8287-ca81-48c9-80b6-0a958f704b67... 2018-04-08 15:16:31,345 - baro_tests.barometer - DEBUG - Interval: 1 2018-04-08 15:16:31,345 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-04-08 15:16:31,345 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-04-08 15:16:43,588 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-04-08 15:16:43,588 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523200204.57 ... 2018-04-08 15:16:43,588 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:16:43,588 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523200204.57 ... 2018-04-08 15:16:43,589 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:16:43,589 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523200204.57 ... 2018-04-08 15:16:43,589 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:16:43,589 - baro_tests.barometer - INFO - OK 2018-04-08 15:16:43,589 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-04-08 15:16:46,593 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-04-08 15:16:58,749 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-04-08 15:16:58,750 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523200204.567, 1523200205.567), ('ovs_stats-br0.br0', 'if_errors', 1523200204.567, 1523200205.567), ('ovs_stats-br0.br0', 'if_packets', 1523200204.567, 1523200205.567)] 2018-04-08 15:16:58,750 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1523200219.567, 1523200220.567), ('ovs_stats-br0.br0', 'if_errors', 1523200219.567, 1523200220.567), ('ovs_stats-br0.br0', 'if_packets', 1523200219.567, 1523200220.568)] 2018-04-08 15:16:58,750 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1523200204.57 - ovs_stats-br0.br0 if_dropped 1523200219.57 ... 2018-04-08 15:16:58,750 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:16:58,751 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1523200204.57 - ovs_stats-br0.br0 if_errors 1523200219.57 ... 2018-04-08 15:16:58,751 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:16:58,751 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1523200204.57 - ovs_stats-br0.br0 if_packets 1523200219.57 ... 2018-04-08 15:16:58,751 - baro_tests.barometer - DEBUG - OK 2018-04-08 15:16:58,751 - baro_tests.barometer - INFO - OK 2018-04-08 15:16:58,752 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-04-08 15:17:11,041 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-04-08 15:17:11,043 - baro_tests.barometer - INFO - ====================================================================== 2018-04-08 15:17:11,043 - baro_tests.barometer - INFO - +----------------------------------+ 2018-04-08 15:17:11,044 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-04-08 15:17:11,045 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-08 15:17:11,046 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-04-08 15:17:11,046 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-08 15:17:11,046 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-04-08 15:17:11,047 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | FAIL | 2018-04-08 15:17:11,048 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | PASS | 2018-04-08 15:17:11,049 - baro_tests.barometer - INFO - | IN:Mcelog | PASS | FAIL | 2018-04-08 15:17:11,049 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | PASS | 2018-04-08 15:17:11,050 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-04-08 15:17:11,051 - baro_tests.barometer - INFO - | IN:Mcelog | PASS | PASS | 2018-04-08 15:17:11,052 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-04-08 15:17:11,052 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-04-08 15:17:11,052 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-04-08 15:17:11,052 - baro_tests.barometer - INFO - ====================================================================== 2018-04-08 15:17:11,052 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-04-08 15:17:11,052 - baro_tests.barometer - ERROR - Overall Result is Fail