2018-05-05 15:55:11,814 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-05-05 15:55:11,815 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-05-05 15:55:16,831 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-05-05 15:55:16,832 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.9', 'overcloud-controller-1: 192.30.9.5', 'overcloud-controller-2: 192.30.9.6'] 2018-05-05 15:55:16,832 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.4', 'overcloud-novacompute-1: 192.30.9.10'] 2018-05-05 15:55:16,832 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-05-05 15:55:31,864 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-0... 2018-05-05 15:55:31,955 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-0... 2018-05-05 15:55:32,474 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-0. 2018-05-05 15:55:32,505 - baro_tests.barometer - INFO - Checking if mcelog is enabled on node-overcloud-novacompute-1... 2018-05-05 15:55:32,573 - baro_tests.barometer - INFO - Mcelog will be enabled on node-overcloud-novacompute-1... 2018-05-05 15:55:33,068 - baro_tests.barometer - INFO - Mcelog was installed on node-overcloud-novacompute-1. 2018-05-05 15:55:46,894 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-05-05 15:55:47,223 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-05-05 15:55:47,224 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-05-05 15:57:45,510 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-05-05 15:57:59,211 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-05-05 15:57:59,212 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-05-05 15:59:33,230 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-05-05 16:00:00,418 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:00:37,861 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-05-05 16:00:37,862 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-05-05 16:00:37,862 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-05-05 16:01:32,302 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-05-05 16:02:13,224 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:02:50,547 - baro_tests.barometer - INFO - PASS 2018-05-05 16:02:50,547 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Intel RDT. 2018-05-05 16:03:44,920 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-05-05 16:04:12,300 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:04:49,602 - baro_tests.barometer - INFO - PASS 2018-05-05 16:04:49,602 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Mcelog. 2018-05-05 16:05:44,829 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-05-05 16:06:11,826 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:06:14,533 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-05-05 16:06:17,827 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-05-05 16:06:20,973 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-05-05 16:07:15,917 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-05-05 16:07:43,255 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:07:46,007 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-05-05 16:07:48,676 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-05-05 16:07:51,290 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-05-05 16:08:46,163 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-05-05 16:09:14,071 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-05-05 16:09:33,329 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-05-05 16:09:33,330 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-05-05 16:10:28,398 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-05-05 16:10:54,460 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-05-05 16:10:56,851 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-05-05 16:10:59,366 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-05-05 16:11:01,848 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-05-05 16:11:01,849 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-05-05 16:11:01,849 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-05-05 16:11:14,377 - baro_tests.barometer - INFO - Stopping collectd service... 2018-05-05 16:11:25,139 - baro_tests.barometer - INFO - Starting collectd service... 2018-05-05 16:11:35,677 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-05-05 16:12:39,407 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-05-05 16:13:06,085 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node ffd1174e-07c3-4c55-bfbc-8f9bc65a64ae... 2018-05-05 16:13:06,085 - baro_tests.barometer - DEBUG - Interval: 10 2018-05-05 16:13:06,086 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-05-05 16:13:06,086 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-05-05 16:13:19,549 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:13:19,549 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1525536360.62 ... 2018-05-05 16:13:19,550 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:19,550 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1525536360.62 ... 2018-05-05 16:13:19,550 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:19,550 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1525536360.62 ... 2018-05-05 16:13:19,550 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:19,551 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1525536360.62 ... 2018-05-05 16:13:19,551 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:19,551 - baro_tests.barometer - INFO - OK 2018-05-05 16:13:19,551 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-05-05 16:13:31,562 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-05-05 16:13:46,598 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:13:46,599 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1525536360.621, 1525536370.621), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1525536360.621, 1525536370.621), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1525536360.621, 1525536370.621), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1525536360.621, 1525536370.621)] 2018-05-05 16:13:46,599 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1525536390.621, 1525536400.621), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1525536390.621, 1525536400.621), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1525536390.621, 1525536400.621), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1525536390.621, 1525536400.621)] 2018-05-05 16:13:46,599 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1525536360.62 - hugepages-mm-2048Kb vmpage_number-free 1525536390.62 ... 2018-05-05 16:13:46,600 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:46,600 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1525536360.62 - hugepages-mm-2048Kb vmpage_number-used 1525536390.62 ... 2018-05-05 16:13:46,600 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:46,600 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1525536360.62 - hugepages-node0-2048Kb vmpage_number-free 1525536390.62 ... 2018-05-05 16:13:46,601 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:46,601 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1525536360.62 - hugepages-node0-2048Kb vmpage_number-used 1525536390.62 ... 2018-05-05 16:13:46,601 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:13:46,601 - baro_tests.barometer - INFO - OK 2018-05-05 16:13:46,602 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-05-05 16:14:41,571 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-05-05 16:15:22,108 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node ffd1174e-07c3-4c55-bfbc-8f9bc65a64ae... 2018-05-05 16:15:22,108 - baro_tests.barometer - DEBUG - Interval: 10 2018-05-05 16:15:22,109 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-05-05 16:15:22,109 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-05-05 16:15:34,780 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:15:34,781 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1525536490.62 ... 2018-05-05 16:15:34,781 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:15:34,781 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1525536490.62 ... 2018-05-05 16:15:34,782 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:15:34,782 - baro_tests.barometer - INFO - OK 2018-05-05 16:15:34,782 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-05-05 16:15:46,794 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-05-05 16:16:00,154 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:16:00,155 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1525536490.621, 1525536500.621), ('intel_rdt-0-2', 'ipc', 1525536490.621, 1525536500.621)] 2018-05-05 16:16:00,157 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1525536520.621, 1525536530.621), ('intel_rdt-0-2', 'ipc', 1525536520.621, 1525536530.621)] 2018-05-05 16:16:00,157 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1525536490.62 - intel_rdt-0-2 bytes-llc 1525536520.62 ... 2018-05-05 16:16:00,157 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:16:00,158 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1525536490.62 - intel_rdt-0-2 ipc 1525536520.62 ... 2018-05-05 16:16:00,158 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:16:00,158 - baro_tests.barometer - INFO - OK 2018-05-05 16:16:00,158 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-05-05 16:16:55,021 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-05-05 16:17:20,980 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node ffd1174e-07c3-4c55-bfbc-8f9bc65a64ae... 2018-05-05 16:17:20,980 - baro_tests.barometer - DEBUG - Interval: 1 2018-05-05 16:17:20,981 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-05-05 16:17:20,983 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-05-05 16:17:34,375 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:17:34,375 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1525536627.67 ... 2018-05-05 16:17:34,375 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:34,375 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1525536627.67 ... 2018-05-05 16:17:34,376 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:34,376 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1525536627.67 ... 2018-05-05 16:17:34,376 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:34,377 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1525536627.67 ... 2018-05-05 16:17:34,377 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:34,377 - baro_tests.barometer - INFO - OK 2018-05-05 16:17:34,377 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-05-05 16:17:37,380 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-05-05 16:17:51,431 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:17:51,432 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1525536627.67, 1525536628.67), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1525536627.67, 1525536628.67), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1525536627.67, 1525536628.67), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1525536627.67, 1525536628.67)] 2018-05-05 16:17:51,434 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1525536644.67, 1525536645.67), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1525536644.67, 1525536645.67), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1525536644.67, 1525536645.67), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1525536644.67, 1525536645.67)] 2018-05-05 16:17:51,437 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1525536627.67 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1525536644.67 ... 2018-05-05 16:17:51,438 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:51,438 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1525536627.67 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1525536644.67 ... 2018-05-05 16:17:51,438 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:51,439 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1525536627.67 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1525536644.67 ... 2018-05-05 16:17:51,439 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:51,439 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1525536627.67 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1525536644.67 ... 2018-05-05 16:17:51,440 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:17:51,440 - baro_tests.barometer - INFO - OK 2018-05-05 16:17:51,440 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Mcelog. 2018-05-05 16:18:45,807 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-05-05 16:19:10,942 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node ffd1174e-07c3-4c55-bfbc-8f9bc65a64ae... 2018-05-05 16:19:10,942 - baro_tests.barometer - DEBUG - Interval: 1 2018-05-05 16:19:10,943 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-05-05 16:19:10,943 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-05-05 16:19:23,924 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:19:23,926 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1525536737.67 ... 2018-05-05 16:19:23,928 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:19:23,928 - baro_tests.barometer - INFO - OK 2018-05-05 16:19:23,928 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-05-05 16:19:26,931 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-05-05 16:19:40,587 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:19:40,588 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1525536737.67, 1525536738.67)] 2018-05-05 16:19:40,588 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1525536753.67, 1525536754.67)] 2018-05-05 16:19:40,588 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1525536737.67 - ovs_events-br0 gauge-link_status 1525536753.67 ... 2018-05-05 16:19:40,588 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:19:40,588 - baro_tests.barometer - INFO - OK 2018-05-05 16:19:40,589 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-05-05 16:20:33,866 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-05-05 16:21:00,219 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node ffd1174e-07c3-4c55-bfbc-8f9bc65a64ae... 2018-05-05 16:21:00,220 - baro_tests.barometer - DEBUG - Interval: 1 2018-05-05 16:21:00,220 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-05-05 16:21:00,221 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-05-05 16:21:13,647 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:21:13,647 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1525536846.67 ... 2018-05-05 16:21:13,648 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:21:13,648 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1525536846.67 ... 2018-05-05 16:21:13,649 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:21:13,649 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1525536846.67 ... 2018-05-05 16:21:13,649 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:21:13,649 - baro_tests.barometer - INFO - OK 2018-05-05 16:21:13,649 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-05-05 16:21:16,652 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-05-05 16:21:30,874 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:21:30,874 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1525536846.67, 1525536847.67), ('ovs_stats-br0.br0', 'if_errors', 1525536846.67, 1525536847.67), ('ovs_stats-br0.br0', 'if_packets', 1525536846.67, 1525536847.67)] 2018-05-05 16:21:30,875 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1525536863.67, 1525536864.67), ('ovs_stats-br0.br0', 'if_errors', 1525536863.67, 1525536865.67), ('ovs_stats-br0.br0', 'if_packets', 1525536863.67, 1525536865.67)] 2018-05-05 16:21:30,875 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1525536846.67 - ovs_stats-br0.br0 if_dropped 1525536863.67 ... 2018-05-05 16:21:30,876 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:21:30,876 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1525536846.67 - ovs_stats-br0.br0 if_errors 1525536863.67 ... 2018-05-05 16:21:30,876 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:21:30,876 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1525536846.67 - ovs_stats-br0.br0 if_packets 1525536863.67 ... 2018-05-05 16:21:30,877 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:21:30,877 - baro_tests.barometer - INFO - OK 2018-05-05 16:21:30,877 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-05-05 16:21:44,330 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-05-05 16:21:57,284 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-05-05 16:21:57,284 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-05-05 16:23:21,024 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-05-05 16:23:46,894 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:24:24,206 - baro_tests.barometer - INFO - PASS 2018-05-05 16:24:24,207 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Hugepages. 2018-05-05 16:25:17,403 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-05-05 16:25:56,899 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:26:34,315 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-05-05 16:26:34,315 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-05-05 16:26:34,315 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-05-05 16:27:28,960 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-05-05 16:27:53,843 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:28:31,533 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-05-05 16:28:31,533 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-05-05 16:28:31,534 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-05-05 16:29:24,892 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-05-05 16:29:50,130 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:29:52,784 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-05-05 16:29:55,491 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-05-05 16:29:58,181 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-05-05 16:30:51,976 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-05-05 16:31:18,218 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-05-05 16:31:55,928 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-05-05 16:31:55,929 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-05-05 16:31:55,932 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-05-05 16:32:51,708 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-05-05 16:33:18,364 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-05-05 16:33:37,518 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-05-05 16:33:37,519 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-05-05 16:34:30,559 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-05-05 16:34:56,996 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-05-05 16:34:59,377 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-05-05 16:35:01,791 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-05-05 16:35:04,233 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-05-05 16:35:04,233 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-05-05 16:35:04,233 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-05-05 16:35:17,656 - baro_tests.barometer - INFO - Stopping collectd service... 2018-05-05 16:35:28,189 - baro_tests.barometer - INFO - Starting collectd service... 2018-05-05 16:35:38,526 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-05-05 16:36:42,313 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-05-05 16:37:09,283 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node d5d7f53a-9774-46bd-a3eb-d048f43a5a4f... 2018-05-05 16:37:09,284 - baro_tests.barometer - DEBUG - Interval: 10 2018-05-05 16:37:09,285 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-05-05 16:37:09,285 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-05-05 16:37:24,011 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:37:24,011 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1525537800.65 ... 2018-05-05 16:37:24,012 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:24,012 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1525537800.65 ... 2018-05-05 16:37:24,012 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:24,012 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1525537800.65 ... 2018-05-05 16:37:24,012 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:24,012 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1525537800.65 ... 2018-05-05 16:37:24,013 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:24,013 - baro_tests.barometer - INFO - OK 2018-05-05 16:37:24,013 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-05-05 16:37:36,016 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-05-05 16:37:49,904 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:37:49,904 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1525537800.651, 1525537810.652), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1525537800.651, 1525537810.652), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1525537800.651, 1525537810.652), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1525537800.651, 1525537810.652)] 2018-05-05 16:37:49,905 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1525537830.651, 1525537840.651), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1525537830.651, 1525537840.651), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1525537830.652, 1525537840.651), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1525537830.652, 1525537840.651)] 2018-05-05 16:37:49,905 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1525537800.65 - hugepages-mm-2048Kb vmpage_number-free 1525537830.65 ... 2018-05-05 16:37:49,905 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:49,905 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1525537800.65 - hugepages-mm-2048Kb vmpage_number-used 1525537830.65 ... 2018-05-05 16:37:49,905 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:49,906 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1525537800.65 - hugepages-node0-2048Kb vmpage_number-free 1525537830.65 ... 2018-05-05 16:37:49,906 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:49,907 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1525537800.65 - hugepages-node0-2048Kb vmpage_number-used 1525537830.65 ... 2018-05-05 16:37:49,907 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:37:49,908 - baro_tests.barometer - INFO - OK 2018-05-05 16:37:49,909 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-05-05 16:38:41,494 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-05-05 16:39:20,001 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node d5d7f53a-9774-46bd-a3eb-d048f43a5a4f... 2018-05-05 16:39:20,002 - baro_tests.barometer - DEBUG - Interval: 10 2018-05-05 16:39:20,002 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-05-05 16:39:20,002 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-05-05 16:39:33,578 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:39:33,578 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1525537930.65 ... 2018-05-05 16:39:33,579 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:39:33,582 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1525537930.65 ... 2018-05-05 16:39:33,582 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:39:33,583 - baro_tests.barometer - INFO - OK 2018-05-05 16:39:33,583 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-05-05 16:39:45,594 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-05-05 16:39:58,397 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:39:58,397 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1525537930.651, 1525537940.651), ('intel_rdt-0-2', 'ipc', 1525537930.651, 1525537940.651)] 2018-05-05 16:39:58,398 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1525537950.651, 1525537960.651), ('intel_rdt-0-2', 'ipc', 1525537950.651, 1525537960.651)] 2018-05-05 16:39:58,398 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1525537930.65 - intel_rdt-0-2 bytes-llc 1525537950.65 ... 2018-05-05 16:39:58,398 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:39:58,399 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1525537930.65 - intel_rdt-0-2 ipc 1525537950.65 ... 2018-05-05 16:39:58,399 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:39:58,399 - baro_tests.barometer - INFO - OK 2018-05-05 16:39:58,399 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-05-05 16:40:51,335 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-05-05 16:41:18,310 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node d5d7f53a-9774-46bd-a3eb-d048f43a5a4f... 2018-05-05 16:41:18,310 - baro_tests.barometer - DEBUG - Interval: 1 2018-05-05 16:41:18,311 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-05-05 16:41:18,311 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-05-05 16:41:31,624 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:41:31,624 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1525538061.57 ... 2018-05-05 16:41:31,625 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:31,627 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1525538061.57 ... 2018-05-05 16:41:31,627 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:31,627 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1525538062.57 ... 2018-05-05 16:41:31,627 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:31,628 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1525538062.57 ... 2018-05-05 16:41:31,628 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:31,628 - baro_tests.barometer - INFO - OK 2018-05-05 16:41:31,628 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-05-05 16:41:34,630 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-05-05 16:41:48,295 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:41:48,295 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1525538061.57, 1525538062.57), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1525538061.57, 1525538062.57), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1525538062.57, 1525538063.571), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1525538062.57, 1525538063.571)] 2018-05-05 16:41:48,297 - baro_tests.barometer - DEBUG - [('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-corrected_memory_errors', 1525538078.57, 1525538079.57), ('mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'errors-uncorrected_memory_errors', 1525538078.57, 1525538079.57), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-corrected_memory_errors', 1525538078.57, 1525538079.57), ('mcelog-SOCKET_0_CHANNEL_any_DIMM_any', 'errors-uncorrected_memory_errors', 1525538078.57, 1525538079.57)] 2018-05-05 16:41:48,298 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1525538061.57 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-corrected_memory_errors 1525538078.57 ... 2018-05-05 16:41:48,298 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:48,298 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1525538061.57 - mcelog-SOCKET_0_CHANNEL_0_DIMM_any errors-uncorrected_memory_errors 1525538078.57 ... 2018-05-05 16:41:48,298 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:48,299 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1525538062.57 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-corrected_memory_errors 1525538078.57 ... 2018-05-05 16:41:48,299 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:48,299 - baro_tests.barometer - DEBUG - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1525538062.57 - mcelog-SOCKET_0_CHANNEL_any_DIMM_any errors-uncorrected_memory_errors 1525538078.57 ... 2018-05-05 16:41:48,299 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:41:48,299 - baro_tests.barometer - INFO - OK 2018-05-05 16:41:48,300 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Mcelog. 2018-05-05 16:42:39,390 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-05-05 16:43:05,386 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node d5d7f53a-9774-46bd-a3eb-d048f43a5a4f... 2018-05-05 16:43:05,389 - baro_tests.barometer - DEBUG - Interval: 1 2018-05-05 16:43:05,389 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-05-05 16:43:05,390 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-05-05 16:43:18,296 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:43:18,297 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1525538168.57 ... 2018-05-05 16:43:18,297 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:43:18,297 - baro_tests.barometer - INFO - OK 2018-05-05 16:43:18,299 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-05-05 16:43:21,302 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-05-05 16:43:34,927 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:43:34,930 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1525538168.57, 1525538169.57)] 2018-05-05 16:43:34,932 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1525538185.57, 1525538186.57)] 2018-05-05 16:43:34,933 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1525538168.57 - ovs_events-br0 gauge-link_status 1525538185.57 ... 2018-05-05 16:43:34,934 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:43:34,938 - baro_tests.barometer - INFO - OK 2018-05-05 16:43:34,940 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-05-05 16:44:27,674 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-05-05 16:44:53,110 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node d5d7f53a-9774-46bd-a3eb-d048f43a5a4f... 2018-05-05 16:44:53,111 - baro_tests.barometer - DEBUG - Interval: 1 2018-05-05 16:44:53,111 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-05-05 16:44:53,111 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-05-05 16:45:06,515 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-05-05 16:45:06,519 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1525538276.57 ... 2018-05-05 16:45:06,525 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:45:06,527 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1525538276.57 ... 2018-05-05 16:45:06,529 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:45:06,530 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1525538277.57 ... 2018-05-05 16:45:06,530 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:45:06,530 - baro_tests.barometer - INFO - OK 2018-05-05 16:45:06,530 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-05-05 16:45:09,534 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-05-05 16:45:23,435 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-05-05 16:45:23,436 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1525538276.57, 1525538277.57), ('ovs_stats-br0.br0', 'if_errors', 1525538276.57, 1525538277.57), ('ovs_stats-br0.br0', 'if_packets', 1525538277.57, 1525538278.57)] 2018-05-05 16:45:23,436 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1525538293.57, 1525538294.57), ('ovs_stats-br0.br0', 'if_errors', 1525538293.57, 1525538294.57), ('ovs_stats-br0.br0', 'if_packets', 1525538293.57, 1525538295.57)] 2018-05-05 16:45:23,436 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1525538276.57 - ovs_stats-br0.br0 if_dropped 1525538293.57 ... 2018-05-05 16:45:23,437 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:45:23,437 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1525538276.57 - ovs_stats-br0.br0 if_errors 1525538293.57 ... 2018-05-05 16:45:23,437 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:45:23,437 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1525538277.57 - ovs_stats-br0.br0 if_packets 1525538293.57 ... 2018-05-05 16:45:23,437 - baro_tests.barometer - DEBUG - OK 2018-05-05 16:45:23,437 - baro_tests.barometer - INFO - OK 2018-05-05 16:45:23,437 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-05-05 16:45:36,483 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-05-05 16:45:36,483 - baro_tests.barometer - INFO - ====================================================================== 2018-05-05 16:45:36,484 - baro_tests.barometer - INFO - +----------------------------------+ 2018-05-05 16:45:36,485 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-05-05 16:45:36,488 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-05-05 16:45:36,488 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-05-05 16:45:36,491 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-05-05 16:45:36,493 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-05-05 16:45:36,494 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | PASS | 2018-05-05 16:45:36,497 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | FAIL | 2018-05-05 16:45:36,499 - baro_tests.barometer - INFO - | IN:Mcelog | PASS | FAIL | 2018-05-05 16:45:36,499 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-05-05 16:45:36,500 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-05-05 16:45:36,500 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-05-05 16:45:36,501 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-05-05 16:45:36,502 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-05-05 16:45:36,502 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-05-05 16:45:36,502 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-05-05 16:45:36,502 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-05-05 16:45:36,502 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-05-05 16:45:36,503 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-05-05 16:45:36,503 - baro_tests.barometer - INFO - | IN:Mcelog | PASS | PASS | 2018-05-05 16:45:36,504 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-05-05 16:45:36,504 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-05-05 16:45:36,505 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-05-05 16:45:36,505 - baro_tests.barometer - INFO - ====================================================================== 2018-05-05 16:45:36,506 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-05-05 16:45:36,507 - baro_tests.barometer - ERROR - Overall Result is Fail