May 18 16:28:24 ctl01 rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="1309" x-info="http://www.rsyslog.com"] start
May 18 16:28:24 ctl01 loadkeys[460]: Loading /etc/console-setup/cached.kmap.gz
May 18 16:28:24 ctl01 systemd-modules-load[463]: Inserted module 'iscsi_tcp'
May 18 16:28:24 ctl01 systemd[1]: Started LVM2 metadata daemon.
May 18 16:28:24 ctl01 systemd[1]: Starting udev Coldplug all Devices...
May 18 16:28:24 ctl01 systemd[1]: Starting Load/Save Random Seed...
May 18 16:28:24 ctl01 systemd[1]: Starting Flush Journal to Persistent Storage...
May 18 16:28:24 ctl01 systemd-modules-load[463]: Inserted module 'ib_iser'
May 18 16:28:24 ctl01 systemd[1]: Starting Create Static Device Nodes in /dev...
May 18 16:28:24 ctl01 systemd[1]: Started Load Kernel Modules.
May 18 16:28:24 ctl01 systemd[1]: Started Load/Save Random Seed.
May 18 16:28:24 ctl01 systemd[1]: Mounting Configuration File System...
May 18 16:28:24 ctl01 systemd[1]: Mounting FUSE Control File System...
May 18 16:28:24 ctl01 rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
May 18 16:28:24 ctl01 systemd[1]: Starting Apply Kernel Variables...
May 18 16:28:24 ctl01 rsyslogd: rsyslogd's groupid changed to 108
May 18 16:28:24 ctl01 rsyslogd: rsyslogd's userid changed to 104
May 18 16:28:24 ctl01 systemd[1]: Mounted Configuration File System.
May 18 16:28:24 ctl01 systemd[1]: Mounted FUSE Control File System.
May 18 16:28:24 ctl01 systemd[1]: Started Create Static Device Nodes in /dev.
May 18 16:28:24 ctl01 systemd[1]: Started Apply Kernel Variables.
May 18 16:28:24 ctl01 systemd[1]: Started Flush Journal to Persistent Storage.
May 18 16:28:24 ctl01 systemd[1]: Starting udev Kernel Device Manager...
May 18 16:28:24 ctl01 rsyslogd-2039: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
May 18 16:28:24 ctl01 rsyslogd-2007: action 'action 11' suspended, next retry is Sat May 18 16:28:54 2019 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
May 18 16:28:24 ctl01 kernel: [    0.000000] Linux version 4.15.0-50-generic (buildd@lgw01-amd64-029) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) #54~16.04.1-Ubuntu SMP Wed May 8 15:55:19 UTC 2019 (Ubuntu 4.15.0-50.54~16.04.1-generic 4.15.18)
May 18 16:28:24 ctl01 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-50-generic root=UUID=1bbe14b0-e1b4-4636-83ed-ba0fa2a5acca ro console=tty1 console=ttyS0
May 18 16:28:24 ctl01 kernel: [    0.000000] KERNEL supported cpus:
May 18 16:28:24 ctl01 kernel: [    0.000000]   Intel GenuineIntel
May 18 16:28:24 ctl01 kernel: [    0.000000]   AMD AuthenticAMD
May 18 16:28:24 ctl01 kernel: [    0.000000]   Centaur CentaurHauls
May 18 16:28:24 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
May 18 16:28:24 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
May 18 16:28:24 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
May 18 16:28:24 ctl01 kernel: [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
May 18 16:28:24 ctl01 kernel: [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
May 18 16:28:24 ctl01 kernel: [    0.000000] e820: BIOS-provided physical RAM map:
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
May 18 16:28:24 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000003bfffffff] usable
May 18 16:28:24 ctl01 kernel: [    0.000000] NX (Execute Disable) protection: active
May 18 16:28:24 ctl01 kernel: [    0.000000] SMBIOS 2.8 present.
May 18 16:28:24 ctl01 kernel: [    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
May 18 16:28:24 ctl01 kernel: [    0.000000] Hypervisor detected: KVM
May 18 16:28:24 ctl01 kernel: [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
May 18 16:28:24 ctl01 kernel: [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
May 18 16:28:24 ctl01 kernel: [    0.000000] e820: last_pfn = 0x3c0000 max_arch_pfn = 0x400000000
May 18 16:28:24 ctl01 kernel: [    0.000000] MTRR default type: write-back
May 18 16:28:24 ctl01 kernel: [    0.000000] MTRR fixed ranges enabled:
May 18 16:28:24 ctl01 kernel: [    0.000000]   00000-9FFFF write-back
May 18 16:28:24 ctl01 kernel: [    0.000000]   A0000-BFFFF uncachable
May 18 16:28:24 ctl01 kernel: [    0.000000]   C0000-FFFFF write-protect
May 18 16:28:24 ctl01 kernel: [    0.000000] MTRR variable ranges enabled:
May 18 16:28:24 ctl01 systemd[1]: Started udev Coldplug all Devices.
May 18 16:28:24 ctl01 kernel: [    0.000000]   0 base 00C0000000 mask FFC0000000 uncachable
May 18 16:28:24 ctl01 kernel: [    0.000000]   1 disabled
May 18 16:28:24 ctl01 kernel: [    0.000000]   2 disabled
May 18 16:28:24 ctl01 kernel: [    0.000000]   3 disabled
May 18 16:28:24 ctl01 kernel: [    0.000000]   4 disabled
May 18 16:28:24 ctl01 kernel: [    0.000000]   5 disabled
May 18 16:28:24 ctl01 systemd[1]: Started udev Kernel Device Manager.
May 18 16:28:24 ctl01 systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
May 18 16:28:24 ctl01 systemd[1]: Found device /dev/ttyS0.
May 18 16:28:24 ctl01 systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
May 18 16:28:24 ctl01 systemd[1]: Reached target Local File Systems (Pre).
May 18 16:28:24 ctl01 systemd[1]: Reached target Local File Systems.
May 18 16:28:24 ctl01 systemd[1]: Starting Commit a transient machine-id on disk...
May 18 16:28:24 ctl01 systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
May 18 16:28:24 ctl01 systemd[1]: Starting Create Volatile Files and Directories...
May 18 16:28:24 ctl01 systemd[1]: Starting Initial cloud-init job (pre-networking)...
May 18 16:28:24 ctl01 systemd-tmpfiles[559]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
May 18 16:28:24 ctl01 systemd[1]: Starting Set console font and keymap...
May 18 16:28:24 ctl01 systemd[1]: Starting LSB: AppArmor initialization...
May 18 16:28:24 ctl01 systemd[1]: Started Tell Plymouth To Write Out Runtime Data.
May 18 16:28:24 ctl01 systemd[1]: Started Create Volatile Files and Directories.
May 18 16:28:24 ctl01 systemd[1]: Starting Network Time Synchronization...
May 18 16:28:24 ctl01 systemd[1]: Starting Update UTMP about System Boot/Shutdown...
May 18 16:28:24 ctl01 systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
May 18 16:28:24 ctl01 systemd[1]: Started Commit a transient machine-id on disk.
May 18 16:28:24 ctl01 systemd[1]: Started Network Time Synchronization.
May 18 16:28:24 ctl01 systemd[1]: Reached target System Time Synchronized.
May 18 16:28:24 ctl01 systemd[1]: Started Update UTMP about System Boot/Shutdown.
May 18 16:28:24 ctl01 apparmor[580]:  * Starting AppArmor profiles
May 18 16:28:24 ctl01 apparmor[580]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
May 18 16:28:24 ctl01 apparmor[580]:    ...done.
May 18 16:28:24 ctl01 systemd[1]: Started LSB: AppArmor initialization.
May 18 16:28:24 ctl01 systemd[1]: Started Set console font and keymap.
May 18 16:28:24 ctl01 systemd[1]: Created slice system-getty.slice.
May 18 16:28:24 ctl01 cloud-init[562]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'init-local' at Sat, 18 May 2019 16:28:15 +0000. Up 5.50 seconds.
May 18 16:28:24 ctl01 systemd[1]: Started Initial cloud-init job (pre-networking).
May 18 16:28:24 ctl01 systemd[1]: Reached target Network (Pre).
May 18 16:28:24 ctl01 systemd[1]: Starting Raise network interfaces...
May 18 16:28:24 ctl01 dhclient[956]: Internet Systems Consortium DHCP Client 4.3.3
May 18 16:28:24 ctl01 ifup[865]: Internet Systems Consortium DHCP Client 4.3.3
May 18 16:28:24 ctl01 dhclient[956]: Copyright 2004-2015 Internet Systems Consortium.
May 18 16:28:24 ctl01 ifup[865]: Copyright 2004-2015 Internet Systems Consortium.
May 18 16:28:24 ctl01 dhclient[956]: All rights reserved.
May 18 16:28:24 ctl01 ifup[865]: All rights reserved.
May 18 16:28:24 ctl01 dhclient[956]: For info, please visit https://www.isc.org/software/dhcp/
May 18 16:28:24 ctl01 ifup[865]: For info, please visit https://www.isc.org/software/dhcp/
May 18 16:28:24 ctl01 dhclient[956]: 
May 18 16:28:24 ctl01 dhclient[956]: Listening on LPF/ens3/52:54:00:9d:ad:df
May 18 16:28:24 ctl01 ifup[865]: Listening on LPF/ens3/52:54:00:9d:ad:df
May 18 16:28:24 ctl01 dhclient[956]: Sending on   LPF/ens3/52:54:00:9d:ad:df
May 18 16:28:24 ctl01 ifup[865]: Sending on   LPF/ens3/52:54:00:9d:ad:df
May 18 16:28:24 ctl01 dhclient[956]: Sending on   Socket/fallback
May 18 16:28:24 ctl01 ifup[865]: Sending on   Socket/fallback
May 18 16:28:24 ctl01 dhclient[956]: DHCPDISCOVER on ens3 to 255.255.255.255 port 67 interval 3 (xid=0x4e53c940)
May 18 16:28:24 ctl01 ifup[865]: DHCPDISCOVER on ens3 to 255.255.255.255 port 67 interval 3 (xid=0x4e53c940)
May 18 16:28:24 ctl01 dhclient[956]: DHCPDISCOVER on ens3 to 255.255.255.255 port 67 interval 6 (xid=0x4e53c940)
May 18 16:28:24 ctl01 ifup[865]: DHCPDISCOVER on ens3 to 255.255.255.255 port 67 interval 6 (xid=0x4e53c940)
May 18 16:28:24 ctl01 dhclient[956]: DHCPREQUEST of 192.168.11.149 on ens3 to 255.255.255.255 port 67 (xid=0x40c9534e)
May 18 16:28:24 ctl01 ifup[865]: DHCPREQUEST of 192.168.11.149 on ens3 to 255.255.255.255 port 67 (xid=0x40c9534e)
May 18 16:28:24 ctl01 dhclient[956]: DHCPOFFER of 192.168.11.149 from 192.168.11.3
May 18 16:28:24 ctl01 ifup[865]: DHCPOFFER of 192.168.11.149 from 192.168.11.3
May 18 16:28:24 ctl01 dhclient[956]: DHCPACK of 192.168.11.149 from 192.168.11.3
May 18 16:28:24 ctl01 kernel: [    0.000000]   6 disabled
May 18 16:28:24 ctl01 kernel: [    0.000000]   7 disabled
May 18 16:28:24 ctl01 kernel: [    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
May 18 16:28:24 ctl01 kernel: [    0.000000] e820: last_pfn = 0xbffdf max_arch_pfn = 0x400000000
May 18 16:28:24 ctl01 kernel: [    0.000000] found SMP MP-table at [mem 0x000f6560-0x000f656f] mapped at [        (ptrval)]
May 18 16:28:24 ctl01 kernel: [    0.000000] Scanning 1 areas for low memory corruption
May 18 16:28:24 ctl01 kernel: [    0.000000] Base memory trampoline at [        (ptrval)] 99000 size 24576
May 18 16:28:24 ctl01 kernel: [    0.000000] Using GB pages for direct mapping
May 18 16:28:24 ctl01 kernel: [    0.000000] BRK [0x21653f000, 0x21653ffff] PGTABLE
May 18 16:28:24 ctl01 kernel: [    0.000000] BRK [0x216540000, 0x216540fff] PGTABLE
May 18 16:28:24 ctl01 kernel: [    0.000000] BRK [0x216541000, 0x216541fff] PGTABLE
May 18 16:28:24 ctl01 kernel: [    0.000000] BRK [0x216542000, 0x216542fff] PGTABLE
May 18 16:28:24 ctl01 kernel: [    0.000000] BRK [0x216543000, 0x216543fff] PGTABLE
May 18 16:28:24 ctl01 kernel: [    0.000000] BRK [0x216544000, 0x216544fff] PGTABLE
May 18 16:28:24 ctl01 kernel: [    0.000000] RAMDISK: [mem 0x3177e000-0x34bb6fff]
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: Early table checksum verification disabled
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: RSDP 0x00000000000F6510 000014 (v00 BOCHS )
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: RSDT 0x00000000BFFE14CC 000030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: FACP 0x00000000BFFE0854 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: DSDT 0x00000000BFFDFC80 000BD4 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: FACS 0x00000000BFFDFC40 000040
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: SSDT 0x00000000BFFE08C8 000B54 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: APIC 0x00000000BFFE141C 0000B0 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
May 18 16:28:24 ctl01 kernel: [    0.000000] No NUMA configuration found
May 18 16:28:24 ctl01 kernel: [    0.000000] Faking a node at [mem 0x0000000000000000-0x00000003bfffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] NODE_DATA(0) allocated [mem 0x3bffd5000-0x3bfffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] kvm-clock: cpu 0, msr 3:bff54001, primary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
May 18 16:28:24 ctl01 kernel: [    0.000000] kvm-clock: using sched offset of 1553257833 cycles
May 18 16:28:24 ctl01 kernel: [    0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
May 18 16:28:24 ctl01 kernel: [    0.000000] Zone ranges:
May 18 16:28:24 ctl01 kernel: [    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000]   Normal   [mem 0x0000000100000000-0x00000003bfffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000]   Device   empty
May 18 16:28:24 ctl01 kernel: [    0.000000] Movable zone start for each node
May 18 16:28:24 ctl01 kernel: [    0.000000] Early memory node ranges
May 18 16:28:24 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
May 18 16:28:24 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000100000-0x00000000bffdefff]
May 18 16:28:24 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000100000000-0x00000003bfffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] Reserved but unavailable: 131 pages
May 18 16:28:24 ctl01 kernel: [    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000003bfffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] On node 0 totalpages: 3669885
May 18 16:28:24 ctl01 kernel: [    0.000000]   DMA zone: 64 pages used for memmap
May 18 16:28:24 ctl01 kernel: [    0.000000]   DMA zone: 21 pages reserved
May 18 16:28:24 ctl01 kernel: [    0.000000]   DMA zone: 3998 pages, LIFO batch:0
May 18 16:28:24 ctl01 kernel: [    0.000000]   DMA32 zone: 12224 pages used for memmap
May 18 16:28:24 ctl01 kernel: [    0.000000]   DMA32 zone: 782303 pages, LIFO batch:31
May 18 16:28:24 ctl01 kernel: [    0.000000]   Normal zone: 45056 pages used for memmap
May 18 16:28:24 ctl01 kernel: [    0.000000]   Normal zone: 2883584 pages, LIFO batch:31
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: PM-Timer IO Port: 0x608
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
May 18 16:28:24 ctl01 kernel: [    0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: IRQ0 used by override.
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: IRQ5 used by override.
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: IRQ9 used by override.
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: IRQ10 used by override.
May 18 16:28:24 ctl01 kernel: [    0.000000] ACPI: IRQ11 used by override.
May 18 16:28:24 ctl01 kernel: [    0.000000] Using ACPI (MADT) for SMP configuration information
May 18 16:28:24 ctl01 kernel: [    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xbffdf000-0xbfffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
May 18 16:28:24 ctl01 kernel: [    0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices
May 18 16:28:24 ctl01 kernel: [    0.000000] Booting paravirtualized kernel on KVM
May 18 16:28:24 ctl01 kernel: [    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
May 18 16:28:24 ctl01 kernel: [    0.000000] random: get_random_bytes called from start_kernel+0x99/0x51b with crng_init=0
May 18 16:28:24 ctl01 kernel: [    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
May 18 16:28:24 ctl01 kernel: [    0.000000] percpu: Embedded 46 pages/cpu @        (ptrval) s151552 r8192 d28672 u262144
May 18 16:28:24 ctl01 kernel: [    0.000000] pcpu-alloc: s151552 r8192 d28672 u262144 alloc=1*2097152
May 18 16:28:24 ctl01 kernel: [    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
May 18 16:28:24 ctl01 kernel: [    0.000000] KVM setup async PF for cpu 0
May 18 16:28:24 ctl01 kernel: [    0.000000] kvm-stealtime: cpu 0, msr 3bfc24040
May 18 16:28:24 ctl01 kernel: [    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
May 18 16:28:24 ctl01 kernel: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 3612520
May 18 16:28:24 ctl01 kernel: [    0.000000] Policy zone: Normal
May 18 16:28:24 ctl01 kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-50-generic root=UUID=1bbe14b0-e1b4-4636-83ed-ba0fa2a5acca ro console=tty1 console=ttyS0
May 18 16:28:24 ctl01 kernel: [    0.000000] Calgary: detecting Calgary via BIOS EBDA area
May 18 16:28:24 ctl01 kernel: [    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
May 18 16:28:24 ctl01 kernel: [    0.000000] Memory: 14300480K/14679540K available (12300K kernel code, 2473K rwdata, 4288K rodata, 2408K init, 2416K bss, 379060K reserved, 0K cma-reserved)
May 18 16:28:24 ctl01 kernel: [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
May 18 16:28:24 ctl01 kernel: [    0.000000] Kernel/User page tables isolation: enabled
May 18 16:28:24 ctl01 kernel: [    0.000000] ftrace: allocating 39241 entries in 154 pages
May 18 16:28:24 ctl01 kernel: [    0.004000] Hierarchical RCU implementation.
May 18 16:28:24 ctl01 kernel: [    0.004000] 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=8.
May 18 16:28:24 ctl01 kernel: [    0.004000] 	Tasks RCU enabled.
May 18 16:28:24 ctl01 kernel: [    0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
May 18 16:28:24 ctl01 kernel: [    0.004000] NR_IRQS: 524544, nr_irqs: 488, preallocated irqs: 16
May 18 16:28:24 ctl01 kernel: [    0.004000] Console: colour VGA+ 80x25
May 18 16:28:24 ctl01 kernel: [    0.004000] console [tty1] enabled
May 18 16:28:24 ctl01 kernel: [    0.004000] console [ttyS0] enabled
May 18 16:28:24 ctl01 kernel: [    0.004000] ACPI: Core revision 20170831
May 18 16:28:24 ctl01 kernel: [    0.004000] ACPI: 2 ACPI AML tables successfully acquired and loaded
May 18 16:28:24 ctl01 kernel: [    0.004005] APIC: Switch to symmetric I/O mode setup
May 18 16:28:24 ctl01 kernel: [    0.005400] x2apic enabled
May 18 16:28:24 ctl01 kernel: [    0.006406] Switched APIC routing to physical x2apic.
May 18 16:28:24 ctl01 kernel: [    0.008870] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
May 18 16:28:24 ctl01 kernel: [    0.010165] tsc: Detected 2799.994 MHz processor
May 18 16:28:24 ctl01 kernel: [    0.011173] Calibrating delay loop (skipped) preset value.. 5599.98 BogoMIPS (lpj=11199976)
May 18 16:28:24 ctl01 kernel: [    0.012003] pid_max: default: 32768 minimum: 301
May 18 16:28:24 ctl01 kernel: [    0.013048] Security Framework initialized
May 18 16:28:24 ctl01 kernel: [    0.013962] Yama: becoming mindful.
May 18 16:28:24 ctl01 kernel: [    0.014805] AppArmor: AppArmor initialized
May 18 16:28:24 ctl01 kernel: [    0.022161] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
May 18 16:28:24 ctl01 kernel: [    0.026185] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
May 18 16:28:24 ctl01 kernel: [    0.028035] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
May 18 16:28:24 ctl01 kernel: [    0.029988] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
May 18 16:28:24 ctl01 kernel: [    0.031831] mce: CPU supports 10 MCE banks
May 18 16:28:24 ctl01 kernel: [    0.032047] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
May 18 16:28:24 ctl01 kernel: [    0.033194] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
May 18 16:28:24 ctl01 kernel: [    0.034459] Spectre V2 : Mitigation: Full generic retpoline
May 18 16:28:24 ctl01 kernel: [    0.036002] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
May 18 16:28:24 ctl01 kernel: [    0.037746] Spectre V2 : Enabling Restricted Speculation for firmware calls
May 18 16:28:24 ctl01 kernel: [    0.039202] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
May 18 16:28:24 ctl01 kernel: [    0.040003] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
May 18 16:28:24 ctl01 kernel: [    0.042037] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
May 18 16:28:24 ctl01 kernel: [    0.050344] Freeing SMP alternatives memory: 36K
May 18 16:28:24 ctl01 kernel: [    0.052641] TSC deadline timer enabled
May 18 16:28:24 ctl01 kernel: [    0.052645] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (family: 0x6, model: 0x3e, stepping: 0x4)
May 18 16:28:24 ctl01 kernel: [    0.054813] Performance Events: IvyBridge events, Intel PMU driver.
May 18 16:28:24 ctl01 kernel: [    0.056000] ... version:                2
May 18 16:28:24 ctl01 kernel: [    0.056005] ... bit width:              48
May 18 16:28:24 ctl01 kernel: [    0.056935] ... generic registers:      4
May 18 16:28:24 ctl01 kernel: [    0.057855] ... value mask:             0000ffffffffffff
May 18 16:28:24 ctl01 kernel: [    0.059005] ... max period:             000000007fffffff
May 18 16:28:24 ctl01 kernel: [    0.060004] ... fixed-purpose events:   3
May 18 16:28:24 ctl01 kernel: [    0.060906] ... event mask:             000000070000000f
May 18 16:28:24 ctl01 kernel: [    0.062131] Hierarchical SRCU implementation.
May 18 16:28:24 ctl01 kernel: [    0.063850] smp: Bringing up secondary CPUs ...
May 18 16:28:24 ctl01 kernel: [    0.064109] x86: Booting SMP configuration:
May 18 16:28:24 ctl01 kernel: [    0.065062] .... node  #0, CPUs:      #1
May 18 16:28:24 ctl01 kernel: [    0.004000] kvm-clock: cpu 1, msr 3:bff54041, secondary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.068031] KVM setup async PF for cpu 1
May 18 16:28:24 ctl01 kernel: [    0.068903] kvm-stealtime: cpu 1, msr 3bfc64040
May 18 16:28:24 ctl01 kernel: [    0.069942]  #2
May 18 16:28:24 ctl01 kernel: [    0.004000] kvm-clock: cpu 2, msr 3:bff54081, secondary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.076052] KVM setup async PF for cpu 2
May 18 16:28:24 ctl01 kernel: [    0.076945] kvm-stealtime: cpu 2, msr 3bfca4040
May 18 16:28:24 ctl01 kernel: [    0.077971]  #3
May 18 16:28:24 ctl01 kernel: [    0.004000] kvm-clock: cpu 3, msr 3:bff540c1, secondary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.084037] KVM setup async PF for cpu 3
May 18 16:28:24 ctl01 kernel: [    0.084913] kvm-stealtime: cpu 3, msr 3bfce4040
May 18 16:28:24 ctl01 kernel: [    0.085927]  #4
May 18 16:28:24 ctl01 kernel: [    0.004000] kvm-clock: cpu 4, msr 3:bff54101, secondary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.092035] KVM setup async PF for cpu 4
May 18 16:28:24 ctl01 kernel: [    0.092913] kvm-stealtime: cpu 4, msr 3bfd24040
May 18 16:28:24 ctl01 kernel: [    0.093951]  #5
May 18 16:28:24 ctl01 kernel: [    0.004000] kvm-clock: cpu 5, msr 3:bff54141, secondary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.100073] KVM setup async PF for cpu 5
May 18 16:28:24 ctl01 kernel: [    0.102136] kvm-stealtime: cpu 5, msr 3bfd64040
May 18 16:28:24 ctl01 kernel: [    0.104117]  #6
May 18 16:28:24 ctl01 kernel: [    0.004000] kvm-clock: cpu 6, msr 3:bff54181, secondary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.108025] KVM setup async PF for cpu 6
May 18 16:28:24 ctl01 kernel: [    0.108896] kvm-stealtime: cpu 6, msr 3bfda4040
May 18 16:28:24 ctl01 kernel: [    0.109901]  #7
May 18 16:28:24 ctl01 kernel: [    0.004000] kvm-clock: cpu 7, msr 3:bff541c1, secondary cpu clock
May 18 16:28:24 ctl01 kernel: [    0.116035] KVM setup async PF for cpu 7
May 18 16:28:24 ctl01 kernel: [    0.116924] kvm-stealtime: cpu 7, msr 3bfde4040
May 18 16:28:24 ctl01 kernel: [    0.124013] smp: Brought up 1 node, 8 CPUs
May 18 16:28:24 ctl01 kernel: [    0.125349] smpboot: Max logical packages: 8
May 18 16:28:24 ctl01 kernel: [    0.126690] smpboot: Total of 8 processors activated (44799.90 BogoMIPS)
May 18 16:28:24 ctl01 kernel: [    0.128549] devtmpfs: initialized
May 18 16:28:24 ctl01 ifup[865]: DHCPACK of 192.168.11.149 from 192.168.11.3
May 18 16:28:24 ctl01 dhclient[956]: bound to 192.168.11.149 -- renewal in 1769 seconds.
May 18 16:28:24 ctl01 ifup[865]: bound to 192.168.11.149 -- renewal in 1769 seconds.
May 18 16:28:24 ctl01 systemd[1]: Started Raise network interfaces.
May 18 16:28:24 ctl01 systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
May 18 16:28:24 ctl01 systemd[1]: Reached target Network.
May 18 16:28:24 ctl01 cloud-init[1069]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'init' at Sat, 18 May 2019 16:28:20 +0000. Up 9.67 seconds.
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: | Device |   Up  |          Address           |      Mask     | Scope  |     Hw-Address    |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |  ens3  |  True |       192.168.11.149       | 255.255.255.0 | global | 52:54:00:9d:ad:df |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |  ens3  |  True | fe80::5054:ff:fe9d:addf/64 |       .       |  link  | 52:54:00:9d:ad:df |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |  ens4  | False |             .              |       .       |   .    | 52:54:00:f8:33:c0 |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |  ens5  | False |             .              |       .       |   .    | 52:54:00:60:db:e8 |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |  ens6  | False |             .              |       .       |   .    | 52:54:00:8b:60:5a |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |   lo   |  True |         127.0.0.1          |   255.0.0.0   |  host  |         .         |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |   lo   |  True |          ::1/128           |       .       |  host  |         .         |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
May 18 16:28:24 ctl01 kernel: [    0.129204] x86/mm: Memory block size: 128MB
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
May 18 16:28:24 ctl01 kernel: [    0.133098] evm: security.selinux
May 18 16:28:24 ctl01 kernel: [    0.134240] evm: security.SMACK64
May 18 16:28:24 ctl01 kernel: [    0.135120] evm: security.SMACK64EXEC
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
May 18 16:28:24 ctl01 kernel: [    0.135953] evm: security.SMACK64TRANSMUTE
May 18 16:28:24 ctl01 kernel: [    0.136006] evm: security.SMACK64MMAP
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
May 18 16:28:24 ctl01 kernel: [    0.136872] evm: security.apparmor
May 18 16:28:24 ctl01 kernel: [    0.137684] evm: security.ima
May 18 16:28:24 ctl01 kernel: [    0.138400] evm: security.capability
May 18 16:28:24 ctl01 kernel: [    0.139264] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |   0   |   0.0.0.0    | 192.168.11.3 |    0.0.0.0    |    ens3   |   UG  |
May 18 16:28:24 ctl01 kernel: [    0.140015] futex hash table entries: 2048 (order: 5, 131072 bytes)
May 18 16:28:24 ctl01 kernel: [    0.141480] pinctrl core: initialized pinctrl subsystem
May 18 16:28:24 ctl01 kernel: [    0.142795] RTC time: 16:28:10, date: 05/18/19
May 18 16:28:24 ctl01 kernel: [    0.144731] NET: Registered protocol family 16
May 18 16:28:24 ctl01 kernel: [    0.145795] audit: initializing netlink subsys (disabled)
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |   1   | 192.168.11.0 |   0.0.0.0    | 255.255.255.0 |    ens3   |   U   |
May 18 16:28:24 ctl01 kernel: [    0.147001] audit: type=2000 audit(1558196890.829:1): state=initialized audit_enabled=0 res=1
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
May 18 16:28:24 ctl01 kernel: [    0.148018] cpuidle: using governor ladder
May 18 16:28:24 ctl01 kernel: [    0.148999] cpuidle: using governor menu
May 18 16:28:24 ctl01 kernel: [    0.152361] ACPI: bus type PCI registered
May 18 16:28:24 ctl01 kernel: [    0.153289] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
May 18 16:28:24 ctl01 kernel: [    0.154789] PCI: Using configuration type 1 for base access
May 18 16:28:24 ctl01 kernel: [    0.156042] core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +-------+-------------+---------+-----------+-------+
May 18 16:28:24 ctl01 kernel: [    0.158806] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
May 18 16:28:24 ctl01 kernel: [    0.160012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: | Route | Destination | Gateway | Interface | Flags |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +-------+-------------+---------+-----------+-------+
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |   0   |  fe80::/64  |    ::   |    ens3   |   U   |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |   2   |    local    |    ::   |    ens3   |   U   |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: |   3   |   ff00::/8  |    ::   |    ens3   |   U   |
May 18 16:28:24 ctl01 cloud-init[1069]: ci-info: +-------+-------------+---------+-----------+-------+
May 18 16:28:24 ctl01 cloud-init[1069]: Generating public/private rsa key pair.
May 18 16:28:24 ctl01 cloud-init[1069]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
May 18 16:28:24 ctl01 cloud-init[1069]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
May 18 16:28:24 ctl01 kernel: [    0.161470] ACPI: Added _OSI(Module Device)
May 18 16:28:24 ctl01 kernel: [    0.161470] ACPI: Added _OSI(Processor Device)
May 18 16:28:24 ctl01 kernel: [    0.164006] ACPI: Added _OSI(3.0 _SCP Extensions)
May 18 16:28:24 ctl01 kernel: [    0.165062] ACPI: Added _OSI(Processor Aggregator Device)
May 18 16:28:24 ctl01 kernel: [    0.166244] ACPI: Added _OSI(Linux-Dell-Video)
May 18 16:28:24 ctl01 kernel: [    0.167229] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
May 18 16:28:24 ctl01 kernel: [    0.168005] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
May 18 16:28:24 ctl01 kernel: [    0.170843] ACPI: Interpreter enabled
May 18 16:28:24 ctl01 kernel: [    0.172019] ACPI: (supports S0 S5)
May 18 16:28:24 ctl01 kernel: [    0.172937] ACPI: Using IOAPIC for interrupt routing
May 18 16:28:24 ctl01 kernel: [    0.174193] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
May 18 16:28:24 ctl01 kernel: [    0.176645] ACPI: Enabled 16 GPEs in block 00 to 0F
May 18 16:28:24 ctl01 cloud-init[1069]: The key fingerprint is:
May 18 16:28:24 ctl01 cloud-init[1069]: SHA256:Ez6++5/VZl4797VJCi+Gf6qY4TqK45FLn3d6xuMMQEY root@ctl01
May 18 16:28:24 ctl01 cloud-init[1069]: The key's randomart image is:
May 18 16:28:24 ctl01 cloud-init[1069]: +---[RSA 2048]----+
May 18 16:28:24 ctl01 cloud-init[1069]: |   E             |
May 18 16:28:24 ctl01 cloud-init[1069]: |  .              |
May 18 16:28:24 ctl01 cloud-init[1069]: |   o    .        |
May 18 16:28:24 ctl01 cloud-init[1069]: |  o    . .       |
May 18 16:28:24 ctl01 cloud-init[1069]: |   .    S        |
May 18 16:28:24 ctl01 cloud-init[1069]: |  . .  . o     . |
May 18 16:28:24 ctl01 cloud-init[1069]: | +   .... ..  ..*|
May 18 16:28:24 ctl01 cloud-init[1069]: |..= .o+*+o oo+o=B|
May 18 16:28:24 ctl01 cloud-init[1069]: |.+.+ooO*++=+*o +*|
May 18 16:28:24 ctl01 cloud-init[1069]: +----[SHA256]-----+
May 18 16:28:24 ctl01 cloud-init[1069]: Generating public/private dsa key pair.
May 18 16:28:24 ctl01 cloud-init[1069]: Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
May 18 16:28:24 ctl01 kernel: [    0.182157] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
May 18 16:28:24 ctl01 kernel: [    0.183675] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
May 18 16:28:24 ctl01 cloud-init[1069]: Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
May 18 16:28:24 ctl01 cloud-init[1069]: The key fingerprint is:
May 18 16:28:24 ctl01 cloud-init[1069]: SHA256:/Fd9AcuHWhEJmN2GiYSQugT+0APdcYw826mtzT8/OPk root@ctl01
May 18 16:28:24 ctl01 cloud-init[1069]: The key's randomart image is:
May 18 16:28:24 ctl01 kernel: [    0.184014] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
May 18 16:28:24 ctl01 cloud-init[1069]: +---[DSA 1024]----+
May 18 16:28:24 ctl01 kernel: [    0.185614] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
May 18 16:28:24 ctl01 kernel: [    0.188407] acpiphp: Slot [3] registered
May 18 16:28:24 ctl01 kernel: [    0.189472] acpiphp: Slot [4] registered
May 18 16:28:24 ctl01 kernel: [    0.190545] acpiphp: Slot [5] registered
May 18 16:28:24 ctl01 kernel: [    0.191612] acpiphp: Slot [6] registered
May 18 16:28:24 ctl01 kernel: [    0.192058] acpiphp: Slot [7] registered
May 18 16:28:24 ctl01 cloud-init[1069]: |  . oo*.o.=.=+o  |
May 18 16:28:24 ctl01 cloud-init[1069]: | o . *.o + +.+=  |
May 18 16:28:24 ctl01 kernel: [    0.193126] acpiphp: Slot [9] registered
May 18 16:28:24 ctl01 cloud-init[1069]: |. + . + .   .= o |
May 18 16:28:24 ctl01 kernel: [    0.194196] acpiphp: Slot [10] registered
May 18 16:28:24 ctl01 kernel: [    0.195283] acpiphp: Slot [11] registered
May 18 16:28:24 ctl01 kernel: [    0.196056] acpiphp: Slot [12] registered
May 18 16:28:24 ctl01 kernel: [    0.197118] acpiphp: Slot [13] registered
May 18 16:28:24 ctl01 cloud-init[1069]: | o = . +    o ...|
May 18 16:28:24 ctl01 cloud-init[1069]: |  + o o S  .  . o|
May 18 16:28:24 ctl01 kernel: [    0.198219] acpiphp: Slot [14] registered
May 18 16:28:24 ctl01 cloud-init[1069]: |   o . . .   .  .|
May 18 16:28:24 ctl01 kernel: [    0.199292] acpiphp: Slot [15] registered
May 18 16:28:24 ctl01 kernel: [    0.200056] acpiphp: Slot [16] registered
May 18 16:28:24 ctl01 kernel: [    0.201125] acpiphp: Slot [17] registered
May 18 16:28:24 ctl01 kernel: [    0.202228] acpiphp: Slot [18] registered
May 18 16:28:24 ctl01 kernel: [    0.203289] acpiphp: Slot [19] registered
May 18 16:28:24 ctl01 cloud-init[1069]: |      +   + .    |
May 18 16:28:24 ctl01 cloud-init[1069]: |     . o = o     |
May 18 16:28:24 ctl01 kernel: [    0.204056] acpiphp: Slot [20] registered
May 18 16:28:24 ctl01 kernel: [    0.205145] acpiphp: Slot [21] registered
May 18 16:28:24 ctl01 kernel: [    0.206228] acpiphp: Slot [22] registered
May 18 16:28:24 ctl01 kernel: [    0.207293] acpiphp: Slot [23] registered
May 18 16:28:24 ctl01 kernel: [    0.208058] acpiphp: Slot [24] registered
May 18 16:28:24 ctl01 cloud-init[1069]: |        ..=E.    |
May 18 16:28:24 ctl01 kernel: [    0.209149] acpiphp: Slot [25] registered
May 18 16:28:24 ctl01 cloud-init[1069]: +----[SHA256]-----+
May 18 16:28:24 ctl01 kernel: [    0.210616] acpiphp: Slot [26] registered
May 18 16:28:24 ctl01 kernel: [    0.211677] acpiphp: Slot [27] registered
May 18 16:28:24 ctl01 kernel: [    0.212056] acpiphp: Slot [28] registered
May 18 16:28:24 ctl01 cloud-init[1069]: Generating public/private ecdsa key pair.
May 18 16:28:24 ctl01 kernel: [    0.213173] acpiphp: Slot [29] registered
May 18 16:28:24 ctl01 kernel: [    0.214247] acpiphp: Slot [30] registered
May 18 16:28:24 ctl01 kernel: [    0.215305] acpiphp: Slot [31] registered
May 18 16:28:24 ctl01 kernel: [    0.216024] PCI host bridge to bus 0000:00
May 18 16:28:24 ctl01 kernel: [    0.217080] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
May 18 16:28:24 ctl01 kernel: [    0.218683] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
May 18 16:28:24 ctl01 kernel: [    0.220007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
May 18 16:28:24 ctl01 kernel: [    0.221810] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
May 18 16:28:24 ctl01 kernel: [    0.223626] pci_bus 0000:00: root bus resource [bus 00-ff]
May 18 16:28:24 ctl01 cloud-init[1069]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
May 18 16:28:24 ctl01 kernel: [    0.224060] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
May 18 16:28:24 ctl01 kernel: [    0.224742] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
May 18 16:28:24 ctl01 cloud-init[1069]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
May 18 16:28:24 ctl01 kernel: [    0.225566] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
May 18 16:28:24 ctl01 cloud-init[1069]: The key fingerprint is:
May 18 16:28:24 ctl01 kernel: [    0.234090] pci 0000:00:01.1: reg 0x20: [io  0xc140-0xc14f]
May 18 16:28:24 ctl01 cloud-init[1069]: SHA256:eWKUgkAe7j2Ncxp/4leOHzHuNA3jccmKpDQxHyIf5UI root@ctl01
May 18 16:28:24 ctl01 kernel: [    0.238198] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
May 18 16:28:24 ctl01 cloud-init[1069]: The key's randomart image is:
May 18 16:28:24 ctl01 kernel: [    0.239886] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
May 18 16:28:24 ctl01 cloud-init[1069]: +---[ECDSA 256]---+
May 18 16:28:24 ctl01 kernel: [    0.240008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
May 18 16:28:24 ctl01 kernel: [    0.241704] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
May 18 16:28:24 ctl01 kernel: [    0.243505] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
May 18 16:28:24 ctl01 kernel: [    0.244128] pci 0000:00:01.3: quirk: [io  0x0600-0x063f] claimed by PIIX4 ACPI
May 18 16:28:24 ctl01 kernel: [    0.246143] pci 0000:00:01.3: quirk: [io  0x0700-0x070f] claimed by PIIX4 SMB
May 18 16:28:24 ctl01 kernel: [    0.248157] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000
May 18 16:28:24 ctl01 kernel: [    0.251494] pci 0000:00:02.0: reg 0x10: [mem 0xfc000000-0xfdffffff pref]
May 18 16:28:24 ctl01 cloud-init[1069]: | .+   E .        |
May 18 16:28:24 ctl01 kernel: [    0.253495] pci 0000:00:02.0: reg 0x14: [mem 0xfebd0000-0xfebd0fff]
May 18 16:28:24 ctl01 kernel: [    0.267929] pci 0000:00:02.0: reg 0x30: [mem 0xfebc0000-0xfebcffff pref]
May 18 16:28:24 ctl01 kernel: [    0.268239] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
May 18 16:28:24 ctl01 kernel: [    0.270670] pci 0000:00:03.0: reg 0x10: [io  0xc040-0xc05f]
May 18 16:28:24 ctl01 cloud-init[1069]: | o o o o .       |
May 18 16:28:24 ctl01 kernel: [    0.272008] pci 0000:00:03.0: reg 0x14: [mem 0xfebd1000-0xfebd1fff]
May 18 16:28:24 ctl01 kernel: [    0.283842] pci 0000:00:03.0: reg 0x30: [mem 0xfeac0000-0xfeafffff pref]
May 18 16:28:24 ctl01 kernel: [    0.284312] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000
May 18 16:28:24 ctl01 cloud-init[1069]: |  o o B =        |
May 18 16:28:24 ctl01 cloud-init[1069]: | . . = X o . .   |
May 18 16:28:24 ctl01 cloud-init[1069]: |  . * * S B +    |
May 18 16:28:24 ctl01 kernel: [    0.287717] pci 0000:00:04.0: reg 0x10: [io  0xc060-0xc07f]
May 18 16:28:24 ctl01 cloud-init[1069]: |     O = *.X     |
May 18 16:28:24 ctl01 kernel: [    0.289182] pci 0000:00:04.0: reg 0x14: [mem 0xfebd2000-0xfebd2fff]
May 18 16:28:24 ctl01 kernel: [    0.302228] pci 0000:00:04.0: reg 0x30: [mem 0xfeb00000-0xfeb3ffff pref]
May 18 16:28:24 ctl01 kernel: [    0.302711] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000
May 18 16:28:24 ctl01 kernel: [    0.305135] pci 0000:00:05.0: reg 0x10: [io  0xc080-0xc09f]
May 18 16:28:24 ctl01 kernel: [    0.307428] pci 0000:00:05.0: reg 0x14: [mem 0xfebd3000-0xfebd3fff]
May 18 16:28:24 ctl01 kernel: [    0.319362] pci 0000:00:05.0: reg 0x30: [mem 0xfeb40000-0xfeb7ffff pref]
May 18 16:28:24 ctl01 kernel: [    0.320488] pci 0000:00:06.0: [1af4:1000] type 00 class 0x020000
May 18 16:28:24 ctl01 kernel: [    0.322898] pci 0000:00:06.0: reg 0x10: [io  0xc0a0-0xc0bf]
May 18 16:28:24 ctl01 cloud-init[1069]: |    . + o+B .    |
May 18 16:28:24 ctl01 kernel: [    0.325146] pci 0000:00:06.0: reg 0x14: [mem 0xfebd4000-0xfebd4fff]
May 18 16:28:24 ctl01 kernel: [    0.337111] pci 0000:00:06.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref]
May 18 16:28:24 ctl01 cloud-init[1069]: |     . ooo.o     |
May 18 16:28:24 ctl01 kernel: [    0.337589] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000
May 18 16:28:24 ctl01 cloud-init[1069]: |      .. .o      |
May 18 16:28:24 ctl01 kernel: [    0.340007] pci 0000:00:07.0: reg 0x10: [io  0xc000-0xc03f]
May 18 16:28:24 ctl01 cloud-init[1069]: +----[SHA256]-----+
May 18 16:28:24 ctl01 kernel: [    0.342329] pci 0000:00:07.0: reg 0x14: [mem 0xfebd5000-0xfebd5fff]
May 18 16:28:24 ctl01 kernel: [    0.354857] pci 0000:00:08.0: [8086:2934] type 00 class 0x0c0300
May 18 16:28:24 ctl01 cloud-init[1069]: Generating public/private ed25519 key pair.
May 18 16:28:24 ctl01 kernel: [    0.360636] pci 0000:00:08.0: reg 0x20: [io  0xc0c0-0xc0df]
May 18 16:28:24 ctl01 kernel: [    0.363272] pci 0000:00:08.1: [8086:2935] type 00 class 0x0c0300
May 18 16:28:24 ctl01 kernel: [    0.369785] pci 0000:00:08.1: reg 0x20: [io  0xc0e0-0xc0ff]
May 18 16:28:24 ctl01 kernel: [    0.372293] pci 0000:00:08.2: [8086:2936] type 00 class 0x0c0300
May 18 16:28:24 ctl01 kernel: [    0.378931] pci 0000:00:08.2: reg 0x20: [io  0xc100-0xc11f]
May 18 16:28:24 ctl01 cloud-init[1069]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
May 18 16:28:24 ctl01 cloud-init[1069]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
May 18 16:28:24 ctl01 kernel: [    0.381510] pci 0000:00:08.7: [8086:293a] type 00 class 0x0c0320
May 18 16:28:24 ctl01 kernel: [    0.382750] pci 0000:00:08.7: reg 0x10: [mem 0xfebd6000-0xfebd6fff]
May 18 16:28:24 ctl01 kernel: [    0.391169] pci 0000:00:09.0: [1af4:1002] type 00 class 0x00ff00
May 18 16:28:24 ctl01 kernel: [    0.392008] pci 0000:00:09.0: reg 0x10: [io  0xc120-0xc13f]
May 18 16:28:24 ctl01 cloud-init[1069]: The key fingerprint is:
May 18 16:28:24 ctl01 kernel: [    0.402383] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
May 18 16:28:24 ctl01 kernel: [    0.404129] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
May 18 16:28:24 ctl01 cloud-init[1069]: SHA256:nj+ThnCTL9CoJdUUdUC4klZFu+9nrkRiT/RbwAHI+Lg root@ctl01
May 18 16:28:24 ctl01 kernel: [    0.405770] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
May 18 16:28:24 ctl01 kernel: [    0.407422] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
May 18 16:28:24 ctl01 kernel: [    0.408087] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
May 18 16:28:24 ctl01 cloud-init[1069]: The key's randomart image is:
May 18 16:28:24 ctl01 kernel: [    0.411103] SCSI subsystem initialized
May 18 16:28:24 ctl01 kernel: [    0.412106] libata version 3.00 loaded.
May 18 16:28:24 ctl01 cloud-init[1069]: +--[ED25519 256]--+
May 18 16:28:24 ctl01 kernel: [    0.412106] pci 0000:00:02.0: vgaarb: setting as boot VGA device
May 18 16:28:24 ctl01 cloud-init[1069]: |        .X*oo..  |
May 18 16:28:24 ctl01 kernel: [    0.413645] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
May 18 16:28:24 ctl01 kernel: [    0.416010] pci 0000:00:02.0: vgaarb: bridge control possible
May 18 16:28:24 ctl01 kernel: [    0.417544] vgaarb: loaded
May 18 16:28:24 ctl01 kernel: [    0.418387] ACPI: bus type USB registered
May 18 16:28:24 ctl01 kernel: [    0.419522] usbcore: registered new interface driver usbfs
May 18 16:28:24 ctl01 cloud-init[1069]: |        = oo . . |
May 18 16:28:24 ctl01 kernel: [    0.420017] usbcore: registered new interface driver hub
May 18 16:28:24 ctl01 cloud-init[1069]: |       = +. . o  |
May 18 16:28:24 ctl01 kernel: [    0.421507] usbcore: registered new device driver usb
May 18 16:28:24 ctl01 kernel: [    0.422969] EDAC MC: Ver: 3.0.0
May 18 16:28:24 ctl01 kernel: [    0.424462] PCI: Using ACPI for IRQ routing
May 18 16:28:24 ctl01 kernel: [    0.425213] PCI: pci_cache_line_size set to 64 bytes
May 18 16:28:24 ctl01 cloud-init[1069]: |      = + .o . . |
May 18 16:28:24 ctl01 kernel: [    0.425589] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
May 18 16:28:24 ctl01 kernel: [    0.425590] e820: reserve RAM buffer [mem 0xbffdf000-0xbfffffff]
May 18 16:28:24 ctl01 cloud-init[1069]: |     o +So+ o . .|
May 18 16:28:24 ctl01 kernel: [    0.425714] NetLabel: Initializing
May 18 16:28:24 ctl01 kernel: [    0.426701] NetLabel:  domain hash size = 128
May 18 16:28:24 ctl01 kernel: [    0.428007] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
May 18 16:28:24 ctl01 cloud-init[1069]: |    . =.Eo *   o |
May 18 16:28:24 ctl01 kernel: [    0.429517] NetLabel:  unlabeled traffic allowed by default
May 18 16:28:24 ctl01 kernel: [    0.431028] clocksource: Switched to clocksource kvm-clock
May 18 16:28:24 ctl01 kernel: [    0.443232] VFS: Disk quotas dquot_6.6.0
May 18 16:28:24 ctl01 cloud-init[1069]: |     + +o+ .+ .  |
May 18 16:28:24 ctl01 cloud-init[1069]: |    .   o.*o  o  |
May 18 16:28:24 ctl01 kernel: [    0.444368] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
May 18 16:28:24 ctl01 kernel: [    0.446257] AppArmor: AppArmor Filesystem Enabled
May 18 16:28:24 ctl01 kernel: [    0.447565] pnp: PnP ACPI init
May 18 16:28:24 ctl01 cloud-init[1069]: |         o.oo=.  |
May 18 16:28:24 ctl01 kernel: [    0.448532] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
May 18 16:28:24 ctl01 kernel: [    0.448587] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
May 18 16:28:24 ctl01 cloud-init[1069]: +----[SHA256]-----+
May 18 16:28:24 ctl01 kernel: [    0.448627] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active)
May 18 16:28:24 ctl01 kernel: [    0.448670] pnp 00:03: [dma 2]
May 18 16:28:24 ctl01 systemd[1]: Started Initial cloud-init job (metadata service crawler).
May 18 16:28:24 ctl01 kernel: [    0.448689] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active)
May 18 16:28:24 ctl01 kernel: [    0.448810] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
May 18 16:28:24 ctl01 kernel: [    0.449158] pnp: PnP ACPI: found 5 devices
May 18 16:28:24 ctl01 kernel: [    0.458708] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
May 18 16:28:24 ctl01 systemd[1]: Reached target System Initialization.
May 18 16:28:24 ctl01 kernel: [    0.461023] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
May 18 16:28:24 ctl01 kernel: [    0.461025] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
May 18 16:28:24 ctl01 kernel: [    0.461027] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
May 18 16:28:24 ctl01 kernel: [    0.461028] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
May 18 16:28:24 ctl01 kernel: [    0.461106] NET: Registered protocol family 2
May 18 16:28:24 ctl01 systemd[1]: Listening on UUID daemon activation socket.
May 18 16:28:24 ctl01 systemd[1]: Listening on D-Bus System Message Bus Socket.
May 18 16:28:24 ctl01 systemd[1]: Starting LXD - unix socket.
May 18 16:28:24 ctl01 systemd[1]: Started Daily Cleanup of Temporary Directories.
May 18 16:28:24 ctl01 systemd[1]: Started ACPI Events Check.
May 18 16:28:24 ctl01 systemd[1]: Reached target Paths.
May 18 16:28:24 ctl01 systemd[1]: Starting Socket activation for snappy daemon.
May 18 16:28:24 ctl01 systemd[1]: Listening on ACPID Listen Socket.
May 18 16:28:24 ctl01 systemd[1]: Started Daily apt download activities.
May 18 16:28:24 ctl01 systemd[1]: Started Daily apt upgrade and clean activities.
May 18 16:28:24 ctl01 systemd[1]: Reached target Timers.
May 18 16:28:24 ctl01 systemd[1]: Reached target Cloud-config availability.
May 18 16:28:24 ctl01 systemd[1]: Reached target Network is Online.
May 18 16:28:24 ctl01 systemd[1]: Starting iSCSI initiator daemon (iscsid)...
May 18 16:28:24 ctl01 systemd[1]: Listening on LXD - unix socket.
May 18 16:28:24 ctl01 systemd[1]: Listening on Socket activation for snappy daemon.
May 18 16:28:24 ctl01 systemd[1]: Reached target Sockets.
May 18 16:28:24 ctl01 kernel: [    0.463133] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
May 18 16:28:24 ctl01 systemd[1]: Reached target Basic System.
May 18 16:28:24 ctl01 systemd[1]: Starting LXD - container startup/shutdown...
May 18 16:28:24 ctl01 kernel: [    0.465295] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
May 18 16:28:24 ctl01 systemd[1]: Starting LSB: Record successful boot for GRUB...
May 18 16:28:24 ctl01 kernel: [    0.467661] TCP: Hash tables configured (established 131072 bind 65536)
May 18 16:28:24 ctl01 kernel: [    0.469400] UDP hash table entries: 8192 (order: 6, 262144 bytes)
May 18 16:28:24 ctl01 kernel: [    0.471033] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
May 18 16:28:24 ctl01 kernel: [    0.472810] NET: Registered protocol family 1
May 18 16:28:24 ctl01 systemd[1]: Started D-Bus System Message Bus.
May 18 16:28:24 ctl01 kernel: [    0.474008] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
May 18 16:28:24 ctl01 dbus[1181]: [system] AppArmor D-Bus mediation is enabled
May 18 16:28:24 ctl01 kernel: [    0.475573] pci 0000:00:01.0: PIIX3: Enabling Passive Release
May 18 16:28:24 ctl01 kernel: [    0.477101] pci 0000:00:01.0: Activating ISA DMA hang workarounds
May 18 16:28:24 ctl01 kernel: [    0.478772] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
May 18 16:28:24 ctl01 systemd[1]: Starting Accounts Service...
May 18 16:28:24 ctl01 kernel: [    0.517421] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
May 18 16:28:24 ctl01 kernel: [    0.580068] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
May 18 16:28:24 ctl01 kernel: [    0.642951] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
May 18 16:28:24 ctl01 systemd[1]: Started Deferred execution scheduler.
May 18 16:28:24 ctl01 kernel: [    0.705414] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
May 18 16:28:24 ctl01 kernel: [    0.737384] PCI: CLS 0 bytes, default 64
May 18 16:28:24 ctl01 kernel: [    0.737422] Unpacking initramfs...
May 18 16:28:24 ctl01 kernel: [    1.473711] Freeing initrd memory: 53476K
May 18 16:28:24 ctl01 kernel: [    1.474753] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
May 18 16:28:24 ctl01 systemd[1]: Starting Pollinate to seed the pseudo random number generator...
May 18 16:28:24 ctl01 kernel: [    1.476161] software IO TLB [mem 0xbbfdf000-0xbffdf000] (64MB) mapped at [        (ptrval)-        (ptrval)]
May 18 16:28:24 ctl01 kernel: [    1.478263] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x285c3aeaff3, max_idle_ns: 440795255742 ns
May 18 16:28:24 ctl01 kernel: [    1.480409] Scanning for low memory corruption every 60 seconds
May 18 16:28:24 ctl01 systemd[1]: Starting /etc/rc.local Compatibility...
May 18 16:28:24 ctl01 kernel: [    1.482753] Initialise system trusted keyrings
May 18 16:28:24 ctl01 kernel: [    1.483789] Key type blacklist registered
May 18 16:28:24 ctl01 kernel: [    1.484973] workingset: timestamp_bits=36 max_order=22 bucket_order=0
May 18 16:28:24 ctl01 systemd[1]: Starting Snappy daemon...
May 18 16:28:24 ctl01 kernel: [    1.487518] zbud: loaded
May 18 16:28:24 ctl01 kernel: [    1.489094] squashfs: version 4.0 (2009/01/31) Phillip Lougher
May 18 16:28:24 ctl01 systemd[1]: Starting Thermal Daemon Service...
May 18 16:28:24 ctl01 kernel: [    1.490898] fuse init (API version 7.26)
May 18 16:28:24 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:28:24 ctl01 kernel: [    1.496207] Key type asymmetric registered
May 18 16:28:24 ctl01 kernel: [    1.498327] Asymmetric key parser 'x509' registered
May 18 16:28:24 ctl01 kernel: [    1.500593] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
May 18 16:28:24 ctl01 kernel: [    1.504040] io scheduler noop registered
May 18 16:28:24 ctl01 kernel: [    1.505694] io scheduler deadline registered
May 18 16:28:24 ctl01 kernel: [    1.507613] io scheduler cfq registered (default)
May 18 16:28:24 ctl01 kernel: [    1.510499] intel_idle: Please enable MWAIT in BIOS SETUP
May 18 16:28:24 ctl01 kernel: [    1.510621] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
May 18 16:28:24 ctl01 kernel: [    1.513779] ACPI: Power Button [PWRF]
May 18 16:28:24 ctl01 systemd[1]: Starting LSB: MD monitoring daemon...
May 18 16:28:24 ctl01 kernel: [    1.565294] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
May 18 16:28:24 ctl01 kernel: [    1.613625] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
May 18 16:28:24 ctl01 systemd[1]: Starting The Salt Minion...
May 18 16:28:24 ctl01 kernel: [    1.654076] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
May 18 16:28:24 ctl01 kernel: [    1.686785] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
May 18 16:28:24 ctl01 kernel: [    1.719499] virtio-pci 0000:00:07.0: virtio_pci: leaving for legacy driver
May 18 16:28:24 ctl01 systemd[1]: Started FUSE filesystem for LXC.
May 18 16:28:24 ctl01 kernel: [    1.751684] virtio-pci 0000:00:09.0: virtio_pci: leaving for legacy driver
May 18 16:28:24 ctl01 kernel: [    1.754329] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
May 18 16:28:24 ctl01 systemd[1]: Started Regular background program processing daemon.
May 18 16:28:24 ctl01 kernel: [    1.780167] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
May 18 16:28:24 ctl01 kernel: [    1.783226] Linux agpgart interface v0.103
May 18 16:28:24 ctl01 kernel: [    1.787517] loop: module loaded
May 18 16:28:24 ctl01 kernel: [    1.788384] ata_piix 0000:00:01.1: version 2.13
May 18 16:28:24 ctl01 systemd[1]: Starting System Logging Service...
May 18 16:28:24 ctl01 kernel: [    1.789738] scsi host0: ata_piix
May 18 16:28:24 ctl01 kernel: [    1.790790] scsi host1: ata_piix
May 18 16:28:24 ctl01 kernel: [    1.791592] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc140 irq 14
May 18 16:28:24 ctl01 kernel: [    1.793031] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc148 irq 15
May 18 16:28:24 ctl01 kernel: [    1.794535] libphy: Fixed MDIO Bus: probed
May 18 16:28:24 ctl01 kernel: [    1.795601] tun: Universal TUN/TAP device driver, 1.6
May 18 16:28:24 ctl01 systemd[1]: Starting Login Service...
May 18 16:28:24 ctl01 systemd[1]: Started Unattended Upgrades Shutdown.
May 18 16:28:24 ctl01 kernel: [    1.796745] PPP generic driver version 2.4.2
May 18 16:28:24 ctl01 cron[1298]: (CRON) INFO (pidfile fd = 3)
May 18 16:28:24 ctl01 kernel: [    1.797754] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
May 18 16:28:24 ctl01 kernel: [    1.806602] ehci-pci: EHCI PCI platform driver
May 18 16:28:24 ctl01 kernel: [    1.838841] ehci-pci 0000:00:08.7: EHCI Host Controller
May 18 16:28:24 ctl01 kernel: [    1.840000] ehci-pci 0000:00:08.7: new USB bus registered, assigned bus number 1
May 18 16:28:24 ctl01 systemd[1]: Started /etc/rc.local Compatibility.
May 18 16:28:24 ctl01 kernel: [    1.841819] ehci-pci 0000:00:08.7: irq 11, io mem 0xfebd6000
May 18 16:28:24 ctl01 mdadm[1272]:  * Starting MD monitoring service mdadm --monitor
May 18 16:28:24 ctl01 kernel: [    1.856091] ehci-pci 0000:00:08.7: USB 2.0 started, EHCI 1.00
May 18 16:28:24 ctl01 kernel: [    1.857747] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
May 18 16:28:24 ctl01 pollinate[1231]: client sent challenge to [https://entropy.ubuntu.com/]
May 18 16:28:24 ctl01 kernel: [    1.859597] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:28:24 ctl01 kernel: [    1.861580] usb usb1: Product: EHCI Host Controller
May 18 16:28:24 ctl01 kernel: [    1.862963] usb usb1: Manufacturer: Linux 4.15.0-50-generic ehci_hcd
May 18 16:28:24 ctl01 kernel: [    1.864682] usb usb1: SerialNumber: 0000:00:08.7
May 18 16:28:24 ctl01 systemd[1]: Started Login Service.
May 18 16:28:24 ctl01 kernel: [    1.866096] hub 1-0:1.0: USB hub found
May 18 16:28:24 ctl01 kernel: [    1.867144] hub 1-0:1.0: 6 ports detected
May 18 16:28:24 ctl01 kernel: [    1.868279] ehci-platform: EHCI generic platform driver
May 18 16:28:24 ctl01 kernel: [    1.869437] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
May 18 16:28:24 ctl01 kernel: [    1.870747] ohci-pci: OHCI PCI platform driver
May 18 16:28:24 ctl01 acpid: starting up with netlink and the input layer
May 18 16:28:24 ctl01 kernel: [    1.871753] ohci-platform: OHCI generic platform driver
May 18 16:28:24 ctl01 iscsid: iSCSI logger with pid=1362 started!
May 18 16:28:24 ctl01 kernel: [    1.872897] uhci_hcd: USB Universal Host Controller Interface driver
May 18 16:28:24 ctl01 kernel: [    1.907340] uhci_hcd 0000:00:08.0: UHCI Host Controller
May 18 16:28:24 ctl01 kernel: [    1.908589] uhci_hcd 0000:00:08.0: new USB bus registered, assigned bus number 2
May 18 16:28:24 ctl01 kernel: [    1.910341] uhci_hcd 0000:00:08.0: detected 2 ports
May 18 16:28:24 ctl01 kernel: [    1.911577] uhci_hcd 0000:00:08.0: irq 11, io base 0x0000c0c0
May 18 16:28:24 ctl01 systemd[1]: Started iSCSI initiator daemon (iscsid).
May 18 16:28:24 ctl01 kernel: [    1.912993] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
May 18 16:28:24 ctl01 kernel: [    1.914626] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:28:24 ctl01 kernel: [    1.916382] usb usb2: Product: UHCI Host Controller
May 18 16:28:24 ctl01 kernel: [    1.917704] usb usb2: Manufacturer: Linux 4.15.0-50-generic uhci_hcd
May 18 16:28:24 ctl01 kernel: [    1.919205] usb usb2: SerialNumber: 0000:00:08.0
May 18 16:28:24 ctl01 kernel: [    1.920485] hub 2-0:1.0: USB hub found
May 18 16:28:24 ctl01 systemd[1]: Starting Login to default iSCSI targets...
May 18 16:28:24 ctl01 kernel: [    1.921465] hub 2-0:1.0: 2 ports detected
May 18 16:28:24 ctl01 acpid: 1 rule loaded
May 18 16:28:24 ctl01 kernel: [    1.952809] ata1.01: NODEV after polling detection
May 18 16:28:24 ctl01 kernel: [    1.953145] ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
May 18 16:28:24 ctl01 kernel: [    1.955102] ata1.00: configured for MWDMA2
May 18 16:28:24 ctl01 kernel: [    1.956463] uhci_hcd 0000:00:08.1: UHCI Host Controller
May 18 16:28:24 ctl01 kernel: [    1.956469] uhci_hcd 0000:00:08.1: new USB bus registered, assigned bus number 3
May 18 16:28:24 ctl01 acpid: waiting for events: event logging is off
May 18 16:28:24 ctl01 kernel: [    1.956500] uhci_hcd 0000:00:08.1: detected 2 ports
May 18 16:28:24 ctl01 cron[1298]: (CRON) INFO (Running @reboot jobs)
May 18 16:28:24 ctl01 kernel: [    1.956599] uhci_hcd 0000:00:08.1: irq 10, io base 0x0000c0e0
May 18 16:28:24 ctl01 kernel: [    1.956706] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
May 18 16:28:24 ctl01 lxcfs[1282]: hierarchies:
May 18 16:28:24 ctl01 kernel: [    1.956708] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:28:24 ctl01 kernel: [    1.956709] usb usb3: Product: UHCI Host Controller
May 18 16:28:24 ctl01 kernel: [    1.956710] usb usb3: Manufacturer: Linux 4.15.0-50-generic uhci_hcd
May 18 16:28:24 ctl01 lxcfs[1282]:   0: fd:   5: devices
May 18 16:28:24 ctl01 kernel: [    1.956711] usb usb3: SerialNumber: 0000:00:08.1
May 18 16:28:24 ctl01 kernel: [    1.956794] hub 3-0:1.0: USB hub found
May 18 16:28:24 ctl01 kernel: [    1.956799] hub 3-0:1.0: 2 ports detected
May 18 16:28:24 ctl01 kernel: [    1.972320] scsi 0:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
May 18 16:28:24 ctl01 kernel: [    1.975348] sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
May 18 16:28:24 ctl01 kernel: [    1.976829] cdrom: Uniform CD-ROM driver Revision: 3.20
May 18 16:28:24 ctl01 lxcfs[1282]:   1: fd:   6: rdma
May 18 16:28:24 ctl01 lxcfs[1282]:   2: fd:   7: cpu,cpuacct
May 18 16:28:24 ctl01 kernel: [    1.978164] sr 0:0:0:0: Attached scsi CD-ROM sr0
May 18 16:28:24 ctl01 kernel: [    1.978210] sr 0:0:0:0: Attached scsi generic sg0 type 5
May 18 16:28:24 ctl01 lxcfs[1282]:   3: fd:   8: hugetlb
May 18 16:28:24 ctl01 kernel: [    1.989127] uhci_hcd 0000:00:08.2: UHCI Host Controller
May 18 16:28:24 ctl01 lxcfs[1282]:   4: fd:   9: net_cls,net_prio
May 18 16:28:24 ctl01 kernel: [    1.990395] uhci_hcd 0000:00:08.2: new USB bus registered, assigned bus number 4
May 18 16:28:24 ctl01 kernel: [    1.992186] uhci_hcd 0000:00:08.2: detected 2 ports
May 18 16:28:24 ctl01 lxcfs[1282]:   5: fd:  10: pids
May 18 16:28:24 ctl01 kernel: [    1.993442] uhci_hcd 0000:00:08.2: irq 10, io base 0x0000c100
May 18 16:28:24 ctl01 kernel: [    1.994880] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
May 18 16:28:24 ctl01 lxcfs[1282]:   6: fd:  11: perf_event
May 18 16:28:24 ctl01 kernel: [    1.996481] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:28:24 ctl01 kernel: [    1.998202] usb usb4: Product: UHCI Host Controller
May 18 16:28:24 ctl01 kernel: [    1.999541] usb usb4: Manufacturer: Linux 4.15.0-50-generic uhci_hcd
May 18 16:28:24 ctl01 kernel: [    2.001041] usb usb4: SerialNumber: 0000:00:08.2
May 18 16:28:24 ctl01 lxcfs[1282]:   7: fd:  12: memory
May 18 16:28:24 ctl01 kernel: [    2.002319] hub 4-0:1.0: USB hub found
May 18 16:28:24 ctl01 kernel: [    2.003309] hub 4-0:1.0: 2 ports detected
May 18 16:28:24 ctl01 kernel: [    2.004499] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
May 18 16:28:24 ctl01 lxcfs[1282]:   8: fd:  13: cpuset
May 18 16:28:24 ctl01 kernel: [    2.007384] serio: i8042 KBD port at 0x60,0x64 irq 1
May 18 16:28:24 ctl01 kernel: [    2.008660] serio: i8042 AUX port at 0x60,0x64 irq 12
May 18 16:28:24 ctl01 kernel: [    2.010054] mousedev: PS/2 mouse device common for all mice
May 18 16:28:24 ctl01 lxcfs[1282]:   9: fd:  14: blkio
May 18 16:28:24 ctl01 kernel: [    2.012112] rtc_cmos 00:00: RTC can wake from S4
May 18 16:28:24 ctl01 lxcfs[1282]:  10: fd:  15: freezer
May 18 16:28:24 ctl01 kernel: [    2.013749] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
May 18 16:28:24 ctl01 kernel: [    2.015253] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
May 18 16:28:24 ctl01 lxcfs[1282]:  11: fd:  16: name=systemd
May 18 16:28:24 ctl01 kernel: [    2.017245] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
May 18 16:28:24 ctl01 kernel: [    2.018667] i2c /dev entries driver
May 18 16:28:24 ctl01 mdadm[1272]:    ...done.
May 18 16:28:24 ctl01 systemd[1]: Started LSB: MD monitoring daemon.
May 18 16:28:24 ctl01 iscsiadm[1377]: iscsiadm: No records found
May 18 16:28:24 ctl01 systemd[1]: Started Login to default iSCSI targets.
May 18 16:28:24 ctl01 systemd[1]: Reached target Remote File Systems (Pre).
May 18 16:28:24 ctl01 systemd[1]: Reached target Remote File Systems.
May 18 16:28:24 ctl01 systemd[1]: Starting LSB: automatic crash report generation...
May 18 16:28:24 ctl01 systemd[1]: Starting LSB: Set the CPU Frequency Scaling governor to "ondemand"...
May 18 16:28:24 ctl01 systemd[1]: Starting LSB: daemon to balance interrupts for SMP systems...
May 18 16:28:24 ctl01 systemd[1]: Starting Permit User Sessions...
May 18 16:28:24 ctl01 systemd[1]: Started LSB: Record successful boot for GRUB.
May 18 16:28:24 ctl01 systemd[1]: Started LSB: Set the CPU Frequency Scaling governor to "ondemand".
May 18 16:28:24 ctl01 systemd[1]: Started Permit User Sessions.
May 18 16:28:24 ctl01 kernel: [    2.019630] pcie_mp2_amd: AMD(R) PCI-E MP2 Communication Driver Version: 1.0
May 18 16:28:24 ctl01 irqbalance[1420]:  * Starting SMP IRQ Balancer: irqbalance
May 18 16:28:24 ctl01 kernel: [    2.021212] device-mapper: uevent: version 1.0.3
May 18 16:28:24 ctl01 kernel: [    2.022406] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com
May 18 16:28:24 ctl01 kernel: [    2.024781] ledtrig-cpu: registered to indicate activity on CPUs
May 18 16:28:24 ctl01 systemd[1]: Starting Hold until boot process finishes up...
May 18 16:28:24 ctl01 kernel: [    2.026847] NET: Registered protocol family 10
May 18 16:28:24 ctl01 apport[1407]:  * Starting automatic crash report generation: apport
May 18 16:28:24 ctl01 kernel: [    2.034612] Segment Routing with IPv6
May 18 16:28:24 ctl01 kernel: [    2.035556] NET: Registered protocol family 17
May 18 16:28:24 ctl01 systemd[1]: Starting Terminate Plymouth Boot Screen...
May 18 16:28:24 ctl01 kernel: [    2.036805] Key type dns_resolver registered
May 18 16:28:24 ctl01 kernel: [    2.039063] RAS: Correctable Errors collector initialized.
May 18 16:28:24 ctl01 systemd[1]: Started Hold until boot process finishes up.
May 18 16:28:24 ctl01 kernel: [    2.040378] sched_clock: Marking stable (2040360584, 0)->(2589119662, -548759078)
May 18 16:28:24 ctl01 kernel: [    2.042959] registered taskstats version 1
May 18 16:28:24 ctl01 kernel: [    2.044059] Loading compiled-in X.509 certificates
May 18 16:28:24 ctl01 kernel: [    2.049683] Loaded X.509 cert 'Build time autogenerated kernel key: 96649165ed7c6e7d68b41a1fadd9ef21d69b21a5'
May 18 16:28:24 ctl01 kernel: [    2.052497] zswap: loaded using pool lzo/zbud
May 18 16:28:24 ctl01 kernel: [    2.060616] Key type big_key registered
May 18 16:28:24 ctl01 kernel: [    2.061674] Key type trusted registered
May 18 16:28:24 ctl01 kernel: [    2.065357] Key type encrypted registered
May 18 16:28:24 ctl01 kernel: [    2.066548] AppArmor: AppArmor sha1 policy hashing enabled
May 18 16:28:24 ctl01 systemd[1]: Started Getty on tty1.
May 18 16:28:24 ctl01 kernel: [    2.068139] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
May 18 16:28:24 ctl01 kernel: [    2.069882] ima: Allocated hash algorithm: sha1
May 18 16:28:24 ctl01 kernel: [    2.071230] evm: HMAC attrs: 0x1
May 18 16:28:24 ctl01 kernel: [    2.072761]   Magic number: 3:968:489
May 18 16:28:24 ctl01 kernel: [    2.073894] rtc_cmos 00:00: setting system clock to 2019-05-18 16:28:12 UTC (1558196892)
May 18 16:28:24 ctl01 systemd[1]: Started Serial Getty on ttyS0.
May 18 16:28:24 ctl01 kernel: [    2.075997] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
May 18 16:28:24 ctl01 systemd[1]: Reached target Login Prompts.
May 18 16:28:24 ctl01 kernel: [    2.077349] EDD information not available.
May 18 16:28:24 ctl01 systemd[1]: Starting Set console scheme...
May 18 16:28:24 ctl01 kernel: [    2.453019] Freeing unused kernel image memory: 2408K
May 18 16:28:24 ctl01 systemd[1]: Started Terminate Plymouth Boot Screen.
May 18 16:28:24 ctl01 kernel: [    2.468091] Write protecting the kernel read-only data: 20480k
May 18 16:28:24 ctl01 kernel: [    2.472668] Freeing unused kernel image memory: 2008K
May 18 16:28:24 ctl01 kernel: [    2.475536] Freeing unused kernel image memory: 1856K
May 18 16:28:24 ctl01 kernel: [    2.486342] x86/mm: Checked W+X mappings: passed, no W+X pages found.
May 18 16:28:24 ctl01 kernel: [    2.487739] x86/mm: Checking user space page tables
May 18 16:28:24 ctl01 kernel: [    2.496458] x86/mm: Checked W+X mappings: passed, no W+X pages found.
May 18 16:28:24 ctl01 kernel: [    2.525219] random: systemd-udevd: uninitialized urandom read (16 bytes read)
May 18 16:28:24 ctl01 kernel: [    2.526692] random: udevadm: uninitialized urandom read (16 bytes read)
May 18 16:28:24 ctl01 kernel: [    2.526730] random: udevadm: uninitialized urandom read (16 bytes read)
May 18 16:28:24 ctl01 kernel: [    2.583091] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
May 18 16:28:24 ctl01 kernel: [    2.583413]  vda: vda1
May 18 16:28:24 ctl01 kernel: [    2.586393] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
May 18 16:28:24 ctl01 kernel: [    2.592608] FDC 0 is a S82078B
May 18 16:28:24 ctl01 kernel: [    2.596498] AVX version of gcm_enc/dec engaged.
May 18 16:28:24 ctl01 kernel: [    2.597546] AES CTR mode by8 optimization enabled
May 18 16:28:24 ctl01 kernel: [    2.599052] [TTM] Zone  kernel: Available graphics memory: 7180132 kiB
May 18 16:28:24 ctl01 kernel: [    2.600508] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
May 18 16:28:24 ctl01 kernel: [    2.601919] [TTM] Initializing pool allocator
May 18 16:28:24 ctl01 kernel: [    2.603023] [TTM] Initializing DMA pool allocator
May 18 16:28:24 ctl01 kernel: [    2.605201] [drm] fb mappable at 0xFC000000
May 18 16:28:24 ctl01 kernel: [    2.606172] [drm] vram aper at 0xFC000000
May 18 16:28:24 ctl01 kernel: [    2.607109] [drm] size 33554432
May 18 16:28:24 ctl01 kernel: [    2.607869] [drm] fb depth is 24
May 18 16:28:24 ctl01 kernel: [    2.608652] [drm]    pitch is 3072
May 18 16:28:24 ctl01 kernel: [    2.609957] fbcon: cirrusdrmfb (fb0) is primary device
May 18 16:28:24 ctl01 kernel: [    2.613758] virtio_net virtio0 ens3: renamed from eth0
May 18 16:28:24 ctl01 kernel: [    2.624272] virtio_net virtio1 ens4: renamed from eth1
May 18 16:28:24 ctl01 kernel: [    2.630875] Console: switching to colour frame buffer device 128x48
May 18 16:28:24 ctl01 kernel: [    2.637378] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device
May 18 16:28:24 ctl01 kernel: [    2.652283] virtio_net virtio3 ens6: renamed from eth3
May 18 16:28:24 ctl01 kernel: [    2.656080] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0
May 18 16:28:24 ctl01 kernel: [    2.676150] virtio_net virtio2 ens5: renamed from eth2
May 18 16:28:24 ctl01 kernel: [    2.844027] raid6: sse2x1   gen()  7119 MB/s
May 18 16:28:24 ctl01 kernel: [    2.892027] raid6: sse2x1   xor()  5708 MB/s
May 18 16:28:24 ctl01 kernel: [    2.940031] raid6: sse2x2   gen()  7512 MB/s
May 18 16:28:24 ctl01 kernel: [    2.988029] raid6: sse2x2   xor()  5323 MB/s
May 18 16:28:24 ctl01 kernel: [    3.036023] raid6: sse2x4   gen()  9727 MB/s
May 18 16:28:24 ctl01 kernel: [    3.084025] raid6: sse2x4   xor()  7761 MB/s
May 18 16:28:24 ctl01 kernel: [    3.084733] raid6: using algorithm sse2x4 gen() 9727 MB/s
May 18 16:28:24 ctl01 kernel: [    3.085610] raid6: .... xor() 7761 MB/s, rmw enabled
May 18 16:28:24 ctl01 kernel: [    3.086419] raid6: using ssse3x2 recovery algorithm
May 18 16:28:24 ctl01 kernel: [    3.088928] xor: automatically using best checksumming function   avx       
May 18 16:28:24 ctl01 kernel: [    3.092215] async_tx: api initialized (async)
May 18 16:28:24 ctl01 kernel: [    3.153692] Btrfs loaded, crc32c=crc32c-intel
May 18 16:28:24 ctl01 kernel: [    3.194876] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
May 18 16:28:24 ctl01 kernel: [    3.335872] random: fast init done
May 18 16:28:24 ctl01 kernel: [    3.787149] EXT4-fs (vda1): re-mounted. Opts: (null)
May 18 16:28:24 ctl01 kernel: [    3.800702] Loading iSCSI transport class v2.0-870.
May 18 16:28:24 ctl01 kernel: [    3.820653] iscsi: registered transport (tcp)
May 18 16:28:24 ctl01 kernel: [    3.862576] iscsi: registered transport (iser)
May 18 16:28:24 ctl01 kernel: [    4.057008] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0
May 18 16:28:24 ctl01 kernel: [    4.251255] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
May 18 16:28:24 ctl01 kernel: [    4.251263] EDAC sbridge:  Ver: 1.1.2 
May 18 16:28:24 ctl01 kernel: [    4.526769] audit: type=1400 audit(1558196894.948:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=749 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.591381] audit: type=1400 audit(1558196895.012:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lxd/lxd-bridge-proxy" pid=739 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.620280] audit: type=1400 audit(1558196895.044:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=748 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.698186] audit: type=1400 audit(1558196895.120:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=740 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.698569] audit: type=1400 audit(1558196895.120:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=740 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.698926] audit: type=1400 audit(1558196895.120:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=740 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.699275] audit: type=1400 audit(1558196895.120:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=740 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.750208] audit: type=1400 audit(1558196895.172:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=743 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    4.750405] audit: type=1400 audit(1558196895.172:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=743 comm="apparmor_parser"
May 18 16:28:24 ctl01 kernel: [    5.831405] ISO 9660 Extensions: Microsoft Joliet Level 3
May 18 16:28:24 ctl01 kernel: [    5.930967] ISO 9660 Extensions: RRIP_1991A
May 18 16:28:24 ctl01 kernel: [   11.227524] EXT4-fs (vda1): resizing filesystem from 786171 to 26214139 blocks
May 18 16:28:24 ctl01 kernel: [   12.000142] EXT4-fs (vda1): resized filesystem to 26214139
May 18 16:28:24 ctl01 kernel: [   14.215899] new mount options do not match the existing superblock, will be ignored
May 18 16:28:24 ctl01 systemd[1]: Started System Logging Service.
May 18 16:28:24 ctl01 apport[1407]:    ...done.
May 18 16:28:24 ctl01 systemd[1]: Started LSB: automatic crash report generation.
May 18 16:28:24 ctl01 systemd[1]: Started Set console scheme.
May 18 16:28:24 ctl01 irqbalance[1420]:    ...done.
May 18 16:28:24 ctl01 systemd[1]: Started LSB: daemon to balance interrupts for SMP systems.
May 18 16:28:24 ctl01 dbus[1181]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkitd.service'
May 18 16:28:24 ctl01 systemd[1]: Starting Authenticate and Authorize Users to Run Privileged Tasks...
May 18 16:28:24 ctl01 systemd[1]: Started Thermal Daemon Service.
May 18 16:28:25 ctl01 thermald[1257]: NO RAPL sysfs present
May 18 16:28:25 ctl01 thermald[1257]: 13 CPUID levels; family:model:stepping 0x6:3e:4 (6:62:4)
May 18 16:28:25 ctl01 thermald[1257]:  Need Linux PowerCap sysfs
May 18 16:28:25 ctl01 thermald[1257]: Unsupported cpu model, use thermal-conf.xml file or run with --ignore-cpuid-check
May 18 16:28:25 ctl01 thermald[1257]: THD engine start failed
May 18 16:28:25 ctl01 polkitd[1545]: started daemon version 0.105 using authority implementation `local' version `0.105'
May 18 16:28:25 ctl01 dbus[1181]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'
May 18 16:28:25 ctl01 systemd[1]: Started Authenticate and Authorize Users to Run Privileged Tasks.
May 18 16:28:25 ctl01 accounts-daemon[1221]: started daemon version 0.6.40
May 18 16:28:25 ctl01 systemd[1]: Started Accounts Service.
May 18 16:28:25 ctl01 systemd[1]: Started LXD - container startup/shutdown.
May 18 16:28:25 ctl01 snapd[1251]: AppArmor status: apparmor is enabled and all features are available
May 18 16:28:25 ctl01 snapd[1251]: helpers.go:145: error trying to compare the snap system key: system-key missing on disk
May 18 16:28:25 ctl01 snapd[1251]: daemon.go:379: started snapd/2.38 (series 16; classic) ubuntu/16.04 (amd64) linux/4.15.0-50-generic.
May 18 16:28:25 ctl01 systemd[1]: Started Snappy daemon.
May 18 16:28:25 ctl01 systemd[1]: Starting Wait until snapd is fully seeded...
May 18 16:28:25 ctl01 iscsid: iSCSI daemon with pid=1364 started!
May 18 16:28:25 ctl01 pollinate[1231]: client verified challenge/response with [https://entropy.ubuntu.com/]
May 18 16:28:25 ctl01 pollinate[1231]: client hashed response from [https://entropy.ubuntu.com/]
May 18 16:28:25 ctl01 pollinate[1231]: client successfully seeded [/dev/urandom]
May 18 16:28:25 ctl01 systemd[1]: Started Pollinate to seed the pseudo random number generator.
May 18 16:28:25 ctl01 systemd[1]: Starting OpenBSD Secure Shell server...
May 18 16:28:25 ctl01 systemd[1]: Started OpenBSD Secure Shell server.
May 18 16:28:25 ctl01 systemd[1]: Started The Salt Minion.
May 18 16:28:26 ctl01 systemd[1]: Started Wait until snapd is fully seeded.
May 18 16:28:26 ctl01 systemd[1]: Starting Apply the settings specified in cloud-config...
May 18 16:28:26 ctl01 systemd[1]: Reached target Multi-User System.
May 18 16:28:26 ctl01 systemd[1]: Reached target Graphical Interface.
May 18 16:28:26 ctl01 systemd[1]: Starting Update UTMP about System Runlevel Changes...
May 18 16:28:26 ctl01 systemd[1]: Started Update UTMP about System Runlevel Changes.
May 18 16:28:26 ctl01 salt-minion[1275]: [ERROR   ] DNS lookup or connection check of 'salt' failed.
May 18 16:28:26 ctl01 salt-minion[1275]: [ERROR   ] Master hostname: 'salt' not found or not responsive. Retrying in 30 seconds
May 18 16:28:26 ctl01 cloud-init[1691]: Generating locales (this might take a while)...
May 18 16:28:27 ctl01 cloud-init[1691]:   en_US.UTF-8... done
May 18 16:28:27 ctl01 cloud-init[1691]: Generation complete.
May 18 16:28:28 ctl01 cloud-init[1691]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'modules:config' at Sat, 18 May 2019 16:28:26 +0000. Up 16.41 seconds.
May 18 16:28:28 ctl01 systemd[1]: Started Apply the settings specified in cloud-config.
May 18 16:28:28 ctl01 systemd[1]: Starting Execute cloud user/final scripts...
May 18 16:28:28 ctl01 systemd[1]: Stopping The Salt Minion...
May 18 16:28:28 ctl01 salt-minion[1275]: [WARNING ] Minion received a SIGTERM. Exiting.
May 18 16:28:29 ctl01 salt-minion[1275]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited.
May 18 16:28:29 ctl01 systemd[1]: Stopped The Salt Minion.
May 18 16:28:29 ctl01 systemd[1]: Starting The Salt Minion...
May 18 16:28:29 ctl01 systemd[1]: Started The Salt Minion.
May 18 16:28:29 ctl01 ec2: 
May 18 16:28:29 ctl01 ec2: #############################################################
May 18 16:28:29 ctl01 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
May 18 16:28:29 ctl01 ec2: 1024 SHA256:/Fd9AcuHWhEJmN2GiYSQugT+0APdcYw826mtzT8/OPk root@ctl01 (DSA)
May 18 16:28:29 ctl01 ec2: 256 SHA256:eWKUgkAe7j2Ncxp/4leOHzHuNA3jccmKpDQxHyIf5UI root@ctl01 (ECDSA)
May 18 16:28:29 ctl01 ec2: 256 SHA256:nj+ThnCTL9CoJdUUdUC4klZFu+9nrkRiT/RbwAHI+Lg root@ctl01 (ED25519)
May 18 16:28:29 ctl01 ec2: 2048 SHA256:Ez6++5/VZl4797VJCi+Gf6qY4TqK45FLn3d6xuMMQEY root@ctl01 (RSA)
May 18 16:28:29 ctl01 ec2: -----END SSH HOST KEY FINGERPRINTS-----
May 18 16:28:29 ctl01 ec2: #############################################################
May 18 16:28:29 ctl01 cloud-init[1799]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'modules:final' at Sat, 18 May 2019 16:28:28 +0000. Up 18.31 seconds.
May 18 16:28:29 ctl01 cloud-init[1799]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 finished at Sat, 18 May 2019 16:28:29 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 19.36 seconds
May 18 16:28:29 ctl01 systemd[1]: Started Execute cloud user/final scripts.
May 18 16:28:29 ctl01 systemd[1]: Reached target Cloud-init target.
May 18 16:28:29 ctl01 systemd[1]: Startup finished in 3.383s (kernel) + 16.033s (userspace) = 19.416s.
May 18 16:28:30 ctl01 snapd[1251]: daemon.go:611: gracefully waiting for running hooks
May 18 16:28:30 ctl01 snapd[1251]: daemon.go:613: done waiting for running hooks
May 18 16:28:30 ctl01 snapd[1251]: daemon stop requested to wait for socket activation
May 18 16:28:45 ctl01 systemd[1]: Time has been changed
May 18 16:28:45 ctl01 systemd-timesyncd[663]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
May 18 16:29:01 ctl01 kernel: [   50.304850] random: crng init done
May 18 16:29:01 ctl01 kernel: [   50.304855] random: 7 urandom warning(s) missed due to ratelimiting
May 18 16:29:11 ctl01 salt-minion[1836]: [ERROR   ] Error while bringing up minion for multi-master. Is master at 192.168.11.2 responding?
May 18 16:34:24 ctl01 salt-minion[1836]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
May 18 16:34:29 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y remove telnet.
May 18 16:34:33 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install smartmontools.
May 18 16:34:35 ctl01 systemd[1]: Reloading.
May 18 16:34:35 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:36 ctl01 systemd[1]: Reloading.
May 18 16:34:36 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:36 ctl01 systemd[1]: Reloading.
May 18 16:34:36 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:36 ctl01 systemd[1]: Reloading.
May 18 16:34:36 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:36 ctl01 systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
May 18 16:34:36 ctl01 smartd[4132]: smartd 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-50-generic] (local build)
May 18 16:34:36 ctl01 smartd[4132]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
May 18 16:34:36 ctl01 smartd[4132]: Opened configuration file /etc/smartd.conf
May 18 16:34:36 ctl01 smartd[4132]: Drive: DEVICESCAN, implied '-a' Directive on line 21 of file /etc/smartd.conf
May 18 16:34:36 ctl01 smartd[4132]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
May 18 16:34:36 ctl01 smartd[4132]: Problem creating device name scan list
May 18 16:34:36 ctl01 smartd[4132]: In the system's table of devices NO devices found to scan
May 18 16:34:36 ctl01 smartd[4132]: Unable to monitor any SMART enabled devices. Try debug (-d) option. Exiting...
May 18 16:34:36 ctl01 systemd[1]: smartd.service: Main process exited, code=exited, status=17/n/a
May 18 16:34:36 ctl01 systemd[1]: smartd.service: Unit entered failed state.
May 18 16:34:36 ctl01 systemd[1]: smartd.service: Failed with result 'exit-code'.
May 18 16:34:36 ctl01 systemd[1]: Reloading.
May 18 16:34:36 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:36 ctl01 systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
May 18 16:34:36 ctl01 smartd[4176]: smartd 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-50-generic] (local build)
May 18 16:34:36 ctl01 smartd[4176]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
May 18 16:34:36 ctl01 smartd[4176]: Opened configuration file /etc/smartd.conf
May 18 16:34:36 ctl01 smartd[4176]: Drive: DEVICESCAN, implied '-a' Directive on line 21 of file /etc/smartd.conf
May 18 16:34:36 ctl01 smartd[4176]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
May 18 16:34:36 ctl01 smartd[4176]: Problem creating device name scan list
May 18 16:34:36 ctl01 smartd[4176]: In the system's table of devices NO devices found to scan
May 18 16:34:36 ctl01 smartd[4176]: Unable to monitor any SMART enabled devices. Try debug (-d) option. Exiting...
May 18 16:34:36 ctl01 systemd[1]: smartd.service: Main process exited, code=exited, status=17/n/a
May 18 16:34:36 ctl01 systemd[1]: smartd.service: Unit entered failed state.
May 18 16:34:36 ctl01 systemd[1]: smartd.service: Failed with result 'exit-code'.
May 18 16:34:36 ctl01 systemd[1]: Reloading.
May 18 16:34:37 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:38 ctl01 dbus[1181]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
May 18 16:34:38 ctl01 systemd[1]: Starting Time & Date Service...
May 18 16:34:38 ctl01 dbus[1181]: [system] Successfully activated service 'org.freedesktop.timedate1'
May 18 16:34:38 ctl01 systemd[1]: Started Time & Date Service.
May 18 16:34:39 ctl01 salt-minion[1836]: [WARNING ] State for file: /boot/grub/grub.cfg - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:40 ctl01 kernel: [  389.169633] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
May 18 16:34:44 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install sysfsutils.
May 18 16:34:45 ctl01 systemd[1]: Reloading.
May 18 16:34:46 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:46 ctl01 systemd[1]: Reloading.
May 18 16:34:46 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:46 ctl01 systemd[1]: Reloading.
May 18 16:34:46 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:46 ctl01 systemd[1]: Starting LSB: Set sysfs variables from /etc/sysfs.conf...
May 18 16:34:46 ctl01 sysfsutils[5135]:  * Setting sysfs variables...
May 18 16:34:46 ctl01 sysfsutils[5135]:    ...done.
May 18 16:34:46 ctl01 systemd[1]: Started LSB: Set sysfs variables from /etc/sysfs.conf.
May 18 16:34:47 ctl01 systemd[1]: Reloading.
May 18 16:34:47 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:48 ctl01 systemd[1]: Started /bin/systemctl stop ondemand.service.
May 18 16:34:48 ctl01 systemd[1]: Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"...
May 18 16:34:48 ctl01 systemd[1]: Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand".
May 18 16:34:49 ctl01 systemd[1]: Started /usr/sbin/update-rc.d -f ondemand remove.
May 18 16:34:49 ctl01 systemd[1]: Reloading.
May 18 16:34:49 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:49 ctl01 dbus[1181]: [system] Activating via systemd: service name='org.freedesktop.locale1' unit='dbus-org.freedesktop.locale1.service'
May 18 16:34:49 ctl01 systemd[1]: Starting Locale Service...
May 18 16:34:49 ctl01 dbus[1181]: [system] Successfully activated service 'org.freedesktop.locale1'
May 18 16:34:49 ctl01 systemd[1]: Started Locale Service.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
May 18 16:34:49 ctl01 systemd[1]: Reloading.
May 18 16:34:49 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/shadow - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/gshadow - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/group- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/group - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/passwd- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/passwd - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/gshadow- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:49 ctl01 salt-minion[1836]: [WARNING ] State for file: /etc/shadow- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:34:50 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install openvswitch-switch bridge-utils.
May 18 16:34:52 ctl01 systemd[1]: Reloading.
May 18 16:34:52 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:53 ctl01 systemd[1]: Reloading.
May 18 16:34:53 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:53 ctl01 systemd[1]: Reloading.
May 18 16:34:53 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:53 ctl01 systemd[1]: Reloading.
May 18 16:34:53 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:53 ctl01 systemd[1]: Starting Open vSwitch Internal Unit...
May 18 16:34:53 ctl01 ovs-ctl[6147]:  * /etc/openvswitch/conf.db does not exist
May 18 16:34:53 ctl01 ovs-ctl[6147]:  * Creating empty database /etc/openvswitch/conf.db
May 18 16:34:53 ctl01 ovs-ctl[6147]:  * Starting ovsdb-server
May 18 16:34:53 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1
May 18 16:34:53 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.5.5 "external-ids:system-id=\"6373cd72-d76d-4de9-8e9c-6557c19a2afa\"" "system-type=\"Ubuntu\"" "system-version=\"16.04-xenial\""
May 18 16:34:53 ctl01 ovs-ctl[6147]:  * Configuring Open vSwitch system IDs
May 18 16:34:53 ctl01 ovs-ctl[6147]:  * Inserting openvswitch module
May 18 16:34:53 ctl01 kernel: [  402.769024] openvswitch: Open vSwitch switching datapath
May 18 16:34:53 ctl01 ovs-ctl[6147]:  * Starting ovs-vswitchd
May 18 16:34:53 ctl01 ovs-ctl[6147]:  * Enabling remote OVSDB managers
May 18 16:34:53 ctl01 systemd[1]: Started Open vSwitch Internal Unit.
May 18 16:34:53 ctl01 systemd[1]: Starting Open vSwitch...
May 18 16:34:53 ctl01 systemd[1]: Started Open vSwitch.
May 18 16:34:53 ctl01 systemd[1]: Reloading.
May 18 16:34:53 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:54 ctl01 systemd[1]: Reloading.
May 18 16:34:54 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:57 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install bridge-utils.
May 18 16:34:58 ctl01 systemd[1]: Reloading.
May 18 16:34:58 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:58 ctl01 systemd[1]: Started /bin/systemctl enable networking.service.
May 18 16:34:58 ctl01 systemd[1]: Reloading.
May 18 16:34:58 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:58 ctl01 systemd[1]: Reloading.
May 18 16:34:58 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:58 ctl01 systemd[1]: Reloading.
May 18 16:34:58 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:34:58 ctl01 salt-minion[1836]: [WARNING ] The network state sls is requiring a reboot of the system to properly apply network configuration.
May 18 16:34:58 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install vlan.
May 18 16:34:59 ctl01 kernel: [  408.279436] 8021q: 802.1Q VLAN Support v1.8
May 18 16:34:59 ctl01 kernel: [  408.279449] 8021q: adding VLAN 0 to HW filter on device ens3
May 18 16:34:59 ctl01 kernel: [  408.287913] 8021q: adding VLAN 0 to HW filter on device ens5
May 18 16:34:59 ctl01 systemd[1]: Reloading OpenBSD Secure Shell server.
May 18 16:34:59 ctl01 systemd[1]: Reloaded OpenBSD Secure Shell server.
May 18 16:34:59 ctl01 systemd[1]: Started ifup for ens5.1000.
May 18 16:34:59 ctl01 systemd[1]: Found device /sys/subsystem/net/devices/ens5.1000.
May 18 16:34:59 ctl01 sh[6848]: ifup: waiting for lock on /run/network/ifstate.ens5
May 18 16:34:59 ctl01 sh[6848]: ifup: interface ens5.1000 already configured
May 18 16:35:00 ctl01 salt-minion[1836]: [ERROR   ] Command '['umount', '/dev/shm']' failed with return code: 32
May 18 16:35:00 ctl01 salt-minion[1836]: [ERROR   ] stderr: umount: /dev/shm: target is busy
May 18 16:35:00 ctl01 salt-minion[1836]:         (In some cases useful info about processes that
May 18 16:35:00 ctl01 salt-minion[1836]:          use the device is found by lsof(8) or fuser(1).)
May 18 16:35:00 ctl01 salt-minion[1836]: [ERROR   ] retcode: 32
May 18 16:35:01 ctl01 systemd[1]: Stopped target Cloud-init target.
May 18 16:35:01 ctl01 systemd[1]: Stopping Time & Date Service...
May 18 16:35:01 ctl01 systemd[1]: Stopped Execute cloud user/final scripts.
May 18 16:35:01 ctl01 systemd[1]: Stopped Apply the settings specified in cloud-config.
May 18 16:35:01 ctl01 systemd[1]: Stopped target Cloud-config availability.
May 18 16:35:01 ctl01 systemd[1]: Stopping Locale Service...
May 18 16:35:01 ctl01 systemd[1]: Stopped target Graphical Interface.
May 18 16:35:01 ctl01 systemd[1]: Stopping Accounts Service...
May 18 16:35:01 ctl01 systemd[1]: Stopped target Multi-User System.
May 18 16:35:01 ctl01 systemd[1]: Stopping LXD - container startup/shutdown...
May 18 16:35:01 ctl01 systemd[1]: Stopping Regular background program processing daemon...
May 18 16:35:01 ctl01 rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="1309" x-info="http://www.rsyslog.com"] exiting on signal 15.
May 18 16:35:19 ctl01 rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="2034" x-info="http://www.rsyslog.com"] start
May 18 16:35:19 ctl01 rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
May 18 16:35:19 ctl01 rsyslogd: rsyslogd's groupid changed to 108
May 18 16:35:19 ctl01 systemd[1]: Started LVM2 metadata daemon.
May 18 16:35:19 ctl01 rsyslogd: rsyslogd's userid changed to 104
May 18 16:35:19 ctl01 systemd[1]: Starting udev Coldplug all Devices...
May 18 16:35:19 ctl01 systemd[1]: Starting Load/Save Random Seed...
May 18 16:35:19 ctl01 systemd-modules-load[456]: Inserted module 'iscsi_tcp'
May 18 16:35:19 ctl01 systemd[1]: Starting Flush Journal to Persistent Storage...
May 18 16:35:19 ctl01 systemd[1]: Started Load/Save Random Seed.
May 18 16:35:19 ctl01 loadkeys[450]: Loading /etc/console-setup/cached.kmap.gz
May 18 16:35:19 ctl01 systemd[1]: Started Set console keymap.
May 18 16:35:19 ctl01 systemd[1]: Started Create Static Device Nodes in /dev.
May 18 16:35:19 ctl01 systemd[1]: Starting udev Kernel Device Manager...
May 18 16:35:19 ctl01 systemd[1]: Started Flush Journal to Persistent Storage.
May 18 16:35:19 ctl01 systemd-modules-load[456]: Inserted module 'ib_iser'
May 18 16:35:19 ctl01 systemd[1]: Started udev Coldplug all Devices.
May 18 16:35:19 ctl01 kernel: [    0.000000] Linux version 4.15.0-50-generic (buildd@lgw01-amd64-029) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) #54~16.04.1-Ubuntu SMP Wed May 8 15:55:19 UTC 2019 (Ubuntu 4.15.0-50.54~16.04.1-generic 4.15.18)
May 18 16:35:19 ctl01 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-50-generic root=UUID=1bbe14b0-e1b4-4636-83ed-ba0fa2a5acca ro console=tty1 console=ttyS0
May 18 16:35:19 ctl01 kernel: [    0.000000] KERNEL supported cpus:
May 18 16:35:19 ctl01 kernel: [    0.000000]   Intel GenuineIntel
May 18 16:35:19 ctl01 kernel: [    0.000000]   AMD AuthenticAMD
May 18 16:35:19 ctl01 kernel: [    0.000000]   Centaur CentaurHauls
May 18 16:35:19 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
May 18 16:35:19 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
May 18 16:35:19 ctl01 systemd-modules-load[456]: Inserted module 'nf_conntrack'
May 18 16:35:19 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
May 18 16:35:19 ctl01 kernel: [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
May 18 16:35:19 ctl01 kernel: [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
May 18 16:35:19 ctl01 kernel: [    0.000000] e820: BIOS-provided physical RAM map:
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
May 18 16:35:19 ctl01 systemd[1]: Started Load Kernel Modules.
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
May 18 16:35:19 ctl01 systemd[1]: Mounting Configuration File System...
May 18 16:35:19 ctl01 systemd[1]: Starting Apply Kernel Variables...
May 18 16:35:19 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000003bfffffff] usable
May 18 16:35:19 ctl01 kernel: [    0.000000] NX (Execute Disable) protection: active
May 18 16:35:19 ctl01 kernel: [    0.000000] SMBIOS 2.8 present.
May 18 16:35:19 ctl01 kernel: [    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
May 18 16:35:19 ctl01 kernel: [    0.000000] Hypervisor detected: KVM
May 18 16:35:19 ctl01 rsyslogd-2039: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
May 18 16:35:19 ctl01 kernel: [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
May 18 16:35:19 ctl01 kernel: [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
May 18 16:35:19 ctl01 kernel: [    0.000000] e820: last_pfn = 0x3c0000 max_arch_pfn = 0x400000000
May 18 16:35:19 ctl01 kernel: [    0.000000] MTRR default type: write-back
May 18 16:35:19 ctl01 systemd[1]: Mounting FUSE Control File System...
May 18 16:35:19 ctl01 systemd[1]: Mounted Configuration File System.
May 18 16:35:19 ctl01 systemd[1]: Mounted FUSE Control File System.
May 18 16:35:19 ctl01 rsyslogd-2007: action 'action 11' suspended, next retry is Sat May 18 16:35:49 2019 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
May 18 16:35:19 ctl01 systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
May 18 16:35:19 ctl01 systemd[1]: Started udev Kernel Device Manager.
May 18 16:35:19 ctl01 systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
May 18 16:35:19 ctl01 systemd[1]: Reached target Local File Systems (Pre).
May 18 16:35:19 ctl01 systemd[1]: Reached target Local File Systems.
May 18 16:35:19 ctl01 systemd[1]: Starting Initial cloud-init job (pre-networking)...
May 18 16:35:19 ctl01 systemd[1]: Starting Create Volatile Files and Directories...
May 18 16:35:19 ctl01 systemd[1]: Starting LSB: AppArmor initialization...
May 18 16:35:19 ctl01 systemd-tmpfiles[548]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
May 18 16:35:19 ctl01 systemd[1]: Starting Set console font and keymap...
May 18 16:35:19 ctl01 systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
May 18 16:35:19 ctl01 systemd[1]: Started Apply Kernel Variables.
May 18 16:35:19 ctl01 systemd[1]: Started Tell Plymouth To Write Out Runtime Data.
May 18 16:35:19 ctl01 systemd[1]: Found device /dev/ttyS0.
May 18 16:35:19 ctl01 apparmor[549]:  * Starting AppArmor profiles
May 18 16:35:19 ctl01 systemd[1]: Started Create Volatile Files and Directories.
May 18 16:35:19 ctl01 systemd[1]: Starting Network Time Synchronization...
May 18 16:35:19 ctl01 systemd[1]: Starting Update UTMP about System Boot/Shutdown...
May 18 16:35:19 ctl01 systemd[1]: Started Network Time Synchronization.
May 18 16:35:19 ctl01 systemd[1]: Started Update UTMP about System Boot/Shutdown.
May 18 16:35:19 ctl01 systemd[1]: Reached target System Time Synchronized.
May 18 16:35:19 ctl01 systemd[1]: Started Set console font and keymap.
May 18 16:35:19 ctl01 systemd[1]: Created slice system-getty.slice.
May 18 16:35:19 ctl01 systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
May 18 16:35:19 ctl01 systemd-udevd[523]: Could not generate persistent MAC address for br-mgmt: No such file or directory
May 18 16:35:19 ctl01 systemd[1]: Found device Virtio network device.
May 18 16:35:19 ctl01 systemd[1]: Found device Virtio network device.
May 18 16:35:19 ctl01 systemd[1]: Found device /sys/subsystem/net/devices/br-mgmt.
May 18 16:35:19 ctl01 systemd[1]: Found device Virtio network device.
May 18 16:35:19 ctl01 apparmor[549]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
May 18 16:35:19 ctl01 apparmor[549]:    ...done.
May 18 16:35:19 ctl01 systemd[1]: Started LSB: AppArmor initialization.
May 18 16:35:19 ctl01 cloud-init[540]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'init-local' at Sat, 18 May 2019 16:35:15 +0000. Up 8.43 seconds.
May 18 16:35:19 ctl01 systemd[1]: Started Initial cloud-init job (pre-networking).
May 18 16:35:19 ctl01 systemd[1]: Reached target Network (Pre).
May 18 16:35:19 ctl01 systemd[1]: Started ifup for ens4.
May 18 16:35:19 ctl01 systemd[1]: Started ifup for ens5.
May 18 16:35:19 ctl01 systemd[1]: Starting Raise network interfaces...
May 18 16:35:19 ctl01 systemd[1]: Started ifup for ens3.
May 18 16:35:19 ctl01 systemd[1]: Started ifup for br-mgmt.
May 18 16:35:19 ctl01 sh[1169]: Waiting for br-mgmt to get ready (MAXWAIT is 32 seconds).
May 18 16:35:19 ctl01 sh[1137]: WARNING:  Could not open /proc/net/vlan/config.  Maybe you need to load the 8021q module, or maybe you are not using PROCFS??
May 18 16:35:19 ctl01 ifup[1220]: /sbin/ifup: waiting for lock on /run/network/ifstate.ens5
May 18 16:35:19 ctl01 sh[1137]: Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
May 18 16:35:19 ctl01 sh[1137]: Added VLAN with VID == 1000 to IF -:ens5:-
May 18 16:35:19 ctl01 ifup[1220]: /sbin/ifup: waiting for lock on /run/network/ifstate.br-mgmt
May 18 16:35:19 ctl01 systemd[1]: Started ifup for ens5.1000.
May 18 16:35:19 ctl01 systemd[1]: Found device /sys/subsystem/net/devices/ens5.1000.
May 18 16:35:19 ctl01 sh[1578]: Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
May 18 16:35:19 ctl01 systemd[1]: Started Raise network interfaces.
May 18 16:35:19 ctl01 systemd[1]: Reached target Network.
May 18 16:35:19 ctl01 systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
May 18 16:35:19 ctl01 cloud-init[1861]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'init' at Sat, 18 May 2019 16:35:17 +0000. Up 9.51 seconds.
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: ++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-----------+-------+----------------------------+---------------+--------+-------------------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   Device  |   Up  |          Address           |      Mask     | Scope  |     Hw-Address    |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-----------+-------+----------------------------+---------------+--------+-------------------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |  br-mgmt  |  True |        172.16.10.36        | 255.255.255.0 | global | 52:54:00:f8:33:c0 |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |  br-mgmt  |  True | fe80::5054:ff:fef8:33c0/64 |       .       |  link  | 52:54:00:f8:33:c0 |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |    ens3   |  True |       192.168.11.21        | 255.255.255.0 | global | 52:54:00:9d:ad:df |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |    ens3   |  True | fe80::5054:ff:fe9d:addf/64 |       .       |  link  | 52:54:00:9d:ad:df |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |    ens4   |  True | fe80::5054:ff:fef8:33c0/64 |       .       |  link  | 52:54:00:f8:33:c0 |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |    ens5   |  True | fe80::5054:ff:fe60:dbe8/64 |       .       |  link  | 52:54:00:60:db:e8 |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: | ens5.1000 |  True | fe80::5054:ff:fe60:dbe8/64 |       .       |  link  | 52:54:00:60:db:e8 |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |    ens6   | False |             .              |       .       |   .    | 52:54:00:8b:60:5a |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |     lo    |  True |         127.0.0.1          |   255.0.0.0   |  host  |         .         |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |     lo    |  True |          ::1/128           |       .       |  host  |         .         |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-----------+-------+----------------------------+---------------+--------+-------------------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   0   |   0.0.0.0    | 192.168.11.3 |    0.0.0.0    |    ens3   |   UG  |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   1   |  10.254.0.0  | 172.16.10.36 |  255.255.0.0  |  br-mgmt  |   UG  |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   2   | 172.16.10.0  |   0.0.0.0    | 255.255.255.0 |  br-mgmt  |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   3   | 192.168.11.0 |   0.0.0.0    | 255.255.255.0 |    ens3   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-------+-------------+---------+-----------+-------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: | Route | Destination | Gateway | Interface | Flags |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-------+-------------+---------+-----------+-------+
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   0   |  fe80::/64  |    ::   |    ens4   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   1   |  fe80::/64  |    ::   |  br-mgmt  |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   2   |  fe80::/64  |    ::   |    ens5   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   3   |  fe80::/64  |    ::   |    ens3   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   4   |  fe80::/64  |    ::   | ens5.1000 |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   6   |    local    |    ::   |    ens4   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   7   |   ff00::/8  |    ::   |    ens4   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   8   |   ff00::/8  |    ::   |  br-mgmt  |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   9   |   ff00::/8  |    ::   |    ens5   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   10  |   ff00::/8  |    ::   |    ens3   |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: |   11  |   ff00::/8  |    ::   | ens5.1000 |   U   |
May 18 16:35:19 ctl01 cloud-init[1861]: ci-info: +-------+-------------+---------+-----------+-------+
May 18 16:35:19 ctl01 systemd[1]: Started Initial cloud-init job (metadata service crawler).
May 18 16:35:19 ctl01 systemd[1]: Reached target Network is Online.
May 18 16:35:19 ctl01 systemd[1]: Starting iSCSI initiator daemon (iscsid)...
May 18 16:35:19 ctl01 systemd[1]: Reached target System Initialization.
May 18 16:35:19 ctl01 systemd[1]: Starting LXD - unix socket.
May 18 16:35:19 ctl01 systemd[1]: Started Daily Cleanup of Temporary Directories.
May 18 16:35:19 ctl01 systemd[1]: Starting Socket activation for snappy daemon.
May 18 16:35:19 ctl01 systemd[1]: Started ACPI Events Check.
May 18 16:35:19 ctl01 systemd[1]: Reached target Paths.
May 18 16:35:19 ctl01 systemd[1]: Listening on UUID daemon activation socket.
May 18 16:35:19 ctl01 systemd[1]: Started Daily apt download activities.
May 18 16:35:19 ctl01 systemd[1]: Listening on ACPID Listen Socket.
May 18 16:35:19 ctl01 systemd[1]: Started Daily apt upgrade and clean activities.
May 18 16:35:19 ctl01 systemd[1]: Reached target Timers.
May 18 16:35:19 ctl01 systemd[1]: Listening on D-Bus System Message Bus Socket.
May 18 16:35:19 ctl01 systemd[1]: Reached target Cloud-config availability.
May 18 16:35:19 ctl01 systemd[1]: Listening on LXD - unix socket.
May 18 16:35:19 ctl01 systemd[1]: Listening on Socket activation for snappy daemon.
May 18 16:35:19 ctl01 systemd[1]: Reached target Sockets.
May 18 16:35:19 ctl01 systemd[1]: Reached target Basic System.
May 18 16:35:19 ctl01 systemd[1]: Starting LSB: Set sysfs variables from /etc/sysfs.conf...
May 18 16:35:19 ctl01 systemd[1]: Started Deferred execution scheduler.
May 18 16:35:19 ctl01 systemd[1]: Starting Accounts Service...
May 18 16:35:19 ctl01 systemd[1]: Started D-Bus System Message Bus.
May 18 16:35:19 ctl01 iscsid: iSCSI logger with pid=1968 started!
May 18 16:35:19 ctl01 dbus[1959]: [system] AppArmor D-Bus mediation is enabled
May 18 16:35:19 ctl01 systemd[1]: Starting LSB: Record successful boot for GRUB...
May 18 16:35:19 ctl01 systemd[1]: Starting Thermal Daemon Service...
May 18 16:35:19 ctl01 systemd[1]: Started FUSE filesystem for LXC.
May 18 16:35:19 ctl01 systemd[1]: Started Unattended Upgrades Shutdown.
May 18 16:35:19 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:35:19 ctl01 systemd[1]: Starting OpenBSD Secure Shell server...
May 18 16:35:19 ctl01 dbus[1959]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkitd.service'
May 18 16:35:19 ctl01 systemd[1]: Starting /etc/rc.local Compatibility...
May 18 16:35:19 ctl01 systemd[1]: Starting System Logging Service...
May 18 16:35:19 ctl01 systemd[1]: Starting LSB: MD monitoring daemon...
May 18 16:35:19 ctl01 systemd[1]: Starting Open vSwitch Internal Unit...
May 18 16:35:19 ctl01 systemd[1]: Starting Login Service...
May 18 16:35:19 ctl01 systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
May 18 16:35:19 ctl01 systemd[1]: Started Regular background program processing daemon.
May 18 16:35:19 ctl01 lxcfs[2004]: hierarchies:
May 18 16:35:19 ctl01 lxcfs[2004]:   0: fd:   5: memory
May 18 16:35:19 ctl01 lxcfs[2004]:   1: fd:   6: devices
May 18 16:35:19 ctl01 lxcfs[2004]:   2: fd:   7: net_cls,net_prio
May 18 16:35:19 ctl01 lxcfs[2004]:   3: fd:   8: pids
May 18 16:35:19 ctl01 lxcfs[2004]:   4: fd:   9: perf_event
May 18 16:35:19 ctl01 lxcfs[2004]:   5: fd:  10: cpu,cpuacct
May 18 16:35:19 ctl01 lxcfs[2004]:   6: fd:  11: rdma
May 18 16:35:19 ctl01 lxcfs[2004]:   7: fd:  12: blkio
May 18 16:35:19 ctl01 lxcfs[2004]:   8: fd:  13: cpuset
May 18 16:35:19 ctl01 lxcfs[2004]:   9: fd:  14: hugetlb
May 18 16:35:19 ctl01 lxcfs[2004]:  10: fd:  15: freezer
May 18 16:35:19 ctl01 lxcfs[2004]:  11: fd:  16: name=systemd
May 18 16:35:19 ctl01 systemd[1]: Starting Snappy daemon...
May 18 16:35:19 ctl01 cron[2085]: (CRON) INFO (pidfile fd = 3)
May 18 16:35:19 ctl01 systemd[1]: Starting The Salt Minion...
May 18 16:35:19 ctl01 systemd[1]: Starting LXD - container startup/shutdown...
May 18 16:35:19 ctl01 systemd[1]: Started iSCSI initiator daemon (iscsid).
May 18 16:35:19 ctl01 systemd[1]: Started /etc/rc.local Compatibility.
May 18 16:35:19 ctl01 sysfsutils[1944]:  * Setting sysfs variables...
May 18 16:35:19 ctl01 mdadm[2042]:  * Starting MD monitoring service mdadm --monitor
May 18 16:35:19 ctl01 acpid: starting up with netlink and the input layer
May 18 16:35:19 ctl01 acpid: 1 rule loaded
May 18 16:35:19 ctl01 acpid: waiting for events: event logging is off
May 18 16:35:19 ctl01 systemd[1]: Starting Authenticate and Authorize Users to Run Privileged Tasks...
May 18 16:35:19 ctl01 systemd[1]: Starting Login to default iSCSI targets...
May 18 16:35:19 ctl01 systemd[1]: Started Login Service.
May 18 16:35:19 ctl01 sysfsutils[1944]:    ...done.
May 18 16:35:19 ctl01 systemd[1]: Started LSB: Set sysfs variables from /etc/sysfs.conf.
May 18 16:35:19 ctl01 cron[2085]: (CRON) INFO (Running @reboot jobs)
May 18 16:35:19 ctl01 systemd[1]: Started Thermal Daemon Service.
May 18 16:35:19 ctl01 mdadm[2042]:    ...done.
May 18 16:35:19 ctl01 systemd[1]: Started LSB: MD monitoring daemon.
May 18 16:35:19 ctl01 thermald[1996]: NO RAPL sysfs present
May 18 16:35:19 ctl01 thermald[1996]: 13 CPUID levels; family:model:stepping 0x6:3e:4 (6:62:4)
May 18 16:35:19 ctl01 thermald[1996]:  Need Linux PowerCap sysfs
May 18 16:35:19 ctl01 iscsiadm[2169]: iscsiadm: No records found
May 18 16:35:19 ctl01 systemd[1]: Started System Logging Service.
May 18 16:35:19 ctl01 kernel: [    0.000000] MTRR fixed ranges enabled:
May 18 16:35:19 ctl01 kernel: [    0.000000]   00000-9FFFF write-back
May 18 16:35:19 ctl01 kernel: [    0.000000]   A0000-BFFFF uncachable
May 18 16:35:19 ctl01 kernel: [    0.000000]   C0000-FFFFF write-protect
May 18 16:35:19 ctl01 kernel: [    0.000000] MTRR variable ranges enabled:
May 18 16:35:19 ctl01 kernel: [    0.000000]   0 base 00C0000000 mask FFC0000000 uncachable
May 18 16:35:19 ctl01 kernel: [    0.000000]   1 disabled
May 18 16:35:19 ctl01 kernel: [    0.000000]   2 disabled
May 18 16:35:19 ctl01 kernel: [    0.000000]   3 disabled
May 18 16:35:19 ctl01 kernel: [    0.000000]   4 disabled
May 18 16:35:19 ctl01 kernel: [    0.000000]   5 disabled
May 18 16:35:19 ctl01 kernel: [    0.000000]   6 disabled
May 18 16:35:19 ctl01 kernel: [    0.000000]   7 disabled
May 18 16:35:19 ctl01 kernel: [    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
May 18 16:35:19 ctl01 kernel: [    0.000000] e820: last_pfn = 0xbffdf max_arch_pfn = 0x400000000
May 18 16:35:19 ctl01 kernel: [    0.000000] found SMP MP-table at [mem 0x000f6560-0x000f656f] mapped at [        (ptrval)]
May 18 16:35:19 ctl01 kernel: [    0.000000] Scanning 1 areas for low memory corruption
May 18 16:35:19 ctl01 kernel: [    0.000000] Base memory trampoline at [        (ptrval)] 99000 size 24576
May 18 16:35:19 ctl01 kernel: [    0.000000] Using GB pages for direct mapping
May 18 16:35:19 ctl01 kernel: [    0.000000] BRK [0x16db3f000, 0x16db3ffff] PGTABLE
May 18 16:35:19 ctl01 kernel: [    0.000000] BRK [0x16db40000, 0x16db40fff] PGTABLE
May 18 16:35:19 ctl01 kernel: [    0.000000] BRK [0x16db41000, 0x16db41fff] PGTABLE
May 18 16:35:19 ctl01 kernel: [    0.000000] BRK [0x16db42000, 0x16db42fff] PGTABLE
May 18 16:35:19 ctl01 kernel: [    0.000000] BRK [0x16db43000, 0x16db43fff] PGTABLE
May 18 16:35:19 ctl01 kernel: [    0.000000] BRK [0x16db44000, 0x16db44fff] PGTABLE
May 18 16:35:19 ctl01 kernel: [    0.000000] RAMDISK: [mem 0x3177e000-0x34bb6fff]
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: Early table checksum verification disabled
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: RSDP 0x00000000000F6510 000014 (v00 BOCHS )
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: RSDT 0x00000000BFFE14CC 000030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: FACP 0x00000000BFFE0854 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: DSDT 0x00000000BFFDFC80 000BD4 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: FACS 0x00000000BFFDFC40 000040
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: SSDT 0x00000000BFFE08C8 000B54 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: APIC 0x00000000BFFE141C 0000B0 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
May 18 16:35:19 ctl01 kernel: [    0.000000] No NUMA configuration found
May 18 16:35:19 ctl01 kernel: [    0.000000] Faking a node at [mem 0x0000000000000000-0x00000003bfffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] NODE_DATA(0) allocated [mem 0x3bffd5000-0x3bfffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] kvm-clock: cpu 0, msr 3:bff54001, primary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
May 18 16:35:19 ctl01 kernel: [    0.000000] kvm-clock: using sched offset of 416695675705 cycles
May 18 16:35:19 ctl01 kernel: [    0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
May 18 16:35:19 ctl01 kernel: [    0.000000] Zone ranges:
May 18 16:35:19 ctl01 kernel: [    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000]   Normal   [mem 0x0000000100000000-0x00000003bfffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000]   Device   empty
May 18 16:35:19 ctl01 kernel: [    0.000000] Movable zone start for each node
May 18 16:35:19 ctl01 kernel: [    0.000000] Early memory node ranges
May 18 16:35:19 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
May 18 16:35:19 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000100000-0x00000000bffdefff]
May 18 16:35:19 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000100000000-0x00000003bfffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] Reserved but unavailable: 131 pages
May 18 16:35:19 ctl01 kernel: [    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000003bfffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] On node 0 totalpages: 3669885
May 18 16:35:19 ctl01 kernel: [    0.000000]   DMA zone: 64 pages used for memmap
May 18 16:35:19 ctl01 kernel: [    0.000000]   DMA zone: 21 pages reserved
May 18 16:35:19 ctl01 kernel: [    0.000000]   DMA zone: 3998 pages, LIFO batch:0
May 18 16:35:19 ctl01 kernel: [    0.000000]   DMA32 zone: 12224 pages used for memmap
May 18 16:35:19 ctl01 kernel: [    0.000000]   DMA32 zone: 782303 pages, LIFO batch:31
May 18 16:35:19 ctl01 kernel: [    0.000000]   Normal zone: 45056 pages used for memmap
May 18 16:35:19 ctl01 kernel: [    0.000000]   Normal zone: 2883584 pages, LIFO batch:31
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: PM-Timer IO Port: 0x608
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
May 18 16:35:19 ctl01 kernel: [    0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: IRQ0 used by override.
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: IRQ5 used by override.
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: IRQ9 used by override.
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: IRQ10 used by override.
May 18 16:35:19 ctl01 kernel: [    0.000000] ACPI: IRQ11 used by override.
May 18 16:35:19 ctl01 kernel: [    0.000000] Using ACPI (MADT) for SMP configuration information
May 18 16:35:19 ctl01 kernel: [    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xbffdf000-0xbfffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
May 18 16:35:19 ctl01 kernel: [    0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices
May 18 16:35:19 ctl01 kernel: [    0.000000] Booting paravirtualized kernel on KVM
May 18 16:35:19 ctl01 kernel: [    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
May 18 16:35:19 ctl01 kernel: [    0.000000] random: get_random_bytes called from start_kernel+0x99/0x51b with crng_init=0
May 18 16:35:19 ctl01 kernel: [    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
May 18 16:35:19 ctl01 kernel: [    0.000000] percpu: Embedded 46 pages/cpu @        (ptrval) s151552 r8192 d28672 u262144
May 18 16:35:19 ctl01 kernel: [    0.000000] pcpu-alloc: s151552 r8192 d28672 u262144 alloc=1*2097152
May 18 16:35:19 ctl01 kernel: [    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
May 18 16:35:19 ctl01 kernel: [    0.000000] KVM setup async PF for cpu 0
May 18 16:35:19 ctl01 kernel: [    0.000000] kvm-stealtime: cpu 0, msr 3bfc24040
May 18 16:35:19 ctl01 kernel: [    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
May 18 16:35:19 ctl01 kernel: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 3612520
May 18 16:35:19 ctl01 kernel: [    0.000000] Policy zone: Normal
May 18 16:35:19 ctl01 kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-50-generic root=UUID=1bbe14b0-e1b4-4636-83ed-ba0fa2a5acca ro console=tty1 console=ttyS0
May 18 16:35:19 ctl01 kernel: [    0.000000] Calgary: detecting Calgary via BIOS EBDA area
May 18 16:35:19 ctl01 kernel: [    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
May 18 16:35:19 ctl01 kernel: [    0.000000] Memory: 14300480K/14679540K available (12300K kernel code, 2473K rwdata, 4288K rodata, 2408K init, 2416K bss, 379060K reserved, 0K cma-reserved)
May 18 16:35:19 ctl01 kernel: [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
May 18 16:35:19 ctl01 kernel: [    0.000000] Kernel/User page tables isolation: enabled
May 18 16:35:19 ctl01 kernel: [    0.000000] ftrace: allocating 39241 entries in 154 pages
May 18 16:35:19 ctl01 kernel: [    0.004000] Hierarchical RCU implementation.
May 18 16:35:19 ctl01 kernel: [    0.004000] 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=8.
May 18 16:35:19 ctl01 kernel: [    0.004000] 	Tasks RCU enabled.
May 18 16:35:19 ctl01 kernel: [    0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
May 18 16:35:19 ctl01 kernel: [    0.004000] NR_IRQS: 524544, nr_irqs: 488, preallocated irqs: 16
May 18 16:35:19 ctl01 kernel: [    0.004000] Console: colour VGA+ 80x25
May 18 16:35:19 ctl01 kernel: [    0.004000] console [tty1] enabled
May 18 16:35:19 ctl01 kernel: [    0.004000] console [ttyS0] enabled
May 18 16:35:19 ctl01 kernel: [    0.004000] ACPI: Core revision 20170831
May 18 16:35:19 ctl01 kernel: [    0.004000] ACPI: 2 ACPI AML tables successfully acquired and loaded
May 18 16:35:19 ctl01 kernel: [    0.004008] APIC: Switch to symmetric I/O mode setup
May 18 16:35:19 ctl01 kernel: [    0.005405] x2apic enabled
May 18 16:35:19 ctl01 kernel: [    0.006406] Switched APIC routing to physical x2apic.
May 18 16:35:19 ctl01 kernel: [    0.008887] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
May 18 16:35:19 ctl01 kernel: [    0.010189] tsc: Detected 2799.994 MHz processor
May 18 16:35:19 ctl01 kernel: [    0.011199] Calibrating delay loop (skipped) preset value.. 5599.98 BogoMIPS (lpj=11199976)
May 18 16:35:19 ctl01 kernel: [    0.012003] pid_max: default: 32768 minimum: 301
May 18 16:35:19 ctl01 kernel: [    0.013052] Security Framework initialized
May 18 16:35:19 ctl01 kernel: [    0.013970] Yama: becoming mindful.
May 18 16:35:19 ctl01 kernel: [    0.014816] AppArmor: AppArmor initialized
May 18 16:35:19 ctl01 kernel: [    0.019146] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
May 18 16:35:19 ctl01 kernel: [    0.021447] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
May 18 16:35:19 ctl01 kernel: [    0.024053] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
May 18 16:35:19 ctl01 kernel: [    0.025546] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
May 18 16:35:19 ctl01 kernel: [    0.027410] mce: CPU supports 10 MCE banks
May 18 16:35:19 ctl01 kernel: [    0.028044] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
May 18 16:35:19 ctl01 kernel: [    0.029193] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
May 18 16:35:19 ctl01 kernel: [    0.030458] Spectre V2 : Mitigation: Full generic retpoline
May 18 16:35:19 ctl01 kernel: [    0.032002] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
May 18 16:35:19 ctl01 kernel: [    0.033756] Spectre V2 : Enabling Restricted Speculation for firmware calls
May 18 16:35:19 ctl01 kernel: [    0.035224] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
May 18 16:35:19 ctl01 kernel: [    0.036003] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
May 18 16:35:19 ctl01 kernel: [    0.038049] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
May 18 16:35:19 ctl01 kernel: [    0.046440] Freeing SMP alternatives memory: 36K
May 18 16:35:19 ctl01 kernel: [    0.048752] TSC deadline timer enabled
May 18 16:35:19 ctl01 kernel: [    0.048756] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (family: 0x6, model: 0x3e, stepping: 0x4)
May 18 16:35:19 ctl01 kernel: [    0.050948] Performance Events: IvyBridge events, Intel PMU driver.
May 18 16:35:19 ctl01 kernel: [    0.052000] ... version:                2
May 18 16:35:19 ctl01 kernel: [    0.052005] ... bit width:              48
May 18 16:35:19 ctl01 kernel: [    0.052930] ... generic registers:      4
May 18 16:35:19 ctl01 kernel: [    0.053834] ... value mask:             0000ffffffffffff
May 18 16:35:19 ctl01 kernel: [    0.054983] ... max period:             000000007fffffff
May 18 16:35:19 ctl01 kernel: [    0.056004] ... fixed-purpose events:   3
May 18 16:35:19 ctl01 kernel: [    0.056906] ... event mask:             000000070000000f
May 18 16:35:19 ctl01 kernel: [    0.058105] Hierarchical SRCU implementation.
May 18 16:35:19 ctl01 kernel: [    0.059950] smp: Bringing up secondary CPUs ...
May 18 16:35:19 ctl01 kernel: [    0.060165] x86: Booting SMP configuration:
May 18 16:35:19 ctl01 kernel: [    0.061106] .... node  #0, CPUs:      #1
May 18 16:35:19 ctl01 kernel: [    0.004000] kvm-clock: cpu 1, msr 3:bff54041, secondary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.064024] KVM setup async PF for cpu 1
May 18 16:35:19 ctl01 kernel: [    0.064896] kvm-stealtime: cpu 1, msr 3bfc64040
May 18 16:35:19 ctl01 kernel: [    0.065910]  #2
May 18 16:35:19 ctl01 kernel: [    0.004000] kvm-clock: cpu 2, msr 3:bff54081, secondary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.068029] KVM setup async PF for cpu 2
May 18 16:35:19 ctl01 kernel: [    0.068917] kvm-stealtime: cpu 2, msr 3bfca4040
May 18 16:35:19 ctl01 kernel: [    0.069932]  #3
May 18 16:35:19 ctl01 kernel: [    0.004000] kvm-clock: cpu 3, msr 3:bff540c1, secondary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.072055] KVM setup async PF for cpu 3
May 18 16:35:19 ctl01 kernel: [    0.072956] kvm-stealtime: cpu 3, msr 3bfce4040
May 18 16:35:19 ctl01 kernel: [    0.073962]  #4
May 18 16:35:19 ctl01 kernel: [    0.004000] kvm-clock: cpu 4, msr 3:bff54101, secondary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.076030] KVM setup async PF for cpu 4
May 18 16:35:19 ctl01 kernel: [    0.076899] kvm-stealtime: cpu 4, msr 3bfd24040
May 18 16:35:19 ctl01 kernel: [    0.077910]  #5
May 18 16:35:19 ctl01 kernel: [    0.004000] kvm-clock: cpu 5, msr 3:bff54141, secondary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.080029] KVM setup async PF for cpu 5
May 18 16:35:19 ctl01 kernel: [    0.080902] kvm-stealtime: cpu 5, msr 3bfd64040
May 18 16:35:19 ctl01 kernel: [    0.081911]  #6
May 18 16:35:19 ctl01 kernel: [    0.004000] kvm-clock: cpu 6, msr 3:bff54181, secondary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.088052] KVM setup async PF for cpu 6
May 18 16:35:19 ctl01 kernel: [    0.088913] kvm-stealtime: cpu 6, msr 3bfda4040
May 18 16:35:19 ctl01 kernel: [    0.089911]  #7
May 18 16:35:19 ctl01 kernel: [    0.004000] kvm-clock: cpu 7, msr 3:bff541c1, secondary cpu clock
May 18 16:35:19 ctl01 kernel: [    0.092072] KVM setup async PF for cpu 7
May 18 16:35:19 ctl01 kernel: [    0.094055] kvm-stealtime: cpu 7, msr 3bfde4040
May 18 16:35:19 ctl01 kernel: [    0.112024] smp: Brought up 1 node, 8 CPUs
May 18 16:35:19 ctl01 kernel: [    0.112972] smpboot: Max logical packages: 8
May 18 16:35:19 ctl01 kernel: [    0.113938] smpboot: Total of 8 processors activated (44799.90 BogoMIPS)
May 18 16:35:19 ctl01 kernel: [    0.116928] devtmpfs: initialized
May 18 16:35:19 ctl01 kernel: [    0.116928] x86/mm: Memory block size: 128MB
May 18 16:35:19 ctl01 kernel: [    0.120186] evm: security.selinux
May 18 16:35:19 ctl01 kernel: [    0.120979] evm: security.SMACK64
May 18 16:35:19 ctl01 kernel: [    0.121745] evm: security.SMACK64EXEC
May 18 16:35:19 ctl01 kernel: [    0.122594] evm: security.SMACK64TRANSMUTE
May 18 16:35:19 ctl01 kernel: [    0.123508] evm: security.SMACK64MMAP
May 18 16:35:19 ctl01 kernel: [    0.124006] evm: security.apparmor
May 18 16:35:19 ctl01 kernel: [    0.124790] evm: security.ima
May 18 16:35:19 ctl01 kernel: [    0.125497] evm: security.capability
May 18 16:35:19 ctl01 kernel: [    0.126374] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
May 18 16:35:19 ctl01 kernel: [    0.128022] futex hash table entries: 2048 (order: 5, 131072 bytes)
May 18 16:35:19 ctl01 kernel: [    0.129457] pinctrl core: initialized pinctrl subsystem
May 18 16:35:19 ctl01 kernel: [    0.130788] RTC time: 16:35:07, date: 05/18/19
May 18 16:35:19 ctl01 kernel: [    0.132704] NET: Registered protocol family 16
May 18 16:35:19 ctl01 kernel: [    0.133791] audit: initializing netlink subsys (disabled)
May 18 16:35:19 ctl01 kernel: [    0.135013] audit: type=2000 audit(1558197305.909:1): state=initialized audit_enabled=0 res=1
May 18 16:35:19 ctl01 kernel: [    0.136010] cpuidle: using governor ladder
May 18 16:35:19 ctl01 kernel: [    0.136976] cpuidle: using governor menu
May 18 16:35:19 ctl01 kernel: [    0.138143] ACPI: bus type PCI registered
May 18 16:35:19 ctl01 kernel: [    0.140007] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
May 18 16:35:19 ctl01 kernel: [    0.141516] PCI: Using configuration type 1 for base access
May 18 16:35:19 ctl01 kernel: [    0.142749] core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off
May 18 16:35:19 ctl01 kernel: [    0.145502] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
May 18 16:35:19 ctl01 kernel: [    0.145502] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
May 18 16:35:19 ctl01 kernel: [    0.148129] ACPI: Added _OSI(Module Device)
May 18 16:35:19 ctl01 kernel: [    0.149074] ACPI: Added _OSI(Processor Device)
May 18 16:35:19 ctl01 kernel: [    0.150057] ACPI: Added _OSI(3.0 _SCP Extensions)
May 18 16:35:19 ctl01 kernel: [    0.151088] ACPI: Added _OSI(Processor Aggregator Device)
May 18 16:35:19 ctl01 kernel: [    0.152006] ACPI: Added _OSI(Linux-Dell-Video)
May 18 16:35:19 ctl01 kernel: [    0.152986] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
May 18 16:35:19 ctl01 kernel: [    0.154129] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
May 18 16:35:19 ctl01 kernel: [    0.156915] ACPI: Interpreter enabled
May 18 16:35:19 ctl01 kernel: [    0.157776] ACPI: (supports S0 S5)
May 18 16:35:19 ctl01 kernel: [    0.158568] ACPI: Using IOAPIC for interrupt routing
May 18 16:35:19 ctl01 kernel: [    0.159680] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
May 18 16:35:19 ctl01 kernel: [    0.160553] ACPI: Enabled 16 GPEs in block 00 to 0F
May 18 16:35:19 ctl01 kernel: [    0.165624] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
May 18 16:35:19 ctl01 kernel: [    0.166953] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
May 18 16:35:19 ctl01 kernel: [    0.168011] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
May 18 16:35:19 ctl01 kernel: [    0.169410] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
May 18 16:35:19 ctl01 kernel: [    0.172131] acpiphp: Slot [3] registered
May 18 16:35:19 ctl01 kernel: [    0.173070] acpiphp: Slot [4] registered
May 18 16:35:19 ctl01 kernel: [    0.174010] acpiphp: Slot [5] registered
May 18 16:35:19 ctl01 kernel: [    0.174943] acpiphp: Slot [6] registered
May 18 16:35:19 ctl01 kernel: [    0.175872] acpiphp: Slot [7] registered
May 18 16:35:19 ctl01 kernel: [    0.176066] acpiphp: Slot [9] registered
May 18 16:35:19 ctl01 kernel: [    0.177010] acpiphp: Slot [10] registered
May 18 16:35:19 ctl01 kernel: [    0.177968] acpiphp: Slot [11] registered
May 18 16:35:19 ctl01 kernel: [    0.178915] acpiphp: Slot [12] registered
May 18 16:35:19 ctl01 kernel: [    0.179859] acpiphp: Slot [13] registered
May 18 16:35:19 ctl01 kernel: [    0.180050] acpiphp: Slot [14] registered
May 18 16:35:19 ctl01 kernel: [    0.180998] acpiphp: Slot [15] registered
May 18 16:35:19 ctl01 kernel: [    0.181940] acpiphp: Slot [16] registered
May 18 16:35:19 ctl01 kernel: [    0.182884] acpiphp: Slot [17] registered
May 18 16:35:19 ctl01 kernel: [    0.183845] acpiphp: Slot [18] registered
May 18 16:35:19 ctl01 kernel: [    0.184049] acpiphp: Slot [19] registered
May 18 16:35:19 ctl01 kernel: [    0.184990] acpiphp: Slot [20] registered
May 18 16:35:19 ctl01 kernel: [    0.185935] acpiphp: Slot [21] registered
May 18 16:35:19 ctl01 kernel: [    0.186895] acpiphp: Slot [22] registered
May 18 16:35:19 ctl01 kernel: [    0.187842] acpiphp: Slot [23] registered
May 18 16:35:19 ctl01 kernel: [    0.188056] acpiphp: Slot [24] registered
May 18 16:35:19 ctl01 kernel: [    0.189002] acpiphp: Slot [25] registered
May 18 16:35:19 ctl01 kernel: [    0.189947] acpiphp: Slot [26] registered
May 18 16:35:19 ctl01 kernel: [    0.190897] acpiphp: Slot [27] registered
May 18 16:35:19 ctl01 kernel: [    0.191842] acpiphp: Slot [28] registered
May 18 16:35:19 ctl01 kernel: [    0.192055] acpiphp: Slot [29] registered
May 18 16:35:19 ctl01 kernel: [    0.193000] acpiphp: Slot [30] registered
May 18 16:35:19 ctl01 kernel: [    0.193948] acpiphp: Slot [31] registered
May 18 16:35:19 ctl01 kernel: [    0.194869] PCI host bridge to bus 0000:00
May 18 16:35:19 ctl01 kernel: [    0.196008] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
May 18 16:35:19 ctl01 kernel: [    0.197612] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
May 18 16:35:19 ctl01 kernel: [    0.199195] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
May 18 16:35:19 ctl01 kernel: [    0.200006] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
May 18 16:35:19 ctl01 kernel: [    0.201803] pci_bus 0000:00: root bus resource [bus 00-ff]
May 18 16:35:19 ctl01 kernel: [    0.203159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
May 18 16:35:19 ctl01 kernel: [    0.203786] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
May 18 16:35:19 ctl01 kernel: [    0.204619] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
May 18 16:35:19 ctl01 kernel: [    0.211778] pci 0000:00:01.1: reg 0x20: [io  0xc140-0xc14f]
May 18 16:35:19 ctl01 kernel: [    0.215196] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
May 18 16:35:19 ctl01 kernel: [    0.216006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
May 18 16:35:19 ctl01 kernel: [    0.217380] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
May 18 16:35:19 ctl01 kernel: [    0.218864] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
May 18 16:35:19 ctl01 kernel: [    0.220261] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
May 18 16:35:19 ctl01 kernel: [    0.220818] pci 0000:00:01.3: quirk: [io  0x0600-0x063f] claimed by PIIX4 ACPI
May 18 16:35:19 ctl01 kernel: [    0.222391] pci 0000:00:01.3: quirk: [io  0x0700-0x070f] claimed by PIIX4 SMB
May 18 16:35:19 ctl01 kernel: [    0.224178] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000
May 18 16:35:19 ctl01 kernel: [    0.227010] pci 0000:00:02.0: reg 0x10: [mem 0xfc000000-0xfdffffff pref]
May 18 16:35:19 ctl01 kernel: [    0.229370] pci 0000:00:02.0: reg 0x14: [mem 0xfebd0000-0xfebd0fff]
May 18 16:35:19 ctl01 kernel: [    0.242595] pci 0000:00:02.0: reg 0x30: [mem 0xfebc0000-0xfebcffff pref]
May 18 16:35:19 ctl01 kernel: [    0.242928] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
May 18 16:35:19 ctl01 kernel: [    0.245166] pci 0000:00:03.0: reg 0x10: [io  0xc040-0xc05f]
May 18 16:35:19 ctl01 kernel: [    0.248006] pci 0000:00:03.0: reg 0x14: [mem 0xfebd1000-0xfebd1fff]
May 18 16:35:19 ctl01 kernel: [    0.260008] pci 0000:00:03.0: reg 0x30: [mem 0xfeac0000-0xfeafffff pref]
May 18 16:35:19 ctl01 kernel: [    0.260480] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000
May 18 16:35:19 ctl01 kernel: [    0.262767] pci 0000:00:04.0: reg 0x10: [io  0xc060-0xc07f]
May 18 16:35:19 ctl01 kernel: [    0.264029] pci 0000:00:04.0: reg 0x14: [mem 0xfebd2000-0xfebd2fff]
May 18 16:35:19 ctl01 kernel: [    0.276009] pci 0000:00:04.0: reg 0x30: [mem 0xfeb00000-0xfeb3ffff pref]
May 18 16:35:19 ctl01 kernel: [    0.276501] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000
May 18 16:35:19 ctl01 kernel: [    0.278762] pci 0000:00:05.0: reg 0x10: [io  0xc080-0xc09f]
May 18 16:35:19 ctl01 kernel: [    0.282990] pci 0000:00:05.0: reg 0x14: [mem 0xfebd3000-0xfebd3fff]
May 18 16:35:19 ctl01 kernel: [    0.294323] pci 0000:00:05.0: reg 0x30: [mem 0xfeb40000-0xfeb7ffff pref]
May 18 16:35:19 ctl01 kernel: [    0.294782] pci 0000:00:06.0: [1af4:1000] type 00 class 0x020000
May 18 16:35:19 ctl01 kernel: [    0.296007] pci 0000:00:06.0: reg 0x10: [io  0xc0a0-0xc0bf]
May 18 16:35:19 ctl01 kernel: [    0.298219] pci 0000:00:06.0: reg 0x14: [mem 0xfebd4000-0xfebd4fff]
May 18 16:35:19 ctl01 kernel: [    0.309423] pci 0000:00:06.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref]
May 18 16:35:19 ctl01 kernel: [    0.309910] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000
May 18 16:35:19 ctl01 kernel: [    0.312007] pci 0000:00:07.0: reg 0x10: [io  0xc000-0xc03f]
May 18 16:35:19 ctl01 kernel: [    0.314363] pci 0000:00:07.0: reg 0x14: [mem 0xfebd5000-0xfebd5fff]
May 18 16:35:19 ctl01 kernel: [    0.327856] pci 0000:00:08.0: [8086:2934] type 00 class 0x0c0300
May 18 16:35:19 ctl01 kernel: [    0.333175] pci 0000:00:08.0: reg 0x20: [io  0xc0c0-0xc0df]
May 18 16:35:19 ctl01 kernel: [    0.336042] pci 0000:00:08.1: [8086:2935] type 00 class 0x0c0300
May 18 16:35:19 ctl01 kernel: [    0.341718] pci 0000:00:08.1: reg 0x20: [io  0xc0e0-0xc0ff]
May 18 16:35:19 ctl01 kernel: [    0.344252] pci 0000:00:08.2: [8086:2936] type 00 class 0x0c0300
May 18 16:35:19 ctl01 kernel: [    0.351533] pci 0000:00:08.2: reg 0x20: [io  0xc100-0xc11f]
May 18 16:35:19 ctl01 kernel: [    0.354057] pci 0000:00:08.7: [8086:293a] type 00 class 0x0c0320
May 18 16:35:19 ctl01 kernel: [    0.355288] pci 0000:00:08.7: reg 0x10: [mem 0xfebd6000-0xfebd6fff]
May 18 16:35:19 ctl01 kernel: [    0.363155] pci 0000:00:09.0: [1af4:1002] type 00 class 0x00ff00
May 18 16:35:19 ctl01 kernel: [    0.364007] pci 0000:00:09.0: reg 0x10: [io  0xc120-0xc13f]
May 18 16:35:19 ctl01 kernel: [    0.372998] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
May 18 16:35:19 ctl01 kernel: [    0.374469] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
May 18 16:35:19 ctl01 kernel: [    0.375908] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
May 18 16:35:19 ctl01 kernel: [    0.376148] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
May 18 16:35:19 ctl01 kernel: [    0.377520] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
May 18 16:35:19 ctl01 kernel: [    0.380079] SCSI subsystem initialized
May 18 16:35:19 ctl01 kernel: [    0.381103] libata version 3.00 loaded.
May 18 16:35:19 ctl01 kernel: [    0.381103] pci 0000:00:02.0: vgaarb: setting as boot VGA device
May 18 16:35:19 ctl01 kernel: [    0.381405] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
May 18 16:35:19 ctl01 kernel: [    0.384008] pci 0000:00:02.0: vgaarb: bridge control possible
May 18 16:35:19 ctl01 kernel: [    0.385329] vgaarb: loaded
May 18 16:35:19 ctl01 kernel: [    0.386071] ACPI: bus type USB registered
May 18 16:35:19 ctl01 kernel: [    0.387084] usbcore: registered new interface driver usbfs
May 18 16:35:19 ctl01 kernel: [    0.388015] usbcore: registered new interface driver hub
May 18 16:35:19 ctl01 kernel: [    0.389289] usbcore: registered new device driver usb
May 18 16:35:19 ctl01 kernel: [    0.390569] EDAC MC: Ver: 3.0.0
May 18 16:35:19 ctl01 kernel: [    0.392038] PCI: Using ACPI for IRQ routing
May 18 16:35:19 ctl01 kernel: [    0.393054] PCI: pci_cache_line_size set to 64 bytes
May 18 16:35:19 ctl01 kernel: [    0.393383] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
May 18 16:35:19 ctl01 kernel: [    0.393385] e820: reserve RAM buffer [mem 0xbffdf000-0xbfffffff]
May 18 16:35:19 ctl01 kernel: [    0.393505] NetLabel: Initializing
May 18 16:35:19 ctl01 kernel: [    0.394373] NetLabel:  domain hash size = 128
May 18 16:35:19 ctl01 kernel: [    0.395413] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
May 18 16:35:19 ctl01 kernel: [    0.396023] NetLabel:  unlabeled traffic allowed by default
May 18 16:35:19 ctl01 kernel: [    0.397334] clocksource: Switched to clocksource kvm-clock
May 18 16:35:19 ctl01 kernel: [    0.411249] VFS: Disk quotas dquot_6.6.0
May 18 16:35:19 ctl01 kernel: [    0.412253] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
May 18 16:35:19 ctl01 kernel: [    0.413949] AppArmor: AppArmor Filesystem Enabled
May 18 16:35:19 ctl01 kernel: [    0.415100] pnp: PnP ACPI init
May 18 16:35:19 ctl01 kernel: [    0.415952] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
May 18 16:35:19 ctl01 kernel: [    0.416017] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
May 18 16:35:19 ctl01 kernel: [    0.416058] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active)
May 18 16:35:19 ctl01 kernel: [    0.416096] pnp 00:03: [dma 2]
May 18 16:35:19 ctl01 kernel: [    0.416115] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active)
May 18 16:35:19 ctl01 kernel: [    0.416224] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
May 18 16:35:19 ctl01 kernel: [    0.416534] pnp: PnP ACPI: found 5 devices
May 18 16:35:19 ctl01 kernel: [    0.424474] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
May 18 16:35:19 ctl01 kernel: [    0.426354] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
May 18 16:35:19 ctl01 kernel: [    0.426356] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
May 18 16:35:19 ctl01 kernel: [    0.426357] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
May 18 16:35:19 ctl01 kernel: [    0.426358] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
May 18 16:35:19 ctl01 kernel: [    0.426424] NET: Registered protocol family 2
May 18 16:35:19 ctl01 kernel: [    0.427690] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
May 18 16:35:19 ctl01 kernel: [    0.429495] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
May 18 16:35:19 ctl01 kernel: [    0.431069] TCP: Hash tables configured (established 131072 bind 65536)
May 18 16:35:19 ctl01 kernel: [    0.432538] UDP hash table entries: 8192 (order: 6, 262144 bytes)
May 18 16:35:19 ctl01 kernel: [    0.433877] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
May 18 16:35:19 ctl01 kernel: [    0.435353] NET: Registered protocol family 1
May 18 16:35:19 ctl01 kernel: [    0.436337] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
May 18 16:35:19 ctl01 kernel: [    0.437614] pci 0000:00:01.0: PIIX3: Enabling Passive Release
May 18 16:35:19 ctl01 kernel: [    0.438872] pci 0000:00:01.0: Activating ISA DMA hang workarounds
May 18 16:35:19 ctl01 kernel: [    0.440245] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
May 18 16:35:19 ctl01 kernel: [    0.472539] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
May 18 16:35:19 ctl01 kernel: [    0.535056] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
May 18 16:35:19 ctl01 kernel: [    0.597811] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
May 18 16:35:19 ctl01 kernel: [    0.661688] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
May 18 16:35:19 ctl01 kernel: [    0.694062] PCI: CLS 0 bytes, default 64
May 18 16:35:19 ctl01 kernel: [    0.694121] Unpacking initramfs...
May 18 16:35:19 ctl01 kernel: [    1.420041] Freeing initrd memory: 53476K
May 18 16:35:19 ctl01 kernel: [    1.422527] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
May 18 16:35:19 ctl01 kernel: [    1.424259] software IO TLB [mem 0xbbfdf000-0xbffdf000] (64MB) mapped at [        (ptrval)-        (ptrval)]
May 18 16:35:19 ctl01 kernel: [    1.426351] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x285c3aeaff3, max_idle_ns: 440795255742 ns
May 18 16:35:19 ctl01 kernel: [    1.428491] Scanning for low memory corruption every 60 seconds
May 18 16:35:19 ctl01 kernel: [    1.430627] Initialise system trusted keyrings
May 18 16:35:19 ctl01 kernel: [    1.431656] Key type blacklist registered
May 18 16:35:19 ctl01 kernel: [    1.432709] workingset: timestamp_bits=36 max_order=22 bucket_order=0
May 18 16:35:19 ctl01 kernel: [    1.435322] zbud: loaded
May 18 16:35:19 ctl01 kernel: [    1.436693] squashfs: version 4.0 (2009/01/31) Phillip Lougher
May 18 16:35:19 ctl01 kernel: [    1.438224] fuse init (API version 7.26)
May 18 16:35:19 ctl01 kernel: [    1.441240] Key type asymmetric registered
May 18 16:35:19 ctl01 kernel: [    1.442353] Asymmetric key parser 'x509' registered
May 18 16:35:19 ctl01 kernel: [    1.443496] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
May 18 16:35:19 ctl01 kernel: [    1.445242] io scheduler noop registered
May 18 16:35:19 ctl01 kernel: [    1.446140] io scheduler deadline registered
May 18 16:35:19 ctl01 kernel: [    1.447130] io scheduler cfq registered (default)
May 18 16:35:19 ctl01 kernel: [    1.448611] intel_idle: Please enable MWAIT in BIOS SETUP
May 18 16:35:19 ctl01 kernel: [    1.448694] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
May 18 16:35:19 ctl01 kernel: [    1.450312] ACPI: Power Button [PWRF]
May 18 16:35:19 ctl01 kernel: [    1.482341] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
May 18 16:35:19 ctl01 kernel: [    1.515073] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
May 18 16:35:19 ctl01 kernel: [    1.547805] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
May 18 16:35:19 ctl01 kernel: [    1.580854] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
May 18 16:35:19 ctl01 kernel: [    1.613567] virtio-pci 0000:00:07.0: virtio_pci: leaving for legacy driver
May 18 16:35:19 ctl01 kernel: [    1.645667] virtio-pci 0000:00:09.0: virtio_pci: leaving for legacy driver
May 18 16:35:19 ctl01 kernel: [    1.648328] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
May 18 16:35:19 ctl01 kernel: [    1.674182] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
May 18 16:35:19 ctl01 kernel: [    1.677794] Linux agpgart interface v0.103
May 18 16:35:19 ctl01 kernel: [    1.681530] loop: module loaded
May 18 16:35:19 ctl01 kernel: [    1.682399] ata_piix 0000:00:01.1: version 2.13
May 18 16:35:19 ctl01 kernel: [    1.683797] scsi host0: ata_piix
May 18 16:35:19 ctl01 kernel: [    1.684780] scsi host1: ata_piix
May 18 16:35:19 ctl01 kernel: [    1.685567] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc140 irq 14
May 18 16:35:19 ctl01 kernel: [    1.686970] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc148 irq 15
May 18 16:35:19 ctl01 kernel: [    1.688479] libphy: Fixed MDIO Bus: probed
May 18 16:35:19 ctl01 kernel: [    1.689499] tun: Universal TUN/TAP device driver, 1.6
May 18 16:35:19 ctl01 kernel: [    1.690681] PPP generic driver version 2.4.2
May 18 16:35:19 ctl01 kernel: [    1.691736] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
May 18 16:35:19 ctl01 kernel: [    1.700523] ehci-pci: EHCI PCI platform driver
May 18 16:35:19 ctl01 kernel: [    1.732687] ehci-pci 0000:00:08.7: EHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.733835] ehci-pci 0000:00:08.7: new USB bus registered, assigned bus number 1
May 18 16:35:19 ctl01 kernel: [    1.735630] ehci-pci 0000:00:08.7: irq 11, io mem 0xfebd6000
May 18 16:35:19 ctl01 kernel: [    1.752038] ehci-pci 0000:00:08.7: USB 2.0 started, EHCI 1.00
May 18 16:35:19 ctl01 kernel: [    1.753320] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
May 18 16:35:19 ctl01 kernel: [    1.754751] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:35:19 ctl01 kernel: [    1.756328] usb usb1: Product: EHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.757405] usb usb1: Manufacturer: Linux 4.15.0-50-generic ehci_hcd
May 18 16:35:19 ctl01 kernel: [    1.758745] usb usb1: SerialNumber: 0000:00:08.7
May 18 16:35:19 ctl01 kernel: [    1.759929] hub 1-0:1.0: USB hub found
May 18 16:35:19 ctl01 kernel: [    1.760812] hub 1-0:1.0: 6 ports detected
May 18 16:35:19 ctl01 kernel: [    1.761937] ehci-platform: EHCI generic platform driver
May 18 16:35:19 ctl01 kernel: [    1.763084] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
May 18 16:35:19 ctl01 kernel: [    1.764404] ohci-pci: OHCI PCI platform driver
May 18 16:35:19 ctl01 kernel: [    1.765395] ohci-platform: OHCI generic platform driver
May 18 16:35:19 ctl01 kernel: [    1.766528] uhci_hcd: USB Universal Host Controller Interface driver
May 18 16:35:19 ctl01 kernel: [    1.799438] uhci_hcd 0000:00:08.0: UHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.800592] uhci_hcd 0000:00:08.0: new USB bus registered, assigned bus number 2
May 18 16:35:19 ctl01 kernel: [    1.802209] uhci_hcd 0000:00:08.0: detected 2 ports
May 18 16:35:19 ctl01 kernel: [    1.803353] uhci_hcd 0000:00:08.0: irq 11, io base 0x0000c0c0
May 18 16:35:19 ctl01 kernel: [    1.804671] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
May 18 16:35:19 ctl01 kernel: [    1.806160] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:35:19 ctl01 kernel: [    1.807801] usb usb2: Product: UHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.808940] usb usb2: Manufacturer: Linux 4.15.0-50-generic uhci_hcd
May 18 16:35:19 ctl01 kernel: [    1.810310] usb usb2: SerialNumber: 0000:00:08.0
May 18 16:35:19 ctl01 kernel: [    1.811505] hub 2-0:1.0: USB hub found
May 18 16:35:19 ctl01 kernel: [    1.812410] hub 2-0:1.0: 2 ports detected
May 18 16:35:19 ctl01 kernel: [    1.848146] uhci_hcd 0000:00:08.1: UHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.849055] ata1.01: NODEV after polling detection
May 18 16:35:19 ctl01 kernel: [    1.849450] ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
May 18 16:35:19 ctl01 kernel: [    1.850127] ata1.00: configured for MWDMA2
May 18 16:35:19 ctl01 kernel: [    1.852032] uhci_hcd 0000:00:08.1: new USB bus registered, assigned bus number 3
May 18 16:35:19 ctl01 kernel: [    1.852048] scsi 0:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
May 18 16:35:19 ctl01 kernel: [    1.855664] uhci_hcd 0000:00:08.1: detected 2 ports
May 18 16:35:19 ctl01 kernel: [    1.856933] uhci_hcd 0000:00:08.1: irq 10, io base 0x0000c0e0
May 18 16:35:19 ctl01 kernel: [    1.858592] sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
May 18 16:35:19 ctl01 kernel: [    1.858733] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
May 18 16:35:19 ctl01 kernel: [    1.858734] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:35:19 ctl01 kernel: [    1.858736] usb usb3: Product: UHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.858737] usb usb3: Manufacturer: Linux 4.15.0-50-generic uhci_hcd
May 18 16:35:19 ctl01 kernel: [    1.858738] usb usb3: SerialNumber: 0000:00:08.1
May 18 16:35:19 ctl01 kernel: [    1.858903] hub 3-0:1.0: USB hub found
May 18 16:35:19 ctl01 kernel: [    1.858908] hub 3-0:1.0: 2 ports detected
May 18 16:35:19 ctl01 kernel: [    1.869239] cdrom: Uniform CD-ROM driver Revision: 3.20
May 18 16:35:19 ctl01 kernel: [    1.870644] sr 0:0:0:0: Attached scsi CD-ROM sr0
May 18 16:35:19 ctl01 kernel: [    1.870866] sr 0:0:0:0: Attached scsi generic sg0 type 5
May 18 16:35:19 ctl01 kernel: [    1.891394] uhci_hcd 0000:00:08.2: UHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.892645] uhci_hcd 0000:00:08.2: new USB bus registered, assigned bus number 4
May 18 16:35:19 ctl01 kernel: [    1.894268] uhci_hcd 0000:00:08.2: detected 2 ports
May 18 16:35:19 ctl01 kernel: [    1.895434] uhci_hcd 0000:00:08.2: irq 10, io base 0x0000c100
May 18 16:35:19 ctl01 kernel: [    1.896755] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
May 18 16:35:19 ctl01 kernel: [    1.898206] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 18 16:35:19 ctl01 kernel: [    1.899799] usb usb4: Product: UHCI Host Controller
May 18 16:35:19 ctl01 kernel: [    1.901009] usb usb4: Manufacturer: Linux 4.15.0-50-generic uhci_hcd
May 18 16:35:19 ctl01 kernel: [    1.902368] usb usb4: SerialNumber: 0000:00:08.2
May 18 16:35:19 ctl01 kernel: [    1.903534] hub 4-0:1.0: USB hub found
May 18 16:35:19 ctl01 kernel: [    1.904435] hub 4-0:1.0: 2 ports detected
May 18 16:35:19 ctl01 kernel: [    1.905541] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
May 18 16:35:19 ctl01 kernel: [    1.908236] serio: i8042 KBD port at 0x60,0x64 irq 1
May 18 16:35:19 ctl01 kernel: [    1.909383] serio: i8042 AUX port at 0x60,0x64 irq 12
May 18 16:35:19 ctl01 kernel: [    1.910666] mousedev: PS/2 mouse device common for all mice
May 18 16:35:19 ctl01 kernel: [    1.912212] rtc_cmos 00:00: RTC can wake from S4
May 18 16:35:19 ctl01 kernel: [    1.913891] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
May 18 16:35:19 ctl01 kernel: [    1.915791] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
May 18 16:35:19 ctl01 kernel: [    1.917298] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
May 18 16:35:19 ctl01 kernel: [    1.918853] i2c /dev entries driver
May 18 16:35:19 ctl01 kernel: [    1.919755] pcie_mp2_amd: AMD(R) PCI-E MP2 Communication Driver Version: 1.0
May 18 16:35:19 ctl01 kernel: [    1.921429] device-mapper: uevent: version 1.0.3
May 18 16:35:19 ctl01 kernel: [    1.922674] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com
May 18 16:35:19 ctl01 kernel: [    1.925158] ledtrig-cpu: registered to indicate activity on CPUs
May 18 16:35:19 ctl01 kernel: [    1.926991] NET: Registered protocol family 10
May 18 16:35:19 ctl01 kernel: [    1.932290] Segment Routing with IPv6
May 18 16:35:19 ctl01 kernel: [    1.933230] NET: Registered protocol family 17
May 18 16:35:19 ctl01 kernel: [    1.934396] Key type dns_resolver registered
May 18 16:35:19 ctl01 kernel: [    1.936075] RAS: Correctable Errors collector initialized.
May 18 16:35:19 ctl01 kernel: [    1.937297] sched_clock: Marking stable (1936021750, 0)->(2433868264, -497846514)
May 18 16:35:19 ctl01 kernel: [    1.939487] registered taskstats version 1
May 18 16:35:19 ctl01 kernel: [    1.940469] Loading compiled-in X.509 certificates
May 18 16:35:19 ctl01 kernel: [    1.944207] Loaded X.509 cert 'Build time autogenerated kernel key: 96649165ed7c6e7d68b41a1fadd9ef21d69b21a5'
May 18 16:35:19 ctl01 kernel: [    1.946329] zswap: loaded using pool lzo/zbud
May 18 16:35:19 ctl01 kernel: [    1.951713] Key type big_key registered
May 18 16:35:19 ctl01 kernel: [    1.952656] Key type trusted registered
May 18 16:35:19 ctl01 kernel: [    1.955368] Key type encrypted registered
May 18 16:35:19 ctl01 kernel: [    1.956332] AppArmor: AppArmor sha1 policy hashing enabled
May 18 16:35:19 ctl01 kernel: [    1.957535] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
May 18 16:35:19 ctl01 kernel: [    1.958872] ima: Allocated hash algorithm: sha1
May 18 16:35:19 ctl01 kernel: [    1.959913] evm: HMAC attrs: 0x1
May 18 16:35:19 ctl01 kernel: [    1.961129]   Magic number: 3:74:591
May 18 16:35:19 ctl01 kernel: [    1.962249] rtc_cmos 00:00: setting system clock to 2019-05-18 16:35:09 UTC (1558197309)
May 18 16:35:19 ctl01 kernel: [    1.964191] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
May 18 16:35:19 ctl01 kernel: [    1.965609] EDD information not available.
May 18 16:35:19 ctl01 kernel: [    2.331728] Freeing unused kernel image memory: 2408K
May 18 16:35:19 ctl01 kernel: [    2.368112] Write protecting the kernel read-only data: 20480k
May 18 16:35:19 ctl01 kernel: [    2.373264] Freeing unused kernel image memory: 2008K
May 18 16:35:19 ctl01 kernel: [    2.376379] Freeing unused kernel image memory: 1856K
May 18 16:35:19 ctl01 kernel: [    2.389276] x86/mm: Checked W+X mappings: passed, no W+X pages found.
May 18 16:35:19 ctl01 kernel: [    2.390655] x86/mm: Checking user space page tables
May 18 16:35:19 ctl01 kernel: [    2.399654] x86/mm: Checked W+X mappings: passed, no W+X pages found.
May 18 16:35:19 ctl01 kernel: [    2.417826] random: systemd-udevd: uninitialized urandom read (16 bytes read)
May 18 16:35:19 ctl01 kernel: [    2.419121] random: udevadm: uninitialized urandom read (16 bytes read)
May 18 16:35:19 ctl01 kernel: [    2.419154] random: udevadm: uninitialized urandom read (16 bytes read)
May 18 16:35:19 ctl01 kernel: [    2.473809]  vda: vda1
May 18 16:35:19 ctl01 kernel: [    2.484624] FDC 0 is a S82078B
May 18 16:35:19 ctl01 kernel: [    2.488543] AVX version of gcm_enc/dec engaged.
May 18 16:35:19 ctl01 kernel: [    2.489589] AES CTR mode by8 optimization enabled
May 18 16:35:19 ctl01 kernel: [    2.491862] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
May 18 16:35:19 ctl01 kernel: [    2.495376] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
May 18 16:35:19 ctl01 kernel: [    2.498268] virtio_net virtio0 ens3: renamed from eth0
May 18 16:35:19 ctl01 kernel: [    2.501223] [TTM] Zone  kernel: Available graphics memory: 7180132 kiB
May 18 16:35:19 ctl01 kernel: [    2.502687] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
May 18 16:35:19 ctl01 kernel: [    2.504095] [TTM] Initializing pool allocator
May 18 16:35:19 ctl01 kernel: [    2.505127] [TTM] Initializing DMA pool allocator
May 18 16:35:19 ctl01 kernel: [    2.506482] [drm] fb mappable at 0xFC000000
May 18 16:35:19 ctl01 kernel: [    2.507449] [drm] vram aper at 0xFC000000
May 18 16:35:19 ctl01 kernel: [    2.508379] [drm] size 33554432
May 18 16:35:19 ctl01 kernel: [    2.509128] [drm] fb depth is 24
May 18 16:35:19 ctl01 kernel: [    2.509892] [drm]    pitch is 3072
May 18 16:35:19 ctl01 kernel: [    2.510803] fbcon: cirrusdrmfb (fb0) is primary device
May 18 16:35:19 ctl01 kernel: [    2.532212] Console: switching to colour frame buffer device 128x48
May 18 16:35:19 ctl01 kernel: [    2.536800] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device
May 18 16:35:19 ctl01 kernel: [    2.540287] virtio_net virtio1 ens4: renamed from eth1
May 18 16:35:19 ctl01 kernel: [    2.552046] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0
May 18 16:35:19 ctl01 kernel: [    2.560194] virtio_net virtio3 ens6: renamed from eth3
May 18 16:35:19 ctl01 kernel: [    2.584234] virtio_net virtio2 ens5: renamed from eth2
May 18 16:35:19 ctl01 kernel: [    2.740029] raid6: sse2x1   gen()  5425 MB/s
May 18 16:35:19 ctl01 kernel: [    2.788026] raid6: sse2x1   xor()  5888 MB/s
May 18 16:35:19 ctl01 kernel: [    2.836042] raid6: sse2x2   gen()  6897 MB/s
May 18 16:35:19 ctl01 kernel: [    2.884038] raid6: sse2x2   xor()  6211 MB/s
May 18 16:35:19 ctl01 kernel: [    2.932028] raid6: sse2x4   gen()  9727 MB/s
May 18 16:35:19 ctl01 kernel: [    2.980023] raid6: sse2x4   xor()  7698 MB/s
May 18 16:35:19 ctl01 kernel: [    2.980723] raid6: using algorithm sse2x4 gen() 9727 MB/s
May 18 16:35:19 ctl01 kernel: [    2.981599] raid6: .... xor() 7698 MB/s, rmw enabled
May 18 16:35:19 ctl01 kernel: [    2.982400] raid6: using ssse3x2 recovery algorithm
May 18 16:35:19 ctl01 kernel: [    2.984967] xor: automatically using best checksumming function   avx       
May 18 16:35:19 ctl01 kernel: [    2.988217] async_tx: api initialized (async)
May 18 16:35:19 ctl01 kernel: [    3.051445] Btrfs loaded, crc32c=crc32c-intel
May 18 16:35:19 ctl01 kernel: [    3.350076] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
May 18 16:35:19 ctl01 kernel: [    3.971105] random: fast init done
May 18 16:35:19 ctl01 kernel: [    5.040110] EXT4-fs (vda1): re-mounted. Opts: (null)
May 18 16:35:19 ctl01 kernel: [    5.060464] Loading iSCSI transport class v2.0-870.
May 18 16:35:19 ctl01 kernel: [    5.116107] iscsi: registered transport (tcp)
May 18 16:35:19 ctl01 kernel: [    5.176777] iscsi: registered transport (iser)
May 18 16:35:19 ctl01 kernel: [    5.190908] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
May 18 16:35:19 ctl01 kernel: [    5.817662] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0
May 18 16:35:19 ctl01 kernel: [    6.069440] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
May 18 16:35:19 ctl01 kernel: [    6.069447] EDAC sbridge:  Ver: 1.1.2 
May 18 16:35:19 ctl01 kernel: [    6.129054] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
May 18 16:35:19 ctl01 kernel: [    6.131081] br-mgmt: port 1(ens4) entered blocking state
May 18 16:35:19 ctl01 kernel: [    6.131083] br-mgmt: port 1(ens4) entered disabled state
May 18 16:35:19 ctl01 kernel: [    6.131162] device ens4 entered promiscuous mode
May 18 16:35:19 ctl01 kernel: [    6.248531] audit: type=1400 audit(1558197313.784:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=1069 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.266769] audit: type=1400 audit(1558197313.800:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=1071 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.266774] audit: type=1400 audit(1558197313.800:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=1071 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.269246] audit: type=1400 audit(1558197313.804:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=1067 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.269252] audit: type=1400 audit(1558197313.804:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=1067 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.269257] audit: type=1400 audit(1558197313.804:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=1067 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.269260] audit: type=1400 audit(1558197313.804:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=1067 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.276679] audit: type=1400 audit(1558197313.812:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=1073 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.283178] audit: type=1400 audit(1558197313.816:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=1068 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    6.283181] audit: type=1400 audit(1558197313.816:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=1068 comm="apparmor_parser"
May 18 16:35:19 ctl01 kernel: [    8.540492] ISO 9660 Extensions: Microsoft Joliet Level 3
May 18 16:35:19 ctl01 kernel: [    8.544995] ISO 9660 Extensions: RRIP_1991A
May 18 16:35:19 ctl01 kernel: [    8.822975] br-mgmt: port 1(ens4) entered blocking state
May 18 16:35:19 ctl01 kernel: [    8.822977] br-mgmt: port 1(ens4) entered forwarding state
May 18 16:35:19 ctl01 kernel: [    8.823011] IPv6: ADDRCONF(NETDEV_UP): br-mgmt: link is not ready
May 18 16:35:19 ctl01 kernel: [    8.823036] IPv6: ADDRCONF(NETDEV_CHANGE): br-mgmt: link becomes ready
May 18 16:35:19 ctl01 kernel: [    8.888674] 8021q: 802.1Q VLAN Support v1.8
May 18 16:35:19 ctl01 kernel: [    8.888687] 8021q: adding VLAN 0 to HW filter on device ens3
May 18 16:35:19 ctl01 kernel: [    8.888747] 8021q: adding VLAN 0 to HW filter on device ens4
May 18 16:35:19 ctl01 kernel: [    8.888785] 8021q: adding VLAN 0 to HW filter on device ens5
May 18 16:35:19 ctl01 kernel: [   11.797168] new mount options do not match the existing superblock, will be ignored
May 18 16:35:19 ctl01 systemd[1]: Started Login to default iSCSI targets.
May 18 16:35:19 ctl01 smartd[2073]: smartd 6.5 2016-01-24 r4214 [x86_64-linux-4.15.0-50-generic] (local build)
May 18 16:35:19 ctl01 smartd[2073]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
May 18 16:35:19 ctl01 systemd[1]: Reached target Remote File Systems (Pre).
May 18 16:35:19 ctl01 systemd[1]: Reached target Remote File Systems.
May 18 16:35:19 ctl01 systemd[1]: Starting LSB: automatic crash report generation...
May 18 16:35:19 ctl01 systemd[1]: Starting LSB: daemon to balance interrupts for SMP systems...
May 18 16:35:19 ctl01 systemd[1]: Starting Permit User Sessions...
May 18 16:35:20 ctl01 smartd[2073]: Opened configuration file /etc/smartd.conf
May 18 16:35:20 ctl01 irqbalance[2207]:  * Starting SMP IRQ Balancer: irqbalance
May 18 16:35:20 ctl01 systemd[1]: Started Permit User Sessions.
May 18 16:35:20 ctl01 smartd[2073]: Drive: DEVICESCAN, implied '-a' Directive on line 21 of file /etc/smartd.conf
May 18 16:35:20 ctl01 smartd[2073]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
May 18 16:35:20 ctl01 smartd[2073]: Problem creating device name scan list
May 18 16:35:20 ctl01 smartd[2073]: In the system's table of devices NO devices found to scan
May 18 16:35:20 ctl01 smartd[2073]: Unable to monitor any SMART enabled devices. Try debug (-d) option. Exiting...
May 18 16:35:20 ctl01 systemd[1]: smartd.service: Main process exited, code=exited, status=17/n/a
May 18 16:35:20 ctl01 systemd[1]: smartd.service: Unit entered failed state.
May 18 16:35:20 ctl01 systemd[1]: smartd.service: Failed with result 'exit-code'.
May 18 16:35:20 ctl01 systemd[1]: Starting Terminate Plymouth Boot Screen...
May 18 16:35:20 ctl01 systemd[1]: Starting Hold until boot process finishes up...
May 18 16:35:20 ctl01 systemd[1]: Started Hold until boot process finishes up.
May 18 16:35:20 ctl01 systemd[1]: Starting Set console scheme...
May 18 16:35:20 ctl01 systemd[1]: Started Getty on tty1.
May 18 16:35:20 ctl01 systemd[1]: Started Serial Getty on ttyS0.
May 18 16:35:20 ctl01 systemd[1]: Reached target Login Prompts.
May 18 16:35:20 ctl01 systemd[1]: Started Terminate Plymouth Boot Screen.
May 18 16:35:20 ctl01 systemd[1]: Started LSB: Record successful boot for GRUB.
May 18 16:35:20 ctl01 polkitd[2142]: started daemon version 0.105 using authority implementation `local' version `0.105'
May 18 16:35:20 ctl01 dbus[1959]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'
May 18 16:35:20 ctl01 systemd[1]: Started Authenticate and Authorize Users to Run Privileged Tasks.
May 18 16:35:20 ctl01 accounts-daemon[1954]: started daemon version 0.6.40
May 18 16:35:20 ctl01 systemd[1]: Started Set console scheme.
May 18 16:35:20 ctl01 systemd[1]: Started Accounts Service.
May 18 16:35:20 ctl01 irqbalance[2207]:    ...done.
May 18 16:35:20 ctl01 systemd[1]: Started LSB: daemon to balance interrupts for SMP systems.
May 18 16:35:20 ctl01 thermald[1996]: Unsupported cpu model, use thermal-conf.xml file or run with --ignore-cpuid-check
May 18 16:35:20 ctl01 thermald[1996]: THD engine start failed
May 18 16:35:20 ctl01 apport[2202]:  * Starting automatic crash report generation: apport
May 18 16:35:20 ctl01 systemd[1]: Started OpenBSD Secure Shell server.
May 18 16:35:20 ctl01 apport[2202]:    ...done.
May 18 16:35:20 ctl01 systemd[1]: Started LSB: automatic crash report generation.
May 18 16:35:20 ctl01 iscsid: iSCSI daemon with pid=1970 started!
May 18 16:35:20 ctl01 snapd[2096]: AppArmor status: apparmor is enabled and all features are available
May 18 16:35:20 ctl01 ovs-ctl[2051]:  * Starting ovsdb-server
May 18 16:35:20 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1
May 18 16:35:20 ctl01 snapd[2096]: daemon.go:379: started snapd/2.38 (series 16; classic) ubuntu/16.04 (amd64) linux/4.15.0-50-generic.
May 18 16:35:20 ctl01 systemd[1]: Started LXD - container startup/shutdown.
May 18 16:35:20 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.5.5 "external-ids:system-id=\"6373cd72-d76d-4de9-8e9c-6557c19a2afa\"" "system-type=\"Ubuntu\"" "system-version=\"16.04-xenial\""
May 18 16:35:20 ctl01 ovs-ctl[2051]:  * Configuring Open vSwitch system IDs
May 18 16:35:20 ctl01 kernel: [   13.199954] openvswitch: Open vSwitch switching datapath
May 18 16:35:20 ctl01 ovs-ctl[2051]:  * Inserting openvswitch module
May 18 16:35:20 ctl01 systemd[1]: Started Snappy daemon.
May 18 16:35:20 ctl01 systemd[1]: Starting Wait until snapd is fully seeded...
May 18 16:35:20 ctl01 ovs-ctl[2051]:  * Starting ovs-vswitchd
May 18 16:35:20 ctl01 systemd[1]: Started Wait until snapd is fully seeded.
May 18 16:35:20 ctl01 systemd[1]: Starting Apply the settings specified in cloud-config...
May 18 16:35:20 ctl01 ovs-ctl[2051]:  * Enabling remote OVSDB managers
May 18 16:35:20 ctl01 systemd[1]: Started Open vSwitch Internal Unit.
May 18 16:35:20 ctl01 systemd[1]: Starting Open vSwitch...
May 18 16:35:20 ctl01 systemd[1]: Started Open vSwitch.
May 18 16:35:21 ctl01 cloud-init[2457]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'modules:config' at Sat, 18 May 2019 16:35:21 +0000. Up 13.74 seconds.
May 18 16:35:21 ctl01 systemd[1]: Started Apply the settings specified in cloud-config.
May 18 16:35:21 ctl01 systemd[1]: Started The Salt Minion.
May 18 16:35:21 ctl01 systemd[1]: Reached target Multi-User System.
May 18 16:35:21 ctl01 systemd[1]: Reached target Graphical Interface.
May 18 16:35:21 ctl01 systemd[1]: Starting Execute cloud user/final scripts...
May 18 16:35:21 ctl01 systemd[1]: Starting Update UTMP about System Runlevel Changes...
May 18 16:35:21 ctl01 systemd[1]: Started Update UTMP about System Runlevel Changes.
May 18 16:35:22 ctl01 cloud-init[2511]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 running 'modules:final' at Sat, 18 May 2019 16:35:22 +0000. Up 14.58 seconds.
May 18 16:35:22 ctl01 cloud-init[2511]: Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~16.04.1 finished at Sat, 18 May 2019 16:35:22 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 14.71 seconds
May 18 16:35:22 ctl01 systemd[1]: Started Execute cloud user/final scripts.
May 18 16:35:22 ctl01 systemd[1]: Reached target Cloud-init target.
May 18 16:35:22 ctl01 systemd[1]: Startup finished in 3.988s (kernel) + 10.770s (userspace) = 14.759s.
May 18 16:35:25 ctl01 snapd[2096]: daemon.go:611: gracefully waiting for running hooks
May 18 16:35:25 ctl01 snapd[2096]: daemon.go:613: done waiting for running hooks
May 18 16:35:25 ctl01 snapd[2096]: daemon stop requested to wait for socket activation
May 18 16:35:38 ctl01 kernel: [   30.563428] random: crng init done
May 18 16:35:38 ctl01 kernel: [   30.563433] random: 7 urandom warning(s) missed due to ratelimiting
May 18 16:35:42 ctl01 systemd[1]: Time has been changed
May 18 16:35:42 ctl01 systemd-timesyncd[577]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
May 18 16:35:48 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef dist-upgrade.
May 18 16:36:16 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install python-oauth python-m2crypto.
May 18 16:36:20 ctl01 systemd[1]: Reloading.
May 18 16:36:20 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:36:21 ctl01 salt-minion[2107]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
May 18 16:36:22 ctl01 salt-minion[2107]: ................................................................................................................................................................++
May 18 16:36:22 ctl01 salt-minion[2107]: ..++
May 18 16:36:24 ctl01 salt-minion[2107]: [WARNING ] State for file: /etc/kubernetes/ssl/ca-kubernetes.crt - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:36:24 ctl01 salt-minion[2107]: .....................................................++
May 18 16:36:24 ctl01 salt-minion[2107]: ..................................................++
May 18 16:36:25 ctl01 salt-minion[2107]: ...........++
May 18 16:36:25 ctl01 salt-minion[2107]: .............................................++
May 18 16:36:26 ctl01 salt-minion[2107]: [WARNING ] State for file: /var/lib/etcd/ca.pem - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
May 18 16:36:27 ctl01 salt-minion[2107]: ........................................................................................................++
May 18 16:36:27 ctl01 salt-minion[2107]: ............................................................................................................++
May 18 16:36:28 ctl01 salt-minion[2107]: ...........++
May 18 16:36:28 ctl01 salt-minion[2107]: ...................++
May 18 16:36:29 ctl01 salt-minion[2107]: ..................................................................................++
May 18 16:36:30 ctl01 salt-minion[2107]: ............................................++
May 18 16:36:30 ctl01 salt-minion[2107]: .................++
May 18 16:36:31 ctl01 salt-minion[2107]: .............................................................................................................................++
May 18 16:36:32 ctl01 salt-minion[2107]: ....................................++
May 18 16:36:34 ctl01 salt-minion[2107]: ...................................................................................................................................................................................................................................................................................................................................................................................................................++
May 18 16:36:35 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install ntp.
May 18 16:36:37 ctl01 systemd[1]: Reloading.
May 18 16:36:37 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:36:38 ctl01 systemd[1]: Reloading.
May 18 16:36:38 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:36:38 ctl01 systemd[1]: Reloading.
May 18 16:36:38 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:36:38 ctl01 systemd[1]: Starting LSB: Start NTP daemon...
May 18 16:36:38 ctl01 ntp[4697]:  * Starting NTP server ntpd
May 18 16:36:38 ctl01 ntpd[4716]: ntpd 4.2.8p4@1.3265-o Fri Jul  6 20:10:51 UTC 2018 (1): Starting
May 18 16:36:38 ctl01 ntpd[4716]: Command line: /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 112:116
May 18 16:36:38 ctl01 ntp[4697]:    ...done.
May 18 16:36:38 ctl01 systemd[1]: Started LSB: Start NTP daemon.
May 18 16:36:38 ctl01 ntpd[4720]: proto: precision = 0.061 usec (-24)
May 18 16:36:38 ctl01 ntpd[4720]: Listen and drop on 0 v6wildcard [::]:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen and drop on 1 v4wildcard 0.0.0.0:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 2 lo 127.0.0.1:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 3 ens3 192.168.11.21:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 4 br-mgmt 172.16.10.36:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 5 lo [::1]:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 6 ens3 [fe80::5054:ff:fe9d:addf%2]:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 7 ens4 [fe80::5054:ff:fef8:33c0%3]:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 8 ens5 [fe80::5054:ff:fe60:dbe8%4]:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 9 br-mgmt [fe80::5054:ff:fef8:33c0%6]:123
May 18 16:36:38 ctl01 ntpd[4720]: Listen normally on 10 ens5.1000 [fe80::5054:ff:fe60:dbe8%7]:123
May 18 16:36:38 ctl01 ntpd[4720]: Listening on routing socket on fd #27 for interface updates
May 18 16:36:38 ctl01 kernel: [   92.304928] kauditd_printk_skb: 3 callbacks suppressed
May 18 16:36:38 ctl01 kernel: [   92.304931] audit: type=1400 audit(1558197398.387:15): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/ntpd" pid=4729 comm="apparmor_parser"
May 18 16:36:38 ctl01 systemd[1]: Reloading.
May 18 16:36:38 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:36:39 ctl01 ntpd[4720]: Soliciting pool server 193.228.143.12
May 18 16:36:40 ctl01 ntpd[4720]: Soliciting pool server 83.168.200.198
May 18 16:36:40 ctl01 ntpd[4720]: Soliciting pool server 193.228.143.13
May 18 16:36:40 ctl01 systemd[1]: Started /bin/systemctl restart ntp.service.
May 18 16:36:40 ctl01 systemd[1]: Stopping LSB: Start NTP daemon...
May 18 16:36:40 ctl01 ntp[4863]:  * Stopping NTP server ntpd
May 18 16:36:40 ctl01 ntpd[4720]: ntpd exiting on signal 15 (Terminated)
May 18 16:36:40 ctl01 ntp[4863]:    ...done.
May 18 16:36:40 ctl01 ntpd[4720]: 193.228.143.12 local addr 192.168.11.21 -> <null>
May 18 16:36:40 ctl01 ntpd[4720]: 83.168.200.198 local addr 192.168.11.21 -> <null>
May 18 16:36:40 ctl01 ntpd[4720]: 193.228.143.13 local addr 192.168.11.21 -> <null>
May 18 16:36:40 ctl01 systemd[1]: Stopped LSB: Start NTP daemon.
May 18 16:36:40 ctl01 systemd[1]: Starting LSB: Start NTP daemon...
May 18 16:36:40 ctl01 ntp[4875]:  * Starting NTP server ntpd
May 18 16:36:40 ctl01 ntpd[4895]: ntpd 4.2.8p4@1.3265-o Fri Jul  6 20:10:51 UTC 2018 (1): Starting
May 18 16:36:40 ctl01 ntpd[4895]: Command line: /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 112:116
May 18 16:36:40 ctl01 ntp[4875]:    ...done.
May 18 16:36:40 ctl01 systemd[1]: Started LSB: Start NTP daemon.
May 18 16:36:40 ctl01 ntpd[4898]: proto: precision = 0.061 usec (-24)
May 18 16:36:40 ctl01 ntpd[4898]: restrict 0.0.0.0: KOD does nothing without LIMITED.
May 18 16:36:40 ctl01 ntpd[4898]: restrict ::: KOD does nothing without LIMITED.
May 18 16:36:40 ctl01 ntpd[4898]: switching logging to file /var/log/ntp.log
May 18 16:36:44 ctl01 salt-minion[2107]: [INFO    ] Executing command ['systemctl', 'status', 'salt-minion.service', '-n', '0'] in directory '/root'
May 18 16:36:44 ctl01 salt-minion[2107]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'restart', 'salt-minion.service'] in directory '/root'
May 18 16:36:44 ctl01 systemd[1]: Started /bin/systemctl restart salt-minion.service.
May 18 16:36:44 ctl01 systemd[1]: Stopping The Salt Minion...
May 18 16:36:44 ctl01 salt-minion[2107]: [WARNING ] Minion received a SIGTERM. Exiting.
May 18 16:36:44 ctl01 salt-minion[2107]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited.
May 18 16:36:44 ctl01 systemd[1]: Stopped The Salt Minion.
May 18 16:36:44 ctl01 systemd[1]: Starting The Salt Minion...
May 18 16:36:44 ctl01 systemd[1]: Started The Salt Minion.
May 18 16:36:44 ctl01 salt-minion[2107]: local:
May 18 16:36:44 ctl01 salt-minion[2107]:     True
May 18 16:36:45 ctl01 salt-minion[5014]: [INFO    ] Setting up the Salt Minion "ctl01.mcp-k8s-calico-noha.local"
May 18 16:36:45 ctl01 salt-minion[5014]: [INFO    ] Starting up the Salt Minion
May 18 16:36:45 ctl01 salt-minion[5014]: [INFO    ] Starting pull socket on /var/run/salt/minion/minion_event_1899fe9592_pull.ipc
May 18 16:36:45 ctl01 salt-minion[5014]: [INFO    ] Creating minion process manager
May 18 16:36:47 ctl01 salt-minion[5014]: [INFO    ] Executing command ['date', '+%z'] in directory '/root'
May 18 16:36:47 ctl01 salt-minion[5014]: [INFO    ] Updating job settings for scheduled job: __mine_interval
May 18 16:36:47 ctl01 salt-minion[5014]: [INFO    ] Added mine.update to scheduler
May 18 16:36:47 ctl01 salt-minion[5014]: [INFO    ] Minion is starting as user 'root'
May 18 16:36:47 ctl01 salt-minion[5014]: [INFO    ] Minion is ready to receive requests!
May 18 16:36:56 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190518163656898519
May 18 16:36:56 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 5105
May 18 16:36:58 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:36:58 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/server/service.sls'
May 18 16:36:58 ctl01 salt-minion[5014]: [INFO    ] Executing command '. /var/lib/etcd/configenv; etcdctl cluster-health > /dev/null 2>&1; echo $?' in directory '/root'
May 18 16:36:58 ctl01 salt-minion[5014]: [ERROR   ] Command '. /var/lib/etcd/configenv; etcdctl cluster-health > /dev/null 2>&1; echo $?' failed with return code: 2
May 18 16:36:58 ctl01 salt-minion[5014]: [ERROR   ] stdout: /bin/sh: 1: .: Can't open /var/lib/etcd/configenv
May 18 16:36:58 ctl01 salt-minion[5014]: [ERROR   ] retcode: 2
May 18 16:36:58 ctl01 salt-minion[5014]: [INFO    ] Running state [etcd_support_packages] at time 16:36:58.835804
May 18 16:36:58 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [etcd_support_packages]
May 18 16:36:58 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:36:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['apt-cache', '-q', 'policy', 'python-etcd'] in directory '/root'
May 18 16:36:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['apt-get', '-q', 'update'] in directory '/root'
May 18 16:37:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
May 18 16:37:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'python-etcd'] in directory '/root'
May 18 16:37:01 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install python-etcd.
May 18 16:37:03 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:37:03 ctl01 salt-minion[5014]: [INFO    ] Made the following changes:
May 18 16:37:03 ctl01 salt-minion[5014]: 'python-etcd' changed from 'absent' to '0.4.2-2'
May 18 16:37:03 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:37:03 ctl01 salt-minion[5014]: [INFO    ] Completed state [etcd_support_packages] at time 16:37:03.868276 duration_in_ms=5032.473
May 18 16:37:03 ctl01 salt-minion[5014]: [INFO    ] Running state [/tmp/etcd/bin] at time 16:37:03.871355
May 18 16:37:03 ctl01 salt-minion[5014]: [INFO    ] Executing state archive.extracted for [/tmp/etcd/bin]
May 18 16:37:06 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', 'x', '--strip=1', '-f', '/var/cache/salt/minion/extrn_files/base/github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz'] in directory '/tmp/etcd/bin/'
May 18 16:37:06 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
May 18 16:37:06 ctl01 salt-minion[5014]: [INFO    ] {'extracted_files': 'no tar output so far', 'directories_created': ['/tmp/etcd/bin/']}
May 18 16:37:06 ctl01 salt-minion[5014]: [INFO    ] Completed state [/tmp/etcd/bin] at time 16:37:06.853233 duration_in_ms=2981.87
May 18 16:37:06 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/local/bin/etcd] at time 16:37:06.854239
May 18 16:37:06 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/local/bin/etcd]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:07 ctl01 salt-minion[5014]: New file
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/local/bin/etcd] at time 16:37:07.175199 duration_in_ms=320.96
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/local/bin/etcdctl] at time 16:37:07.175636
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/local/bin/etcdctl]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:07 ctl01 salt-minion[5014]: New file
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/local/bin/etcdctl] at time 16:37:07.327660 duration_in_ms=152.024
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Running state [etcd] at time 16:37:07.329395
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Executing state user.present for [etcd]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] User etcd is present and up to date
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [etcd] at time 16:37:07.339579 duration_in_ms=10.184
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/etcd.service] at time 16:37:07.339816
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/etcd.service]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/systemd/etcd.service'
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:07 ctl01 salt-minion[5014]: New file
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/etcd.service] at time 16:37:07.376539 duration_in_ms=36.723
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/etcd] at time 16:37:07.376771
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/etcd]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/default'
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:07 ctl01 salt-minion[5014]: New file
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/etcd] at time 16:37:07.468607 duration_in_ms=91.836
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Running state [/var/lib/etcd/] at time 16:37:07.468867
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/var/lib/etcd/]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Directory /var/lib/etcd is in the correct state
May 18 16:37:07 ctl01 salt-minion[5014]: Directory /var/lib/etcd updated
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [/var/lib/etcd/] at time 16:37:07.473799 duration_in_ms=4.933
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Running state [/var/lib/etcd/configenv] at time 16:37:07.474334
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/var/lib/etcd/configenv]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/configenv'
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:07 ctl01 salt-minion[5014]: New file
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [/var/lib/etcd/configenv] at time 16:37:07.530796 duration_in_ms=56.462
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Running state [/var/lib/etcd/configenvv3] at time 16:37:07.531136
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/var/lib/etcd/configenvv3]
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/configenvv3'
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:07 ctl01 salt-minion[5014]: New file
May 18 16:37:07 ctl01 salt-minion[5014]: [INFO    ] Completed state [/var/lib/etcd/configenvv3] at time 16:37:07.582310 duration_in_ms=51.174
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Running state [etcd] at time 16:37:08.060307
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [etcd]
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'etcd.service', '-n', '0'] in directory '/root'
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 systemd[1]: Started /bin/systemctl start etcd.service.
May 18 16:37:08 ctl01 systemd[1]: Starting etcd - highly-available key value store...
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://172.16.10.36:4001
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_CERT_FILE=/var/lib/etcd/etcd-server.crt
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/default
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=5000
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=250
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.16.10.36:2380
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_INITIAL_CLUSTER=ctl01=https://172.16.10.36:2380
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=IN7KaRMSo3xkGxkjAAPtkRkAgqN4ZNRq
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_KEY_FILE=/var/lib/etcd/etcd-server.key
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://172.16.10.36:4001,http://127.0.0.1:4001
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://172.16.10.36:2380
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_NAME=ctl01
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_PEER_CERT_FILE=/var/lib/etcd/etcd-server.crt
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_PEER_KEY_FILE=/var/lib/etcd/etcd-server.key
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/var/lib/etcd/ca.pem
May 18 16:37:08 ctl01 etcd[5687]: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/var/lib/etcd/ca.pem
May 18 16:37:08 ctl01 etcd[5687]: setting maximum number of CPUs to 8, total number of available CPUs is 8
May 18 16:37:08 ctl01 etcd[5687]: peerTLS: cert = /var/lib/etcd/etcd-server.crt, key = /var/lib/etcd/etcd-server.key, ca = , trusted-ca = /var/lib/etcd/ca.pem, client-cert-auth = true, crl-file = 
May 18 16:37:08 ctl01 etcd[5687]: listening for peers on https://172.16.10.36:2380
May 18 16:37:08 ctl01 etcd[5687]: The scheme of client url http://127.0.0.1:4001 is HTTP while peer key/cert files are presented. Ignored key/cert files.
May 18 16:37:08 ctl01 etcd[5687]: The scheme of client url http://127.0.0.1:4001 is HTTP while client cert auth (--client-cert-auth) is enabled. Ignored client cert auth for this url.
May 18 16:37:08 ctl01 etcd[5687]: listening for client requests on 127.0.0.1:4001
May 18 16:37:08 ctl01 etcd[5687]: listening for client requests on 172.16.10.36:4001
May 18 16:37:08 ctl01 etcd[5687]: name = ctl01
May 18 16:37:08 ctl01 etcd[5687]: data dir = /var/lib/etcd/default
May 18 16:37:08 ctl01 etcd[5687]: member dir = /var/lib/etcd/default/member
May 18 16:37:08 ctl01 etcd[5687]: heartbeat = 250ms
May 18 16:37:08 ctl01 etcd[5687]: election = 5000ms
May 18 16:37:08 ctl01 etcd[5687]: snapshot count = 100000
May 18 16:37:08 ctl01 etcd[5687]: advertise client URLs = https://172.16.10.36:4001
May 18 16:37:08 ctl01 etcd[5687]: initial advertise peer URLs = https://172.16.10.36:2380
May 18 16:37:08 ctl01 etcd[5687]: initial cluster = ctl01=https://172.16.10.36:2380
May 18 16:37:08 ctl01 etcd[5687]: starting member 411212f5fea59a9f in cluster 8f5ffbb80be06f34
May 18 16:37:08 ctl01 etcd[5687]: 411212f5fea59a9f became follower at term 0
May 18 16:37:08 ctl01 etcd[5687]: newRaft 411212f5fea59a9f [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
May 18 16:37:08 ctl01 etcd[5687]: 411212f5fea59a9f became follower at term 1
May 18 16:37:08 ctl01 etcd[5687]: simple token is not cryptographically signed
May 18 16:37:08 ctl01 etcd[5687]: starting server... [version: 3.3.12, cluster version: to_be_decided]
May 18 16:37:08 ctl01 etcd[5687]: 411212f5fea59a9f as single-node; fast-forwarding 19 ticks (election ticks 20)
May 18 16:37:08 ctl01 etcd[5687]: added member 411212f5fea59a9f [https://172.16.10.36:2380] to cluster 8f5ffbb80be06f34
May 18 16:37:08 ctl01 etcd[5687]: ClientTLS: cert = /var/lib/etcd/etcd-server.crt, key = /var/lib/etcd/etcd-server.key, ca = , trusted-ca = /var/lib/etcd/ca.pem, client-cert-auth = true, crl-file = 
May 18 16:37:08 ctl01 etcd[5687]: 411212f5fea59a9f is starting a new election at term 1
May 18 16:37:08 ctl01 etcd[5687]: 411212f5fea59a9f became candidate at term 2
May 18 16:37:08 ctl01 etcd[5687]: 411212f5fea59a9f received MsgVoteResp from 411212f5fea59a9f at term 2
May 18 16:37:08 ctl01 etcd[5687]: 411212f5fea59a9f became leader at term 2
May 18 16:37:08 ctl01 etcd[5687]: raft.node: 411212f5fea59a9f elected leader 411212f5fea59a9f at term 2
May 18 16:37:08 ctl01 etcd[5687]: setting up the initial cluster version to 3.3
May 18 16:37:08 ctl01 etcd[5687]: set the initial cluster version to 3.3
May 18 16:37:08 ctl01 etcd[5687]: enabled capabilities for version 3.3
May 18 16:37:08 ctl01 etcd[5687]: published {Name:ctl01 ClientURLs:[https://172.16.10.36:4001]} to cluster 8f5ffbb80be06f34
May 18 16:37:08 ctl01 etcd[5687]: ready to serve client requests
May 18 16:37:08 ctl01 systemd[1]: Started etcd - highly-available key value store.
May 18 16:37:08 ctl01 etcd[5687]: ready to serve client requests
May 18 16:37:08 ctl01 etcd[5687]: serving insecure client requests on 127.0.0.1:4001, this is strongly discouraged!
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 etcd[5687]: serving client requests on 172.16.10.36:4001
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 systemd[1]: Started /bin/systemctl enable etcd.service.
May 18 16:37:08 ctl01 systemd[1]: Reloading.
May 18 16:37:08 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] {'etcd': True}
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Completed state [etcd] at time 16:37:08.927694 duration_in_ms=867.388
May 18 16:37:08 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163656898519
May 18 16:37:09 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command cmd.run with jid 20190518163709613169
May 18 16:37:09 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 5761
May 18 16:37:09 ctl01 salt-minion[5014]: [INFO    ] Executing command '. /var/lib/etcd/configenv && etcdctl cluster-health' in directory '/root'
May 18 16:37:09 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163709613169
May 18 16:37:10 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190518163710380922
May 18 16:37:10 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 5777
May 18 16:37:16 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:37:16 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/kube-addons.sls'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons] at time 16:37:17.205533
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/etc/kubernetes/addons]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] {'/etc/kubernetes/addons': 'New Dir'}
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons] at time 16:37:17.211971 duration_in_ms=6.438
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:37:17.212230
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-kube-controllers.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/calico/calico-kube-controllers.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:37:17.327067 duration_in_ms=114.836
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:37:17.327511
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-rbac.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/calico/calico-rbac.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:37:17.376392 duration_in_ms=48.88
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:37:17.376756
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-svc.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-svc.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:37:17.479926 duration_in_ms=103.169
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:37:17.480417
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-server.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-server.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:37:17.614822 duration_in_ms=134.406
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:37:17.615114
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-agent.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-agent.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:37:17.719790 duration_in_ms=104.676
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:37:17.720079
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-serviceaccount.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:37:17.748142 duration_in_ms=28.063
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:37:17.748405
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-roles.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-roles.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:37:17.777538 duration_in_ms=29.132
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:37:17.777858
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/prometheus/prometheus-roles.yml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/prometheus/prometheus-roles.yml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:37:17.809019 duration_in_ms=31.162
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/coredns] at time 16:37:17.809282
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/coredns]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/coredns is not present
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns] at time 16:37:17.810176 duration_in_ms=0.894
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true] at time 16:37:17.811231
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state cmd.run for [kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command 'kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] {'pid': 5858, 'retcode': 0, 'stderr': '/bin/sh: 1: kubectl: not found', 'stdout': 'coredns is absent. OK'}
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true] at time 16:37:17.824375 duration_in_ms=13.144
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-svc.yaml] at time 16:37:17.824699
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-svc.yaml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-svc.yaml'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:17 ctl01 salt-minion[5014]: New file
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-svc.yaml] at time 16:37:17.932141 duration_in_ms=107.441
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-rc.yaml] at time 16:37:17.932629
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-rc.yaml]
May 18 16:37:17 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-rc.yaml'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:18 ctl01 salt-minion[5014]: New file
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-rc.yaml] at time 16:37:18.081229 duration_in_ms=148.6
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-sa.yaml] at time 16:37:18.081592
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-sa.yaml]
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-sa.yaml'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:18 ctl01 salt-minion[5014]: New file
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-sa.yaml] at time 16:37:18.111043 duration_in_ms=29.451
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-autoscaler.yaml] at time 16:37:18.111286
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-autoscaler.yaml]
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-autoscaler.yaml'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:18 ctl01 salt-minion[5014]: New file
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-autoscaler.yaml] at time 16:37:18.219442 duration_in_ms=108.155
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-autoscaler-rbac.yaml] at time 16:37:18.219738
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-autoscaler-rbac.yaml]
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-autoscaler-rbac.yaml'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:18 ctl01 salt-minion[5014]: New file
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-autoscaler-rbac.yaml] at time 16:37:18.252156 duration_in_ms=32.417
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-clusterrole.yaml] at time 16:37:18.252493
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-clusterrole.yaml]
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-clusterrole.yaml'
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:18 ctl01 salt-minion[5014]: New file
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-clusterrole.yaml] at time 16:37:18.281689 duration_in_ms=29.196
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/metrics-server] at time 16:37:18.281923
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/metrics-server]
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/metrics-server is not present
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/metrics-server] at time 16:37:18.282708 duration_in_ms=0.785
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163710380922
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190518163718874370
May 18 16:37:18 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 5896
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/init.sls'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/calico.sls'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/service.sls'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/_common.sls'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/kube-proxy.sls'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/calicoctl] at time 16:37:20.726897
May 18 16:37:20 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/bin/calicoctl]
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:29 ctl01 salt-minion[5014]: New file
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/calicoctl] at time 16:37:29.126805 duration_in_ms=8399.908
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/birdcl] at time 16:37:29.127077
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/bin/birdcl]
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:29 ctl01 salt-minion[5014]: New file
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/birdcl] at time 16:37:29.474271 duration_in_ms=347.193
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] Running state [/opt/cni/bin/calico] at time 16:37:29.474531
May 18 16:37:29 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico]
May 18 16:37:33 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190518163733934997
May 18 16:37:33 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 5972
May 18 16:37:33 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163733934997
May 18 16:37:36 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:36 ctl01 salt-minion[5014]: New file
May 18 16:37:36 ctl01 salt-minion[5014]: [INFO    ] Completed state [/opt/cni/bin/calico] at time 16:37:36.991334 duration_in_ms=7516.801
May 18 16:37:36 ctl01 salt-minion[5014]: [INFO    ] Running state [/opt/cni/bin/calico-ipam] at time 16:37:36.991794
May 18 16:37:36 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico-ipam]
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:44 ctl01 salt-minion[5014]: New file
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Completed state [/opt/cni/bin/calico-ipam] at time 16:37:44.249616 duration_in_ms=7257.822
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/cni/net.d/10-calico.conf] at time 16:37:44.249886
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/cni/net.d/10-calico.conf]
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/calico.conf'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:44 ctl01 salt-minion[5014]: New file
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/cni/net.d/10-calico.conf] at time 16:37:44.368503 duration_in_ms=118.616
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/calico/network-environment] at time 16:37:44.368836
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/calico/network-environment]
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/network-environment.pool'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:44 ctl01 salt-minion[5014]: New file
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/calico/network-environment] at time 16:37:44.459415 duration_in_ms=90.58
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/calico/calicoctl.cfg] at time 16:37:44.459705
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/calico/calicoctl.cfg]
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/calicoctl.cfg.pool'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:44 ctl01 salt-minion[5014]: New file
May 18 16:37:44 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/calico/calicoctl.cfg] at time 16:37:44.547209 duration_in_ms=87.504
May 18 16:37:45 ctl01 salt-minion[5014]: [INFO    ] Running state [containerd=1.2.5-3~u16.04+mcp] at time 16:37:45.208868
May 18 16:37:45 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [containerd=1.2.5-3~u16.04+mcp]
May 18 16:37:45 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:37:45 ctl01 salt-minion[5014]: [INFO    ] Executing command ['apt-cache', '-q', 'policy', 'containerd=1.2.5-3~u16.04+mcp'] in directory '/root'
May 18 16:37:45 ctl01 salt-minion[5014]: [INFO    ] Executing command ['apt-get', '-q', 'update'] in directory '/root'
May 18 16:37:47 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
May 18 16:37:47 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'containerd=1.2.5-3~u16.04+mcp'] in directory '/root'
May 18 16:37:47 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install containerd=1.2.5-3~u16.04+mcp.
May 18 16:37:54 ctl01 systemd[1]: Reloading.
May 18 16:37:54 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:37:54 ctl01 systemd[1]: Reloading.
May 18 16:37:54 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:37:54 ctl01 systemd[1]: Starting containerd container runtime...
May 18 16:37:54 ctl01 systemd[1]: Started containerd container runtime.
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.802921221Z" level=info msg="starting containerd" revision= version=1.2.5-3
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.803511657Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.803653634Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.803834346Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.804108390Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
May 18 16:37:54 ctl01 kernel: [  168.754945] aufs 4.15-20180219
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.843604637Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.843774231Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.844037087Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.844362804Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.844405875Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.844498791Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.844523901Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887211874Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887313275Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887478525Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887561585Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887622542Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887682259Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887720585Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887752092Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887801702Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.887837306Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.888062860Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.888243314Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889049584Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889098384Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889171856Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889200953Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889226016Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889249864Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889274096Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889303039Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889328319Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889354176Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889382962Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889622517Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889660911Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889686688Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889711625Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.889813108Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:127.0.0.1 StreamServerPort:0 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.890034183Z" level=info msg="Connect containerd service"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.890469470Z" level=info msg="Get image filesystem path "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs""
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.891733112Z" level=info msg="Start subscribing containerd event"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.891884635Z" level=info msg="Start recovering state"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.892167350Z" level=info msg="Start event monitor"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.892229684Z" level=info msg="Start snapshots syncer"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.892261998Z" level=info msg="Start streaming server"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.891596420Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.892914217Z" level=info msg=serving... address="/run/containerd/containerd.sock"
May 18 16:37:54 ctl01 containerd[6838]: time="2019-05-18T16:37:54.900737383Z" level=info msg="containerd successfully booted in 0.090428s"
May 18 16:37:55 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:37:55 ctl01 salt-minion[5014]: [INFO    ] Made the following changes:
May 18 16:37:55 ctl01 salt-minion[5014]: 'containerd' changed from 'absent' to '1.2.5-3~u16.04+mcp'
May 18 16:37:55 ctl01 salt-minion[5014]: 'runc' changed from 'absent' to '1.0.0~rc7-2~u16.04+mcp'
May 18 16:37:55 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Completed state [containerd=1.2.5-3~u16.04+mcp] at time 16:37:56.034750 duration_in_ms=10825.882
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/containerd/config.toml] at time 16:37:56.041260
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/containerd/config.toml]
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/containerd/config.toml'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:56 ctl01 salt-minion[5014]: New file
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/containerd/config.toml] at time 16:37:56.147102 duration_in_ms=105.842
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Running state [containerd] at time 16:37:56.593620
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [containerd]
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'containerd.service', '-n', '0'] in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'containerd.service'] in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'containerd.service'] in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] The service containerd is already running
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Completed state [containerd] at time 16:37:56.628187 duration_in_ms=34.567
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Running state [containerd] at time 16:37:56.628449
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing state service.mod_watch for [containerd]
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'containerd.service'] in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'restart', 'containerd.service'] in directory '/root'
May 18 16:37:56 ctl01 systemd[1]: Started /bin/systemctl restart containerd.service.
May 18 16:37:56 ctl01 systemd[1]: Stopping containerd container runtime...
May 18 16:37:56 ctl01 containerd[6838]: time="2019-05-18T16:37:56.653456344Z" level=info msg="Stop CRI service"
May 18 16:37:56 ctl01 containerd[6838]: time="2019-05-18T16:37:56.653563474Z" level=info msg="Stop CRI service"
May 18 16:37:56 ctl01 containerd[6838]: time="2019-05-18T16:37:56.653598380Z" level=info msg="Event monitor stopped"
May 18 16:37:56 ctl01 systemd[1]: Stopped containerd container runtime.
May 18 16:37:56 ctl01 systemd[1]: Starting containerd container runtime...
May 18 16:37:56 ctl01 systemd[1]: Started containerd container runtime.
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] {'containerd': True}
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Completed state [containerd] at time 16:37:56.664615 duration_in_ms=36.166
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/calico-node.service] at time 16:37:56.665372
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/calico-node.service]
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/calico-node.service.ctr'
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.690408552Z" level=info msg="starting containerd" revision= version=1.2.5-3
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.690781319Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.690824435Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.690981699Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.691500497Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.693874487Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.693914134Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.693985753Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694149510Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694166736Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694184241Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694192403Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694277218Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694297655Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694351754Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694370026Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694384074Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694398474Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694413117Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694426620Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694440373Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694457527Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694500337Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694543034Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694966105Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.694992132Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695043214Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695065280Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695082080Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695096114Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695118698Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695140246Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695155390Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695168421Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695181036Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695231654Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695248881Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695262023Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695274459Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695343547Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:127.0.0.1 StreamServerPort:0 EnableSelinux:false SandboxImage:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695690962Z" level=info msg="Connect containerd service"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.695836041Z" level=info msg="Get image filesystem path "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs""
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.696414365Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.696492196Z" level=info msg="Start subscribing containerd event"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.696551078Z" level=info msg="Start recovering state"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.696650558Z" level=info msg="Start event monitor"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.696664584Z" level=info msg="Start snapshots syncer"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.696672188Z" level=info msg="Start streaming server"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.697543616Z" level=info msg=serving... address="/run/containerd/containerd.sock"
May 18 16:37:56 ctl01 containerd[6966]: time="2019-05-18T16:37:56.698039001Z" level=info msg="containerd successfully booted in 0.008078s"
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:37:56 ctl01 salt-minion[5014]: New file
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/calico-node.service] at time 16:37:56.705665 duration_in_ms=40.291
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Running state [/var/lib/calico] at time 16:37:56.705928
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/var/lib/calico]
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] {'/var/lib/calico': 'New Dir'}
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Completed state [/var/lib/calico] at time 16:37:56.707580 duration_in_ms=1.651
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Running state [/var/log/calico] at time 16:37:56.707802
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/var/log/calico]
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] {'/var/log/calico': 'New Dir'}
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Completed state [/var/log/calico] at time 16:37:56.709355 duration_in_ms=1.553
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Running state [calico-node] at time 16:37:56.711516
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [calico-node]
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'calico-node.service', '-n', '0'] in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'calico-node.service'] in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
May 18 16:37:56 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'calico-node.service'] in directory '/root'
May 18 16:37:56 ctl01 systemd[1]: Started /bin/systemctl start calico-node.service.
May 18 16:37:56 ctl01 systemd[1]: Starting calico-node...
May 18 16:37:56 ctl01 ctr[7018]: ctr: container "calico-node" in namespace "default": not found
May 18 16:37:56 ctl01 ctr[7038]: time="2019-05-18T16:37:56Z" level=error msg="failed to delete container "calico-node"" error="container "calico-node" in namespace "default": not found"
May 18 16:37:56 ctl01 ctr[7038]: ctr: container "calico-node" in namespace "default": not found
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.1 s                                                                        total:   0.0 B (0.0 B/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.2 s                                                                        total:   0.0 B (0.0 B/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.3 s                                                                        total:   0.0 B (0.0 B/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.4 s                                                                        total:   0.0 B (0.0 B/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.5 s                                                                        total:   0.0 B (0.0 B/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.3 KiB
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.6 s                                                                        total:   0.0 B (0.0 B/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/13.7 MiB
May 18 16:37:57 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/3.2 KiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        waiting        |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        waiting        |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/47.1 KiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        waiting        |#033[32m#033[0m--------------------------------------|
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.7 s                                                                        total:  1.3 Ki (1.9 KiB/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/13.7 MiB
May 18 16:37:57 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/3.2 KiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.9 MiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.8 MiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        downloading    |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m| 47.1 KiB/47.1 KiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/2.1 MiB
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.8 s                                                                        total:  48.4 K (60.4 KiB/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------| 151.8 Ki/13.7 MiB
May 18 16:37:57 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m#033[0m--------------------------------------|  9.0 KiB/1.9 MiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m#033[0m--------------------------------------| 40.1 KiB/1.8 MiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m#033[0m--------------------------------------|  8.0 KiB/2.1 MiB
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 0.9 s                                                                        total:  260.7  (288.8 KiB/s)
May 18 16:37:57 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+#033[0m-------------------------------------| 542.1 Ki/13.7 MiB
May 18 16:37:57 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++#033[0m------------------------------------| 105.2 Ki/1.9 MiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++#033[0m-----------------------------------| 181.5 Ki/1.8 MiB
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:57 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+#033[0m-------------------------------------| 64.2 KiB/2.1 MiB
May 18 16:37:57 ctl01 ctr[7050]: elapsed: 1.0 s                                                                        total:  944.7  (943.5 KiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++#033[0m------------------------------------| 813.6 Ki/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++#033[0m--------------------------------| 335.7 Ki/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++#033[0m--------------------------------| 328.6 Ki/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m++++#033[0m----------------------------------| 243.7 Ki/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.1 s                                                                        total:  1.7 Mi (1.6 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++#033[0m------------------------------------|  1.0 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++#033[0m-----------------------------| 462.9 Ki/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++#033[0m------------------------------| 426.2 Ki/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++#033[0m-------------------------| 772.6 Ki/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.2 s                                                                        total:  2.7 Mi (2.2 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++#033[0m-----------------------------------|  1.2 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++#033[0m---------------------------| 574.6 Ki/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++++#033[0m----------------------------| 522.3 Ki/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++#033[0m---------------------| 1010.2 K/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.3 s                                                                        total:  3.3 Mi (2.5 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++#033[0m-----------------------------------|  1.3 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++++#033[0m-------------------------| 701.9 Ki/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++#033[0m-------------------------| 646.8 Ki/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++#033[0m-----------------|  1.2 MiB/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.4 s                                                                        total:  3.9 Mi (2.8 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++#033[0m-----------------------------------|  1.4 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++#033[0m----------------------| 816.5 Ki/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++#033[0m-----------------------| 762.7 Ki/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++#033[0m-------------|  1.4 MiB/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.5 s                                                                        total:  4.4 Mi (2.9 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++#033[0m----------------------------------|  1.5 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++#033[0m--------------------| 932.4 Ki/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++++#033[0m---------------------| 877.3 Ki/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++++++#033[0m---------|  1.6 MiB/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.6 s                                                                        total:  5.0 Mi (3.1 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++#033[0m----------------------------------|  1.6 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++++#033[0m------------------| 1014.4 K/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++++++#033[0m-------------------| 976.2 Ki/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++++++++++#033[0m-----|  1.8 MiB/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.7 s                                                                        total:  5.5 Mi (3.2 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++#033[0m----------------------------------|  1.8 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++++++#033[0m----------------|  1.1 MiB/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++++++++++++++++#033[0m----------------|  1.1 MiB/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++++++++++++++#033[0m-|  2.1 MiB/2.1 MiB
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.8 s                                                                        total:  6.0 Mi (3.3 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++#033[0m---------------------------------|  1.9 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++++++++++++++++#033[0m-------------|  1.2 MiB/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++++++++++++++++++#033[0m--------------|  1.2 MiB/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 1.9 s                                                                        total:  6.5 Mi (3.4 MiB/s)
May 18 16:37:58 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++#033[0m---------------------------------|  2.0 MiB/13.7 MiB
May 18 16:37:58 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++++++++++++++++++#033[0m-----------|  1.4 MiB/1.9 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++++++++++++++#033[0m-----------|  1.3 MiB/1.8 MiB
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:58 ctl01 ctr[7050]: elapsed: 2.0 s                                                                        total:  6.9 Mi (3.4 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++#033[0m--------------------------------|  2.3 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++++++++++++++++++++++#033[0m-------|  1.5 MiB/1.9 MiB
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++++++++++++++++++#033[0m-------|  1.5 MiB/1.8 MiB
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.1 s                                                                        total:  7.5 Mi (3.6 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++#033[0m--------------------------------|  2.5 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++++++++++++++++++#033[0m----|  1.7 MiB/1.9 MiB
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++++++++++++++++++++++++++++++#033[0m--|  1.8 MiB/1.8 MiB
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.2 s                                                                        total:  8.1 Mi (3.7 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++#033[0m-------------------------------|  2.8 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|  1.9 MiB/1.9 MiB
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.3 s                                                                        total:  8.6 Mi (3.7 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++#033[0m------------------------------|  3.0 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.4 s                                                                        total:  8.8 Mi (3.7 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++#033[0m------------------------------|  3.1 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.5 s                                                                        total:  9.0 Mi (3.6 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++#033[0m-----------------------------|  3.3 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.6 s                                                                        total:  9.1 Mi (3.5 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++#033[0m-----------------------------|  3.4 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.7 s                                                                        total:  9.3 Mi (3.4 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++#033[0m-----------------------------|  3.6 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.8 s                                                                        total:  9.4 Mi (3.4 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++#033[0m----------------------------|  3.7 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 2.9 s                                                                        total:  9.6 Mi (3.3 MiB/s)
May 18 16:37:59 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++#033[0m----------------------------|  3.9 MiB/13.7 MiB
May 18 16:37:59 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:37:59 ctl01 ctr[7050]: elapsed: 3.0 s                                                                        total:  9.7 Mi (3.2 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++#033[0m---------------------------|  4.1 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.1 s                                                                        total:  9.9 Mi (3.2 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++#033[0m---------------------------|  4.2 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.2 s                                                                        total:  10.1 M (3.1 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++#033[0m--------------------------|  4.4 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.3 s                                                                        total:  10.2 M (3.1 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++#033[0m--------------------------|  4.5 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.4 s                                                                        total:  10.4 M (3.0 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++#033[0m--------------------------|  4.7 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.5 s                                                                        total:  10.5 M (3.0 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++#033[0m-------------------------|  4.8 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.6 s                                                                        total:  10.7 M (3.0 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++#033[0m-------------------------|  4.9 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.7 s                                                                        total:  10.8 M (2.9 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++#033[0m------------------------|  5.1 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.8 s                                                                        total:  10.9 M (2.9 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++#033[0m------------------------|  5.2 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 3.9 s                                                                        total:  11.0 M (2.8 MiB/s)
May 18 16:38:00 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++#033[0m------------------------|  5.3 MiB/13.7 MiB
May 18 16:38:00 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:00 ctl01 ctr[7050]: elapsed: 4.0 s                                                                        total:  11.2 M (2.8 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++#033[0m-----------------------|  5.5 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.1 s                                                                        total:  11.3 M (2.8 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++#033[0m-----------------------|  5.6 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.2 s                                                                        total:  11.5 M (2.7 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++#033[0m----------------------|  5.8 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.3 s                                                                        total:  11.7 M (2.7 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++#033[0m---------------------|  6.2 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.4 s                                                                        total:  12.1 M (2.7 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++#033[0m--------------------|  6.5 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.5 s                                                                        total:  12.4 M (2.7 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++#033[0m-------------------|  6.9 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.6 s                                                                        total:  12.7 M (2.8 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++#033[0m------------------|  7.3 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.7 s                                                                        total:  13.1 M (2.8 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++#033[0m-----------------|  7.6 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.8 s                                                                        total:  13.5 M (2.8 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++#033[0m----------------|  8.0 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 4.9 s                                                                        total:  13.9 M (2.8 MiB/s)
May 18 16:38:01 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++#033[0m---------------|  8.4 MiB/13.7 MiB
May 18 16:38:01 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:01 ctl01 ctr[7050]: elapsed: 5.0 s                                                                        total:  14.2 M (2.8 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++#033[0m--------------|  8.8 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.1 s                                                                        total:  14.6 M (2.9 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++#033[0m-------------|  9.1 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.2 s                                                                        total:  15.0 M (2.9 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++#033[0m------------|  9.5 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.3 s                                                                        total:  15.4 M (2.9 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++#033[0m-----------|  9.9 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.4 s                                                                        total:  15.7 M (2.9 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++#033[0m----------| 10.2 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.5 s                                                                        total:  16.1 M (2.9 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++++#033[0m---------| 10.6 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.6 s                                                                        total:  16.5 M (2.9 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++++#033[0m--------| 11.0 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.7 s                                                                        total:  16.8 M (3.0 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++++++#033[0m-------| 11.4 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.8 s                                                                        total:  17.2 M (3.0 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++++++#033[0m------| 11.7 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 5.9 s                                                                        total:  17.6 M (3.0 MiB/s)
May 18 16:38:02 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++++++++#033[0m-----| 12.2 MiB/13.7 MiB
May 18 16:38:02 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:02 ctl01 ctr[7050]: elapsed: 6.0 s                                                                        total:  18.0 M (3.0 MiB/s)
May 18 16:38:03 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++++++++++#033[0m---| 12.7 MiB/13.7 MiB
May 18 16:38:03 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: elapsed: 6.1 s                                                                        total:  18.6 M (3.0 MiB/s)
May 18 16:38:03 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++++++++++#033[0m--| 13.2 MiB/13.7 MiB
May 18 16:38:03 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: elapsed: 6.2 s                                                                        total:  19.1 M (3.1 MiB/s)
May 18 16:38:03 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++++++++++++#033[0m-| 13.7 MiB/13.7 MiB
May 18 16:38:03 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: elapsed: 6.3 s                                                                        total:  19.6 M (3.1 MiB/s)
May 18 16:38:03 ctl01 ctr[7050]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
May 18 16:38:03 ctl01 ctr[7050]: elapsed: 6.4 s                                                                        total:  19.6 M (3.1 MiB/s)
May 18 16:38:03 ctl01 ctr[7050]: unpacking linux/amd64 sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18...
May 18 16:38:04 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190518163804157635
May 18 16:38:04 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 7086
May 18 16:38:04 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163804157635
May 18 16:38:05 ctl01 etcd[5687]: proto: no coders for int
May 18 16:38:05 ctl01 etcd[5687]: proto: no encoder for ValueSize int [GetProperties]
May 18 16:38:05 ctl01 ctr[7050]: done
May 18 16:38:05 ctl01 systemd[1]: Started calico-node.
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'calico-node.service'] in directory '/root'
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'calico-node.service'] in directory '/root'
May 18 16:38:05 ctl01 systemd[1]: Started /bin/systemctl enable calico-node.service.
May 18 16:38:05 ctl01 systemd[1]: Reloading.
May 18 16:38:05 ctl01 containerd[6966]: time="2019-05-18T16:38:05.702568050Z" level=info msg="shim containerd-shim started" address="/containerd-shim/default/calico-node/shim.sock" debug=false pid=7127
May 18 16:38:05 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] {'calico-node': True}
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Completed state [calico-node] at time 16:38:05.796498 duration_in_ms=9084.981
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Running state [curl] at time 16:38:05.798324
May 18 16:38:05 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [curl]
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.826 [INFO][9] startup.go 264: Early log level set to info
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.826 [INFO][9] startup.go 280: Using NODENAME environment for node name
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.826 [INFO][9] startup.go 292: Determined node name: ctl01
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.881 [INFO][9] startup.go 105: Skipping datastore connection test
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.882 [INFO][9] startup.go 365: Building new node resource Name="ctl01"
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.883 [INFO][9] startup.go 380: Initialize BGP data
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.883 [INFO][9] startup.go 474: Using IPv4 address from environment: IP=172.16.10.36
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.883 [INFO][9] startup.go 507: IPv4 address 172.16.10.36 discovered on interface br-mgmt
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.883 [INFO][9] startup.go 450: Node IPv4 changed, will check for conflicts
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.884 [INFO][9] startup.go 640: Using AS number specified in environment (AS=64512)
May 18 16:38:05 ctl01 ctr[7095]: 2019-05-18 16:38:05.891 [INFO][9] startup.go 189: Using node name: ctl01
May 18 16:38:06 ctl01 ctr[7095]: Calico node started successfully
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Completed state [curl] at time 16:38:06.030413 duration_in_ms=232.088
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Running state [git] at time 16:38:06.030730
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [git]
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Completed state [git] at time 16:38:06.040296 duration_in_ms=9.565
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Running state [apt-transport-https] at time 16:38:06.040535
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [apt-transport-https]
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Completed state [apt-transport-https] at time 16:38:06.052504 duration_in_ms=11.969
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Running state [python-apt] at time 16:38:06.052749
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [python-apt]
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Completed state [python-apt] at time 16:38:06.062103 duration_in_ms=9.354
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Running state [socat] at time 16:38:06.062352
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [socat]
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
May 18 16:38:06 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'socat'] in directory '/root'
May 18 16:38:06 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install socat.
May 18 16:38:07 ctl01 kernel: [  181.133467] Netfilter messages via NETLINK v0.30.
May 18 16:38:07 ctl01 kernel: [  181.137691] ip_set: protocol 6
May 18 16:38:07 ctl01 kernel: [  181.340247] ip6_tables: (C) 2000-2006 Netfilter Core Team
May 18 16:38:07 ctl01 kernel: [  181.343145] ip_tables: (C) 2000-2006 Netfilter Core Team
May 18 16:38:08 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:38:08 ctl01 salt-minion[5014]: [INFO    ] Made the following changes:
May 18 16:38:08 ctl01 salt-minion[5014]: 'socat' changed from 'absent' to '1.7.3.1-1'
May 18 16:38:08 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:38:08 ctl01 salt-minion[5014]: [INFO    ] Completed state [socat] at time 16:38:08.571952 duration_in_ms=2509.599
May 18 16:38:08 ctl01 salt-minion[5014]: [INFO    ] Running state [openssl] at time 16:38:08.578113
May 18 16:38:08 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [openssl]
May 18 16:38:09 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:38:09 ctl01 salt-minion[5014]: [INFO    ] Completed state [openssl] at time 16:38:09.311256 duration_in_ms=733.142
May 18 16:38:09 ctl01 salt-minion[5014]: [INFO    ] Running state [conntrack] at time 16:38:09.311596
May 18 16:38:09 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [conntrack]
May 18 16:38:09 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
May 18 16:38:09 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'conntrack'] in directory '/root'
May 18 16:38:09 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install conntrack.
May 18 16:38:11 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:38:11 ctl01 salt-minion[5014]: [INFO    ] Made the following changes:
May 18 16:38:11 ctl01 salt-minion[5014]: 'conntrack' changed from 'absent' to '1:1.4.3-3'
May 18 16:38:11 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:38:11 ctl01 salt-minion[5014]: [INFO    ] Completed state [conntrack] at time 16:38:11.557457 duration_in_ms=2245.859
May 18 16:38:11 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/hyperkube] at time 16:38:11.562433
May 18 16:38:11 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/bin/hyperkube]
May 18 16:38:34 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190518163834237263
May 18 16:38:34 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 8203
May 18 16:38:34 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163834237263
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:43 ctl01 salt-minion[5014]: New file
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/hyperkube] at time 16:38:43.566390 duration_in_ms=32003.956
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/kubectl] at time 16:38:43.567323
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] Executing state file.symlink for [/usr/bin/kubectl]
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] {'new': '/usr/bin/kubectl'}
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/kubectl] at time 16:38:43.626283 duration_in_ms=58.959
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] Running state [/tmp/crictl] at time 16:38:43.630941
May 18 16:38:43 ctl01 salt-minion[5014]: [INFO    ] Executing state archive.extracted for [/tmp/crictl]
May 18 16:38:47 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', 'xz', '-f', '/var/cache/salt/minion/extrn_files/base/github.com/kubernetes-sigs/cri-tools/releases/download/v1.12.0/crictl-v1.12.0-linux-amd64.tar.gz'] in directory '/tmp/crictl/'
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] {'extracted_files': 'no tar output so far', 'directories_created': ['/tmp/crictl/']}
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/tmp/crictl] at time 16:38:48.118177 duration_in_ms=4487.229
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/local/bin/crictl] at time 16:38:48.119345
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/local/bin/crictl]
May 18 16:38:48 ctl01 salt-minion[5014]: [WARNING ] Use of argument owner found, "owner" is invalid, please use "user"
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:48 ctl01 salt-minion[5014]: New file
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/local/bin/crictl] at time 16:38:48.405744 duration_in_ms=286.399
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/crictl.yaml] at time 16:38:48.406018
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/crictl.yaml]
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:48 ctl01 salt-minion[5014]: New file
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/crictl.yaml] at time 16:38:48.434648 duration_in_ms=28.629
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/criproxy] at time 16:38:48.434978
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/criproxy]
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] File /etc/criproxy is not present
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/criproxy] at time 16:38:48.436087 duration_in_ms=1.108
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Running state [criproxy] at time 16:38:48.919172
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing state service.dead for [criproxy]
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'criproxy.service', '-n', '0'] in directory '/root'
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] The named service criproxy is not available
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [criproxy] at time 16:38:48.933953 duration_in_ms=14.782
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kubelet.service] at time 16:38:48.934311
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kubelet.service]
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kubelet.service'
May 18 16:38:48 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:49 ctl01 salt-minion[5014]: New file
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kubelet.service] at time 16:38:49.244939 duration_in_ms=310.627
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/config] at time 16:38:49.245269
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/kubernetes/config]
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/config is not present
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/config] at time 16:38:49.246860 duration_in_ms=1.591
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kubelet] at time 16:38:49.247309
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/kubelet]
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kubelet/default.master'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:49 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:50 ctl01 salt-minion[5014]: New file
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kubelet] at time 16:38:50.126675 duration_in_ms=879.365
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/kubelet.kubeconfig] at time 16:38:50.127171
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/kubelet.kubeconfig]
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kubelet/kubelet.kubeconfig.master'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:50 ctl01 salt-minion[5014]: New file
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/kubelet.kubeconfig] at time 16:38:50.748950 duration_in_ms=621.777
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/manifests] at time 16:38:50.749651
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/etc/kubernetes/manifests]
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] {'/etc/kubernetes/manifests': 'New Dir'}
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/manifests] at time 16:38:50.752392 duration_in_ms=2.742
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Running state [kubelet] at time 16:38:50.754655
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [kubelet]
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kubelet.service', '-n', '0'] in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 systemd[1]: Started /bin/systemctl start kubelet.service.
May 18 16:38:50 ctl01 systemd[1]: Started Kubernetes Kubelet Server.
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 systemd[1]: Started /bin/systemctl enable kubelet.service.
May 18 16:38:50 ctl01 systemd[1]: Reloading.
May 18 16:38:50 ctl01 kubelet[8501]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 18 16:38:50 ctl01 kubelet[8501]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 18 16:38:50 ctl01 kubelet[8501]: Flag --allow-privileged has been deprecated, will be removed in a future version
May 18 16:38:50 ctl01 kubelet[8501]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 18 16:38:50 ctl01 kubelet[8501]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 18 16:38:50 ctl01 kubelet[8501]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943343    8501 flags.go:33] FLAG: --address="172.16.10.36"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943422    8501 flags.go:33] FLAG: --allow-privileged="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943431    8501 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943445    8501 flags.go:33] FLAG: --alsologtostderr="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943460    8501 flags.go:33] FLAG: --anonymous-auth="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943471    8501 flags.go:33] FLAG: --application-metrics-count-limit="100"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943482    8501 flags.go:33] FLAG: --authentication-token-webhook="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943493    8501 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943507    8501 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943516    8501 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943522    8501 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943528    8501 flags.go:33] FLAG: --azure-container-registry-config=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943534    8501 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943541    8501 flags.go:33] FLAG: --bootstrap-checkpoint-path=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943553    8501 flags.go:33] FLAG: --bootstrap-kubeconfig=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943558    8501 flags.go:33] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943565    8501 flags.go:33] FLAG: --cgroup-driver="cgroupfs"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943570    8501 flags.go:33] FLAG: --cgroup-root=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943576    8501 flags.go:33] FLAG: --cgroups-per-qos="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943581    8501 flags.go:33] FLAG: --chaos-chance="0"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943590    8501 flags.go:33] FLAG: --client-ca-file=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943595    8501 flags.go:33] FLAG: --cloud-config=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943600    8501 flags.go:33] FLAG: --cloud-provider=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943606    8501 flags.go:33] FLAG: --cluster-dns="[10.254.0.10]"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943615    8501 flags.go:33] FLAG: --cluster-domain="cluster.local"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943621    8501 flags.go:33] FLAG: --cni-bin-dir="/opt/cni/bin"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943627    8501 flags.go:33] FLAG: --cni-conf-dir="/etc/cni/net.d"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943633    8501 flags.go:33] FLAG: --config=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943639    8501 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943645    8501 flags.go:33] FLAG: --container-log-max-files="5"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943653    8501 flags.go:33] FLAG: --container-log-max-size="10Mi"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943659    8501 flags.go:33] FLAG: --container-runtime="remote"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943665    8501 flags.go:33] FLAG: --container-runtime-endpoint="unix:///run/containerd/containerd.sock"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943677    8501 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943684    8501 flags.go:33] FLAG: --containerized="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943690    8501 flags.go:33] FLAG: --contention-profiling="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943696    8501 flags.go:33] FLAG: --cpu-cfs-quota="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943701    8501 flags.go:33] FLAG: --cpu-cfs-quota-period="100ms"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943707    8501 flags.go:33] FLAG: --cpu-manager-policy="none"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943713    8501 flags.go:33] FLAG: --cpu-manager-reconcile-period="10s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943719    8501 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943725    8501 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943732    8501 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943737    8501 flags.go:33] FLAG: --docker-only="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943743    8501 flags.go:33] FLAG: --docker-root="/var/lib/docker"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943749    8501 flags.go:33] FLAG: --docker-tls="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943755    8501 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943760    8501 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943766    8501 flags.go:33] FLAG: --docker-tls-key="key.pem"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943771    8501 flags.go:33] FLAG: --dynamic-config-dir=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943779    8501 flags.go:33] FLAG: --enable-controller-attach-detach="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943784    8501 flags.go:33] FLAG: --enable-debugging-handlers="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943790    8501 flags.go:33] FLAG: --enable-load-reader="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943796    8501 flags.go:33] FLAG: --enable-server="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943801    8501 flags.go:33] FLAG: --enforce-node-allocatable="[pods]"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943812    8501 flags.go:33] FLAG: --event-burst="10"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943818    8501 flags.go:33] FLAG: --event-qps="5"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943824    8501 flags.go:33] FLAG: --event-storage-age-limit="default=0"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943830    8501 flags.go:33] FLAG: --event-storage-event-limit="default=0"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943836    8501 flags.go:33] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943854    8501 flags.go:33] FLAG: --eviction-max-pod-grace-period="0"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943861    8501 flags.go:33] FLAG: --eviction-minimum-reclaim=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943869    8501 flags.go:33] FLAG: --eviction-pressure-transition-period="5m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943876    8501 flags.go:33] FLAG: --eviction-soft=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943882    8501 flags.go:33] FLAG: --eviction-soft-grace-period=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943893    8501 flags.go:33] FLAG: --exit-on-lock-contention="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943900    8501 flags.go:33] FLAG: --experimental-allocatable-ignore-eviction="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943906    8501 flags.go:33] FLAG: --experimental-bootstrap-kubeconfig=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943911    8501 flags.go:33] FLAG: --experimental-check-node-capabilities-before-mount="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943917    8501 flags.go:33] FLAG: --experimental-dockershim="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943923    8501 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943929    8501 flags.go:33] FLAG: --experimental-fail-swap-on="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943934    8501 flags.go:33] FLAG: --experimental-kernel-memcg-notification="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943940    8501 flags.go:33] FLAG: --experimental-mounter-path=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943946    8501 flags.go:33] FLAG: --fail-swap-on="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943951    8501 flags.go:33] FLAG: --feature-gates=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943959    8501 flags.go:33] FLAG: --file-check-frequency="20s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943965    8501 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943971    8501 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943977    8501 flags.go:33] FLAG: --healthz-bind-address="127.0.0.1"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943983    8501 flags.go:33] FLAG: --healthz-port="10248"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943989    8501 flags.go:33] FLAG: --help="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.943995    8501 flags.go:33] FLAG: --host-ipc-sources="[*]"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944008    8501 flags.go:33] FLAG: --host-network-sources="[*]"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944017    8501 flags.go:33] FLAG: --host-pid-sources="[*]"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944024    8501 flags.go:33] FLAG: --hostname-override="ctl01"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944030    8501 flags.go:33] FLAG: --housekeeping-interval="10s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944036    8501 flags.go:33] FLAG: --http-check-frequency="20s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944042    8501 flags.go:33] FLAG: --image-gc-high-threshold="85"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944048    8501 flags.go:33] FLAG: --image-gc-low-threshold="80"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944053    8501 flags.go:33] FLAG: --image-pull-progress-deadline="1m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944059    8501 flags.go:33] FLAG: --image-service-endpoint=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944064    8501 flags.go:33] FLAG: --iptables-drop-bit="15"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944070    8501 flags.go:33] FLAG: --iptables-masquerade-bit="14"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944075    8501 flags.go:33] FLAG: --keep-terminated-pod-volumes="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944081    8501 flags.go:33] FLAG: --kube-api-burst="10"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944086    8501 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944097    8501 flags.go:33] FLAG: --kube-api-qps="5"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944103    8501 flags.go:33] FLAG: --kube-reserved=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944110    8501 flags.go:33] FLAG: --kube-reserved-cgroup=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944115    8501 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/kubelet.kubeconfig"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944122    8501 flags.go:33] FLAG: --kubelet-cgroups=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944127    8501 flags.go:33] FLAG: --lock-file=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944132    8501 flags.go:33] FLAG: --log-backtrace-at=":0"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944139    8501 flags.go:33] FLAG: --log-cadvisor-usage="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944145    8501 flags.go:33] FLAG: --log-dir=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944150    8501 flags.go:33] FLAG: --log-file=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944156    8501 flags.go:33] FLAG: --log-flush-frequency="5s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944162    8501 flags.go:33] FLAG: --logtostderr="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944167    8501 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944174    8501 flags.go:33] FLAG: --make-iptables-util-chains="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944179    8501 flags.go:33] FLAG: --manifest-url=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944185    8501 flags.go:33] FLAG: --manifest-url-header=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944194    8501 flags.go:33] FLAG: --master-service-namespace="default"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944200    8501 flags.go:33] FLAG: --max-open-files="1000000"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944213    8501 flags.go:33] FLAG: --max-pods="110"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944220    8501 flags.go:33] FLAG: --maximum-dead-containers="-1"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944226    8501 flags.go:33] FLAG: --maximum-dead-containers-per-container="1"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944231    8501 flags.go:33] FLAG: --minimum-container-ttl-duration="0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944237    8501 flags.go:33] FLAG: --minimum-image-ttl-duration="2m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944243    8501 flags.go:33] FLAG: --network-plugin="cni"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944248    8501 flags.go:33] FLAG: --network-plugin-mtu="0"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944258    8501 flags.go:33] FLAG: --node-ip="172.16.10.36"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944269    8501 flags.go:33] FLAG: --node-labels="node-role.kubernetes.io/master=true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944287    8501 flags.go:33] FLAG: --node-status-max-images="50"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944297    8501 flags.go:33] FLAG: --node-status-update-frequency="10s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944308    8501 flags.go:33] FLAG: --non-masquerade-cidr="10.0.0.0/8"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944318    8501 flags.go:33] FLAG: --oom-score-adj="-999"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944325    8501 flags.go:33] FLAG: --pod-cidr=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944336    8501 flags.go:33] FLAG: --pod-infra-container-image="docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944346    8501 flags.go:33] FLAG: --pod-manifest-path="/etc/kubernetes/manifests"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944352    8501 flags.go:33] FLAG: --pod-max-pids="-1"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944358    8501 flags.go:33] FLAG: --pods-per-core="0"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944364    8501 flags.go:33] FLAG: --port="10250"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944370    8501 flags.go:33] FLAG: --protect-kernel-defaults="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944375    8501 flags.go:33] FLAG: --provider-id=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944381    8501 flags.go:33] FLAG: --qos-reserved=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944387    8501 flags.go:33] FLAG: --read-only-port="10255"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944393    8501 flags.go:33] FLAG: --really-crash-for-testing="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944399    8501 flags.go:33] FLAG: --redirect-container-streaming="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944404    8501 flags.go:33] FLAG: --register-node="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944410    8501 flags.go:33] FLAG: --register-schedulable="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944415    8501 flags.go:33] FLAG: --register-with-taints=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944423    8501 flags.go:33] FLAG: --registry-burst="10"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944428    8501 flags.go:33] FLAG: --registry-qps="5"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944434    8501 flags.go:33] FLAG: --resolv-conf="/etc/resolv.conf"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944440    8501 flags.go:33] FLAG: --root-dir="/var/lib/kubelet"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944450    8501 flags.go:33] FLAG: --rotate-certificates="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944456    8501 flags.go:33] FLAG: --rotate-server-certificates="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944462    8501 flags.go:33] FLAG: --runonce="false"
May 18 16:38:50 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944468    8501 flags.go:33] FLAG: --runtime-cgroups=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944473    8501 flags.go:33] FLAG: --runtime-request-timeout="2m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944479    8501 flags.go:33] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944485    8501 flags.go:33] FLAG: --serialize-image-pulls="true"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944490    8501 flags.go:33] FLAG: --stderrthreshold="2"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944496    8501 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944502    8501 flags.go:33] FLAG: --storage-driver-db="cadvisor"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944508    8501 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944514    8501 flags.go:33] FLAG: --storage-driver-password="root"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944520    8501 flags.go:33] FLAG: --storage-driver-secure="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944526    8501 flags.go:33] FLAG: --storage-driver-table="stats"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944536    8501 flags.go:33] FLAG: --storage-driver-user="root"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944542    8501 flags.go:33] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944548    8501 flags.go:33] FLAG: --sync-frequency="1m0s"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944554    8501 flags.go:33] FLAG: --system-cgroups=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944559    8501 flags.go:33] FLAG: --system-reserved=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944566    8501 flags.go:33] FLAG: --system-reserved-cgroup=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944571    8501 flags.go:33] FLAG: --tls-cert-file=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944576    8501 flags.go:33] FLAG: --tls-cipher-suites="[]"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944586    8501 flags.go:33] FLAG: --tls-min-version=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944592    8501 flags.go:33] FLAG: --tls-private-key-file=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944598    8501 flags.go:33] FLAG: --v="2"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944603    8501 flags.go:33] FLAG: --version="false"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944612    8501 flags.go:33] FLAG: --vmodule=""
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944618    8501 flags.go:33] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944625    8501 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
May 18 16:38:50 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944657    8501 feature_gate.go:206] feature gates: &{map[]}
May 18 16:38:50 ctl01 kubelet[8501]: W0518 16:38:50.944677    8501 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
May 18 16:38:50 ctl01 kubelet[8501]: W0518 16:38:50.944691    8501 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
May 18 16:38:50 ctl01 kubelet[8501]: W0518 16:38:50.944702    8501 server.go:182] Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead
May 18 16:38:50 ctl01 kubelet[8501]: I0518 16:38:50.944752    8501 feature_gate.go:206] feature gates: &{map[]}
May 18 16:38:50 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] {'kubelet': True}
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Completed state [kubelet] at time 16:38:50.998394 duration_in_ms=243.738
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/logrotate.d/kubernetes] at time 16:38:50.998815
May 18 16:38:50 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/logrotate.d/kubernetes]
May 18 16:38:51 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/logrotate'
May 18 16:38:51 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:51 ctl01 salt-minion[5014]: New file
May 18 16:38:51 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/logrotate.d/kubernetes] at time 16:38:51.028621 duration_in_ms=29.806
May 18 16:38:51 ctl01 salt-minion[5014]: [INFO    ] Running state [/opt/cni/bin] at time 16:38:51.028932
May 18 16:38:51 ctl01 salt-minion[5014]: [INFO    ] Executing state archive.extracted for [/opt/cni/bin]
May 18 16:38:51 ctl01 systemd[1]: Started Kubernetes systemd probe.
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.931536    8501 mount_linux.go:180] Detected OS with systemd
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.932038    8501 server.go:407] Version: v1.13.5-3+98374c02d2d8c1
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.932435    8501 feature_gate.go:206] feature gates: &{map[]}
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.932879    8501 feature_gate.go:206] feature gates: &{map[]}
May 18 16:38:51 ctl01 kubelet[8501]: W0518 16:38:51.933317    8501 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
May 18 16:38:51 ctl01 kubelet[8501]: W0518 16:38:51.933769    8501 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.934621    8501 plugins.go:103] No cloud provider specified.
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.935016    8501 server.go:523] No cloud provider specified: "" from the config file: ""
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.941818    8501 manager.go:155] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.943147    8501 fs.go:142] Filesystem UUIDs: map[2019-05-18-18-27-50-00:/dev/sr0 1bbe14b0-e1b4-4636-83ed-ba0fa2a5acca:/dev/vda1]
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.943465    8501 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /dev/vda1:{mountpoint:/ major:252 minor:1 fsType:ext4 blockSize:0}]
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.948964    8501 manager.go:229] Machine: {NumCores:8 CpuFrequency:2799994 MemoryCapacity:14704910336 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:4ecec05e06d644b28e1705daa5bcdc49 SystemUUID:7E8D7D3A-48AC-482E-984E-F72F98E53F3A BootID:a59f6ad2-b116-4960-96df-a733b7cb1bd8 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:24 Capacity:1470492672 Type:vfs Inodes:1795033 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:104022159360 Type:vfs Inodes:12800000 HasInodes:true}] DiskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:br-mgmt MacAddress:52:54:00:f8:33:c0 Speed:0 Mtu:9000} {Name:ens3 MacAddress:52:54:00:9d:ad:df Speed:-1 Mtu:9000} {Name:ens4 MacAddress:52:54:00:f8:33:c0 Speed:-1 Mtu:9000} {Name:ens5 MacAddress:52:54:00:60:db:e8 Speed:-1 Mtu:9000} {Name:ens5.1000 MacAddress:52:54:00:60:db:e8 Speed:-1 Mtu:9000} {Name:ens6 MacAddress:52:54:00:8b:60:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:14704910336 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:2 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:3 Memory:0 Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:4 Memory:0 Cores:[{Id:0 Threads:[4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:5 Memory:0 Cores:[{Id:0 Threads:[5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:6 Memory:0 Cores:
May 18 16:38:51 ctl01 kubelet[8501]: [{Id:0 Threads:[6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:7 Memory:0 Cores:[{Id:0 Threads:[7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.949833    8501 manager.go:235] Version: {KernelVersion:4.15.0-50-generic ContainerOsVersion:Ubuntu 16.04.6 LTS DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:}
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.949946    8501 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.950228    8501 container_manager_linux.go:248] container manager verified user specified cgroup-root exists: []
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.950259    8501 container_manager_linux.go:253] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms}
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.950410    8501 container_manager_linux.go:272] Creating device plugin manager: true
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.950422    8501 manager.go:109] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.950512    8501 state_mem.go:36] [cpumanager] initializing new in-memory state store
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.965086    8501 server.go:941] Using root directory: /var/lib/kubelet
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.965129    8501 kubelet.go:281] Adding pod path: /etc/kubernetes/manifests
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.965172    8501 file.go:68] Watching path "/etc/kubernetes/manifests"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.965188    8501 kubelet.go:306] Watching apiserver
May 18 16:38:51 ctl01 kubelet[8501]: E0518 16:38:51.966199    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:51 ctl01 kubelet[8501]: E0518 16:38:51.966381    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:51 ctl01 kubelet[8501]: E0518 16:38:51.966617    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.973624    8501 kuberuntime_manager.go:198] Container runtime containerd initialized, version: 1.2.5-3, apiVersion: v1alpha2
May 18 16:38:51 ctl01 kubelet[8501]: W0518 16:38:51.973977    8501 probe.go:271] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974253    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974274    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/empty-dir"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974284    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974294    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/git-repo"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974346    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/host-path"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974356    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/nfs"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974375    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/secret"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974385    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/iscsi"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974398    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974413    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974424    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974448    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/quobyte"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974460    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/cephfs"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974475    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/downward-api"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974486    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974497    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/flocker"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974509    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974521    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/configmap"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974533    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974545    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974557    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974569    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/projected"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974592    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974605    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974623    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/local-volume"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974651    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.974704    8501 plugins.go:547] Loaded volume plugin "kubernetes.io/csi"
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.975002    8501 server.go:999] Started kubelet
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.975229    8501 server.go:137] Starting to listen on 172.16.10.36:10250
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.975070    8501 server.go:157] Starting to listen read-only on 172.16.10.36:10255
May 18 16:38:51 ctl01 kubelet[8501]: E0518 16:38:51.975759    8501 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.976202    8501 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.976274    8501 status_manager.go:152] Starting to sync pod status with apiserver
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.976301    8501 kubelet.go:1829] Starting kubelet main sync loop.
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.976345    8501 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg has yet to be successful]
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.977137    8501 volume_manager.go:246] The desired_state_of_world populator starts
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.977148    8501 volume_manager.go:248] Starting Kubelet Volume Manager
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.978021    8501 desired_state_of_world_populator.go:130] Desired state populator starts to run
May 18 16:38:51 ctl01 kubelet[8501]: E0518 16:38:51.978593    8501 cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
May 18 16:38:51 ctl01 kubelet[8501]: E0518 16:38:51.978611    8501 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.979459    8501 server.go:333] Adding debug handlers to kubelet server.
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.981485    8501 factory.go:136] Registering containerd factory
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.983286    8501 factory.go:54] Registering systemd factory
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.983746    8501 factory.go:97] Registering Raw factory
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.984157    8501 manager.go:1222] Started watching for new ooms in manager
May 18 16:38:51 ctl01 kubelet[8501]: I0518 16:38:51.986062    8501 manager.go:365] Starting recovery of all containers
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.035673    8501 manager.go:370] Recovery completed
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.076536    8501 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet]
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.078946    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.078996    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.079436    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.080650    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.080717    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.080781    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.080832    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.081229    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.093764    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.094086    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.094930    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.094969    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.095010    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.095087    8501 cpu_manager.go:155] [cpumanager] starting with none policy
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.095100    8501 cpu_manager.go:156] [cpumanager] reconciling every 10s
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.095118    8501 policy_none.go:42] [cpumanager] none policy: Start
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.095575    8501 container_manager_linux.go:376] Updating kernel flag: kernel/panic, expected value: 10, actual value: 60
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.095650    8501 container_manager_linux.go:376] Updating kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.095821    8501 container_manager_linux.go:376] Updating kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.111260    8501 manager.go:196] Starting Device Plugin manager
May 18 16:38:52 ctl01 kubelet[8501]: W0518 16:38:52.111306    8501 manager.go:537] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.111511    8501 manager.go:231] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock"
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.111580    8501 plugin_watcher.go:90] Plugin Watcher Start at /var/lib/kubelet/plugins_registry
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.112000    8501 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.179174    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.276805    8501 kubelet.go:1908] SyncLoop (ADD, "file"): ""
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.279398    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.281454    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.281747    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.282581    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.282610    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.282622    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.282640    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.283007    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.379576    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.479713    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.580012    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.680150    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.683174    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.683540    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.684506    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.684565    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.684585    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: I0518 16:38:52.684616    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.685040    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.780358    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.880510    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.966959    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.968145    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.969321    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:52 ctl01 kubelet[8501]: E0518 16:38:52.980661    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.080848    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.181009    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.281230    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.381436    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.481587    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: I0518 16:38:53.485208    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:38:53 ctl01 kubelet[8501]: I0518 16:38:53.485504    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:38:53 ctl01 kubelet[8501]: I0518 16:38:53.486761    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:38:53 ctl01 kubelet[8501]: I0518 16:38:53.486903    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:38:53 ctl01 kubelet[8501]: I0518 16:38:53.486933    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:38:53 ctl01 kubelet[8501]: I0518 16:38:53.487018    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.487927    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.581733    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.682217    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.782801    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.883056    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.967663    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.968570    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.969776    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:53 ctl01 kubelet[8501]: E0518 16:38:53.983217    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.083358    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.183574    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.283785    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.383977    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.484170    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.584313    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.684521    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.784686    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.884850    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.968582    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.969351    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.970536    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:54 ctl01 kubelet[8501]: E0518 16:38:54.984982    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.085241    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: I0518 16:38:55.088187    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:38:55 ctl01 kubelet[8501]: I0518 16:38:55.088514    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:38:55 ctl01 kubelet[8501]: I0518 16:38:55.089933    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:38:55 ctl01 kubelet[8501]: I0518 16:38:55.090009    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:38:55 ctl01 kubelet[8501]: I0518 16:38:55.090044    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:38:55 ctl01 kubelet[8501]: I0518 16:38:55.090125    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.090758    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.185455    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.285663    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.385947    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.486218    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.586488    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.686692    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.786919    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.887193    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.969363    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.970209    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.971564    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:55 ctl01 kubelet[8501]: E0518 16:38:55.987388    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.087745    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.187957    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.288209    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.388479    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.488641    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.588811    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.688957    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.789219    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.889524    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.970168    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.970965    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.972066    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:56 ctl01 kubelet[8501]: E0518 16:38:56.989727    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.089918    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.175007    8501 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.190132    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.290305    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.390536    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.490723    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.590932    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.691103    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.791306    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', 'xz', '-f', '/var/cache/salt/minion/extrn_files/base/docker-prod-local.artifactory.mirantis.com/artifactory/binary-prod-local/mirantis/kubernetes/containernetworking-plugins/containernetworking-plugins_v0.7.2-173-g8db2808.tar.gz'] in directory '/opt/cni/bin/'
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.891548    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.971067    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.972004    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.972887    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:57 ctl01 kubelet[8501]: E0518 16:38:57.991751    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.091904    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.192147    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 kubelet[8501]: I0518 16:38:58.290979    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:38:58 ctl01 kubelet[8501]: I0518 16:38:58.291265    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:38:58 ctl01 kubelet[8501]: I0518 16:38:58.292170    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:38:58 ctl01 kubelet[8501]: I0518 16:38:58.292228    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:38:58 ctl01 kubelet[8501]: I0518 16:38:58.292242    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:38:58 ctl01 kubelet[8501]: I0518 16:38:58.292269    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.292297    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.292607    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.392547    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] {'extracted_files': 'no tar output so far'}
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Completed state [/opt/cni/bin] at time 16:38:58.419176 duration_in_ms=7390.238
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/proxy.kubeconfig] at time 16:38:58.419547
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/proxy.kubeconfig]
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-proxy/proxy.kubeconfig'
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.492768    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.592980    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.693130    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.793341    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.893492    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.971984    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.972844    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.973727    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:58 ctl01 kubelet[8501]: E0518 16:38:58.993681    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:59 ctl01 salt-minion[5014]: New file
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/proxy.kubeconfig] at time 16:38:59.008462 duration_in_ms=588.913
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kube-proxy.service] at time 16:38:59.008883
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-proxy.service]
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-proxy.service'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:59 ctl01 salt-minion[5014]: New file
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kube-proxy.service] at time 16:38:59.039921 duration_in_ms=31.038
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kube-proxy] at time 16:38:59.040158
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/kube-proxy]
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:38:59 ctl01 salt-minion[5014]: New file
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kube-proxy] at time 16:38:59.042275 duration_in_ms=2.117
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Running state [kube-proxy] at time 16:38:59.043928
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [kube-proxy]
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kube-proxy.service', '-n', '0'] in directory '/root'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.093903    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 systemd[1]: Started /bin/systemctl start kube-proxy.service.
May 18 16:38:59 ctl01 systemd[1]: Started Kubernetes Kube-Proxy Server.
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 systemd[1]: Started /bin/systemctl enable kube-proxy.service.
May 18 16:38:59 ctl01 systemd[1]: Reloading.
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.194123    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233172    8698 flags.go:33] FLAG: --alsologtostderr="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233378    8698 flags.go:33] FLAG: --application-metrics-count-limit="100"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233401    8698 flags.go:33] FLAG: --azure-container-registry-config=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233409    8698 flags.go:33] FLAG: --bind-address="0.0.0.0"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233417    8698 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233424    8698 flags.go:33] FLAG: --cleanup="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233431    8698 flags.go:33] FLAG: --cleanup-iptables="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233437    8698 flags.go:33] FLAG: --cleanup-ipvs="true"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233442    8698 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233453    8698 flags.go:33] FLAG: --cluster-cidr="192.168.0.0/16"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233459    8698 flags.go:33] FLAG: --config=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233464    8698 flags.go:33] FLAG: --config-sync-period="15m0s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233472    8698 flags.go:33] FLAG: --conntrack-max="0"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233479    8698 flags.go:33] FLAG: --conntrack-max-per-core="32768"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233485    8698 flags.go:33] FLAG: --conntrack-min="131072"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233491    8698 flags.go:33] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233497    8698 flags.go:33] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233502    8698 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233509    8698 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233515    8698 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233521    8698 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233526    8698 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233532    8698 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233537    8698 flags.go:33] FLAG: --docker-only="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233543    8698 flags.go:33] FLAG: --docker-root="/var/lib/docker"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233549    8698 flags.go:33] FLAG: --docker-tls="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233554    8698 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233560    8698 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233565    8698 flags.go:33] FLAG: --docker-tls-key="key.pem"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233570    8698 flags.go:33] FLAG: --enable-load-reader="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233576    8698 flags.go:33] FLAG: --event-storage-age-limit="default=0"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233581    8698 flags.go:33] FLAG: --event-storage-event-limit="default=0"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233587    8698 flags.go:33] FLAG: --feature-gates=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233596    8698 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233668    8698 flags.go:33] FLAG: --healthz-bind-address="0.0.0.0:10256"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233678    8698 flags.go:33] FLAG: --healthz-port="10256"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233704    8698 flags.go:33] FLAG: --help="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233711    8698 flags.go:33] FLAG: --hostname-override=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233717    8698 flags.go:33] FLAG: --housekeeping-interval="10s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233723    8698 flags.go:33] FLAG: --iptables-masquerade-bit="14"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233728    8698 flags.go:33] FLAG: --iptables-min-sync-period="0s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233734    8698 flags.go:33] FLAG: --iptables-sync-period="30s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233740    8698 flags.go:33] FLAG: --ipvs-exclude-cidrs="[]"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233755    8698 flags.go:33] FLAG: --ipvs-min-sync-period="0s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233761    8698 flags.go:33] FLAG: --ipvs-scheduler=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233767    8698 flags.go:33] FLAG: --ipvs-sync-period="30s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233772    8698 flags.go:33] FLAG: --kube-api-burst="10"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233778    8698 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233784    8698 flags.go:33] FLAG: --kube-api-qps="5"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233792    8698 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/proxy.kubeconfig"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233799    8698 flags.go:33] FLAG: --log-backtrace-at=":0"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233807    8698 flags.go:33] FLAG: --log-cadvisor-usage="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233813    8698 flags.go:33] FLAG: --log-dir=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233818    8698 flags.go:33] FLAG: --log-file=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233844    8698 flags.go:33] FLAG: --log-flush-frequency="5s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233853    8698 flags.go:33] FLAG: --logtostderr="true"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233859    8698 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.233866    8698 flags.go:33] FLAG: --masquerade-all="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234406    8698 flags.go:33] FLAG: --master=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234739    8698 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234769    8698 flags.go:33] FLAG: --mesos-agent-timeout="10s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234776    8698 flags.go:33] FLAG: --metrics-bind-address="127.0.0.1:10249"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234782    8698 flags.go:33] FLAG: --metrics-port="10249"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234788    8698 flags.go:33] FLAG: --nodeport-addresses="[]"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234798    8698 flags.go:33] FLAG: --oom-score-adj="-999"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234804    8698 flags.go:33] FLAG: --profiling="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234821    8698 flags.go:33] FLAG: --proxy-mode="iptables"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234830    8698 flags.go:33] FLAG: --proxy-port-range=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234837    8698 flags.go:33] FLAG: --resource-container="/kube-proxy"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234843    8698 flags.go:33] FLAG: --skip-headers="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234849    8698 flags.go:33] FLAG: --stderrthreshold="2"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234854    8698 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234860    8698 flags.go:33] FLAG: --storage-driver-db="cadvisor"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234866    8698 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234872    8698 flags.go:33] FLAG: --storage-driver-password="root"
May 18 16:38:59 ctl01 kernel: [  233.158419] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
May 18 16:38:59 ctl01 kernel: [  233.158462] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234901    8698 flags.go:33] FLAG: --storage-driver-secure="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234907    8698 flags.go:33] FLAG: --storage-driver-table="stats"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234912    8698 flags.go:33] FLAG: --storage-driver-user="root"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234918    8698 flags.go:33] FLAG: --udp-timeout="250ms"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234924    8698 flags.go:33] FLAG: --v="2"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234930    8698 flags.go:33] FLAG: --version="false"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234939    8698 flags.go:33] FLAG: --vmodule=""
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.234945    8698 flags.go:33] FLAG: --write-config-to=""
May 18 16:38:59 ctl01 kube-proxy[8698]: W0518 16:38:59.234955    8698 server.go:198] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.235021    8698 feature_gate.go:206] feature gates: &{map[]}
May 18 16:38:59 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
May 18 16:38:59 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
May 18 16:38:59 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] {'kube-proxy': True}
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Completed state [kube-proxy] at time 16:38:59.278795 duration_in_ms=234.866
May 18 16:38:59 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163718874370
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.294306    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.394494    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kernel: [  233.396141] IPVS: ipvs loaded.
May 18 16:38:59 ctl01 kernel: [  233.400841] IPVS: [rr] scheduler registered.
May 18 16:38:59 ctl01 kernel: [  233.404086] IPVS: [wrr] scheduler registered.
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.494639    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kernel: [  233.406755] IPVS: [sh] scheduler registered.
May 18 16:38:59 ctl01 kube-proxy[8698]: W0518 16:38:59.504056    8698 node.go:103] Failed to retrieve node info: Get https://172.16.10.36:443/api/v1/nodes/ctl01: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.504090    8698 server_others.go:148] Using iptables Proxier.
May 18 16:38:59 ctl01 kube-proxy[8698]: W0518 16:38:59.504163    8698 proxier.go:314] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.504939    8698 server_others.go:178] Tearing down inactive rules.
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.522028    8698 server.go:483] Version: v1.13.5-3+98374c02d2d8c1
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.529619    8698 server.go:509] Running in resource-only container "/kube-proxy"
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.530602    8698 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 262144
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.530652    8698 conntrack.go:52] Setting nf_conntrack_max to 262144
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.530795    8698 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.530869    8698 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.531034    8698 config.go:102] Starting endpoints config controller
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.531047    8698 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.531060    8698 config.go:202] Starting service config controller
May 18 16:38:59 ctl01 kube-proxy[8698]: I0518 16:38:59.531094    8698 controller_utils.go:1027] Waiting for caches to sync for service config controller
May 18 16:38:59 ctl01 kube-proxy[8698]: E0518 16:38:59.531601    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:59 ctl01 kube-proxy[8698]: E0518 16:38:59.531612    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:59 ctl01 kube-proxy[8698]: E0518 16:38:59.531812    8698 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.594781    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.694963    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.795221    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.895507    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.972837    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.973777    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.974767    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:38:59 ctl01 kubelet[8501]: E0518 16:38:59.995668    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kube-proxy[8698]: E0518 16:39:00.054511    8698 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.095841    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.196005    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.296166    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.396458    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.496619    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kube-proxy[8698]: E0518 16:39:00.532220    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:00 ctl01 kube-proxy[8698]: E0518 16:39:00.533225    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.596886    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.697060    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.797267    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.897449    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.973621    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.974871    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.975974    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:00 ctl01 kubelet[8501]: E0518 16:39:00.997620    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.097859    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.198038    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.298188    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.398350    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.498517    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kube-proxy[8698]: E0518 16:39:01.533116    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:01 ctl01 kube-proxy[8698]: E0518 16:39:01.533822    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.598774    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.698942    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.799106    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.899251    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.974205    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.975317    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.976398    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:01 ctl01 kubelet[8501]: E0518 16:39:01.999400    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.099574    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.112321    8501 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.199828    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.300081    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.400326    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.500652    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kube-proxy[8698]: E0518 16:39:02.533831    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:02 ctl01 kube-proxy[8698]: E0518 16:39:02.534686    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.600861    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.701036    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.801207    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.901430    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.974872    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.975747    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:02 ctl01 kubelet[8501]: E0518 16:39:02.976736    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.001647    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.101849    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.202014    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.302168    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.402440    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.502702    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kube-proxy[8698]: E0518 16:39:03.534443    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:03 ctl01 kube-proxy[8698]: E0518 16:39:03.535436    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.602884    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.703101    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.803270    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.903424    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.975373    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.976405    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:03 ctl01 kubelet[8501]: E0518 16:39:03.977488    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.003622    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.103851    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.204037    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.304163    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.404331    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.504489    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kube-proxy[8698]: E0518 16:39:04.535085    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:04 ctl01 kube-proxy[8698]: E0518 16:39:04.535948    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.604656    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: I0518 16:39:04.692773    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:04 ctl01 kubelet[8501]: I0518 16:39:04.693009    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:04 ctl01 kubelet[8501]: I0518 16:39:04.693780    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:04 ctl01 kubelet[8501]: I0518 16:39:04.693814    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:04 ctl01 kubelet[8501]: I0518 16:39:04.693827    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:04 ctl01 kubelet[8501]: I0518 16:39:04.693853    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.694098    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.704780    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.804972    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.905154    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.975959    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.976842    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:04 ctl01 kubelet[8501]: E0518 16:39:04.977898    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.005288    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.105479    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.205674    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.305895    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.406171    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.506422    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kube-proxy[8698]: E0518 16:39:05.535999    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:05 ctl01 kube-proxy[8698]: E0518 16:39:05.536604    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.606655    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.706934    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.807160    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.907491    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.976628    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.977536    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:05 ctl01 kubelet[8501]: E0518 16:39:05.978585    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.007704    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.107871    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.208033    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.308209    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.408430    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.508678    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kube-proxy[8698]: E0518 16:39:06.536715    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:06 ctl01 kube-proxy[8698]: E0518 16:39:06.537609    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.609065    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.709315    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.809543    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.909883    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.977817    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.978500    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:06 ctl01 kubelet[8501]: E0518 16:39:06.979268    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.010146    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.110343    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.175890    8501 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.210942    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.311151    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.411950    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.512348    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kube-proxy[8698]: E0518 16:39:07.537583    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:07 ctl01 kube-proxy[8698]: E0518 16:39:07.538428    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.613371    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.714338    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.815241    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.915465    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.978529    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.979508    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:07 ctl01 kubelet[8501]: E0518 16:39:07.980490    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.016211    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.116481    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.216819    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.317202    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.417453    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.517713    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kube-proxy[8698]: E0518 16:39:08.538234    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:08 ctl01 kube-proxy[8698]: E0518 16:39:08.539093    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.617888    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.718107    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.818280    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.918441    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.979140    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.980159    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:08 ctl01 kubelet[8501]: E0518 16:39:08.981202    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.018628    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.118810    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.218958    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.319093    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.419455    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.519667    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kube-proxy[8698]: E0518 16:39:09.539437    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:09 ctl01 kube-proxy[8698]: E0518 16:39:09.540134    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.619945    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.720291    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.820593    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.920938    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.980301    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.981457    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:09 ctl01 kubelet[8501]: E0518 16:39:09.982440    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.021273    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kube-proxy[8698]: E0518 16:39:10.055212    8698 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.121634    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.221940    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.322262    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.422568    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.522939    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kube-proxy[8698]: E0518 16:39:10.540495    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:10 ctl01 kube-proxy[8698]: E0518 16:39:10.541615    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.623194    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.723479    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.823786    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.924039    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.981215    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.982430    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:10 ctl01 kubelet[8501]: E0518 16:39:10.983483    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.024319    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.124692    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.225075    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.325418    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.425783    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.526168    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kube-proxy[8698]: E0518 16:39:11.541462    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:11 ctl01 kube-proxy[8698]: E0518 16:39:11.542446    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.626514    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: I0518 16:39:11.694467    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:11 ctl01 kubelet[8501]: I0518 16:39:11.694966    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:11 ctl01 kubelet[8501]: I0518 16:39:11.696890    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:11 ctl01 kubelet[8501]: I0518 16:39:11.696979    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:11 ctl01 kubelet[8501]: I0518 16:39:11.697009    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:11 ctl01 kubelet[8501]: I0518 16:39:11.697176    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.697833    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.726699    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.827092    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.927363    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.982487    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.983518    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:11 ctl01 kubelet[8501]: E0518 16:39:11.984102    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.027703    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.112707    8501 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.128027    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.228406    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.328711    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.429042    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.529350    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kube-proxy[8698]: E0518 16:39:12.542501    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:12 ctl01 kube-proxy[8698]: E0518 16:39:12.543416    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.629702    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.729993    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.830350    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.930643    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.983712    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.984272    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:12 ctl01 kubelet[8501]: E0518 16:39:12.985681    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.031028    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.131404    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.231637    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.331877    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.432112    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.532401    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kube-proxy[8698]: E0518 16:39:13.543576    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:13 ctl01 kube-proxy[8698]: E0518 16:39:13.544443    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.632706    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.732929    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.833287    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.933544    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.984892    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.985889    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:13 ctl01 kubelet[8501]: E0518 16:39:13.987613    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.033923    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.134113    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.234405    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.334689    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.435070    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.535319    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kube-proxy[8698]: E0518 16:39:14.544491    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:14 ctl01 kube-proxy[8698]: E0518 16:39:14.545405    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.635622    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command cmd.run with jid 20190518163914657623
May 18 16:39:14 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 8840
May 18 16:39:14 ctl01 salt-minion[5014]: [INFO    ] Executing command 'calicoctl node status' in directory '/root'
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.736545    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163914657623
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.836873    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.937113    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.985830    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.986712    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:14 ctl01 kubelet[8501]: E0518 16:39:14.988370    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.037422    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.137771    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.238024    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.338426    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command cmd.run with jid 20190518163915381705
May 18 16:39:15 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 8860
May 18 16:39:15 ctl01 salt-minion[5014]: [INFO    ] Executing command 'calicoctl get ippool' in directory '/root'
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.438654    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163915381705
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.538862    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kube-proxy[8698]: E0518 16:39:15.545113    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:15 ctl01 kube-proxy[8698]: E0518 16:39:15.546057    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.639173    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.739403    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.839741    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.939923    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.986469    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.987417    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:15 ctl01 kubelet[8501]: E0518 16:39:15.988788    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.040252    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.140614    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190518163916176622
May 18 16:39:16 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 8884
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.240922    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.341263    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.441617    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.541928    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kube-proxy[8698]: E0518 16:39:16.545950    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:16 ctl01 kube-proxy[8698]: E0518 16:39:16.547083    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.642135    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.742340    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.842531    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.942789    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.987047    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.988109    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:16 ctl01 kubelet[8501]: E0518 16:39:16.989684    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.043076    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.143307    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.176971    8501 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.243461    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.343629    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.443869    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.544021    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kube-proxy[8698]: E0518 16:39:17.546758    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:17 ctl01 kube-proxy[8698]: E0518 16:39:17.547523    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.644293    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.744541    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.844735    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.944881    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.987708    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.988728    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:17 ctl01 kubelet[8501]: E0518 16:39:17.990161    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.045170    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.145507    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.245870    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.346059    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.446383    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.546663    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kube-proxy[8698]: E0518 16:39:18.547706    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:18 ctl01 kube-proxy[8698]: E0518 16:39:18.548650    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.646954    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: I0518 16:39:18.698153    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:18 ctl01 kubelet[8501]: I0518 16:39:18.698481    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:18 ctl01 kubelet[8501]: I0518 16:39:18.699760    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:18 ctl01 kubelet[8501]: I0518 16:39:18.699847    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:18 ctl01 kubelet[8501]: I0518 16:39:18.699879    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:18 ctl01 kubelet[8501]: I0518 16:39:18.699925    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.700496    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.747242    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.847569    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.947889    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.988540    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.989463    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:18 ctl01 kubelet[8501]: E0518 16:39:18.990988    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.048374    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.148618    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.248828    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.349047    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.449254    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kube-proxy[8698]: E0518 16:39:19.548611    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:19 ctl01 kube-proxy[8698]: E0518 16:39:19.549542    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.549438    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.649623    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.749809    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.850086    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.950281    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.989515    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.990282    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:19 ctl01 kubelet[8501]: E0518 16:39:19.991644    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.050484    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kube-proxy[8698]: E0518 16:39:20.056036    8698 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.150769    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.251035    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.351230    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.451537    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kube-proxy[8698]: E0518 16:39:20.549414    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:20 ctl01 kube-proxy[8698]: E0518 16:39:20.550147    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.551693    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.651833    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.751998    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.852201    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.952407    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.990134    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.991112    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:20 ctl01 kubelet[8501]: E0518 16:39:20.992285    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.052755    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.153086    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.253369    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.353636    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.453789    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kube-proxy[8698]: E0518 16:39:21.550690    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:21 ctl01 kube-proxy[8698]: E0518 16:39:21.551323    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.554053    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.654396    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.754635    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.854935    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.955841    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.990861    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.991645    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:21 ctl01 kubelet[8501]: E0518 16:39:21.992670    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.056038    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.112939    8501 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.156232    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.256424    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.356722    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.456934    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/server/setup.sls'
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Running state [/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16] at time 16:39:22.532694
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Executing state etcd.set for [/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16]
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Starting new HTTP connection (1): 127.0.0.1
May 18 16:39:22 ctl01 salt-minion[5014]: [ERROR   ] etcd: Key not found : /calico
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Starting new HTTP connection (1): 127.0.0.1
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] {'/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16': '{"masquerade":true,"cidr":"192.168.0.0/16"}'}
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Completed state [/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16] at time 16:39:22.540209 duration_in_ms=7.516
May 18 16:39:22 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163916176622
May 18 16:39:22 ctl01 kube-proxy[8698]: E0518 16:39:22.551513    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:22 ctl01 kube-proxy[8698]: E0518 16:39:22.552495    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.557183    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.657491    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.757852    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.858261    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.958585    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.992644    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.993167    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:22 ctl01 kubelet[8501]: E0518 16:39:22.994975    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.058858    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.159183    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190518163923178309
May 18 16:39:23 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 8901
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.259498    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.359736    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.460034    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kube-proxy[8698]: E0518 16:39:23.552458    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:23 ctl01 kube-proxy[8698]: E0518 16:39:23.553347    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.560242    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.660436    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.760630    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.860819    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.961096    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.993627    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.994890    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:23 ctl01 kubelet[8501]: E0518 16:39:23.995619    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.061281    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.161540    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.261718    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.361881    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/init.sls'
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/init.sls'
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.462120    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:24 ctl01 kube-proxy[8698]: E0518 16:39:24.553352    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:24 ctl01 kube-proxy[8698]: E0518 16:39:24.554149    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.562278    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.662476    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.762727    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/service.sls'
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.862938    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.963216    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:24 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.994592    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.995740    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:24 ctl01 kubelet[8501]: E0518 16:39:24.996939    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.063523    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.163763    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.263990    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.364217    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.464527    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 kube-proxy[8698]: E0518 16:39:25.554381    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:25 ctl01 kube-proxy[8698]: E0518 16:39:25.555569    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.564720    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.666292    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 kubelet[8501]: I0518 16:39:25.700823    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:25 ctl01 kubelet[8501]: I0518 16:39:25.701226    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:25 ctl01 kubelet[8501]: I0518 16:39:25.703063    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:25 ctl01 kubelet[8501]: I0518 16:39:25.703138    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:25 ctl01 kubelet[8501]: I0518 16:39:25.703180    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:25 ctl01 kubelet[8501]: I0518 16:39:25.703241    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.703943    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.766514    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.866817    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.966977    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.995613    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.996929    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:25 ctl01 kubelet[8501]: E0518 16:39:25.997666    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.067294    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.167514    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.267873    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.368136    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.468358    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 kube-proxy[8698]: E0518 16:39:26.555230    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:26 ctl01 kube-proxy[8698]: E0518 16:39:26.556370    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:26 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.568688    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.668935    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.769318    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.869595    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.969886    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.996547    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.997882    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:26 ctl01 kubelet[8501]: E0518 16:39:26.998880    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.070155    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/controller.sls'
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.170344    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.177552    8501 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:27 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.270626    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.370917    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.471192    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:27 ctl01 kube-proxy[8698]: E0518 16:39:27.555843    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:27 ctl01 kube-proxy[8698]: E0518 16:39:27.557248    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.571429    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.671609    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.771864    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.872125    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.972398    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.997295    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.998444    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:27 ctl01 kubelet[8501]: E0518 16:39:27.999538    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.072571    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 salt-minion[5014]: [INFO    ] Executing command 'date "+%FT%TZ"' in directory '/root'
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.172768    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/setup.sls'
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.273008    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.373164    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.473377    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:28 ctl01 kube-proxy[8698]: E0518 16:39:28.556878    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:28 ctl01 kube-proxy[8698]: E0518 16:39:28.557959    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.573683    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.673974    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.774250    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.874543    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.974779    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:28 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.998310    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:28 ctl01 kubelet[8501]: E0518 16:39:28.999253    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.000297    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.075073    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.175277    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.275534    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.375761    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.476082    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 kube-proxy[8698]: I0518 16:39:29.531182    8698 proxier.go:645] Not syncing iptables until Services and Endpoints have been received from master
May 18 16:39:29 ctl01 kube-proxy[8698]: E0518 16:39:29.557736    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:29 ctl01 kube-proxy[8698]: E0518 16:39:29.558715    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.576370    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.676570    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.776884    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.877174    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.977399    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.999112    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:29 ctl01 kubelet[8501]: E0518 16:39:29.999844    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.001093    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:30 ctl01 kube-proxy[8698]: E0518 16:39:30.056847    8698 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.077592    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.177774    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.278112    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.378391    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.478561    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 kube-proxy[8698]: E0518 16:39:30.558938    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:30 ctl01 kube-proxy[8698]: E0518 16:39:30.559697    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:30 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.578729    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.678879    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.779211    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.879503    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:30 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:30 ctl01 kubelet[8501]: E0518 16:39:30.979693    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.000131    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.000937    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.002035    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.079854    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.180112    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.280435    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.380623    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/control/init.sls'
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.480775    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:31 ctl01 kube-proxy[8698]: E0518 16:39:31.560066    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:31 ctl01 kube-proxy[8698]: E0518 16:39:31.560898    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.580979    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.681161    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.781469    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/control/service.sls'
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.881691    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:31 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:31 ctl01 kubelet[8501]: E0518 16:39:31.981835    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.000905    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.001840    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.002937    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:32 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.082122    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.113207    8501 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.182398    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/control/role.sls'
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.282568    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.382928    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.483246    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:32 ctl01 kube-proxy[8698]: E0518 16:39:32.560647    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:32 ctl01 kube-proxy[8698]: E0518 16:39:32.561705    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.583424    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.683618    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: I0518 16:39:32.704310    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:32 ctl01 kubelet[8501]: I0518 16:39:32.704691    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:32 ctl01 kubelet[8501]: I0518 16:39:32.706077    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:32 ctl01 kubelet[8501]: I0518 16:39:32.706134    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:32 ctl01 kubelet[8501]: I0518 16:39:32.706152    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:32 ctl01 kubelet[8501]: I0518 16:39:32.706235    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.708587    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.783903    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.884118    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:32 ctl01 kubelet[8501]: E0518 16:39:32.984406    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.001864    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.002824    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.003881    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.084669    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.184848    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.285020    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.385153    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [curl] at time 16:39:33.453942
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [curl]
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.485358    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 kube-proxy[8698]: E0518 16:39:33.561327    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:33 ctl01 kube-proxy[8698]: E0518 16:39:33.562167    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.585503    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.685714    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.785923    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Completed state [curl] at time 16:39:33.822693 duration_in_ms=368.75
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [git] at time 16:39:33.823090
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [git]
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Completed state [git] at time 16:39:33.835413 duration_in_ms=12.323
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [apt-transport-https] at time 16:39:33.835705
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [apt-transport-https]
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Completed state [apt-transport-https] at time 16:39:33.847746 duration_in_ms=12.041
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [python-apt] at time 16:39:33.848042
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [python-apt]
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Completed state [python-apt] at time 16:39:33.859794 duration_in_ms=11.752
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [socat] at time 16:39:33.860103
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [socat]
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Completed state [socat] at time 16:39:33.872016 duration_in_ms=11.913
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [openssl] at time 16:39:33.872302
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [openssl]
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Completed state [openssl] at time 16:39:33.884221 duration_in_ms=11.918
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [conntrack] at time 16:39:33.884508
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [conntrack]
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.886138    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] All specified packages are already installed
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Completed state [conntrack] at time 16:39:33.894645 duration_in_ms=10.137
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/hyperkube] at time 16:39:33.897444
May 18 16:39:33 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/bin/hyperkube]
May 18 16:39:33 ctl01 kubelet[8501]: E0518 16:39:33.986328    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.002539    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.003768    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.004768    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.086544    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.186752    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.286971    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.387328    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.487562    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kube-proxy[8698]: E0518 16:39:34.562018    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:34 ctl01 kube-proxy[8698]: E0518 16:39:34.562930    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.587810    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] File /usr/bin/hyperkube is in the correct state
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/hyperkube] at time 16:39:34.630510 duration_in_ms=733.066
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/kubectl] at time 16:39:34.631957
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Executing state file.symlink for [/usr/bin/kubectl]
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Symlink /usr/bin/kubectl is present and owned by root:root
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/kubectl] at time 16:39:34.634013 duration_in_ms=2.056
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Running state [containerd=1.2.5-3~u16.04+mcp] at time 16:39:34.634255
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Executing state pkg.installed for [containerd=1.2.5-3~u16.04+mcp]
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Executing command ['apt-cache', '-q', 'policy', 'containerd=1.2.5-3~u16.04+mcp'] in directory '/root'
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.688030    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 salt-minion[5014]: [INFO    ] Executing command ['apt-get', '-q', 'update'] in directory '/root'
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.788215    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.888495    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:34 ctl01 kubelet[8501]: E0518 16:39:34.988784    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.003579    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.004433    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.005539    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.089030    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.189270    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.289464    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.389807    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.490165    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kube-proxy[8698]: E0518 16:39:35.562783    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:35 ctl01 kube-proxy[8698]: E0518 16:39:35.563560    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.590525    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.690835    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.791181    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.891513    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:35 ctl01 kubelet[8501]: E0518 16:39:35.991809    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.004275    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.005781    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.007043    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.092008    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.192155    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.292435    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.392675    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.492880    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kube-proxy[8698]: E0518 16:39:36.563474    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:36 ctl01 kube-proxy[8698]: E0518 16:39:36.564338    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.593099    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.693282    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.793481    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.893734    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:36 ctl01 kubelet[8501]: E0518 16:39:36.993955    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.004937    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.006377    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.007658    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'containerd=1.2.5-3~u16.04+mcp'] in directory '/root'
May 18 16:39:37 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install containerd=1.2.5-3~u16.04+mcp.
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.094196    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.178374    8501 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.194434    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.294668    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.394833    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] 1 targeted package was installed/updated.
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Completed state [containerd=1.2.5-3~u16.04+mcp] at time 16:39:37.416440 duration_in_ms=2782.185
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/containerd/config.toml] at time 16:39:37.416812
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/containerd/config.toml]
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.495023    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 kube-proxy[8698]: E0518 16:39:37.564365    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:37 ctl01 kube-proxy[8698]: E0518 16:39:37.565446    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.595329    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.695471    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] File /etc/containerd/config.toml is in the correct state
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/containerd/config.toml] at time 16:39:37.794667 duration_in_ms=377.854
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.795736    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Running state [containerd] at time 16:39:37.797023
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [containerd]
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'containerd.service', '-n', '0'] in directory '/root'
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'containerd.service'] in directory '/root'
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'containerd.service'] in directory '/root'
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] The service containerd is already running
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Completed state [containerd] at time 16:39:37.852216 duration_in_ms=55.192
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Running state [/tmp/crictl] at time 16:39:37.855472
May 18 16:39:37 ctl01 salt-minion[5014]: [INFO    ] Executing state archive.extracted for [/tmp/crictl]
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.896080    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:37 ctl01 kubelet[8501]: E0518 16:39:37.996380    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.006007    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.007262    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.008526    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.096557    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.196721    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] All files in archive are already present
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Completed state [/tmp/crictl] at time 16:39:38.247872 duration_in_ms=392.4
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/local/bin/crictl] at time 16:39:38.249258
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/local/bin/crictl]
May 18 16:39:38 ctl01 salt-minion[5014]: [WARNING ] Use of argument owner found, "owner" is invalid, please use "user"
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190518163938262435
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 10203
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.296950    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163938262435
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.397193    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] File /usr/local/bin/crictl is in the correct state
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/local/bin/crictl] at time 16:39:38.473675 duration_in_ms=224.417
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/crictl.yaml] at time 16:39:38.473946
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/crictl.yaml]
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] File /etc/crictl.yaml is in the correct state
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/crictl.yaml] at time 16:39:38.476046 duration_in_ms=2.1
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/criproxy] at time 16:39:38.476271
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/criproxy]
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] File /etc/criproxy is not present
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/criproxy] at time 16:39:38.477023 duration_in_ms=0.752
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Running state [criproxy] at time 16:39:38.477249
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing state service.dead for [criproxy]
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'criproxy.service', '-n', '0'] in directory '/root'
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] The named service criproxy is not available
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Completed state [criproxy] at time 16:39:38.495691 duration_in_ms=18.44
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kubelet.service] at time 16:39:38.496160
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kubelet.service]
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.497468    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:38 ctl01 kube-proxy[8698]: E0518 16:39:38.565028    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:38 ctl01 kube-proxy[8698]: E0518 16:39:38.565947    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.597654    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.697863    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.798235    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] File /etc/systemd/system/kubelet.service is in the correct state
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kubelet.service] at time 16:39:38.852647 duration_in_ms=356.486
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/config] at time 16:39:38.853184
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/kubernetes/config]
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/config is not present
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/config] at time 16:39:38.854709 duration_in_ms=1.525
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kubelet] at time 16:39:38.855097
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/kubelet]
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.898604    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:38 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:38 ctl01 kubelet[8501]: E0518 16:39:38.998787    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.006666    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.007743    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.009054    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.099023    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.199228    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.299379    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.399544    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.499766    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 kube-proxy[8698]: E0518 16:39:39.565601    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:39 ctl01 kube-proxy[8698]: E0518 16:39:39.566523    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.599982    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.700126    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 kubelet[8501]: I0518 16:39:39.708855    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:39 ctl01 kubelet[8501]: I0518 16:39:39.709075    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:39 ctl01 kubelet[8501]: I0518 16:39:39.709921    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:39 ctl01 kubelet[8501]: I0518 16:39:39.709965    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:39 ctl01 kubelet[8501]: I0518 16:39:39.709981    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:39 ctl01 kubelet[8501]: I0518 16:39:39.710005    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.710402    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] File /etc/default/kubelet is in the correct state
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kubelet] at time 16:39:39.748149 duration_in_ms=893.052
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/kubelet.kubeconfig] at time 16:39:39.748646
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/kubelet.kubeconfig]
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.800344    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:39 ctl01 kubelet[8501]: E0518 16:39:39.900573    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:39 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.000793    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.007320    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.008209    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.009559    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:40 ctl01 kube-proxy[8698]: E0518 16:39:40.057835    8698 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.101011    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.201223    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.301442    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/kubelet.kubeconfig is in the correct state
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/kubelet.kubeconfig] at time 16:39:40.377092 duration_in_ms=628.446
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/manifests] at time 16:39:40.377420
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/etc/kubernetes/manifests]
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Directory /etc/kubernetes/manifests is in the correct state
May 18 16:39:40 ctl01 salt-minion[5014]: Directory /etc/kubernetes/manifests updated
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/manifests] at time 16:39:40.378908 duration_in_ms=1.489
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Running state [kubelet] at time 16:39:40.383014
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [kubelet]
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kubelet.service', '-n', '0'] in directory '/root'
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.401617    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kubelet.service'] in directory '/root'
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] The service kubelet is already running
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Completed state [kubelet] at time 16:39:40.429595 duration_in_ms=46.58
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/logrotate.d/kubernetes] at time 16:39:40.430182
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/logrotate.d/kubernetes]
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] File /etc/logrotate.d/kubernetes is in the correct state
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/logrotate.d/kubernetes] at time 16:39:40.457958 duration_in_ms=27.776
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Running state [/opt/cni/bin] at time 16:39:40.458311
May 18 16:39:40 ctl01 salt-minion[5014]: [INFO    ] Executing state archive.extracted for [/opt/cni/bin]
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.501848    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 kube-proxy[8698]: E0518 16:39:40.566246    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:40 ctl01 kube-proxy[8698]: E0518 16:39:40.567155    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.602029    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.702213    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.802407    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:40 ctl01 kubelet[8501]: E0518 16:39:40.902594    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.002766    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.007880    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.008905    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.009925    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.102941    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.203132    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.303302    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', 'xz', '-f', '/var/cache/salt/minion/extrn_files/base/docker-prod-local.artifactory.mirantis.com/artifactory/binary-prod-local/mirantis/kubernetes/containernetworking-plugins/containernetworking-plugins_v0.7.2-173-g8db2808.tar.gz'] in directory '/opt/cni/bin/'
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.403471    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.503666    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kube-proxy[8698]: E0518 16:39:41.566873    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:41 ctl01 kube-proxy[8698]: E0518 16:39:41.567812    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.603844    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.704036    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.804236    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 kubelet[8501]: E0518 16:39:41.904419    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] {'extracted_files': 'no tar output so far'}
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Completed state [/opt/cni/bin] at time 16:39:41.991577 duration_in_ms=1533.266
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons] at time 16:39:41.992090
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/etc/kubernetes/addons]
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Directory /etc/kubernetes/addons is in the correct state
May 18 16:39:41 ctl01 salt-minion[5014]: Directory /etc/kubernetes/addons updated
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons] at time 16:39:41.994459 duration_in_ms=2.369
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:39:41.994802
May 18 16:39:41 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-kube-controllers.yml]
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.004775    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.008811    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.009520    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.010331    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.104955    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.113386    8501 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.205158    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.305510    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/calico/calico-kube-controllers.yml is in the correct state
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:39:42.394293 duration_in_ms=399.49
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:39:42.394683
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-rbac.yml]
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.405773    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/calico/calico-rbac.yml is in the correct state
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:39:42.419364 duration_in_ms=24.681
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:39:42.419681
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-svc.yml]
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.505952    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 kube-proxy[8698]: E0518 16:39:42.567768    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:42 ctl01 kube-proxy[8698]: E0518 16:39:42.568733    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.606359    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.706582    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.806819    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-svc.yml is in the correct state
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:39:42.809198 duration_in_ms=389.516
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:39:42.809699
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-server.yml]
May 18 16:39:42 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:42 ctl01 kubelet[8501]: E0518 16:39:42.907085    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.007313    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.009389    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.010502    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.011771    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.107485    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.207690    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.307937    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.408255    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-server.yml is in the correct state
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:39:43.477664 duration_in_ms=667.964
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:39:43.478223
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-agent.yml]
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.508520    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:43 ctl01 kube-proxy[8698]: E0518 16:39:43.568552    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:43 ctl01 kube-proxy[8698]: E0518 16:39:43.569477    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.608753    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.709014    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.809295    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-agent.yml is in the correct state
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:39:43.840650 duration_in_ms=362.426
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:39:43.840974
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml]
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml is in the correct state
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:39:43.859732 duration_in_ms=18.758
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:39:43.859979
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-roles.yml]
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-roles.yml is in the correct state
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:39:43.878154 duration_in_ms=18.175
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:39:43.878508
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/prometheus/prometheus-roles.yml]
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/prometheus/prometheus-roles.yml is in the correct state
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:39:43.901117 duration_in_ms=22.609
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/dns] at time 16:39:43.901354
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/dns]
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] {'removed': '/etc/kubernetes/addons/dns'}
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/dns] at time 16:39:43.902392 duration_in_ms=1.039
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Running state [kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true] at time 16:39:43.903712
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing state cmd.run for [kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true]
May 18 16:39:43 ctl01 salt-minion[5014]: [INFO    ] Executing command 'kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true' in directory '/root'
May 18 16:39:43 ctl01 kubelet[8501]: E0518 16:39:43.909533    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.009702    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.009955    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.011053    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.012962    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] {'pid': 10627, 'retcode': 0, 'stderr': 'The connection to the server localhost:8080 was refused - did you specify the right host or port?', 'stdout': 'kube-dns is absent. OK'}
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Completed state [kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true] at time 16:39:44.037922 duration_in_ms=134.209
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-cm.yml] at time 16:39:44.038391
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-cm.yml]
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-cm.yml'
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.109876    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.210088    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.310308    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.410474    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.510658    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 kube-proxy[8698]: E0518 16:39:44.569513    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:44 ctl01 kube-proxy[8698]: E0518 16:39:44.570302    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.610925    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:44 ctl01 salt-minion[5014]: New file
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-cm.yml] at time 16:39:44.616909 duration_in_ms=578.516
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-deploy.yml] at time 16:39:44.617307
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-deploy.yml]
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-deploy.yml'
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.711206    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.811455    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:44 ctl01 kubelet[8501]: E0518 16:39:44.911791    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:44 ctl01 salt-minion[5014]: New file
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-deploy.yml] at time 16:39:44.987867 duration_in_ms=370.56
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-svc.yml] at time 16:39:44.988183
May 18 16:39:44 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-svc.yml]
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-svc.yml'
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.011012    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.011729    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.012484    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.013891    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.112636    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.212819    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.313004    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:45 ctl01 salt-minion[5014]: New file
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-svc.yml] at time 16:39:45.340078 duration_in_ms=351.894
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-rbac.yml] at time 16:39:45.340522
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-rbac.yml]
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-rbac.yml'
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.413326    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.513611    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:45 ctl01 kube-proxy[8698]: E0518 16:39:45.570439    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:45 ctl01 kube-proxy[8698]: E0518 16:39:45.571259    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.613862    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:45 ctl01 salt-minion[5014]: New file
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-rbac.yml] at time 16:39:45.686315 duration_in_ms=345.793
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/metrics-server] at time 16:39:45.686633
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/metrics-server]
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/addons/metrics-server is not present
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/metrics-server] at time 16:39:45.687582 duration_in_ms=0.949
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Running state [/srv/kubernetes/known_tokens.csv] at time 16:39:45.687838
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/srv/kubernetes/known_tokens.csv]
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/known_tokens.csv'
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.714122    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.814283    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 kubelet[8501]: E0518 16:39:45.914441    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:45 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.012432    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.013487    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.014615    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.014686    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.114856    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.215144    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.315328    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:46 ctl01 salt-minion[5014]: New file
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Completed state [/srv/kubernetes/known_tokens.csv] at time 16:39:46.384018 duration_in_ms=696.179
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Running state [/srv/kubernetes/basic_auth.csv] at time 16:39:46.384476
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/srv/kubernetes/basic_auth.csv]
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/basic_auth.csv'
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.415619    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.515967    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 kube-proxy[8698]: E0518 16:39:46.571048    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:46 ctl01 kube-proxy[8698]: E0518 16:39:46.572322    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.616301    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:46 ctl01 kubelet[8501]: I0518 16:39:46.710600    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:46 ctl01 kubelet[8501]: I0518 16:39:46.710890    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:46 ctl01 kubelet[8501]: I0518 16:39:46.714461    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:46 ctl01 kubelet[8501]: I0518 16:39:46.714494    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:46 ctl01 kubelet[8501]: I0518 16:39:46.714508    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:46 ctl01 kubelet[8501]: I0518 16:39:46.714529    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.714846    8501 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.716460    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:46 ctl01 salt-minion[5014]: New file
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Completed state [/srv/kubernetes/basic_auth.csv] at time 16:39:46.783998 duration_in_ms=399.521
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kube-apiserver] at time 16:39:46.784508
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/kube-apiserver]
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:46 ctl01 salt-minion[5014]: New file
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kube-apiserver] at time 16:39:46.788384 duration_in_ms=3.877
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/scheduler.kubeconfig] at time 16:39:46.788623
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/scheduler.kubeconfig]
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-scheduler/scheduler.kubeconfig'
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.816697    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:46 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:46 ctl01 kubelet[8501]: E0518 16:39:46.916991    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.013590    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.014300    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.015646    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.017226    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.117431    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.179068    8501 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.217609    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.317889    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.418212    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:47 ctl01 salt-minion[5014]: New file
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/scheduler.kubeconfig] at time 16:39:47.484206 duration_in_ms=695.582
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/controller-manager.kubeconfig] at time 16:39:47.484638
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/controller-manager.kubeconfig]
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-controller-manager/controller-manager.kubeconfig'
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.518429    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:47 ctl01 kube-proxy[8698]: E0518 16:39:47.571884    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:47 ctl01 kube-proxy[8698]: E0518 16:39:47.572923    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.618620    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.718830    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.819119    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:47 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:39:47 ctl01 kubelet[8501]: E0518 16:39:47.919458    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.014813    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.016223    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.016622    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.019776    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.120096    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/controller-manager.kubeconfig] at time 16:39:48.175577 duration_in_ms=690.938
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kube-controller-manager] at time 16:39:48.175891
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/kube-controller-manager]
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kube-controller-manager] at time 16:39:48.178234 duration_in_ms=2.343
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kube-scheduler] at time 16:39:48.178506
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/kube-scheduler]
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kube-scheduler] at time 16:39:48.180682 duration_in_ms=2.177
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kube-apiserver.service] at time 16:39:48.180911
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-apiserver.service]
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-apiserver.service'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kube-apiserver.service] at time 16:39:48.212575 duration_in_ms=31.664
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kube-scheduler.service] at time 16:39:48.212812
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-scheduler.service]
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.220365    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-scheduler.service'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kube-scheduler.service] at time 16:39:48.246698 duration_in_ms=33.885
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kube-controller-manager.service] at time 16:39:48.246932
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-controller-manager.service]
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-controller-manager.service'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kube-controller-manager.service] at time 16:39:48.281153 duration_in_ms=34.221
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/ssl/kubernetes-server.crt] at time 16:39:48.281422
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/ssl/kubernetes-server.crt]
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** '_certs/kubernetes/kubernetes-server.crt'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/ssl/kubernetes-server.crt] at time 16:39:48.305696 duration_in_ms=24.273
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/ssl/kubernetes-server.key] at time 16:39:48.306022
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/ssl/kubernetes-server.key]
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.320567    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** '_certs/kubernetes/kubernetes-server.key'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/ssl/kubernetes-server.key] at time 16:39:48.330462 duration_in_ms=24.44
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/ssl/kubernetes-server.pem] at time 16:39:48.330810
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/ssl/kubernetes-server.pem]
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** '_certs/kubernetes/kubernetes-server.pem'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:39:48 ctl01 salt-minion[5014]: New file
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/ssl/kubernetes-server.pem] at time 16:39:48.354059 duration_in_ms=23.248
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Running state [kube-apiserver] at time 16:39:48.362095
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [kube-apiserver]
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kube-apiserver.service', '-n', '0'] in directory '/root'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-apiserver.service'] in directory '/root'
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.420811    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-apiserver.service'] in directory '/root'
May 18 16:39:48 ctl01 systemd[1]: Started /bin/systemctl start kube-apiserver.service.
May 18 16:39:48 ctl01 systemd[1]: Starting Kubernetes API Server...
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.521112    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 kube-apiserver[11092]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
May 18 16:39:48 ctl01 kube-apiserver[11092]: Flag --insecure-port has been deprecated, This flag will be removed in a future version.
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570527   11092 flags.go:33] FLAG: --address="0.0.0.0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570598   11092 flags.go:33] FLAG: --admission-control="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570619   11092 flags.go:33] FLAG: --admission-control-config-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570626   11092 flags.go:33] FLAG: --advertise-address="172.16.10.36"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570633   11092 flags.go:33] FLAG: --allow-privileged="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570640   11092 flags.go:33] FLAG: --alsologtostderr="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570647   11092 flags.go:33] FLAG: --anonymous-auth="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570653   11092 flags.go:33] FLAG: --api-audiences="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570660   11092 flags.go:33] FLAG: --apiserver-count="1"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570667   11092 flags.go:33] FLAG: --application-metrics-count-limit="100"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570673   11092 flags.go:33] FLAG: --audit-dynamic-configuration="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570679   11092 flags.go:33] FLAG: --audit-log-batch-buffer-size="10000"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570687   11092 flags.go:33] FLAG: --audit-log-batch-max-size="1"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570698   11092 flags.go:33] FLAG: --audit-log-batch-max-wait="0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570709   11092 flags.go:33] FLAG: --audit-log-batch-throttle-burst="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570716   11092 flags.go:33] FLAG: --audit-log-batch-throttle-enable="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570721   11092 flags.go:33] FLAG: --audit-log-batch-throttle-qps="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570729   11092 flags.go:33] FLAG: --audit-log-format="json"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570735   11092 flags.go:33] FLAG: --audit-log-maxage="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570740   11092 flags.go:33] FLAG: --audit-log-maxbackup="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570746   11092 flags.go:33] FLAG: --audit-log-maxsize="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570751   11092 flags.go:33] FLAG: --audit-log-mode="blocking"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570757   11092 flags.go:33] FLAG: --audit-log-path=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570762   11092 flags.go:33] FLAG: --audit-log-truncate-enabled="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570768   11092 flags.go:33] FLAG: --audit-log-truncate-max-batch-size="10485760"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570775   11092 flags.go:33] FLAG: --audit-log-truncate-max-event-size="102400"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570781   11092 flags.go:33] FLAG: --audit-log-version="audit.k8s.io/v1"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570787   11092 flags.go:33] FLAG: --audit-policy-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570792   11092 flags.go:33] FLAG: --audit-webhook-batch-buffer-size="10000"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570798   11092 flags.go:33] FLAG: --audit-webhook-batch-initial-backoff="10s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570804   11092 flags.go:33] FLAG: --audit-webhook-batch-max-size="400"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570810   11092 flags.go:33] FLAG: --audit-webhook-batch-max-wait="30s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570815   11092 flags.go:33] FLAG: --audit-webhook-batch-throttle-burst="15"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570831   11092 flags.go:33] FLAG: --audit-webhook-batch-throttle-enable="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570837   11092 flags.go:33] FLAG: --audit-webhook-batch-throttle-qps="10"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570845   11092 flags.go:33] FLAG: --audit-webhook-config-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570850   11092 flags.go:33] FLAG: --audit-webhook-initial-backoff="10s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570856   11092 flags.go:33] FLAG: --audit-webhook-mode="batch"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570862   11092 flags.go:33] FLAG: --audit-webhook-truncate-enabled="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570867   11092 flags.go:33] FLAG: --audit-webhook-truncate-max-batch-size="10485760"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570874   11092 flags.go:33] FLAG: --audit-webhook-truncate-max-event-size="102400"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570879   11092 flags.go:33] FLAG: --audit-webhook-version="audit.k8s.io/v1"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570885   11092 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570891   11092 flags.go:33] FLAG: --authentication-token-webhook-config-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570897   11092 flags.go:33] FLAG: --authorization-mode="[Node,RBAC]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570907   11092 flags.go:33] FLAG: --authorization-policy-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570913   11092 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 18 16:39:48 ctl01 kube-proxy[8698]: E0518 16:39:48.572731    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:48 ctl01 kube-proxy[8698]: E0518 16:39:48.573354    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570919   11092 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570925   11092 flags.go:33] FLAG: --authorization-webhook-config-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570930   11092 flags.go:33] FLAG: --azure-container-registry-config=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570936   11092 flags.go:33] FLAG: --basic-auth-file="/srv/kubernetes/basic_auth.csv"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570942   11092 flags.go:33] FLAG: --bind-address="0.0.0.0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570947   11092 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570954   11092 flags.go:33] FLAG: --cert-dir="/var/run/kubernetes"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570960   11092 flags.go:33] FLAG: --client-ca-file="/etc/kubernetes/ssl/ca-kubernetes.crt"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570966   11092 flags.go:33] FLAG: --cloud-config=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570971   11092 flags.go:33] FLAG: --cloud-provider=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570976   11092 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570985   11092 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570992   11092 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.570998   11092 flags.go:33] FLAG: --contention-profiling="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571004   11092 flags.go:33] FLAG: --cors-allowed-origins="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571010   11092 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571016   11092 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571022   11092 flags.go:33] FLAG: --default-watch-cache-size="100"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571037   11092 flags.go:33] FLAG: --delete-collection-workers="1"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571043   11092 flags.go:33] FLAG: --deserialization-cache-size="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571048   11092 flags.go:33] FLAG: --disable-admission-plugins="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571057   11092 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571063   11092 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571069   11092 flags.go:33] FLAG: --docker-only="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571074   11092 flags.go:33] FLAG: --docker-root="/var/lib/docker"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571080   11092 flags.go:33] FLAG: --docker-tls="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571085   11092 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571090   11092 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571096   11092 flags.go:33] FLAG: --docker-tls-key="key.pem"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571101   11092 flags.go:33] FLAG: --enable-admission-plugins="[NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,DefaultStorageClass]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571112   11092 flags.go:33] FLAG: --enable-aggregator-routing="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571118   11092 flags.go:33] FLAG: --enable-bootstrap-token-auth="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571123   11092 flags.go:33] FLAG: --enable-garbage-collector="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571129   11092 flags.go:33] FLAG: --enable-load-reader="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571134   11092 flags.go:33] FLAG: --enable-logs-handler="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571140   11092 flags.go:33] FLAG: --enable-swagger-ui="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571152   11092 flags.go:33] FLAG: --encryption-provider-config=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571158   11092 flags.go:33] FLAG: --endpoint-reconciler-type="lease"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571163   11092 flags.go:33] FLAG: --etcd-cafile="/var/lib/etcd/ca.pem"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571169   11092 flags.go:33] FLAG: --etcd-certfile="/var/lib/etcd/etcd-client.crt"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571175   11092 flags.go:33] FLAG: --etcd-compaction-interval="5m0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571181   11092 flags.go:33] FLAG: --etcd-count-metric-poll-period="1m0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571186   11092 flags.go:33] FLAG: --etcd-keyfile="/var/lib/etcd/etcd-client.key"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571192   11092 flags.go:33] FLAG: --etcd-prefix="/registry"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571197   11092 flags.go:33] FLAG: --etcd-servers="[https://172.16.10.36:4001]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571206   11092 flags.go:33] FLAG: --etcd-servers-overrides="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571213   11092 flags.go:33] FLAG: --event-storage-age-limit="default=0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571219   11092 flags.go:33] FLAG: --event-storage-event-limit="default=0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571224   11092 flags.go:33] FLAG: --event-ttl="1h0m0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571237   11092 flags.go:33] FLAG: --experimental-encryption-provider-config=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571251   11092 flags.go:33] FLAG: --external-hostname=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571257   11092 flags.go:33] FLAG: --feature-gates=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571265   11092 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571271   11092 flags.go:33] FLAG: --help="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571277   11092 flags.go:33] FLAG: --housekeeping-interval="10s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571283   11092 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571288   11092 flags.go:33] FLAG: --insecure-bind-address="0.0.0.0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571300   11092 flags.go:33] FLAG: --insecure-port="8080"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571305   11092 flags.go:33] FLAG: --kubelet-certificate-authority=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571311   11092 flags.go:33] FLAG: --kubelet-client-certificate=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571315   11092 flags.go:33] FLAG: --kubelet-client-key=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571320   11092 flags.go:33] FLAG: --kubelet-https="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571326   11092 flags.go:33] FLAG: --kubelet-port="10250"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571333   11092 flags.go:33] FLAG: --kubelet-preferred-address-types="[Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571344   11092 flags.go:33] FLAG: --kubelet-read-only-port="10255"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571350   11092 flags.go:33] FLAG: --kubelet-timeout="5s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571355   11092 flags.go:33] FLAG: --kubernetes-service-node-port="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571361   11092 flags.go:33] FLAG: --log-backtrace-at=":0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571390   11092 flags.go:33] FLAG: --log-cadvisor-usage="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571397   11092 flags.go:33] FLAG: --log-dir=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571403   11092 flags.go:33] FLAG: --log-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571408   11092 flags.go:33] FLAG: --log-flush-frequency="5s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571414   11092 flags.go:33] FLAG: --logtostderr="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571419   11092 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571426   11092 flags.go:33] FLAG: --master-service-namespace="default"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571431   11092 flags.go:33] FLAG: --max-connection-bytes-per-sec="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571437   11092 flags.go:33] FLAG: --max-mutating-requests-inflight="200"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571443   11092 flags.go:33] FLAG: --max-requests-inflight="400"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571448   11092 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571454   11092 flags.go:33] FLAG: --mesos-agent-timeout="10s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571459   11092 flags.go:33] FLAG: --min-request-timeout="1800"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571465   11092 flags.go:33] FLAG: --oidc-ca-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571480   11092 flags.go:33] FLAG: --oidc-client-id=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571485   11092 flags.go:33] FLAG: --oidc-groups-claim=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571491   11092 flags.go:33] FLAG: --oidc-groups-prefix=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571496   11092 flags.go:33] FLAG: --oidc-issuer-url=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571501   11092 flags.go:33] FLAG: --oidc-required-claim=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571510   11092 flags.go:33] FLAG: --oidc-signing-algs="[RS256]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571518   11092 flags.go:33] FLAG: --oidc-username-claim="sub"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571524   11092 flags.go:33] FLAG: --oidc-username-prefix=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571529   11092 flags.go:33] FLAG: --port="8080"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571535   11092 flags.go:33] FLAG: --profiling="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571541   11092 flags.go:33] FLAG: --proxy-client-cert-file="/etc/kubernetes/ssl/kube-aggregator-proxy-client.crt"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571547   11092 flags.go:33] FLAG: --proxy-client-key-file="/etc/kubernetes/ssl/kube-aggregator-proxy-client.key"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571554   11092 flags.go:33] FLAG: --repair-malformed-updates="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571559   11092 flags.go:33] FLAG: --request-timeout="1m0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571565   11092 flags.go:33] FLAG: --requestheader-allowed-names="[system:kube-controller-manager]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571576   11092 flags.go:33] FLAG: --requestheader-client-ca-file="/etc/kubernetes/ssl/ca-kubernetes.crt"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571582   11092 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[X-Remote-Extra-]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571589   11092 flags.go:33] FLAG: --requestheader-group-headers="[X-Remote-Group]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571599   11092 flags.go:33] FLAG: --requestheader-username-headers="[X-Remote-User]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571606   11092 flags.go:33] FLAG: --runtime-config=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571614   11092 flags.go:33] FLAG: --secure-port="443"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571620   11092 flags.go:33] FLAG: --service-account-api-audiences="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571629   11092 flags.go:33] FLAG: --service-account-issuer=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571643   11092 flags.go:33] FLAG: --service-account-key-file="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571651   11092 flags.go:33] FLAG: --service-account-lookup="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571656   11092 flags.go:33] FLAG: --service-account-max-token-expiration="0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571662   11092 flags.go:33] FLAG: --service-account-signing-key-file=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571667   11092 flags.go:33] FLAG: --service-cluster-ip-range="10.254.0.0/16"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571675   11092 flags.go:33] FLAG: --service-node-port-range="30000-32767"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571684   11092 flags.go:33] FLAG: --skip-headers="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571689   11092 flags.go:33] FLAG: --ssh-keyfile=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571695   11092 flags.go:33] FLAG: --ssh-user=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571710   11092 flags.go:33] FLAG: --stderrthreshold="2"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571716   11092 flags.go:33] FLAG: --storage-backend=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571721   11092 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571727   11092 flags.go:33] FLAG: --storage-driver-db="cadvisor"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571732   11092 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571738   11092 flags.go:33] FLAG: --storage-driver-password="root"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571743   11092 flags.go:33] FLAG: --storage-driver-secure="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571749   11092 flags.go:33] FLAG: --storage-driver-table="stats"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571754   11092 flags.go:33] FLAG: --storage-driver-user="root"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571759   11092 flags.go:33] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571766   11092 flags.go:33] FLAG: --storage-versions="admission.k8s.io/v1beta1,admissionregistration.k8s.io/v1beta1,apps/v1,auditregistration.k8s.io/v1alpha1,authentication.k8s.io/v1,authorization.k8s.io/v1,autoscaling/v1,batch/v1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,events.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,scheduling.k8s.io/v1beta1,settings.k8s.io/v1alpha1,storage.k8s.io/v1,v1"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571952   11092 flags.go:33] FLAG: --target-ram-mb="0"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571965   11092 flags.go:33] FLAG: --tls-cert-file="/etc/kubernetes/ssl/kubernetes-server.crt"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571972   11092 flags.go:33] FLAG: --tls-cipher-suites="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571983   11092 flags.go:33] FLAG: --tls-min-version=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571989   11092 flags.go:33] FLAG: --tls-private-key-file="/etc/kubernetes/ssl/kubernetes-server.key"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.571995   11092 flags.go:33] FLAG: --tls-sni-cert-key="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572003   11092 flags.go:33] FLAG: --token-auth-file="/srv/kubernetes/known_tokens.csv"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572009   11092 flags.go:33] FLAG: --v="2"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572015   11092 flags.go:33] FLAG: --version="false"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572023   11092 flags.go:33] FLAG: --vmodule=""
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572028   11092 flags.go:33] FLAG: --watch-cache="true"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572034   11092 flags.go:33] FLAG: --watch-cache-sizes="[]"
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572051   11092 server.go:557] external host was not specified, using 172.16.10.36
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572618   11092 server.go:600] Initializing cache sizes based on 0MB limit
May 18 16:39:48 ctl01 kube-apiserver[11092]: I0518 16:39:48.572744   11092 server.go:146] Version: v1.13.5-3+98374c02d2d8c1
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.621286    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.721492    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.821640    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:48 ctl01 kubelet[8501]: E0518 16:39:48.921797    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.021949    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.122078    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.136590   11092 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.136770   11092 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.138437   11092 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.138463   11092 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.199832   11092 store.go:1414] Monitoring customresourcedefinitions.apiextensions.k8s.io count at <storage-prefix>//apiextensions.k8s.io/customresourcedefinitions
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.220460   11092 master.go:228] Using reconciler: lease
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.222252    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.322454    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.334633   11092 store.go:1414] Monitoring podtemplates count at <storage-prefix>//podtemplates
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.384510   11092 store.go:1414] Monitoring events count at <storage-prefix>//events
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.422700    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.436009   11092 store.go:1414] Monitoring limitranges count at <storage-prefix>//limitranges
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.490971   11092 store.go:1414] Monitoring resourcequotas count at <storage-prefix>//resourcequotas
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.522872    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.550594   11092 store.go:1414] Monitoring secrets count at <storage-prefix>//secrets
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.599886   11092 store.go:1414] Monitoring persistentvolumes count at <storage-prefix>//persistentvolumes
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.623028    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.651530   11092 store.go:1414] Monitoring persistentvolumeclaims count at <storage-prefix>//persistentvolumeclaims
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.704358   11092 store.go:1414] Monitoring configmaps count at <storage-prefix>//configmaps
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.723170    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.758713   11092 store.go:1414] Monitoring namespaces count at <storage-prefix>//namespaces
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.811801   11092 store.go:1414] Monitoring endpoints count at <storage-prefix>//services/endpoints
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.823349    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.867305   11092 store.go:1414] Monitoring nodes count at <storage-prefix>//minions
May 18 16:39:49 ctl01 kubelet[8501]: E0518 16:39:49.923555    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.933692   11092 store.go:1414] Monitoring pods count at <storage-prefix>//pods
May 18 16:39:49 ctl01 kube-apiserver[11092]: I0518 16:39:49.989025   11092 store.go:1414] Monitoring serviceaccounts count at <storage-prefix>//serviceaccounts
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.023836    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.054583   11092 store.go:1414] Monitoring services count at <storage-prefix>//services/specs
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.124081    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.224278    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.225014   11092 store.go:1414] Monitoring replicationcontrollers count at <storage-prefix>//controllers
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.324500    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.326371   11092 master.go:407] Skipping disabled API group "auditregistration.k8s.io".
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.326419   11092 master.go:415] Enabling API group "authentication.k8s.io".
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.326442   11092 master.go:415] Enabling API group "authorization.k8s.io".
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.377641   11092 store.go:1414] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.424830    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.453397   11092 store.go:1414] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.505814   11092 store.go:1414] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.505840   11092 master.go:415] Enabling API group "autoscaling".
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.525018    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.555611   11092 store.go:1414] Monitoring jobs.batch count at <storage-prefix>//jobs
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.607837   11092 store.go:1414] Monitoring cronjobs.batch count at <storage-prefix>//cronjobs
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.607885   11092 master.go:415] Enabling API group "batch".
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.625226    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.666270   11092 store.go:1414] Monitoring certificatesigningrequests.certificates.k8s.io count at <storage-prefix>//certificatesigningrequests
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.666328   11092 master.go:415] Enabling API group "certificates.k8s.io".
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.720928   11092 store.go:1414] Monitoring leases.coordination.k8s.io count at <storage-prefix>//leases
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.720996   11092 master.go:415] Enabling API group "coordination.k8s.io".
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.725370    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.774378   11092 store.go:1414] Monitoring replicationcontrollers count at <storage-prefix>//controllers
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.822938   11092 store.go:1414] Monitoring daemonsets.apps count at <storage-prefix>//daemonsets
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.825492    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.872053   11092 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.923548   11092 store.go:1414] Monitoring ingresses.extensions count at <storage-prefix>//ingress
May 18 16:39:50 ctl01 kubelet[8501]: E0518 16:39:50.925710    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:50 ctl01 kube-apiserver[11092]: I0518 16:39:50.970961   11092 store.go:1414] Monitoring podsecuritypolicies.policy count at <storage-prefix>//podsecuritypolicy
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.025993    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.036371   11092 store.go:1414] Monitoring replicasets.apps count at <storage-prefix>//replicasets
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.116476   11092 store.go:1414] Monitoring networkpolicies.networking.k8s.io count at <storage-prefix>//networkpolicies
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.116510   11092 master.go:415] Enabling API group "extensions".
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.126204    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.172197   11092 store.go:1414] Monitoring networkpolicies.networking.k8s.io count at <storage-prefix>//networkpolicies
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.172245   11092 master.go:415] Enabling API group "networking.k8s.io".
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.221341   11092 store.go:1414] Monitoring poddisruptionbudgets.policy count at <storage-prefix>//poddisruptionbudgets
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.226343    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.270390   11092 store.go:1414] Monitoring podsecuritypolicies.policy count at <storage-prefix>//podsecuritypolicy
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.270468   11092 master.go:415] Enabling API group "policy".
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.319617   11092 store.go:1414] Monitoring roles.rbac.authorization.k8s.io count at <storage-prefix>//roles
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.326532    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.367510   11092 store.go:1414] Monitoring rolebindings.rbac.authorization.k8s.io count at <storage-prefix>//rolebindings
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.417560   11092 store.go:1414] Monitoring clusterroles.rbac.authorization.k8s.io count at <storage-prefix>//clusterroles
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.426652    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.465918   11092 store.go:1414] Monitoring clusterrolebindings.rbac.authorization.k8s.io count at <storage-prefix>//clusterrolebindings
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.513843   11092 store.go:1414] Monitoring roles.rbac.authorization.k8s.io count at <storage-prefix>//roles
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.526872    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.568255   11092 store.go:1414] Monitoring rolebindings.rbac.authorization.k8s.io count at <storage-prefix>//rolebindings
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.614850   11092 store.go:1414] Monitoring clusterroles.rbac.authorization.k8s.io count at <storage-prefix>//clusterroles
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.627082    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.669366   11092 store.go:1414] Monitoring clusterrolebindings.rbac.authorization.k8s.io count at <storage-prefix>//clusterrolebindings
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.669409   11092 master.go:415] Enabling API group "rbac.authorization.k8s.io".
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.722601   11092 store.go:1414] Monitoring priorityclasses.scheduling.k8s.io count at <storage-prefix>//priorityclasses
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.722630   11092 master.go:415] Enabling API group "scheduling.k8s.io".
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.722653   11092 master.go:407] Skipping disabled API group "settings.k8s.io".
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.727273    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.784617   11092 store.go:1414] Monitoring storageclasses.storage.k8s.io count at <storage-prefix>//storageclasses
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.827477    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.844445   11092 store.go:1414] Monitoring volumeattachments.storage.k8s.io count at <storage-prefix>//volumeattachments
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.904188   11092 store.go:1414] Monitoring storageclasses.storage.k8s.io count at <storage-prefix>//storageclasses
May 18 16:39:51 ctl01 kubelet[8501]: E0518 16:39:51.927719    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.954141   11092 store.go:1414] Monitoring volumeattachments.storage.k8s.io count at <storage-prefix>//volumeattachments
May 18 16:39:51 ctl01 kube-apiserver[11092]: I0518 16:39:51.954177   11092 master.go:415] Enabling API group "storage.k8s.io".
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.000107   11092 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.027945    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.055128   11092 store.go:1414] Monitoring statefulsets.apps count at <storage-prefix>//statefulsets
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.104794   11092 store.go:1414] Monitoring controllerrevisions.apps count at <storage-prefix>//controllerrevisions
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.113642    8501 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.128152    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.154706   11092 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.205270   11092 store.go:1414] Monitoring statefulsets.apps count at <storage-prefix>//statefulsets
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.228445    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.255283   11092 store.go:1414] Monitoring daemonsets.apps count at <storage-prefix>//daemonsets
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.312487   11092 store.go:1414] Monitoring replicasets.apps count at <storage-prefix>//replicasets
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.328660    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.362588   11092 store.go:1414] Monitoring controllerrevisions.apps count at <storage-prefix>//controllerrevisions
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.420337   11092 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.428909    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.464484   11092 store.go:1414] Monitoring statefulsets.apps count at <storage-prefix>//statefulsets
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.516112   11092 store.go:1414] Monitoring daemonsets.apps count at <storage-prefix>//daemonsets
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.529192    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.569084   11092 store.go:1414] Monitoring replicasets.apps count at <storage-prefix>//replicasets
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.614830   11092 store.go:1414] Monitoring controllerrevisions.apps count at <storage-prefix>//controllerrevisions
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.614871   11092 master.go:415] Enabling API group "apps".
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.629453    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.674498   11092 store.go:1414] Monitoring validatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//validatingwebhookconfigurations
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.725502   11092 store.go:1414] Monitoring mutatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//mutatingwebhookconfigurations
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.725531   11092 master.go:415] Enabling API group "admissionregistration.k8s.io".
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.729749    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.776896   11092 store.go:1414] Monitoring events count at <storage-prefix>//events
May 18 16:39:52 ctl01 kube-apiserver[11092]: I0518 16:39:52.776928   11092 master.go:415] Enabling API group "events.k8s.io".
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.830034    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:52 ctl01 kube-apiserver[11092]: W0518 16:39:52.883655   11092 genericapiserver.go:338] Skipping API batch/v2alpha1 because it has no resources.
May 18 16:39:52 ctl01 kubelet[8501]: E0518 16:39:52.930460    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.030754    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kube-apiserver[11092]: W0518 16:39:53.069399   11092 genericapiserver.go:338] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
May 18 16:39:53 ctl01 kube-apiserver[11092]: W0518 16:39:53.083785   11092 genericapiserver.go:338] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
May 18 16:39:53 ctl01 kube-apiserver[11092]: W0518 16:39:53.098886   11092 genericapiserver.go:338] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.130962    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kube-apiserver[11092]: W0518 16:39:53.194839   11092 genericapiserver.go:338] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.231141    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kube-apiserver[11092]: [restful] 2019/05/18 16:39:53 log.go:33: [restful/swagger] listing is available at https://172.16.10.36:443/swaggerapi
May 18 16:39:53 ctl01 kube-apiserver[11092]: [restful] 2019/05/18 16:39:53 log.go:33: [restful/swagger] https://172.16.10.36:443/swaggerui/ is mapped to folder /swagger-ui/
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.331333    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.431550    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.531746    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.631908    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kubelet[8501]: I0518 16:39:53.715058    8501 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 18 16:39:53 ctl01 kubelet[8501]: I0518 16:39:53.715346    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:39:53 ctl01 kubelet[8501]: I0518 16:39:53.716279    8501 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
May 18 16:39:53 ctl01 kubelet[8501]: I0518 16:39:53.716314    8501 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
May 18 16:39:53 ctl01 kubelet[8501]: I0518 16:39:53.716327    8501 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
May 18 16:39:53 ctl01 kubelet[8501]: I0518 16:39:53.716349    8501 kubelet_node_status.go:72] Attempting to register node ctl01
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.732093    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.832287    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:53 ctl01 kubelet[8501]: E0518 16:39:53.932522    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.032760    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.133000    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.233176    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.333422    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.433656    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.533862    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.634065    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.734316    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.834594    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:54 ctl01 kubelet[8501]: E0518 16:39:54.934804    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.035002    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.135181    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.235422    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.335593    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.435747    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.535924    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kube-apiserver[11092]: [restful] 2019/05/18 16:39:55 log.go:33: [restful/swagger] listing is available at https://172.16.10.36:443/swaggerapi
May 18 16:39:55 ctl01 kube-apiserver[11092]: [restful] 2019/05/18 16:39:55 log.go:33: [restful/swagger] https://172.16.10.36:443/swaggerui/ is mapped to folder /swagger-ui/
May 18 16:39:55 ctl01 kube-apiserver[11092]: I0518 16:39:55.621647   11092 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
May 18 16:39:55 ctl01 kube-apiserver[11092]: I0518 16:39:55.621682   11092 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.636127    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kube-apiserver[11092]: I0518 16:39:55.677990   11092 store.go:1414] Monitoring apiservices.apiregistration.k8s.io count at <storage-prefix>//apiregistration.k8s.io/apiservices
May 18 16:39:55 ctl01 kube-apiserver[11092]: I0518 16:39:55.730116   11092 store.go:1414] Monitoring apiservices.apiregistration.k8s.io count at <storage-prefix>//apiregistration.k8s.io/apiservices
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.736313    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.836608    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:55 ctl01 kubelet[8501]: E0518 16:39:55.936805    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.036961    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.137409    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.237580    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.337914    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.438200    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.538388    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.638547    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.738684    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.838843    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:56 ctl01 kubelet[8501]: E0518 16:39:56.939019    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.039246    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.139411    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.239622    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.339812    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.440167    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.540620    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.640887    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.741156    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.841436    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:57 ctl01 kubelet[8501]: E0518 16:39:57.941835    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.042124    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.142454    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.242802    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.343125    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.443432    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.543622    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.643895    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.744198    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.844566    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:58 ctl01 kubelet[8501]: E0518 16:39:58.944869    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.015905    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.016894    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.018100    8501 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.045127    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.145306    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.245505    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.345685    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.445889    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kube-proxy[8698]: I0518 16:39:59.531297    8698 proxier.go:645] Not syncing iptables until Services and Endpoints have been received from master
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.546116    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kube-proxy[8698]: E0518 16:39:59.573615    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 18 16:39:59 ctl01 kube-proxy[8698]: E0518 16:39:59.574527    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.646303    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.746479    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.846660    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:39:59 ctl01 kubelet[8501]: E0518 16:39:59.946806    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.047105    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kube-proxy[8698]: E0518 16:40:00.058734    8698 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: net/http: TLS handshake timeout' (may retry after sleeping)
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.147326    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.247486    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.347662    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.447863    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.548112    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.602716   11092 deprecated_insecure_serving.go:51] Serving insecurely on [::]:8080
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.604460   11092 secure_serving.go:116] Serving securely on [::]:443
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.604616   11092 controller.go:84] Starting OpenAPI AggregationController
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.604661   11092 autoregister_controller.go:136] Starting autoregister controller
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.604692   11092 cache.go:32] Waiting for caches to sync for autoregister controller
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.604712   11092 available_controller.go:283] Starting AvailableConditionController
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.604737   11092 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.604776   11092 crd_finalizer.go:242] Starting CRDFinalizer
May 18 16:40:00 ctl01 systemd[1]: Started Kubernetes API Server.
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.605819   11092 customresource_discovery_controller.go:203] Starting DiscoveryController
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.605856   11092 naming_controller.go:284] Starting NamingConditionController
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.605876   11092 establishing_controller.go:73] Starting EstablishingController
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.606900   11092 crdregistration_controller.go:112] Starting crd-autoregister controller
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.606914   11092 controller_utils.go:1027] Waiting for caches to sync for crd-autoregister controller
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-apiserver.service'] in directory '/root'
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.631638   11092 apiservice_controller.go:90] Starting APIServiceRegistrationController
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.632018   11092 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.633742   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41700: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.634733   11092 log.go:172] http: TLS handshake error from 172.16.10.56:55488: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.635632   11092 log.go:172] http: TLS handshake error from 172.16.10.36:60036: EOF
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.648266    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.653973   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41686: EOF
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-apiserver.service'] in directory '/root'
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.669760   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41688: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.672626   11092 log.go:172] http: TLS handshake error from 172.16.10.36:59992: EOF
May 18 16:40:00 ctl01 systemd[1]: Started /bin/systemctl enable kube-apiserver.service.
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.681356   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41690: EOF
May 18 16:40:00 ctl01 systemd[1]: Reloading.
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.689072   11092 log.go:172] http: TLS handshake error from 172.16.10.56:55490: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.701070   11092 log.go:172] http: TLS handshake error from 172.16.10.36:59994: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.704074   11092 log.go:172] http: TLS handshake error from 172.16.10.36:59996: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.714188   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41694: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.727641   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41696: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.735827   11092 log.go:172] http: TLS handshake error from 172.16.10.56:55486: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.737685   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41692: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.739248   11092 log.go:172] http: TLS handshake error from 172.16.10.56:55492: EOF
May 18 16:40:00 ctl01 kubelet[8501]: E0518 16:40:00.748424    8501 kubelet.go:2266] node "ctl01" not found
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.753071   11092 log.go:172] http: TLS handshake error from 172.16.10.55:41698: EOF
May 18 16:40:00 ctl01 kubelet[8501]: I0518 16:40:00.753606    8501 kubelet_node_status.go:75] Successfully registered node ctl01
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.753704   11092 log.go:172] http: TLS handshake error from 172.16.10.56:55494: EOF
May 18 16:40:00 ctl01 kubelet[8501]: I0518 16:40:00.755124    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.768426   11092 log.go:172] http: TLS handshake error from 172.16.10.36:60014: EOF
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.784977   11092 log.go:172] http: TLS handshake error from 172.16.10.36:60016: EOF
May 18 16:40:00 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
May 18 16:40:00 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
May 18 16:40:00 ctl01 kubelet[8501]: I0518 16:40:00.801267    8501 kubelet.go:1908] SyncLoop (ADD, "api"): ""
May 18 16:40:00 ctl01 kube-apiserver[11092]: W0518 16:40:00.803121   11092 lease.go:222] Resetting endpoints for master service "kubernetes" to [172.16.10.36]
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.803981   11092 controller.go:608] quota admission added evaluator for: endpoints
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.804841   11092 cache.go:39] Caches are synced for AvailableConditionController controller
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.805941   11092 cache.go:39] Caches are synced for autoregister controller
May 18 16:40:00 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.810624   11092 controller_utils.go:1034] Caches are synced for crd-autoregister controller
May 18 16:40:00 ctl01 kube-proxy[8698]: E0518 16:40:00.815227    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] {'kube-apiserver': True}
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Completed state [kube-apiserver] at time 16:40:00.821343 duration_in_ms=12459.247
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Running state [kube-scheduler] at time 16:40:00.827728
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [kube-scheduler]
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kube-scheduler.service', '-n', '0'] in directory '/root'
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.831899   11092 cacher.go:605] cacher (*apiregistration.APIService): 1 objects queued in incoming channel.
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.831928   11092 cacher.go:605] cacher (*apiregistration.APIService): 2 objects queued in incoming channel.
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.832493   11092 cache.go:39] Caches are synced for APIServiceRegistrationController controller
May 18 16:40:00 ctl01 kube-proxy[8698]: E0518 16:40:00.832593    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-proxy" cannot list resource "endpoints" in API group "" at the cluster scope
May 18 16:40:00 ctl01 kube-apiserver[11092]: I0518 16:40:00.833614   11092 cacher.go:605] cacher (*apiregistration.APIService): 1 objects queued in incoming channel.
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:00 ctl01 systemd[1]: Started /bin/systemctl start kube-scheduler.service.
May 18 16:40:00 ctl01 kubelet[8501]: I0518 16:40:00.910265    8501 reconciler.go:154] Reconciler: start to sync state
May 18 16:40:00 ctl01 systemd[1]: Started Kubernetes Scheduler Plugin.
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:00 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:00 ctl01 systemd[1]: Started /bin/systemctl enable kube-scheduler.service.
May 18 16:40:00 ctl01 systemd[1]: Reloading.
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057738   11188 flags.go:33] FLAG: --address="0.0.0.0"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057910   11188 flags.go:33] FLAG: --algorithm-provider=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057930   11188 flags.go:33] FLAG: --alsologtostderr="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057942   11188 flags.go:33] FLAG: --application-metrics-count-limit="100"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057951   11188 flags.go:33] FLAG: --authentication-kubeconfig=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057959   11188 flags.go:33] FLAG: --authentication-skip-lookup="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057970   11188 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057980   11188 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="true"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.057988   11188 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058008   11188 flags.go:33] FLAG: --authorization-kubeconfig=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058017   11188 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058025   11188 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058033   11188 flags.go:33] FLAG: --azure-container-registry-config=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058041   11188 flags.go:33] FLAG: --bind-address="0.0.0.0"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058051   11188 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058060   11188 flags.go:33] FLAG: --cert-dir=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058069   11188 flags.go:33] FLAG: --client-ca-file=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058077   11188 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058091   11188 flags.go:33] FLAG: --config=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058099   11188 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058108   11188 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058116   11188 flags.go:33] FLAG: --contention-profiling="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058124   11188 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058151   11188 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058165   11188 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058173   11188 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058181   11188 flags.go:33] FLAG: --docker-only="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058189   11188 flags.go:33] FLAG: --docker-root="/var/lib/docker"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058197   11188 flags.go:33] FLAG: --docker-tls="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058205   11188 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058213   11188 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058222   11188 flags.go:33] FLAG: --docker-tls-key="key.pem"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058231   11188 flags.go:33] FLAG: --enable-load-reader="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058238   11188 flags.go:33] FLAG: --event-storage-age-limit="default=0"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058247   11188 flags.go:33] FLAG: --event-storage-event-limit="default=0"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058254   11188 flags.go:33] FLAG: --failure-domains="kubernetes.io/hostname,failure-domain.beta.kubernetes.io/zone,failure-domain.beta.kubernetes.io/region"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058266   11188 flags.go:33] FLAG: --feature-gates=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058278   11188 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058287   11188 flags.go:33] FLAG: --hard-pod-affinity-symmetric-weight="1"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058297   11188 flags.go:33] FLAG: --help="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058305   11188 flags.go:33] FLAG: --housekeeping-interval="10s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058313   11188 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058323   11188 flags.go:33] FLAG: --kube-api-burst="100"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058331   11188 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058339   11188 flags.go:33] FLAG: --kube-api-qps="50"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058350   11188 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/scheduler.kubeconfig"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058359   11188 flags.go:33] FLAG: --leader-elect="true"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058366   11188 flags.go:33] FLAG: --leader-elect-lease-duration="15s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058375   11188 flags.go:33] FLAG: --leader-elect-renew-deadline="10s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058383   11188 flags.go:33] FLAG: --leader-elect-resource-lock="endpoints"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058391   11188 flags.go:33] FLAG: --leader-elect-retry-period="2s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058399   11188 flags.go:33] FLAG: --lock-object-name="kube-scheduler"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058406   11188 flags.go:33] FLAG: --lock-object-namespace="kube-system"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058414   11188 flags.go:33] FLAG: --log-backtrace-at=":0"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058427   11188 flags.go:33] FLAG: --log-cadvisor-usage="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058435   11188 flags.go:33] FLAG: --log-dir=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058443   11188 flags.go:33] FLAG: --log-file=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058461   11188 flags.go:33] FLAG: --log-flush-frequency="5s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058469   11188 flags.go:33] FLAG: --logtostderr="true"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058477   11188 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058486   11188 flags.go:33] FLAG: --master=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058494   11188 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058503   11188 flags.go:33] FLAG: --mesos-agent-timeout="10s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058511   11188 flags.go:33] FLAG: --policy-config-file=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058518   11188 flags.go:33] FLAG: --policy-configmap=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058526   11188 flags.go:33] FLAG: --policy-configmap-namespace="kube-system"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058534   11188 flags.go:33] FLAG: --port="10251"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058544   11188 flags.go:33] FLAG: --profiling="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058551   11188 flags.go:33] FLAG: --requestheader-allowed-names="[]"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058566   11188 flags.go:33] FLAG: --requestheader-client-ca-file=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058574   11188 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058590   11188 flags.go:33] FLAG: --requestheader-group-headers="[x-remote-group]"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058602   11188 flags.go:33] FLAG: --requestheader-username-headers="[x-remote-user]"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058617   11188 flags.go:33] FLAG: --scheduler-name="default-scheduler"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058626   11188 flags.go:33] FLAG: --secure-port="10259"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058634   11188 flags.go:33] FLAG: --skip-headers="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058643   11188 flags.go:33] FLAG: --stderrthreshold="2"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058650   11188 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058659   11188 flags.go:33] FLAG: --storage-driver-db="cadvisor"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058667   11188 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058675   11188 flags.go:33] FLAG: --storage-driver-password="root"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058682   11188 flags.go:33] FLAG: --storage-driver-secure="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058690   11188 flags.go:33] FLAG: --storage-driver-table="stats"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058698   11188 flags.go:33] FLAG: --storage-driver-user="root"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058706   11188 flags.go:33] FLAG: --tls-cert-file=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058713   11188 flags.go:33] FLAG: --tls-cipher-suites="[]"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058726   11188 flags.go:33] FLAG: --tls-min-version=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058733   11188 flags.go:33] FLAG: --tls-private-key-file=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058741   11188 flags.go:33] FLAG: --tls-sni-cert-key="[]"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058760   11188 flags.go:33] FLAG: --use-legacy-policy-config="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058769   11188 flags.go:33] FLAG: --v="2"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058777   11188 flags.go:33] FLAG: --version="false"
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058789   11188 flags.go:33] FLAG: --vmodule=""
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.058798   11188 flags.go:33] FLAG: --write-config-to=""
May 18 16:40:01 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
May 18 16:40:01 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
May 18 16:40:01 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] {'kube-scheduler': True}
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Completed state [kube-scheduler] at time 16:40:01.093781 duration_in_ms=266.053
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Running state [kube-controller-manager] at time 16:40:01.098443
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [kube-controller-manager]
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kube-controller-manager.service', '-n', '0'] in directory '/root'
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 systemd[1]: Started /bin/systemctl start kube-controller-manager.service.
May 18 16:40:01 ctl01 systemd[1]: Started Kubernetes Controller Manager.
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 systemd[1]: Started /bin/systemctl enable kube-controller-manager.service.
May 18 16:40:01 ctl01 systemd[1]: Reloading.
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.257333   11188 serving.go:318] Generated self-signed cert in-memory
May 18 16:40:01 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
May 18 16:40:01 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307554   11262 flags.go:33] FLAG: --address="0.0.0.0"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307721   11262 flags.go:33] FLAG: --allocate-node-cidrs="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307741   11262 flags.go:33] FLAG: --allow-untagged-cloud="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307748   11262 flags.go:33] FLAG: --alsologtostderr="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307755   11262 flags.go:33] FLAG: --application-metrics-count-limit="100"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307762   11262 flags.go:33] FLAG: --attach-detach-reconcile-sync-period="1m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307769   11262 flags.go:33] FLAG: --authentication-kubeconfig=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307775   11262 flags.go:33] FLAG: --authentication-skip-lookup="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307872   11262 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307882   11262 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307888   11262 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307898   11262 flags.go:33] FLAG: --authorization-kubeconfig=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307904   11262 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307910   11262 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307915   11262 flags.go:33] FLAG: --azure-container-registry-config=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307920   11262 flags.go:33] FLAG: --bind-address="0.0.0.0"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307927   11262 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307933   11262 flags.go:33] FLAG: --cert-dir=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307938   11262 flags.go:33] FLAG: --cidr-allocator-type="RangeAllocator"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307944   11262 flags.go:33] FLAG: --client-ca-file=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307949   11262 flags.go:33] FLAG: --cloud-config=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307954   11262 flags.go:33] FLAG: --cloud-provider=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307959   11262 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307968   11262 flags.go:33] FLAG: --cluster-cidr=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307973   11262 flags.go:33] FLAG: --cluster-name="kubernetes"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307979   11262 flags.go:33] FLAG: --cluster-signing-cert-file="/etc/kubernetes/ca/ca.pem"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307985   11262 flags.go:33] FLAG: --cluster-signing-key-file="/etc/kubernetes/ca/ca.key"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307990   11262 flags.go:33] FLAG: --concurrent-deployment-syncs="5"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.307997   11262 flags.go:33] FLAG: --concurrent-endpoint-syncs="5"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308003   11262 flags.go:33] FLAG: --concurrent-gc-syncs="20"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308008   11262 flags.go:33] FLAG: --concurrent-namespace-syncs="10"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308013   11262 flags.go:33] FLAG: --concurrent-replicaset-syncs="5"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308019   11262 flags.go:33] FLAG: --concurrent-resource-quota-syncs="5"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308024   11262 flags.go:33] FLAG: --concurrent-service-syncs="1"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308030   11262 flags.go:33] FLAG: --concurrent-serviceaccount-token-syncs="5"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308035   11262 flags.go:33] FLAG: --concurrent-ttl-after-finished-syncs="5"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308040   11262 flags.go:33] FLAG: --concurrent_rc_syncs="5"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308045   11262 flags.go:33] FLAG: --configure-cloud-routes="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308051   11262 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308057   11262 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308062   11262 flags.go:33] FLAG: --contention-profiling="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308068   11262 flags.go:33] FLAG: --controller-start-interval="0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308082   11262 flags.go:33] FLAG: --controllers="[*]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308098   11262 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308104   11262 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308110   11262 flags.go:33] FLAG: --deleting-pods-burst="0"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308116   11262 flags.go:33] FLAG: --deleting-pods-qps="0.1"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308125   11262 flags.go:33] FLAG: --deployment-controller-sync-period="30s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308131   11262 flags.go:33] FLAG: --disable-attach-detach-reconcile-sync="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308136   11262 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308142   11262 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308147   11262 flags.go:33] FLAG: --docker-only="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308152   11262 flags.go:33] FLAG: --docker-root="/var/lib/docker"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308158   11262 flags.go:33] FLAG: --docker-tls="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308163   11262 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308169   11262 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308174   11262 flags.go:33] FLAG: --docker-tls-key="key.pem"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308179   11262 flags.go:33] FLAG: --enable-dynamic-provisioning="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308184   11262 flags.go:33] FLAG: --enable-garbage-collector="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308190   11262 flags.go:33] FLAG: --enable-hostpath-provisioner="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308195   11262 flags.go:33] FLAG: --enable-load-reader="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308200   11262 flags.go:33] FLAG: --enable-taint-manager="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308205   11262 flags.go:33] FLAG: --event-storage-age-limit="default=0"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308210   11262 flags.go:33] FLAG: --event-storage-event-limit="default=0"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308216   11262 flags.go:33] FLAG: --experimental-cluster-signing-duration="8760h0m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308221   11262 flags.go:33] FLAG: --external-cloud-volume-plugin=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308226   11262 flags.go:33] FLAG: --feature-gates=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308235   11262 flags.go:33] FLAG: --flex-volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308241   11262 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308247   11262 flags.go:33] FLAG: --help="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308252   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-cpu-initialization-period="5m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308258   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-downscale-delay="5m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308263   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-downscale-stabilization="5m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308269   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-initial-readiness-delay="30s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308283   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-sync-period="15s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308289   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-tolerance="0.1"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308297   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-upscale-delay="3m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308302   11262 flags.go:33] FLAG: --horizontal-pod-autoscaler-use-rest-clients="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308308   11262 flags.go:33] FLAG: --housekeeping-interval="10s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308313   11262 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308321   11262 flags.go:33] FLAG: --kube-api-burst="30"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308326   11262 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308332   11262 flags.go:33] FLAG: --kube-api-qps="20"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308338   11262 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/controller-manager.kubeconfig"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308345   11262 flags.go:33] FLAG: --large-cluster-size-threshold="50"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308350   11262 flags.go:33] FLAG: --leader-elect="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308355   11262 flags.go:33] FLAG: --leader-elect-lease-duration="15s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308361   11262 flags.go:33] FLAG: --leader-elect-renew-deadline="10s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308366   11262 flags.go:33] FLAG: --leader-elect-resource-lock="endpoints"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308372   11262 flags.go:33] FLAG: --leader-elect-retry-period="2s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308377   11262 flags.go:33] FLAG: --log-backtrace-at=":0"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308385   11262 flags.go:33] FLAG: --log-cadvisor-usage="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308391   11262 flags.go:33] FLAG: --log-dir=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308396   11262 flags.go:33] FLAG: --log-file=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308401   11262 flags.go:33] FLAG: --log-flush-frequency="5s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308407   11262 flags.go:33] FLAG: --logtostderr="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308412   11262 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308418   11262 flags.go:33] FLAG: --master=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308423   11262 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308429   11262 flags.go:33] FLAG: --mesos-agent-timeout="10s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308434   11262 flags.go:33] FLAG: --min-resync-period="12h0m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308439   11262 flags.go:33] FLAG: --namespace-sync-period="5m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308445   11262 flags.go:33] FLAG: --node-cidr-mask-size="24"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308450   11262 flags.go:33] FLAG: --node-eviction-rate="0.1"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308457   11262 flags.go:33] FLAG: --node-monitor-grace-period="40s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308462   11262 flags.go:33] FLAG: --node-monitor-period="5s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308477   11262 flags.go:33] FLAG: --node-startup-grace-period="1m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308482   11262 flags.go:33] FLAG: --node-sync-period="0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308488   11262 flags.go:33] FLAG: --pod-eviction-timeout="5m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308493   11262 flags.go:33] FLAG: --port="10252"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308499   11262 flags.go:33] FLAG: --profiling="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308504   11262 flags.go:33] FLAG: --pv-recycler-increment-timeout-nfs="30"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308509   11262 flags.go:33] FLAG: --pv-recycler-minimum-timeout-hostpath="60"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308515   11262 flags.go:33] FLAG: --pv-recycler-minimum-timeout-nfs="300"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308521   11262 flags.go:33] FLAG: --pv-recycler-pod-template-filepath-hostpath=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308526   11262 flags.go:33] FLAG: --pv-recycler-pod-template-filepath-nfs=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308530   11262 flags.go:33] FLAG: --pv-recycler-timeout-increment-hostpath="30"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308536   11262 flags.go:33] FLAG: --pvclaimbinder-sync-period="15s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308541   11262 flags.go:33] FLAG: --register-retry-count="10"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308546   11262 flags.go:33] FLAG: --requestheader-allowed-names="[]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308553   11262 flags.go:33] FLAG: --requestheader-client-ca-file=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308558   11262 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308571   11262 flags.go:33] FLAG: --requestheader-group-headers="[x-remote-group]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308578   11262 flags.go:33] FLAG: --requestheader-username-headers="[x-remote-user]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308587   11262 flags.go:33] FLAG: --resource-quota-sync-period="5m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308593   11262 flags.go:33] FLAG: --root-ca-file="/etc/kubernetes/ssl/ca-kubernetes.crt"
May 18 16:40:01 ctl01 systemd[1]: Started ACPI event daemon.
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308599   11262 flags.go:33] FLAG: --route-reconciliation-period="10s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308605   11262 flags.go:33] FLAG: --secondary-node-eviction-rate="0.01"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308611   11262 flags.go:33] FLAG: --secure-port="10257"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308617   11262 flags.go:33] FLAG: --service-account-private-key-file="/etc/kubernetes/ssl/kubernetes-server.key"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308623   11262 flags.go:33] FLAG: --service-cluster-ip-range=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308628   11262 flags.go:33] FLAG: --skip-headers="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308633   11262 flags.go:33] FLAG: --stderrthreshold="2"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308638   11262 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308644   11262 flags.go:33] FLAG: --storage-driver-db="cadvisor"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308649   11262 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308655   11262 flags.go:33] FLAG: --storage-driver-password="root"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308660   11262 flags.go:33] FLAG: --storage-driver-secure="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308674   11262 flags.go:33] FLAG: --storage-driver-table="stats"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308680   11262 flags.go:33] FLAG: --storage-driver-user="root"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308686   11262 flags.go:33] FLAG: --terminated-pod-gc-threshold="12500"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308691   11262 flags.go:33] FLAG: --tls-cert-file=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308697   11262 flags.go:33] FLAG: --tls-cipher-suites="[]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308703   11262 flags.go:33] FLAG: --tls-min-version=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308708   11262 flags.go:33] FLAG: --tls-private-key-file=""
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308714   11262 flags.go:33] FLAG: --tls-sni-cert-key="[]"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308722   11262 flags.go:33] FLAG: --unhealthy-zone-threshold="0.55"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308728   11262 flags.go:33] FLAG: --use-service-account-credentials="true"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308733   11262 flags.go:33] FLAG: --v="2"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308739   11262 flags.go:33] FLAG: --version="false"
May 18 16:40:01 ctl01 kube-controller-manager[11262]: I0518 16:40:01.308747   11262 flags.go:33] FLAG: --vmodule=""
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] {'kube-controller-manager': True}
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Completed state [kube-controller-manager] at time 16:40:01.339140 duration_in_ms=240.696
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Running state [kubectl create ns "netchecker"] at time 16:40:01.339832
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing state cmd.run for [kubectl create ns "netchecker"]
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command 'kubectl get ns -o=custom-columns=NAME:.metadata.name | grep -v NAME | grep "netchecker"' in directory '/root'
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.608415   11092 storage_scheduling.go:91] created PriorityClass system-node-critical with value 2000001000
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing command 'kubectl create ns "netchecker"' in directory '/root'
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.615637   11092 storage_scheduling.go:91] created PriorityClass system-cluster-critical with value 2000000000
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.615658   11092 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.642453   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/cluster-admin
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.645908   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:discovery
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.649509   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:basic-user
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.652495   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/admin
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.655564   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/edit
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.658855   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/view
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.662995   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-admin
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.668929   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-edit
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.673245   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-view
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.676684   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:heapster
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.679446   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.681978   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-problem-detector
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.684538   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-proxier
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.687172   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kubelet-api-admin
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.689968   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-bootstrapper
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.692711   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:auth-delegator
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.695394   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-aggregator
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.698234   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-controller-manager
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.701219   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-scheduler
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.703866   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-dns
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.706727   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:persistent-volume-provisioner
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.709730   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:csi-external-attacher
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.712847   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aws-cloud-provider
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.716110   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:nodeclient
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.719615   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.723870   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:volume-scheduler
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.726644   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:csi-external-provisioner
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.729689   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:attachdetach-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.732759   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.735786   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:cronjob-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.738450   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:daemon-set-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.741128   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:deployment-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.743890   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:disruption-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.746872   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:endpoint-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.749554   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:expand-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.752335   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.755231   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.757865   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:job-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.760770   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:namespace-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.764436   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:node-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.767861   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.771075   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.774255   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:replicaset-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.778571   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:replication-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.781659   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:resourcequota-controller
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] {'pid': 11349, 'retcode': 0, 'stderr': '', 'stdout': 'namespace/netchecker created'}
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.784663   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:route-controller
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Completed state [kubectl create ns "netchecker"] at time 16:40:01.784651 duration_in_ms=444.819
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/calicoctl] at time 16:40:01.785087
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/bin/calicoctl]
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.788168   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:service-account-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.791189   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:service-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.794440   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:statefulset-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.797365   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:ttl-controller
May 18 16:40:01 ctl01 kube-proxy[8698]: E0518 16:40:01.816810    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.827788   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:certificate-controller
May 18 16:40:01 ctl01 kube-proxy[8698]: E0518 16:40:01.834091    8698 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-proxy" cannot list resource "endpoints" in API group "" at the cluster scope
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.868129   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.907886   11092 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pv-protection-controller
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] File /usr/bin/calicoctl is in the correct state
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/calicoctl] at time 16:40:01.921665 duration_in_ms=136.577
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/birdcl] at time 16:40:01.921949
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/usr/bin/birdcl]
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] File /usr/bin/birdcl is in the correct state
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/birdcl] at time 16:40:01.925745 duration_in_ms=3.796
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Running state [/opt/cni/bin/calico] at time 16:40:01.925999
May 18 16:40:01 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico]
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.947989   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/cluster-admin
May 18 16:40:01 ctl01 kube-scheduler[11188]: W0518 16:40:01.965426   11188 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
May 18 16:40:01 ctl01 kube-scheduler[11188]: W0518 16:40:01.965537   11188 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
May 18 16:40:01 ctl01 kube-scheduler[11188]: W0518 16:40:01.965566   11188 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.972704   11188 server.go:150] Version: v1.13.5-3+98374c02d2d8c1
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.972750   11188 defaults.go:210] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.973017   11188 factory.go:1157] Creating scheduler from algorithm provider 'DefaultProvider'
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.973043   11188 factory.go:1256] Creating scheduler with fit predicates 'map[CheckVolumeBinding:{} NoVolumeZoneConflict:{} MaxEBSVolumeCount:{} MaxCSIVolumeCountPred:{} CheckNodeUnschedulable:{} NoDiskConflict:{} MaxGCEPDVolumeCount:{} MatchInterPodAffinity:{} PodToleratesNodeTaints:{} MaxAzureDiskVolumeCount:{} GeneralPredicates:{}]' and priority functions 'map[TaintTolerationPriority:{} ImageLocalityPriority:{} SelectorSpreadPriority:{} InterPodAffinityPriority:{} LeastRequestedPriority:{} BalancedResourceAllocation:{} NodePreferAvoidPodsPriority:{} NodeAffinityPriority:{}]'
May 18 16:40:01 ctl01 kube-scheduler[11188]: W0518 16:40:01.973711   11188 authorization.go:47] Authorization is disabled
May 18 16:40:01 ctl01 kube-scheduler[11188]: W0518 16:40:01.973724   11188 authentication.go:55] Authentication is disabled
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.973735   11188 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251
May 18 16:40:01 ctl01 kube-scheduler[11188]: I0518 16:40:01.974197   11188 secure_serving.go:116] Serving securely on [::]:10259
May 18 16:40:01 ctl01 kube-apiserver[11092]: I0518 16:40:01.999564   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:discovery
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.042262   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:basic-user
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.046356   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.046487   11188 reflector.go:125] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.046589   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] File /opt/cni/bin/calico is in the correct state
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Completed state [/opt/cni/bin/calico] at time 16:40:02.049282 duration_in_ms=123.281
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Running state [/opt/cni/bin/calico-ipam] at time 16:40:02.049566
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico-ipam]
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.061652   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.062709   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.068242   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node-proxier
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.074099   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.075127   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.078197   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.081043   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
May 18 16:40:02 ctl01 kube-scheduler[11188]: E0518 16:40:02.087653   11188 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.108660   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-controller-manager
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.148328   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-dns
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] File /opt/cni/bin/calico-ipam is in the correct state
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Completed state [/opt/cni/bin/calico-ipam] at time 16:40:02.169166 duration_in_ms=119.599
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/cni/net.d/10-calico.conf] at time 16:40:02.169459
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/cni/net.d/10-calico.conf]
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.189407   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-scheduler
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.230067   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:aws-cloud-provider
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.268944   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.309513   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:volume-scheduler
May 18 16:40:02 ctl01 kube-controller-manager[11262]: I0518 16:40:02.314010   11262 serving.go:318] Generated self-signed cert in-memory
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.349949   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:attachdetach-controller
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.390388   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.428147   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:cronjob-controller
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.468170   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:daemon-set-controller
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.507765   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:deployment-controller
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.548046   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:disruption-controller
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.587919   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpoint-controller
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.628377   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.668352   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.709289   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.749512   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.790795   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] File /etc/cni/net.d/10-calico.conf is in the correct state
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/cni/net.d/10-calico.conf] at time 16:40:02.806310 duration_in_ms=636.849
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/calico/network-environment] at time 16:40:02.806644
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/calico/network-environment]
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.828726   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller
May 18 16:40:02 ctl01 kube-proxy[8698]: I0518 16:40:02.831285    8698 controller_utils.go:1034] Caches are synced for service config controller
May 18 16:40:02 ctl01 kube-proxy[8698]: I0518 16:40:02.831419    8698 proxier.go:645] Not syncing iptables until Services and Endpoints have been received from master
May 18 16:40:02 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.868189   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.908640   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
May 18 16:40:02 ctl01 kube-proxy[8698]: I0518 16:40:02.931224    8698 controller_utils.go:1034] Caches are synced for endpoints config controller
May 18 16:40:02 ctl01 kube-proxy[8698]: I0518 16:40:02.931338    8698 service.go:309] Adding new service port "default/kubernetes:https" at 10.254.0.1:443/TCP
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.948020   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replicaset-controller
May 18 16:40:02 ctl01 kube-apiserver[11092]: I0518 16:40:02.988264   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replication-controller
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.028093   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:resourcequota-controller
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.068166   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:route-controller
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.109119   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-account-controller
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.149888   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-controller
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] File /etc/calico/network-environment is in the correct state
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/calico/network-environment] at time 16:40:03.181463 duration_in_ms=374.818
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/calico/calicoctl.cfg] at time 16:40:03.181797
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/calico/calicoctl.cfg]
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.188164   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:statefulset-controller
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.228455   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:ttl-controller
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.269465   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:certificate-controller
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.308034   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.348016   11092 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pv-protection-controller
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.386869   11092 controller.go:608] quota admission added evaluator for: roles.rbac.authorization.k8s.io
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.393859   11092 storage_rbac.go:246] created role.rbac.authorization.k8s.io/extension-apiserver-authentication-reader in kube-system
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.428144   11092 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.468106   11092 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] File /etc/calico/calicoctl.cfg is in the correct state
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/calico/calicoctl.cfg] at time 16:40:03.507179 duration_in_ms=325.381
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.507984   11092 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/calico-node.service] at time 16:40:03.508475
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/calico-node.service]
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] File /etc/systemd/system/calico-node.service is in the correct state
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/calico-node.service] at time 16:40:03.543263 duration_in_ms=34.788
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Running state [/var/lib/calico] at time 16:40:03.543531
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/var/lib/calico]
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Directory /var/lib/calico is in the correct state
May 18 16:40:03 ctl01 salt-minion[5014]: Directory /var/lib/calico updated
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Completed state [/var/lib/calico] at time 16:40:03.544804 duration_in_ms=1.273
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Running state [/var/log/calico] at time 16:40:03.545028
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/var/log/calico]
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Directory /var/log/calico is in the correct state
May 18 16:40:03 ctl01 salt-minion[5014]: Directory /var/log/calico updated
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Completed state [/var/log/calico] at time 16:40:03.546226 duration_in_ms=1.198
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.548540   11092 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Running state [calico-node] at time 16:40:03.549490
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [calico-node]
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'calico-node.service', '-n', '0'] in directory '/root'
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'calico-node.service'] in directory '/root'
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.588582   11092 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] The service calico-node is already running
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Completed state [calico-node] at time 16:40:03.603079 duration_in_ms=53.587
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/proxy.kubeconfig] at time 16:40:03.603741
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/proxy.kubeconfig]
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.630614   11092 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.666777   11092 controller.go:608] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.668456   11092 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.708525   11092 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.748074   11092 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.788235   11092 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:03 ctl01 kube-controller-manager[11262]: W0518 16:40:03.813594   11262 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
May 18 16:40:03 ctl01 kube-controller-manager[11262]: W0518 16:40:03.813686   11262 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
May 18 16:40:03 ctl01 kube-controller-manager[11262]: W0518 16:40:03.813710   11262 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.813731   11262 controllermanager.go:151] Version: v1.13.5-3+98374c02d2d8c1
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.814284   11262 secure_serving.go:116] Serving securely on [::]:10257
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.814757   11262 deprecated_insecure_serving.go:51] Serving insecurely on [::]:10252
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.814959   11262 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-controller-manager...
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.827796   11092 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.863749   11262 leaderelection.go:214] successfully acquired lease kube-system/kube-controller-manager
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.868189   11092 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.869236   11262 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"962f6ddf-798b-11e9-a4a5-5254009daddf", APIVersion:"v1", ResourceVersion:"208", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' ctl01_96284d33-798b-11e9-9bb4-5254009daddf became leader
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.898370   11262 plugins.go:103] No cloud provider specified.
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.902332   11262 controllermanager.go:501] Starting "statefulset"
May 18 16:40:03 ctl01 kube-controller-manager[11262]: I0518 16:40:03.902353   11262 controller_utils.go:1027] Waiting for caches to sync for tokens controller
May 18 16:40:03 ctl01 kube-apiserver[11092]: I0518 16:40:03.910468   11092 controller.go:608] quota admission added evaluator for: serviceaccounts
May 18 16:40:03 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:03 ctl01 kube-scheduler[11188]: I0518 16:40:03.975281   11188 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.002604   11262 controller_utils.go:1034] Caches are synced for tokens controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.037269   11262 controllermanager.go:516] Started "statefulset"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.037308   11262 controllermanager.go:501] Starting "csrsigning"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.037324   11262 stateful_set.go:151] Starting stateful set controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: W0518 16:40:04.037334   11262 controllermanager.go:508] Skipping "csrsigning"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.037338   11262 controller_utils.go:1027] Waiting for caches to sync for stateful set controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.037342   11262 controllermanager.go:501] Starting "nodeipam"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: W0518 16:40:04.037349   11262 controllermanager.go:508] Skipping "nodeipam"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.037359   11262 controllermanager.go:501] Starting "persistentvolume-expander"
May 18 16:40:04 ctl01 kube-scheduler[11188]: I0518 16:40:04.075428   11188 controller_utils.go:1034] Caches are synced for scheduler controller
May 18 16:40:04 ctl01 kube-scheduler[11188]: I0518 16:40:04.075472   11188 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-scheduler...
May 18 16:40:04 ctl01 kube-scheduler[11188]: I0518 16:40:04.083166   11188 leaderelection.go:214] successfully acquired lease kube-system/kube-scheduler
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.087969   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088002   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088015   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088035   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088047   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088057   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088073   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088098   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088111   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088121   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088131   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088140   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088150   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088281   11262 controllermanager.go:516] Started "persistentvolume-expander"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088297   11262 controllermanager.go:501] Starting "endpoint"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088403   11262 expand_controller.go:153] Starting expand controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.088413   11262 controller_utils.go:1027] Waiting for caches to sync for expand controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.125186   11262 controllermanager.go:516] Started "endpoint"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.125215   11262 controllermanager.go:501] Starting "daemonset"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.125241   11262 endpoints_controller.go:160] Starting endpoint controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.125258   11262 controller_utils.go:1027] Waiting for caches to sync for endpoint controller
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.186723   11262 controllermanager.go:516] Started "daemonset"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.186755   11262 controllermanager.go:501] Starting "job"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.186893   11262 daemon_controller.go:269] Starting daemon sets controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.186905   11262 controller_utils.go:1027] Waiting for caches to sync for daemon sets controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.235861   11262 controllermanager.go:516] Started "job"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.235894   11262 controllermanager.go:501] Starting "deployment"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.236101   11262 job_controller.go:143] Starting job controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.236114   11262 controller_utils.go:1027] Waiting for caches to sync for job controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.275021   11262 controllermanager.go:516] Started "deployment"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.275043   11262 controllermanager.go:501] Starting "disruption"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.275053   11262 deployment_controller.go:152] Starting deployment controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.275068   11262 controller_utils.go:1027] Waiting for caches to sync for deployment controller
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] File /etc/kubernetes/proxy.kubeconfig is in the correct state
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/proxy.kubeconfig] at time 16:40:04.290851 duration_in_ms=687.109
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kube-proxy.service] at time 16:40:04.291196
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-proxy.service]
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] File /etc/systemd/system/kube-proxy.service is in the correct state
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.314594   11262 controllermanager.go:516] Started "disruption"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.314614   11262 controllermanager.go:501] Starting "service"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.314755   11262 disruption.go:288] Starting disruption controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.314770   11262 controller_utils.go:1027] Waiting for caches to sync for disruption controller
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kube-proxy.service] at time 16:40:04.314623 duration_in_ms=23.427
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kube-proxy] at time 16:40:04.314870
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/default/kube-proxy]
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] File /etc/default/kube-proxy is in the correct state
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kube-proxy] at time 16:40:04.316824 duration_in_ms=1.954
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Running state [kube-proxy] at time 16:40:04.320238
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing state service.running for [kube-proxy]
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kube-proxy.service', '-n', '0'] in directory '/root'
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-proxy.service'] in directory '/root'
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] The service kube-proxy is already running
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Completed state [kube-proxy] at time 16:40:04.359114 duration_in_ms=38.875
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Running state [/srv/kubernetes] at time 16:40:04.361063
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing state file.directory for [/srv/kubernetes]
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Directory /srv/kubernetes is in the correct state
May 18 16:40:04 ctl01 salt-minion[5014]: Directory /srv/kubernetes updated
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Completed state [/srv/kubernetes] at time 16:40:04.362598 duration_in_ms=1.535
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Running state [/srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:40:04.363536
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml]
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/rolebinding.yml'
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:40:04 ctl01 salt-minion[5014]: New file
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Completed state [/srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:40:04.406135 duration_in_ms=42.585
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Running state [kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:40:04.407421
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing state cmd.run for [kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml]
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Executing command 'kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml' in directory '/root'
May 18 16:40:04 ctl01 kube-controller-manager[11262]: E0518 16:40:04.423442   11262 core.go:76] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
May 18 16:40:04 ctl01 kube-controller-manager[11262]: W0518 16:40:04.423832   11262 controllermanager.go:508] Skipping "service"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.424230   11262 controllermanager.go:501] Starting "attachdetach"
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] {'pid': 11666, 'retcode': 0, 'stderr': '', 'stdout': 'clusterrolebinding.rbac.authorization.k8s.io/root-cluster-admin-binding created'}
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Completed state [kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:40:04.699673 duration_in_ms=292.251
May 18 16:40:04 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518163923178309
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820641   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820697   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820718   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820768   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820788   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820805   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820822   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820838   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820860   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820877   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820894   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/iscsi"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820911   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.820929   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/csi"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.821225   11262 controllermanager.go:516] Started "attachdetach"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.821253   11262 controllermanager.go:501] Starting "clusterrole-aggregation"
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.821379   11262 attach_detach_controller.go:315] Starting attach detach controller
May 18 16:40:04 ctl01 kube-controller-manager[11262]: I0518 16:40:04.821397   11262 controller_utils.go:1027] Waiting for caches to sync for attach detach controller
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.076967   11262 controllermanager.go:516] Started "clusterrole-aggregation"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.077030   11262 controllermanager.go:501] Starting "namespace"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.077121   11262 clusterroleaggregation_controller.go:148] Starting ClusterRoleAggregator
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.077152   11262 controller_utils.go:1027] Waiting for caches to sync for ClusterRoleAggregator controller
May 18 16:40:05 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190518164005342440
May 18 16:40:05 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 11688
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.364277   11262 controllermanager.go:516] Started "namespace"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.364304   11262 controllermanager.go:501] Starting "csrapproving"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.364372   11262 namespace_controller.go:186] Starting namespace controller
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.364400   11262 controller_utils.go:1027] Waiting for caches to sync for namespace controller
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.574701   11262 controllermanager.go:516] Started "csrapproving"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.574746   11262 controllermanager.go:501] Starting "csrcleaner"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.574820   11262 certificate_controller.go:113] Starting certificate controller
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.574838   11262 controller_utils.go:1027] Waiting for caches to sync for certificate controller
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.717258   11262 controllermanager.go:516] Started "csrcleaner"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: W0518 16:40:05.717279   11262 controllermanager.go:495] "bootstrapsigner" is disabled
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.717289   11262 controllermanager.go:501] Starting "replicationcontroller"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.717325   11262 cleaner.go:81] Starting CSR cleaner controller
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.975731   11262 controllermanager.go:516] Started "replicationcontroller"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.975756   11262 controllermanager.go:501] Starting "garbagecollector"
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.975795   11262 replica_set.go:182] Starting replicationcontroller controller
May 18 16:40:05 ctl01 kube-controller-manager[11262]: I0518 16:40:05.975806   11262 controller_utils.go:1027] Waiting for caches to sync for ReplicationController controller
May 18 16:40:06 ctl01 kube-controller-manager[11262]: W0518 16:40:06.367802   11262 garbagecollector.go:649] failed to discover preferred resources: the cache has not been filled yet
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.368278   11262 controllermanager.go:516] Started "garbagecollector"
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.368305   11262 controllermanager.go:501] Starting "ttl"
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.368521   11262 garbagecollector.go:133] Starting garbage collector controller
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.368534   11262 controller_utils.go:1027] Waiting for caches to sync for garbage collector controller
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.368549   11262 graph_builder.go:308] GraphBuilder running
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.375540   11262 garbagecollector.go:204] syncing garbage collector with updated resources from discovery (attempt 1): added: [/v1, Resource=configmaps /v1, Resource=endpoints /v1, Resource=events /v1, Resource=limitranges /v1, Resource=namespaces /v1, Resource=nodes /v1, Resource=persistentvolumeclaims /v1, Resource=persistentvolumes /v1, Resource=pods /v1, Resource=podtemplates /v1, Resource=replicationcontrollers /v1, Resource=resourcequotas /v1, Resource=secrets /v1, Resource=serviceaccounts /v1, Resource=services admissionregistration.k8s.io/v1beta1, Resource=mutatingwebhookconfigurations admissionregistration.k8s.io/v1beta1, Resource=validatingwebhookconfigurations apiextensions.k8s.io/v1beta1, Resource=customresourcedefinitions apiregistration.k8s.io/v1, Resource=apiservices apps/v1, Resource=controllerrevisions apps/v1, Resource=daemonsets apps/v1, Resource=deployments apps/v1, Resource=replicasets apps/v1, Resource=statefulsets autoscaling/v1, Resource=horizontalpodautoscalers batch/v1, Resource=jobs batch/v1beta1, Resource=cronjobs certificates.k8s.io/v1beta1, Resource=certificatesigningrequests coordination.k8s.io/v1beta1, Resource=leases events.k8s.io/v1beta1, Resource=events extensions/v1beta1, Resource=daemonsets extensions/v1beta1, Resource=deployments extensions/v1beta1, Resource=ingresses extensions/v1beta1, Resource=networkpolicies extensions/v1beta1, Resource=podsecuritypolicies extensions/v1beta1, Resource=replicasets networking.k8s.io/v1, Resource=networkpolicies policy/v1beta1, Resource=poddisruptionbudgets policy/v1beta1, Resource=podsecuritypolicies rbac.authorization.k8s.io/v1, Resource=clusterrolebindings rbac.authorization.k8s.io/v1, Resource=clusterroles rbac.authorization.k8s.io/v1, Resource=rolebindings rbac.authorization.k8s.io/v1, Resource=roles scheduling.k8s.io/v1beta1, Resource=priorityclasses storage.k8s.io/v1, Resource=storageclasses storage.k8s.io/v1, Resource=volumeattachments], removed: []
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.623120   11262 controllermanager.go:516] Started "ttl"
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.623178   11262 controllermanager.go:501] Starting "nodelifecycle"
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.623307   11262 ttl_controller.go:116] Starting TTL controller
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.623331   11262 controller_utils.go:1027] Waiting for caches to sync for TTL controller
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.871767   11262 node_lifecycle_controller.go:272] Sending events to api server.
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.871878   11262 node_lifecycle_controller.go:312] Controller is using taint based evictions.
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.871944   11262 taint_manager.go:175] Sending events to api server.
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.872023   11262 node_lifecycle_controller.go:378] Controller will taint node by condition.
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.872087   11262 controllermanager.go:516] Started "nodelifecycle"
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.872099   11262 controllermanager.go:501] Starting "cronjob"
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.872134   11262 node_lifecycle_controller.go:423] Starting node controller
May 18 16:40:06 ctl01 kube-controller-manager[11262]: I0518 16:40:06.872143   11262 controller_utils.go:1027] Waiting for caches to sync for taint controller
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.123602   11262 controllermanager.go:516] Started "cronjob"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.123649   11262 controllermanager.go:501] Starting "persistentvolume-binder"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.123739   11262 cronjob_controller.go:92] Starting CronJob Manager
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374460   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/host-path"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374500   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/nfs"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374516   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374529   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374554   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/quobyte"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374563   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374572   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/flocker"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374588   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374601   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374611   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/local-volume"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374620   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374629   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374638   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374656   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374667   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374676   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374685   11262 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374763   11262 controllermanager.go:516] Started "persistentvolume-binder"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374776   11262 controllermanager.go:501] Starting "root-ca-cert-publisher"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: W0518 16:40:07.374784   11262 controllermanager.go:508] Skipping "root-ca-cert-publisher"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374791   11262 controllermanager.go:501] Starting "resourcequota"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374858   11262 pv_controller_base.go:271] Starting persistent volume controller
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.374873   11262 controller_utils.go:1027] Waiting for caches to sync for persistent volume controller
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.631895   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for statefulsets.apps
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.632350   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.632605   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
May 18 16:40:07 ctl01 kube-controller-manager[11262]: W0518 16:40:07.632824   11262 shared_informer.go:311] resyncPeriod 57070904745119 is smaller than resyncCheckPeriod 69600562682185 and the informer has already started. Changing it to 69600562682185
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633118   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633201   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for serviceaccounts
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633272   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpoints
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633323   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633353   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633414   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.extensions
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633448   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.extensions
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633477   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.extensions
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633501   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for controllerrevisions.apps
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633532   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.extensions
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633558   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.apps
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633590   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for events.events.k8s.io
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633642   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.apps
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633672   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for limitranges
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633701   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for podtemplates
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633724   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.apps
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633756   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for jobs.batch
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633792   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for cronjobs.batch
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633820   11262 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
May 18 16:40:07 ctl01 kube-controller-manager[11262]: E0518 16:40:07.633842   11262 resource_quota_controller.go:171] initial monitor sync has error: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633855   11262 controllermanager.go:516] Started "resourcequota"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633867   11262 controllermanager.go:501] Starting "serviceaccount"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633921   11262 resource_quota_controller.go:276] Starting resource quota controller
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633932   11262 controller_utils.go:1027] Waiting for caches to sync for resource quota controller
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.633947   11262 resource_quota_monitor.go:301] QuotaMonitor running
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.872393   11262 controllermanager.go:516] Started "serviceaccount"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.872773   11262 controllermanager.go:501] Starting "horizontalpodautoscaling"
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.872425   11262 serviceaccounts_controller.go:115] Starting service account controller
May 18 16:40:07 ctl01 kube-controller-manager[11262]: I0518 16:40:07.873410   11262 controller_utils.go:1027] Waiting for caches to sync for service account controller
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.566620   11262 controllermanager.go:516] Started "horizontalpodautoscaling"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.566656   11262 controllermanager.go:501] Starting "ttl-after-finished"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: W0518 16:40:08.566667   11262 controllermanager.go:508] Skipping "ttl-after-finished"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.566674   11262 controllermanager.go:501] Starting "podgc"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.566707   11262 horizontal.go:156] Starting HPA controller
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.566717   11262 controller_utils.go:1027] Waiting for caches to sync for HPA controller
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.821259   11262 controllermanager.go:516] Started "podgc"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: W0518 16:40:08.821284   11262 controllermanager.go:495] "tokencleaner" is disabled
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.821295   11262 controllermanager.go:501] Starting "route"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.821302   11262 core.go:151] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
May 18 16:40:08 ctl01 kube-controller-manager[11262]: W0518 16:40:08.821310   11262 controllermanager.go:508] Skipping "route"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.821317   11262 controllermanager.go:501] Starting "replicaset"
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.821353   11262 gc_controller.go:76] Starting GC controller
May 18 16:40:08 ctl01 kube-controller-manager[11262]: I0518 16:40:08.821364   11262 controller_utils.go:1027] Waiting for caches to sync for GC controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.075196   11262 controllermanager.go:516] Started "replicaset"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.075239   11262 controllermanager.go:501] Starting "pvc-protection"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.075299   11262 replica_set.go:182] Starting replicaset controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.075314   11262 controller_utils.go:1027] Waiting for caches to sync for ReplicaSet controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.127203   11262 resource_quota_controller.go:427] syncing resource quota controller with updated resources from discovery: map[events.k8s.io/v1beta1, Resource=events:{} rbac.authorization.k8s.io/v1, Resource=rolebindings:{} /v1, Resource=replicationcontrollers:{} extensions/v1beta1, Resource=deployments:{} /v1, Resource=serviceaccounts:{} /v1, Resource=endpoints:{} /v1, Resource=configmaps:{} apps/v1, Resource=statefulsets:{} /v1, Resource=secrets:{} /v1, Resource=resourcequotas:{} apps/v1, Resource=replicasets:{} apps/v1, Resource=deployments:{} rbac.authorization.k8s.io/v1, Resource=roles:{} coordination.k8s.io/v1beta1, Resource=leases:{} /v1, Resource=limitranges:{} autoscaling/v1, Resource=horizontalpodautoscalers:{} extensions/v1beta1, Resource=replicasets:{} batch/v1beta1, Resource=cronjobs:{} networking.k8s.io/v1, Resource=networkpolicies:{} /v1, Resource=podtemplates:{} /v1, Resource=events:{} /v1, Resource=services:{} extensions/v1beta1, Resource=daemonsets:{} apps/v1, Resource=controllerrevisions:{} batch/v1, Resource=jobs:{} policy/v1beta1, Resource=poddisruptionbudgets:{} /v1, Resource=pods:{} extensions/v1beta1, Resource=ingresses:{} apps/v1, Resource=daemonsets:{} /v1, Resource=persistentvolumeclaims:{} extensions/v1beta1, Resource=networkpolicies:{}]
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.325474   11262 controllermanager.go:516] Started "pvc-protection"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.325502   11262 controllermanager.go:501] Starting "pv-protection"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.325517   11262 pvc_protection_controller.go:99] Starting PVC protection controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.325547   11262 controller_utils.go:1027] Waiting for caches to sync for PVC protection controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.575067   11262 controllermanager.go:516] Started "pv-protection"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.575728   11262 pv_protection_controller.go:81] Starting PV protection controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.577283   11262 controller_utils.go:1027] Waiting for caches to sync for PV protection controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: E0518 16:40:09.581354   11262 resource_quota_controller.go:437] failed to sync resource monitors: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.594750   11262 controller_utils.go:1027] Waiting for caches to sync for garbage collector controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: W0518 16:40:09.594779   11262 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="ctl01" does not exist
May 18 16:40:09 ctl01 kube-controller-manager[11262]: W0518 16:40:09.594945   11262 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="cmp002" does not exist
May 18 16:40:09 ctl01 kube-controller-manager[11262]: W0518 16:40:09.594960   11262 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="cmp001" does not exist
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.621570   11262 controller_utils.go:1034] Caches are synced for GC controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.623485   11262 controller_utils.go:1034] Caches are synced for TTL controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.625490   11262 controller_utils.go:1034] Caches are synced for endpoint controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.625731   11262 controller_utils.go:1034] Caches are synced for PVC protection controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.629454   11262 ttl_controller.go:271] Changed ttl annotation for node cmp002 to 0 seconds
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.631301   11262 ttl_controller.go:271] Changed ttl annotation for node cmp001 to 0 seconds
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.633614   11262 ttl_controller.go:271] Changed ttl annotation for node ctl01 to 0 seconds
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.636367   11262 controller_utils.go:1034] Caches are synced for job controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.637540   11262 controller_utils.go:1034] Caches are synced for stateful set controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.664626   11262 controller_utils.go:1034] Caches are synced for namespace controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.666887   11262 controller_utils.go:1034] Caches are synced for HPA controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672376   11262 controller_utils.go:1034] Caches are synced for taint controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672478   11262 node_lifecycle_controller.go:665] Controller observed a new Node: "ctl01"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672522   11262 controller_utils.go:212] Recording Registered Node ctl01 in Controller event message for node ctl01
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672564   11262 node_lifecycle_controller.go:1222] Initializing eviction metric for zone:
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672592   11262 node_lifecycle_controller.go:665] Controller observed a new Node: "cmp002"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672607   11262 controller_utils.go:212] Recording Registered Node cmp002 in Controller event message for node cmp002
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672625   11262 node_lifecycle_controller.go:665] Controller observed a new Node: "cmp001"
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672638   11262 controller_utils.go:212] Recording Registered Node cmp001 in Controller event message for node cmp001
May 18 16:40:09 ctl01 kube-controller-manager[11262]: W0518 16:40:09.672680   11262 node_lifecycle_controller.go:895] Missing timestamp for Node ctl01. Assuming now as a timestamp.
May 18 16:40:09 ctl01 kube-controller-manager[11262]: W0518 16:40:09.672724   11262 node_lifecycle_controller.go:895] Missing timestamp for Node cmp002. Assuming now as a timestamp.
May 18 16:40:09 ctl01 kube-controller-manager[11262]: W0518 16:40:09.672758   11262 node_lifecycle_controller.go:895] Missing timestamp for Node cmp001. Assuming now as a timestamp.
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.672792   11262 node_lifecycle_controller.go:1122] Controller detected that zone  is now in state Normal.
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.673304   11262 taint_manager.go:198] Starting NoExecuteTaintManager
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.673854   11262 controller_utils.go:1034] Caches are synced for service account controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.674035   11262 event.go:221] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"cmp001", UID:"979e2ca9-798b-11e9-a4a5-5254009daddf", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node cmp001 event: Registered Node cmp001 in Controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.674083   11262 event.go:221] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ctl01", UID:"9454c802-798b-11e9-a4a5-5254009daddf", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ctl01 event: Registered Node ctl01 in Controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.674109   11262 event.go:221] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"cmp002", UID:"9457e9f9-798b-11e9-a4a5-5254009daddf", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node cmp002 event: Registered Node cmp002 in Controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.675070   11262 controller_utils.go:1034] Caches are synced for certificate controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.675253   11262 controller_utils.go:1034] Caches are synced for deployment controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.676105   11262 controller_utils.go:1034] Caches are synced for ReplicationController controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.676368   11262 controller_utils.go:1034] Caches are synced for ReplicaSet controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.677378   11262 controller_utils.go:1034] Caches are synced for ClusterRoleAggregator controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.687011   11262 controller_utils.go:1034] Caches are synced for daemon sets controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.714907   11262 controller_utils.go:1034] Caches are synced for disruption controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.714923   11262 disruption.go:296] Sending events to api server.
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.775077   11262 controller_utils.go:1034] Caches are synced for persistent volume controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.777789   11262 controller_utils.go:1034] Caches are synced for PV protection controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.788707   11262 controller_utils.go:1034] Caches are synced for expand controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.821632   11262 controller_utils.go:1034] Caches are synced for attach detach controller
May 18 16:40:09 ctl01 kube-controller-manager[11262]: I0518 16:40:09.834178   11262 controller_utils.go:1034] Caches are synced for resource quota controller
May 18 16:40:10 ctl01 kube-controller-manager[11262]: I0518 16:40:10.168835   11262 controller_utils.go:1034] Caches are synced for garbage collector controller
May 18 16:40:10 ctl01 kube-controller-manager[11262]: I0518 16:40:10.168869   11262 garbagecollector.go:142] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
May 18 16:40:10 ctl01 kube-controller-manager[11262]: I0518 16:40:10.195110   11262 controller_utils.go:1034] Caches are synced for garbage collector controller
May 18 16:40:10 ctl01 kube-controller-manager[11262]: I0518 16:40:10.195168   11262 garbagecollector.go:245] synced garbage collector
May 18 16:40:10 ctl01 kubelet[8501]: I0518 16:40:10.783781    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:40:11 ctl01 salt-minion[5014]: [INFO    ] Loading fresh modules for state activity
May 18 16:40:11 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:11 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:11 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/kubeconfig.sh] at time 16:40:12.181977
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/kubeconfig.sh]
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kubeconfig.sh'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:40:12 ctl01 salt-minion[5014]: New file
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/kubeconfig.sh] at time 16:40:12.547943 duration_in_ms=365.967
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config] at time 16:40:12.549504
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing state cmd.run for [/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config]
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing command '/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config' in directory '/root'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] {'pid': 11813, 'retcode': 0, 'stderr': '', 'stdout': ''}
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config] at time 16:40:12.593710 duration_in_ms=44.205
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/addons/namespace.yaml] at time 16:40:12.594297
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/namespace.yaml]
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addon-manager/namespace.yaml'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:40:12 ctl01 salt-minion[5014]: New file
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/addons/namespace.yaml] at time 16:40:12.621473 duration_in_ms=27.175
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/kubernetes/manifests/kube-addon-manager.yml] at time 16:40:12.621828
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/etc/kubernetes/manifests/kube-addon-manager.yml]
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/manifest/kube-addon-manager.yml'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:40:12 ctl01 salt-minion[5014]: New file
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/kubernetes/manifests/kube-addon-manager.yml] at time 16:40:12.994078 duration_in_ms=372.249
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/default/kube-addon-manager] at time 16:40:12.994540
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/default/kube-addon-manager]
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] File /etc/default/kube-addon-manager is not present
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/default/kube-addon-manager] at time 16:40:12.995859 duration_in_ms=1.319
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Running state [/usr/bin/kube-addons.sh] at time 16:40:12.996193
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/usr/bin/kube-addons.sh]
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] File /usr/bin/kube-addons.sh is not present
May 18 16:40:12 ctl01 salt-minion[5014]: [INFO    ] Completed state [/usr/bin/kube-addons.sh] at time 16:40:12.997339 duration_in_ms=1.147
May 18 16:40:13 ctl01 kubelet[8501]: E0518 16:40:13.001512    8501 file.go:108] Unable to process watch event: can't process config file "/etc/kubernetes/manifests/kube-addon-manager.ymlTiCuhB": open /etc/kubernetes/manifests/kube-addon-manager.ymlTiCuhB: no such file or directory
May 18 16:40:13 ctl01 kubelet[8501]: E0518 16:40:13.001548    8501 file.go:108] Unable to process watch event: can't process config file "/etc/kubernetes/manifests/kube-addon-manager.ymlTiCuhB": open /etc/kubernetes/manifests/kube-addon-manager.ymlTiCuhB: no such file or directory
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.001666    8501 kubelet.go:1908] SyncLoop (ADD, "file"): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.004902    8501 kubelet.go:1918] SyncLoop (REMOVE, "file"): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.005664    8501 kubelet.go:2120] Failed to delete pod "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)", err: pod not found
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.005696    8501 kubelet.go:1908] SyncLoop (ADD, "file"): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.018069    8501 kubelet.go:1908] SyncLoop (ADD, "api"): "kube-addon-manager-ctl01_kube-system(9ba419ef-798b-11e9-a4a5-5254009daddf)"
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.049020    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-addons") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.049062    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "varlog" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-varlog") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.149314    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-addons") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.149376    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "varlog" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-varlog") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.149429    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "varlog" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-varlog") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.149523    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "addons" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-addons") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
May 18 16:40:13 ctl01 kubelet[8501]: I0518 16:40:13.318616    8501 kuberuntime_manager.go:397] No sandbox for pod "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)" can be found. Need to start a new one
May 18 16:40:13 ctl01 containerd[6966]: time="2019-05-18T16:40:13.319227315Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-addon-manager-ctl01,Uid:b6e9bf37167122649376fb470187a295,Namespace:kube-system,Attempt:0,}"
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Running state [kube-addon-manager] at time 16:40:13.448552
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Executing state service.dead for [kube-addon-manager]
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Executing command ['systemctl', 'status', 'kube-addon-manager.service', '-n', '0'] in directory '/root'
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] The named service kube-addon-manager is not available
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Completed state [kube-addon-manager] at time 16:40:13.474387 duration_in_ms=25.835
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Running state [/etc/systemd/system/kube-addon-manager.service] at time 16:40:13.474883
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Executing state file.absent for [/etc/systemd/system/kube-addon-manager.service]
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] File /etc/systemd/system/kube-addon-manager.service is not present
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Completed state [/etc/systemd/system/kube-addon-manager.service] at time 16:40:13.476097 duration_in_ms=1.214
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Running state [/srv/kubernetes/conformance.yml] at time 16:40:13.476392
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Executing state file.managed for [/srv/kubernetes/conformance.yml]
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/conformance/conformance.yml'
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] File changed:
May 18 16:40:13 ctl01 salt-minion[5014]: New file
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Completed state [/srv/kubernetes/conformance.yml] at time 16:40:13.826877 duration_in_ms=350.485
May 18 16:40:13 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518164005342440
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.176084305Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3,Labels:map[string]string{},}"
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.184059120Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:478b5c5586708a45bfb7aab46b7a298c4eb65c3442311e7868f49ba269f358a4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.184687542Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.230406201Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:478b5c5586708a45bfb7aab46b7a298c4eb65c3442311e7868f49ba269f358a4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.232859771Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.234762839Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64@sha256:9e2b9d9d64bab9f4d80790ec6c6fe09cdb5714d43bf23357e0ed0d0ab512fffd,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.260077051Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824/shim.sock" debug=false pid=11904
May 18 16:40:14 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command cp.push with jid 20190518164014446560
May 18 16:40:14 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 11959
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.486968790Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-addon-manager-ctl01,Uid:b6e9bf37167122649376fb470187a295,Namespace:kube-system,Attempt:0,} returns sandbox id "56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824""
May 18 16:40:14 ctl01 kubelet[8501]: I0518 16:40:14.488341    8501 provider.go:116] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 18 16:40:14 ctl01 containerd[6966]: time="2019-05-18T16:40:14.488900080Z" level=info msg="PullImage "k8s.gcr.io/kube-addon-manager:v8.9""
May 18 16:40:14 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518164014446560
May 18 16:40:15 ctl01 kubelet[8501]: I0518 16:40:15.083819    8501 kubelet.go:1953] SyncLoop (PLEG): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)", event: &pleg.PodLifecycleEvent{ID:"b6e9bf37167122649376fb470187a295", Type:"ContainerStarted", Data:"56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824"}
May 18 16:40:15 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command cp.push_dir with jid 20190518164015245335
May 18 16:40:15 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 11976
May 18 16:40:15 ctl01 salt-minion[5014]: [INFO    ] Returning information for job: 20190518164015245335
May 18 16:40:16 ctl01 containerd[6966]: time="2019-05-18T16:40:16.095918710Z" level=info msg="ImageCreate event &ImageCreate{Name:k8s.gcr.io/kube-addon-manager:v8.9,Labels:map[string]string{},}"
May 18 16:40:16 ctl01 containerd[6966]: time="2019-05-18T16:40:16.104631541Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:13da3ecfc10d338d91f3fecf494fac162c704fc284006ec289d660ec7cbba3e4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:16 ctl01 containerd[6966]: time="2019-05-18T16:40:16.106079955Z" level=info msg="ImageUpdate event &ImageUpdate{Name:k8s.gcr.io/kube-addon-manager:v8.9,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:18 ctl01 containerd[6966]: time="2019-05-18T16:40:18.941228055Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:13da3ecfc10d338d91f3fecf494fac162c704fc284006ec289d660ec7cbba3e4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:18 ctl01 containerd[6966]: time="2019-05-18T16:40:18.943578377Z" level=info msg="ImageUpdate event &ImageUpdate{Name:k8s.gcr.io/kube-addon-manager:v8.9,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:18 ctl01 containerd[6966]: time="2019-05-18T16:40:18.946191609Z" level=info msg="ImageCreate event &ImageCreate{Name:k8s.gcr.io/kube-addon-manager@sha256:2fd1daf3d3cf0e94a753f2263b60dbb0d42b107b5cde0c75ee3fc5c830e016e4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:18 ctl01 containerd[6966]: time="2019-05-18T16:40:18.946923143Z" level=info msg="PullImage "k8s.gcr.io/kube-addon-manager:v8.9" returns image reference "sha256:13da3ecfc10d338d91f3fecf494fac162c704fc284006ec289d660ec7cbba3e4""
May 18 16:40:18 ctl01 containerd[6966]: time="2019-05-18T16:40:18.950262452Z" level=info msg="CreateContainer within sandbox "56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824" for container &ContainerMetadata{Name:kube-addon-manager,Attempt:0,}"
May 18 16:40:19 ctl01 kernel: [  312.963290] audit: type=1400 audit(1558197619.043:16): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cri-containerd.apparmor.d" pid=12068 comm="apparmor_parser"
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.064358213Z" level=info msg="CreateContainer within sandbox "56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824" for &ContainerMetadata{Name:kube-addon-manager,Attempt:0,} returns container id "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a""
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.065028874Z" level=info msg="StartContainer for "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a""
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.066006497Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a/shim.sock" debug=false pid=12069
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.244693592Z" level=info msg="Finish piping stdout of container "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a""
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.244752626Z" level=info msg="Finish piping stderr of container "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a""
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.255897246Z" level=info msg="StartContainer for "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a" returns successfully"
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.258748073Z" level=info msg="CreateContainer within sandbox "56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824" for container &ContainerMetadata{Name:kube-addon-manager,Attempt:0,}"
May 18 16:40:19 ctl01 kubelet[8501]: I0518 16:40:19.259813    8501 kubelet.go:1953] SyncLoop (PLEG): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)", event: &pleg.PodLifecycleEvent{ID:"b6e9bf37167122649376fb470187a295", Type:"ContainerStarted", Data:"388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a"}
May 18 16:40:19 ctl01 containerd[6966]: time="2019-05-18T16:40:19.271817350Z" level=error msg="CreateContainer within sandbox "56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824" for &ContainerMetadata{Name:kube-addon-manager,Attempt:0,} failed" error="failed to reserve container name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0": name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0" is reserved for "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a""
May 18 16:40:19 ctl01 kubelet[8501]: E0518 16:40:19.272117    8501 remote_runtime.go:191] CreateContainer in sandbox "56c2d3eee7a4cabdab7786800287f2c5c69cc8e4a2f7349f0ededf466c4f9824" from runtime service failed: rpc error: code = Unknown desc = failed to reserve container name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0": name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0" is reserved for "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a"
May 18 16:40:19 ctl01 kubelet[8501]: E0518 16:40:19.272200    8501 kuberuntime_manager.go:749] container start failed: CreateContainerError: failed to reserve container name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0": name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0" is reserved for "388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a"
May 18 16:40:19 ctl01 kubelet[8501]: E0518 16:40:19.272288    8501 pod_workers.go:190] Error syncing pod b6e9bf37167122649376fb470187a295 ("kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"), skipping: failed to "StartContainer" for "kube-addon-manager" with CreateContainerError: "failed to reserve container name \"kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0\": name \"kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0\" is reserved for \"388c0d75469522161c21f4a30f3d638bb1ae727d91c8827b5897e6545a8e596a\""
May 18 16:40:19 ctl01 kube-apiserver[11092]: I0518 16:40:19.665757   11092 controller.go:608] quota admission added evaluator for: namespaces
May 18 16:40:20 ctl01 kube-apiserver[11092]: I0518 16:40:20.584413   11092 controller.go:608] quota admission added evaluator for: deployments.extensions
May 18 16:40:20 ctl01 kube-apiserver[11092]: I0518 16:40:20.591713   11092 controller.go:608] quota admission added evaluator for: replicasets.apps
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.593631   11262 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"calico-kube-controllers", UID:"a0271778-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"373", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set calico-kube-controllers-996f9b774 to 1
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.593783   11262 replica_set.go:477] Too few replicas for ReplicaSet kube-system/calico-kube-controllers-996f9b774, need 1, creating 1
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.602664   11262 deployment_controller.go:484] Error syncing deployment kube-system/calico-kube-controllers: Operation cannot be fulfilled on deployments.apps "calico-kube-controllers": the object has been modified; please apply your changes to the latest version and try again
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.632355   11262 replica_set.go:477] Too few replicas for ReplicaSet kube-system/coredns-7f8f94c97b, need 2, creating 2
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.632778   11262 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"a02dc5fc-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"386", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-7f8f94c97b to 2
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.637481   11262 replica_set.go:514] Slow-start failure. Skipping creation of 2 pods, decrementing expectations for ReplicaSet kube-system/coredns-7f8f94c97b
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.637544   11262 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-7f8f94c97b", UID:"a02e2e9d-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"387", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-7f8f94c97b-" is forbidden: error looking up service account kube-system/coredns: serviceaccount "coredns" not found
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.640219   11262 deployment_controller.go:484] Error syncing deployment kube-system/coredns: Operation cannot be fulfilled on deployments.apps "coredns": the object has been modified; please apply your changes to the latest version and try again
May 18 16:40:20 ctl01 kube-controller-manager[11262]: E0518 16:40:20.663984   11262 replica_set.go:450] Sync "kube-system/coredns-7f8f94c97b" failed with pods "coredns-7f8f94c97b-" is forbidden: error looking up service account kube-system/coredns: serviceaccount "coredns" not found
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.664082   11262 replica_set.go:477] Too few replicas for ReplicaSet kube-system/coredns-7f8f94c97b, need 2, creating 2
May 18 16:40:20 ctl01 kube-proxy[8698]: I0518 16:40:20.686175    8698 service.go:309] Adding new service port "kube-system/coredns:dns" at 10.254.0.10:53/UDP
May 18 16:40:20 ctl01 kube-proxy[8698]: I0518 16:40:20.686214    8698 service.go:309] Adding new service port "kube-system/coredns:dns-tcp" at 10.254.0.10:53/TCP
May 18 16:40:20 ctl01 kube-apiserver[11092]: I0518 16:40:20.699849   11092 controller.go:608] quota admission added evaluator for: daemonsets.extensions
May 18 16:40:20 ctl01 kube-apiserver[11092]: I0518 16:40:20.709055   11092 controller.go:608] quota admission added evaluator for: controllerrevisions.apps
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.715854   11262 event.go:221] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"netchecker", Name:"netchecker-agent", UID:"a0389dc2-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"402", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-agent-qmnnm
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.719358   11262 event.go:221] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"netchecker", Name:"netchecker-agent", UID:"a0389dc2-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"402", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-agent-strkg
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.719401   11262 event.go:221] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"netchecker", Name:"netchecker-agent", UID:"a0389dc2-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"402", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-agent-7gfsf
May 18 16:40:20 ctl01 kubelet[8501]: I0518 16:40:20.745735    8501 kubelet.go:1908] SyncLoop (ADD, "api"): "netchecker-agent-qmnnm_netchecker(a03adeca-798b-11e9-a4a5-5254009daddf)"
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.761553   11262 replica_set.go:477] Too few replicas for ReplicaSet netchecker/netchecker-server-7876fb46d4, need 1, creating 1
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.761788   11262 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"netchecker", Name:"netchecker-server", UID:"a0409fd3-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"419", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set netchecker-server-7876fb46d4 to 1
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.765514   11262 replica_set.go:514] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet netchecker/netchecker-server-7876fb46d4
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.765553   11262 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"netchecker", Name:"netchecker-server-7876fb46d4", UID:"a041d704-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"422", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "netchecker-server-7876fb46d4-" is forbidden: error looking up service account netchecker/netchecker: serviceaccount "netchecker" not found
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.768159   11262 deployment_controller.go:484] Error syncing deployment netchecker/netchecker-server: Operation cannot be fulfilled on deployments.apps "netchecker-server": the object has been modified; please apply your changes to the latest version and try again
May 18 16:40:20 ctl01 kube-controller-manager[11262]: E0518 16:40:20.771571   11262 replica_set.go:450] Sync "netchecker/netchecker-server-7876fb46d4" failed with pods "netchecker-server-7876fb46d4-" is forbidden: error looking up service account netchecker/netchecker: serviceaccount "netchecker" not found
May 18 16:40:20 ctl01 kube-controller-manager[11262]: I0518 16:40:20.772027   11262 replica_set.go:477] Too few replicas for ReplicaSet netchecker/netchecker-server-7876fb46d4, need 1, creating 1
May 18 16:40:20 ctl01 kube-proxy[8698]: I0518 16:40:20.784617    8698 service.go:309] Adding new service port "netchecker/netchecker:" at 10.254.26.165:80/TCP
May 18 16:40:20 ctl01 kubelet[8501]: I0518 16:40:20.796311    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:40:20 ctl01 kube-proxy[8698]: I0518 16:40:20.810593    8698 proxier.go:1427] Opened local port "nodePort for netchecker/netchecker:" (:32541/tcp)
May 18 16:40:20 ctl01 kubelet[8501]: I0518 16:40:20.878830    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-dtgft" (UniqueName: "kubernetes.io/secret/a03adeca-798b-11e9-a4a5-5254009daddf-default-token-dtgft") pod "netchecker-agent-qmnnm" (UID: "a03adeca-798b-11e9-a4a5-5254009daddf")
May 18 16:40:20 ctl01 kubelet[8501]: I0518 16:40:20.979153    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "default-token-dtgft" (UniqueName: "kubernetes.io/secret/a03adeca-798b-11e9-a4a5-5254009daddf-default-token-dtgft") pod "netchecker-agent-qmnnm" (UID: "a03adeca-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/a03adeca-798b-11e9-a4a5-5254009daddf/volumes/kubernetes.io~secret/default-token-dtgft.
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:20.991096    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "default-token-dtgft" (UniqueName: "kubernetes.io/secret/a03adeca-798b-11e9-a4a5-5254009daddf-default-token-dtgft") pod "netchecker-agent-qmnnm" (UID: "a03adeca-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.053631    8501 kuberuntime_manager.go:397] No sandbox for pod "netchecker-agent-qmnnm_netchecker(a03adeca-798b-11e9-a4a5-5254009daddf)" can be found. Need to start a new one
May 18 16:40:21 ctl01 containerd[6966]: time="2019-05-18T16:40:21.054570809Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:netchecker-agent-qmnnm,Uid:a03adeca-798b-11e9-a4a5-5254009daddf,Namespace:netchecker,Attempt:0,}"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.365 [INFO][12323] calico.go 75: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"netchecker", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"ctl01", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"netchecker-agent-qmnnm", ContainerID:"fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b"}}
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.422 [INFO][12323] k8s.go 60: Extracted identifiers for CmdAddK8s ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Namespace="netchecker" Pod="netchecker-agent-qmnnm" WorkloadEndpoint="ctl01-k8s-netchecker--agent--qmnnm-eth0"
May 18 16:40:21 ctl01 containerd[6966]: Calico CNI IPAM request count IPv4=1 IPv6=0
May 18 16:40:21 ctl01 containerd[6966]: Calico CNI IPAM handle=calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.509 [INFO][12339] calico-ipam.go 186: Auto assigning IP ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" HandleID="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Workload="ctl01-k8s-netchecker--agent--qmnnm-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0xc4206860d0), Attrs:map[string]string(nil), Hostname:"ctl01", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}}
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.509 [INFO][12339] ipam.go 70: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'ctl01'
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.510 [INFO][12339] ipam.go 254: Looking up existing affinities for host handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.512 [INFO][12339] ipam.go 265: Ran out of existing affine blocks for host handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.514 [INFO][12339] ipam.go 324: No more affine blocks, but need to allocate 1 more addresses - allocate another block handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.514 [INFO][12339] ipam.go 328: Looking for an unclaimed block handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.514 [INFO][12339] ipam_block_reader_writer.go 106: Found free block: 192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.514 [INFO][12339] ipam.go 340: Found unclaimed block host="ctl01" subnet=192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.514 [INFO][12339] ipam_block_reader_writer.go 122: Trying to create affinity in pending state host="ctl01" subnet=192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.516 [INFO][12339] ipam_block_reader_writer.go 152: Successfully created pending affinity for block host="ctl01" subnet=192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.516 [INFO][12339] ipam.go 118: Attempting to load block cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.518 [INFO][12339] ipam.go 123: The referenced block doesn't exist, trying to create it cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.519 [INFO][12339] ipam.go 130: Wrote affinity as pending cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.520 [INFO][12339] ipam.go 139: Attempting to claim the block cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.520 [INFO][12339] ipam_block_reader_writer.go 175: Attempting to create a new block host="ctl01" subnet=192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.521 [INFO][12339] ipam_block_reader_writer.go 217: Successfully created block
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.521 [INFO][12339] ipam_block_reader_writer.go 228: Confirming affinity host="ctl01" subnet=192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.522 [INFO][12339] ipam_block_reader_writer.go 243: Successfully confirmed affinity host="ctl01" subnet=192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.522 [INFO][12339] ipam.go 372: Claimed new block &{BlockKey(cidr=192.168.237.0/26) 0xc42038abd0 445 0s} - assigning 1 addresses host="ctl01" subnet=192.168.237.0/26
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.522 [INFO][12339] ipam.go 677: Attempting to assign 1 addresses from block block=192.168.237.0/26 handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.522 [INFO][12339] ipam.go 1110: Creating new handle: calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.523 [INFO][12339] ipam.go 700: Writing block in order to claim IPs block=192.168.237.0/26 handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.524 [INFO][12339] ipam.go 710: Successfully claimed IPs: [192.168.237.0] block=192.168.237.0/26 handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.524 [INFO][12339] ipam.go 456: Auto-assigned 1 out of 1 IPv4s: [192.168.237.0] handle="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" host="ctl01"
May 18 16:40:21 ctl01 containerd[6966]: Calico CNI IPAM assigned addresses IPv4=[192.168.237.0] IPv6=[]
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.524 [INFO][12339] calico-ipam.go 214: IPAM Result ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" HandleID="calico-k8s-network.fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Workload="ctl01-k8s-netchecker--agent--qmnnm-eth0" result.IPs=[]*current.IPConfig{(*current.IPConfig)(0xc420092600)}
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.526 [INFO][12323] k8s.go 365: Populated endpoint ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Namespace="netchecker" Pod="netchecker-agent-qmnnm" WorkloadEndpoint="ctl01-k8s-netchecker--agent--qmnnm-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--agent--qmnnm-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"", Pod:"netchecker-agent-qmnnm", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.0/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
May 18 16:40:21 ctl01 containerd[6966]: Calico CNI using IPs: [192.168.237.0/32]
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.527 [INFO][12323] network.go 75: Setting the host side veth name to cali6f3049528bd ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Namespace="netchecker" Pod="netchecker-agent-qmnnm" WorkloadEndpoint="ctl01-k8s-netchecker--agent--qmnnm-eth0"
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.531 [INFO][12323] network.go 380: Disabling IPv4 forwarding ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Namespace="netchecker" Pod="netchecker-agent-qmnnm" WorkloadEndpoint="ctl01-k8s-netchecker--agent--qmnnm-eth0"
May 18 16:40:21 ctl01 kernel: [  315.443498] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 18 16:40:21 ctl01 kernel: [  315.473107] IPv6: ADDRCONF(NETDEV_UP): cali6f3049528bd: link is not ready
May 18 16:40:21 ctl01 kernel: [  315.473130] IPv6: ADDRCONF(NETDEV_CHANGE): cali6f3049528bd: link becomes ready
May 18 16:40:21 ctl01 kernel: [  315.473223] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.560 [INFO][12323] k8s.go 392: Added Mac, interface name, and active container ID to endpoint ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Namespace="netchecker" Pod="netchecker-agent-qmnnm" WorkloadEndpoint="ctl01-k8s-netchecker--agent--qmnnm-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--agent--qmnnm-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b", Pod:"netchecker-agent-qmnnm", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.0/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"cali6f3049528bd", MAC:"a2:cb:67:5a:bd:40", Ports:[]v3.EndpointPort(nil)}}
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.572 [INFO][12323] k8s.go 424: Wrote updated endpoint to datastore ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Namespace="netchecker" Pod="netchecker-agent-qmnnm" WorkloadEndpoint="ctl01-k8s-netchecker--agent--qmnnm-eth0"
May 18 16:40:21 ctl01 containerd[6966]: Calico CNI creating profile: calico-k8s-network
May 18 16:40:21 ctl01 containerd[6966]: 2019-05-18 16:40:21.576 [INFO][12323] calico.go 366: Creating profile ContainerID="fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" Namespace="netchecker" Pod="netchecker-agent-qmnnm" WorkloadEndpoint="ctl01-k8s-netchecker--agent--qmnnm-" profile=&v3.Profile{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"calico-k8s-network", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.ProfileSpec{Ingress:[]v3.Rule{v3.Rule{Action:"Allow", IPVersion:(*int)(nil), Protocol:(*numorstring.Protocol)(nil), ICMP:(*v3.ICMPFields)(nil), NotProtocol:(*numorstring.Protocol)(nil), NotICMP:(*v3.ICMPFields)(nil), Source:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Ports:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, Destination:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Ports:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, HTTP:(*v3.HTTPMatch)(nil)}}, Egress:[]v3.Rule{v3.Rule{Action:"Allow", IPVersion:(*int)(nil), Protocol:(*numorstring.Protocol)(nil), ICMP:(*v3.ICMPFields)(nil), NotProtocol:(*numorstring.Protocol)(nil), NotICMP:(*v3.ICMPFields)(nil), Source:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Ports:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, Destination:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Por
May 18 16:40:21 ctl01 containerd[6966]: ts:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, HTTP:(*v3.HTTPMatch)(nil)}}, LabelsToApply:map[string]string{"calico-k8s-network":""}}}
May 18 16:40:21 ctl01 containerd[6966]: time="2019-05-18T16:40:21.591768772Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b/shim.sock" debug=false pid=12416
May 18 16:40:21 ctl01 kube-controller-manager[11262]: I0518 16:40:21.604294   11262 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"calico-kube-controllers-996f9b774", UID:"a0283410-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"374", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: calico-kube-controllers-996f9b774-srxg7
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.609438    8501 kubelet.go:1908] SyncLoop (ADD, "api"): "calico-kube-controllers-996f9b774-srxg7_kube-system(a0c25f7b-798b-11e9-a4a5-5254009daddf)"
May 18 16:40:21 ctl01 kube-controller-manager[11262]: I0518 16:40:21.673259   11262 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-7f8f94c97b", UID:"a02e2e9d-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"392", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-7f8f94c97b-j2drj
May 18 16:40:21 ctl01 kube-controller-manager[11262]: I0518 16:40:21.675825   11262 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-7f8f94c97b", UID:"a02e2e9d-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"392", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-7f8f94c97b-6ssv4
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.680811    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/a0c25f7b-798b-11e9-a4a5-5254009daddf-etcd-certs") pod "calico-kube-controllers-996f9b774-srxg7" (UID: "a0c25f7b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.680911    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-kube-controllers-token-n9mnh" (UniqueName: "kubernetes.io/secret/a0c25f7b-798b-11e9-a4a5-5254009daddf-calico-kube-controllers-token-n9mnh") pod "calico-kube-controllers-996f9b774-srxg7" (UID: "a0c25f7b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.682090    8501 kubelet.go:1908] SyncLoop (ADD, "api"): "coredns-7f8f94c97b-6ssv4_kube-system(a0cd531b-798b-11e9-a4a5-5254009daddf)"
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.781090    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/a0cd531b-798b-11e9-a4a5-5254009daddf-config-volume") pod "coredns-7f8f94c97b-6ssv4" (UID: "a0cd531b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.781139    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-7c42f" (UniqueName: "kubernetes.io/secret/a0cd531b-798b-11e9-a4a5-5254009daddf-coredns-token-7c42f") pod "coredns-7f8f94c97b-6ssv4" (UID: "a0cd531b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.781251    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/a0c25f7b-798b-11e9-a4a5-5254009daddf-etcd-certs") pod "calico-kube-controllers-996f9b774-srxg7" (UID: "a0c25f7b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.781298    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "calico-kube-controllers-token-n9mnh" (UniqueName: "kubernetes.io/secret/a0c25f7b-798b-11e9-a4a5-5254009daddf-calico-kube-controllers-token-n9mnh") pod "calico-kube-controllers-996f9b774-srxg7" (UID: "a0c25f7b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.781779    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/a0c25f7b-798b-11e9-a4a5-5254009daddf-etcd-certs") pod "calico-kube-controllers-996f9b774-srxg7" (UID: "a0c25f7b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kube-controller-manager[11262]: I0518 16:40:21.784067   11262 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"netchecker", Name:"netchecker-server-7876fb46d4", UID:"a041d704-798b-11e9-a4a5-5254009daddf", APIVersion:"apps/v1", ResourceVersion:"427", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-server-7876fb46d4-nqgxn
May 18 16:40:21 ctl01 containerd[6966]: time="2019-05-18T16:40:21.785376214Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:netchecker-agent-qmnnm,Uid:a03adeca-798b-11e9-a4a5-5254009daddf,Namespace:netchecker,Attempt:0,} returns sandbox id "fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b""
May 18 16:40:21 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/a0c25f7b-798b-11e9-a4a5-5254009daddf/volumes/kubernetes.io~secret/calico-kube-controllers-token-n9mnh.
May 18 16:40:21 ctl01 containerd[6966]: time="2019-05-18T16:40:21.790853553Z" level=info msg="PullImage "mirantis/k8s-netchecker-agent:stable""
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.789275    8501 kubelet.go:1908] SyncLoop (ADD, "api"): "netchecker-server-7876fb46d4-nqgxn_netchecker(a0dd9201-798b-11e9-a4a5-5254009daddf)"
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.791290    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "calico-kube-controllers-token-n9mnh" (UniqueName: "kubernetes.io/secret/a0c25f7b-798b-11e9-a4a5-5254009daddf-calico-kube-controllers-token-n9mnh") pod "calico-kube-controllers-996f9b774-srxg7" (UID: "a0c25f7b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: W0518 16:40:21.797447    8501 raw.go:87] Error while processing event ("/sys/fs/cgroup/memory/system.slice/run-re389e267581a435882882e9585471cfb.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-re389e267581a435882882e9585471cfb.scope: no such file or directory
May 18 16:40:21 ctl01 kubelet[8501]: W0518 16:40:21.797500    8501 raw.go:87] Error while processing event ("/sys/fs/cgroup/devices/system.slice/run-re389e267581a435882882e9585471cfb.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/system.slice/run-re389e267581a435882882e9585471cfb.scope: no such file or directory
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.881855    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/a0cd531b-798b-11e9-a4a5-5254009daddf-config-volume") pod "coredns-7f8f94c97b-6ssv4" (UID: "a0cd531b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.881962    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "coredns-token-7c42f" (UniqueName: "kubernetes.io/secret/a0cd531b-798b-11e9-a4a5-5254009daddf-coredns-token-7c42f") pod "coredns-7f8f94c97b-6ssv4" (UID: "a0cd531b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.882031    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/a0dd9201-798b-11e9-a4a5-5254009daddf-etcd-certs") pod "netchecker-server-7876fb46d4-nqgxn" (UID: "a0dd9201-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.882092    8501 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "netchecker-token-g52fc" (UniqueName: "kubernetes.io/secret/a0dd9201-798b-11e9-a4a5-5254009daddf-netchecker-token-g52fc") pod "netchecker-server-7876fb46d4-nqgxn" (UID: "a0dd9201-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.889201    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "config-volume" (UniqueName: "kubernetes.io/configmap/a0cd531b-798b-11e9-a4a5-5254009daddf-config-volume") pod "coredns-7f8f94c97b-6ssv4" (UID: "a0cd531b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/a0cd531b-798b-11e9-a4a5-5254009daddf/volumes/kubernetes.io~secret/coredns-token-7c42f.
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.900527    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "coredns-token-7c42f" (UniqueName: "kubernetes.io/secret/a0cd531b-798b-11e9-a4a5-5254009daddf-coredns-token-7c42f") pod "coredns-7f8f94c97b-6ssv4" (UID: "a0cd531b-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: W0518 16:40:21.919884    8501 container.go:409] Failed to create summary reader for "/system.slice/run-re389e267581a435882882e9585471cfb.scope": none of the resources are being tracked.
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.922327    8501 kuberuntime_manager.go:397] No sandbox for pod "calico-kube-controllers-996f9b774-srxg7_kube-system(a0c25f7b-798b-11e9-a4a5-5254009daddf)" can be found. Need to start a new one
May 18 16:40:21 ctl01 containerd[6966]: time="2019-05-18T16:40:21.922932040Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:calico-kube-controllers-996f9b774-srxg7,Uid:a0c25f7b-798b-11e9-a4a5-5254009daddf,Namespace:kube-system,Attempt:0,}"
May 18 16:40:21 ctl01 kubelet[8501]: W0518 16:40:21.924730    8501 raw.go:87] Error while processing event ("/sys/fs/cgroup/cpu,cpuacct/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/cpu,cpuacct/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope: no such file or directory
May 18 16:40:21 ctl01 kubelet[8501]: W0518 16:40:21.924818    8501 raw.go:87] Error while processing event ("/sys/fs/cgroup/blkio/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/blkio/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope: no such file or directory
May 18 16:40:21 ctl01 kubelet[8501]: W0518 16:40:21.924877    8501 raw.go:87] Error while processing event ("/sys/fs/cgroup/memory/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope: no such file or directory
May 18 16:40:21 ctl01 kubelet[8501]: W0518 16:40:21.925009    8501 raw.go:87] Error while processing event ("/sys/fs/cgroup/devices/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/system.slice/run-rcb8ceb6984274ae5abb55591e54a485b.scope: no such file or directory
May 18 16:40:21 ctl01 containerd[6966]: time="2019-05-18T16:40:21.936142680Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/1558e351e01be313da4a15654cbec9c1856fcc7120c3d293ee6b9968877d5a83/shim.sock" debug=false pid=12517
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.982428    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/a0dd9201-798b-11e9-a4a5-5254009daddf-etcd-certs") pod "netchecker-server-7876fb46d4-nqgxn" (UID: "a0dd9201-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.982499    8501 reconciler.go:252] operationExecutor.MountVolume started for volume "netchecker-token-g52fc" (UniqueName: "kubernetes.io/secret/a0dd9201-798b-11e9-a4a5-5254009daddf-netchecker-token-g52fc") pod "netchecker-server-7876fb46d4-nqgxn" (UID: "a0dd9201-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.982582    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/a0dd9201-798b-11e9-a4a5-5254009daddf-etcd-certs") pod "netchecker-server-7876fb46d4-nqgxn" (UID: "a0dd9201-798b-11e9-a4a5-5254009daddf")
May 18 16:40:21 ctl01 kubelet[8501]: I0518 16:40:21.990664    8501 kuberuntime_manager.go:397] No sandbox for pod "coredns-7f8f94c97b-6ssv4_kube-system(a0cd531b-798b-11e9-a4a5-5254009daddf)" can be found. Need to start a new one
May 18 16:40:21 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/a0dd9201-798b-11e9-a4a5-5254009daddf/volumes/kubernetes.io~secret/netchecker-token-g52fc.
May 18 16:40:21 ctl01 containerd[6966]: time="2019-05-18T16:40:21.991039846Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:coredns-7f8f94c97b-6ssv4,Uid:a0cd531b-798b-11e9-a4a5-5254009daddf,Namespace:kube-system,Attempt:0,}"
May 18 16:40:22 ctl01 kubelet[8501]: I0518 16:40:21.999769    8501 operation_generator.go:571] MountVolume.SetUp succeeded for volume "netchecker-token-g52fc" (UniqueName: "kubernetes.io/secret/a0dd9201-798b-11e9-a4a5-5254009daddf-netchecker-token-g52fc") pod "netchecker-server-7876fb46d4-nqgxn" (UID: "a0dd9201-798b-11e9-a4a5-5254009daddf")
May 18 16:40:22 ctl01 kubelet[8501]: I0518 16:40:22.101645    8501 kuberuntime_manager.go:397] No sandbox for pod "netchecker-server-7876fb46d4-nqgxn_netchecker(a0dd9201-798b-11e9-a4a5-5254009daddf)" can be found. Need to start a new one
May 18 16:40:22 ctl01 containerd[6966]: time="2019-05-18T16:40:22.102533202Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:netchecker-server-7876fb46d4-nqgxn,Uid:a0dd9201-798b-11e9-a4a5-5254009daddf,Namespace:netchecker,Attempt:0,}"
May 18 16:40:22 ctl01 containerd[6966]: time="2019-05-18T16:40:22.241361226Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-kube-controllers-996f9b774-srxg7,Uid:a0c25f7b-798b-11e9-a4a5-5254009daddf,Namespace:kube-system,Attempt:0,} returns sandbox id "1558e351e01be313da4a15654cbec9c1856fcc7120c3d293ee6b9968877d5a83""
May 18 16:40:22 ctl01 kubelet[8501]: I0518 16:40:22.265746    8501 kubelet.go:1953] SyncLoop (PLEG): "netchecker-agent-qmnnm_netchecker(a03adeca-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a03adeca-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b"}
May 18 16:40:22 ctl01 kubelet[8501]: I0518 16:40:22.267095    8501 kubelet.go:1953] SyncLoop (PLEG): "calico-kube-controllers-996f9b774-srxg7_kube-system(a0c25f7b-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a0c25f7b-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"1558e351e01be313da4a15654cbec9c1856fcc7120c3d293ee6b9968877d5a83"}
May 18 16:40:22 ctl01 kubelet[8501]: W0518 16:40:22.272371    8501 container.go:409] Failed to create summary reader for "/system.slice/run-r23a597161ee54852923a2149432d3004.scope": none of the resources are being tracked.
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.374 [INFO][12594] calico.go 75: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"kube-system", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"ctl01", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"coredns-7f8f94c97b-6ssv4", ContainerID:"48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8"}}
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.461 [INFO][12594] k8s.go 60: Extracted identifiers for CmdAddK8s ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Namespace="kube-system" Pod="coredns-7f8f94c97b-6ssv4" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0"
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI IPAM request count IPv4=1 IPv6=0
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI IPAM handle=calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.554 [INFO][12608] calico-ipam.go 186: Auto assigning IP ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" HandleID="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Workload="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0xc4202d27d0), Attrs:map[string]string(nil), Hostname:"ctl01", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}}
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.554 [INFO][12608] ipam.go 70: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'ctl01'
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.555 [INFO][12608] ipam.go 254: Looking up existing affinities for host handle="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.556 [INFO][12608] ipam.go 275: Trying affinity for 192.168.237.0/26 handle="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.557 [INFO][12608] ipam.go 118: Attempting to load block cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.557 [INFO][12608] ipam.go 195: Affinity is confirmed and block has been loaded cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.557 [INFO][12608] ipam.go 677: Attempting to assign 1 addresses from block block=192.168.237.0/26 handle="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.558 [INFO][12608] ipam.go 1110: Creating new handle: calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.559 [INFO][12608] ipam.go 700: Writing block in order to claim IPs block=192.168.237.0/26 handle="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.563 [INFO][12608] ipam.go 710: Successfully claimed IPs: [192.168.237.1] block=192.168.237.0/26 handle="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.563 [INFO][12608] ipam.go 307: Block '192.168.237.0/26' provided addresses: [192.168.237.1] handle="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.569 [INFO][12608] ipam.go 456: Auto-assigned 1 out of 1 IPv4s: [192.168.237.1] handle="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI IPAM assigned addresses IPv4=[192.168.237.1] IPv6=[]
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.569 [INFO][12608] calico-ipam.go 214: IPAM Result ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" HandleID="calico-k8s-network.48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Workload="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0" result.IPs=[]*current.IPConfig{(*current.IPConfig)(0xc4200d00c0)}
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.570 [INFO][12594] k8s.go 365: Populated endpoint ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Namespace="kube-system" Pod="coredns-7f8f94c97b-6ssv4" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"", Pod:"coredns-7f8f94c97b-6ssv4", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.1/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI using IPs: [192.168.237.1/32]
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.571 [INFO][12594] network.go 75: Setting the host side veth name to cali899f7e4a841 ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Namespace="kube-system" Pod="coredns-7f8f94c97b-6ssv4" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.572 [INFO][12594] network.go 380: Disabling IPv4 forwarding ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Namespace="kube-system" Pod="coredns-7f8f94c97b-6ssv4" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0"
May 18 16:40:22 ctl01 kernel: [  316.484525] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 18 16:40:22 ctl01 kernel: [  316.484735] IPv6: ADDRCONF(NETDEV_UP): cali899f7e4a841: link is not ready
May 18 16:40:22 ctl01 kernel: [  316.484745] IPv6: ADDRCONF(NETDEV_CHANGE): cali899f7e4a841: link becomes ready
May 18 16:40:22 ctl01 kernel: [  316.484780] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.600 [INFO][12594] k8s.go 392: Added Mac, interface name, and active container ID to endpoint ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Namespace="kube-system" Pod="coredns-7f8f94c97b-6ssv4" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8", Pod:"coredns-7f8f94c97b-6ssv4", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.1/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"cali899f7e4a841", MAC:"ba:c0:7b:27:67:c9", Ports:[]v3.EndpointPort(nil)}}
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.604 [INFO][12594] k8s.go 424: Wrote updated endpoint to datastore ContainerID="48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" Namespace="kube-system" Pod="coredns-7f8f94c97b-6ssv4" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6ssv4-eth0"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.630 [INFO][12638] calico.go 75: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"netchecker", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"ctl01", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"netchecker-server-7876fb46d4-nqgxn", ContainerID:"f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7"}}
May 18 16:40:22 ctl01 containerd[6966]: time="2019-05-18T16:40:22.649404345Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8/shim.sock" debug=false pid=12700
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.708 [INFO][12638] k8s.go 60: Extracted identifiers for CmdAddK8s ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-nqgxn" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0"
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI IPAM request count IPv4=1 IPv6=0
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI IPAM handle=calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.800 [INFO][12749] calico-ipam.go 186: Auto assigning IP ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" HandleID="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Workload="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0xc4202da130), Attrs:map[string]string(nil), Hostname:"ctl01", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}}
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.800 [INFO][12749] ipam.go 70: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'ctl01'
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.801 [INFO][12749] ipam.go 254: Looking up existing affinities for host handle="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.802 [INFO][12749] ipam.go 275: Trying affinity for 192.168.237.0/26 handle="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.803 [INFO][12749] ipam.go 118: Attempting to load block cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.803 [INFO][12749] ipam.go 195: Affinity is confirmed and block has been loaded cidr=192.168.237.0/26 host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.803 [INFO][12749] ipam.go 677: Attempting to assign 1 addresses from block block=192.168.237.0/26 handle="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.804 [INFO][12749] ipam.go 1110: Creating new handle: calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.807 [INFO][12749] ipam.go 700: Writing block in order to claim IPs block=192.168.237.0/26 handle="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.808 [INFO][12749] ipam.go 710: Successfully claimed IPs: [192.168.237.2] block=192.168.237.0/26 handle="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.808 [INFO][12749] ipam.go 307: Block '192.168.237.0/26' provided addresses: [192.168.237.2] handle="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.810 [INFO][12749] ipam.go 456: Auto-assigned 1 out of 1 IPv4s: [192.168.237.2] handle="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" host="ctl01"
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI IPAM assigned addresses IPv4=[192.168.237.2] IPv6=[]
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.810 [INFO][12749] calico-ipam.go 214: IPAM Result ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" HandleID="calico-k8s-network.f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Workload="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0" result.IPs=[]*current.IPConfig{(*current.IPConfig)(0xc42033e480)}
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.812 [INFO][12638] k8s.go 365: Populated endpoint ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-nqgxn" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"", Pod:"netchecker-server-7876fb46d4-nqgxn", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.2/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
May 18 16:40:22 ctl01 containerd[6966]: Calico CNI using IPs: [192.168.237.2/32]
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.812 [INFO][12638] network.go 75: Setting the host side veth name to calidca13f2d506 ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-nqgxn" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0"
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.813 [INFO][12638] network.go 380: Disabling IPv4 forwarding ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-nqgxn" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0"
May 18 16:40:22 ctl01 kernel: [  316.726181] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 18 16:40:22 ctl01 kernel: [  316.726413] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May 18 16:40:22 ctl01 containerd[6966]: time="2019-05-18T16:40:22.839930719Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7f8f94c97b-6ssv4,Uid:a0cd531b-798b-11e9-a4a5-5254009daddf,Namespace:kube-system,Attempt:0,} returns sandbox id "48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8""
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.847 [INFO][12638] k8s.go 392: Added Mac, interface name, and active container ID to endpoint ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-nqgxn" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7", Pod:"netchecker-server-7876fb46d4-nqgxn", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.2/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"calidca13f2d506", MAC:"0a:78:8c:87:e9:25", Ports:[]v3.EndpointPort(nil)}}
May 18 16:40:22 ctl01 containerd[6966]: 2019-05-18 16:40:22.851 [INFO][12638] k8s.go 424: Wrote updated endpoint to datastore ContainerID="f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-nqgxn" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--nqgxn-eth0"
May 18 16:40:22 ctl01 containerd[6966]: time="2019-05-18T16:40:22.877277207Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7/shim.sock" debug=false pid=12834
May 18 16:40:23 ctl01 containerd[6966]: time="2019-05-18T16:40:23.072199782Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:netchecker-server-7876fb46d4-nqgxn,Uid:a0dd9201-798b-11e9-a4a5-5254009daddf,Namespace:netchecker,Attempt:0,} returns sandbox id "f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7""
May 18 16:40:23 ctl01 kubelet[8501]: I0518 16:40:23.273116    8501 kubelet.go:1953] SyncLoop (PLEG): "coredns-7f8f94c97b-6ssv4_kube-system(a0cd531b-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a0cd531b-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8"}
May 18 16:40:23 ctl01 kubelet[8501]: I0518 16:40:23.274402    8501 kubelet.go:1953] SyncLoop (PLEG): "netchecker-server-7876fb46d4-nqgxn_netchecker(a0dd9201-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a0dd9201-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7"}
May 18 16:40:23 ctl01 containerd[6966]: time="2019-05-18T16:40:23.844093640Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-agent:stable,Labels:map[string]string{},}"
May 18 16:40:23 ctl01 containerd[6966]: time="2019-05-18T16:40:23.849481517Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:d16c8d7f4d5bdaecda098c54c655027871da52f292683759f5a32c8ecd43b3f0,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:23 ctl01 containerd[6966]: time="2019-05-18T16:40:23.850255433Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-agent:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.129830021Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:d16c8d7f4d5bdaecda098c54c655027871da52f292683759f5a32c8ecd43b3f0,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.132038313Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-agent:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.134312654Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-agent@sha256:4ac49ebef7eaeaa5a8a19f56faa73740bf4861979aa067d0125867a72846720a,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.134938122Z" level=info msg="PullImage "mirantis/k8s-netchecker-agent:stable" returns image reference "sha256:d16c8d7f4d5bdaecda098c54c655027871da52f292683759f5a32c8ecd43b3f0""
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.136200911Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2""
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.140651991Z" level=info msg="CreateContainer within sandbox "fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" for container &ContainerMetadata{Name:netchecker-agent,Attempt:0,}"
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.237052414Z" level=info msg="CreateContainer within sandbox "fba070da661618022e727feedcdf7ad3d8920dc5a2d7b184ec57bf424686f43b" for &ContainerMetadata{Name:netchecker-agent,Attempt:0,} returns container id "19d5458000809adf1ed0daa37044941ae5b7cd6b1372cb1bdaeca63cf6e2eee2""
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.237711231Z" level=info msg="StartContainer for "19d5458000809adf1ed0daa37044941ae5b7cd6b1372cb1bdaeca63cf6e2eee2""
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.238566243Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/19d5458000809adf1ed0daa37044941ae5b7cd6b1372cb1bdaeca63cf6e2eee2/shim.sock" debug=false pid=12929
May 18 16:40:24 ctl01 containerd[6966]: time="2019-05-18T16:40:24.426992378Z" level=info msg="StartContainer for "19d5458000809adf1ed0daa37044941ae5b7cd6b1372cb1bdaeca63cf6e2eee2" returns successfully"
May 18 16:40:24 ctl01 kubelet[8501]: I0518 16:40:24.429544    8501 kubelet.go:1953] SyncLoop (PLEG): "netchecker-agent-qmnnm_netchecker(a03adeca-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a03adeca-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"19d5458000809adf1ed0daa37044941ae5b7cd6b1372cb1bdaeca63cf6e2eee2"}
May 18 16:40:26 ctl01 containerd[6966]: time="2019-05-18T16:40:26.618315805Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2,Labels:map[string]string{},}"
May 18 16:40:26 ctl01 containerd[6966]: time="2019-05-18T16:40:26.625082070Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:22c16a9aecce95ea9631683dd66ee0329e399e07bb86950239d4a6db1c6d201b,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:26 ctl01 containerd[6966]: time="2019-05-18T16:40:26.625557818Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.886238775Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:22c16a9aecce95ea9631683dd66ee0329e399e07bb86950239d4a6db1c6d201b,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.888520450Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.890399412Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers@sha256:59296255e44bcb3ab00dae468a795dd669c41d6d1f16b8b5a3f2a1271e88e810,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.891070574Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2" returns image reference "sha256:22c16a9aecce95ea9631683dd66ee0329e399e07bb86950239d4a6db1c6d201b""
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.892328959Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96""
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.893451699Z" level=info msg="CreateContainer within sandbox "1558e351e01be313da4a15654cbec9c1856fcc7120c3d293ee6b9968877d5a83" for container &ContainerMetadata{Name:calico-kube-controllers,Attempt:0,}"
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.973824924Z" level=info msg="CreateContainer within sandbox "1558e351e01be313da4a15654cbec9c1856fcc7120c3d293ee6b9968877d5a83" for &ContainerMetadata{Name:calico-kube-controllers,Attempt:0,} returns container id "8422123243b4b6d796e1f11e0c68526e3935bbeb042700e796fcc97e4cc16578""
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.974727638Z" level=info msg="StartContainer for "8422123243b4b6d796e1f11e0c68526e3935bbeb042700e796fcc97e4cc16578""
May 18 16:40:27 ctl01 containerd[6966]: time="2019-05-18T16:40:27.975862372Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/8422123243b4b6d796e1f11e0c68526e3935bbeb042700e796fcc97e4cc16578/shim.sock" debug=false pid=13012
May 18 16:40:28 ctl01 containerd[6966]: time="2019-05-18T16:40:28.380942819Z" level=info msg="StartContainer for "8422123243b4b6d796e1f11e0c68526e3935bbeb042700e796fcc97e4cc16578" returns successfully"
May 18 16:40:28 ctl01 kubelet[8501]: I0518 16:40:28.438305    8501 kubelet.go:1953] SyncLoop (PLEG): "calico-kube-controllers-996f9b774-srxg7_kube-system(a0c25f7b-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a0c25f7b-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"8422123243b4b6d796e1f11e0c68526e3935bbeb042700e796fcc97e4cc16578"}
May 18 16:40:29 ctl01 kube-proxy[8698]: I0518 16:40:29.364472    8698 proxier.go:659] Stale udp service kube-system/coredns:dns -> 10.254.0.10
May 18 16:40:29 ctl01 kernel: [  323.311944] ctnetlink v0.93: registering with nfnetlink.
May 18 16:40:29 ctl01 containerd[6966]: time="2019-05-18T16:40:29.539946572Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96,Labels:map[string]string{},}"
May 18 16:40:29 ctl01 containerd[6966]: time="2019-05-18T16:40:29.551113214Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:31b943183ba2eec561c44c1fdda2a39ab03c0cb72f90a24c1bea4adf0a44bfbc,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:29 ctl01 containerd[6966]: time="2019-05-18T16:40:29.551889747Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.631535464Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:31b943183ba2eec561c44c1fdda2a39ab03c0cb72f90a24c1bea4adf0a44bfbc,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.634597559Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.637694374Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns@sha256:30f28dcd8a8c9a97c206bba187bd9b1e8fbc1cf52ce38f7e937c85a191709376,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.638913691Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96" returns image reference "sha256:31b943183ba2eec561c44c1fdda2a39ab03c0cb72f90a24c1bea4adf0a44bfbc""
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.640466196Z" level=info msg="PullImage "mirantis/k8s-netchecker-server:stable""
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.641798583Z" level=info msg="CreateContainer within sandbox "48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" for container &ContainerMetadata{Name:coredns,Attempt:0,}"
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.715237042Z" level=info msg="CreateContainer within sandbox "48c03006cc29d6d7ddf65c9b1e62b12aa975fe4f2d7719eb097c2ee7222063e8" for &ContainerMetadata{Name:coredns,Attempt:0,} returns container id "60ecae09e6e9b1b70585f13b6f8e7a84755c78b2eee1bd627a7e9fd5bec0b8d5""
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.716502354Z" level=info msg="StartContainer for "60ecae09e6e9b1b70585f13b6f8e7a84755c78b2eee1bd627a7e9fd5bec0b8d5""
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.718181450Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/60ecae09e6e9b1b70585f13b6f8e7a84755c78b2eee1bd627a7e9fd5bec0b8d5/shim.sock" debug=false pid=13132
May 18 16:40:30 ctl01 kubelet[8501]: I0518 16:40:30.808123    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:40:30 ctl01 containerd[6966]: time="2019-05-18T16:40:30.925471996Z" level=info msg="StartContainer for "60ecae09e6e9b1b70585f13b6f8e7a84755c78b2eee1bd627a7e9fd5bec0b8d5" returns successfully"
May 18 16:40:31 ctl01 kubelet[8501]: I0518 16:40:31.444986    8501 kubelet.go:1953] SyncLoop (PLEG): "coredns-7f8f94c97b-6ssv4_kube-system(a0cd531b-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a0cd531b-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"60ecae09e6e9b1b70585f13b6f8e7a84755c78b2eee1bd627a7e9fd5bec0b8d5"}
May 18 16:40:32 ctl01 containerd[6966]: time="2019-05-18T16:40:32.058668158Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-server:stable,Labels:map[string]string{},}"
May 18 16:40:32 ctl01 containerd[6966]: time="2019-05-18T16:40:32.065442489Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:22d27d4fe076561ad51f5e7a0228ab7b1035773a492c95fd4a246b40ba4f9b58,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:32 ctl01 containerd[6966]: time="2019-05-18T16:40:32.066067150Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-server:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:34 ctl01 containerd[6966]: time="2019-05-18T16:40:34.939888091Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:22d27d4fe076561ad51f5e7a0228ab7b1035773a492c95fd4a246b40ba4f9b58,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:34 ctl01 containerd[6966]: time="2019-05-18T16:40:34.943092073Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-server:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:34 ctl01 containerd[6966]: time="2019-05-18T16:40:34.945398724Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-server@sha256:87064b3cb72a8fb419eb714ae892ebd5929649e84499c08111e46e466acc8407,Labels:map[string]string{io.cri-containerd.image: managed,},}"
May 18 16:40:34 ctl01 containerd[6966]: time="2019-05-18T16:40:34.946176241Z" level=info msg="PullImage "mirantis/k8s-netchecker-server:stable" returns image reference "sha256:22d27d4fe076561ad51f5e7a0228ab7b1035773a492c95fd4a246b40ba4f9b58""
May 18 16:40:34 ctl01 containerd[6966]: time="2019-05-18T16:40:34.949845673Z" level=info msg="CreateContainer within sandbox "f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" for container &ContainerMetadata{Name:netchecker-server,Attempt:0,}"
May 18 16:40:35 ctl01 containerd[6966]: time="2019-05-18T16:40:35.029044984Z" level=info msg="CreateContainer within sandbox "f672d39b6ce8897f44c2cd253c6e206bfbfba87d123b8418faacf373430e15e7" for &ContainerMetadata{Name:netchecker-server,Attempt:0,} returns container id "6b690a7386451a58b054d0ab9df1d219379b51212e0b36280755ccff408e733e""
May 18 16:40:35 ctl01 containerd[6966]: time="2019-05-18T16:40:35.029890430Z" level=info msg="StartContainer for "6b690a7386451a58b054d0ab9df1d219379b51212e0b36280755ccff408e733e""
May 18 16:40:35 ctl01 containerd[6966]: time="2019-05-18T16:40:35.031479315Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/6b690a7386451a58b054d0ab9df1d219379b51212e0b36280755ccff408e733e/shim.sock" debug=false pid=13246
May 18 16:40:35 ctl01 containerd[6966]: time="2019-05-18T16:40:35.246816793Z" level=info msg="StartContainer for "6b690a7386451a58b054d0ab9df1d219379b51212e0b36280755ccff408e733e" returns successfully"
May 18 16:40:35 ctl01 kubelet[8501]: I0518 16:40:35.451704    8501 kubelet.go:1953] SyncLoop (PLEG): "netchecker-server-7876fb46d4-nqgxn_netchecker(a0dd9201-798b-11e9-a4a5-5254009daddf)", event: &pleg.PodLifecycleEvent{ID:"a0dd9201-798b-11e9-a4a5-5254009daddf", Type:"ContainerStarted", Data:"6b690a7386451a58b054d0ab9df1d219379b51212e0b36280755ccff408e733e"}
May 18 16:40:40 ctl01 kubelet[8501]: I0518 16:40:40.816618    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:40:50 ctl01 kubelet[8501]: I0518 16:40:50.825519    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:41:00 ctl01 kubelet[8501]: I0518 16:41:00.837044    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:41:10 ctl01 kubelet[8501]: I0518 16:41:10.846685    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:41:20 ctl01 kubelet[8501]: I0518 16:41:20.858045    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:41:30 ctl01 kubelet[8501]: I0518 16:41:30.866635    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:41:40 ctl01 kubelet[8501]: I0518 16:41:40.876203    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:41:50 ctl01 kubelet[8501]: I0518 16:41:50.887281    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:42:00 ctl01 kubelet[8501]: I0518 16:42:00.894742    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:42:10 ctl01 kubelet[8501]: I0518 16:42:10.904664    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:42:20 ctl01 kubelet[8501]: I0518 16:42:20.914093    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:42:30 ctl01 kubelet[8501]: I0518 16:42:30.922786    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:42:40 ctl01 kubelet[8501]: I0518 16:42:40.933323    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:42:44 ctl01 kube-apiserver[11092]: I0518 16:42:44.201570   11092 node_authorizer.go:176] NODE DENY: ctl01.mcp-k8s-calico-noha.local &authorizer.AttributesRecord{User:(*user.DefaultInfo)(0xc0063c6780), Verb:"create", Namespace:"default", APIGroup:"", APIVersion:"v1", Resource:"configmaps", Subresource:"", Name:"", ResourceRequest:true, Path:"/api/v1/namespaces/default/configmaps"}
May 18 16:42:44 ctl01 kube-apiserver[11092]: I0518 16:42:44.286282   11092 node_authorizer.go:176] NODE DENY: ctl01.mcp-k8s-calico-noha.local &authorizer.AttributesRecord{User:(*user.DefaultInfo)(0xc0063c6bc0), Verb:"delete", Namespace:"default", APIGroup:"", APIVersion:"v1", Resource:"configmaps", Subresource:"", Name:"k8s-b015ad23-key", ResourceRequest:true, Path:"/api/v1/namespaces/default/configmaps/k8s-b015ad23-key"}
May 18 16:42:44 ctl01 kube-apiserver[11092]: I0518 16:42:44.440853   11092 node_authorizer.go:176] NODE DENY: ctl01.mcp-k8s-calico-noha.local &authorizer.AttributesRecord{User:(*user.DefaultInfo)(0xc0063c7000), Verb:"delete", Namespace:"default", APIGroup:"", APIVersion:"v1", Resource:"configmaps", Subresource:"", Name:"k8s-b015ad23-key", ResourceRequest:true, Path:"/api/v1/namespaces/default/configmaps/k8s-b015ad23-key"}
May 18 16:42:50 ctl01 kubelet[8501]: I0518 16:42:50.944718    8501 setters.go:72] Using node IP: "172.16.10.36"
May 18 16:42:57 ctl01 salt-minion[5014]: [INFO    ] User sudo_ubuntu Executing command cp.push_dir with jid 20190518164257807017
May 18 16:42:57 ctl01 salt-minion[5014]: [INFO    ] Starting a new job with PID 14004
