Aug 28 16:07:38 ctl01 systemd-modules-load[457]: Inserted module 'iscsi_tcp'
Aug 28 16:07:38 ctl01 keyboard-setup.sh[444]: cannot open file /run/tmpkbd.LYMgjv
Aug 28 16:07:38 ctl01 systemd[1]: Started udev Coldplug all Devices.
Aug 28 16:07:38 ctl01 systemd[1]: Starting Flush Journal to Persistent Storage...
Aug 28 16:07:38 ctl01 systemd[1]: Started Load/Save Random Seed.
Aug 28 16:07:38 ctl01 systemd-modules-load[457]: Inserted module 'ib_iser'
Aug 28 16:07:38 ctl01 systemd[1]: Started Load Kernel Modules.
Aug 28 16:07:38 ctl01 systemd[1]: Mounting FUSE Control File System...
Aug 28 16:07:38 ctl01 systemd[1]: Mounting Kernel Configuration File System...
Aug 28 16:07:38 ctl01 systemd[1]: Starting Apply Kernel Variables...
Aug 28 16:07:38 ctl01 systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Aug 28 16:07:38 ctl01 systemd[1]: Mounted FUSE Control File System.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Linux version 4.15.0-58-generic (buildd@lcy01-amd64-013) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 (Ubuntu 4.15.0-58.64-generic 4.15.18)
Aug 28 16:07:38 ctl01 systemd[1]: Mounted Kernel Configuration File System.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-58-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
Aug 28 16:07:38 ctl01 kernel: [    0.000000] KERNEL supported cpus:
Aug 28 16:07:38 ctl01 systemd[1]: Started Create Static Device Nodes in /dev.
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   Intel GenuineIntel
Aug 28 16:07:38 ctl01 systemd[1]: Starting udev Kernel Device Manager...
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   AMD AuthenticAMD
Aug 28 16:07:38 ctl01 systemd[1]: Started Flush Journal to Persistent Storage.
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   Centaur CentaurHauls
Aug 28 16:07:38 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Aug 28 16:07:38 ctl01 systemd[1]: Started Apply Kernel Variables.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Aug 28 16:07:38 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Aug 28 16:07:38 ctl01 kernel: [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Aug 28 16:07:38 ctl01 systemd[1]: Started udev Kernel Device Manager.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Aug 28 16:07:38 ctl01 systemd[1]: Found device /dev/ttyS0.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] e820: BIOS-provided physical RAM map:
Aug 28 16:07:38 ctl01 systemd-udevd[519]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
Aug 28 16:07:38 ctl01 systemd-udevd[506]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000003bfffffff] usable
Aug 28 16:07:38 ctl01 systemd-udevd[508]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] NX (Execute Disable) protection: active
Aug 28 16:07:38 ctl01 kernel: [    0.000000] SMBIOS 2.8 present.
Aug 28 16:07:38 ctl01 systemd-udevd[510]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:07:38 ctl01 systemd-udevd[522]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:07:38 ctl01 systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Aug 28 16:07:38 ctl01 systemd[1]: Found device /dev/disk/by-label/UEFI.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Hypervisor detected: KVM
Aug 28 16:07:38 ctl01 systemd[1]: Started Set the console keyboard layout.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Aug 28 16:07:38 ctl01 kernel: [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Local File Systems (Pre).
Aug 28 16:07:38 ctl01 kernel: [    0.000000] e820: last_pfn = 0x3c0000 max_arch_pfn = 0x400000000
Aug 28 16:07:38 ctl01 kernel: [    0.000000] MTRR default type: write-back
Aug 28 16:07:38 ctl01 systemd[1]: Mounting /boot/efi...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] MTRR fixed ranges enabled:
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   00000-9FFFF write-back
Aug 28 16:07:38 ctl01 systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   A0000-BFFFF uncachable
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Local Encrypted Volumes.
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   C0000-FFFFF write-protect
Aug 28 16:07:38 ctl01 systemd[1]: Mounted /boot/efi.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] MTRR variable ranges enabled:
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   0 base 00C0000000 mask FFC0000000 uncachable
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   1 disabled
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   2 disabled
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   3 disabled
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Local File Systems.
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   4 disabled
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   5 disabled
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   6 disabled
Aug 28 16:07:38 ctl01 systemd[1]: Starting Set console font and keymap...
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   7 disabled
Aug 28 16:07:38 ctl01 kernel: [    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
Aug 28 16:07:38 ctl01 kernel: [    0.000000] e820: last_pfn = 0xbffdf max_arch_pfn = 0x400000000
Aug 28 16:07:38 ctl01 kernel: [    0.000000] found SMP MP-table at [mem 0x000f6560-0x000f656f] mapped at [        (ptrval)]
Aug 28 16:07:38 ctl01 systemd[1]: Starting Create Volatile Files and Directories...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Scanning 1 areas for low memory corruption
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Base memory trampoline at [        (ptrval)] 99000 size 24576
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Using GB pages for direct mapping
Aug 28 16:07:38 ctl01 systemd[1]: Starting AppArmor initialization...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BRK [0x38773d000, 0x38773dfff] PGTABLE
Aug 28 16:07:38 ctl01 systemd[1]: Starting Commit a transient machine-id on disk...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BRK [0x38773e000, 0x38773efff] PGTABLE
Aug 28 16:07:38 ctl01 systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BRK [0x38773f000, 0x38773ffff] PGTABLE
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BRK [0x387740000, 0x387740fff] PGTABLE
Aug 28 16:07:38 ctl01 systemd[1]: Starting ebtables ruleset management...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BRK [0x387741000, 0x387741fff] PGTABLE
Aug 28 16:07:38 ctl01 kernel: [    0.000000] BRK [0x387742000, 0x387742fff] PGTABLE
Aug 28 16:07:38 ctl01 systemd[1]: Started Create Volatile Files and Directories.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] RAMDISK: [mem 0x35a8b000-0x36d3cfff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: Early table checksum verification disabled
Aug 28 16:07:38 ctl01 systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: RSDP 0x00000000000F6510 000014 (v00 BOCHS )
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: RSDT 0x00000000BFFE14CC 000030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: FACP 0x00000000BFFE0854 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
Aug 28 16:07:38 ctl01 systemd[1]: Starting Network Time Synchronization...
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: DSDT 0x00000000BFFDFC80 000BD4 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: FACS 0x00000000BFFDFC40 000040
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: SSDT 0x00000000BFFE08C8 000B54 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: APIC 0x00000000BFFE141C 0000B0 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
Aug 28 16:07:38 ctl01 systemd[1]: Started Tell Plymouth To Write Out Runtime Data.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
Aug 28 16:07:38 ctl01 kernel: [    0.000000] No NUMA configuration found
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Faking a node at [mem 0x0000000000000000-0x00000003bfffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] NODE_DATA(0) allocated [mem 0x3bffd5000-0x3bfffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] kvm-clock: cpu 0, msr 3:bff54001, primary cpu clock
Aug 28 16:07:38 ctl01 kernel: [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
Aug 28 16:07:38 ctl01 kernel: [    0.000000] kvm-clock: using sched offset of 11232551023 cycles
Aug 28 16:07:38 ctl01 kernel: [    0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Zone ranges:
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   Normal   [mem 0x0000000100000000-0x00000003bfffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   Device   empty
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Movable zone start for each node
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Early memory node ranges
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000100000-0x00000000bffdefff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000100000000-0x00000003bfffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Reserved but unavailable: 98 pages
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000003bfffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] On node 0 totalpages: 3669885
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   DMA zone: 64 pages used for memmap
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   DMA zone: 21 pages reserved
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   DMA zone: 3998 pages, LIFO batch:0
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   DMA32 zone: 12224 pages used for memmap
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   DMA32 zone: 782303 pages, LIFO batch:31
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   Normal zone: 45056 pages used for memmap
Aug 28 16:07:38 ctl01 kernel: [    0.000000]   Normal zone: 2883584 pages, LIFO batch:31
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: PM-Timer IO Port: 0x608
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Aug 28 16:07:38 ctl01 kernel: [    0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: IRQ0 used by override.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: IRQ5 used by override.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: IRQ9 used by override.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: IRQ10 used by override.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ACPI: IRQ11 used by override.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Using ACPI (MADT) for SMP configuration information
Aug 28 16:07:38 ctl01 kernel: [    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xbffdf000-0xbfffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Booting paravirtualized kernel on KVM
Aug 28 16:07:38 ctl01 kernel: [    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
Aug 28 16:07:38 ctl01 kernel: [    0.000000] random: get_random_bytes called from start_kernel+0x99/0x4fd with crng_init=0
Aug 28 16:07:38 ctl01 kernel: [    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
Aug 28 16:07:38 ctl01 systemd[1]: Started Update UTMP about System Boot/Shutdown.
Aug 28 16:07:38 ctl01 apparmor[639]:  * Starting AppArmor profiles
Aug 28 16:07:38 ctl01 systemd[1]: Started ebtables ruleset management.
Aug 28 16:07:38 ctl01 apparmor[639]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Aug 28 16:07:38 ctl01 systemd[1]: Started Commit a transient machine-id on disk.
Aug 28 16:07:38 ctl01 systemd[1]: Started Network Time Synchronization.
Aug 28 16:07:38 ctl01 systemd[1]: Reached target System Time Synchronized.
Aug 28 16:07:38 ctl01 apparmor[639]:    ...done.
Aug 28 16:07:38 ctl01 systemd[1]: Started AppArmor initialization.
Aug 28 16:07:38 ctl01 systemd[1]: Starting Initial cloud-init job (pre-networking)...
Aug 28 16:07:38 ctl01 cloud-init[793]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init-local' at Wed, 28 Aug 2019 16:07:30 +0000. Up 7.58 seconds.
Aug 28 16:07:38 ctl01 systemd[1]: Started Initial cloud-init job (pre-networking).
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Network (Pre).
Aug 28 16:07:38 ctl01 systemd[1]: Starting Raise network interfaces...
Aug 28 16:07:38 ctl01 dhclient[1023]: Internet Systems Consortium DHCP Client 4.3.5
Aug 28 16:07:38 ctl01 ifup[997]: Internet Systems Consortium DHCP Client 4.3.5
Aug 28 16:07:38 ctl01 ifup[997]: Copyright 2004-2016 Internet Systems Consortium.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] percpu: Embedded 46 pages/cpu @        (ptrval) s151552 r8192 d28672 u262144
Aug 28 16:07:38 ctl01 ifup[997]: All rights reserved.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] pcpu-alloc: s151552 r8192 d28672 u262144 alloc=1*2097152
Aug 28 16:07:38 ctl01 kernel: [    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
Aug 28 16:07:38 ctl01 ifup[997]: For info, please visit https://www.isc.org/software/dhcp/
Aug 28 16:07:38 ctl01 dhclient[1023]: Copyright 2004-2016 Internet Systems Consortium.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] KVM setup async PF for cpu 0
Aug 28 16:07:38 ctl01 dhclient[1023]: All rights reserved.
Aug 28 16:07:38 ctl01 kernel: [    0.000000] kvm-stealtime: cpu 0, msr 3bfc24040
Aug 28 16:07:38 ctl01 dhclient[1023]: For info, please visit https://www.isc.org/software/dhcp/
Aug 28 16:07:38 ctl01 kernel: [    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 3612520
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Policy zone: Normal
Aug 28 16:07:38 ctl01 dhclient[1023]: 
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-58-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Calgary: detecting Calgary via BIOS EBDA area
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
Aug 28 16:07:38 ctl01 dhclient[1023]: Listening on LPF/ens3/52:54:00:3f:9b:0c
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Memory: 14334824K/14679540K available (12300K kernel code, 2478K rwdata, 4280K rodata, 2424K init, 2384K bss, 344716K reserved, 0K cma-reserved)
Aug 28 16:07:38 ctl01 kernel: [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
Aug 28 16:07:38 ctl01 ifup[997]: Listening on LPF/ens3/52:54:00:3f:9b:0c
Aug 28 16:07:38 ctl01 kernel: [    0.000000] Kernel/User page tables isolation: enabled
Aug 28 16:07:38 ctl01 kernel: [    0.000000] ftrace: allocating 39243 entries in 154 pages
Aug 28 16:07:38 ctl01 ifup[997]: Sending on   LPF/ens3/52:54:00:3f:9b:0c
Aug 28 16:07:38 ctl01 kernel: [    0.004000] Hierarchical RCU implementation.
Aug 28 16:07:38 ctl01 kernel: [    0.004000] 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=8.
Aug 28 16:07:38 ctl01 ifup[997]: Sending on   Socket/fallback
Aug 28 16:07:38 ctl01 kernel: [    0.004000] 	Tasks RCU enabled.
Aug 28 16:07:38 ctl01 kernel: [    0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
Aug 28 16:07:38 ctl01 dhclient[1023]: Sending on   LPF/ens3/52:54:00:3f:9b:0c
Aug 28 16:07:38 ctl01 ifup[997]: DHCPDISCOVER on ens3 to 255.255.255.255 port 67 interval 3 (xid=0x4fe2207e)
Aug 28 16:07:38 ctl01 kernel: [    0.004000] NR_IRQS: 524544, nr_irqs: 488, preallocated irqs: 16
Aug 28 16:07:38 ctl01 dhclient[1023]: Sending on   Socket/fallback
Aug 28 16:07:38 ctl01 kernel: [    0.004000] Console: colour VGA+ 80x25
Aug 28 16:07:38 ctl01 kernel: [    0.004000] console [tty1] enabled
Aug 28 16:07:38 ctl01 dhclient[1023]: DHCPDISCOVER on ens3 to 255.255.255.255 port 67 interval 3 (xid=0x4fe2207e)
Aug 28 16:07:38 ctl01 kernel: [    0.004000] console [ttyS0] enabled
Aug 28 16:07:38 ctl01 kernel: [    0.004000] ACPI: Core revision 20170831
Aug 28 16:07:38 ctl01 systemd[1]: Started Set console font and keymap.
Aug 28 16:07:38 ctl01 kernel: [    0.004000] ACPI: 2 ACPI AML tables successfully acquired and loaded
Aug 28 16:07:38 ctl01 kernel: [    0.004004] APIC: Switch to symmetric I/O mode setup
Aug 28 16:07:38 ctl01 dhclient[1023]: DHCPREQUEST of 192.168.11.230 on ens3 to 255.255.255.255 port 67 (xid=0x7e20e24f)
Aug 28 16:07:38 ctl01 kernel: [    0.005198] x2apic enabled
Aug 28 16:07:38 ctl01 kernel: [    0.006097] Switched APIC routing to physical x2apic.
Aug 28 16:07:38 ctl01 kernel: [    0.008173] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Aug 28 16:07:38 ctl01 ifup[997]: DHCPREQUEST of 192.168.11.230 on ens3 to 255.255.255.255 port 67 (xid=0x7e20e24f)
Aug 28 16:07:38 ctl01 kernel: [    0.009335] tsc: Detected 2799.994 MHz processor
Aug 28 16:07:38 ctl01 kernel: [    0.010219] Calibrating delay loop (skipped) preset value.. 5599.98 BogoMIPS (lpj=11199976)
Aug 28 16:07:38 ctl01 kernel: [    0.010219] pid_max: default: 32768 minimum: 301
Aug 28 16:07:38 ctl01 ifup[997]: DHCPOFFER of 192.168.11.230 from 192.168.11.3
Aug 28 16:07:38 ctl01 kernel: [    0.012026] Security Framework initialized
Aug 28 16:07:38 ctl01 kernel: [    0.012869] Yama: becoming mindful.
Aug 28 16:07:38 ctl01 ifup[997]: DHCPACK of 192.168.11.230 from 192.168.11.3
Aug 28 16:07:38 ctl01 kernel: [    0.013592] AppArmor: AppArmor initialized
Aug 28 16:07:38 ctl01 kernel: [    0.019320] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Aug 28 16:07:38 ctl01 dhclient[1023]: DHCPOFFER of 192.168.11.230 from 192.168.11.3
Aug 28 16:07:38 ctl01 kernel: [    0.022127] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Aug 28 16:07:38 ctl01 kernel: [    0.024041] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
Aug 28 16:07:38 ctl01 dhclient[1023]: DHCPACK of 192.168.11.230 from 192.168.11.3
Aug 28 16:07:38 ctl01 kernel: [    0.025730] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
Aug 28 16:07:38 ctl01 ifup[997]: Failed to try-reload-or-restart systemd-resolved.service: Unit systemd-resolved.service is masked.
Aug 28 16:07:38 ctl01 kernel: [    0.027316] mce: CPU supports 10 MCE banks
Aug 28 16:07:38 ctl01 kernel: [    0.028036] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
Aug 28 16:07:38 ctl01 kernel: [    0.029048] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
Aug 28 16:07:38 ctl01 dhclient[1023]: bound to 192.168.11.230 -- renewal in 1582 seconds.
Aug 28 16:07:38 ctl01 kernel: [    0.030143] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
Aug 28 16:07:38 ctl01 kernel: [    0.032002] Spectre V2 : Mitigation: Full generic retpoline
Aug 28 16:07:38 ctl01 kernel: [    0.033049] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
Aug 28 16:07:38 ctl01 ifup[997]: bound to 192.168.11.230 -- renewal in 1582 seconds.
Aug 28 16:07:38 ctl01 kernel: [    0.034565] Spectre V2 : Enabling Restricted Speculation for firmware calls
Aug 28 16:07:38 ctl01 kernel: [    0.036008] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
Aug 28 16:07:38 ctl01 kernel: [    0.037586] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
Aug 28 16:07:38 ctl01 systemd[1]: Started Raise network interfaces.
Aug 28 16:07:38 ctl01 kernel: [    0.039343] MDS: Mitigation: Clear CPU buffers
Aug 28 16:07:38 ctl01 kernel: [    0.044803] Freeing SMP alternatives memory: 36K
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Network.
Aug 28 16:07:38 ctl01 kernel: [    0.046729] TSC deadline timer enabled
Aug 28 16:07:38 ctl01 kernel: [    0.046731] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (family: 0x6, model: 0x3e, stepping: 0x4)
Aug 28 16:07:38 ctl01 kernel: [    0.048000] Performance Events: IvyBridge events, Intel PMU driver.
Aug 28 16:07:38 ctl01 systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
Aug 28 16:07:38 ctl01 kernel: [    0.048000] ... version:                2
Aug 28 16:07:38 ctl01 kernel: [    0.048000] ... bit width:              48
Aug 28 16:07:38 ctl01 kernel: [    0.048004] ... generic registers:      4
Aug 28 16:07:38 ctl01 kernel: [    0.048784] ... value mask:             0000ffffffffffff
Aug 28 16:07:38 ctl01 cloud-init[1119]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init' at Wed, 28 Aug 2019 16:07:33 +0000. Up 10.74 seconds.
Aug 28 16:07:38 ctl01 kernel: [    0.049787] ... max period:             000000007fffffff
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
Aug 28 16:07:38 ctl01 kernel: [    0.050794] ... fixed-purpose events:   3
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
Aug 28 16:07:38 ctl01 kernel: [    0.051582] ... event mask:             000000070000000f
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: | Device |   Up  |          Address           |      Mask     | Scope  |     Hw-Address    |
Aug 28 16:07:38 ctl01 kernel: [    0.052038] Hierarchical SRCU implementation.
Aug 28 16:07:38 ctl01 kernel: [    0.053480] smp: Bringing up secondary CPUs ...
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
Aug 28 16:07:38 ctl01 kernel: [    0.054450] x86: Booting SMP configuration:
Aug 28 16:07:38 ctl01 kernel: [    0.055274] .... node  #0, CPUs:      #1
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |  ens3  |  True |       192.168.11.230       | 255.255.255.0 | global | 52:54:00:3f:9b:0c |
Aug 28 16:07:38 ctl01 kernel: [    0.004000] kvm-clock: cpu 1, msr 3:bff54041, secondary cpu clock
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |  ens3  |  True | fe80::5054:ff:fe3f:9b0c/64 |       .       |  link  | 52:54:00:3f:9b:0c |
Aug 28 16:07:38 ctl01 kernel: [    0.060024] KVM setup async PF for cpu 1
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |  ens4  | False |             .              |       .       |   .    | 52:54:00:2c:cd:bc |
Aug 28 16:07:38 ctl01 kernel: [    0.060772] kvm-stealtime: cpu 1, msr 3bfc64040
Aug 28 16:07:38 ctl01 kernel: [    0.061642]  #2
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |  ens5  | False |             .              |       .       |   .    | 52:54:00:9e:c5:f6 |
Aug 28 16:07:38 ctl01 kernel: [    0.004000] kvm-clock: cpu 2, msr 3:bff54081, secondary cpu clock
Aug 28 16:07:38 ctl01 kernel: [    0.064020] KVM setup async PF for cpu 2
Aug 28 16:07:38 ctl01 kernel: [    0.064826] kvm-stealtime: cpu 2, msr 3bfca4040
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |  ens6  | False |             .              |       .       |   .    | 52:54:00:14:c4:1a |
Aug 28 16:07:38 ctl01 kernel: [    0.065767]  #3
Aug 28 16:07:38 ctl01 kernel: [    0.004000] kvm-clock: cpu 3, msr 3:bff540c1, secondary cpu clock
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |   lo   |  True |         127.0.0.1          |   255.0.0.0   |  host  |         .         |
Aug 28 16:07:38 ctl01 kernel: [    0.072056] KVM setup async PF for cpu 3
Aug 28 16:07:38 ctl01 kernel: [    0.074694] kvm-stealtime: cpu 3, msr 3bfce4040
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |   lo   |  True |          ::1/128           |       .       |  host  |         .         |
Aug 28 16:07:38 ctl01 kernel: [    0.076088]  #4
Aug 28 16:07:38 ctl01 kernel: [    0.004000] kvm-clock: cpu 4, msr 3:bff54101, secondary cpu clock
Aug 28 16:07:38 ctl01 kernel: [    0.080018] KVM setup async PF for cpu 4
Aug 28 16:07:38 ctl01 kernel: [    0.080824] kvm-stealtime: cpu 4, msr 3bfd24040
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +--------+-------+----------------------------+---------------+--------+-------------------+
Aug 28 16:07:38 ctl01 kernel: [    0.081728]  #5
Aug 28 16:07:38 ctl01 kernel: [    0.004000] kvm-clock: cpu 5, msr 3:bff54141, secondary cpu clock
Aug 28 16:07:38 ctl01 kernel: [    0.084017] KVM setup async PF for cpu 5
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
Aug 28 16:07:38 ctl01 kernel: [    0.084874] kvm-stealtime: cpu 5, msr 3bfd64040
Aug 28 16:07:38 ctl01 kernel: [    0.085814]  #6
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Aug 28 16:07:38 ctl01 kernel: [    0.004000] kvm-clock: cpu 6, msr 3:bff54181, secondary cpu clock
Aug 28 16:07:38 ctl01 kernel: [    0.088019] KVM setup async PF for cpu 6
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
Aug 28 16:07:38 ctl01 kernel: [    0.088868] kvm-stealtime: cpu 6, msr 3bfda4040
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Aug 28 16:07:38 ctl01 kernel: [    0.096514]  #7
Aug 28 16:07:38 ctl01 kernel: [    0.004000] kvm-clock: cpu 7, msr 3:bff541c1, secondary cpu clock
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |   0   |   0.0.0.0    | 192.168.11.3 |    0.0.0.0    |    ens3   |   UG  |
Aug 28 16:07:38 ctl01 kernel: [    0.100019] KVM setup async PF for cpu 7
Aug 28 16:07:38 ctl01 kernel: [    0.100821] kvm-stealtime: cpu 7, msr 3bfde4040
Aug 28 16:07:38 ctl01 kernel: [    0.101724] smp: Brought up 1 node, 8 CPUs
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |   1   | 192.168.11.0 |   0.0.0.0    | 255.255.255.0 |    ens3   |   U   |
Aug 28 16:07:38 ctl01 kernel: [    0.101724] smpboot: Max logical packages: 8
Aug 28 16:07:38 ctl01 kernel: [    0.101806] smpboot: Total of 8 processors activated (44799.90 BogoMIPS)
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Aug 28 16:07:38 ctl01 kernel: [    0.104871] devtmpfs: initialized
Aug 28 16:07:38 ctl01 kernel: [    0.104871] x86/mm: Memory block size: 128MB
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
Aug 28 16:07:38 ctl01 kernel: [    0.106399] evm: security.selinux
Aug 28 16:07:38 ctl01 kernel: [    0.108006] evm: security.SMACK64
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +-------+-------------+---------+-----------+-------+
Aug 28 16:07:38 ctl01 kernel: [    0.108703] evm: security.SMACK64EXEC
Aug 28 16:07:38 ctl01 kernel: [    0.109459] evm: security.SMACK64TRANSMUTE
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: | Route | Destination | Gateway | Interface | Flags |
Aug 28 16:07:38 ctl01 kernel: [    0.110283] evm: security.SMACK64MMAP
Aug 28 16:07:38 ctl01 kernel: [    0.111039] evm: security.apparmor
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +-------+-------------+---------+-----------+-------+
Aug 28 16:07:38 ctl01 kernel: [    0.111788] evm: security.ima
Aug 28 16:07:38 ctl01 kernel: [    0.112004] evm: security.capability
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |   1   |  fe80::/64  |    ::   |    ens3   |   U   |
Aug 28 16:07:38 ctl01 kernel: [    0.112817] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
Aug 28 16:07:38 ctl01 kernel: [    0.114062] futex hash table entries: 2048 (order: 5, 131072 bytes)
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |   3   |    local    |    ::   |    ens3   |   U   |
Aug 28 16:07:38 ctl01 kernel: [    0.116130] pinctrl core: initialized pinctrl subsystem
Aug 28 16:07:38 ctl01 kernel: [    0.117459] RTC time: 16:07:22, date: 08/28/19
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: |   4   |   ff00::/8  |    ::   |    ens3   |   U   |
Aug 28 16:07:38 ctl01 kernel: [    0.119175] NET: Registered protocol family 16
Aug 28 16:07:38 ctl01 kernel: [    0.120072] audit: initializing netlink subsys (disabled)
Aug 28 16:07:38 ctl01 cloud-init[1119]: ci-info: +-------+-------------+---------+-----------+-------+
Aug 28 16:07:38 ctl01 kernel: [    0.121261] audit: type=2000 audit(1567008442.730:1): state=initialized audit_enabled=0 res=1
Aug 28 16:07:38 ctl01 kernel: [    0.121707] cpuidle: using governor ladder
Aug 28 16:07:38 ctl01 kernel: [    0.121707] cpuidle: using governor menu
Aug 28 16:07:38 ctl01 kernel: [    0.124186] ACPI: bus type PCI registered
Aug 28 16:07:38 ctl01 kernel: [    0.125074] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Aug 28 16:07:38 ctl01 kernel: [    0.126447] PCI: Using configuration type 1 for base access
Aug 28 16:07:38 ctl01 kernel: [    0.127588] core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off
Aug 28 16:07:38 ctl01 kernel: [    0.129092] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
Aug 28 16:07:38 ctl01 kernel: [    0.129429] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
Aug 28 16:07:38 ctl01 kernel: [    0.132505] ACPI: Added _OSI(Module Device)
Aug 28 16:07:38 ctl01 kernel: [    0.133048] ACPI: Added _OSI(Processor Device)
Aug 28 16:07:38 ctl01 kernel: [    0.134011] ACPI: Added _OSI(3.0 _SCP Extensions)
Aug 28 16:07:38 ctl01 kernel: [    0.134992] ACPI: Added _OSI(Processor Aggregator Device)
Aug 28 16:07:38 ctl01 kernel: [    0.136009] ACPI: Added _OSI(Linux-Dell-Video)
Aug 28 16:07:38 ctl01 kernel: [    0.136912] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
Aug 28 16:07:38 ctl01 kernel: [    0.138079] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
Aug 28 16:07:38 ctl01 kernel: [    0.140489] ACPI: Interpreter enabled
Aug 28 16:07:38 ctl01 kernel: [    0.141295] ACPI: (supports S0 S5)
Aug 28 16:07:38 ctl01 kernel: [    0.142131] ACPI: Using IOAPIC for interrupt routing
Aug 28 16:07:38 ctl01 kernel: [    0.143132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Aug 28 16:07:38 ctl01 kernel: [    0.144565] ACPI: Enabled 16 GPEs in block 00 to 0F
Aug 28 16:07:38 ctl01 kernel: [    0.148927] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
Aug 28 16:07:38 ctl01 cloud-init[1119]: Generating public/private rsa key pair.
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key fingerprint is:
Aug 28 16:07:38 ctl01 cloud-init[1119]: SHA256:pfW/uSiLOWPjMDbNdktpPWsm0O/fEyRGqDwhCPAXV8M root@ctl01
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key's randomart image is:
Aug 28 16:07:38 ctl01 cloud-init[1119]: +---[RSA 2048]----+
Aug 28 16:07:38 ctl01 cloud-init[1119]: | .......oo  .    |
Aug 28 16:07:38 ctl01 cloud-init[1119]: |  .  .o. E.. .   |
Aug 28 16:07:38 ctl01 kernel: [    0.150171] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
Aug 28 16:07:38 ctl01 kernel: [    0.151515] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
Aug 28 16:07:38 ctl01 kernel: [    0.152025] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
Aug 28 16:07:38 ctl01 cloud-init[1119]: |   . .  o = .    |
Aug 28 16:07:38 ctl01 kernel: [    0.154539] acpiphp: Slot [3] registered
Aug 28 16:07:38 ctl01 kernel: [    0.155400] acpiphp: Slot [4] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: |    .    B . o . |
Aug 28 16:07:38 ctl01 kernel: [    0.156047] acpiphp: Slot [5] registered
Aug 28 16:07:38 ctl01 kernel: [    0.156956] acpiphp: Slot [6] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: |        S.. o o  |
Aug 28 16:07:38 ctl01 kernel: [    0.157861] acpiphp: Slot [7] registered
Aug 28 16:07:38 ctl01 kernel: [    0.158712] acpiphp: Slot [9] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: |       o. .o . . |
Aug 28 16:07:38 ctl01 kernel: [    0.159585] acpiphp: Slot [10] registered
Aug 28 16:07:38 ctl01 kernel: [    0.160042] acpiphp: Slot [11] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: |      = +.=.o . .|
Aug 28 16:07:38 ctl01 kernel: [    0.160912] acpiphp: Slot [12] registered
Aug 28 16:07:38 ctl01 kernel: [    0.161784] acpiphp: Slot [13] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: |     . ==*o.++ = |
Aug 28 16:07:38 ctl01 kernel: [    0.162652] acpiphp: Slot [14] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: |       o++oB+.=.o|
Aug 28 16:07:38 ctl01 kernel: [    0.163577] acpiphp: Slot [15] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: +----[SHA256]-----+
Aug 28 16:07:38 ctl01 kernel: [    0.164042] acpiphp: Slot [16] registered
Aug 28 16:07:38 ctl01 kernel: [    0.164926] acpiphp: Slot [17] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: Generating public/private dsa key pair.
Aug 28 16:07:38 ctl01 kernel: [    0.165808] acpiphp: Slot [18] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Aug 28 16:07:38 ctl01 kernel: [    0.166702] acpiphp: Slot [19] registered
Aug 28 16:07:38 ctl01 kernel: [    0.167655] acpiphp: Slot [20] registered
Aug 28 16:07:38 ctl01 kernel: [    0.168055] acpiphp: Slot [21] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
Aug 28 16:07:38 ctl01 kernel: [    0.168957] acpiphp: Slot [22] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key fingerprint is:
Aug 28 16:07:38 ctl01 kernel: [    0.169872] acpiphp: Slot [23] registered
Aug 28 16:07:38 ctl01 kernel: [    0.170826] acpiphp: Slot [24] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: SHA256:anOtrDzT6lBks2TSNEy9PaGpyMtTN70xrRFH8M8QUrM root@ctl01
Aug 28 16:07:38 ctl01 kernel: [    0.171672] acpiphp: Slot [25] registered
Aug 28 16:07:38 ctl01 kernel: [    0.172041] acpiphp: Slot [26] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key's randomart image is:
Aug 28 16:07:38 ctl01 kernel: [    0.172871] acpiphp: Slot [27] registered
Aug 28 16:07:38 ctl01 kernel: [    0.173801] acpiphp: Slot [28] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: +---[DSA 1024]----+
Aug 28 16:07:38 ctl01 kernel: [    0.174736] acpiphp: Slot [29] registered
Aug 28 16:07:38 ctl01 kernel: [    0.175918] acpiphp: Slot [30] registered
Aug 28 16:07:38 ctl01 kernel: [    0.176041] acpiphp: Slot [31] registered
Aug 28 16:07:38 ctl01 cloud-init[1119]: |     o+.  oo+    |
Aug 28 16:07:38 ctl01 kernel: [    0.176927] PCI host bridge to bus 0000:00
Aug 28 16:07:38 ctl01 kernel: [    0.177843] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
Aug 28 16:07:38 ctl01 kernel: [    0.179129] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
Aug 28 16:07:38 ctl01 cloud-init[1119]: |     o... .o.+   |
Aug 28 16:07:38 ctl01 kernel: [    0.180005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
Aug 28 16:07:38 ctl01 kernel: [    0.181619] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
Aug 28 16:07:38 ctl01 cloud-init[1119]: |    . B  = oE    |
Aug 28 16:07:38 ctl01 kernel: [    0.183047] pci_bus 0000:00: root bus resource [bus 00-ff]
Aug 28 16:07:38 ctl01 kernel: [    0.184046] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
Aug 28 16:07:38 ctl01 cloud-init[1119]: |     * o+ + .+   |
Aug 28 16:07:38 ctl01 kernel: [    0.184613] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
Aug 28 16:07:38 ctl01 kernel: [    0.185322] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
Aug 28 16:07:38 ctl01 cloud-init[1119]: |   . .o.S. =  o  |
Aug 28 16:07:38 ctl01 kernel: [    0.192006] pci 0000:00:01.1: reg 0x20: [io  0xc140-0xc14f]
Aug 28 16:07:38 ctl01 kernel: [    0.196066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
Aug 28 16:07:38 ctl01 cloud-init[1119]: |    o.o.o.= .    |
Aug 28 16:07:38 ctl01 kernel: [    0.197442] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
Aug 28 16:07:38 ctl01 kernel: [    0.198655] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
Aug 28 16:07:38 ctl01 cloud-init[1119]: |   ..o+o...*     |
Aug 28 16:07:38 ctl01 kernel: [    0.200004] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
Aug 28 16:07:38 ctl01 kernel: [    0.201477] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
Aug 28 16:07:38 ctl01 cloud-init[1119]: |    ++o+..o      |
Aug 28 16:07:38 ctl01 kernel: [    0.201979] pci 0000:00:01.3: quirk: [io  0x0600-0x063f] claimed by PIIX4 ACPI
Aug 28 16:07:38 ctl01 kernel: [    0.203416] pci 0000:00:01.3: quirk: [io  0x0700-0x070f] claimed by PIIX4 SMB
Aug 28 16:07:38 ctl01 kernel: [    0.204254] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000
Aug 28 16:07:38 ctl01 cloud-init[1119]: |     o==o        |
Aug 28 16:07:38 ctl01 kernel: [    0.207165] pci 0000:00:02.0: reg 0x10: [mem 0xfc000000-0xfdffffff pref]
Aug 28 16:07:38 ctl01 kernel: [    0.209290] pci 0000:00:02.0: reg 0x14: [mem 0xfebd0000-0xfebd0fff]
Aug 28 16:07:38 ctl01 cloud-init[1119]: +----[SHA256]-----+
Aug 28 16:07:38 ctl01 cloud-init[1119]: Generating public/private ecdsa key pair.
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key fingerprint is:
Aug 28 16:07:38 ctl01 cloud-init[1119]: SHA256:HYGlII/LYnNdg9eWOHLyVe7rBuYldfyaXxfGuEZwgzA root@ctl01
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key's randomart image is:
Aug 28 16:07:38 ctl01 cloud-init[1119]: +---[ECDSA 256]---+
Aug 28 16:07:38 ctl01 cloud-init[1119]: |    . .  Eo .    |
Aug 28 16:07:38 ctl01 cloud-init[1119]: |     + o.+o=.    |
Aug 28 16:07:38 ctl01 cloud-init[1119]: |    . = O *o.+   |
Aug 28 16:07:38 ctl01 cloud-init[1119]: |   . o B * o+ *  |
Aug 28 16:07:38 ctl01 cloud-init[1119]: |  + + . S ...+ = |
Aug 28 16:07:38 ctl01 cloud-init[1119]: | . +      + o.o o|
Aug 28 16:07:38 ctl01 cloud-init[1119]: |         o +.o oo|
Aug 28 16:07:38 ctl01 kernel: [    0.220006] pci 0000:00:02.0: reg 0x30: [mem 0xfebc0000-0xfebcffff pref]
Aug 28 16:07:38 ctl01 cloud-init[1119]: |          ..o o o|
Aug 28 16:07:38 ctl01 kernel: [    0.220256] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:07:38 ctl01 cloud-init[1119]: |           ..  ..|
Aug 28 16:07:38 ctl01 kernel: [    0.222021] pci 0000:00:03.0: reg 0x10: [io  0xc040-0xc05f]
Aug 28 16:07:38 ctl01 cloud-init[1119]: +----[SHA256]-----+
Aug 28 16:07:38 ctl01 cloud-init[1119]: Generating public/private ed25519 key pair.
Aug 28 16:07:38 ctl01 kernel: [    0.224006] pci 0000:00:03.0: reg 0x14: [mem 0xfebd1000-0xfebd1fff]
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
Aug 28 16:07:38 ctl01 kernel: [    0.233407] pci 0000:00:03.0: reg 0x30: [mem 0xfeac0000-0xfeafffff pref]
Aug 28 16:07:38 ctl01 kernel: [    0.233827] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:07:38 ctl01 cloud-init[1119]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
Aug 28 16:07:38 ctl01 kernel: [    0.236919] pci 0000:00:04.0: reg 0x10: [io  0xc060-0xc07f]
Aug 28 16:07:38 ctl01 kernel: [    0.238686] pci 0000:00:04.0: reg 0x14: [mem 0xfebd2000-0xfebd2fff]
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key fingerprint is:
Aug 28 16:07:38 ctl01 kernel: [    0.248005] pci 0000:00:04.0: reg 0x30: [mem 0xfeb00000-0xfeb3ffff pref]
Aug 28 16:07:38 ctl01 kernel: [    0.248447] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:07:38 ctl01 cloud-init[1119]: SHA256:v66WtajLBh8kpNMrx56KLa6CuaK4hYWT5QBgLEDRF9I root@ctl01
Aug 28 16:07:38 ctl01 kernel: [    0.250243] pci 0000:00:05.0: reg 0x10: [io  0xc080-0xc09f]
Aug 28 16:07:38 ctl01 kernel: [    0.251994] pci 0000:00:05.0: reg 0x14: [mem 0xfebd3000-0xfebd3fff]
Aug 28 16:07:38 ctl01 kernel: [    0.260833] pci 0000:00:05.0: reg 0x30: [mem 0xfeb40000-0xfeb7ffff pref]
Aug 28 16:07:38 ctl01 cloud-init[1119]: The key's randomart image is:
Aug 28 16:07:38 ctl01 kernel: [    0.261216] pci 0000:00:06.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:07:38 ctl01 kernel: [    0.263060] pci 0000:00:06.0: reg 0x10: [io  0xc0a0-0xc0bf]
Aug 28 16:07:38 ctl01 kernel: [    0.264005] pci 0000:00:06.0: reg 0x14: [mem 0xfebd4000-0xfebd4fff]
Aug 28 16:07:38 ctl01 cloud-init[1119]: +--[ED25519 256]--+
Aug 28 16:07:38 ctl01 kernel: [    0.272904] pci 0000:00:06.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref]
Aug 28 16:07:38 ctl01 kernel: [    0.273310] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000
Aug 28 16:07:38 ctl01 cloud-init[1119]: |O+o....          |
Aug 28 16:07:38 ctl01 kernel: [    0.275328] pci 0000:00:07.0: reg 0x10: [io  0xc000-0xc03f]
Aug 28 16:07:38 ctl01 kernel: [    0.277024] pci 0000:00:07.0: reg 0x14: [mem 0xfebd5000-0xfebd5fff]
Aug 28 16:07:38 ctl01 cloud-init[1119]: |+. .oE           |
Aug 28 16:07:38 ctl01 kernel: [    0.287557] pci 0000:00:08.0: [8086:2934] type 00 class 0x0c0300
Aug 28 16:07:38 ctl01 cloud-init[1119]: |o .+.            |
Aug 28 16:07:38 ctl01 kernel: [    0.292874] pci 0000:00:08.0: reg 0x20: [io  0xc0c0-0xc0df]
Aug 28 16:07:38 ctl01 kernel: [    0.294929] pci 0000:00:08.1: [8086:2935] type 00 class 0x0c0300
Aug 28 16:07:38 ctl01 kernel: [    0.299188] pci 0000:00:08.1: reg 0x20: [io  0xc0e0-0xc0ff]
Aug 28 16:07:38 ctl01 cloud-init[1119]: | *o o .          |
Aug 28 16:07:38 ctl01 kernel: [    0.302516] pci 0000:00:08.2: [8086:2936] type 00 class 0x0c0300
Aug 28 16:07:38 ctl01 kernel: [    0.307179] pci 0000:00:08.2: reg 0x20: [io  0xc100-0xc11f]
Aug 28 16:07:38 ctl01 cloud-init[1119]: |+ oo +  S        |
Aug 28 16:07:38 ctl01 kernel: [    0.309134] pci 0000:00:08.7: [8086:293a] type 00 class 0x0c0320
Aug 28 16:07:38 ctl01 cloud-init[1119]: | +. = .  ..      |
Aug 28 16:07:38 ctl01 kernel: [    0.310103] pci 0000:00:08.7: reg 0x10: [mem 0xfebd6000-0xfebd6fff]
Aug 28 16:07:38 ctl01 kernel: [    0.316220] pci 0000:00:09.0: [1af4:1002] type 00 class 0x00ff00
Aug 28 16:07:38 ctl01 cloud-init[1119]: |o..+ + . +..     |
Aug 28 16:07:38 ctl01 kernel: [    0.317217] pci 0000:00:09.0: reg 0x10: [io  0xc120-0xc13f]
Aug 28 16:07:38 ctl01 kernel: [    0.324284] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
Aug 28 16:07:38 ctl01 kernel: [    0.325606] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
Aug 28 16:07:38 ctl01 cloud-init[1119]: |B+  o.o + ..     |
Aug 28 16:07:38 ctl01 kernel: [    0.326864] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
Aug 28 16:07:38 ctl01 kernel: [    0.328054] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
Aug 28 16:07:38 ctl01 cloud-init[1119]: |&=o. .++.oo      |
Aug 28 16:07:38 ctl01 kernel: [    0.329217] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
Aug 28 16:07:38 ctl01 kernel: [    0.331225] SCSI subsystem initialized
Aug 28 16:07:38 ctl01 cloud-init[1119]: +----[SHA256]-----+
Aug 28 16:07:38 ctl01 kernel: [    0.332092] libata version 3.00 loaded.
Aug 28 16:07:38 ctl01 systemd[1]: Started Initial cloud-init job (metadata service crawler).
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Network is Online.
Aug 28 16:07:38 ctl01 kernel: [    0.332092] pci 0000:00:02.0: vgaarb: setting as boot VGA device
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Remote File Systems (Pre).
Aug 28 16:07:38 ctl01 kernel: [    0.333258] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
Aug 28 16:07:38 ctl01 kernel: [    0.334966] pci 0000:00:02.0: vgaarb: bridge control possible
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Remote File Systems.
Aug 28 16:07:38 ctl01 kernel: [    0.336004] vgaarb: loaded
Aug 28 16:07:38 ctl01 kernel: [    0.336609] ACPI: bus type USB registered
Aug 28 16:07:38 ctl01 kernel: [    0.337507] usbcore: registered new interface driver usbfs
Aug 28 16:07:38 ctl01 systemd[1]: Starting Availability of block devices...
Aug 28 16:07:38 ctl01 kernel: [    0.338680] usbcore: registered new interface driver hub
Aug 28 16:07:38 ctl01 kernel: [    0.339764] usbcore: registered new device driver usb
Aug 28 16:07:38 ctl01 systemd[1]: Reached target System Initialization.
Aug 28 16:07:38 ctl01 kernel: [    0.340098] EDAC MC: Ver: 3.0.0
Aug 28 16:07:38 ctl01 kernel: [    0.341099] PCI: Using ACPI for IRQ routing
Aug 28 16:07:38 ctl01 systemd[1]: Listening on Open-iSCSI iscsid Socket.
Aug 28 16:07:38 ctl01 kernel: [    0.341099] PCI: pci_cache_line_size set to 64 bytes
Aug 28 16:07:38 ctl01 kernel: [    0.341160] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
Aug 28 16:07:38 ctl01 systemd[1]: Starting Socket activation for snappy daemon.
Aug 28 16:07:38 ctl01 kernel: [    0.341162] e820: reserve RAM buffer [mem 0xbffdf000-0xbfffffff]
Aug 28 16:07:38 ctl01 kernel: [    0.341247] NetLabel: Initializing
Aug 28 16:07:38 ctl01 systemd[1]: Started ACPI Events Check.
Aug 28 16:07:38 ctl01 kernel: [    0.341980] NetLabel:  domain hash size = 128
Aug 28 16:07:38 ctl01 kernel: [    0.344004] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Paths.
Aug 28 16:07:38 ctl01 kernel: [    0.345162] NetLabel:  unlabeled traffic allowed by default
Aug 28 16:07:38 ctl01 kernel: [    0.346256] clocksource: Switched to clocksource kvm-clock
Aug 28 16:07:38 ctl01 kernel: [    0.355062] VFS: Disk quotas dquot_6.6.0
Aug 28 16:07:38 ctl01 systemd[1]: Listening on ACPID Listen Socket.
Aug 28 16:07:38 ctl01 kernel: [    0.355923] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Aug 28 16:07:38 ctl01 kernel: [    0.357309] AppArmor: AppArmor Filesystem Enabled
Aug 28 16:07:38 ctl01 systemd[1]: Started Daily apt download activities.
Aug 28 16:07:38 ctl01 systemd[1]: Started Message of the Day.
Aug 28 16:07:38 ctl01 kernel: [    0.358257] pnp: PnP ACPI init
Aug 28 16:07:38 ctl01 systemd[1]: Listening on D-Bus System Message Bus Socket.
Aug 28 16:07:38 ctl01 systemd[1]: Listening on UUID daemon activation socket.
Aug 28 16:07:38 ctl01 kernel: [    0.358949] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
Aug 28 16:07:38 ctl01 systemd[1]: Started Daily apt upgrade and clean activities.
Aug 28 16:07:38 ctl01 kernel: [    0.358989] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
Aug 28 16:07:38 ctl01 systemd[1]: Started Daily Cleanup of Temporary Directories.
Aug 28 16:07:38 ctl01 kernel: [    0.359016] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active)
Aug 28 16:07:38 ctl01 kernel: [    0.359045] pnp 00:03: [dma 2]
Aug 28 16:07:38 ctl01 systemd[1]: Started Discard unused blocks once a week.
Aug 28 16:07:38 ctl01 kernel: [    0.359060] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active)
Aug 28 16:07:38 ctl01 kernel: [    0.359141] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
Aug 28 16:07:38 ctl01 systemd[1]: Starting LXD - unix socket.
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Timers.
Aug 28 16:07:38 ctl01 kernel: [    0.359375] pnp: PnP ACPI: found 5 devices
Aug 28 16:07:38 ctl01 kernel: [    0.367397] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Cloud-config availability.
Aug 28 16:07:38 ctl01 kernel: [    0.369115] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
Aug 28 16:07:38 ctl01 systemd[1]: Started Availability of block devices.
Aug 28 16:07:38 ctl01 kernel: [    0.369116] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
Aug 28 16:07:38 ctl01 kernel: [    0.369117] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
Aug 28 16:07:38 ctl01 kernel: [    0.369118] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
Aug 28 16:07:38 ctl01 systemd[1]: Listening on Socket activation for snappy daemon.
Aug 28 16:07:38 ctl01 kernel: [    0.369174] NET: Registered protocol family 2
Aug 28 16:07:38 ctl01 kernel: [    0.370722] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
Aug 28 16:07:38 ctl01 systemd[1]: Listening on LXD - unix socket.
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Sockets.
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Basic System.
Aug 28 16:07:38 ctl01 systemd[1]: Starting Pollinate to seed the pseudo random number generator...
Aug 28 16:07:38 ctl01 systemd[1]: Starting The Salt Minion...
Aug 28 16:07:38 ctl01 systemd[1]: Started D-Bus System Message Bus.
Aug 28 16:07:38 ctl01 dbus-daemon[1242]: [system] AppArmor D-Bus mediation is enabled
Aug 28 16:07:38 ctl01 systemd[1]: Starting Accounts Service...
Aug 28 16:07:38 ctl01 pollinate[1237]: client sent challenge to [https://entropy.ubuntu.com/]
Aug 28 16:07:38 ctl01 systemd[1]: Starting LSB: Record successful boot for GRUB...
Aug 28 16:07:38 ctl01 systemd[1]: Starting Permit User Sessions...
Aug 28 16:07:38 ctl01 systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
Aug 28 16:07:38 ctl01 systemd[1]: Started FUSE filesystem for LXC.
Aug 28 16:07:38 ctl01 systemd[1]: Started Deferred execution scheduler.
Aug 28 16:07:38 ctl01 systemd[1]: Starting LSB: automatic crash report generation...
Aug 28 16:07:38 ctl01 systemd[1]: Starting LXD - container startup/shutdown...
Aug 28 16:07:38 ctl01 systemd[1]: Started irqbalance daemon.
Aug 28 16:07:38 ctl01 systemd[1]: Starting Login Service...
Aug 28 16:07:38 ctl01 systemd[1]: Started Regular background program processing daemon.
Aug 28 16:07:38 ctl01 systemd[1]: Starting Snappy daemon...
Aug 28 16:07:38 ctl01 kernel: [    0.372319] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Aug 28 16:07:38 ctl01 systemd[1]: Starting System Logging Service...
Aug 28 16:07:38 ctl01 kernel: [    0.374077] TCP: Hash tables configured (established 131072 bind 65536)
Aug 28 16:07:38 ctl01 kernel: [    0.375356] UDP hash table entries: 8192 (order: 6, 262144 bytes)
Aug 28 16:07:38 ctl01 systemd[1]: Started Permit User Sessions.
Aug 28 16:07:38 ctl01 kernel: [    0.376545] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
Aug 28 16:07:38 ctl01 kernel: [    0.377848] NET: Registered protocol family 1
Aug 28 16:07:38 ctl01 kernel: [    0.378732] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
Aug 28 16:07:38 ctl01 kernel: [    0.379864] pci 0000:00:01.0: PIIX3: Enabling Passive Release
Aug 28 16:07:38 ctl01 cron[1326]: (CRON) INFO (pidfile fd = 3)
Aug 28 16:07:38 ctl01 kernel: [    0.381016] pci 0000:00:01.0: Activating ISA DMA hang workarounds
Aug 28 16:07:38 ctl01 kernel: [    0.382228] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
Aug 28 16:07:38 ctl01 systemd[1]: Starting Terminate Plymouth Boot Screen...
Aug 28 16:07:38 ctl01 kernel: [    0.409124] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
Aug 28 16:07:38 ctl01 kernel: [    0.460295] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
Aug 28 16:07:38 ctl01 grub-common[1278]:  * Recording successful boot for GRUB
Aug 28 16:07:38 ctl01 kernel: [    0.512946] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
Aug 28 16:07:38 ctl01 kernel: [    0.563747] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
Aug 28 16:07:38 ctl01 lxcfs[1291]: mount namespace: 5
Aug 28 16:07:38 ctl01 kernel: [    0.589590] PCI: CLS 0 bytes, default 64
Aug 28 16:07:38 ctl01 kernel: [    0.589630] Unpacking initramfs...
Aug 28 16:07:38 ctl01 lxcfs[1291]: hierarchies:
Aug 28 16:07:38 ctl01 lxcfs[1291]:   0: fd:   6: perf_event
Aug 28 16:07:38 ctl01 kernel: [    0.824646] Freeing initrd memory: 19144K
Aug 28 16:07:38 ctl01 lxcfs[1291]:   1: fd:   7: blkio
Aug 28 16:07:38 ctl01 kernel: [    0.825551] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Aug 28 16:07:38 ctl01 kernel: [    0.826764] software IO TLB: mapped [mem 0xbbfdf000-0xbffdf000] (64MB)
Aug 28 16:07:38 ctl01 lxcfs[1291]:   2: fd:   8: rdma
Aug 28 16:07:38 ctl01 kernel: [    0.828030] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x285c3aeaff3, max_idle_ns: 440795255742 ns
Aug 28 16:07:38 ctl01 kernel: [    0.829933] Scanning for low memory corruption every 60 seconds
Aug 28 16:07:38 ctl01 lxcfs[1291]:   3: fd:   9: devices
Aug 28 16:07:38 ctl01 kernel: [    0.831727] Initialise system trusted keyrings
Aug 28 16:07:38 ctl01 lxcfs[1291]:   4: fd:  10: freezer
Aug 28 16:07:38 ctl01 kernel: [    0.832779] Key type blacklist registered
Aug 28 16:07:38 ctl01 kernel: [    0.833794] workingset: timestamp_bits=36 max_order=22 bucket_order=0
Aug 28 16:07:38 ctl01 lxcfs[1291]:   5: fd:  11: net_cls,net_prio
Aug 28 16:07:38 ctl01 lxcfs[1291]:   6: fd:  12: pids
Aug 28 16:07:38 ctl01 kernel: [    0.835856] zbud: loaded
Aug 28 16:07:38 ctl01 kernel: [    0.836972] squashfs: version 4.0 (2009/01/31) Phillip Lougher
Aug 28 16:07:38 ctl01 lxcfs[1291]:   7: fd:  13: cpuset
Aug 28 16:07:38 ctl01 kernel: [    0.838302] fuse init (API version 7.26)
Aug 28 16:07:38 ctl01 kernel: [    0.841132] Key type asymmetric registered
Aug 28 16:07:38 ctl01 kernel: [    0.842019] Asymmetric key parser 'x509' registered
Aug 28 16:07:38 ctl01 lxcfs[1291]:   8: fd:  14: hugetlb
Aug 28 16:07:38 ctl01 lxcfs[1291]:   9: fd:  15: cpu,cpuacct
Aug 28 16:07:38 ctl01 kernel: [    0.842997] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
Aug 28 16:07:38 ctl01 lxcfs[1291]:  10: fd:  16: memory
Aug 28 16:07:38 ctl01 kernel: [    0.844607] io scheduler noop registered
Aug 28 16:07:38 ctl01 lxcfs[1291]:  11: fd:  17: name=systemd
Aug 28 16:07:38 ctl01 kernel: [    0.845473] io scheduler deadline registered
Aug 28 16:07:38 ctl01 lxcfs[1291]:  12: fd:  18: unified
Aug 28 16:07:38 ctl01 kernel: [    0.846349] io scheduler cfq registered (default)
Aug 28 16:07:38 ctl01 kernel: [    0.847683] intel_idle: Please enable MWAIT in BIOS SETUP
Aug 28 16:07:38 ctl01 systemd[1]: Starting Hold until boot process finishes up...
Aug 28 16:07:38 ctl01 kernel: [    0.847752] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Aug 28 16:07:38 ctl01 kernel: [    0.849385] ACPI: Power Button [PWRF]
Aug 28 16:07:38 ctl01 apport[1298]:  * Starting automatic crash report generation: apport
Aug 28 16:07:38 ctl01 kernel: [    0.875866] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
Aug 28 16:07:38 ctl01 kernel: [    0.903084] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
Aug 28 16:07:38 ctl01 systemd[1]: Started Hold until boot process finishes up.
Aug 28 16:07:38 ctl01 kernel: [    0.930843] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
Aug 28 16:07:38 ctl01 kernel: [    0.958109] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
Aug 28 16:07:38 ctl01 kernel: [    0.985471] virtio-pci 0000:00:07.0: virtio_pci: leaving for legacy driver
Aug 28 16:07:38 ctl01 systemd[1]: Started Terminate Plymouth Boot Screen.
Aug 28 16:07:38 ctl01 kernel: [    1.011591] virtio-pci 0000:00:09.0: virtio_pci: leaving for legacy driver
Aug 28 16:07:38 ctl01 kernel: [    1.013919] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
Aug 28 16:07:38 ctl01 kernel: [    1.038549] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
Aug 28 16:07:38 ctl01 systemd[1]: Starting Set console scheme...
Aug 28 16:07:38 ctl01 kernel: [    1.042365] Linux agpgart interface v0.103
Aug 28 16:07:38 ctl01 systemd[1]: Started Serial Getty on ttyS0.
Aug 28 16:07:38 ctl01 apport[1298]:    ...done.
Aug 28 16:07:38 ctl01 kernel: [    1.045980] loop: module loaded
Aug 28 16:07:38 ctl01 kernel: [    1.046738] ata_piix 0000:00:01.1: version 2.13
Aug 28 16:07:38 ctl01 kernel: [    1.047861] scsi host0: ata_piix
Aug 28 16:07:38 ctl01 kernel: [    1.048764] scsi host1: ata_piix
Aug 28 16:07:38 ctl01 kernel: [    1.049521] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc140 irq 14
Aug 28 16:07:38 ctl01 kernel: [    1.050759] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc148 irq 15
Aug 28 16:07:38 ctl01 kernel: [    1.052106] libphy: Fixed MDIO Bus: probed
Aug 28 16:07:38 ctl01 kernel: [    1.053099] tun: Universal TUN/TAP device driver, 1.6
Aug 28 16:07:38 ctl01 kernel: [    1.054163] PPP generic driver version 2.4.2
Aug 28 16:07:38 ctl01 kernel: [    1.062536] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Aug 28 16:07:38 ctl01 kernel: [    1.063816] ehci-pci: EHCI PCI platform driver
Aug 28 16:07:38 ctl01 kernel: [    1.091225] ehci-pci 0000:00:08.7: EHCI Host Controller
Aug 28 16:07:38 ctl01 kernel: [    1.092297] ehci-pci 0000:00:08.7: new USB bus registered, assigned bus number 1
Aug 28 16:07:38 ctl01 systemd[1]: Started LSB: automatic crash report generation.
Aug 28 16:07:38 ctl01 kernel: [    1.093971] ehci-pci 0000:00:08.7: irq 11, io mem 0xfebd6000
Aug 28 16:07:38 ctl01 kernel: [    1.108033] ehci-pci 0000:00:08.7: USB 2.0 started, EHCI 1.00
Aug 28 16:07:38 ctl01 kernel: [    1.109211] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Aug 28 16:07:38 ctl01 kernel: [    1.110488] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:07:38 ctl01 kernel: [    1.111924] usb usb1: Product: EHCI Host Controller
Aug 28 16:07:38 ctl01 kernel: [    1.112923] usb usb1: Manufacturer: Linux 4.15.0-58-generic ehci_hcd
Aug 28 16:07:38 ctl01 kernel: [    1.114138] usb usb1: SerialNumber: 0000:00:08.7
Aug 28 16:07:38 ctl01 kernel: [    1.115156] hub 1-0:1.0: USB hub found
Aug 28 16:07:38 ctl01 systemd[1]: Started Login Service.
Aug 28 16:07:38 ctl01 kernel: [    1.115937] hub 1-0:1.0: 6 ports detected
Aug 28 16:07:38 ctl01 kernel: [    1.116966] ehci-platform: EHCI generic platform driver
Aug 28 16:07:38 ctl01 systemd[1]: Started Unattended Upgrades Shutdown.
Aug 28 16:07:38 ctl01 kernel: [    1.118002] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Aug 28 16:07:38 ctl01 kernel: [    1.119171] ohci-pci: OHCI PCI platform driver
Aug 28 16:07:38 ctl01 cron[1326]: (CRON) INFO (Running @reboot jobs)
Aug 28 16:07:38 ctl01 kernel: [    1.120071] ohci-platform: OHCI generic platform driver
Aug 28 16:07:38 ctl01 kernel: [    1.121137] uhci_hcd: USB Universal Host Controller Interface driver
Aug 28 16:07:38 ctl01 dnsmasq[1286]: dnsmasq: syntax check OK.
Aug 28 16:07:38 ctl01 kernel: [    1.148222] uhci_hcd 0000:00:08.0: UHCI Host Controller
Aug 28 16:07:38 ctl01 kernel: [    1.149296] uhci_hcd 0000:00:08.0: new USB bus registered, assigned bus number 2
Aug 28 16:07:38 ctl01 systemd[1]: Started Set console scheme.
Aug 28 16:07:38 ctl01 systemd[1]: Created slice system-getty.slice.
Aug 28 16:07:38 ctl01 kernel: [    1.150732] uhci_hcd 0000:00:08.0: detected 2 ports
Aug 28 16:07:38 ctl01 systemd[1]: Started Getty on tty1.
Aug 28 16:07:38 ctl01 kernel: [    1.151743] uhci_hcd 0000:00:08.0: irq 11, io base 0x0000c0c0
Aug 28 16:07:38 ctl01 kernel: [    1.152951] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Login Prompts.
Aug 28 16:07:38 ctl01 kernel: [    1.154294] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:07:38 ctl01 kernel: [    1.156104] usb usb2: Product: UHCI Host Controller
Aug 28 16:07:38 ctl01 kernel: [    1.157105] usb usb2: Manufacturer: Linux 4.15.0-58-generic uhci_hcd
Aug 28 16:07:38 ctl01 grub-common[1278]:    ...done.
Aug 28 16:07:38 ctl01 kernel: [    1.158510] usb usb2: SerialNumber: 0000:00:08.0
Aug 28 16:07:38 ctl01 kernel: [    1.159531] hub 2-0:1.0: USB hub found
Aug 28 16:07:38 ctl01 kernel: [    1.160329] hub 2-0:1.0: 2 ports detected
Aug 28 16:07:38 ctl01 systemd[1]: Started LSB: Record successful boot for GRUB.
Aug 28 16:07:38 ctl01 kernel: [    1.187016] uhci_hcd 0000:00:08.1: UHCI Host Controller
Aug 28 16:07:38 ctl01 dbus-daemon[1242]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkit.service' requested by ':1.2' (uid=0 pid=1276 comm="/usr/lib/accountsservice/accounts-daemon " label="unconfined")
Aug 28 16:07:38 ctl01 kernel: [    1.188087] uhci_hcd 0000:00:08.1: new USB bus registered, assigned bus number 3
Aug 28 16:07:38 ctl01 kernel: [    1.189581] uhci_hcd 0000:00:08.1: detected 2 ports
Aug 28 16:07:38 ctl01 kernel: [    1.190627] uhci_hcd 0000:00:08.1: irq 10, io base 0x0000c0e0
Aug 28 16:07:38 ctl01 systemd[1]: Starting Authorization Manager...
Aug 28 16:07:38 ctl01 dnsmasq[1481]: started, version 2.79 cachesize 150
Aug 28 16:07:38 ctl01 dnsmasq[1481]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
Aug 28 16:07:38 ctl01 dnsmasq[1481]: reading /etc/resolv.conf
Aug 28 16:07:38 ctl01 dnsmasq[1481]: using nameserver 8.8.8.8#53
Aug 28 16:07:38 ctl01 dnsmasq[1481]: read /etc/hosts - 7 addresses
Aug 28 16:07:38 ctl01 systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.
Aug 28 16:07:38 ctl01 rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.32.0]
Aug 28 16:07:38 ctl01 rsyslogd: rsyslogd's groupid changed to 106
Aug 28 16:07:38 ctl01 rsyslogd: rsyslogd's userid changed to 102
Aug 28 16:07:38 ctl01 rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="1348" x-info="http://www.rsyslog.com"] start
Aug 28 16:07:38 ctl01 systemd[1]: Started System Logging Service.
Aug 28 16:07:38 ctl01 kernel: [    1.191814] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
Aug 28 16:07:38 ctl01 kernel: [    1.193224] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:07:38 ctl01 kernel: [    1.194706] usb usb3: Product: UHCI Host Controller
Aug 28 16:07:38 ctl01 kernel: [    1.195733] usb usb3: Manufacturer: Linux 4.15.0-58-generic uhci_hcd
Aug 28 16:07:38 ctl01 kernel: [    1.197163] usb usb3: SerialNumber: 0000:00:08.1
Aug 28 16:07:38 ctl01 kernel: [    1.198243] hub 3-0:1.0: USB hub found
Aug 28 16:07:38 ctl01 kernel: [    1.199071] hub 3-0:1.0: 2 ports detected
Aug 28 16:07:38 ctl01 kernel: [    1.212750] ata1.01: NODEV after polling detection
Aug 28 16:07:38 ctl01 kernel: [    1.213060] ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
Aug 28 16:07:38 ctl01 kernel: [    1.214698] ata1.00: configured for MWDMA2
Aug 28 16:07:38 ctl01 kernel: [    1.216218] scsi 0:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
Aug 28 16:07:38 ctl01 kernel: [    1.218803] sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
Aug 28 16:07:38 ctl01 kernel: [    1.220216] cdrom: Uniform CD-ROM driver Revision: 3.20
Aug 28 16:07:38 ctl01 kernel: [    1.221948] sr 0:0:0:0: Attached scsi CD-ROM sr0
Aug 28 16:07:38 ctl01 kernel: [    1.222003] sr 0:0:0:0: Attached scsi generic sg0 type 5
Aug 28 16:07:38 ctl01 kernel: [    1.233465] uhci_hcd 0000:00:08.2: UHCI Host Controller
Aug 28 16:07:38 ctl01 kernel: [    1.234623] uhci_hcd 0000:00:08.2: new USB bus registered, assigned bus number 4
Aug 28 16:07:38 ctl01 kernel: [    1.236222] uhci_hcd 0000:00:08.2: detected 2 ports
Aug 28 16:07:38 ctl01 kernel: [    1.237293] uhci_hcd 0000:00:08.2: irq 10, io base 0x0000c100
Aug 28 16:07:38 ctl01 kernel: [    1.238525] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
Aug 28 16:07:38 ctl01 kernel: [    1.239891] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:07:38 ctl01 kernel: [    1.241335] usb usb4: Product: UHCI Host Controller
Aug 28 16:07:38 ctl01 kernel: [    1.242348] usb usb4: Manufacturer: Linux 4.15.0-58-generic uhci_hcd
Aug 28 16:07:38 ctl01 kernel: [    1.243646] usb usb4: SerialNumber: 0000:00:08.2
Aug 28 16:07:38 ctl01 kernel: [    1.244716] hub 4-0:1.0: USB hub found
Aug 28 16:07:38 ctl01 kernel: [    1.245524] hub 4-0:1.0: 2 ports detected
Aug 28 16:07:38 ctl01 kernel: [    1.246520] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
Aug 28 16:07:38 ctl01 kernel: [    1.249102] serio: i8042 KBD port at 0x60,0x64 irq 1
Aug 28 16:07:38 ctl01 kernel: [    1.250144] serio: i8042 AUX port at 0x60,0x64 irq 12
Aug 28 16:07:38 ctl01 kernel: [    1.251295] mousedev: PS/2 mouse device common for all mice
Aug 28 16:07:38 ctl01 kernel: [    1.252691] rtc_cmos 00:00: RTC can wake from S4
Aug 28 16:07:38 ctl01 kernel: [    1.254201] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
Aug 28 16:07:38 ctl01 kernel: [    1.255604] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
Aug 28 16:07:38 ctl01 kernel: [    1.256101] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
Aug 28 16:07:38 ctl01 kernel: [    1.259477] i2c /dev entries driver
Aug 28 16:07:38 ctl01 kernel: [    1.260373] device-mapper: uevent: version 1.0.3
Aug 28 16:07:38 ctl01 kernel: [    1.261595] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com
Aug 28 16:07:38 ctl01 kernel: [    1.263825] ledtrig-cpu: registered to indicate activity on CPUs
Aug 28 16:07:38 ctl01 kernel: [    1.265898] NET: Registered protocol family 10
Aug 28 16:07:38 ctl01 kernel: [    1.272336] Segment Routing with IPv6
Aug 28 16:07:38 ctl01 kernel: [    1.273292] NET: Registered protocol family 17
Aug 28 16:07:38 ctl01 kernel: [    1.274502] Key type dns_resolver registered
Aug 28 16:07:38 ctl01 kernel: [    1.276275] RAS: Correctable Errors collector initialized.
Aug 28 16:07:38 ctl01 kernel: [    1.277583] sched_clock: Marking stable (1276220030, 0)->(1763178965, -486958935)
Aug 28 16:07:38 ctl01 kernel: [    1.280163] registered taskstats version 1
Aug 28 16:07:38 ctl01 kernel: [    1.281342] Loading compiled-in X.509 certificates
Aug 28 16:07:38 ctl01 kernel: [    1.285085] Loaded X.509 cert 'Build time autogenerated kernel key: e343dd3e31c9820cf0adf0156bc2eecdcfc394af'
Aug 28 16:07:38 ctl01 kernel: [    1.286968] zswap: loaded using pool lzo/zbud
Aug 28 16:07:38 ctl01 kernel: [    1.292180] Key type big_key registered
Aug 28 16:07:38 ctl01 kernel: [    1.293109] Key type trusted registered
Aug 28 16:07:38 ctl01 kernel: [    1.296073] Key type encrypted registered
Aug 28 16:07:38 ctl01 kernel: [    1.297093] AppArmor: AppArmor sha1 policy hashing enabled
Aug 28 16:07:38 ctl01 kernel: [    1.298383] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
Aug 28 16:07:38 ctl01 kernel: [    1.299837] ima: Allocated hash algorithm: sha1
Aug 28 16:07:38 ctl01 kernel: [    1.300974] evm: HMAC attrs: 0x1
Aug 28 16:07:38 ctl01 kernel: [    1.302267]   Magic number: 15:877:136
Aug 28 16:07:38 ctl01 kernel: [    1.303218] usb usb4-port1: hash matches
Aug 28 16:07:38 ctl01 kernel: [    1.304502] rtc_cmos 00:00: setting system clock to 2019-08-28 16:07:24 UTC (1567008444)
Aug 28 16:07:38 ctl01 kernel: [    1.306329] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
Aug 28 16:07:38 ctl01 kernel: [    1.307479] EDD information not available.
Aug 28 16:07:38 ctl01 kernel: [    1.310755] Freeing unused kernel image memory: 2424K
Aug 28 16:07:38 ctl01 kernel: [    1.336182] Write protecting the kernel read-only data: 20480k
Aug 28 16:07:38 ctl01 kernel: [    1.339539] Freeing unused kernel image memory: 2008K
Aug 28 16:07:38 ctl01 kernel: [    1.342095] Freeing unused kernel image memory: 1864K
Aug 28 16:07:38 ctl01 kernel: [    1.354004] x86/mm: Checked W+X mappings: passed, no W+X pages found.
Aug 28 16:07:38 ctl01 kernel: [    1.355302] x86/mm: Checking user space page tables
Aug 28 16:07:38 ctl01 kernel: [    1.362926] x86/mm: Checked W+X mappings: passed, no W+X pages found.
Aug 28 16:07:38 ctl01 kernel: [    1.451419] GPT:Primary header thinks Alt. header is not at the end of the disk.
Aug 28 16:07:38 ctl01 kernel: [    1.454119] GPT:4612095 != 209715199
Aug 28 16:07:38 ctl01 kernel: [    1.455609] GPT:Alternate GPT header not at the end of the disk.
Aug 28 16:07:38 ctl01 kernel: [    1.457338] GPT:4612095 != 209715199
Aug 28 16:07:38 ctl01 kernel: [    1.458235] GPT: Use GNU Parted to correct GPT errors.
Aug 28 16:07:38 ctl01 kernel: [    1.459446]  vda: vda1 vda14 vda15
Aug 28 16:07:38 ctl01 kernel: [    1.460951] FDC 0 is a S82078B
Aug 28 16:07:38 ctl01 kernel: [    1.463453] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
Aug 28 16:07:38 ctl01 kernel: [    1.467273] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
Aug 28 16:07:38 ctl01 kernel: [    1.474257] AVX version of gcm_enc/dec engaged.
Aug 28 16:07:38 ctl01 kernel: [    1.475373] AES CTR mode by8 optimization enabled
Aug 28 16:07:38 ctl01 kernel: [    1.485885] virtio_net virtio0 ens3: renamed from eth0
Aug 28 16:07:38 ctl01 kernel: [    1.509789] virtio_net virtio2 ens5: renamed from eth2
Aug 28 16:07:38 ctl01 kernel: [    1.536351] virtio_net virtio3 ens6: renamed from eth3
Aug 28 16:07:38 ctl01 kernel: [    1.556119] virtio_net virtio1 ens4: renamed from eth1
Aug 28 16:07:38 ctl01 kernel: [    3.176037] raid6: sse2x1   gen()  6743 MB/s
Aug 28 16:07:38 ctl01 kernel: [    3.224035] raid6: sse2x1   xor()  4423 MB/s
Aug 28 16:07:38 ctl01 kernel: [    3.272026] raid6: sse2x2   gen()  9522 MB/s
Aug 28 16:07:38 ctl01 kernel: [    3.320030] raid6: sse2x2   xor()  5872 MB/s
Aug 28 16:07:38 ctl01 kernel: [    3.368029] raid6: sse2x4   gen() 10307 MB/s
Aug 28 16:07:38 ctl01 kernel: [    3.416028] raid6: sse2x4   xor()  7194 MB/s
Aug 28 16:07:38 ctl01 kernel: [    3.417097] raid6: using algorithm sse2x4 gen() 10307 MB/s
Aug 28 16:07:38 ctl01 kernel: [    3.418374] raid6: .... xor() 7194 MB/s, rmw enabled
Aug 28 16:07:38 ctl01 kernel: [    3.419542] raid6: using ssse3x2 recovery algorithm
Aug 28 16:07:38 ctl01 kernel: [    3.421992] xor: automatically using best checksumming function   avx       
Aug 28 16:07:38 ctl01 kernel: [    3.424867] async_tx: api initialized (async)
Aug 28 16:07:38 ctl01 kernel: [    3.471706] Btrfs loaded, crc32c=crc32c-intel
Aug 28 16:07:38 ctl01 kernel: [    3.517806] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
Aug 28 16:07:38 ctl01 kernel: [    3.621599] random: fast init done
Aug 28 16:07:38 ctl01 kernel: [    3.817497] ip_tables: (C) 2000-2006 Netfilter Core Team
Aug 28 16:07:38 ctl01 kernel: [    3.827878] random: systemd: uninitialized urandom read (16 bytes read)
Aug 28 16:07:38 ctl01 kernel: [    3.833296] systemd[1]: systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Aug 28 16:07:38 ctl01 kernel: [    3.838225] systemd[1]: Detected virtualization kvm.
Aug 28 16:07:38 ctl01 kernel: [    3.839459] systemd[1]: Detected architecture x86-64.
Aug 28 16:07:38 ctl01 kernel: [    3.840792] random: systemd: uninitialized urandom read (16 bytes read)
Aug 28 16:07:38 ctl01 kernel: [    3.842387] random: systemd: uninitialized urandom read (16 bytes read)
Aug 28 16:07:38 ctl01 kernel: [    3.849665] systemd[1]: Set hostname to <ubuntu>.
Aug 28 16:07:38 ctl01 kernel: [    3.851946] systemd[1]: Initializing machine ID from KVM UUID.
Aug 28 16:07:38 ctl01 kernel: [    3.853387] systemd[1]: Installed transient /etc/machine-id file.
Aug 28 16:07:38 ctl01 kernel: [    4.071092] systemd[1]: Created slice User and Session Slice.
Aug 28 16:07:38 ctl01 kernel: [    4.073524] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
Aug 28 16:07:38 ctl01 kernel: [    4.076773] systemd[1]: Reached target Swap.
Aug 28 16:07:38 ctl01 kernel: [    4.078715] systemd[1]: Created slice System Slice.
Aug 28 16:07:38 ctl01 kernel: [    4.122550] EXT4-fs (vda1): re-mounted. Opts: (null)
Aug 28 16:07:38 ctl01 kernel: [    4.154711] Loading iSCSI transport class v2.0-870.
Aug 28 16:07:38 ctl01 kernel: [    4.184851] iscsi: registered transport (tcp)
Aug 28 16:07:38 ctl01 kernel: [    4.310001] systemd-journald[463]: Received request to flush runtime journal from PID 1
Aug 28 16:07:38 ctl01 kernel: [    4.329547] iscsi: registered transport (iser)
Aug 28 16:07:38 ctl01 kernel: [    6.261737] audit: type=1400 audit(1567008449.452:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=752 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.361066] audit: type=1400 audit(1567008449.552:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=747 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.365035] audit: type=1400 audit(1567008449.556:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=743 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.365405] audit: type=1400 audit(1567008449.556:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=743 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.365800] audit: type=1400 audit(1567008449.556:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=743 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.470096] audit: type=1400 audit(1567008449.660:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=739 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.470501] audit: type=1400 audit(1567008449.660:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=739 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.470905] audit: type=1400 audit(1567008449.660:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=739 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.471297] audit: type=1400 audit(1567008449.660:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=739 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    6.497177] audit: type=1400 audit(1567008449.688:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=744 comm="apparmor_parser"
Aug 28 16:07:38 ctl01 kernel: [    7.716883] ISO 9660 Extensions: Microsoft Joliet Level 3
Aug 28 16:07:38 ctl01 kernel: [    7.719358] ISO 9660 Extensions: RRIP_1991A
Aug 28 16:07:38 ctl01 kernel: [   11.598789] EXT4-fs (vda1): resizing filesystem from 548091 to 26185979 blocks
Aug 28 16:07:38 ctl01 kernel: [   12.198743] EXT4-fs (vda1): resized filesystem to 26185979
Aug 28 16:07:38 ctl01 kernel: [   15.386401] new mount options do not match the existing superblock, will be ignored
Aug 28 16:07:38 ctl01 systemd[1]: Reached target Host and Network Name Lookups.
Aug 28 16:07:38 ctl01 polkitd[1464]: started daemon version 0.105 using authority implementation `local' version `0.105'
Aug 28 16:07:38 ctl01 dbus-daemon[1242]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'
Aug 28 16:07:38 ctl01 systemd[1]: Started Authorization Manager.
Aug 28 16:07:38 ctl01 accounts-daemon[1276]: started daemon version 0.6.45
Aug 28 16:07:38 ctl01 systemd[1]: Started Accounts Service.
Aug 28 16:07:39 ctl01 snapd[1343]: AppArmor status: apparmor is enabled and all features are available
Aug 28 16:07:39 ctl01 pollinate[1237]: client verified challenge/response with [https://entropy.ubuntu.com/]
Aug 28 16:07:39 ctl01 pollinate[1237]: client hashed response from [https://entropy.ubuntu.com/]
Aug 28 16:07:39 ctl01 pollinate[1237]: client successfully seeded [/dev/urandom]
Aug 28 16:07:39 ctl01 systemd[1]: Started Pollinate to seed the pseudo random number generator.
Aug 28 16:07:39 ctl01 systemd[1]: Starting OpenBSD Secure Shell server...
Aug 28 16:07:39 ctl01 snapd[1343]: helpers.go:146: error trying to compare the snap system key: system-key missing on disk
Aug 28 16:07:39 ctl01 systemd[1]: Started LXD - container startup/shutdown.
Aug 28 16:07:39 ctl01 snapd[1343]: daemon.go:379: started snapd/2.39.2+18.04 (series 16; classic) ubuntu/18.04 (amd64) linux/4.15.0-58-generic.
Aug 28 16:07:39 ctl01 systemd[1]: Started OpenBSD Secure Shell server.
Aug 28 16:07:39 ctl01 systemd[1]: Started Snappy daemon.
Aug 28 16:07:39 ctl01 systemd[1]: Starting Wait until snapd is fully seeded...
Aug 28 16:07:39 ctl01 systemd[1]: Started The Salt Minion.
Aug 28 16:07:41 ctl01 systemd[1]: Started Wait until snapd is fully seeded.
Aug 28 16:07:41 ctl01 systemd[1]: Reached target Multi-User System.
Aug 28 16:07:41 ctl01 systemd[1]: Reached target Graphical Interface.
Aug 28 16:07:41 ctl01 systemd[1]: Starting Update UTMP about System Runlevel Changes...
Aug 28 16:07:41 ctl01 systemd[1]: Starting Apply the settings specified in cloud-config...
Aug 28 16:07:41 ctl01 systemd[1]: Started Update UTMP about System Runlevel Changes.
Aug 28 16:07:41 ctl01 kernel: [   18.787263] random: crng init done
Aug 28 16:07:41 ctl01 kernel: [   18.787270] random: 7 urandom warning(s) missed due to ratelimiting
Aug 28 16:07:42 ctl01 salt-minion[1238]: [ERROR   ] DNS lookup or connection check of 'salt' failed.
Aug 28 16:07:42 ctl01 salt-minion[1238]: [ERROR   ] Master hostname: 'salt' not found or not responsive. Retrying in 30 seconds
Aug 28 16:07:43 ctl01 cloud-init[1670]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:config' at Wed, 28 Aug 2019 16:07:41 +0000. Up 18.31 seconds.
Aug 28 16:07:43 ctl01 systemd[1]: Started Apply the settings specified in cloud-config.
Aug 28 16:07:43 ctl01 systemd[1]: Starting Execute cloud user/final scripts...
Aug 28 16:07:44 ctl01 systemd[1]: Reloading.
Aug 28 16:07:44 ctl01 snapd[1343]: daemon.go:611: gracefully waiting for running hooks
Aug 28 16:07:44 ctl01 snapd[1343]: daemon.go:613: done waiting for running hooks
Aug 28 16:07:44 ctl01 snapd[1343]: daemon stop requested to wait for socket activation
Aug 28 16:07:44 ctl01 cloud-init[1745]: Synchronizing state of networking.service with SysV service script with /lib/systemd/systemd-sysv-install.
Aug 28 16:07:44 ctl01 cloud-init[1745]: Executing: /lib/systemd/systemd-sysv-install enable networking
Aug 28 16:07:44 ctl01 systemd[1]: Reloading.
Aug 28 16:07:44 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:07:45 ctl01 cloud-init[1745]: Synchronizing state of salt-minion.service with SysV service script with /lib/systemd/systemd-sysv-install.
Aug 28 16:07:45 ctl01 cloud-init[1745]: Executing: /lib/systemd/systemd-sysv-install enable salt-minion
Aug 28 16:07:45 ctl01 systemd[1]: Reloading.
Aug 28 16:07:45 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:07:45 ctl01 systemd[1]: Stopping The Salt Minion...
Aug 28 16:07:45 ctl01 salt-minion[1238]: [WARNING ] Minion received a SIGTERM. Exiting.
Aug 28 16:07:46 ctl01 salt-minion[1238]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited.
Aug 28 16:07:46 ctl01 systemd[1]: Stopped The Salt Minion.
Aug 28 16:07:46 ctl01 systemd[1]: Starting The Salt Minion...
Aug 28 16:07:46 ctl01 systemd[1]: Started The Salt Minion.
Aug 28 16:07:46 ctl01 ec2: 
Aug 28 16:07:46 ctl01 ec2: #############################################################
Aug 28 16:07:46 ctl01 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
Aug 28 16:07:46 ctl01 ec2: 1024 SHA256:anOtrDzT6lBks2TSNEy9PaGpyMtTN70xrRFH8M8QUrM root@ctl01 (DSA)
Aug 28 16:07:46 ctl01 ec2: 256 SHA256:HYGlII/LYnNdg9eWOHLyVe7rBuYldfyaXxfGuEZwgzA root@ctl01 (ECDSA)
Aug 28 16:07:46 ctl01 ec2: 256 SHA256:v66WtajLBh8kpNMrx56KLa6CuaK4hYWT5QBgLEDRF9I root@ctl01 (ED25519)
Aug 28 16:07:46 ctl01 ec2: 2048 SHA256:pfW/uSiLOWPjMDbNdktpPWsm0O/fEyRGqDwhCPAXV8M root@ctl01 (RSA)
Aug 28 16:07:46 ctl01 ec2: -----END SSH HOST KEY FINGERPRINTS-----
Aug 28 16:07:46 ctl01 ec2: #############################################################
Aug 28 16:07:46 ctl01 cloud-init[1745]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:final' at Wed, 28 Aug 2019 16:07:43 +0000. Up 20.43 seconds.
Aug 28 16:07:46 ctl01 cloud-init[1745]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 finished at Wed, 28 Aug 2019 16:07:46 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 23.09 seconds
Aug 28 16:07:46 ctl01 systemd[1]: Started Execute cloud user/final scripts.
Aug 28 16:07:46 ctl01 systemd[1]: Reached target Cloud-init target.
Aug 28 16:07:46 ctl01 systemd[1]: Startup finished in 3.767s (kernel) + 19.379s (userspace) = 23.146s.
Aug 28 16:07:59 ctl01 systemd-timesyncd[686]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).
Aug 28 16:10:11 ctl01 salt-minion[1979]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
Aug 28 16:10:16 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y remove telnet.
Aug 28 16:10:22 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install smartmontools.
Aug 28 16:10:30 ctl01 systemd[1]: Reloading.
Aug 28 16:10:33 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:10:34 ctl01 systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
Aug 28 16:10:34 ctl01 systemd[1]: Reloading.
Aug 28 16:10:34 ctl01 smartd[4096]: smartd 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-58-generic] (local build)
Aug 28 16:10:34 ctl01 smartd[4096]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Aug 28 16:10:34 ctl01 smartd[4096]: Opened configuration file /etc/smartd.conf
Aug 28 16:10:34 ctl01 smartd[4096]: Drive: DEVICESCAN, implied '-a' Directive on line 21 of file /etc/smartd.conf
Aug 28 16:10:34 ctl01 smartd[4096]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Aug 28 16:10:34 ctl01 smartd[4096]: DEVICESCAN failed: glob(3) aborted matching pattern /dev/discs/disc*
Aug 28 16:10:34 ctl01 smartd[4096]: In the system's table of devices NO devices found to scan
Aug 28 16:10:34 ctl01 smartd[4096]: Unable to monitor any SMART enabled devices. Try debug (-d) option. Exiting...
Aug 28 16:10:35 ctl01 systemd[1]: smartd.service: Main process exited, code=exited, status=17/n/a
Aug 28 16:10:35 ctl01 systemd[1]: smartd.service: Failed with result 'exit-code'.
Aug 28 16:10:35 ctl01 systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
Aug 28 16:10:35 ctl01 smartd[4139]: smartd 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-58-generic] (local build)
Aug 28 16:10:35 ctl01 smartd[4139]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Aug 28 16:10:35 ctl01 smartd[4139]: Opened configuration file /etc/smartd.conf
Aug 28 16:10:35 ctl01 smartd[4139]: Drive: DEVICESCAN, implied '-a' Directive on line 21 of file /etc/smartd.conf
Aug 28 16:10:35 ctl01 smartd[4139]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Aug 28 16:10:35 ctl01 smartd[4139]: DEVICESCAN failed: glob(3) aborted matching pattern /dev/discs/disc*
Aug 28 16:10:35 ctl01 smartd[4139]: In the system's table of devices NO devices found to scan
Aug 28 16:10:35 ctl01 smartd[4139]: Unable to monitor any SMART enabled devices. Try debug (-d) option. Exiting...
Aug 28 16:10:35 ctl01 systemd[1]: smartd.service: Main process exited, code=exited, status=17/n/a
Aug 28 16:10:35 ctl01 systemd[1]: smartd.service: Failed with result 'exit-code'.
Aug 28 16:10:36 ctl01 systemd[1]: Reloading.
Aug 28 16:10:38 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:10:39 ctl01 systemd[1]: Created slice system-postfix.slice.
Aug 28 16:10:39 ctl01 systemd[1]: Starting Postfix Mail Transport Agent (instance -)...
Aug 28 16:10:39 ctl01 configure-instance.sh[4273]: postconf: fatal: open /etc/postfix/main.cf: No such file or directory
Aug 28 16:10:40 ctl01 configure-instance.sh[4273]: postconf: fatal: open /etc/postfix/main.cf: No such file or directory
Aug 28 16:10:41 ctl01 systemd[1]: postfix@-.service: Control process exited, code=exited status=1
Aug 28 16:10:41 ctl01 systemd[1]: postfix@-.service: Failed with result 'exit-code'.
Aug 28 16:10:41 ctl01 systemd[1]: Failed to start Postfix Mail Transport Agent (instance -).
Aug 28 16:10:48 ctl01 systemd[1]: Reloading.
Aug 28 16:10:48 ctl01 systemd[1]: Starting Postfix Mail Transport Agent (instance -)...
Aug 28 16:10:49 ctl01 postfix/postfix-script[4641]: starting the Postfix mail system
Aug 28 16:10:49 ctl01 postfix/master[4643]: daemon started -- version 3.3.0, configuration /etc/postfix
Aug 28 16:10:49 ctl01 systemd[1]: Started Postfix Mail Transport Agent (instance -).
Aug 28 16:10:49 ctl01 systemd[1]: Starting Postfix Mail Transport Agent...
Aug 28 16:10:49 ctl01 systemd[1]: Started Postfix Mail Transport Agent.
Aug 28 16:10:49 ctl01 systemd[1]: Reloading.
Aug 28 16:10:51 ctl01 systemd[1]: Reloading.
Aug 28 16:10:51 ctl01 systemd[1]: Stopping System Logging Service...
Aug 28 16:10:51 ctl01 rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="1348" x-info="http://www.rsyslog.com"] exiting on signal 15.
Aug 28 16:10:51 ctl01 systemd[1]: Stopped System Logging Service.
Aug 28 16:10:51 ctl01 systemd[1]: Starting System Logging Service...
Aug 28 16:10:51 ctl01 rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.32.0]
Aug 28 16:10:51 ctl01 systemd[1]: Started System Logging Service.
Aug 28 16:10:51 ctl01 rsyslogd: rsyslogd's groupid changed to 106
Aug 28 16:10:51 ctl01 rsyslogd: rsyslogd's userid changed to 102
Aug 28 16:10:51 ctl01 rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="5136" x-info="http://www.rsyslog.com"] start
Aug 28 16:10:53 ctl01 dbus-daemon[1242]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.22' (uid=0 pid=5216 comm="timedatectl " label="unconfined")
Aug 28 16:10:53 ctl01 systemd[1]: Starting Time & Date Service...
Aug 28 16:10:53 ctl01 dbus-daemon[1242]: [system] Successfully activated service 'org.freedesktop.timedate1'
Aug 28 16:10:53 ctl01 systemd[1]: Started Time & Date Service.
Aug 28 16:10:54 ctl01 salt-minion[1979]: [WARNING ] State for file: /boot/grub/grub.cfg - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:10:54 ctl01 kernel: [  211.877873] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
Aug 28 16:10:59 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install sysfsutils.
Aug 28 16:11:01 ctl01 systemd[1]: Reloading.
Aug 28 16:11:01 ctl01 systemd[1]: Reloading.
Aug 28 16:11:01 ctl01 systemd[1]: Starting LSB: Set sysfs variables from /etc/sysfs.conf...
Aug 28 16:11:01 ctl01 sysfsutils[5979]:  * Setting sysfs variables...
Aug 28 16:11:01 ctl01 sysfsutils[5979]:    ...done.
Aug 28 16:11:01 ctl01 systemd[1]: Started LSB: Set sysfs variables from /etc/sysfs.conf.
Aug 28 16:11:01 ctl01 systemd[1]: Reloading.
Aug 28 16:11:05 ctl01 systemd[1]: Started /bin/systemctl disable ondemand.service.
Aug 28 16:11:05 ctl01 systemd[1]: Reloading.
Aug 28 16:11:05 ctl01 dbus-daemon[1242]: [system] Activating via systemd: service name='org.freedesktop.locale1' unit='dbus-org.freedesktop.locale1.service' requested by ':1.25' (uid=0 pid=6257 comm="localectl " label="unconfined")
Aug 28 16:11:05 ctl01 systemd[1]: Starting Locale Service...
Aug 28 16:11:05 ctl01 dbus-daemon[1242]: [system] Successfully activated service 'org.freedesktop.locale1'
Aug 28 16:11:05 ctl01 systemd[1]: Started Locale Service.
Aug 28 16:11:05 ctl01 systemd-localed[6264]: Changed locale to LANG=en_US.UTF-8.
Aug 28 16:11:05 ctl01 salt-minion[1979]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
Aug 28 16:11:05 ctl01 systemd[1]: Reloading.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/shadow - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/gshadow - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/group- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/group - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/passwd- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/passwd - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/gshadow- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 salt-minion[1979]: [WARNING ] State for file: /etc/shadow- - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:11:06 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install openvswitch-switch bridge-utils vlan.
Aug 28 16:11:10 ctl01 systemd[1]: Reloading.
Aug 28 16:11:10 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:11:11 ctl01 systemd[1]: Starting Open vSwitch Database Unit...
Aug 28 16:11:11 ctl01 ovs-ctl[6576]:  * /etc/openvswitch/conf.db does not exist
Aug 28 16:11:11 ctl01 ovs-ctl[6576]:  * Creating empty database /etc/openvswitch/conf.db
Aug 28 16:11:11 ctl01 ovs-ctl[6576]:  * Starting ovsdb-server
Aug 28 16:11:11 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.16.1
Aug 28 16:11:11 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.11.1 "external-ids:system-id=\"54f19e88-f36a-46eb-9077-8d0f01b13a75\"" "external-ids:rundir=\"/var/run/openvswitch\"" "system-type=\"ubuntu\"" "system-version=\"18.04\""
Aug 28 16:11:11 ctl01 ovs-ctl[6576]:  * Configuring Open vSwitch system IDs
Aug 28 16:11:11 ctl01 ovs-ctl[6576]:  * Enabling remote OVSDB managers
Aug 28 16:11:11 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . external-ids:hostname=ctl01
Aug 28 16:11:11 ctl01 systemd[1]: Started Open vSwitch Database Unit.
Aug 28 16:11:11 ctl01 systemd[1]: Starting Open vSwitch Forwarding Unit...
Aug 28 16:11:11 ctl01 kernel: [  228.305298] openvswitch: Open vSwitch switching datapath
Aug 28 16:11:11 ctl01 ovs-ctl[6640]:  * Inserting openvswitch module
Aug 28 16:11:11 ctl01 ovs-ctl[6640]:  * Starting ovs-vswitchd
Aug 28 16:11:11 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . external-ids:hostname=ctl01
Aug 28 16:11:11 ctl01 ovs-ctl[6640]:  * Enabling remote OVSDB managers
Aug 28 16:11:11 ctl01 systemd[1]: Started Open vSwitch Forwarding Unit.
Aug 28 16:11:11 ctl01 systemd[1]: Starting Open vSwitch...
Aug 28 16:11:11 ctl01 systemd[1]: Started Open vSwitch.
Aug 28 16:11:11 ctl01 systemd[1]: Reloading.
Aug 28 16:11:16 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install bridge-utils.
Aug 28 16:11:17 ctl01 systemd[1]: Reloading.
Aug 28 16:11:17 ctl01 systemd[1]: Started /bin/systemctl enable networking.service.
Aug 28 16:11:17 ctl01 systemd[1]: Reloading.
Aug 28 16:11:18 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:11:18 ctl01 dnsmasq[1481]: reading /etc/resolv.conf
Aug 28 16:11:18 ctl01 dnsmasq[1481]: using nameserver 8.8.8.8#53
Aug 28 16:11:18 ctl01 salt-minion[1979]: [WARNING ] The network state sls is requiring a reboot of the system to properly apply network configuration.
Aug 28 16:11:18 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install vlan.
Aug 28 16:11:19 ctl01 kernel: [  236.188120] 8021q: 802.1Q VLAN Support v1.8
Aug 28 16:11:19 ctl01 kernel: [  236.188134] 8021q: adding VLAN 0 to HW filter on device ens3
Aug 28 16:11:19 ctl01 kernel: [  236.188222] 8021q: adding VLAN 0 to HW filter on device ens5
Aug 28 16:11:19 ctl01 systemd-udevd[7470]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:11:19 ctl01 systemd[1]: Reloading OpenBSD Secure Shell server.
Aug 28 16:11:19 ctl01 systemd[1]: Reloaded OpenBSD Secure Shell server.
Aug 28 16:11:19 ctl01 systemd[1]: Found device /sys/subsystem/net/devices/ens5.1000.
Aug 28 16:11:19 ctl01 systemd[1]: Started ifup for ens5.1000.
Aug 28 16:11:19 ctl01 sh[7578]: ifup: waiting for lock on /run/network/ifstate.ens5
Aug 28 16:11:19 ctl01 systemd[1]: Reloading Postfix Mail Transport Agent (instance -).
Aug 28 16:11:19 ctl01 postfix/postfix-script[7595]: refreshing the Postfix mail system
Aug 28 16:11:19 ctl01 postfix/master[4643]: reload -- version 3.3.0, configuration /etc/postfix
Aug 28 16:11:19 ctl01 systemd[1]: Reloaded Postfix Mail Transport Agent (instance -).
Aug 28 16:11:19 ctl01 systemd[1]: Reloading Postfix Mail Transport Agent.
Aug 28 16:11:19 ctl01 systemd[1]: Reloaded Postfix Mail Transport Agent.
Aug 28 16:11:19 ctl01 sh[7578]: ifup: interface ens5.1000 already configured
Aug 28 16:11:19 ctl01 salt-minion[1979]: [ERROR   ] Command '['umount', '/dev/shm']' failed with return code: 32
Aug 28 16:11:19 ctl01 salt-minion[1979]: [ERROR   ] stderr: umount: /dev/shm: target is busy.
Aug 28 16:11:19 ctl01 salt-minion[1979]: [ERROR   ] retcode: 32
Aug 28 16:11:23 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef dist-upgrade.
Aug 28 16:11:35 ctl01 systemd[1]: Stopping Authorization Manager...
Aug 28 16:11:35 ctl01 systemd[1]: Stopped target Graphical Interface.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped target Cloud-init target.
Aug 28 16:11:35 ctl01 systemd[1]: Stopping Locale Service...
Aug 28 16:11:35 ctl01 systemd[1]: Stopped target Timers.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped Message of the Day.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped Discard unused blocks once a week.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped Daily apt upgrade and clean activities.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped Daily apt download activities.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped target System Time Synchronized.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped Daily Cleanup of Temporary Directories.
Aug 28 16:11:35 ctl01 systemd[1]: Stopping Availability of block devices...
Aug 28 16:11:35 ctl01 systemd[1]: Stopped Execute cloud user/final scripts.
Aug 28 16:11:35 ctl01 systemd[1]: Stopped target Multi-User System.
Aug 28 16:11:35 ctl01 systemd[1]: Stopping FUSE filesystem for LXC...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping LXD - container startup/shutdown...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping The Salt Minion...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping Regular background program processing daemon...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping OpenBSD Secure Shell server...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping D-Bus System Message Bus...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping Unattended Upgrades Shutdown...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping LSB: Record successful boot for GRUB...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping Deferred execution scheduler...
Aug 28 16:11:35 ctl01 systemd[1]: Stopping System Logging Service...
Aug 28 16:11:35 ctl01 systemd[1]: Stopped Postfix Mail Transport Agent.
Aug 28 16:12:07 ctl01 systemd-modules-load[449]: Inserted module 'iscsi_tcp'
Aug 28 16:12:07 ctl01 systemd-modules-load[449]: Inserted module 'ib_iser'
Aug 28 16:12:07 ctl01 systemd[1]: Started udev Coldplug all Devices.
Aug 28 16:12:07 ctl01 systemd[1]: Starting Flush Journal to Persistent Storage...
Aug 28 16:12:07 ctl01 systemd[1]: Started Set the console keyboard layout.
Aug 28 16:12:07 ctl01 systemd[1]: Started Load/Save Random Seed.
Aug 28 16:12:07 ctl01 systemd-modules-load[449]: Inserted module 'nf_conntrack'
Aug 28 16:12:07 ctl01 systemd[1]: Started Load Kernel Modules.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Linux version 4.15.0-58-generic (buildd@lcy01-amd64-013) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 (Ubuntu 4.15.0-58.64-generic 4.15.18)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-58-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
Aug 28 16:12:07 ctl01 kernel: [    0.000000] KERNEL supported cpus:
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   Intel GenuineIntel
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   AMD AuthenticAMD
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   Centaur CentaurHauls
Aug 28 16:12:07 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Aug 28 16:12:07 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Aug 28 16:12:07 ctl01 kernel: [    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Aug 28 16:12:07 ctl01 kernel: [    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Aug 28 16:12:07 ctl01 kernel: [    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] e820: BIOS-provided physical RAM map:
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000003bfffffff] usable
Aug 28 16:12:07 ctl01 kernel: [    0.000000] NX (Execute Disable) protection: active
Aug 28 16:12:07 ctl01 kernel: [    0.000000] SMBIOS 2.8 present.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Hypervisor detected: KVM
Aug 28 16:12:07 ctl01 kernel: [    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Aug 28 16:12:07 ctl01 kernel: [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
Aug 28 16:12:07 ctl01 kernel: [    0.000000] e820: last_pfn = 0x3c0000 max_arch_pfn = 0x400000000
Aug 28 16:12:07 ctl01 kernel: [    0.000000] MTRR default type: write-back
Aug 28 16:12:07 ctl01 kernel: [    0.000000] MTRR fixed ranges enabled:
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   00000-9FFFF write-back
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   A0000-BFFFF uncachable
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   C0000-FFFFF write-protect
Aug 28 16:12:07 ctl01 kernel: [    0.000000] MTRR variable ranges enabled:
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   0 base 00C0000000 mask FFC0000000 uncachable
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   1 disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   2 disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   3 disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   4 disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   5 disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   6 disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   7 disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
Aug 28 16:12:07 ctl01 kernel: [    0.000000] e820: last_pfn = 0xbffdf max_arch_pfn = 0x400000000
Aug 28 16:12:07 ctl01 kernel: [    0.000000] found SMP MP-table at [mem 0x000f6560-0x000f656f] mapped at [        (ptrval)]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Scanning 1 areas for low memory corruption
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Base memory trampoline at [        (ptrval)] 99000 size 24576
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Using GB pages for direct mapping
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BRK [0x190d3d000, 0x190d3dfff] PGTABLE
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BRK [0x190d3e000, 0x190d3efff] PGTABLE
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BRK [0x190d3f000, 0x190d3ffff] PGTABLE
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BRK [0x190d40000, 0x190d40fff] PGTABLE
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BRK [0x190d41000, 0x190d41fff] PGTABLE
Aug 28 16:12:07 ctl01 kernel: [    0.000000] BRK [0x190d42000, 0x190d42fff] PGTABLE
Aug 28 16:12:07 ctl01 kernel: [    0.000000] RAMDISK: [mem 0x35a8b000-0x36d3cfff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: Early table checksum verification disabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: RSDP 0x00000000000F6510 000014 (v00 BOCHS )
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: RSDT 0x00000000BFFE14CC 000030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: FACP 0x00000000BFFE0854 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: DSDT 0x00000000BFFDFC80 000BD4 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: FACS 0x00000000BFFDFC40 000040
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: SSDT 0x00000000BFFE08C8 000B54 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: APIC 0x00000000BFFE141C 0000B0 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
Aug 28 16:12:07 ctl01 kernel: [    0.000000] No NUMA configuration found
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Faking a node at [mem 0x0000000000000000-0x00000003bfffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] NODE_DATA(0) allocated [mem 0x3bffd5000-0x3bfffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] kvm-clock: cpu 0, msr 3:bff54001, primary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
Aug 28 16:12:07 ctl01 kernel: [    0.000000] kvm-clock: using sched offset of 278514167904 cycles
Aug 28 16:12:07 ctl01 kernel: [    0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Zone ranges:
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   Normal   [mem 0x0000000100000000-0x00000003bfffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   Device   empty
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Movable zone start for each node
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Early memory node ranges
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000000100000-0x00000000bffdefff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   node   0: [mem 0x0000000100000000-0x00000003bfffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Reserved but unavailable: 98 pages
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000003bfffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] On node 0 totalpages: 3669885
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   DMA zone: 64 pages used for memmap
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   DMA zone: 21 pages reserved
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   DMA zone: 3998 pages, LIFO batch:0
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   DMA32 zone: 12224 pages used for memmap
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   DMA32 zone: 782303 pages, LIFO batch:31
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   Normal zone: 45056 pages used for memmap
Aug 28 16:12:07 ctl01 kernel: [    0.000000]   Normal zone: 2883584 pages, LIFO batch:31
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: PM-Timer IO Port: 0x608
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Aug 28 16:12:07 ctl01 kernel: [    0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: IRQ0 used by override.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: IRQ5 used by override.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: IRQ9 used by override.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: IRQ10 used by override.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ACPI: IRQ11 used by override.
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Using ACPI (MADT) for SMP configuration information
Aug 28 16:12:07 ctl01 kernel: [    0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xbffdf000-0xbfffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Booting paravirtualized kernel on KVM
Aug 28 16:12:07 ctl01 kernel: [    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
Aug 28 16:12:07 ctl01 kernel: [    0.000000] random: get_random_bytes called from start_kernel+0x99/0x4fd with crng_init=0
Aug 28 16:12:07 ctl01 kernel: [    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
Aug 28 16:12:07 ctl01 kernel: [    0.000000] percpu: Embedded 46 pages/cpu @        (ptrval) s151552 r8192 d28672 u262144
Aug 28 16:12:07 ctl01 kernel: [    0.000000] pcpu-alloc: s151552 r8192 d28672 u262144 alloc=1*2097152
Aug 28 16:12:07 ctl01 kernel: [    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 
Aug 28 16:12:07 ctl01 kernel: [    0.000000] KVM setup async PF for cpu 0
Aug 28 16:12:07 ctl01 kernel: [    0.000000] kvm-stealtime: cpu 0, msr 3bfc24040
Aug 28 16:12:07 ctl01 kernel: [    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 3612520
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Policy zone: Normal
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-58-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Calgary: detecting Calgary via BIOS EBDA area
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Memory: 14334824K/14679540K available (12300K kernel code, 2478K rwdata, 4280K rodata, 2424K init, 2384K bss, 344716K reserved, 0K cma-reserved)
Aug 28 16:12:07 ctl01 kernel: [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
Aug 28 16:12:07 ctl01 kernel: [    0.000000] Kernel/User page tables isolation: enabled
Aug 28 16:12:07 ctl01 kernel: [    0.000000] ftrace: allocating 39243 entries in 154 pages
Aug 28 16:12:07 ctl01 kernel: [    0.004000] Hierarchical RCU implementation.
Aug 28 16:12:07 ctl01 kernel: [    0.004000] 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=8.
Aug 28 16:12:07 ctl01 kernel: [    0.004000] 	Tasks RCU enabled.
Aug 28 16:12:07 ctl01 kernel: [    0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
Aug 28 16:12:07 ctl01 kernel: [    0.004000] NR_IRQS: 524544, nr_irqs: 488, preallocated irqs: 16
Aug 28 16:12:07 ctl01 kernel: [    0.004000] Console: colour VGA+ 80x25
Aug 28 16:12:07 ctl01 kernel: [    0.004000] console [tty1] enabled
Aug 28 16:12:07 ctl01 kernel: [    0.004000] console [ttyS0] enabled
Aug 28 16:12:07 ctl01 kernel: [    0.004000] ACPI: Core revision 20170831
Aug 28 16:12:07 ctl01 kernel: [    0.004000] ACPI: 2 ACPI AML tables successfully acquired and loaded
Aug 28 16:12:07 ctl01 kernel: [    0.004006] APIC: Switch to symmetric I/O mode setup
Aug 28 16:12:07 ctl01 kernel: [    0.005508] x2apic enabled
Aug 28 16:12:07 ctl01 kernel: [    0.006574] Switched APIC routing to physical x2apic.
Aug 28 16:12:07 ctl01 kernel: [    0.009181] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Aug 28 16:12:07 ctl01 kernel: [    0.010598] tsc: Detected 2799.994 MHz processor
Aug 28 16:12:07 ctl01 kernel: [    0.011706] Calibrating delay loop (skipped) preset value.. 5599.98 BogoMIPS (lpj=11199976)
Aug 28 16:12:07 ctl01 kernel: [    0.012003] pid_max: default: 32768 minimum: 301
Aug 28 16:12:07 ctl01 kernel: [    0.013136] Security Framework initialized
Aug 28 16:12:07 ctl01 kernel: [    0.014153] Yama: becoming mindful.
Aug 28 16:12:07 ctl01 kernel: [    0.016029] AppArmor: AppArmor initialized
Aug 28 16:12:07 ctl01 kernel: [    0.020596] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.023742] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.024053] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.025671] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.028297] mce: CPU supports 10 MCE banks
Aug 28 16:12:07 ctl01 kernel: [    0.029337] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
Aug 28 16:12:07 ctl01 kernel: [    0.030590] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
Aug 28 16:12:07 ctl01 kernel: [    0.032003] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
Aug 28 16:12:07 ctl01 kernel: [    0.033935] Spectre V2 : Mitigation: Full generic retpoline
Aug 28 16:12:07 ctl01 kernel: [    0.036002] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
Aug 28 16:12:07 ctl01 kernel: [    0.037884] Spectre V2 : Enabling Restricted Speculation for firmware calls
Aug 28 16:12:07 ctl01 kernel: [    0.039456] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
Aug 28 16:12:07 ctl01 kernel: [    0.040002] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
Aug 28 16:12:07 ctl01 kernel: [    0.042198] MDS: Mitigation: Clear CPU buffers
Aug 28 16:12:07 ctl01 kernel: [    0.050113] Freeing SMP alternatives memory: 36K
Aug 28 16:12:07 ctl01 kernel: [    0.052479] TSC deadline timer enabled
Aug 28 16:12:07 ctl01 kernel: [    0.052483] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (family: 0x6, model: 0x3e, stepping: 0x4)
Aug 28 16:12:07 ctl01 kernel: [    0.054825] Performance Events: IvyBridge events, Intel PMU driver.
Aug 28 16:12:07 ctl01 kernel: [    0.056000] ... version:                2
Aug 28 16:12:07 ctl01 kernel: [    0.056005] ... bit width:              48
Aug 28 16:12:07 ctl01 kernel: [    0.056999] ... generic registers:      4
Aug 28 16:12:07 ctl01 kernel: [    0.057984] ... value mask:             0000ffffffffffff
Aug 28 16:12:07 ctl01 kernel: [    0.059219] ... max period:             000000007fffffff
Aug 28 16:12:07 ctl01 kernel: [    0.060004] ... fixed-purpose events:   3
Aug 28 16:12:07 ctl01 kernel: [    0.060980] ... event mask:             000000070000000f
Aug 28 16:12:07 ctl01 kernel: [    0.062277] Hierarchical SRCU implementation.
Aug 28 16:12:07 ctl01 kernel: [    0.064137] smp: Bringing up secondary CPUs ...
Aug 28 16:12:07 ctl01 kernel: [    0.065381] x86: Booting SMP configuration:
Aug 28 16:12:07 ctl01 kernel: [    0.066394] .... node  #0, CPUs:      #1
Aug 28 16:12:07 ctl01 kernel: [    0.004000] kvm-clock: cpu 1, msr 3:bff54041, secondary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.072024] KVM setup async PF for cpu 1
Aug 28 16:12:07 ctl01 kernel: [    0.072992] kvm-stealtime: cpu 1, msr 3bfc64040
Aug 28 16:12:07 ctl01 kernel: [    0.074096]  #2
Aug 28 16:12:07 ctl01 kernel: [    0.004000] kvm-clock: cpu 2, msr 3:bff54081, secondary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.076024] KVM setup async PF for cpu 2
Aug 28 16:12:07 ctl01 kernel: [    0.076973] kvm-stealtime: cpu 2, msr 3bfca4040
Aug 28 16:12:07 ctl01 kernel: [    0.078077]  #3
Aug 28 16:12:07 ctl01 kernel: [    0.004000] kvm-clock: cpu 3, msr 3:bff540c1, secondary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.080021] KVM setup async PF for cpu 3
Aug 28 16:12:07 ctl01 kernel: [    0.080983] kvm-stealtime: cpu 3, msr 3bfce4040
Aug 28 16:12:07 ctl01 kernel: [    0.082096]  #4
Aug 28 16:12:07 ctl01 kernel: [    0.004000] kvm-clock: cpu 4, msr 3:bff54101, secondary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.084020] KVM setup async PF for cpu 4
Aug 28 16:12:07 ctl01 kernel: [    0.084972] kvm-stealtime: cpu 4, msr 3bfd24040
Aug 28 16:12:07 ctl01 kernel: [    0.086093]  #5
Aug 28 16:12:07 ctl01 kernel: [    0.004000] kvm-clock: cpu 5, msr 3:bff54141, secondary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.088022] KVM setup async PF for cpu 5
Aug 28 16:12:07 ctl01 kernel: [    0.090648] kvm-stealtime: cpu 5, msr 3bfd64040
Aug 28 16:12:07 ctl01 kernel: [    0.091749]  #6
Aug 28 16:12:07 ctl01 kernel: [    0.004000] kvm-clock: cpu 6, msr 3:bff54181, secondary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.096022] KVM setup async PF for cpu 6
Aug 28 16:12:07 ctl01 kernel: [    0.096983] kvm-stealtime: cpu 6, msr 3bfda4040
Aug 28 16:12:07 ctl01 kernel: [    0.104164]  #7
Aug 28 16:12:07 ctl01 kernel: [    0.004000] kvm-clock: cpu 7, msr 3:bff541c1, secondary cpu clock
Aug 28 16:12:07 ctl01 kernel: [    0.112022] KVM setup async PF for cpu 7
Aug 28 16:12:07 ctl01 kernel: [    0.112968] kvm-stealtime: cpu 7, msr 3bfde4040
Aug 28 16:12:07 ctl01 kernel: [    0.114072] smp: Brought up 1 node, 8 CPUs
Aug 28 16:12:07 ctl01 kernel: [    0.114072] smpboot: Max logical packages: 8
Aug 28 16:12:07 ctl01 kernel: [    0.116007] smpboot: Total of 8 processors activated (44799.90 BogoMIPS)
Aug 28 16:12:07 ctl01 kernel: [    0.118404] devtmpfs: initialized
Aug 28 16:12:07 ctl01 kernel: [    0.118404] x86/mm: Memory block size: 128MB
Aug 28 16:12:07 ctl01 kernel: [    0.121185] evm: security.selinux
Aug 28 16:12:07 ctl01 kernel: [    0.122034] evm: security.SMACK64
Aug 28 16:12:07 ctl01 kernel: [    0.122863] evm: security.SMACK64EXEC
Aug 28 16:12:07 ctl01 kernel: [    0.123778] evm: security.SMACK64TRANSMUTE
Aug 28 16:12:07 ctl01 kernel: [    0.124006] evm: security.SMACK64MMAP
Aug 28 16:12:07 ctl01 kernel: [    0.124911] evm: security.apparmor
Aug 28 16:12:07 ctl01 kernel: [    0.125754] evm: security.ima
Aug 28 16:12:07 ctl01 kernel: [    0.126529] evm: security.capability
Aug 28 16:12:07 ctl01 kernel: [    0.127460] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
Aug 28 16:12:07 ctl01 kernel: [    0.128016] futex hash table entries: 2048 (order: 5, 131072 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.129557] pinctrl core: initialized pinctrl subsystem
Aug 28 16:12:07 ctl01 systemd[1]: Mounting Kernel Configuration File System...
Aug 28 16:12:07 ctl01 kernel: [    0.130970] RTC time: 16:11:51, date: 08/28/19
Aug 28 16:12:07 ctl01 systemd[1]: Mounting FUSE Control File System...
Aug 28 16:12:07 ctl01 kernel: [    0.132854] NET: Registered protocol family 16
Aug 28 16:12:07 ctl01 systemd[1]: Starting Apply Kernel Variables...
Aug 28 16:12:07 ctl01 kernel: [    0.134010] audit: initializing netlink subsys (disabled)
Aug 28 16:12:07 ctl01 kernel: [    0.135301] audit: type=2000 audit(1567008710.054:1): state=initialized audit_enabled=0 res=1
Aug 28 16:12:07 ctl01 kernel: [    0.136025] cpuidle: using governor ladder
Aug 28 16:12:07 ctl01 kernel: [    0.137045] cpuidle: using governor menu
Aug 28 16:12:07 ctl01 systemd[1]: Mounted Kernel Configuration File System.
Aug 28 16:12:07 ctl01 kernel: [    0.138203] ACPI: bus type PCI registered
Aug 28 16:12:07 ctl01 kernel: [    0.140006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Aug 28 16:12:07 ctl01 kernel: [    0.141621] PCI: Using configuration type 1 for base access
Aug 28 16:12:07 ctl01 kernel: [    0.142942] core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off
Aug 28 16:12:07 ctl01 kernel: [    0.145374] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
Aug 28 16:12:07 ctl01 systemd[1]: Mounted FUSE Control File System.
Aug 28 16:12:07 ctl01 kernel: [    0.145548] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
Aug 28 16:12:07 ctl01 systemd[1]: Started Create Static Device Nodes in /dev.
Aug 28 16:12:07 ctl01 kernel: [    0.148127] ACPI: Added _OSI(Module Device)
Aug 28 16:12:07 ctl01 kernel: [    0.149145] ACPI: Added _OSI(Processor Device)
Aug 28 16:12:07 ctl01 kernel: [    0.150220] ACPI: Added _OSI(3.0 _SCP Extensions)
Aug 28 16:12:07 ctl01 kernel: [    0.151335] ACPI: Added _OSI(Processor Aggregator Device)
Aug 28 16:12:07 ctl01 kernel: [    0.152007] ACPI: Added _OSI(Linux-Dell-Video)
Aug 28 16:12:07 ctl01 systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Aug 28 16:12:07 ctl01 kernel: [    0.153071] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
Aug 28 16:12:07 ctl01 kernel: [    0.154305] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
Aug 28 16:12:07 ctl01 kernel: [    0.157215] ACPI: Interpreter enabled
Aug 28 16:12:07 ctl01 kernel: [    0.158150] ACPI: (supports S0 S5)
Aug 28 16:12:07 ctl01 kernel: [    0.159004] ACPI: Using IOAPIC for interrupt routing
Aug 28 16:12:07 ctl01 kernel: [    0.160023] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Aug 28 16:12:07 ctl01 kernel: [    0.162700] ACPI: Enabled 16 GPEs in block 00 to 0F
Aug 28 16:12:07 ctl01 kernel: [    0.167813] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Local File Systems (Pre).
Aug 28 16:12:07 ctl01 systemd[1]: Starting udev Kernel Device Manager...
Aug 28 16:12:07 ctl01 systemd[1]: Started Apply Kernel Variables.
Aug 28 16:12:07 ctl01 systemd[1]: Started Flush Journal to Persistent Storage.
Aug 28 16:12:07 ctl01 systemd[1]: Started udev Kernel Device Manager.
Aug 28 16:12:07 ctl01 systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Local Encrypted Volumes.
Aug 28 16:12:07 ctl01 systemd[1]: Found device /dev/ttyS0.
Aug 28 16:12:07 ctl01 systemd-udevd[502]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:12:07 ctl01 systemd-udevd[508]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:12:07 ctl01 systemd-udevd[503]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:12:07 ctl01 systemd-udevd[497]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:12:07 ctl01 systemd-udevd[501]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:12:07 ctl01 systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Aug 28 16:12:07 ctl01 kernel: [    0.168012] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
Aug 28 16:12:07 ctl01 kernel: [    0.169576] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
Aug 28 16:12:07 ctl01 systemd-udevd[489]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:12:07 ctl01 systemd-udevd[489]: Could not generate persistent MAC address for br-mgmt: No such file or directory
Aug 28 16:12:07 ctl01 systemd[1]: Found device Virtio network device.
Aug 28 16:12:07 ctl01 systemd[1]: Found device Virtio network device.
Aug 28 16:12:07 ctl01 systemd[1]: Found device /sys/subsystem/net/devices/br-mgmt.
Aug 28 16:12:07 ctl01 systemd[1]: Found device Virtio network device.
Aug 28 16:12:07 ctl01 systemd[1]: Found device /dev/disk/by-label/UEFI.
Aug 28 16:12:07 ctl01 systemd[1]: Mounting /boot/efi...
Aug 28 16:12:07 ctl01 systemd[1]: Mounted /boot/efi.
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Local File Systems.
Aug 28 16:12:07 ctl01 systemd[1]: Starting Set console font and keymap...
Aug 28 16:12:07 ctl01 kernel: [    0.171089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
Aug 28 16:12:07 ctl01 systemd[1]: Starting AppArmor initialization...
Aug 28 16:12:07 ctl01 kernel: [    0.172383] acpiphp: Slot [3] registered
Aug 28 16:12:07 ctl01 kernel: [    0.173388] acpiphp: Slot [4] registered
Aug 28 16:12:07 ctl01 systemd[1]: Starting Create Volatile Files and Directories...
Aug 28 16:12:07 ctl01 systemd[1]: Starting ebtables ruleset management...
Aug 28 16:12:07 ctl01 systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
Aug 28 16:12:07 ctl01 kernel: [    0.174409] acpiphp: Slot [5] registered
Aug 28 16:12:07 ctl01 systemd[1]: Started Set console font and keymap.
Aug 28 16:12:07 ctl01 kernel: [    0.175421] acpiphp: Slot [6] registered
Aug 28 16:12:07 ctl01 kernel: [    0.176051] acpiphp: Slot [7] registered
Aug 28 16:12:07 ctl01 kernel: [    0.177074] acpiphp: Slot [9] registered
Aug 28 16:12:07 ctl01 kernel: [    0.178087] acpiphp: Slot [10] registered
Aug 28 16:12:07 ctl01 systemd[1]: Started Tell Plymouth To Write Out Runtime Data.
Aug 28 16:12:07 ctl01 apparmor[927]:  * Starting AppArmor profiles
Aug 28 16:12:07 ctl01 kernel: [    0.179109] acpiphp: Slot [11] registered
Aug 28 16:12:07 ctl01 kernel: [    0.180060] acpiphp: Slot [12] registered
Aug 28 16:12:07 ctl01 systemd[1]: Started Create Volatile Files and Directories.
Aug 28 16:12:07 ctl01 kernel: [    0.181083] acpiphp: Slot [13] registered
Aug 28 16:12:07 ctl01 systemd[1]: Starting Network Time Synchronization...
Aug 28 16:12:07 ctl01 kernel: [    0.182147] acpiphp: Slot [14] registered
Aug 28 16:12:07 ctl01 kernel: [    0.183171] acpiphp: Slot [15] registered
Aug 28 16:12:07 ctl01 kernel: [    0.184050] acpiphp: Slot [16] registered
Aug 28 16:12:07 ctl01 kernel: [    0.185082] acpiphp: Slot [17] registered
Aug 28 16:12:07 ctl01 systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Aug 28 16:12:07 ctl01 kernel: [    0.186115] acpiphp: Slot [18] registered
Aug 28 16:12:07 ctl01 kernel: [    0.187137] acpiphp: Slot [19] registered
Aug 28 16:12:07 ctl01 systemd[1]: Started Update UTMP about System Boot/Shutdown.
Aug 28 16:12:07 ctl01 kernel: [    0.188051] acpiphp: Slot [20] registered
Aug 28 16:12:07 ctl01 kernel: [    0.189093] acpiphp: Slot [21] registered
Aug 28 16:12:07 ctl01 kernel: [    0.190117] acpiphp: Slot [22] registered
Aug 28 16:12:07 ctl01 systemd[1]: Started ebtables ruleset management.
Aug 28 16:12:07 ctl01 apparmor[927]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Aug 28 16:12:07 ctl01 kernel: [    0.191137] acpiphp: Slot [23] registered
Aug 28 16:12:07 ctl01 kernel: [    0.192055] acpiphp: Slot [24] registered
Aug 28 16:12:07 ctl01 kernel: [    0.193094] acpiphp: Slot [25] registered
Aug 28 16:12:07 ctl01 kernel: [    0.194123] acpiphp: Slot [26] registered
Aug 28 16:12:07 ctl01 apparmor[927]:    ...done.
Aug 28 16:12:07 ctl01 kernel: [    0.195140] acpiphp: Slot [27] registered
Aug 28 16:12:07 ctl01 systemd[1]: Started AppArmor initialization.
Aug 28 16:12:07 ctl01 kernel: [    0.196050] acpiphp: Slot [28] registered
Aug 28 16:12:07 ctl01 kernel: [    0.197097] acpiphp: Slot [29] registered
Aug 28 16:12:07 ctl01 kernel: [    0.198139] acpiphp: Slot [30] registered
Aug 28 16:12:07 ctl01 kernel: [    0.199159] acpiphp: Slot [31] registered
Aug 28 16:12:07 ctl01 kernel: [    0.200020] PCI host bridge to bus 0000:00
Aug 28 16:12:07 ctl01 systemd[1]: Starting Initial cloud-init job (pre-networking)...
Aug 28 16:12:07 ctl01 kernel: [    0.201036] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
Aug 28 16:12:07 ctl01 kernel: [    0.202563] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
Aug 28 16:12:07 ctl01 systemd[1]: Started Network Time Synchronization.
Aug 28 16:12:07 ctl01 kernel: [    0.204005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
Aug 28 16:12:07 ctl01 kernel: [    0.205745] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
Aug 28 16:12:07 ctl01 kernel: [    0.207479] pci_bus 0000:00: root bus resource [bus 00-ff]
Aug 28 16:12:07 ctl01 kernel: [    0.208055] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
Aug 28 16:12:07 ctl01 kernel: [    0.208636] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
Aug 28 16:12:07 ctl01 systemd[1]: Reached target System Time Synchronized.
Aug 28 16:12:07 ctl01 kernel: [    0.209415] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
Aug 28 16:12:07 ctl01 kernel: [    0.217013] pci 0000:00:01.1: reg 0x20: [io  0xc140-0xc14f]
Aug 28 16:12:07 ctl01 kernel: [    0.220693] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
Aug 28 16:12:07 ctl01 cloud-init[1074]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init-local' at Wed, 28 Aug 2019 16:12:03 +0000. Up 11.08 seconds.
Aug 28 16:12:07 ctl01 kernel: [    0.222325] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
Aug 28 16:12:07 ctl01 kernel: [    0.223818] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
Aug 28 16:12:07 ctl01 kernel: [    0.224006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
Aug 28 16:12:07 ctl01 kernel: [    0.225744] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
Aug 28 16:12:07 ctl01 kernel: [    0.226348] pci 0000:00:01.3: quirk: [io  0x0600-0x063f] claimed by PIIX4 ACPI
Aug 28 16:12:07 ctl01 systemd[1]: Started Initial cloud-init job (pre-networking).
Aug 28 16:12:07 ctl01 kernel: [    0.228017] pci 0000:00:01.3: quirk: [io  0x0700-0x070f] claimed by PIIX4 SMB
Aug 28 16:12:07 ctl01 kernel: [    0.230058] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000
Aug 28 16:12:07 ctl01 kernel: [    0.232007] pci 0000:00:02.0: reg 0x10: [mem 0xfc000000-0xfdffffff pref]
Aug 28 16:12:07 ctl01 kernel: [    0.235001] pci 0000:00:02.0: reg 0x14: [mem 0xfebd0000-0xfebd0fff]
Aug 28 16:12:07 ctl01 kernel: [    0.249378] pci 0000:00:02.0: reg 0x30: [mem 0xfebc0000-0xfebcffff pref]
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Network (Pre).
Aug 28 16:12:07 ctl01 kernel: [    0.249696] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:12:07 ctl01 kernel: [    0.253140] pci 0000:00:03.0: reg 0x10: [io  0xc040-0xc05f]
Aug 28 16:12:07 ctl01 systemd[1]: Starting Open vSwitch Database Unit...
Aug 28 16:12:07 ctl01 systemd[1]: Started ifup for ens4.
Aug 28 16:12:07 ctl01 kernel: [    0.255426] pci 0000:00:03.0: reg 0x14: [mem 0xfebd1000-0xfebd1fff]
Aug 28 16:12:07 ctl01 systemd[1]: Started ifup for ens5.
Aug 28 16:12:07 ctl01 kernel: [    0.267518] pci 0000:00:03.0: reg 0x30: [mem 0xfeac0000-0xfeafffff pref]
Aug 28 16:12:07 ctl01 kernel: [    0.267963] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:12:07 ctl01 kernel: [    0.270186] pci 0000:00:04.0: reg 0x10: [io  0xc060-0xc07f]
Aug 28 16:12:07 ctl01 kernel: [    0.272038] pci 0000:00:04.0: reg 0x14: [mem 0xfebd2000-0xfebd2fff]
Aug 28 16:12:07 ctl01 systemd[1]: Started ifup for ens3.
Aug 28 16:12:07 ctl01 kernel: [    0.284006] pci 0000:00:04.0: reg 0x30: [mem 0xfeb00000-0xfeb3ffff pref]
Aug 28 16:12:07 ctl01 systemd[1]: Started ifup for br-mgmt.
Aug 28 16:12:07 ctl01 kernel: [    0.286238] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:12:07 ctl01 kernel: [    0.288007] pci 0000:00:05.0: reg 0x10: [io  0xc080-0xc09f]
Aug 28 16:12:07 ctl01 kernel: [    0.290071] pci 0000:00:05.0: reg 0x14: [mem 0xfebd3000-0xfebd3fff]
Aug 28 16:12:07 ctl01 kernel: [    0.303380] pci 0000:00:05.0: reg 0x30: [mem 0xfeb40000-0xfeb7ffff pref]
Aug 28 16:12:07 ctl01 kernel: [    0.303847] pci 0000:00:06.0: [1af4:1000] type 00 class 0x020000
Aug 28 16:12:07 ctl01 sh[1171]: Waiting for br-mgmt to get ready (MAXWAIT is 32 seconds).
Aug 28 16:12:07 ctl01 kernel: [    0.306530] pci 0000:00:06.0: reg 0x10: [io  0xc0a0-0xc0bf]
Aug 28 16:12:07 ctl01 kernel: [    0.308007] pci 0000:00:06.0: reg 0x14: [mem 0xfebd4000-0xfebd4fff]
Aug 28 16:12:07 ctl01 kernel: [    0.321251] pci 0000:00:06.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref]
Aug 28 16:12:07 ctl01 kernel: [    0.321761] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000
Aug 28 16:12:07 ctl01 kernel: [    0.324007] pci 0000:00:07.0: reg 0x10: [io  0xc000-0xc03f]
Aug 28 16:12:07 ctl01 sh[1154]: WARNING:  Could not open /proc/net/vlan/config.  Maybe you need to load the 8021q module, or maybe you are not using PROCFS??
Aug 28 16:12:07 ctl01 sh[1154]: Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Aug 28 16:12:07 ctl01 kernel: [    0.326661] pci 0000:00:07.0: reg 0x14: [mem 0xfebd5000-0xfebd5fff]
Aug 28 16:12:07 ctl01 sh[1154]: Added VLAN with VID == 1000 to IF -:ens5:-
Aug 28 16:12:07 ctl01 kernel: [    0.337766] pci 0000:00:08.0: [8086:2934] type 00 class 0x0c0300
Aug 28 16:12:07 ctl01 systemd-udevd[1429]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:12:07 ctl01 kernel: [    0.344614] pci 0000:00:08.0: reg 0x20: [io  0xc0c0-0xc0df]
Aug 28 16:12:07 ctl01 kernel: [    0.347288] pci 0000:00:08.1: [8086:2935] type 00 class 0x0c0300
Aug 28 16:12:07 ctl01 kernel: [    0.354393] pci 0000:00:08.1: reg 0x20: [io  0xc0e0-0xc0ff]
Aug 28 16:12:07 ctl01 kernel: [    0.356864] pci 0000:00:08.2: [8086:2936] type 00 class 0x0c0300
Aug 28 16:12:07 ctl01 kernel: [    0.364009] pci 0000:00:08.2: reg 0x20: [io  0xc100-0xc11f]
Aug 28 16:12:07 ctl01 ovs-ctl[1143]:  * Starting ovsdb-server
Aug 28 16:12:07 ctl01 kernel: [    0.366812] pci 0000:00:08.7: [8086:293a] type 00 class 0x0c0320
Aug 28 16:12:07 ctl01 kernel: [    0.368007] pci 0000:00:08.7: reg 0x10: [mem 0xfebd6000-0xfebd6fff]
Aug 28 16:12:07 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.16.1
Aug 28 16:12:07 ctl01 kernel: [    0.376925] pci 0000:00:09.0: [1af4:1002] type 00 class 0x00ff00
Aug 28 16:12:07 ctl01 kernel: [    0.378218] pci 0000:00:09.0: reg 0x10: [io  0xc120-0xc13f]
Aug 28 16:12:07 ctl01 kernel: [    0.390456] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
Aug 28 16:12:07 ctl01 systemd[1]: Found device /sys/subsystem/net/devices/ens5.1000.
Aug 28 16:12:07 ctl01 systemd[1]: Started ifup for ens5.1000.
Aug 28 16:12:07 ctl01 kernel: [    0.392059] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
Aug 28 16:12:07 ctl01 kernel: [    0.393603] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
Aug 28 16:12:07 ctl01 kernel: [    0.395152] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
Aug 28 16:12:07 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.11.1 "external-ids:system-id=\"54f19e88-f36a-46eb-9077-8d0f01b13a75\"" "external-ids:rundir=\"/var/run/openvswitch\"" "system-type=\"ubuntu\"" "system-version=\"18.04\""
Aug 28 16:12:07 ctl01 kernel: [    0.396081] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
Aug 28 16:12:07 ctl01 kernel: [    0.398439] SCSI subsystem initialized
Aug 28 16:12:07 ctl01 sh[1551]: Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Aug 28 16:12:07 ctl01 kernel: [    0.399532] libata version 3.00 loaded.
Aug 28 16:12:07 ctl01 kernel: [    0.399532] pci 0000:00:02.0: vgaarb: setting as boot VGA device
Aug 28 16:12:07 ctl01 kernel: [    0.399532] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
Aug 28 16:12:07 ctl01 kernel: [    0.400008] pci 0000:00:02.0: vgaarb: bridge control possible
Aug 28 16:12:07 ctl01 kernel: [    0.401429] vgaarb: loaded
Aug 28 16:12:07 ctl01 ovs-ctl[1143]:  * Configuring Open vSwitch system IDs
Aug 28 16:12:07 ctl01 kernel: [    0.402216] ACPI: bus type USB registered
Aug 28 16:12:07 ctl01 kernel: [    0.403284] usbcore: registered new interface driver usbfs
Aug 28 16:12:07 ctl01 kernel: [    0.404015] usbcore: registered new interface driver hub
Aug 28 16:12:07 ctl01 kernel: [    0.405382] usbcore: registered new device driver usb
Aug 28 16:12:07 ctl01 kernel: [    0.406731] EDAC MC: Ver: 3.0.0
Aug 28 16:12:07 ctl01 kernel: [    0.408092] PCI: Using ACPI for IRQ routing
Aug 28 16:12:07 ctl01 ovs-ctl[1143]:  * Enabling remote OVSDB managers
Aug 28 16:12:07 ctl01 kernel: [    0.409129] PCI: pci_cache_line_size set to 64 bytes
Aug 28 16:12:07 ctl01 kernel: [    0.409478] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
Aug 28 16:12:07 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . external-ids:hostname=ctl01
Aug 28 16:12:07 ctl01 kernel: [    0.409480] e820: reserve RAM buffer [mem 0xbffdf000-0xbfffffff]
Aug 28 16:12:07 ctl01 kernel: [    0.409580] NetLabel: Initializing
Aug 28 16:12:07 ctl01 kernel: [    0.410513] NetLabel:  domain hash size = 128
Aug 28 16:12:07 ctl01 kernel: [    0.411659] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
Aug 28 16:12:07 ctl01 systemd[1]: Started Open vSwitch Database Unit.
Aug 28 16:12:07 ctl01 systemd[1]: Starting Open vSwitch Forwarding Unit...
Aug 28 16:12:07 ctl01 kernel: [    0.412021] NetLabel:  unlabeled traffic allowed by default
Aug 28 16:12:07 ctl01 ovs-ctl[1601]:  * Inserting openvswitch module
Aug 28 16:12:07 ctl01 kernel: [    0.413449] clocksource: Switched to clocksource kvm-clock
Aug 28 16:12:07 ctl01 ovs-ctl[1601]:  * Starting ovs-vswitchd
Aug 28 16:12:07 ctl01 kernel: [    0.425747] VFS: Disk quotas dquot_6.6.0
Aug 28 16:12:07 ctl01 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . external-ids:hostname=ctl01
Aug 28 16:12:07 ctl01 kernel: [    0.426817] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.428634] AppArmor: AppArmor Filesystem Enabled
Aug 28 16:12:07 ctl01 kernel: [    0.430265] pnp: PnP ACPI init
Aug 28 16:12:07 ctl01 ovs-ctl[1601]:  * Enabling remote OVSDB managers
Aug 28 16:12:07 ctl01 systemd[1]: Started Open vSwitch Forwarding Unit.
Aug 28 16:12:07 ctl01 kernel: [    0.431208] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
Aug 28 16:12:07 ctl01 systemd[1]: Starting Raise network interfaces...
Aug 28 16:12:07 ctl01 kernel: [    0.431254] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
Aug 28 16:12:07 ctl01 systemd[1]: Started Raise network interfaces.
Aug 28 16:12:07 ctl01 kernel: [    0.431289] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active)
Aug 28 16:12:07 ctl01 kernel: [    0.431326] pnp 00:03: [dma 2]
Aug 28 16:12:07 ctl01 kernel: [    0.431343] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active)
Aug 28 16:12:07 ctl01 kernel: [    0.431450] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
Aug 28 16:12:07 ctl01 kernel: [    0.431768] pnp: PnP ACPI: found 5 devices
Aug 28 16:12:07 ctl01 systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
Aug 28 16:12:07 ctl01 kernel: [    0.439709] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
Aug 28 16:12:07 ctl01 cloud-init[1893]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init' at Wed, 28 Aug 2019 16:12:04 +0000. Up 12.42 seconds.
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: ++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
Aug 28 16:12:07 ctl01 kernel: [    0.442010] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
Aug 28 16:12:07 ctl01 kernel: [    0.442012] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-----------+-------+----------------------------+---------------+--------+-------------------+
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   Device  |   Up  |          Address           |      Mask     | Scope  |     Hw-Address    |
Aug 28 16:12:07 ctl01 kernel: [    0.442013] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-----------+-------+----------------------------+---------------+--------+-------------------+
Aug 28 16:12:07 ctl01 kernel: [    0.442015] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |  br-mgmt  |  True |        172.16.10.36        | 255.255.255.0 | global | 52:54:00:2c:cd:bc |
Aug 28 16:12:07 ctl01 kernel: [    0.442089] NET: Registered protocol family 2
Aug 28 16:12:07 ctl01 kernel: [    0.444131] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |  br-mgmt  |  True | fe80::5054:ff:fe2c:cdbc/64 |       .       |  link  | 52:54:00:2c:cd:bc |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |    ens3   |  True |       192.168.11.21        | 255.255.255.0 | global | 52:54:00:3f:9b:0c |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |    ens3   |  True | fe80::5054:ff:fe3f:9b0c/64 |       .       |  link  | 52:54:00:3f:9b:0c |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |    ens4   |  True | fe80::5054:ff:fe2c:cdbc/64 |       .       |  link  | 52:54:00:2c:cd:bc |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |    ens5   |  True | fe80::5054:ff:fe9e:c5f6/64 |       .       |  link  | 52:54:00:9e:c5:f6 |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: | ens5.1000 |  True | fe80::5054:ff:fe9e:c5f6/64 |       .       |  link  | 52:54:00:9e:c5:f6 |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |    ens6   | False |             .              |       .       |   .    | 52:54:00:14:c4:1a |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |     lo    |  True |         127.0.0.1          |   255.0.0.0   |  host  |         .         |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |     lo    |  True |          ::1/128           |       .       |  host  |         .         |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-----------+-------+----------------------------+---------------+--------+-------------------+
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   0   |   0.0.0.0    | 192.168.11.3 |    0.0.0.0    |    ens3   |   UG  |
Aug 28 16:12:07 ctl01 kernel: [    0.446277] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.448633] TCP: Hash tables configured (established 131072 bind 65536)
Aug 28 16:12:07 ctl01 kernel: [    0.450385] UDP hash table entries: 8192 (order: 6, 262144 bytes)
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   1   |  10.254.0.0  | 172.16.10.36 |  255.255.0.0  |  br-mgmt  |   UG  |
Aug 28 16:12:07 ctl01 kernel: [    0.452013] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
Aug 28 16:12:07 ctl01 kernel: [    0.453765] NET: Registered protocol family 1
Aug 28 16:12:07 ctl01 kernel: [    0.454963] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
Aug 28 16:12:07 ctl01 kernel: [    0.456524] pci 0000:00:01.0: PIIX3: Enabling Passive Release
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   2   | 172.16.10.0  |   0.0.0.0    | 255.255.255.0 |  br-mgmt  |   U   |
Aug 28 16:12:07 ctl01 kernel: [    0.458047] pci 0000:00:01.0: Activating ISA DMA hang workarounds
Aug 28 16:12:07 ctl01 kernel: [    0.459690] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
Aug 28 16:12:07 ctl01 kernel: [    0.495609] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   3   | 192.168.11.0 |   0.0.0.0    | 255.255.255.0 |    ens3   |   U   |
Aug 28 16:12:07 ctl01 kernel: [    0.564247] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
Aug 28 16:12:07 ctl01 kernel: [    0.629079] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
Aug 28 16:12:07 ctl01 kernel: [    0.691924] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
Aug 28 16:12:07 ctl01 kernel: [    0.723697] PCI: CLS 0 bytes, default 64
Aug 28 16:12:07 ctl01 kernel: [    0.723755] Unpacking initramfs...
Aug 28 16:12:07 ctl01 kernel: [    0.999136] Freeing initrd memory: 19144K
Aug 28 16:12:07 ctl01 kernel: [    1.000215] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
Aug 28 16:12:07 ctl01 kernel: [    1.001710] software IO TLB: mapped [mem 0xbbfdf000-0xbffdf000] (64MB)
Aug 28 16:12:07 ctl01 kernel: [    1.003261] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x285c3aeaff3, max_idle_ns: 440795255742 ns
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-------+-------------+---------+-----------+-------+
Aug 28 16:12:07 ctl01 kernel: [    1.005588] Scanning for low memory corruption every 60 seconds
Aug 28 16:12:07 ctl01 kernel: [    1.007771] Initialise system trusted keyrings
Aug 28 16:12:07 ctl01 kernel: [    1.008883] Key type blacklist registered
Aug 28 16:12:07 ctl01 kernel: [    1.009958] workingset: timestamp_bits=36 max_order=22 bucket_order=0
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: | Route | Destination | Gateway | Interface | Flags |
Aug 28 16:12:07 ctl01 kernel: [    1.012590] zbud: loaded
Aug 28 16:12:07 ctl01 kernel: [    1.013908] squashfs: version 4.0 (2009/01/31) Phillip Lougher
Aug 28 16:12:07 ctl01 kernel: [    1.015460] fuse init (API version 7.26)
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-------+-------------+---------+-----------+-------+
Aug 28 16:12:07 ctl01 kernel: [    1.018938] Key type asymmetric registered
Aug 28 16:12:07 ctl01 kernel: [    1.019950] Asymmetric key parser 'x509' registered
Aug 28 16:12:07 ctl01 kernel: [    1.021152] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
Aug 28 16:12:07 ctl01 kernel: [    1.022979] io scheduler noop registered
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   1   |  fe80::/64  |    ::   |    ens4   |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.023954] io scheduler deadline registered
Aug 28 16:12:07 ctl01 kernel: [    1.025032] io scheduler cfq registered (default)
Aug 28 16:12:07 ctl01 kernel: [    1.026692] intel_idle: Please enable MWAIT in BIOS SETUP
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   2   |  fe80::/64  |    ::   |  br-mgmt  |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.026766] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Aug 28 16:12:07 ctl01 kernel: [    1.028599] ACPI: Power Button [PWRF]
Aug 28 16:12:07 ctl01 kernel: [    1.060422] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
Aug 28 16:12:07 ctl01 kernel: [    1.092768] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
Aug 28 16:12:07 ctl01 kernel: [    1.124981] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
Aug 28 16:12:07 ctl01 kernel: [    1.157241] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   3   |  fe80::/64  |    ::   |    ens3   |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.189911] virtio-pci 0000:00:07.0: virtio_pci: leaving for legacy driver
Aug 28 16:12:07 ctl01 kernel: [    1.221610] virtio-pci 0000:00:09.0: virtio_pci: leaving for legacy driver
Aug 28 16:12:07 ctl01 kernel: [    1.224412] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
Aug 28 16:12:07 ctl01 kernel: [    1.249745] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
Aug 28 16:12:07 ctl01 kernel: [    1.253362] Linux agpgart interface v0.103
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   4   |  fe80::/64  |    ::   |    ens5   |   U   |
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   5   |  fe80::/64  |    ::   | ens5.1000 |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.257220] loop: module loaded
Aug 28 16:12:07 ctl01 kernel: [    1.258138] ata_piix 0000:00:01.1: version 2.13
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   7   |    local    |    ::   |    ens4   |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.259440] scsi host0: ata_piix
Aug 28 16:12:07 ctl01 kernel: [    1.260460] scsi host1: ata_piix
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   8   |   ff00::/8  |    ::   |    ens4   |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.261330] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc140 irq 14
Aug 28 16:12:07 ctl01 kernel: [    1.262892] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc148 irq 15
Aug 28 16:12:07 ctl01 kernel: [    1.264544] libphy: Fixed MDIO Bus: probed
Aug 28 16:12:07 ctl01 kernel: [    1.265633] tun: Universal TUN/TAP device driver, 1.6
Aug 28 16:12:07 ctl01 kernel: [    1.266911] PPP generic driver version 2.4.2
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   9   |   ff00::/8  |    ::   |  br-mgmt  |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.276707] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Aug 28 16:12:07 ctl01 kernel: [    1.278223] ehci-pci: EHCI PCI platform driver
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   10  |   ff00::/8  |    ::   |    ens3   |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.310556] ehci-pci 0000:00:08.7: EHCI Host Controller
Aug 28 16:12:07 ctl01 kernel: [    1.311809] ehci-pci 0000:00:08.7: new USB bus registered, assigned bus number 1
Aug 28 16:12:07 ctl01 kernel: [    1.313780] ehci-pci 0000:00:08.7: irq 11, io mem 0xfebd6000
Aug 28 16:12:07 ctl01 kernel: [    1.328060] ehci-pci 0000:00:08.7: USB 2.0 started, EHCI 1.00
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   11  |   ff00::/8  |    ::   |    ens5   |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.330011] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Aug 28 16:12:07 ctl01 kernel: [    1.332193] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:12:07 ctl01 kernel: [    1.334547] usb usb1: Product: EHCI Host Controller
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: |   12  |   ff00::/8  |    ::   | ens5.1000 |   U   |
Aug 28 16:12:07 ctl01 kernel: [    1.336174] usb usb1: Manufacturer: Linux 4.15.0-58-generic ehci_hcd
Aug 28 16:12:07 ctl01 kernel: [    1.338234] usb usb1: SerialNumber: 0000:00:08.7
Aug 28 16:12:07 ctl01 kernel: [    1.339988] hub 1-0:1.0: USB hub found
Aug 28 16:12:07 ctl01 kernel: [    1.340955] hub 1-0:1.0: 6 ports detected
Aug 28 16:12:07 ctl01 kernel: [    1.342134] ehci-platform: EHCI generic platform driver
Aug 28 16:12:07 ctl01 kernel: [    1.343386] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Aug 28 16:12:07 ctl01 kernel: [    1.344833] ohci-pci: OHCI PCI platform driver
Aug 28 16:12:07 ctl01 cloud-init[1893]: ci-info: +-------+-------------+---------+-----------+-------+
Aug 28 16:12:07 ctl01 kernel: [    1.345908] ohci-platform: OHCI generic platform driver
Aug 28 16:12:07 ctl01 kernel: [    1.347146] uhci_hcd: USB Universal Host Controller Interface driver
Aug 28 16:12:07 ctl01 kernel: [    1.379530] uhci_hcd 0000:00:08.0: UHCI Host Controller
Aug 28 16:12:07 ctl01 systemd[1]: Started Initial cloud-init job (metadata service crawler).
Aug 28 16:12:07 ctl01 kernel: [    1.380780] uhci_hcd 0000:00:08.0: new USB bus registered, assigned bus number 2
Aug 28 16:12:07 ctl01 kernel: [    1.382531] uhci_hcd 0000:00:08.0: detected 2 ports
Aug 28 16:12:07 ctl01 kernel: [    1.383746] uhci_hcd 0000:00:08.0: irq 11, io base 0x0000c0c0
Aug 28 16:12:07 ctl01 kernel: [    1.385176] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
Aug 28 16:12:07 ctl01 systemd[1]: Reached target System Initialization.
Aug 28 16:12:07 ctl01 systemd[1]: Started Daily apt download activities.
Aug 28 16:12:07 ctl01 kernel: [    1.386765] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:12:07 ctl01 kernel: [    1.388481] usb usb2: Product: UHCI Host Controller
Aug 28 16:12:07 ctl01 kernel: [    1.389841] usb usb2: Manufacturer: Linux 4.15.0-58-generic uhci_hcd
Aug 28 16:12:07 ctl01 systemd[1]: Started Daily Cleanup of Temporary Directories.
Aug 28 16:12:07 ctl01 kernel: [    1.391331] usb usb2: SerialNumber: 0000:00:08.0
Aug 28 16:12:07 ctl01 kernel: [    1.392605] hub 2-0:1.0: USB hub found
Aug 28 16:12:07 ctl01 kernel: [    1.393551] hub 2-0:1.0: 2 ports detected
Aug 28 16:12:07 ctl01 kernel: [    1.425663] uhci_hcd 0000:00:08.1: UHCI Host Controller
Aug 28 16:12:07 ctl01 systemd[1]: Started Message of the Day.
Aug 28 16:12:07 ctl01 kernel: [    1.425757] ata1.01: NODEV after polling detection
Aug 28 16:12:07 ctl01 systemd[1]: Listening on ACPID Listen Socket.
Aug 28 16:12:07 ctl01 kernel: [    1.426939] uhci_hcd 0000:00:08.1: new USB bus registered, assigned bus number 3
Aug 28 16:12:07 ctl01 systemd[1]: Starting LXD - unix socket.
Aug 28 16:12:07 ctl01 kernel: [    1.427335] ata1.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
Aug 28 16:12:07 ctl01 kernel: [    1.428756] uhci_hcd 0000:00:08.1: detected 2 ports
Aug 28 16:12:07 ctl01 kernel: [    1.431283] ata1.00: configured for MWDMA2
Aug 28 16:12:07 ctl01 kernel: [    1.432021] uhci_hcd 0000:00:08.1: irq 10, io base 0x0000c0e0
Aug 28 16:12:07 ctl01 kernel: [    1.435093] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
Aug 28 16:12:07 ctl01 systemd[1]: Listening on UUID daemon activation socket.
Aug 28 16:12:07 ctl01 systemd[1]: Listening on D-Bus System Message Bus Socket.
Aug 28 16:12:07 ctl01 kernel: [    1.436831] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:12:07 ctl01 systemd[1]: Starting Socket activation for snappy daemon.
Aug 28 16:12:07 ctl01 kernel: [    1.436890] scsi 0:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
Aug 28 16:12:07 ctl01 systemd[1]: Listening on Open-iSCSI iscsid Socket.
Aug 28 16:12:07 ctl01 kernel: [    1.438702] usb usb3: Product: UHCI Host Controller
Aug 28 16:12:07 ctl01 kernel: [    1.442276] usb usb3: Manufacturer: Linux 4.15.0-58-generic uhci_hcd
Aug 28 16:12:07 ctl01 kernel: [    1.442284] sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
Aug 28 16:12:07 ctl01 kernel: [    1.443861] usb usb3: SerialNumber: 0000:00:08.1
Aug 28 16:12:07 ctl01 kernel: [    1.445578] cdrom: Uniform CD-ROM driver Revision: 3.20
Aug 28 16:12:07 ctl01 systemd[1]: Started ACPI Events Check.
Aug 28 16:12:07 ctl01 kernel: [    1.448208] hub 3-0:1.0: USB hub found
Aug 28 16:12:07 ctl01 kernel: [    1.448247] sr 0:0:0:0: Attached scsi CD-ROM sr0
Aug 28 16:12:07 ctl01 kernel: [    1.449288] hub 3-0:1.0: 2 ports detected
Aug 28 16:12:07 ctl01 kernel: [    1.449340] sr 0:0:0:0: Attached scsi generic sg0 type 5
Aug 28 16:12:07 ctl01 kernel: [    1.483142] uhci_hcd 0000:00:08.2: UHCI Host Controller
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Paths.
Aug 28 16:12:07 ctl01 kernel: [    1.484516] uhci_hcd 0000:00:08.2: new USB bus registered, assigned bus number 4
Aug 28 16:12:07 ctl01 kernel: [    1.486424] uhci_hcd 0000:00:08.2: detected 2 ports
Aug 28 16:12:07 ctl01 kernel: [    1.487752] uhci_hcd 0000:00:08.2: irq 10, io base 0x0000c100
Aug 28 16:12:07 ctl01 systemd[1]: Started Daily apt upgrade and clean activities.
Aug 28 16:12:07 ctl01 kernel: [    1.489304] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
Aug 28 16:12:07 ctl01 kernel: [    1.490999] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 28 16:12:07 ctl01 kernel: [    1.493064] usb usb4: Product: UHCI Host Controller
Aug 28 16:12:07 ctl01 kernel: [    1.494328] usb usb4: Manufacturer: Linux 4.15.0-58-generic uhci_hcd
Aug 28 16:12:07 ctl01 kernel: [    1.495922] usb usb4: SerialNumber: 0000:00:08.2
Aug 28 16:12:07 ctl01 systemd[1]: Started Discard unused blocks once a week.
Aug 28 16:12:07 ctl01 kernel: [    1.497298] hub 4-0:1.0: USB hub found
Aug 28 16:12:07 ctl01 kernel: [    1.498342] hub 4-0:1.0: 2 ports detected
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Timers.
Aug 28 16:12:07 ctl01 kernel: [    1.499576] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
Aug 28 16:12:07 ctl01 kernel: [    1.502647] serio: i8042 KBD port at 0x60,0x64 irq 1
Aug 28 16:12:07 ctl01 kernel: [    1.503965] serio: i8042 AUX port at 0x60,0x64 irq 12
Aug 28 16:12:07 ctl01 kernel: [    1.505451] mousedev: PS/2 mouse device common for all mice
Aug 28 16:12:07 ctl01 kernel: [    1.507157] rtc_cmos 00:00: RTC can wake from S4
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Cloud-config availability.
Aug 28 16:12:07 ctl01 kernel: [    1.508975] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
Aug 28 16:12:07 ctl01 systemd[1]: Listening on LXD - unix socket.
Aug 28 16:12:07 ctl01 kernel: [    1.511176] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
Aug 28 16:12:07 ctl01 kernel: [    1.512909] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
Aug 28 16:12:07 ctl01 kernel: [    1.514584] i2c /dev entries driver
Aug 28 16:12:07 ctl01 kernel: [    1.515590] device-mapper: uevent: version 1.0.3
Aug 28 16:12:07 ctl01 systemd[1]: Listening on Socket activation for snappy daemon.
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Sockets.
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Basic System.
Aug 28 16:12:07 ctl01 systemd[1]: Started Self Monitoring and Reporting Technology (SMART) Daemon.
Aug 28 16:12:07 ctl01 systemd[1]: Starting System Logging Service...
Aug 28 16:12:07 ctl01 systemd[1]: Started Deferred execution scheduler.
Aug 28 16:12:07 ctl01 systemd[1]: Starting LXD - container startup/shutdown...
Aug 28 16:12:07 ctl01 systemd[1]: Starting Snappy daemon...
Aug 28 16:12:07 ctl01 systemd[1]: Starting LSB: Record successful boot for GRUB...
Aug 28 16:12:07 ctl01 systemd[1]: Starting Open vSwitch...
Aug 28 16:12:07 ctl01 systemd[1]: Starting Accounts Service...
Aug 28 16:12:07 ctl01 systemd[1]: Starting LSB: Set sysfs variables from /etc/sysfs.conf...
Aug 28 16:12:07 ctl01 systemd[1]: Started D-Bus System Message Bus.
Aug 28 16:12:07 ctl01 kernel: [    1.516910] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com
Aug 28 16:12:07 ctl01 kernel: [    1.519420] ledtrig-cpu: registered to indicate activity on CPUs
Aug 28 16:12:07 ctl01 kernel: [    1.521452] NET: Registered protocol family 10
Aug 28 16:12:07 ctl01 smartd[1971]: smartd 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-58-generic] (local build)
Aug 28 16:12:07 ctl01 smartd[1971]: Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
Aug 28 16:12:07 ctl01 smartd[1971]: Opened configuration file /etc/smartd.conf
Aug 28 16:12:07 ctl01 kernel: [    1.527431] Segment Routing with IPv6
Aug 28 16:12:07 ctl01 smartd[1971]: Drive: DEVICESCAN, implied '-a' Directive on line 21 of file /etc/smartd.conf
Aug 28 16:12:07 ctl01 kernel: [    1.528456] NET: Registered protocol family 17
Aug 28 16:12:07 ctl01 kernel: [    1.529763] Key type dns_resolver registered
Aug 28 16:12:07 ctl01 kernel: [    1.531690] RAS: Correctable Errors collector initialized.
Aug 28 16:12:07 ctl01 kernel: [    1.533111] sched_clock: Marking stable (1533092586, 0)->(2091003710, -557911124)
Aug 28 16:12:07 ctl01 kernel: [    1.535319] registered taskstats version 1
Aug 28 16:12:07 ctl01 smartd[1971]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Aug 28 16:12:07 ctl01 kernel: [    1.536356] Loading compiled-in X.509 certificates
Aug 28 16:12:07 ctl01 kernel: [    1.540238] Loaded X.509 cert 'Build time autogenerated kernel key: e343dd3e31c9820cf0adf0156bc2eecdcfc394af'
Aug 28 16:12:07 ctl01 kernel: [    1.542506] zswap: loaded using pool lzo/zbud
Aug 28 16:12:07 ctl01 kernel: [    1.548369] Key type big_key registered
Aug 28 16:12:07 ctl01 kernel: [    1.549342] Key type trusted registered
Aug 28 16:12:07 ctl01 smartd[1971]: DEVICESCAN failed: glob(3) aborted matching pattern /dev/discs/disc*
Aug 28 16:12:07 ctl01 kernel: [    1.552665] Key type encrypted registered
Aug 28 16:12:07 ctl01 smartd[1971]: In the system's table of devices NO devices found to scan
Aug 28 16:12:07 ctl01 kernel: [    1.553678] AppArmor: AppArmor sha1 policy hashing enabled
Aug 28 16:12:07 ctl01 smartd[1971]: Unable to monitor any SMART enabled devices. Try debug (-d) option. Exiting...
Aug 28 16:12:07 ctl01 kernel: [    1.555432] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
Aug 28 16:12:07 ctl01 kernel: [    1.556899] ima: Allocated hash algorithm: sha1
Aug 28 16:12:07 ctl01 rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.32.0]
Aug 28 16:12:07 ctl01 rsyslogd: rsyslogd's groupid changed to 106
Aug 28 16:12:07 ctl01 kernel: [    1.558045] evm: HMAC attrs: 0x1
Aug 28 16:12:07 ctl01 kernel: [    1.559287]   Magic number: 15:430:187
Aug 28 16:12:07 ctl01 kernel: [    1.560497] rtc_cmos 00:00: setting system clock to 2019-08-28 16:11:53 UTC (1567008713)
Aug 28 16:12:07 ctl01 rsyslogd: rsyslogd's userid changed to 102
Aug 28 16:12:07 ctl01 rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="1977" x-info="http://www.rsyslog.com"] start
Aug 28 16:12:07 ctl01 kernel: [    1.562564] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
Aug 28 16:12:07 ctl01 kernel: [    1.563959] EDD information not available.
Aug 28 16:12:07 ctl01 kernel: [    1.928196] Freeing unused kernel image memory: 2424K
Aug 28 16:12:07 ctl01 kernel: [    1.964202] Write protecting the kernel read-only data: 20480k
Aug 28 16:12:07 ctl01 kernel: [    1.969565] Freeing unused kernel image memory: 2008K
Aug 28 16:12:07 ctl01 kernel: [    1.973040] Freeing unused kernel image memory: 1864K
Aug 28 16:12:07 ctl01 kernel: [    1.983497] x86/mm: Checked W+X mappings: passed, no W+X pages found.
Aug 28 16:12:07 ctl01 kernel: [    1.984989] x86/mm: Checking user space page tables
Aug 28 16:12:07 ctl01 kernel: [    1.993590] x86/mm: Checked W+X mappings: passed, no W+X pages found.
Aug 28 16:12:07 ctl01 kernel: [    2.108199] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
Aug 28 16:12:07 ctl01 kernel: [    2.108271]  vda: vda1 vda14 vda15
Aug 28 16:12:07 ctl01 kernel: [    2.112539] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
Aug 28 16:12:07 ctl01 kernel: [    2.122770] AVX version of gcm_enc/dec engaged.
Aug 28 16:12:07 ctl01 kernel: [    2.123902] AES CTR mode by8 optimization enabled
Aug 28 16:12:07 ctl01 kernel: [    2.124864] FDC 0 is a S82078B
Aug 28 16:12:07 ctl01 kernel: [    2.130366] virtio_net virtio2 ens5: renamed from eth2
Aug 28 16:12:07 ctl01 kernel: [    2.152366] virtio_net virtio1 ens4: renamed from eth1
Aug 28 16:12:07 ctl01 kernel: [    2.180238] virtio_net virtio0 ens3: renamed from eth0
Aug 28 16:12:07 ctl01 kernel: [    2.220497] virtio_net virtio3 ens6: renamed from eth3
Aug 28 16:12:07 ctl01 kernel: [    3.860030] raid6: sse2x1   gen()  7732 MB/s
Aug 28 16:12:07 ctl01 kernel: [    3.908029] raid6: sse2x1   xor()  5912 MB/s
Aug 28 16:12:07 ctl01 kernel: [    3.956027] raid6: sse2x2   gen()  9366 MB/s
Aug 28 16:12:07 ctl01 kernel: [    4.004024] raid6: sse2x2   xor()  6525 MB/s
Aug 28 16:12:07 ctl01 kernel: [    4.052025] raid6: sse2x4   gen() 11610 MB/s
Aug 28 16:12:07 ctl01 kernel: [    4.100028] raid6: sse2x4   xor()  7744 MB/s
Aug 28 16:12:07 ctl01 kernel: [    4.101106] raid6: using algorithm sse2x4 gen() 11610 MB/s
Aug 28 16:12:07 ctl01 kernel: [    4.102404] raid6: .... xor() 7744 MB/s, rmw enabled
Aug 28 16:12:07 ctl01 kernel: [    4.103591] raid6: using ssse3x2 recovery algorithm
Aug 28 16:12:07 ctl01 kernel: [    4.105994] xor: automatically using best checksumming function   avx       
Aug 28 16:12:07 ctl01 kernel: [    4.108807] async_tx: api initialized (async)
Aug 28 16:12:07 ctl01 kernel: [    4.163230] Btrfs loaded, crc32c=crc32c-intel
Aug 28 16:12:07 ctl01 kernel: [    4.364572] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
Aug 28 16:12:07 ctl01 kernel: [    4.893243] random: fast init done
Aug 28 16:12:07 ctl01 kernel: [    5.654135] ip_tables: (C) 2000-2006 Netfilter Core Team
Aug 28 16:12:07 ctl01 kernel: [    5.663380] random: systemd: uninitialized urandom read (16 bytes read)
Aug 28 16:12:07 ctl01 kernel: [    5.668393] systemd[1]: systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Aug 28 16:12:07 ctl01 kernel: [    5.673184] systemd[1]: Detected virtualization kvm.
Aug 28 16:12:07 ctl01 kernel: [    5.674387] systemd[1]: Detected architecture x86-64.
Aug 28 16:12:07 ctl01 kernel: [    5.675633] random: systemd: uninitialized urandom read (16 bytes read)
Aug 28 16:12:07 ctl01 kernel: [    5.677201] random: systemd: uninitialized urandom read (16 bytes read)
Aug 28 16:12:07 ctl01 kernel: [    5.724103] systemd[1]: Set hostname to <ctl01>.
Aug 28 16:12:07 ctl01 kernel: [    6.410655] systemd[1]: Reached target User and Group Name Lookups.
Aug 28 16:12:07 ctl01 kernel: [    6.413616] systemd[1]: Reached target Swap.
Aug 28 16:12:07 ctl01 kernel: [    6.417365] systemd[1]: Created slice User and Session Slice.
Aug 28 16:12:07 ctl01 kernel: [    6.419992] systemd[1]: Created slice System Slice.
Aug 28 16:12:07 ctl01 kernel: [    6.422296] systemd[1]: Created slice system-postfix.slice.
Aug 28 16:12:07 ctl01 kernel: [    6.424665] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
Aug 28 16:12:07 ctl01 kernel: [    6.569817] Loading iSCSI transport class v2.0-870.
Aug 28 16:12:07 ctl01 kernel: [    6.589033] iscsi: registered transport (tcp)
Aug 28 16:12:07 ctl01 kernel: [    6.592421] EXT4-fs (vda1): re-mounted. Opts: (null)
Aug 28 16:12:07 ctl01 kernel: [    6.633112] iscsi: registered transport (iser)
Aug 28 16:12:07 ctl01 kernel: [    6.728264] systemd-journald[458]: Received request to flush runtime journal from PID 1
Aug 28 16:12:07 ctl01 kernel: [    6.733984] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
Aug 28 16:12:07 ctl01 kernel: [    7.480811] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Aug 28 16:12:07 ctl01 kernel: [    7.482860] br-mgmt: port 1(ens4) entered blocking state
Aug 28 16:12:07 ctl01 kernel: [    7.482862] br-mgmt: port 1(ens4) entered disabled state
Aug 28 16:12:07 ctl01 kernel: [    7.482923] device ens4 entered promiscuous mode
Aug 28 16:12:07 ctl01 kernel: [    8.416215] audit: type=1400 audit(1567008720.352:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=1067 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.418494] audit: type=1400 audit(1567008720.352:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=1068 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.418498] audit: type=1400 audit(1567008720.352:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=1068 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.418501] audit: type=1400 audit(1567008720.352:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=1068 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.422278] audit: type=1400 audit(1567008720.356:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=1069 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.422283] audit: type=1400 audit(1567008720.356:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=1069 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.423694] audit: type=1400 audit(1567008720.356:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=1071 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.423807] audit: type=1400 audit(1567008720.356:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=1065 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.423810] audit: type=1400 audit(1567008720.356:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=1065 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [    8.423813] audit: type=1400 audit(1567008720.356:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=1065 comm="apparmor_parser"
Aug 28 16:12:07 ctl01 kernel: [   11.252824] ISO 9660 Extensions: Microsoft Joliet Level 3
Aug 28 16:12:07 ctl01 kernel: [   11.256406] ISO 9660 Extensions: RRIP_1991A
Aug 28 16:12:07 ctl01 kernel: [   11.485230] br-mgmt: port 1(ens4) entered blocking state
Aug 28 16:12:07 ctl01 kernel: [   11.485232] br-mgmt: port 1(ens4) entered forwarding state
Aug 28 16:12:07 ctl01 kernel: [   11.485271] IPv6: ADDRCONF(NETDEV_UP): br-mgmt: link is not ready
Aug 28 16:12:07 ctl01 kernel: [   11.485304] IPv6: ADDRCONF(NETDEV_CHANGE): br-mgmt: link becomes ready
Aug 28 16:12:07 ctl01 kernel: [   11.577723] 8021q: 802.1Q VLAN Support v1.8
Aug 28 16:12:07 ctl01 kernel: [   11.577731] 8021q: adding VLAN 0 to HW filter on device ens3
Aug 28 16:12:07 ctl01 kernel: [   11.577796] 8021q: adding VLAN 0 to HW filter on device ens4
Aug 28 16:12:07 ctl01 kernel: [   11.577819] 8021q: adding VLAN 0 to HW filter on device ens5
Aug 28 16:12:07 ctl01 kernel: [   11.726628] openvswitch: Open vSwitch switching datapath
Aug 28 16:12:07 ctl01 dbus-daemon[2024]: [system] AppArmor D-Bus mediation is enabled
Aug 28 16:12:07 ctl01 systemd[1]: Starting Login Service...
Aug 28 16:12:07 ctl01 systemd[1]: Started Regular background program processing daemon.
Aug 28 16:12:07 ctl01 systemd[1]: Started FUSE filesystem for LXC.
Aug 28 16:12:07 ctl01 cron[2068]: (CRON) INFO (pidfile fd = 3)
Aug 28 16:12:07 ctl01 systemd[1]: Started irqbalance daemon.
Aug 28 16:12:07 ctl01 systemd[1]: Started System Logging Service.
Aug 28 16:12:07 ctl01 systemd[1]: smartd.service: Main process exited, code=exited, status=17/n/a
Aug 28 16:12:07 ctl01 systemd[1]: smartd.service: Failed with result 'exit-code'.
Aug 28 16:12:07 ctl01 systemd[1]: Started Open vSwitch.
Aug 28 16:12:07 ctl01 systemd[1]: Started Login Service.
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Network.
Aug 28 16:12:07 ctl01 sysfsutils[2013]:  * Setting sysfs variables...
Aug 28 16:12:07 ctl01 grub-common[1996]:  * Recording successful boot for GRUB
Aug 28 16:12:07 ctl01 systemd[1]: Started Unattended Upgrades Shutdown.
Aug 28 16:12:07 ctl01 systemd[1]: Starting The Salt Minion...
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Network is Online.
Aug 28 16:12:07 ctl01 dbus-daemon[2024]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkit.service' requested by ':1.1' (uid=0 pid=2010 comm="/usr/lib/accountsservice/accounts-daemon " label="unconfined")
Aug 28 16:12:07 ctl01 cron[2068]: (CRON) INFO (Running @reboot jobs)
Aug 28 16:12:07 ctl01 sysfsutils[2013]:    ...done.
Aug 28 16:12:07 ctl01 kernel: [   15.657139] new mount options do not match the existing superblock, will be ignored
Aug 28 16:12:07 ctl01 systemd[1]: Starting OpenBSD Secure Shell server...
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Remote File Systems (Pre).
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Remote File Systems.
Aug 28 16:12:07 ctl01 lxcfs[2076]: mount namespace: 5
Aug 28 16:12:07 ctl01 lxcfs[2076]: hierarchies:
Aug 28 16:12:07 ctl01 lxcfs[2076]:   0: fd:   6: perf_event
Aug 28 16:12:07 ctl01 lxcfs[2076]:   1: fd:   7: pids
Aug 28 16:12:07 ctl01 lxcfs[2076]:   2: fd:   8: cpuset
Aug 28 16:12:07 ctl01 lxcfs[2076]:   3: fd:   9: rdma
Aug 28 16:12:07 ctl01 lxcfs[2076]:   4: fd:  10: devices
Aug 28 16:12:07 ctl01 lxcfs[2076]:   5: fd:  11: cpu,cpuacct
Aug 28 16:12:07 ctl01 lxcfs[2076]:   6: fd:  12: blkio
Aug 28 16:12:07 ctl01 lxcfs[2076]:   7: fd:  13: hugetlb
Aug 28 16:12:07 ctl01 lxcfs[2076]:   8: fd:  14: net_cls,net_prio
Aug 28 16:12:07 ctl01 lxcfs[2076]:   9: fd:  15: memory
Aug 28 16:12:07 ctl01 lxcfs[2076]:  10: fd:  16: freezer
Aug 28 16:12:07 ctl01 lxcfs[2076]:  11: fd:  17: name=systemd
Aug 28 16:12:07 ctl01 lxcfs[2076]:  12: fd:  18: unified
Aug 28 16:12:07 ctl01 systemd[1]: Starting Permit User Sessions...
Aug 28 16:12:07 ctl01 systemd[1]: Starting LSB: automatic crash report generation...
Aug 28 16:12:07 ctl01 systemd[1]: Starting Availability of block devices...
Aug 28 16:12:07 ctl01 systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
Aug 28 16:12:07 ctl01 systemd[1]: Started LSB: Set sysfs variables from /etc/sysfs.conf.
Aug 28 16:12:07 ctl01 systemd[1]: Started Availability of block devices.
Aug 28 16:12:07 ctl01 systemd[1]: Starting Authorization Manager...
Aug 28 16:12:07 ctl01 systemd[1]: Started Permit User Sessions.
Aug 28 16:12:07 ctl01 systemd[1]: Starting Terminate Plymouth Boot Screen...
Aug 28 16:12:07 ctl01 systemd[1]: Starting Hold until boot process finishes up...
Aug 28 16:12:07 ctl01 systemd[1]: Started Hold until boot process finishes up.
Aug 28 16:12:07 ctl01 systemd[1]: Starting Set console scheme...
Aug 28 16:12:07 ctl01 systemd[1]: Started Serial Getty on ttyS0.
Aug 28 16:12:07 ctl01 systemd[1]: Started Terminate Plymouth Boot Screen.
Aug 28 16:12:07 ctl01 grub-common[1996]:    ...done.
Aug 28 16:12:07 ctl01 systemd[1]: Started LSB: Record successful boot for GRUB.
Aug 28 16:12:07 ctl01 systemd[1]: Started Set console scheme.
Aug 28 16:12:07 ctl01 systemd[1]: Created slice system-getty.slice.
Aug 28 16:12:07 ctl01 systemd[1]: Started Getty on tty1.
Aug 28 16:12:07 ctl01 systemd[1]: Reached target Login Prompts.
Aug 28 16:12:07 ctl01 apport[2153]:  * Starting automatic crash report generation: apport
Aug 28 16:12:08 ctl01 dnsmasq[2161]: dnsmasq: syntax check OK.
Aug 28 16:12:08 ctl01 apport[2153]:    ...done.
Aug 28 16:12:08 ctl01 systemd[1]: Started LSB: automatic crash report generation.
Aug 28 16:12:08 ctl01 snapd[1990]: AppArmor status: apparmor is enabled and all features are available
Aug 28 16:12:08 ctl01 polkitd[2178]: started daemon version 0.105 using authority implementation `local' version `0.105'
Aug 28 16:12:08 ctl01 dbus-daemon[2024]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'
Aug 28 16:12:08 ctl01 systemd[1]: Started Authorization Manager.
Aug 28 16:12:08 ctl01 accounts-daemon[2010]: started daemon version 0.6.45
Aug 28 16:12:08 ctl01 systemd[1]: Started Accounts Service.
Aug 28 16:12:08 ctl01 dnsmasq[2307]: started, version 2.79 cachesize 150
Aug 28 16:12:08 ctl01 dnsmasq[2307]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
Aug 28 16:12:08 ctl01 dnsmasq[2307]: reading /etc/resolv.conf
Aug 28 16:12:08 ctl01 dnsmasq[2307]: using nameserver 8.8.8.8#53
Aug 28 16:12:08 ctl01 dnsmasq[2307]: read /etc/hosts - 11 addresses
Aug 28 16:12:08 ctl01 systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.
Aug 28 16:12:08 ctl01 systemd[1]: Reached target Host and Network Name Lookups.
Aug 28 16:12:08 ctl01 systemd[1]: Starting Postfix Mail Transport Agent (instance -)...
Aug 28 16:12:08 ctl01 systemd[1]: Started OpenBSD Secure Shell server.
Aug 28 16:12:08 ctl01 snapd[1990]: daemon.go:379: started snapd/2.39.2+18.04 (series 16; classic) ubuntu/18.04 (amd64) linux/4.15.0-58-generic.
Aug 28 16:12:08 ctl01 systemd[1]: Started Snappy daemon.
Aug 28 16:12:08 ctl01 systemd[1]: Starting Wait until snapd is fully seeded...
Aug 28 16:12:08 ctl01 systemd[1]: Started Wait until snapd is fully seeded.
Aug 28 16:12:08 ctl01 systemd[1]: Starting Apply the settings specified in cloud-config...
Aug 28 16:12:08 ctl01 systemd[1]: Started LXD - container startup/shutdown.
Aug 28 16:12:09 ctl01 postfix/postfix-script[2581]: starting the Postfix mail system
Aug 28 16:12:09 ctl01 postfix/master[2585]: daemon started -- version 3.3.0, configuration /etc/postfix
Aug 28 16:12:09 ctl01 systemd[1]: Started Postfix Mail Transport Agent (instance -).
Aug 28 16:12:09 ctl01 systemd[1]: Starting Postfix Mail Transport Agent...
Aug 28 16:12:09 ctl01 systemd[1]: Started Postfix Mail Transport Agent.
Aug 28 16:12:09 ctl01 systemd[1]: Started The Salt Minion.
Aug 28 16:12:09 ctl01 systemd[1]: Reached target Multi-User System.
Aug 28 16:12:09 ctl01 systemd[1]: Reached target Graphical Interface.
Aug 28 16:12:09 ctl01 systemd[1]: Starting Update UTMP about System Runlevel Changes...
Aug 28 16:12:09 ctl01 systemd[1]: Started Update UTMP about System Runlevel Changes.
Aug 28 16:12:09 ctl01 cloud-init[2447]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:config' at Wed, 28 Aug 2019 16:12:09 +0000. Up 17.43 seconds.
Aug 28 16:12:09 ctl01 systemd[1]: Started Apply the settings specified in cloud-config.
Aug 28 16:12:09 ctl01 systemd[1]: Starting Execute cloud user/final scripts...
Aug 28 16:12:10 ctl01 cloud-init[2638]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:final' at Wed, 28 Aug 2019 16:12:09 +0000. Up 18.01 seconds.
Aug 28 16:12:10 ctl01 cloud-init[2638]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 finished at Wed, 28 Aug 2019 16:12:10 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 18.11 seconds
Aug 28 16:12:10 ctl01 systemd[1]: Started Execute cloud user/final scripts.
Aug 28 16:12:10 ctl01 systemd[1]: Reached target Cloud-init target.
Aug 28 16:12:10 ctl01 systemd[1]: Startup finished in 5.498s (kernel) + 12.685s (userspace) = 18.184s.
Aug 28 16:12:13 ctl01 snapd[1990]: daemon.go:611: gracefully waiting for running hooks
Aug 28 16:12:13 ctl01 snapd[1990]: daemon.go:613: done waiting for running hooks
Aug 28 16:12:13 ctl01 snapd[1990]: daemon stop requested to wait for socket activation
Aug 28 16:12:15 ctl01 kernel: [   23.320350] random: crng init done
Aug 28 16:12:15 ctl01 kernel: [   23.320427] random: 7 urandom warning(s) missed due to ratelimiting
Aug 28 16:12:29 ctl01 systemd-timesyncd[977]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com).
Aug 28 16:12:42 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install python-oauth python-m2crypto.
Aug 28 16:12:48 ctl01 systemd[1]: Reloading.
Aug 28 16:12:49 ctl01 salt-minion[2112]: [WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
Aug 28 16:12:50 ctl01 salt-minion[2112]: ................................................................................................++++
Aug 28 16:12:52 ctl01 salt-minion[2112]: ............................................................................................................................................................................................................................................................................................................................................................................++++
Aug 28 16:12:53 ctl01 salt-minion[2112]: [WARNING ] State for file: /etc/kubernetes/ssl/ca-kubernetes.crt - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:12:54 ctl01 salt-minion[2112]: ...........................................................................................................................................................++++
Aug 28 16:12:55 ctl01 salt-minion[2112]: .........................................................................++++
Aug 28 16:12:55 ctl01 salt-minion[2112]: ..++++
Aug 28 16:12:58 ctl01 salt-minion[2112]: ..................................................................................................................................................................................................................................................................................................................................................++++
Aug 28 16:12:58 ctl01 salt-minion[2112]: [WARNING ] State for file: /var/lib/etcd/ca.pem - Neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'True'. As there is no source to replace the file with, 'replace' has been set to 'False' to avoid reading the file unnecessarily.
Aug 28 16:12:59 ctl01 salt-minion[2112]: .............................................................++++
Aug 28 16:12:59 ctl01 salt-minion[2112]: .............++++
Aug 28 16:13:00 ctl01 salt-minion[2112]: ....................++++
Aug 28 16:13:00 ctl01 salt-minion[2112]: ....................................++++
Aug 28 16:13:02 ctl01 salt-minion[2112]: ...............................................................................................................................................................................................................++++
Aug 28 16:13:03 ctl01 salt-minion[2112]: .................................................................................................++++
Aug 28 16:13:04 ctl01 salt-minion[2112]: ...........................++++
Aug 28 16:13:04 ctl01 salt-minion[2112]: ...................................................................................................................................++++
Aug 28 16:13:06 ctl01 salt-minion[2112]: .....................................................................................................................++++
Aug 28 16:13:06 ctl01 salt-minion[2112]: .............................................................................................................++++
Aug 28 16:13:07 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install ntp.
Aug 28 16:13:10 ctl01 systemd[1]: Reloading.
Aug 28 16:13:10 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:13:10 ctl01 systemd[1]: Started ntp-systemd-netif.path.
Aug 28 16:13:10 ctl01 kernel: [   80.609573] kauditd_printk_skb: 5 callbacks suppressed
Aug 28 16:13:10 ctl01 kernel: [   80.609575] audit: type=1400 audit(1567008790.857:17): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/ntpd" pid=3925 comm="apparmor_parser"
Aug 28 16:13:10 ctl01 systemd[1]: Reloading.
Aug 28 16:13:11 ctl01 systemd[1]: message repeated 2 times: [ Reloading.]
Aug 28 16:13:11 ctl01 systemd[1]: Starting Network Time Service...
Aug 28 16:13:11 ctl01 systemd[1]: Stopping Network Time Synchronization...
Aug 28 16:13:11 ctl01 ntpd[4035]: ntpd 4.2.8p10@1.3728-o (1): Starting
Aug 28 16:13:11 ctl01 ntpd[4035]: Command line: /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 112:118
Aug 28 16:13:11 ctl01 systemd[1]: Started Network Time Service.
Aug 28 16:13:11 ctl01 systemd[1]: Reloading.
Aug 28 16:13:11 ctl01 systemd[1]: Stopped Network Time Synchronization.
Aug 28 16:13:11 ctl01 ntpd[4039]: proto: precision = 0.061 usec (-24)
Aug 28 16:13:11 ctl01 ntpd[4039]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): good hash signature
Aug 28 16:13:11 ctl01 ntpd[4039]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): loaded, expire=2019-12-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen and drop on 0 v6wildcard [::]:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen and drop on 1 v4wildcard 0.0.0.0:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 2 lo 127.0.0.1:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 3 ens3 192.168.11.21:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 4 br-mgmt 172.16.10.36:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 5 lo [::1]:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 6 ens3 [fe80::5054:ff:fe3f:9b0c%2]:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 7 ens4 [fe80::5054:ff:fe2c:cdbc%3]:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 8 ens5 [fe80::5054:ff:fe9e:c5f6%4]:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 9 br-mgmt [fe80::5054:ff:fe2c:cdbc%6]:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listen normally on 10 ens5.1000 [fe80::5054:ff:fe9e:c5f6%7]:123
Aug 28 16:13:11 ctl01 ntpd[4039]: Listening on routing socket on fd #27 for interface updates
Aug 28 16:13:12 ctl01 ntpd[4039]: Soliciting pool server 91.209.0.19
Aug 28 16:13:13 ctl01 ntpd[4039]: Soliciting pool server 193.228.143.22
Aug 28 16:13:13 ctl01 ntpd[4039]: Soliciting pool server 194.190.168.1
Aug 28 16:13:14 ctl01 ntpd[4039]: Soliciting pool server 194.58.202.148
Aug 28 16:13:14 ctl01 ntpd[4039]: Soliciting pool server 91.209.0.20
Aug 28 16:13:15 ctl01 systemd[1]: Started /bin/systemctl restart ntp.service.
Aug 28 16:13:15 ctl01 systemd[1]: Stopping Network Time Service...
Aug 28 16:13:15 ctl01 ntpd[4039]: ntpd exiting on signal 15 (Terminated)
Aug 28 16:13:15 ctl01 ntpd[4039]: 91.209.0.19 local addr 192.168.11.21 -> <null>
Aug 28 16:13:15 ctl01 ntpd[4039]: 193.228.143.22 local addr 192.168.11.21 -> <null>
Aug 28 16:13:15 ctl01 ntpd[4039]: 194.58.202.148 local addr 192.168.11.21 -> <null>
Aug 28 16:13:15 ctl01 ntpd[4039]: 91.209.0.20 local addr 192.168.11.21 -> <null>
Aug 28 16:13:15 ctl01 systemd[1]: Stopped Network Time Service.
Aug 28 16:13:15 ctl01 systemd[1]: Starting Network Time Service...
Aug 28 16:13:15 ctl01 ntpd[4368]: ntpd 4.2.8p10@1.3728-o (1): Starting
Aug 28 16:13:15 ctl01 ntpd[4368]: Command line: /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 112:118
Aug 28 16:13:15 ctl01 systemd[1]: Started Network Time Service.
Aug 28 16:13:15 ctl01 ntpd[4371]: proto: precision = 0.063 usec (-24)
Aug 28 16:13:15 ctl01 ntpd[4371]: restrict 0.0.0.0: KOD does nothing without LIMITED.
Aug 28 16:13:15 ctl01 ntpd[4371]: restrict ::: KOD does nothing without LIMITED.
Aug 28 16:13:15 ctl01 ntpd[4371]: switching logging to file /var/log/ntp.log
Aug 28 16:13:18 ctl01 salt-minion[2112]: [INFO    ] Executing command ['systemctl', 'status', 'salt-minion.service', '-n', '0'] in directory '/root'
Aug 28 16:13:19 ctl01 salt-minion[2112]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'restart', 'salt-minion.service'] in directory '/root'
Aug 28 16:13:19 ctl01 systemd[1]: Started /bin/systemctl restart salt-minion.service.
Aug 28 16:13:19 ctl01 systemd[1]: Stopping The Salt Minion...
Aug 28 16:13:19 ctl01 salt-minion[2112]: [WARNING ] Minion received a SIGTERM. Exiting.
Aug 28 16:13:19 ctl01 salt-minion[2112]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited.
Aug 28 16:13:19 ctl01 systemd[1]: Stopped The Salt Minion.
Aug 28 16:13:19 ctl01 systemd[1]: salt-minion.service: Found left-over process 4374 (bash) in control group while starting unit. Ignoring.
Aug 28 16:13:19 ctl01 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Aug 28 16:13:19 ctl01 systemd[1]: salt-minion.service: Found left-over process 4439 (salt-call) in control group while starting unit. Ignoring.
Aug 28 16:13:19 ctl01 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Aug 28 16:13:19 ctl01 systemd[1]: Starting The Salt Minion...
Aug 28 16:13:19 ctl01 systemd[1]: Started The Salt Minion.
Aug 28 16:13:19 ctl01 salt-minion[2112]: local:
Aug 28 16:13:19 ctl01 salt-minion[2112]:     True
Aug 28 16:13:19 ctl01 salt-minion[4490]: [INFO    ] Setting up the Salt Minion "ctl01.mcp-k8s-calico-noha.local"
Aug 28 16:13:20 ctl01 salt-minion[4490]: [INFO    ] Starting up the Salt Minion
Aug 28 16:13:20 ctl01 salt-minion[4490]: [INFO    ] Starting pull socket on /var/run/salt/minion/minion_event_1899fe9592_pull.ipc
Aug 28 16:13:20 ctl01 salt-minion[4490]: [INFO    ] Creating minion process manager
Aug 28 16:13:22 ctl01 salt-minion[4490]: [INFO    ] Executing command ['date', '+%z'] in directory '/root'
Aug 28 16:13:22 ctl01 salt-minion[4490]: [INFO    ] Updating job settings for scheduled job: __mine_interval
Aug 28 16:13:22 ctl01 salt-minion[4490]: [INFO    ] Added mine.update to scheduler
Aug 28 16:13:22 ctl01 salt-minion[4490]: [INFO    ] Minion is starting as user 'root'
Aug 28 16:13:22 ctl01 salt-minion[4490]: [INFO    ] Minion is ready to receive requests!
Aug 28 16:13:31 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190828161331067379
Aug 28 16:13:31 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 4580
Aug 28 16:13:32 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:13:32 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/server/service.sls'
Aug 28 16:13:32 ctl01 salt-minion[4490]: [INFO    ] Executing command '. /var/lib/etcd/configenv; etcdctl cluster-health > /dev/null 2>&1; echo $?' in directory '/root'
Aug 28 16:13:32 ctl01 salt-minion[4490]: [ERROR   ] Command '. /var/lib/etcd/configenv; etcdctl cluster-health > /dev/null 2>&1; echo $?' failed with return code: 2
Aug 28 16:13:32 ctl01 salt-minion[4490]: [ERROR   ] stdout: /bin/sh: 1: .: Can't open /var/lib/etcd/configenv
Aug 28 16:13:32 ctl01 salt-minion[4490]: [ERROR   ] retcode: 2
Aug 28 16:13:33 ctl01 salt-minion[4490]: [INFO    ] Running state [etcd_support_packages] at time 16:13:33.124862
Aug 28 16:13:33 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [etcd_support_packages]
Aug 28 16:13:33 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:13:33 ctl01 salt-minion[4490]: [INFO    ] Executing command ['apt-cache', '-q', 'policy', 'python-etcd'] in directory '/root'
Aug 28 16:13:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['apt-get', '-q', 'update'] in directory '/root'
Aug 28 16:13:37 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
Aug 28 16:13:37 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'python-etcd'] in directory '/root'
Aug 28 16:13:37 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install python-etcd.
Aug 28 16:13:40 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:13:40 ctl01 salt-minion[4490]: [INFO    ] Made the following changes:
Aug 28 16:13:40 ctl01 salt-minion[4490]: 'python-etcd' changed from 'absent' to '0.4.3-2'
Aug 28 16:13:40 ctl01 salt-minion[4490]: 'python-dnspython' changed from 'absent' to '1.15.0-1'
Aug 28 16:13:40 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:13:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [etcd_support_packages] at time 16:13:40.946977 duration_in_ms=7822.113
Aug 28 16:13:40 ctl01 salt-minion[4490]: [INFO    ] Running state [/tmp/etcd/bin] at time 16:13:40.951883
Aug 28 16:13:40 ctl01 salt-minion[4490]: [INFO    ] Executing state archive.extracted for [/tmp/etcd/bin]
Aug 28 16:13:43 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', 'x', '--strip=1', '-f', '/var/cache/salt/minion/extrn_files/base/github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz'] in directory '/tmp/etcd/bin/'
Aug 28 16:13:43 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
Aug 28 16:13:43 ctl01 salt-minion[4490]: [INFO    ] {'extracted_files': 'no tar output so far', 'directories_created': ['/tmp/etcd/bin/']}
Aug 28 16:13:43 ctl01 salt-minion[4490]: [INFO    ] Completed state [/tmp/etcd/bin] at time 16:13:43.847154 duration_in_ms=2895.267
Aug 28 16:13:43 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/local/bin/etcd] at time 16:13:43.848078
Aug 28 16:13:43 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/local/bin/etcd]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:44 ctl01 salt-minion[4490]: New file
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/local/bin/etcd] at time 16:13:44.067436 duration_in_ms=219.357
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/local/bin/etcdctl] at time 16:13:44.068037
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/local/bin/etcdctl]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:44 ctl01 salt-minion[4490]: New file
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/local/bin/etcdctl] at time 16:13:44.238751 duration_in_ms=170.713
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Running state [etcd] at time 16:13:44.240540
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Executing state user.present for [etcd]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] User etcd is present and up to date
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [etcd] at time 16:13:44.250778 duration_in_ms=10.238
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/etcd.service] at time 16:13:44.251119
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/etcd.service]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/systemd/etcd.service'
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:44 ctl01 salt-minion[4490]: New file
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/etcd.service] at time 16:13:44.285030 duration_in_ms=33.91
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/etcd] at time 16:13:44.285378
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/etcd]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/default'
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:44 ctl01 salt-minion[4490]: New file
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/etcd] at time 16:13:44.366185 duration_in_ms=80.807
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/var/lib/etcd/] at time 16:13:44.366531
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/var/lib/etcd/]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Directory /var/lib/etcd is in the correct state
Aug 28 16:13:44 ctl01 salt-minion[4490]: Directory /var/lib/etcd updated
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/var/lib/etcd/] at time 16:13:44.371625 duration_in_ms=5.093
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/var/lib/etcd/configenv] at time 16:13:44.372281
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/var/lib/etcd/configenv]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/configenv'
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:44 ctl01 salt-minion[4490]: New file
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/var/lib/etcd/configenv] at time 16:13:44.430523 duration_in_ms=58.242
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/var/lib/etcd/configenvv3] at time 16:13:44.430974
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/var/lib/etcd/configenvv3]
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/files/configenvv3'
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:44 ctl01 salt-minion[4490]: New file
Aug 28 16:13:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/var/lib/etcd/configenvv3] at time 16:13:44.482387 duration_in_ms=51.413
Aug 28 16:13:45 ctl01 salt-minion[4490]: [INFO    ] Running state [etcd] at time 16:13:45.050621
Aug 28 16:13:45 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [etcd]
Aug 28 16:13:45 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'etcd.service', '-n', '0'] in directory '/root'
Aug 28 16:13:45 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'etcd.service'] in directory '/root'
Aug 28 16:13:45 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
Aug 28 16:13:45 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'etcd.service'] in directory '/root'
Aug 28 16:13:45 ctl01 systemd[1]: Started /bin/systemctl start etcd.service.
Aug 28 16:13:45 ctl01 systemd[1]: Starting etcd - highly-available key value store...
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://172.16.10.36:4001
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_CERT_FILE=/var/lib/etcd/etcd-server.crt
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/default
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=5000
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=250
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.16.10.36:2380
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_INITIAL_CLUSTER=ctl01=https://172.16.10.36:2380
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=IN7KaRMSo3xkGxkjAAPtkRkAgqN4ZNRq
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_KEY_FILE=/var/lib/etcd/etcd-server.key
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://172.16.10.36:4001,http://127.0.0.1:4001
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://172.16.10.36:2380
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_NAME=ctl01
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_PEER_CERT_FILE=/var/lib/etcd/etcd-server.crt
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_PEER_KEY_FILE=/var/lib/etcd/etcd-server.key
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/var/lib/etcd/ca.pem
Aug 28 16:13:45 ctl01 etcd[5283]: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/var/lib/etcd/ca.pem
Aug 28 16:13:45 ctl01 etcd[5283]: etcd Version: 3.3.12
Aug 28 16:13:45 ctl01 etcd[5283]: Git SHA: d57e8b8
Aug 28 16:13:45 ctl01 etcd[5283]: Go Version: go1.10.8
Aug 28 16:13:45 ctl01 etcd[5283]: Go OS/Arch: linux/amd64
Aug 28 16:13:45 ctl01 etcd[5283]: setting maximum number of CPUs to 8, total number of available CPUs is 8
Aug 28 16:13:45 ctl01 etcd[5283]: peerTLS: cert = /var/lib/etcd/etcd-server.crt, key = /var/lib/etcd/etcd-server.key, ca = , trusted-ca = /var/lib/etcd/ca.pem, client-cert-auth = true, crl-file = 
Aug 28 16:13:45 ctl01 etcd[5283]: listening for peers on https://172.16.10.36:2380
Aug 28 16:13:45 ctl01 etcd[5283]: The scheme of client url http://127.0.0.1:4001 is HTTP while peer key/cert files are presented. Ignored key/cert files.
Aug 28 16:13:45 ctl01 etcd[5283]: The scheme of client url http://127.0.0.1:4001 is HTTP while client cert auth (--client-cert-auth) is enabled. Ignored client cert auth for this url.
Aug 28 16:13:45 ctl01 etcd[5283]: listening for client requests on 127.0.0.1:4001
Aug 28 16:13:45 ctl01 etcd[5283]: listening for client requests on 172.16.10.36:4001
Aug 28 16:13:45 ctl01 etcd[5283]: name = ctl01
Aug 28 16:13:45 ctl01 etcd[5283]: data dir = /var/lib/etcd/default
Aug 28 16:13:45 ctl01 etcd[5283]: member dir = /var/lib/etcd/default/member
Aug 28 16:13:45 ctl01 etcd[5283]: heartbeat = 250ms
Aug 28 16:13:45 ctl01 etcd[5283]: election = 5000ms
Aug 28 16:13:45 ctl01 etcd[5283]: snapshot count = 100000
Aug 28 16:13:45 ctl01 etcd[5283]: advertise client URLs = https://172.16.10.36:4001
Aug 28 16:13:45 ctl01 etcd[5283]: initial advertise peer URLs = https://172.16.10.36:2380
Aug 28 16:13:45 ctl01 etcd[5283]: initial cluster = ctl01=https://172.16.10.36:2380
Aug 28 16:13:45 ctl01 etcd[5283]: starting member 411212f5fea59a9f in cluster 8f5ffbb80be06f34
Aug 28 16:13:45 ctl01 etcd[5283]: 411212f5fea59a9f became follower at term 0
Aug 28 16:13:45 ctl01 etcd[5283]: newRaft 411212f5fea59a9f [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Aug 28 16:13:45 ctl01 etcd[5283]: 411212f5fea59a9f became follower at term 1
Aug 28 16:13:45 ctl01 etcd[5283]: simple token is not cryptographically signed
Aug 28 16:13:45 ctl01 etcd[5283]: starting server... [version: 3.3.12, cluster version: to_be_decided]
Aug 28 16:13:45 ctl01 etcd[5283]: 411212f5fea59a9f as single-node; fast-forwarding 19 ticks (election ticks 20)
Aug 28 16:13:45 ctl01 etcd[5283]: added member 411212f5fea59a9f [https://172.16.10.36:2380] to cluster 8f5ffbb80be06f34
Aug 28 16:13:45 ctl01 etcd[5283]: ClientTLS: cert = /var/lib/etcd/etcd-server.crt, key = /var/lib/etcd/etcd-server.key, ca = , trusted-ca = /var/lib/etcd/ca.pem, client-cert-auth = true, crl-file = 
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190828161346118721
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 5312
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161346118721
Aug 28 16:13:46 ctl01 etcd[5283]: 411212f5fea59a9f is starting a new election at term 1
Aug 28 16:13:46 ctl01 etcd[5283]: 411212f5fea59a9f became candidate at term 2
Aug 28 16:13:46 ctl01 etcd[5283]: 411212f5fea59a9f received MsgVoteResp from 411212f5fea59a9f at term 2
Aug 28 16:13:46 ctl01 etcd[5283]: 411212f5fea59a9f became leader at term 2
Aug 28 16:13:46 ctl01 etcd[5283]: raft.node: 411212f5fea59a9f elected leader 411212f5fea59a9f at term 2
Aug 28 16:13:46 ctl01 etcd[5283]: setting up the initial cluster version to 3.3
Aug 28 16:13:46 ctl01 etcd[5283]: ready to serve client requests
Aug 28 16:13:46 ctl01 etcd[5283]: ready to serve client requests
Aug 28 16:13:46 ctl01 etcd[5283]: published {Name:ctl01 ClientURLs:[https://172.16.10.36:4001]} to cluster 8f5ffbb80be06f34
Aug 28 16:13:46 ctl01 systemd[1]: Started etcd - highly-available key value store.
Aug 28 16:13:46 ctl01 etcd[5283]: serving insecure client requests on 127.0.0.1:4001, this is strongly discouraged!
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'etcd.service'] in directory '/root'
Aug 28 16:13:46 ctl01 etcd[5283]: serving client requests on 172.16.10.36:4001
Aug 28 16:13:46 ctl01 etcd[5283]: set the initial cluster version to 3.3
Aug 28 16:13:46 ctl01 etcd[5283]: enabled capabilities for version 3.3
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'etcd.service'] in directory '/root'
Aug 28 16:13:46 ctl01 systemd[1]: Started /bin/systemctl enable etcd.service.
Aug 28 16:13:46 ctl01 systemd[1]: Reloading.
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'etcd.service'] in directory '/root'
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] {'etcd': True}
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Completed state [etcd] at time 16:13:46.470643 duration_in_ms=1420.022
Aug 28 16:13:46 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161331067379
Aug 28 16:13:47 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command cmd.run with jid 20190828161347174244
Aug 28 16:13:47 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 5378
Aug 28 16:13:47 ctl01 salt-minion[4490]: [INFO    ] Executing command '. /var/lib/etcd/configenv && etcdctl cluster-health' in directory '/root'
Aug 28 16:13:47 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161347174244
Aug 28 16:13:48 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190828161348041078
Aug 28 16:13:48 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 5392
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/kube-addons.sls'
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons] at time 16:13:54.973948
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/etc/kubernetes/addons]
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] {'/etc/kubernetes/addons': 'New Dir'}
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons] at time 16:13:54.987467 duration_in_ms=13.52
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:13:54.988057
Aug 28 16:13:54 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-kube-controllers.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/calico/calico-kube-controllers.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:13:55.128351 duration_in_ms=140.292
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:13:55.128682
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-rbac.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/calico/calico-rbac.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:13:55.183012 duration_in_ms=54.328
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:13:55.183571
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-svc.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-svc.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:13:55.291179 duration_in_ms=107.607
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:13:55.291637
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-server.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-server.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:13:55.459513 duration_in_ms=167.873
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:13:55.460245
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-agent.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-agent.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:13:55.573840 duration_in_ms=113.595
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:13:55.574334
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-serviceaccount.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:13:55.607143 duration_in_ms=32.808
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:13:55.607387
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-roles.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/netchecker/netchecker-roles.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:13:55.634913 duration_in_ms=27.525
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:13:55.635216
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/prometheus/prometheus-roles.yml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/prometheus/prometheus-roles.yml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:13:55.666509 duration_in_ms=31.293
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/coredns] at time 16:13:55.666765
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/coredns]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/coredns is not present
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns] at time 16:13:55.667592 duration_in_ms=0.827
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true] at time 16:13:55.668599
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state cmd.run for [kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command 'kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] {'pid': 5473, 'retcode': 0, 'stderr': '/bin/sh: 1: kubectl: not found', 'stdout': 'coredns is absent. OK'}
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [kubectl -n kube-system delete svc coredns > /dev/null || echo "coredns is absent. OK" && true] at time 16:13:55.682400 duration_in_ms=13.801
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-svc.yaml] at time 16:13:55.683429
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-svc.yaml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-svc.yaml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-svc.yaml] at time 16:13:55.791890 duration_in_ms=108.461
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-rc.yaml] at time 16:13:55.792429
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-rc.yaml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-rc.yaml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-rc.yaml] at time 16:13:55.914173 duration_in_ms=121.742
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-sa.yaml] at time 16:13:55.914661
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-sa.yaml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-sa.yaml'
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:55 ctl01 salt-minion[4490]: New file
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-sa.yaml] at time 16:13:55.947705 duration_in_ms=33.045
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-autoscaler.yaml] at time 16:13:55.948010
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-autoscaler.yaml]
Aug 28 16:13:55 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-autoscaler.yaml'
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:56 ctl01 salt-minion[4490]: New file
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-autoscaler.yaml] at time 16:13:56.053677 duration_in_ms=105.665
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-autoscaler-rbac.yaml] at time 16:13:56.054178
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-autoscaler-rbac.yaml]
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-autoscaler-rbac.yaml'
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:56 ctl01 salt-minion[4490]: New file
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-autoscaler-rbac.yaml] at time 16:13:56.089743 duration_in_ms=35.565
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/dns/kubedns-clusterrole.yaml] at time 16:13:56.090055
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/dns/kubedns-clusterrole.yaml]
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/dns/kubedns-clusterrole.yaml'
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:13:56 ctl01 salt-minion[4490]: New file
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/dns/kubedns-clusterrole.yaml] at time 16:13:56.124179 duration_in_ms=34.125
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/metrics-server] at time 16:13:56.124450
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/metrics-server]
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/metrics-server is not present
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/metrics-server] at time 16:13:56.125238 duration_in_ms=0.788
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161348041078
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190828161356768692
Aug 28 16:13:56 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 5511
Aug 28 16:13:57 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:13:57 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/init.sls'
Aug 28 16:13:57 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:57 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/calico.sls'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/service.sls'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/_common.sls'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/pool/kube-proxy.sls'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/calicoctl] at time 16:13:58.630511
Aug 28 16:13:58 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/bin/calicoctl]
Aug 28 16:14:09 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:09 ctl01 salt-minion[4490]: New file
Aug 28 16:14:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/calicoctl] at time 16:14:09.566557 duration_in_ms=10936.046
Aug 28 16:14:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/birdcl] at time 16:14:09.566917
Aug 28 16:14:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/bin/birdcl]
Aug 28 16:14:10 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:10 ctl01 salt-minion[4490]: New file
Aug 28 16:14:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/birdcl] at time 16:14:10.041252 duration_in_ms=474.334
Aug 28 16:14:10 ctl01 salt-minion[4490]: [INFO    ] Running state [/opt/cni/bin/calico] at time 16:14:10.041566
Aug 28 16:14:10 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico]
Aug 28 16:14:11 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190828161411825253
Aug 28 16:14:11 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 5586
Aug 28 16:14:11 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161411825253
Aug 28 16:14:17 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:17 ctl01 salt-minion[4490]: New file
Aug 28 16:14:17 ctl01 salt-minion[4490]: [INFO    ] Completed state [/opt/cni/bin/calico] at time 16:14:17.245385 duration_in_ms=7203.818
Aug 28 16:14:17 ctl01 salt-minion[4490]: [INFO    ] Running state [/opt/cni/bin/calico-ipam] at time 16:14:17.245736
Aug 28 16:14:17 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico-ipam]
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:22 ctl01 salt-minion[4490]: New file
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Completed state [/opt/cni/bin/calico-ipam] at time 16:14:22.204464 duration_in_ms=4958.728
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/cni/net.d/10-calico.conf] at time 16:14:22.204724
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/cni/net.d/10-calico.conf]
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/calico.conf'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:22 ctl01 salt-minion[4490]: New file
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/cni/net.d/10-calico.conf] at time 16:14:22.324616 duration_in_ms=119.892
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/calico/network-environment] at time 16:14:22.324940
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/calico/network-environment]
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/network-environment.pool'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:22 ctl01 salt-minion[4490]: New file
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/calico/network-environment] at time 16:14:22.417303 duration_in_ms=92.362
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/calico/calicoctl.cfg] at time 16:14:22.417623
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/calico/calicoctl.cfg]
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/calicoctl.cfg.pool'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:22 ctl01 salt-minion[4490]: New file
Aug 28 16:14:22 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/calico/calicoctl.cfg] at time 16:14:22.528871 duration_in_ms=111.247
Aug 28 16:14:23 ctl01 salt-minion[4490]: [INFO    ] Running state [containerd] at time 16:14:23.182232
Aug 28 16:14:23 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [containerd]
Aug 28 16:14:23 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:14:24 ctl01 salt-minion[4490]: [INFO    ] Executing command ['apt-cache', '-q', 'policy', 'containerd'] in directory '/root'
Aug 28 16:14:24 ctl01 salt-minion[4490]: [INFO    ] Executing command ['apt-get', '-q', 'update'] in directory '/root'
Aug 28 16:14:26 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
Aug 28 16:14:26 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'containerd'] in directory '/root'
Aug 28 16:14:26 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install containerd.
Aug 28 16:14:31 ctl01 systemd[1]: Reloading.
Aug 28 16:14:31 ctl01 systemd[1]: Reloading.
Aug 28 16:14:31 ctl01 systemd[1]: Starting containerd container runtime...
Aug 28 16:14:31 ctl01 systemd[1]: Started containerd container runtime.
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.401114736Z" level=info msg="starting containerd" revision= version="1.2.6-0ubuntu1~18.04.2"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.401569511Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.401741035Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.401950610Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.402016719Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.417161324Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.417279562Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.417516959Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.417783453Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.417811208Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.417876479Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.417890129Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Aug 28 16:14:31 ctl01 kernel: [  161.164098] aufs 4.15-20180219
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431426170Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431464933Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431509661Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431530989Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431545481Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431564424Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431581590Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431596207Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431609203Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431622976Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431754445Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.431845659Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432467256Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432515362Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432571070Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432589560Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432603361Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432616329Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432973898Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.432992834Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433006451Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433019948Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433033266Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433183033Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433203990Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433217709Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433241983Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433315898Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:127.0.0.1 StreamServerPort:0 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433393488Z" level=info msg="Connect containerd service"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.433572372Z" level=info msg="Get image filesystem path "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs""
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.434084849Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.434181489Z" level=info msg="Start subscribing containerd event"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.434953738Z" level=info msg=serving... address="/run/containerd/containerd.sock"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.434994603Z" level=info msg="containerd successfully booted in 0.034351s"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.435308390Z" level=info msg="Start recovering state"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.435464580Z" level=info msg="Start event monitor"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.435496754Z" level=info msg="Start snapshots syncer"
Aug 28 16:14:31 ctl01 containerd[6278]: time="2019-08-28T16:14:31.435518659Z" level=info msg="Start streaming server"
Aug 28 16:14:33 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Made the following changes:
Aug 28 16:14:34 ctl01 salt-minion[4490]: 'containerd' changed from 'absent' to '1.2.6-0ubuntu1~18.04.2'
Aug 28 16:14:34 ctl01 salt-minion[4490]: 'runc' changed from 'absent' to '1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Completed state [containerd] at time 16:14:34.078777 duration_in_ms=10896.546
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/containerd/config.toml] at time 16:14:34.083546
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/containerd/config.toml]
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/containerd/config.toml'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:34 ctl01 salt-minion[4490]: New file
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/containerd/config.toml] at time 16:14:34.195236 duration_in_ms=111.689
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Running state [containerd] at time 16:14:34.713999
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [containerd]
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'containerd.service', '-n', '0'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'containerd.service'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'containerd.service'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] The service containerd is already running
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Completed state [containerd] at time 16:14:34.765604 duration_in_ms=51.604
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Running state [containerd] at time 16:14:34.766093
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing state service.mod_watch for [containerd]
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'containerd.service'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'restart', 'containerd.service'] in directory '/root'
Aug 28 16:14:34 ctl01 systemd[1]: Started /bin/systemctl restart containerd.service.
Aug 28 16:14:34 ctl01 systemd[1]: Stopping containerd container runtime...
Aug 28 16:14:34 ctl01 containerd[6278]: time="2019-08-28T16:14:34.808534753Z" level=info msg="Stop CRI service"
Aug 28 16:14:34 ctl01 systemd[1]: Stopped containerd container runtime.
Aug 28 16:14:34 ctl01 systemd[1]: Starting containerd container runtime...
Aug 28 16:14:34 ctl01 systemd[1]: Started containerd container runtime.
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] {'containerd': True}
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Completed state [containerd] at time 16:14:34.819550 duration_in_ms=53.458
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/calico-node.service] at time 16:14:34.820553
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/calico-node.service]
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/calico/calico-node.service.ctr'
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.851643789Z" level=info msg="starting containerd" revision= version="1.2.6-0ubuntu1~18.04.2"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.852101814Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.852148582Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.852345548Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.852415970Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854137925Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854186366Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854277685Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854476695Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854509789Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854529548Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854539627Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854652118Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854679901Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854722744Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854743850Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854759397Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854774550Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854789573Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854804014Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854817536Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854831694Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854874922Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.854925092Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855305824Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855347852Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855405115Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855422924Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855436681Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855449288Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855462315Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855475553Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855488076Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855502927Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855515933Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855562909Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855587553Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855601897Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855614617Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855700613Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:127.0.0.1 StreamServerPort:0 EnableSelinux:false SandboxImage:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855785819Z" level=info msg="Connect containerd service"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.855986011Z" level=info msg="Get image filesystem path "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs""
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.856890799Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.857056330Z" level=info msg="Start subscribing containerd event"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.857195459Z" level=info msg="Start recovering state"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.857313299Z" level=info msg=serving... address="/run/containerd/containerd.sock"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.857376716Z" level=info msg="containerd successfully booted in 0.006335s"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.857484233Z" level=info msg="Start event monitor"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.857572451Z" level=info msg="Start snapshots syncer"
Aug 28 16:14:34 ctl01 containerd[6590]: time="2019-08-28T16:14:34.857634569Z" level=info msg="Start streaming server"
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:14:34 ctl01 salt-minion[4490]: New file
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/calico-node.service] at time 16:14:34.866578 duration_in_ms=46.024
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Running state [/var/lib/calico] at time 16:14:34.866875
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/var/lib/calico]
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] {'/var/lib/calico': 'New Dir'}
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Completed state [/var/lib/calico] at time 16:14:34.868570 duration_in_ms=1.694
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Running state [/var/log/calico] at time 16:14:34.868815
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/var/log/calico]
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] {'/var/log/calico': 'New Dir'}
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Completed state [/var/log/calico] at time 16:14:34.871041 duration_in_ms=2.226
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Running state [calico-node] at time 16:14:34.873298
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [calico-node]
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'calico-node.service', '-n', '0'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'calico-node.service'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
Aug 28 16:14:34 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'calico-node.service'] in directory '/root'
Aug 28 16:14:34 ctl01 systemd[1]: Started /bin/systemctl start calico-node.service.
Aug 28 16:14:34 ctl01 systemd[1]: Starting calico-node...
Aug 28 16:14:35 ctl01 ctr[6639]: ctr: container "calico-node" in namespace "default": not found
Aug 28 16:14:35 ctl01 ctr[6658]: time="2019-08-28T16:14:35Z" level=error msg="failed to delete container "calico-node"" error="container "calico-node" in namespace "default": not found"
Aug 28 16:14:35 ctl01 ctr[6658]: ctr: container "calico-node" in namespace "default": not found
Aug 28 16:14:35 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:35 ctl01 ctr[6667]: elapsed: 0.1 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:35 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:35 ctl01 ctr[6667]: elapsed: 0.2 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:35 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:35 ctl01 ctr[6667]: elapsed: 0.3 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:35 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:35 ctl01 ctr[6667]: elapsed: 0.4 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:35 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:35 ctl01 ctr[6667]: elapsed: 0.5 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:35 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:35 ctl01 ctr[6667]: elapsed: 0.6 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:35 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:35 ctl01 ctr[6667]: elapsed: 0.7 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 0.8 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 0.9 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.0 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.1 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.2 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.3 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.4 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.5 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 etcd[5283]: proto: no coders for int
Aug 28 16:14:36 ctl01 etcd[5283]: proto: no encoder for ValueSize int [GetProperties]
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.6 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:36 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:36 ctl01 ctr[6667]: elapsed: 1.7 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 1.8 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 1.9 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.0 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.1 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.2 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.3 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.4 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.5 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.6 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:37 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolving      |#033[32m#033[0m--------------------------------------|
Aug 28 16:14:37 ctl01 ctr[6667]: elapsed: 2.7 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.3 KiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 2.8 s                                                                        total:   0.0 B (0.0 B/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/3.2 KiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/47.1 KiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 2.9 s                                                                        total:  1.3 Ki (472.0 B/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/3.2 KiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m#033[0m--------------------------------------| 37.3 KiB/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/47.1 KiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------|    0.0 B/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.0 s                                                                        total:  38.6 K (12.9 KiB/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m#033[0m--------------------------------------| 26.0 KiB/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m#033[0m--------------------------------------| 33.1 KiB/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       downloading    |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|  3.2 KiB/3.2 KiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++#033[0m---------------------------------| 338.5 Ki/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        downloading    |#033[32m+#033[0m-------------------------------------|  2.0 KiB/47.1 KiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------|  7.6 KiB/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.1 s                                                                        total:  411.7  (132.8 KiB/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++#033[0m-----------------------------------| 195.7 Ki/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++#033[0m----------------------------------| 209.8 Ki/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++#033[0m-------------------------| 771.2 Ki/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------| 109.4 Ki/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.2 s                                                                        total:  1.3 Mi (417.8 KiB/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++#033[0m---------------------------| 566.2 Ki/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++#033[0m--------------------------------| 327.2 Ki/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++#033[0m---------------------| 1001.7 K/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------| 238.1 Ki/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.3 s                                                                        total:  2.1 Mi (661.8 KiB/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++++++#033[0m-----------------------| 792.4 Ki/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++#033[0m------------------------------| 441.7 Ki/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++#033[0m-----------------|  1.2 MiB/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m#033[0m--------------------------------------| 351.2 Ki/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.4 s                                                                        total:  2.8 Mi (841.4 KiB/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++++#033[0m------------------|  1.0 MiB/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++#033[0m---------------------------| 554.8 Ki/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++#033[0m-------------|  1.4 MiB/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+#033[0m-------------------------------------| 468.6 Ki/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.5 s                                                                        total:  3.5 Mi (1016.5 KiB/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++++++++#033[0m--------------|  1.2 MiB/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++#033[0m-------------------------| 669.4 Ki/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++++++#033[0m---------|  1.6 MiB/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+#033[0m-------------------------------------| 581.7 Ki/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.6 s                                                                        total:  4.1 Mi (1.1 MiB/s)
Aug 28 16:14:38 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m+++++++++++++++++++++++++++++#033[0m---------|  1.4 MiB/1.9 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++#033[0m-----------------------| 785.3 Ki/1.8 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++++++++++#033[0m-----|  1.9 MiB/2.1 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:38 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+#033[0m-------------------------------------| 696.3 Ki/13.7 MiB
Aug 28 16:14:38 ctl01 ctr[6667]: elapsed: 3.7 s                                                                        total:  4.8 Mi (1.3 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        downloading    |#033[32m++++++++++++++++++++++++++++++++++#033[0m----|  1.7 MiB/1.9 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++++++++++++#033[0m--------------------| 901.3 Ki/1.8 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        downloading    |#033[32m+++++++++++++++++++++++++++++++++++++#033[0m-|  2.1 MiB/2.1 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++#033[0m------------------------------------| 810.8 Ki/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 3.8 s                                                                        total:  5.5 Mi (1.4 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++++++++#033[0m-----------------|  1.0 MiB/1.8 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++#033[0m------------------------------------| 962.1 Ki/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 3.9 s                                                                        total:  6.0 Mi (1.5 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++++++++++++#033[0m-------------|  1.2 MiB/1.8 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++#033[0m-----------------------------------|  1.2 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.0 s                                                                        total:  6.4 Mi (1.6 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m++++++++++++++++++++++++++++++++#033[0m------|  1.6 MiB/1.8 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++#033[0m----------------------------------|  1.5 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.1 s                                                                        total:  7.1 Mi (1.7 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        downloading    |#033[32m+++++++++++++++++++++++++++++++++++++#033[0m-|  1.8 MiB/1.8 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++#033[0m----------------------------------|  1.8 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.2 s                                                                        total:  7.6 Mi (1.8 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++#033[0m--------------------------------|  2.3 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.3 s                                                                        total:  8.2 Mi (1.9 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++#033[0m------------------------------|  2.9 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.4 s                                                                        total:  8.8 Mi (2.0 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++#033[0m-----------------------------|  3.5 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.5 s                                                                        total:  9.3 Mi (2.1 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++#033[0m----------------------------|  4.0 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.6 s                                                                        total:  9.8 Mi (2.1 MiB/s)
Aug 28 16:14:39 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:39 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++#033[0m--------------------------|  4.6 MiB/13.7 MiB
Aug 28 16:14:39 ctl01 ctr[6667]: elapsed: 4.7 s                                                                        total:  10.4 M (2.2 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++#033[0m------------------------|  5.1 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 4.8 s                                                                        total:  11.0 M (2.3 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++#033[0m-----------------------|  5.6 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 4.9 s                                                                        total:  11.4 M (2.3 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++#033[0m----------------------|  6.0 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.0 s                                                                        total:  11.9 M (2.4 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++#033[0m--------------------|  6.7 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.1 s                                                                        total:  12.6 M (2.5 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++#033[0m------------------|  7.4 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.2 s                                                                        total:  13.2 M (2.5 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++#033[0m-----------------|  7.9 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.3 s                                                                        total:  13.8 M (2.6 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++#033[0m---------------|  8.5 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.4 s                                                                        total:  14.3 M (2.7 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++#033[0m-------------|  9.1 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.5 s                                                                        total:  14.9 M (2.7 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++#033[0m------------|  9.6 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.6 s                                                                        total:  15.5 M (2.8 MiB/s)
Aug 28 16:14:40 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:40 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++#033[0m----------| 10.2 MiB/13.7 MiB
Aug 28 16:14:40 ctl01 ctr[6667]: elapsed: 5.7 s                                                                        total:  16.0 M (2.8 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++#033[0m----------| 10.4 MiB/13.7 MiB
Aug 28 16:14:41 ctl01 ctr[6667]: elapsed: 5.8 s                                                                        total:  16.2 M (2.8 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++++#033[0m--------| 11.2 MiB/13.7 MiB
Aug 28 16:14:41 ctl01 ctr[6667]: elapsed: 5.9 s                                                                        total:  17.0 M (2.9 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++++++#033[0m------| 11.8 MiB/13.7 MiB
Aug 28 16:14:41 ctl01 ctr[6667]: elapsed: 6.0 s                                                                        total:  17.6 M (2.9 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m++++++++++++++++++++++++++++++++++#033[0m----| 12.3 MiB/13.7 MiB
Aug 28 16:14:41 ctl01 ctr[6667]: elapsed: 6.1 s                                                                        total:  18.2 M (3.0 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++++++++++#033[0m---| 12.8 MiB/13.7 MiB
Aug 28 16:14:41 ctl01 ctr[6667]: elapsed: 6.2 s                                                                        total:  18.6 M (3.0 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        downloading    |#033[32m+++++++++++++++++++++++++++++++++++++#033[0m-| 13.4 MiB/13.7 MiB
Aug 28 16:14:41 ctl01 ctr[6667]: elapsed: 6.3 s                                                                        total:  19.2 M (3.1 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/node:v3.3.2: resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: manifest-sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18:     done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:3f90cdf570685ae358ac0456a9d07fffc96858427c71d754f82e222d96f1c683:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:b788fa0813576d69f1efd12e893b97db7d007b5b93ca1cce6663f96ba1ab6488:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: config-sha256:4e9be81e3a5948d40df6358fdae2cc0dde85a0085723666c50ee7d15427a9b48:       done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:37159c5154b88277f12fe9aa20d728ca5c92fd38e6e707660ee27eef281de923:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: layer-sha256:798a8ef97f6bac88a42e34bf15d8e412242025372c4f567590579ba37d381b2c:        done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 28 16:14:41 ctl01 ctr[6667]: elapsed: 6.4 s                                                                        total:  19.2 M (3.0 MiB/s)
Aug 28 16:14:41 ctl01 ctr[6667]: unpacking linux/amd64 sha256:4b3e3750deeb97cf6f68e5d021f60891a0562f7412efdb545599e6ea505eaf18...
Aug 28 16:14:41 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190828161441852956
Aug 28 16:14:41 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 6707
Aug 28 16:14:41 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161441852956
Aug 28 16:14:43 ctl01 ctr[6667]: done
Aug 28 16:14:43 ctl01 systemd[1]: Started calico-node.
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'calico-node.service'] in directory '/root'
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
Aug 28 16:14:43 ctl01 containerd[6590]: time="2019-08-28T16:14:43.762033900Z" level=info msg="shim containerd-shim started" address="/containerd-shim/default/calico-node/shim.sock" debug=false pid=6736
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'calico-node.service'] in directory '/root'
Aug 28 16:14:43 ctl01 systemd[1]: Started /bin/systemctl enable calico-node.service.
Aug 28 16:14:43 ctl01 systemd[1]: Reloading.
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.878 [INFO][9] startup.go 264: Early log level set to info
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.879 [INFO][9] startup.go 280: Using NODENAME environment for node name
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.879 [INFO][9] startup.go 292: Determined node name: ctl01
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.937 [INFO][9] startup.go 105: Skipping datastore connection test
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.939 [INFO][9] startup.go 365: Building new node resource Name="ctl01"
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.939 [INFO][9] startup.go 380: Initialize BGP data
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.939 [INFO][9] startup.go 474: Using IPv4 address from environment: IP=172.16.10.36
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.942 [INFO][9] startup.go 507: IPv4 address 172.16.10.36 discovered on interface br-mgmt
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.942 [INFO][9] startup.go 450: Node IPv4 changed, will check for conflicts
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.943 [INFO][9] startup.go 640: Using AS number specified in environment (AS=64512)
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] {'calico-node': True}
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Completed state [calico-node] at time 16:14:43.959404 duration_in_ms=9086.105
Aug 28 16:14:43 ctl01 ctr[6712]: 2019-08-28 16:14:43.960 [INFO][9] startup.go 189: Using node name: ctl01
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Running state [curl] at time 16:14:43.961384
Aug 28 16:14:43 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [curl]
Aug 28 16:14:44 ctl01 ctr[6712]: Calico node started successfully
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [curl] at time 16:14:44.153983 duration_in_ms=192.599
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Running state [git] at time 16:14:44.154318
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [git]
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [git] at time 16:14:44.164261 duration_in_ms=9.943
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Running state [apt-transport-https] at time 16:14:44.164523
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [apt-transport-https]
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [apt-transport-https] at time 16:14:44.174010 duration_in_ms=9.486
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Running state [python-apt] at time 16:14:44.174254
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [python-apt]
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [python-apt] at time 16:14:44.186400 duration_in_ms=12.146
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Running state [socat] at time 16:14:44.186650
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [socat]
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
Aug 28 16:14:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'socat'] in directory '/root'
Aug 28 16:14:44 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install socat.
Aug 28 16:14:45 ctl01 kernel: [  175.007466] Netfilter messages via NETLINK v0.30.
Aug 28 16:14:45 ctl01 kernel: [  175.015169] ip_set: protocol 6
Aug 28 16:14:45 ctl01 kernel: [  175.172423] ip6_tables: (C) 2000-2006 Netfilter Core Team
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Made the following changes:
Aug 28 16:14:48 ctl01 salt-minion[4490]: 'socat' changed from 'absent' to '1.7.3.2-2ubuntu2'
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [socat] at time 16:14:48.219371 duration_in_ms=4032.719
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Running state [openssl] at time 16:14:48.230738
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [openssl]
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [openssl] at time 16:14:48.922354 duration_in_ms=691.617
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Running state [conntrack] at time 16:14:48.922662
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [conntrack]
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
Aug 28 16:14:48 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'conntrack'] in directory '/root'
Aug 28 16:14:48 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install conntrack.
Aug 28 16:14:52 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:14:53 ctl01 salt-minion[4490]: [INFO    ] Made the following changes:
Aug 28 16:14:53 ctl01 salt-minion[4490]: 'conntrack' changed from 'absent' to '1:1.4.4+snapshot20161117-6ubuntu2'
Aug 28 16:14:53 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:14:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [conntrack] at time 16:14:53.098805 duration_in_ms=4176.142
Aug 28 16:14:53 ctl01 salt-minion[4490]: [INFO    ] Running state [nfs-common] at time 16:14:53.106778
Aug 28 16:14:53 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [nfs-common]
Aug 28 16:14:53 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
Aug 28 16:14:53 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'nfs-common'] in directory '/root'
Aug 28 16:14:53 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install nfs-common.
Aug 28 16:14:56 ctl01 systemd[1]: Reloading.
Aug 28 16:14:57 ctl01 systemd[1]: message repeated 4 times: [ Reloading.]
Aug 28 16:14:57 ctl01 systemd[1]: Listening on RPCbind Server Activation Socket.
Aug 28 16:14:57 ctl01 systemd[1]: Starting RPC bind portmap service...
Aug 28 16:14:57 ctl01 systemd[1]: Started RPC bind portmap service.
Aug 28 16:14:57 ctl01 systemd[1]: Reached target RPC Port Mapper.
Aug 28 16:14:58 ctl01 systemd[1]: Reloading.
Aug 28 16:14:59 ctl01 systemd[1]: message repeated 4 times: [ Reloading.]
Aug 28 16:15:02 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:15:02 ctl01 salt-minion[4490]: [INFO    ] Made the following changes:
Aug 28 16:15:02 ctl01 salt-minion[4490]: 'keyutils' changed from 'absent' to '1.5.9-9.2ubuntu2'
Aug 28 16:15:02 ctl01 salt-minion[4490]: 'nfs-common' changed from 'absent' to '1:1.3.4-2.1ubuntu5.2'
Aug 28 16:15:02 ctl01 salt-minion[4490]: 'rpcbind' changed from 'absent' to '0.2.3-0.6'
Aug 28 16:15:02 ctl01 salt-minion[4490]: 'libtirpc1' changed from 'absent' to '0.2.5-1.2ubuntu0.1'
Aug 28 16:15:02 ctl01 salt-minion[4490]: 'nfs-client' changed from 'absent' to '1'
Aug 28 16:15:02 ctl01 salt-minion[4490]: 'libnfsidmap2' changed from 'absent' to '0.25-5.1'
Aug 28 16:15:02 ctl01 salt-minion[4490]: 'portmap' changed from 'absent' to '1'
Aug 28 16:15:02 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:15:02 ctl01 salt-minion[4490]: [INFO    ] Completed state [nfs-common] at time 16:15:02.260359 duration_in_ms=9153.577
Aug 28 16:15:02 ctl01 salt-minion[4490]: [INFO    ] Running state [cifs-utils] at time 16:15:02.276323
Aug 28 16:15:02 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [cifs-utils]
Aug 28 16:15:03 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
Aug 28 16:15:03 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'cifs-utils'] in directory '/root'
Aug 28 16:15:03 ctl01 systemd[1]: Started /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install cifs-utils.
Aug 28 16:15:11 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190828161511972210
Aug 28 16:15:11 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 9238
Aug 28 16:15:12 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161511972210
Aug 28 16:15:13 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:15:13 ctl01 salt-minion[4490]: [INFO    ] Made the following changes:
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python2.7-ldb' changed from 'absent' to '1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python-ldb' changed from 'absent' to '2:1.2.3-1ubuntu0.1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libtdb1' changed from 'absent' to '1.3.15-2'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libavahi-common3' changed from 'absent' to '0.7-3.1ubuntu1.2'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python2.7-talloc' changed from 'absent' to '1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libavahi-client3' changed from 'absent' to '0.7-3.1ubuntu1.2'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libwbclient0' changed from 'absent' to '2:4.7.6+dfsg~ubuntu-0ubuntu2.11'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libavahi-common-data' changed from 'absent' to '0.7-3.1ubuntu1.2'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libcups2' changed from 'absent' to '2.2.7-1ubuntu2.7'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'cifs-utils' changed from 'absent' to '2:6.8-1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'samba-common' changed from 'absent' to '2:4.7.6+dfsg~ubuntu-0ubuntu2.11'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python2.7-tdb' changed from 'absent' to '1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'samba-libs' changed from 'absent' to '2:4.7.6+dfsg~ubuntu-0ubuntu2.11'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libldb1' changed from 'absent' to '2:1.2.3-1ubuntu0.1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libtevent0' changed from 'absent' to '0.9.34-1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python-talloc' changed from 'absent' to '2.1.10-2ubuntu1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'samba-common-bin' changed from 'absent' to '2:4.7.6+dfsg~ubuntu-0ubuntu2.11'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python-samba' changed from 'absent' to '2:4.7.6+dfsg~ubuntu-0ubuntu2.11'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libtalloc2' changed from 'absent' to '2.1.10-2ubuntu1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python2.7-samba' changed from 'absent' to '1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'libjansson4' changed from 'absent' to '2.11-1'
Aug 28 16:15:13 ctl01 salt-minion[4490]: 'python-tdb' changed from 'absent' to '1.3.15-2'
Aug 28 16:15:13 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:15:13 ctl01 salt-minion[4490]: [INFO    ] Completed state [cifs-utils] at time 16:15:13.405786 duration_in_ms=11129.461
Aug 28 16:15:13 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/hyperkube] at time 16:15:13.411181
Aug 28 16:15:13 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/bin/hyperkube]
Aug 28 16:15:41 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190828161541990032
Aug 28 16:15:42 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 9317
Aug 28 16:15:42 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161541990032
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:06 ctl01 salt-minion[4490]: New file
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/hyperkube] at time 16:16:06.227626 duration_in_ms=52816.444
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/kubectl] at time 16:16:06.228717
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] Executing state file.symlink for [/usr/bin/kubectl]
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] {'new': '/usr/bin/kubectl'}
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/kubectl] at time 16:16:06.290363 duration_in_ms=61.646
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] Running state [/tmp/crictl] at time 16:16:06.293624
Aug 28 16:16:06 ctl01 salt-minion[4490]: [INFO    ] Executing state archive.extracted for [/tmp/crictl]
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', 'xz', '-f', '/var/cache/salt/minion/extrn_files/base/github.com/kubernetes-sigs/cri-tools/releases/download/v1.12.0/crictl-v1.12.0-linux-amd64.tar.gz'] in directory '/tmp/crictl/'
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] {'extracted_files': 'no tar output so far', 'directories_created': ['/tmp/crictl/']}
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Completed state [/tmp/crictl] at time 16:16:08.706157 duration_in_ms=2412.528
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/local/bin/crictl] at time 16:16:08.707621
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/local/bin/crictl]
Aug 28 16:16:08 ctl01 salt-minion[4490]: [WARNING ] Use of argument owner found, "owner" is invalid, please use "user"
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:08 ctl01 salt-minion[4490]: New file
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/local/bin/crictl] at time 16:16:08.989144 duration_in_ms=281.523
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/crictl.yaml] at time 16:16:08.989459
Aug 28 16:16:08 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/crictl.yaml]
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:09 ctl01 salt-minion[4490]: New file
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/crictl.yaml] at time 16:16:09.018245 duration_in_ms=28.786
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/criproxy] at time 16:16:09.018493
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/criproxy]
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] File /etc/criproxy is not present
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/criproxy] at time 16:16:09.019323 duration_in_ms=0.83
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Running state [criproxy] at time 16:16:09.604890
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing state service.dead for [criproxy]
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'criproxy.service', '-n', '0'] in directory '/root'
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] The named service criproxy is not available
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [criproxy] at time 16:16:09.623389 duration_in_ms=18.499
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kubelet.service] at time 16:16:09.623757
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kubelet.service]
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kubelet.service'
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:09 ctl01 salt-minion[4490]: New file
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kubelet.service] at time 16:16:09.981418 duration_in_ms=357.661
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/config] at time 16:16:09.981756
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/kubernetes/config]
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/config is not present
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/config] at time 16:16:09.982679 duration_in_ms=0.924
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kubelet] at time 16:16:09.982938
Aug 28 16:16:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/kubelet]
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kubelet/default.master'
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:10 ctl01 salt-minion[4490]: New file
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kubelet] at time 16:16:10.920704 duration_in_ms=937.766
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/kubelet.kubeconfig] at time 16:16:10.921035
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/kubelet.kubeconfig]
Aug 28 16:16:10 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kubelet/kubelet.kubeconfig.master'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:11 ctl01 salt-minion[4490]: New file
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/kubelet.kubeconfig] at time 16:16:11.553365 duration_in_ms=632.329
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/manifests] at time 16:16:11.553845
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/etc/kubernetes/manifests]
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] {'/etc/kubernetes/manifests': 'New Dir'}
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/manifests] at time 16:16:11.556984 duration_in_ms=3.139
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Running state [kubelet] at time 16:16:11.560728
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [kubelet]
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kubelet.service', '-n', '0'] in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 systemd[1]: Started /bin/systemctl start kubelet.service.
Aug 28 16:16:11 ctl01 systemd[1]: Started Kubernetes Kubelet Server.
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 systemd[1]: Started /bin/systemctl enable kubelet.service.
Aug 28 16:16:11 ctl01 systemd[1]: Reloading.
Aug 28 16:16:11 ctl01 kubelet[9620]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:11 ctl01 kubelet[9620]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:11 ctl01 kubelet[9620]: Flag --allow-privileged has been deprecated, will be removed in a future version
Aug 28 16:16:11 ctl01 kubelet[9620]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:11 ctl01 kubelet[9620]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:11 ctl01 kubelet[9620]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844177    9620 flags.go:33] FLAG: --address="172.16.10.36"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844235    9620 flags.go:33] FLAG: --allow-privileged="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844243    9620 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844256    9620 flags.go:33] FLAG: --alsologtostderr="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844267    9620 flags.go:33] FLAG: --anonymous-auth="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844274    9620 flags.go:33] FLAG: --application-metrics-count-limit="100"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844280    9620 flags.go:33] FLAG: --authentication-token-webhook="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844286    9620 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844294    9620 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844300    9620 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844306    9620 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844312    9620 flags.go:33] FLAG: --azure-container-registry-config=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844317    9620 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844324    9620 flags.go:33] FLAG: --bootstrap-checkpoint-path=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844329    9620 flags.go:33] FLAG: --bootstrap-kubeconfig=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844334    9620 flags.go:33] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844340    9620 flags.go:33] FLAG: --cgroup-driver="cgroupfs"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844346    9620 flags.go:33] FLAG: --cgroup-root=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844351    9620 flags.go:33] FLAG: --cgroups-per-qos="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844356    9620 flags.go:33] FLAG: --chaos-chance="0"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844364    9620 flags.go:33] FLAG: --client-ca-file=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844369    9620 flags.go:33] FLAG: --cloud-config=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844374    9620 flags.go:33] FLAG: --cloud-provider=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844379    9620 flags.go:33] FLAG: --cluster-dns="[10.254.0.10]"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844387    9620 flags.go:33] FLAG: --cluster-domain="cluster.local"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844393    9620 flags.go:33] FLAG: --cni-bin-dir="/opt/cni/bin"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844399    9620 flags.go:33] FLAG: --cni-conf-dir="/etc/cni/net.d"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844404    9620 flags.go:33] FLAG: --config=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844410    9620 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844416    9620 flags.go:33] FLAG: --container-log-max-files="5"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844429    9620 flags.go:33] FLAG: --container-log-max-size="10Mi"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844434    9620 flags.go:33] FLAG: --container-runtime="remote"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844440    9620 flags.go:33] FLAG: --container-runtime-endpoint="unix:///run/containerd/containerd.sock"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844451    9620 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844457    9620 flags.go:33] FLAG: --containerized="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844463    9620 flags.go:33] FLAG: --contention-profiling="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844468    9620 flags.go:33] FLAG: --cpu-cfs-quota="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844473    9620 flags.go:33] FLAG: --cpu-cfs-quota-period="100ms"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844479    9620 flags.go:33] FLAG: --cpu-manager-policy="none"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844485    9620 flags.go:33] FLAG: --cpu-manager-reconcile-period="10s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844490    9620 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844496    9620 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844502    9620 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844507    9620 flags.go:33] FLAG: --docker-only="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844513    9620 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844518    9620 flags.go:33] FLAG: --docker-tls="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844524    9620 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844530    9620 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844535    9620 flags.go:33] FLAG: --docker-tls-key="key.pem"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844540    9620 flags.go:33] FLAG: --dynamic-config-dir=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844548    9620 flags.go:33] FLAG: --enable-controller-attach-detach="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844553    9620 flags.go:33] FLAG: --enable-debugging-handlers="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844559    9620 flags.go:33] FLAG: --enable-load-reader="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844564    9620 flags.go:33] FLAG: --enable-server="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844569    9620 flags.go:33] FLAG: --enforce-node-allocatable="[pods]"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844580    9620 flags.go:33] FLAG: --event-burst="10"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844586    9620 flags.go:33] FLAG: --event-qps="5"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844591    9620 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844596    9620 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844602    9620 flags.go:33] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844620    9620 flags.go:33] FLAG: --eviction-max-pod-grace-period="0"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844626    9620 flags.go:33] FLAG: --eviction-minimum-reclaim=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844638    9620 flags.go:33] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844644    9620 flags.go:33] FLAG: --eviction-soft=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844650    9620 flags.go:33] FLAG: --eviction-soft-grace-period=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844660    9620 flags.go:33] FLAG: --exit-on-lock-contention="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844666    9620 flags.go:33] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844671    9620 flags.go:33] FLAG: --experimental-bootstrap-kubeconfig=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844676    9620 flags.go:33] FLAG: --experimental-check-node-capabilities-before-mount="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844682    9620 flags.go:33] FLAG: --experimental-dockershim="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844687    9620 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844693    9620 flags.go:33] FLAG: --experimental-fail-swap-on="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844698    9620 flags.go:33] FLAG: --experimental-kernel-memcg-notification="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844703    9620 flags.go:33] FLAG: --experimental-mounter-path=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844709    9620 flags.go:33] FLAG: --fail-swap-on="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844714    9620 flags.go:33] FLAG: --feature-gates=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844722    9620 flags.go:33] FLAG: --file-check-frequency="20s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844727    9620 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844733    9620 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844739    9620 flags.go:33] FLAG: --healthz-bind-address="127.0.0.1"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844745    9620 flags.go:33] FLAG: --healthz-port="10248"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844750    9620 flags.go:33] FLAG: --help="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844756    9620 flags.go:33] FLAG: --host-ipc-sources="[*]"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844763    9620 flags.go:33] FLAG: --host-network-sources="[*]"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844773    9620 flags.go:33] FLAG: --host-pid-sources="[*]"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844779    9620 flags.go:33] FLAG: --hostname-override="ctl01"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844785    9620 flags.go:33] FLAG: --housekeeping-interval="10s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844790    9620 flags.go:33] FLAG: --http-check-frequency="20s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844796    9620 flags.go:33] FLAG: --image-gc-high-threshold="85"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844802    9620 flags.go:33] FLAG: --image-gc-low-threshold="80"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844807    9620 flags.go:33] FLAG: --image-pull-progress-deadline="1m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844813    9620 flags.go:33] FLAG: --image-service-endpoint=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844818    9620 flags.go:33] FLAG: --iptables-drop-bit="15"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844824    9620 flags.go:33] FLAG: --iptables-masquerade-bit="14"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844835    9620 flags.go:33] FLAG: --keep-terminated-pod-volumes="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844840    9620 flags.go:33] FLAG: --kube-api-burst="10"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844846    9620 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844856    9620 flags.go:33] FLAG: --kube-api-qps="5"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844862    9620 flags.go:33] FLAG: --kube-reserved=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844868    9620 flags.go:33] FLAG: --kube-reserved-cgroup=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844873    9620 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/kubelet.kubeconfig"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844879    9620 flags.go:33] FLAG: --kubelet-cgroups=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844884    9620 flags.go:33] FLAG: --lock-file=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844889    9620 flags.go:33] FLAG: --log-backtrace-at=":0"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844896    9620 flags.go:33] FLAG: --log-cadvisor-usage="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844902    9620 flags.go:33] FLAG: --log-dir=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844907    9620 flags.go:33] FLAG: --log-file=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844912    9620 flags.go:33] FLAG: --log-flush-frequency="5s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844918    9620 flags.go:33] FLAG: --logtostderr="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844923    9620 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844929    9620 flags.go:33] FLAG: --make-iptables-util-chains="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844935    9620 flags.go:33] FLAG: --manifest-url=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844940    9620 flags.go:33] FLAG: --manifest-url-header=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844949    9620 flags.go:33] FLAG: --master-service-namespace="default"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844954    9620 flags.go:33] FLAG: --max-open-files="1000000"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844962    9620 flags.go:33] FLAG: --max-pods="110"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844967    9620 flags.go:33] FLAG: --maximum-dead-containers="-1"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844973    9620 flags.go:33] FLAG: --maximum-dead-containers-per-container="1"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844978    9620 flags.go:33] FLAG: --minimum-container-ttl-duration="0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844984    9620 flags.go:33] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844989    9620 flags.go:33] FLAG: --network-plugin="cni"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.844994    9620 flags.go:33] FLAG: --network-plugin-mtu="0"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845000    9620 flags.go:33] FLAG: --node-ip="172.16.10.36"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845005    9620 flags.go:33] FLAG: --node-labels="node-role.kubernetes.io/master=true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845015    9620 flags.go:33] FLAG: --node-status-max-images="50"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845020    9620 flags.go:33] FLAG: --node-status-update-frequency="10s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845030    9620 flags.go:33] FLAG: --non-masquerade-cidr="10.0.0.0/8"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845035    9620 flags.go:33] FLAG: --oom-score-adj="-999"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845041    9620 flags.go:33] FLAG: --pod-cidr=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845051    9620 flags.go:33] FLAG: --pod-infra-container-image="docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845060    9620 flags.go:33] FLAG: --pod-manifest-path="/etc/kubernetes/manifests"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845066    9620 flags.go:33] FLAG: --pod-max-pids="-1"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845072    9620 flags.go:33] FLAG: --pods-per-core="0"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845078    9620 flags.go:33] FLAG: --port="10250"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845083    9620 flags.go:33] FLAG: --protect-kernel-defaults="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845089    9620 flags.go:33] FLAG: --provider-id=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845094    9620 flags.go:33] FLAG: --qos-reserved=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845100    9620 flags.go:33] FLAG: --read-only-port="10255"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845105    9620 flags.go:33] FLAG: --really-crash-for-testing="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845110    9620 flags.go:33] FLAG: --redirect-container-streaming="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845116    9620 flags.go:33] FLAG: --register-node="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845121    9620 flags.go:33] FLAG: --register-schedulable="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845126    9620 flags.go:33] FLAG: --register-with-taints=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845133    9620 flags.go:33] FLAG: --registry-burst="10"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845139    9620 flags.go:33] FLAG: --registry-qps="5"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845144    9620 flags.go:33] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845150    9620 flags.go:33] FLAG: --root-dir="/var/lib/kubelet"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845156    9620 flags.go:33] FLAG: --rotate-certificates="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845161    9620 flags.go:33] FLAG: --rotate-server-certificates="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845166    9620 flags.go:33] FLAG: --runonce="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845171    9620 flags.go:33] FLAG: --runtime-cgroups=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845176    9620 flags.go:33] FLAG: --runtime-request-timeout="2m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845182    9620 flags.go:33] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845188    9620 flags.go:33] FLAG: --serialize-image-pulls="true"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845193    9620 flags.go:33] FLAG: --stderrthreshold="2"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845199    9620 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845205    9620 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845210    9620 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845221    9620 flags.go:33] FLAG: --storage-driver-password="root"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845226    9620 flags.go:33] FLAG: --storage-driver-secure="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845232    9620 flags.go:33] FLAG: --storage-driver-table="stats"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845242    9620 flags.go:33] FLAG: --storage-driver-user="root"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845247    9620 flags.go:33] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845253    9620 flags.go:33] FLAG: --sync-frequency="1m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845259    9620 flags.go:33] FLAG: --system-cgroups=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845264    9620 flags.go:33] FLAG: --system-reserved=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845270    9620 flags.go:33] FLAG: --system-reserved-cgroup=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845275    9620 flags.go:33] FLAG: --tls-cert-file=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845280    9620 flags.go:33] FLAG: --tls-cipher-suites="[]"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845290    9620 flags.go:33] FLAG: --tls-min-version=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845295    9620 flags.go:33] FLAG: --tls-private-key-file=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845300    9620 flags.go:33] FLAG: --v="2"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845306    9620 flags.go:33] FLAG: --version="false"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845314    9620 flags.go:33] FLAG: --vmodule=""
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845320    9620 flags.go:33] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845327    9620 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845362    9620 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:11 ctl01 kubelet[9620]: W0828 16:16:11.845381    9620 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
Aug 28 16:16:11 ctl01 kubelet[9620]: W0828 16:16:11.845393    9620 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
Aug 28 16:16:11 ctl01 kubelet[9620]: W0828 16:16:11.845405    9620 server.go:182] Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead
Aug 28 16:16:11 ctl01 kubelet[9620]: I0828 16:16:11.845463    9620 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:11 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:16:11 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] {'kubelet': True}
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Completed state [kubelet] at time 16:16:11.938213 duration_in_ms=377.484
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/logrotate.d/kubernetes] at time 16:16:11.938610
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/logrotate.d/kubernetes]
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/logrotate'
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:11 ctl01 salt-minion[4490]: New file
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/logrotate.d/kubernetes] at time 16:16:11.966629 duration_in_ms=28.018
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Running state [/opt/cni/bin] at time 16:16:11.966880
Aug 28 16:16:11 ctl01 salt-minion[4490]: [INFO    ] Executing state archive.extracted for [/opt/cni/bin]
Aug 28 16:16:12 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190828161612207694
Aug 28 16:16:12 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 9698
Aug 28 16:16:12 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161612207694
Aug 28 16:16:12 ctl01 systemd[1]: Started Kubernetes systemd probe.
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.695486    9620 mount_linux.go:180] Detected OS with systemd
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.695594    9620 server.go:407] Version: v1.13.5-3+98374c02d2d8c1
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.695667    9620 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.695751    9620 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:12 ctl01 kubelet[9620]: W0828 16:16:12.695777    9620 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
Aug 28 16:16:12 ctl01 kubelet[9620]: W0828 16:16:12.695796    9620 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.695976    9620 plugins.go:103] No cloud provider specified.
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.695999    9620 server.go:523] No cloud provider specified: "" from the config file: ""
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.703157    9620 manager.go:155] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.704590    9620 fs.go:142] Filesystem UUIDs: map[2019-08-28-18-06-57-00:/dev/sr0 9430be73-33d9-4f4e-beff-4a37ae3601c5:/dev/vda1 C738-8EF9:/dev/vda15]
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.704634    9620 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /dev/vda1:{mountpoint:/ major:252 minor:1 fsType:ext4 blockSize:0}]
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.709931    9620 manager.go:229] Machine: {NumCores:8 CpuFrequency:2799994 MemoryCapacity:14704947200 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:41672cdd030f48a18c2eb70f309172a9 SystemUUID:41672CDD-030F-48A1-8C2E-B70F309172A9 BootID:fefbdff1-5a6b-4a85-9190-6007d0de5d83 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:24 Capacity:1470496768 Type:vfs Inodes:1795037 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:103880232960 Type:vfs Inodes:12902400 HasInodes:true}] DiskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:br-mgmt MacAddress:52:54:00:2c:cd:bc Speed:0 Mtu:9000} {Name:ens3 MacAddress:52:54:00:3f:9b:0c Speed:-1 Mtu:9000} {Name:ens4 MacAddress:52:54:00:2c:cd:bc Speed:-1 Mtu:9000} {Name:ens5 MacAddress:52:54:00:9e:c5:f6 Speed:-1 Mtu:9000} {Name:ens5.1000 MacAddress:52:54:00:9e:c5:f6 Speed:-1 Mtu:9000} {Name:ens6 MacAddress:52:54:00:14:c4:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:14704947200 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:2 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:3 Memory:0 Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:4 Memory:0 Cores:[{Id:0 Threads:[4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:5 Memory:0 Cores:[{Id:0 Threads:[5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:6 Memory:0 Cores:[{Id:0 Threads:[6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:7 Memory:0 Cores:[{Id:0 Threads:[7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.710451    9620 manager.go:235] Version: {KernelVersion:4.15.0-58-generic ContainerOsVersion:Ubuntu 18.04.3 LTS DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:}
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.710602    9620 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.710959    9620 container_manager_linux.go:248] container manager verified user specified cgroup-root exists: []
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.710983    9620 container_manager_linux.go:253] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms}
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.711129    9620 container_manager_linux.go:272] Creating device plugin manager: true
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.711142    9620 manager.go:109] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.711250    9620 state_mem.go:36] [cpumanager] initializing new in-memory state store
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.723281    9620 server.go:941] Using root directory: /var/lib/kubelet
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.723323    9620 kubelet.go:281] Adding pod path: /etc/kubernetes/manifests
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.723365    9620 file.go:68] Watching path "/etc/kubernetes/manifests"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.723390    9620 kubelet.go:306] Watching apiserver
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.724180    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.724270    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.724325    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.734102    9620 kuberuntime_manager.go:198] Container runtime containerd initialized, version: 1.2.6-0ubuntu1~18.04.2, apiVersion: v1alpha2
Aug 28 16:16:12 ctl01 kubelet[9620]: W0828 16:16:12.734804    9620 probe.go:271] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735052    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735077    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/empty-dir"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735088    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735098    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/git-repo"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735113    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/host-path"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735123    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/nfs"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735133    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/secret"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735143    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/iscsi"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735154    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735169    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735178    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735187    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/quobyte"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735196    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/cephfs"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735213    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/downward-api"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735253    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735267    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/flocker"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735277    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735286    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/configmap"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735296    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735323    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735337    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735348    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/projected"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735366    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735376    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735386    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/local-volume"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735396    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735409    9620 plugins.go:547] Loaded volume plugin "kubernetes.io/csi"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.735667    9620 server.go:999] Started kubelet
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.736051    9620 server.go:137] Starting to listen on 172.16.10.36:10250
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.736410    9620 server.go:157] Starting to listen read-only on 172.16.10.36:10255
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.738393    9620 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.738444    9620 status_manager.go:152] Starting to sync pod status with apiserver
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.738465    9620 kubelet.go:1829] Starting kubelet main sync loop.
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.738479    9620 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg has yet to be successful]
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.738693    9620 cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.738715    9620 desired_state_of_world_populator.go:130] Desired state populator starts to run
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.738728    9620 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.738699    9620 volume_manager.go:246] The desired_state_of_world populator starts
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.738860    9620 volume_manager.go:248] Starting Kubelet Volume Manager
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.739065    9620 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.739340    9620 server.go:333] Adding debug handlers to kubelet server.
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.746364    9620 factory.go:136] Registering containerd factory
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.747584    9620 factory.go:54] Registering systemd factory
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.747983    9620 factory.go:97] Registering Raw factory
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.748361    9620 manager.go:1222] Started watching for new ooms in manager
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.750447    9620 manager.go:365] Starting recovery of all containers
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.800839    9620 manager.go:370] Recovery completed
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.838648    9620 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet]
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.838968    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.838981    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.839350    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.840453    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.840489    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.840505    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.840528    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.841012    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.868784    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.869013    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.869718    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.869756    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.869770    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.869836    9620 cpu_manager.go:155] [cpumanager] starting with none policy
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.869845    9620 cpu_manager.go:156] [cpumanager] reconciling every 10s
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.869861    9620 policy_none.go:42] [cpumanager] none policy: Start
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.870257    9620 container_manager_linux.go:376] Updating kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.870352    9620 container_manager_linux.go:376] Updating kernel flag: kernel/panic, expected value: 10, actual value: 60
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.870416    9620 container_manager_linux.go:376] Updating kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.884518    9620 manager.go:196] Starting Device Plugin manager
Aug 28 16:16:12 ctl01 kubelet[9620]: W0828 16:16:12.884566    9620 manager.go:537] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.884680    9620 manager.go:231] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 28 16:16:12 ctl01 kubelet[9620]: I0828 16:16:12.884729    9620 plugin_watcher.go:90] Plugin Watcher Start at /var/lib/kubelet/plugins_registry
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.885485    9620 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
Aug 28 16:16:12 ctl01 kubelet[9620]: E0828 16:16:12.939181    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.038875    9620 kubelet.go:1908] SyncLoop (ADD, "file"): ""
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.039318    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.041234    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.041494    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.042466    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.042611    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.042717    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.042847    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.043235    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.139567    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.239841    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.339995    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.440166    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.443530    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.443872    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.444977    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.445017    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.445032    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: I0828 16:16:13.445062    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.445487    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.540312    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.640495    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.725208    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.726085    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.727030    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.740760    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.840957    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:13 ctl01 kubelet[9620]: E0828 16:16:13.941183    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.041423    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.141770    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.241931    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: I0828 16:16:14.245670    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:14 ctl01 kubelet[9620]: I0828 16:16:14.245936    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:14 ctl01 kubelet[9620]: I0828 16:16:14.247052    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:14 ctl01 kubelet[9620]: I0828 16:16:14.247183    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:14 ctl01 kubelet[9620]: I0828 16:16:14.247475    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:14 ctl01 kubelet[9620]: I0828 16:16:14.247669    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.248195    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.342081    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.442227    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.542358    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.642506    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.725760    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.726709    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.727785    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.742617    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.842772    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:14 ctl01 kubelet[9620]: E0828 16:16:14.942958    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.043137    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.143418    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.243555    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.343744    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.443954    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.544198    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.644474    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.726522    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.727498    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.728531    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.744633    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.844837    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:15 ctl01 kubelet[9620]: I0828 16:16:15.848511    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:15 ctl01 kubelet[9620]: I0828 16:16:15.849034    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:15 ctl01 kubelet[9620]: I0828 16:16:15.850284    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:15 ctl01 kubelet[9620]: I0828 16:16:15.850331    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:15 ctl01 kubelet[9620]: I0828 16:16:15.850348    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:15 ctl01 kubelet[9620]: I0828 16:16:15.850378    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.850617    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:15 ctl01 kubelet[9620]: E0828 16:16:15.945196    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.045417    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.145578    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.245780    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.345976    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.446237    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.546524    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.646834    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.727230    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.728326    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.729351    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.747031    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.847167    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:16 ctl01 kubelet[9620]: E0828 16:16:16.947335    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.047614    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.147770    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.247950    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.348113    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.448368    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.548551    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.648772    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.728001    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.728773    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.729824    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.748994    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.849164    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:17 ctl01 kubelet[9620]: E0828 16:16:17.949328    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.049478    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.149663    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.249860    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.350033    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', 'xz', '-f', '/var/cache/salt/minion/extrn_files/base/docker-prod-local.artifactory.mirantis.com/artifactory/binary-prod-local/mirantis/kubernetes/containernetworking-plugins/containernetworking-plugins_v0.7.2-173-g8db2808.tar.gz'] in directory '/opt/cni/bin/'
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.450209    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.550369    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.650601    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.728650    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.729617    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.730753    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.750767    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.850936    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:18 ctl01 kubelet[9620]: E0828 16:16:18.951102    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 kubelet[9620]: I0828 16:16:19.050994    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.051233    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 kubelet[9620]: I0828 16:16:19.051535    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:19 ctl01 kubelet[9620]: I0828 16:16:19.052649    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:19 ctl01 kubelet[9620]: I0828 16:16:19.052800    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:19 ctl01 kubelet[9620]: I0828 16:16:19.052927    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:19 ctl01 kubelet[9620]: I0828 16:16:19.053059    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.053651    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] {'extracted_files': 'no tar output so far'}
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/opt/cni/bin] at time 16:16:19.072779 duration_in_ms=7105.893
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/proxy.kubeconfig] at time 16:16:19.073181
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/proxy.kubeconfig]
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-proxy/proxy.kubeconfig'
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.151517    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.251681    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.351891    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.452205    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.552397    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.652575    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.729893    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.730408    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.731657    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.752801    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:19 ctl01 salt-minion[4490]: New file
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/proxy.kubeconfig] at time 16:16:19.770256 duration_in_ms=697.075
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kube-proxy.service] at time 16:16:19.770593
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-proxy.service]
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-proxy.service'
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:19 ctl01 salt-minion[4490]: New file
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kube-proxy.service] at time 16:16:19.806342 duration_in_ms=35.749
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kube-proxy] at time 16:16:19.806610
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/kube-proxy]
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:19 ctl01 salt-minion[4490]: New file
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kube-proxy] at time 16:16:19.810253 duration_in_ms=3.643
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Running state [kube-proxy] at time 16:16:19.812748
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [kube-proxy]
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kube-proxy.service', '-n', '0'] in directory '/root'
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.853086    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:19 ctl01 systemd[1]: Started /bin/systemctl start kube-proxy.service.
Aug 28 16:16:19 ctl01 systemd[1]: Started Kubernetes Kube-Proxy Server.
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:19 ctl01 kubelet[9620]: E0828 16:16:19.953272    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:19 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:20 ctl01 systemd[1]: Started /bin/systemctl enable kube-proxy.service.
Aug 28 16:16:20 ctl01 systemd[1]: Reloading.
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.053443    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067564    9844 flags.go:33] FLAG: --alsologtostderr="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067782    9844 flags.go:33] FLAG: --application-metrics-count-limit="100"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067816    9844 flags.go:33] FLAG: --azure-container-registry-config=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067825    9844 flags.go:33] FLAG: --bind-address="0.0.0.0"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067834    9844 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067840    9844 flags.go:33] FLAG: --cleanup="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067848    9844 flags.go:33] FLAG: --cleanup-iptables="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067854    9844 flags.go:33] FLAG: --cleanup-ipvs="true"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067860    9844 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067871    9844 flags.go:33] FLAG: --cluster-cidr="192.168.0.0/16"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067877    9844 flags.go:33] FLAG: --config=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067883    9844 flags.go:33] FLAG: --config-sync-period="15m0s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067890    9844 flags.go:33] FLAG: --conntrack-max="0"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067929    9844 flags.go:33] FLAG: --conntrack-max-per-core="32768"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067936    9844 flags.go:33] FLAG: --conntrack-min="131072"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067943    9844 flags.go:33] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067949    9844 flags.go:33] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067961    9844 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067968    9844 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067974    9844 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067980    9844 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067986    9844 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067992    9844 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.067997    9844 flags.go:33] FLAG: --docker-only="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068003    9844 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068009    9844 flags.go:33] FLAG: --docker-tls="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068014    9844 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068020    9844 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068025    9844 flags.go:33] FLAG: --docker-tls-key="key.pem"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068031    9844 flags.go:33] FLAG: --enable-load-reader="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068036    9844 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068042    9844 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068048    9844 flags.go:33] FLAG: --feature-gates=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068057    9844 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068064    9844 flags.go:33] FLAG: --healthz-bind-address="0.0.0.0:10256"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068071    9844 flags.go:33] FLAG: --healthz-port="10256"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068077    9844 flags.go:33] FLAG: --help="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068083    9844 flags.go:33] FLAG: --hostname-override=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068089    9844 flags.go:33] FLAG: --housekeeping-interval="10s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068095    9844 flags.go:33] FLAG: --iptables-masquerade-bit="14"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068101    9844 flags.go:33] FLAG: --iptables-min-sync-period="0s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068116    9844 flags.go:33] FLAG: --iptables-sync-period="30s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068122    9844 flags.go:33] FLAG: --ipvs-exclude-cidrs="[]"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068135    9844 flags.go:33] FLAG: --ipvs-min-sync-period="0s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068141    9844 flags.go:33] FLAG: --ipvs-scheduler=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068147    9844 flags.go:33] FLAG: --ipvs-sync-period="30s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068152    9844 flags.go:33] FLAG: --kube-api-burst="10"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068158    9844 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068164    9844 flags.go:33] FLAG: --kube-api-qps="5"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068178    9844 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/proxy.kubeconfig"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068185    9844 flags.go:33] FLAG: --log-backtrace-at=":0"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068193    9844 flags.go:33] FLAG: --log-cadvisor-usage="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068199    9844 flags.go:33] FLAG: --log-dir=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068205    9844 flags.go:33] FLAG: --log-file=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068210    9844 flags.go:33] FLAG: --log-flush-frequency="5s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068216    9844 flags.go:33] FLAG: --logtostderr="true"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068221    9844 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068227    9844 flags.go:33] FLAG: --masquerade-all="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068233    9844 flags.go:33] FLAG: --master=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068238    9844 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068255    9844 flags.go:33] FLAG: --mesos-agent-timeout="10s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068260    9844 flags.go:33] FLAG: --metrics-bind-address="127.0.0.1:10249"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068266    9844 flags.go:33] FLAG: --metrics-port="10249"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068272    9844 flags.go:33] FLAG: --nodeport-addresses="[]"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068285    9844 flags.go:33] FLAG: --oom-score-adj="-999"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068291    9844 flags.go:33] FLAG: --profiling="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068297    9844 flags.go:33] FLAG: --proxy-mode="iptables"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068305    9844 flags.go:33] FLAG: --proxy-port-range=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068321    9844 flags.go:33] FLAG: --resource-container="/kube-proxy"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068327    9844 flags.go:33] FLAG: --skip-headers="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068333    9844 flags.go:33] FLAG: --stderrthreshold="2"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068339    9844 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068345    9844 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068365    9844 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068371    9844 flags.go:33] FLAG: --storage-driver-password="root"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068377    9844 flags.go:33] FLAG: --storage-driver-secure="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068383    9844 flags.go:33] FLAG: --storage-driver-table="stats"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068388    9844 flags.go:33] FLAG: --storage-driver-user="root"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068394    9844 flags.go:33] FLAG: --udp-timeout="250ms"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068399    9844 flags.go:33] FLAG: --v="2"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068405    9844 flags.go:33] FLAG: --version="false"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068414    9844 flags.go:33] FLAG: --vmodule=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068420    9844 flags.go:33] FLAG: --write-config-to=""
Aug 28 16:16:20 ctl01 kube-proxy[9844]: W0828 16:16:20.068431    9844 server.go:198] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.068473    9844 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:20 ctl01 kernel: [  269.830655] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Aug 28 16:16:20 ctl01 kernel: [  269.830699] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Aug 28 16:16:20 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:16:20 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.153647    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
Aug 28 16:16:20 ctl01 salt-minion[4490]: [INFO    ] {'kube-proxy': True}
Aug 28 16:16:20 ctl01 salt-minion[4490]: [INFO    ] Completed state [kube-proxy] at time 16:16:20.194694 duration_in_ms=381.944
Aug 28 16:16:20 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161356768692
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.253859    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.354194    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kernel: [  270.103499] IPVS: ipvs loaded.
Aug 28 16:16:20 ctl01 kernel: [  270.109296] IPVS: [rr] scheduler registered.
Aug 28 16:16:20 ctl01 kernel: [  270.113546] IPVS: [wrr] scheduler registered.
Aug 28 16:16:20 ctl01 kernel: [  270.117786] IPVS: [sh] scheduler registered.
Aug 28 16:16:20 ctl01 kube-proxy[9844]: W0828 16:16:20.386642    9844 node.go:103] Failed to retrieve node info: Get https://172.16.10.36:443/api/v1/nodes/ctl01: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.386674    9844 server_others.go:148] Using iptables Proxier.
Aug 28 16:16:20 ctl01 kube-proxy[9844]: W0828 16:16:20.386762    9844 proxier.go:314] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.386831    9844 server_others.go:178] Tearing down inactive rules.
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.407394    9844 server.go:483] Version: v1.13.5-3+98374c02d2d8c1
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.414861    9844 server.go:509] Running in resource-only container "/kube-proxy"
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.415831    9844 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 262144
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.415930    9844 conntrack.go:52] Setting nf_conntrack_max to 262144
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.416078    9844 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.416151    9844 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.416310    9844 config.go:102] Starting endpoints config controller
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.416332    9844 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.416377    9844 config.go:202] Starting service config controller
Aug 28 16:16:20 ctl01 kube-proxy[9844]: I0828 16:16:20.416403    9844 controller_utils.go:1027] Waiting for caches to sync for service config controller
Aug 28 16:16:20 ctl01 kube-proxy[9844]: E0828 16:16:20.416709    9844 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:20 ctl01 kube-proxy[9844]: E0828 16:16:20.416880    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:20 ctl01 kube-proxy[9844]: E0828 16:16:20.416965    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.454505    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.554794    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.655131    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.730662    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.731685    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.732769    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.755378    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.855667    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:20 ctl01 kubelet[9620]: E0828 16:16:20.956097    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.056379    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.156775    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.257148    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.357574    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kube-proxy[9844]: E0828 16:16:21.417654    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:21 ctl01 kube-proxy[9844]: E0828 16:16:21.418420    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.457937    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command cmd.run with jid 20190828161621502835
Aug 28 16:16:21 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 9960
Aug 28 16:16:21 ctl01 salt-minion[4490]: [INFO    ] Executing command 'calicoctl node status' in directory '/root'
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.558635    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161621502835
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.659078    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.731485    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.732282    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.733190    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.759348    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.808167    9620 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.859514    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:21 ctl01 kubelet[9620]: E0828 16:16:21.959827    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.060156    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command cmd.run with jid 20190828161622147548
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.160351    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 9979
Aug 28 16:16:22 ctl01 salt-minion[4490]: [INFO    ] Executing command 'calicoctl get ippool' in directory '/root'
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.260628    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161622147548
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.360988    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 kube-proxy[9844]: E0828 16:16:22.418962    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:22 ctl01 kube-proxy[9844]: E0828 16:16:22.419844    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.461304    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.561599    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.661961    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.732705    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.733511    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.734574    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.762305    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.862569    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.885640    9620 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
Aug 28 16:16:22 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190828161622927403
Aug 28 16:16:22 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 10000
Aug 28 16:16:22 ctl01 kubelet[9620]: E0828 16:16:22.962818    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.063179    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.163512    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.263821    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.364081    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kube-proxy[9844]: E0828 16:16:23.420225    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:23 ctl01 kube-proxy[9844]: E0828 16:16:23.420999    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.464386    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.564659    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.664868    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.733294    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.734586    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.735549    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.765139    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.865352    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:23 ctl01 kubelet[9620]: E0828 16:16:23.965548    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.065751    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.165965    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.266163    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.366370    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kube-proxy[9844]: E0828 16:16:24.420735    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:24 ctl01 kube-proxy[9844]: E0828 16:16:24.421770    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.466572    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.566733    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.666935    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.733791    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.735039    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.736213    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.767144    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.867421    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:24 ctl01 kubelet[9620]: E0828 16:16:24.967697    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.067874    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.168044    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.268239    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.368384    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kube-proxy[9844]: E0828 16:16:25.421297    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:25 ctl01 kube-proxy[9844]: E0828 16:16:25.422221    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:25 ctl01 kubelet[9620]: I0828 16:16:25.453908    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:25 ctl01 kubelet[9620]: I0828 16:16:25.454122    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:25 ctl01 kubelet[9620]: I0828 16:16:25.454928    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:25 ctl01 kubelet[9620]: I0828 16:16:25.454963    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:25 ctl01 kubelet[9620]: I0828 16:16:25.454976    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:25 ctl01 kubelet[9620]: I0828 16:16:25.454997    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.455276    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.468529    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.568701    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.669028    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.734223    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.735446    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.736582    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.769168    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.869348    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:25 ctl01 kubelet[9620]: E0828 16:16:25.969676    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.069953    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.170222    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.270628    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.370916    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kube-proxy[9844]: E0828 16:16:26.422405    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:26 ctl01 kube-proxy[9844]: E0828 16:16:26.423264    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.471257    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.571564    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.671845    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.735202    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.736578    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.737267    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.772113    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:26 ctl01 kubelet[9620]: E0828 16:16:26.872761    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:26.972905    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.073049    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.173246    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.273391    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.373538    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kube-proxy[9844]: E0828 16:16:27.423095    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:27 ctl01 kube-proxy[9844]: E0828 16:16:27.424213    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.473745    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.574039    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.674251    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.735777    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.737121    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.738327    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.774578    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.874742    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:27 ctl01 kubelet[9620]: E0828 16:16:27.975026    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.075219    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.175361    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.275592    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.375791    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kube-proxy[9844]: E0828 16:16:28.424075    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:28 ctl01 kube-proxy[9844]: E0828 16:16:28.424907    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.476127    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.576510    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.676850    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.736779    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.737918    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.738964    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.777138    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kube-proxy[9844]: E0828 16:16:28.844192    9844 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.877425    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:28 ctl01 kubelet[9620]: E0828 16:16:28.977683    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.077908    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.178188    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.278491    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:16:29 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'etcd/server/setup.sls'
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.378746    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kube-proxy[9844]: E0828 16:16:29.424745    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:29 ctl01 kube-proxy[9844]: E0828 16:16:29.425726    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:29 ctl01 salt-minion[4490]: [INFO    ] Running state [/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16] at time 16:16:29.477200
Aug 28 16:16:29 ctl01 salt-minion[4490]: [INFO    ] Executing state etcd.set for [/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16]
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.478988    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 salt-minion[4490]: [ERROR   ] etcd: Key not found : /calico
Aug 28 16:16:29 ctl01 salt-minion[4490]: [INFO    ] {'/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16': '{"masquerade":true,"cidr":"192.168.0.0/16"}'}
Aug 28 16:16:29 ctl01 salt-minion[4490]: [INFO    ] Completed state [/calico/ipam/v2/assignment/ipv4/block/192.168.0.0-16] at time 16:16:29.483432 duration_in_ms=6.232
Aug 28 16:16:29 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161622927403
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.579362    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.679694    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.737708    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.738667    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.739568    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.779880    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.880091    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:29 ctl01 kubelet[9620]: E0828 16:16:29.980319    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190828161630039595
Aug 28 16:16:30 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 10012
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.080579    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.180883    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.281182    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.381332    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kube-proxy[9844]: E0828 16:16:30.425331    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:30 ctl01 kube-proxy[9844]: E0828 16:16:30.426260    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.481530    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.581714    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.681924    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.738344    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.739270    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.740340    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.782125    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.882340    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:30 ctl01 kubelet[9620]: E0828 16:16:30.982531    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.082702    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.182920    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/init.sls'
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.283125    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/init.sls'
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.383334    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:31 ctl01 kube-proxy[9844]: E0828 16:16:31.425929    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:31 ctl01 kube-proxy[9844]: E0828 16:16:31.426985    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.483676    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.584050    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.684272    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.739112    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.740087    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.740971    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.784500    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/service.sls'
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.809514    9620 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.884664    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:31 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:31 ctl01 kubelet[9620]: E0828 16:16:31.984870    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.085080    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.185290    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.285486    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.385742    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 kube-proxy[9844]: E0828 16:16:32.426542    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:32 ctl01 kube-proxy[9844]: E0828 16:16:32.427404    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:32 ctl01 kubelet[9620]: I0828 16:16:32.455467    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:32 ctl01 kubelet[9620]: I0828 16:16:32.455880    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:32 ctl01 kubelet[9620]: I0828 16:16:32.457097    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:32 ctl01 kubelet[9620]: I0828 16:16:32.457189    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:32 ctl01 kubelet[9620]: I0828 16:16:32.457219    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:32 ctl01 kubelet[9620]: I0828 16:16:32.457264    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.457896    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.485965    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.586283    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.686556    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.739997    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.740664    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.741786    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.786713    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.885881    9620 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.886866    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:32 ctl01 kubelet[9620]: E0828 16:16:32.987082    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.087273    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.187518    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.287696    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.387896    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 kube-proxy[9844]: E0828 16:16:33.427085    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:33 ctl01 kube-proxy[9844]: E0828 16:16:33.428127    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:33 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.488082    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.588442    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.688723    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.740738    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.741695    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.742811    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.788967    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.889230    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:33 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/controller.sls'
Aug 28 16:16:33 ctl01 kubelet[9620]: E0828 16:16:33.989524    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.089755    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.189998    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.290205    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.390414    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 kube-proxy[9844]: E0828 16:16:34.427786    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:34 ctl01 kube-proxy[9844]: E0828 16:16:34.428693    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.490709    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.590911    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.691132    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.741645    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.742588    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.744007    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.791427    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command 'date "+%FT%TZ"' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.891744    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/master/setup.sls'
Aug 28 16:16:34 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:34 ctl01 kubelet[9620]: E0828 16:16:34.992064    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.092359    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.192584    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.292811    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.393039    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 kube-proxy[9844]: E0828 16:16:35.428455    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:35 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:35 ctl01 kube-proxy[9844]: E0828 16:16:35.429495    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.493364    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.593583    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.693816    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.742444    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.743266    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.744580    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.794052    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.894301    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:35 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:35 ctl01 kubelet[9620]: E0828 16:16:35.994537    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.094787    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.195081    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.295355    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.395736    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 kube-proxy[9844]: E0828 16:16:36.429235    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:36 ctl01 kube-proxy[9844]: E0828 16:16:36.430032    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:36 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.496044    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.596336    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.696563    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.743449    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.744341    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.745315    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:36 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.796818    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.897136    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:36 ctl01 kubelet[9620]: E0828 16:16:36.997469    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.097673    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.197929    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.298214    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.398410    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 kube-proxy[9844]: E0828 16:16:37.430206    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:37 ctl01 kube-proxy[9844]: E0828 16:16:37.431195    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.498686    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.598890    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.699146    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.744066    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:37 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.745282    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.746270    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.799339    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.899548    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:37 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:37 ctl01 kubelet[9620]: E0828 16:16:37.999791    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/control/init.sls'
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.100003    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.200332    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.300507    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.400800    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 kube-proxy[9844]: E0828 16:16:38.431048    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:38 ctl01 kube-proxy[9844]: E0828 16:16:38.431824    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/control/service.sls'
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.500984    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.601205    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.701426    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.744617    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.745760    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.746957    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.801688    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 kube-proxy[9844]: E0828 16:16:38.845208    9844 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/control/role.sls'
Aug 28 16:16:38 ctl01 kubelet[9620]: E0828 16:16:38.901952    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:38 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.002123    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.102377    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.202651    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.302824    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.403081    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 kube-proxy[9844]: E0828 16:16:39.431814    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:39 ctl01 kube-proxy[9844]: E0828 16:16:39.432720    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:39 ctl01 kubelet[9620]: I0828 16:16:39.458113    9620 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:39 ctl01 kubelet[9620]: I0828 16:16:39.458378    9620 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:39 ctl01 kubelet[9620]: I0828 16:16:39.459149    9620 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:39 ctl01 kubelet[9620]: I0828 16:16:39.459182    9620 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:39 ctl01 kubelet[9620]: I0828 16:16:39.459196    9620 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:39 ctl01 kubelet[9620]: I0828 16:16:39.459232    9620 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.459532    9620 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.503279    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.603477    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.703665    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.745215    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.746150    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.747473    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.803867    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:39 ctl01 salt-minion[4490]: [INFO    ] Running state [curl] at time 16:16:39.900642
Aug 28 16:16:39 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [curl]
Aug 28 16:16:39 ctl01 salt-minion[4490]: [INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/root'
Aug 28 16:16:39 ctl01 kubelet[9620]: E0828 16:16:39.904031    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.004246    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.104448    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.204660    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.304870    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.405048    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kube-proxy[9844]: E0828 16:16:40.432412    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:40 ctl01 kube-proxy[9844]: E0828 16:16:40.433759    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.505279    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.605505    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.705656    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [curl] at time 16:16:40.726672 duration_in_ms=826.033
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [git] at time 16:16:40.727018
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [git]
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [git] at time 16:16:40.736494 duration_in_ms=9.476
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [apt-transport-https] at time 16:16:40.736733
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [apt-transport-https]
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.745872    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [apt-transport-https] at time 16:16:40.746183 duration_in_ms=9.45
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [python-apt] at time 16:16:40.746424
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.746795    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [python-apt]
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.747946    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [python-apt] at time 16:16:40.755855 duration_in_ms=9.431
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [socat] at time 16:16:40.756116
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [socat]
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [socat] at time 16:16:40.765283 duration_in_ms=9.166
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [openssl] at time 16:16:40.765518
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [openssl]
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [openssl] at time 16:16:40.774928 duration_in_ms=9.409
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [conntrack] at time 16:16:40.775167
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [conntrack]
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [conntrack] at time 16:16:40.784794 duration_in_ms=9.627
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [nfs-common] at time 16:16:40.785028
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [nfs-common]
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [nfs-common] at time 16:16:40.794199 duration_in_ms=9.171
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [cifs-utils] at time 16:16:40.794437
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [cifs-utils]
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Completed state [cifs-utils] at time 16:16:40.803723 duration_in_ms=9.286
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.805816    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/hyperkube] at time 16:16:40.806951
Aug 28 16:16:40 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/bin/hyperkube]
Aug 28 16:16:40 ctl01 kubelet[9620]: E0828 16:16:40.905990    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.006214    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.106398    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.206631    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.306896    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.407128    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kube-proxy[9844]: E0828 16:16:41.433047    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:41 ctl01 kube-proxy[9844]: E0828 16:16:41.434180    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.507361    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] File /usr/bin/hyperkube is in the correct state
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/hyperkube] at time 16:16:41.533253 duration_in_ms=726.301
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/kubectl] at time 16:16:41.534376
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing state file.symlink for [/usr/bin/kubectl]
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Symlink /usr/bin/kubectl is present and owned by root:root
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/kubectl] at time 16:16:41.536429 duration_in_ms=2.054
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Running state [containerd] at time 16:16:41.536663
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing state pkg.installed for [containerd]
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] All specified packages are already installed
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Completed state [containerd] at time 16:16:41.546232 duration_in_ms=9.569
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/containerd/config.toml] at time 16:16:41.546468
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/containerd/config.toml]
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.607581    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.707760    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.746796    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.747590    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.748588    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.807969    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.810384    9620 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] File /etc/containerd/config.toml is in the correct state
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/containerd/config.toml] at time 16:16:41.890027 duration_in_ms=343.558
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Running state [containerd] at time 16:16:41.892431
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [containerd]
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'containerd.service', '-n', '0'] in directory '/root'
Aug 28 16:16:41 ctl01 kubelet[9620]: E0828 16:16:41.908147    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'containerd.service'] in directory '/root'
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'containerd.service'] in directory '/root'
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] The service containerd is already running
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Completed state [containerd] at time 16:16:41.958875 duration_in_ms=66.444
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Running state [/tmp/crictl] at time 16:16:41.961036
Aug 28 16:16:41 ctl01 salt-minion[4490]: [INFO    ] Executing state archive.extracted for [/tmp/crictl]
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.008339    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.108536    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.208734    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] All files in archive are already present
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Completed state [/tmp/crictl] at time 16:16:42.276212 duration_in_ms=315.176
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/local/bin/crictl] at time 16:16:42.277302
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/local/bin/crictl]
Aug 28 16:16:42 ctl01 salt-minion[4490]: [WARNING ] Use of argument owner found, "owner" is invalid, please use "user"
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.308992    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.409373    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 kube-proxy[9844]: E0828 16:16:42.434235    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:42 ctl01 kube-proxy[9844]: E0828 16:16:42.434785    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.509778    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.610010    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] File /usr/local/bin/crictl is in the correct state
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/local/bin/crictl] at time 16:16:42.702323 duration_in_ms=425.02
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/crictl.yaml] at time 16:16:42.702593
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/crictl.yaml]
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] File /etc/crictl.yaml is in the correct state
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/crictl.yaml] at time 16:16:42.704651 duration_in_ms=2.058
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/criproxy] at time 16:16:42.704899
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/criproxy]
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] File /etc/criproxy is not present
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/criproxy] at time 16:16:42.705689 duration_in_ms=0.79
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Running state [criproxy] at time 16:16:42.705922
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing state service.dead for [criproxy]
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'criproxy.service', '-n', '0'] in directory '/root'
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.710265    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] The named service criproxy is not available
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Completed state [criproxy] at time 16:16:42.726496 duration_in_ms=20.573
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kubelet.service] at time 16:16:42.727009
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kubelet.service]
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.747665    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.748660    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.749869    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.810527    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.886120    9620 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
Aug 28 16:16:42 ctl01 kubelet[9620]: E0828 16:16:42.911039    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:42 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.011242    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] File /etc/systemd/system/kubelet.service is in the correct state
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kubelet.service] at time 16:16:43.044016 duration_in_ms=317.007
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/config] at time 16:16:43.044367
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/kubernetes/config]
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/config is not present
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/config] at time 16:16:43.045287 duration_in_ms=0.92
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kubelet] at time 16:16:43.045531
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/kubelet]
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.111458    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.211624    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.311842    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.412006    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:43 ctl01 kube-proxy[9844]: E0828 16:16:43.435069    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:43 ctl01 kube-proxy[9844]: E0828 16:16:43.436218    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.512171    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.612395    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.712609    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.748465    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.749206    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.750513    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.812840    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:43 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:43 ctl01 kubelet[9620]: E0828 16:16:43.913805    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.014067    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] File /etc/default/kubelet is in the correct state
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kubelet] at time 16:16:44.028020 duration_in_ms=982.489
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/kubelet.kubeconfig] at time 16:16:44.028476
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/kubelet.kubeconfig]
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.114265    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.214509    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.314821    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.415019    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 kube-proxy[9844]: E0828 16:16:44.436339    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 kube-proxy[9844]: E0828 16:16:44.436894    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.515233    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.615412    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/kubelet.kubeconfig is in the correct state
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/kubelet.kubeconfig] at time 16:16:44.629239 duration_in_ms=600.762
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/manifests] at time 16:16:44.629823
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/etc/kubernetes/manifests]
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Directory /etc/kubernetes/manifests is in the correct state
Aug 28 16:16:44 ctl01 salt-minion[4490]: Directory /etc/kubernetes/manifests updated
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/manifests] at time 16:16:44.632239 duration_in_ms=2.416
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Running state [kubelet] at time 16:16:44.636787
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [kubelet]
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kubelet.service', '-n', '0'] in directory '/root'
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kubelet.service'] in directory '/root'
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kubelet.service'] in directory '/root'
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.715690    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 systemd[1]: Started /bin/systemctl start kubelet.service.
Aug 28 16:16:44 ctl01 systemd[1]: kubelet.service: Found left-over process 9620 (hyperkube) in control group while starting unit. Ignoring.
Aug 28 16:16:44 ctl01 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Aug 28 16:16:44 ctl01 systemd[1]: Started Kubernetes Kubelet Server.
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kubelet.service'] in directory '/root'
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.749264    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.750144    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.752119    9620 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kubelet.service'] in directory '/root'
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] {'kubelet': True}
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [kubelet] at time 16:16:44.793098 duration_in_ms=156.31
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/logrotate.d/kubernetes] at time 16:16:44.793640
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/logrotate.d/kubernetes]
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.815968    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] File /etc/logrotate.d/kubernetes is in the correct state
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/logrotate.d/kubernetes] at time 16:16:44.822387 duration_in_ms=28.747
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Running state [/opt/cni/bin] at time 16:16:44.822784
Aug 28 16:16:44 ctl01 salt-minion[4490]: [INFO    ] Executing state archive.extracted for [/opt/cni/bin]
Aug 28 16:16:44 ctl01 kubelet[11101]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:44 ctl01 kubelet[11101]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:44 ctl01 kubelet[11101]: Flag --allow-privileged has been deprecated, will be removed in a future version
Aug 28 16:16:44 ctl01 kubelet[11101]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:44 ctl01 kubelet[11101]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:44 ctl01 kubelet[11101]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861035   11101 flags.go:33] FLAG: --address="172.16.10.36"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861092   11101 flags.go:33] FLAG: --allow-privileged="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861101   11101 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861113   11101 flags.go:33] FLAG: --alsologtostderr="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861120   11101 flags.go:33] FLAG: --anonymous-auth="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861126   11101 flags.go:33] FLAG: --application-metrics-count-limit="100"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861132   11101 flags.go:33] FLAG: --authentication-token-webhook="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861138   11101 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861146   11101 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861153   11101 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861159   11101 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861165   11101 flags.go:33] FLAG: --azure-container-registry-config=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861171   11101 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861177   11101 flags.go:33] FLAG: --bootstrap-checkpoint-path=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861182   11101 flags.go:33] FLAG: --bootstrap-kubeconfig=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861188   11101 flags.go:33] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861194   11101 flags.go:33] FLAG: --cgroup-driver="cgroupfs"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861199   11101 flags.go:33] FLAG: --cgroup-root=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861205   11101 flags.go:33] FLAG: --cgroups-per-qos="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861210   11101 flags.go:33] FLAG: --chaos-chance="0"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861218   11101 flags.go:33] FLAG: --client-ca-file=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861224   11101 flags.go:33] FLAG: --cloud-config=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861229   11101 flags.go:33] FLAG: --cloud-provider=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861234   11101 flags.go:33] FLAG: --cluster-dns="[10.254.0.10]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861242   11101 flags.go:33] FLAG: --cluster-domain="cluster.local"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861248   11101 flags.go:33] FLAG: --cni-bin-dir="/opt/cni/bin"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861254   11101 flags.go:33] FLAG: --cni-conf-dir="/etc/cni/net.d"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861259   11101 flags.go:33] FLAG: --config=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861265   11101 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861271   11101 flags.go:33] FLAG: --container-log-max-files="5"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861279   11101 flags.go:33] FLAG: --container-log-max-size="10Mi"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861284   11101 flags.go:33] FLAG: --container-runtime="remote"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861297   11101 flags.go:33] FLAG: --container-runtime-endpoint="unix:///run/containerd/containerd.sock"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861309   11101 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861316   11101 flags.go:33] FLAG: --containerized="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861321   11101 flags.go:33] FLAG: --contention-profiling="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861327   11101 flags.go:33] FLAG: --cpu-cfs-quota="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861332   11101 flags.go:33] FLAG: --cpu-cfs-quota-period="100ms"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861338   11101 flags.go:33] FLAG: --cpu-manager-policy="none"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861344   11101 flags.go:33] FLAG: --cpu-manager-reconcile-period="10s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861350   11101 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861356   11101 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861362   11101 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861368   11101 flags.go:33] FLAG: --docker-only="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861373   11101 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861379   11101 flags.go:33] FLAG: --docker-tls="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861385   11101 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861390   11101 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861396   11101 flags.go:33] FLAG: --docker-tls-key="key.pem"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861402   11101 flags.go:33] FLAG: --dynamic-config-dir=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861409   11101 flags.go:33] FLAG: --enable-controller-attach-detach="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861415   11101 flags.go:33] FLAG: --enable-debugging-handlers="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861420   11101 flags.go:33] FLAG: --enable-load-reader="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861426   11101 flags.go:33] FLAG: --enable-server="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861431   11101 flags.go:33] FLAG: --enforce-node-allocatable="[pods]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861442   11101 flags.go:33] FLAG: --event-burst="10"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861448   11101 flags.go:33] FLAG: --event-qps="5"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861453   11101 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861459   11101 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861465   11101 flags.go:33] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861481   11101 flags.go:33] FLAG: --eviction-max-pod-grace-period="0"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861487   11101 flags.go:33] FLAG: --eviction-minimum-reclaim=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861495   11101 flags.go:33] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861501   11101 flags.go:33] FLAG: --eviction-soft=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861512   11101 flags.go:33] FLAG: --eviction-soft-grace-period=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861523   11101 flags.go:33] FLAG: --exit-on-lock-contention="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861529   11101 flags.go:33] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861535   11101 flags.go:33] FLAG: --experimental-bootstrap-kubeconfig=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861541   11101 flags.go:33] FLAG: --experimental-check-node-capabilities-before-mount="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861547   11101 flags.go:33] FLAG: --experimental-dockershim="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861553   11101 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861559   11101 flags.go:33] FLAG: --experimental-fail-swap-on="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861564   11101 flags.go:33] FLAG: --experimental-kernel-memcg-notification="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861570   11101 flags.go:33] FLAG: --experimental-mounter-path=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861575   11101 flags.go:33] FLAG: --fail-swap-on="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861580   11101 flags.go:33] FLAG: --feature-gates=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861589   11101 flags.go:33] FLAG: --file-check-frequency="20s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861594   11101 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861600   11101 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861606   11101 flags.go:33] FLAG: --healthz-bind-address="127.0.0.1"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861612   11101 flags.go:33] FLAG: --healthz-port="10248"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861618   11101 flags.go:33] FLAG: --help="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861624   11101 flags.go:33] FLAG: --host-ipc-sources="[*]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861631   11101 flags.go:33] FLAG: --host-network-sources="[*]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861640   11101 flags.go:33] FLAG: --host-pid-sources="[*]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861647   11101 flags.go:33] FLAG: --hostname-override="ctl01"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861653   11101 flags.go:33] FLAG: --housekeeping-interval="10s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861658   11101 flags.go:33] FLAG: --http-check-frequency="20s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861664   11101 flags.go:33] FLAG: --image-gc-high-threshold="85"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861669   11101 flags.go:33] FLAG: --image-gc-low-threshold="80"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861675   11101 flags.go:33] FLAG: --image-pull-progress-deadline="1m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861680   11101 flags.go:33] FLAG: --image-service-endpoint=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861686   11101 flags.go:33] FLAG: --iptables-drop-bit="15"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861691   11101 flags.go:33] FLAG: --iptables-masquerade-bit="14"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861697   11101 flags.go:33] FLAG: --keep-terminated-pod-volumes="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861703   11101 flags.go:33] FLAG: --kube-api-burst="10"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861714   11101 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861725   11101 flags.go:33] FLAG: --kube-api-qps="5"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861731   11101 flags.go:33] FLAG: --kube-reserved=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861737   11101 flags.go:33] FLAG: --kube-reserved-cgroup=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861743   11101 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/kubelet.kubeconfig"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861749   11101 flags.go:33] FLAG: --kubelet-cgroups=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861754   11101 flags.go:33] FLAG: --lock-file=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861759   11101 flags.go:33] FLAG: --log-backtrace-at=":0"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861766   11101 flags.go:33] FLAG: --log-cadvisor-usage="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861772   11101 flags.go:33] FLAG: --log-dir=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861777   11101 flags.go:33] FLAG: --log-file=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861783   11101 flags.go:33] FLAG: --log-flush-frequency="5s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861789   11101 flags.go:33] FLAG: --logtostderr="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861794   11101 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861801   11101 flags.go:33] FLAG: --make-iptables-util-chains="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861806   11101 flags.go:33] FLAG: --manifest-url=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861812   11101 flags.go:33] FLAG: --manifest-url-header=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861821   11101 flags.go:33] FLAG: --master-service-namespace="default"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861826   11101 flags.go:33] FLAG: --max-open-files="1000000"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861834   11101 flags.go:33] FLAG: --max-pods="110"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861840   11101 flags.go:33] FLAG: --maximum-dead-containers="-1"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861846   11101 flags.go:33] FLAG: --maximum-dead-containers-per-container="1"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861851   11101 flags.go:33] FLAG: --minimum-container-ttl-duration="0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861857   11101 flags.go:33] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861863   11101 flags.go:33] FLAG: --network-plugin="cni"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861868   11101 flags.go:33] FLAG: --network-plugin-mtu="0"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861874   11101 flags.go:33] FLAG: --node-ip="172.16.10.36"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861880   11101 flags.go:33] FLAG: --node-labels="node-role.kubernetes.io/master=true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861889   11101 flags.go:33] FLAG: --node-status-max-images="50"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861895   11101 flags.go:33] FLAG: --node-status-update-frequency="10s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861901   11101 flags.go:33] FLAG: --non-masquerade-cidr="10.0.0.0/8"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861906   11101 flags.go:33] FLAG: --oom-score-adj="-999"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861917   11101 flags.go:33] FLAG: --pod-cidr=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861928   11101 flags.go:33] FLAG: --pod-infra-container-image="docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861937   11101 flags.go:33] FLAG: --pod-manifest-path="/etc/kubernetes/manifests"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861943   11101 flags.go:33] FLAG: --pod-max-pids="-1"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861949   11101 flags.go:33] FLAG: --pods-per-core="0"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861955   11101 flags.go:33] FLAG: --port="10250"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861960   11101 flags.go:33] FLAG: --protect-kernel-defaults="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861966   11101 flags.go:33] FLAG: --provider-id=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861971   11101 flags.go:33] FLAG: --qos-reserved=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861977   11101 flags.go:33] FLAG: --read-only-port="10255"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861983   11101 flags.go:33] FLAG: --really-crash-for-testing="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861988   11101 flags.go:33] FLAG: --redirect-container-streaming="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861994   11101 flags.go:33] FLAG: --register-node="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.861999   11101 flags.go:33] FLAG: --register-schedulable="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862005   11101 flags.go:33] FLAG: --register-with-taints=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862012   11101 flags.go:33] FLAG: --registry-burst="10"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862018   11101 flags.go:33] FLAG: --registry-qps="5"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862023   11101 flags.go:33] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862029   11101 flags.go:33] FLAG: --root-dir="/var/lib/kubelet"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862035   11101 flags.go:33] FLAG: --rotate-certificates="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862040   11101 flags.go:33] FLAG: --rotate-server-certificates="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862045   11101 flags.go:33] FLAG: --runonce="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862051   11101 flags.go:33] FLAG: --runtime-cgroups=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862056   11101 flags.go:33] FLAG: --runtime-request-timeout="2m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862062   11101 flags.go:33] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862068   11101 flags.go:33] FLAG: --serialize-image-pulls="true"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862073   11101 flags.go:33] FLAG: --stderrthreshold="2"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862079   11101 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862084   11101 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862090   11101 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862096   11101 flags.go:33] FLAG: --storage-driver-password="root"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862102   11101 flags.go:33] FLAG: --storage-driver-secure="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862112   11101 flags.go:33] FLAG: --storage-driver-table="stats"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862123   11101 flags.go:33] FLAG: --storage-driver-user="root"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862129   11101 flags.go:33] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862135   11101 flags.go:33] FLAG: --sync-frequency="1m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862141   11101 flags.go:33] FLAG: --system-cgroups=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862146   11101 flags.go:33] FLAG: --system-reserved=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862152   11101 flags.go:33] FLAG: --system-reserved-cgroup=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862158   11101 flags.go:33] FLAG: --tls-cert-file=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862163   11101 flags.go:33] FLAG: --tls-cipher-suites="[]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862173   11101 flags.go:33] FLAG: --tls-min-version=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862178   11101 flags.go:33] FLAG: --tls-private-key-file=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862184   11101 flags.go:33] FLAG: --v="2"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862189   11101 flags.go:33] FLAG: --version="false"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862198   11101 flags.go:33] FLAG: --vmodule=""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862204   11101 flags.go:33] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862211   11101 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862248   11101 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:44 ctl01 kubelet[11101]: W0828 16:16:44.862266   11101 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
Aug 28 16:16:44 ctl01 kubelet[11101]: W0828 16:16:44.862279   11101 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
Aug 28 16:16:44 ctl01 kubelet[11101]: W0828 16:16:44.862291   11101 server.go:182] Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.862334   11101 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:44 ctl01 systemd[1]: Started Kubernetes systemd probe.
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.875492   11101 mount_linux.go:180] Detected OS with systemd
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.875542   11101 server.go:407] Version: v1.13.5-3+98374c02d2d8c1
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.875578   11101 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.875628   11101 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:44 ctl01 kubelet[11101]: W0828 16:16:44.875647   11101 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
Aug 28 16:16:44 ctl01 kubelet[11101]: W0828 16:16:44.875659   11101 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.875729   11101 plugins.go:103] No cloud provider specified.
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.875743   11101 server.go:523] No cloud provider specified: "" from the config file: ""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.882035   11101 manager.go:155] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.883127   11101 fs.go:142] Filesystem UUIDs: map[2019-08-28-18-06-57-00:/dev/sr0 9430be73-33d9-4f4e-beff-4a37ae3601c5:/dev/vda1 C738-8EF9:/dev/vda15]
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.883150   11101 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /dev/vda1:{mountpoint:/ major:252 minor:1 fsType:ext4 blockSize:0}]
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.888495   11101 manager.go:229] Machine: {NumCores:8 CpuFrequency:2799994 MemoryCapacity:14704947200 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:41672cdd030f48a18c2eb70f309172a9 SystemUUID:41672CDD-030F-48A1-8C2E-B70F309172A9 BootID:fefbdff1-5a6b-4a85-9190-6007d0de5d83 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:24 Capacity:1470496768 Type:vfs Inodes:1795037 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:103880232960 Type:vfs Inodes:12902400 HasInodes:true}] DiskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:br-mgmt MacAddress:52:54:00:2c:cd:bc Speed:0 Mtu:9000} {Name:ens3 MacAddress:52:54:00:3f:9b:0c Speed:-1 Mtu:9000} {Name:ens4 MacAddress:52:54:00:2c:cd:bc Speed:-1 Mtu:9000} {Name:ens5 MacAddress:52:54:00:9e:c5:f6 Speed:-1 Mtu:9000} {Name:ens5.1000 MacAddress:52:54:00:9e:c5:f6 Speed:-1 Mtu:9000} {Name:ens6 MacAddress:52:54:00:14:c4:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:14704947200 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:2 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:3 Memory:0 Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:4 Memory:0 Cores:[{Id:0 Threads:[4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:5 Memory:0 Cores:[{Id:0 Threads:[5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:6 Memory:0 Cores:[{Id:0 Threads:[6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:7 Memory:0 Cores:[{Id:0 Threads:[7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.889004   11101 manager.go:235] Version: {KernelVersion:4.15.0-58-generic ContainerOsVersion:Ubuntu 18.04.3 LTS DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:}
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.889239   11101 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.889643   11101 container_manager_linux.go:248] container manager verified user specified cgroup-root exists: []
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.889792   11101 container_manager_linux.go:253] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms}
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.890109   11101 container_manager_linux.go:272] Creating device plugin manager: true
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.890250   11101 manager.go:109] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.890405   11101 state_mem.go:36] [cpumanager] initializing new in-memory state store
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.890642   11101 state_mem.go:84] [cpumanager] updated default cpuset: ""
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.890793   11101 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.890940   11101 state_checkpoint.go:100] [cpumanager] state checkpoint: restored state from checkpoint
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.891060   11101 state_checkpoint.go:101] [cpumanager] state checkpoint: defaultCPUSet:
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.891283   11101 server.go:941] Using root directory: /var/lib/kubelet
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.891430   11101 kubelet.go:281] Adding pod path: /etc/kubernetes/manifests
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.891587   11101 file.go:68] Watching path "/etc/kubernetes/manifests"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.891746   11101 kubelet.go:306] Watching apiserver
Aug 28 16:16:44 ctl01 kubelet[11101]: E0828 16:16:44.892342   11101 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 kubelet[11101]: E0828 16:16:44.892348   11101 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 kubelet[11101]: E0828 16:16:44.892519   11101 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.895293   11101 kuberuntime_manager.go:198] Container runtime containerd initialized, version: 1.2.6-0ubuntu1~18.04.2, apiVersion: v1alpha2
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.895973   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.896124   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/empty-dir"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.896262   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.896398   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/git-repo"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.896528   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/host-path"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.896666   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/nfs"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.896794   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/secret"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.896922   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/iscsi"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897051   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897184   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897311   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897436   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/quobyte"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897568   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/cephfs"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897696   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/downward-api"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897823   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.897948   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/flocker"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898082   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898209   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/configmap"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898335   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898461   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898594   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898720   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/projected"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898856   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.898988   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.899116   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/local-volume"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.899241   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.899373   11101 plugins.go:547] Loaded volume plugin "kubernetes.io/csi"
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.900080   11101 server.go:137] Starting to listen on 172.16.10.36:10250
Aug 28 16:16:44 ctl01 kubelet[11101]: E0828 16:16:44.900322   11101 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.900550   11101 server.go:157] Starting to listen read-only on 172.16.10.36:10255
Aug 28 16:16:44 ctl01 kubelet[11101]: E0828 16:16:44.900798   11101 cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
Aug 28 16:16:44 ctl01 kubelet[11101]: E0828 16:16:44.900834   11101 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.901735   11101 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.909249   11101 volume_manager.go:246] The desired_state_of_world populator starts
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.909182   11101 server.go:999] Started kubelet
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.909397   11101 desired_state_of_world_populator.go:130] Desired state populator starts to run
Aug 28 16:16:44 ctl01 kubelet[11101]: I0828 16:16:44.909382   11101 volume_manager.go:248] Starting Kubelet Volume Manager
Aug 28 16:16:44 ctl01 kubelet[11101]: F0828 16:16:44.909724   11101 server.go:165] listen tcp 172.16.10.36:10255: bind: address already in use
Aug 28 16:16:44 ctl01 kubelet[9620]: E0828 16:16:44.916121    9620 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:44 ctl01 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Aug 28 16:16:44 ctl01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Aug 28 16:16:45 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command saltutil.find_job with jid 20190828161645117610
Aug 28 16:16:45 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 11158
Aug 28 16:16:45 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161645117610
Aug 28 16:16:45 ctl01 kube-proxy[9844]: E0828 16:16:45.436946    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:45 ctl01 kube-proxy[9844]: E0828 16:16:45.438419    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:45 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', 'xz', '-f', '/var/cache/salt/minion/extrn_files/base/docker-prod-local.artifactory.mirantis.com/artifactory/binary-prod-local/mirantis/kubernetes/containernetworking-plugins/containernetworking-plugins_v0.7.2-173-g8db2808.tar.gz'] in directory '/opt/cni/bin/'
Aug 28 16:16:46 ctl01 kube-proxy[9844]: E0828 16:16:46.437691    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:46 ctl01 kube-proxy[9844]: E0828 16:16:46.438851    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Executing command ['tar', '--version'] in directory '/root'
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] {'extracted_files': 'no tar output so far'}
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Completed state [/opt/cni/bin] at time 16:16:46.702979 duration_in_ms=1880.191
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons] at time 16:16:46.703673
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/etc/kubernetes/addons]
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Directory /etc/kubernetes/addons is in the correct state
Aug 28 16:16:46 ctl01 salt-minion[4490]: Directory /etc/kubernetes/addons updated
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons] at time 16:16:46.706513 duration_in_ms=2.84
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:16:46.706934
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-kube-controllers.yml]
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:46 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/calico/calico-kube-controllers.yml is in the correct state
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-kube-controllers.yml] at time 16:16:47.100470 duration_in_ms=393.536
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:16:47.100801
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/calico/calico-rbac.yml]
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/calico/calico-rbac.yml is in the correct state
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/calico/calico-rbac.yml] at time 16:16:47.125212 duration_in_ms=24.411
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:16:47.125453
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-svc.yml]
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:47 ctl01 kube-proxy[9844]: E0828 16:16:47.438678    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:47 ctl01 kube-proxy[9844]: E0828 16:16:47.439686    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-svc.yml is in the correct state
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-svc.yml] at time 16:16:47.505188 duration_in_ms=379.735
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:16:47.505541
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-server.yml]
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:47 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-server.yml is in the correct state
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-server.yml] at time 16:16:48.254477 duration_in_ms=748.935
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:16:48.254822
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-agent.yml]
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:48 ctl01 kube-proxy[9844]: E0828 16:16:48.439710    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:48 ctl01 kube-proxy[9844]: E0828 16:16:48.440643    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-agent.yml is in the correct state
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-agent.yml] at time 16:16:48.609428 duration_in_ms=354.604
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:16:48.610132
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml]
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml is in the correct state
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-serviceaccount.yml] at time 16:16:48.639109 duration_in_ms=28.978
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:16:48.639488
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/netchecker/netchecker-roles.yml]
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/netchecker/netchecker-roles.yml is in the correct state
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/netchecker/netchecker-roles.yml] at time 16:16:48.663571 duration_in_ms=24.083
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:16:48.663840
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/prometheus/prometheus-roles.yml]
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/prometheus/prometheus-roles.yml is in the correct state
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/prometheus/prometheus-roles.yml] at time 16:16:48.685525 duration_in_ms=21.685
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/dns] at time 16:16:48.685780
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/dns]
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] {'removed': '/etc/kubernetes/addons/dns'}
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/dns] at time 16:16:48.686851 duration_in_ms=1.071
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Running state [kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true] at time 16:16:48.688261
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing state cmd.run for [kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true]
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing command 'kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true' in directory '/root'
Aug 28 16:16:48 ctl01 kube-proxy[9844]: E0828 16:16:48.846029    9844 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] {'pid': 11356, 'retcode': 0, 'stderr': 'The connection to the server localhost:8080 was refused - did you specify the right host or port?', 'stdout': 'kube-dns is absent. OK'}
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Completed state [kubectl -n kube-system delete svc kube-dns > /dev/null || echo "kube-dns is absent. OK" && true] at time 16:16:48.847882 duration_in_ms=159.62
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-cm.yml] at time 16:16:48.848508
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-cm.yml]
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-cm.yml'
Aug 28 16:16:48 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:49 ctl01 kube-proxy[9844]: E0828 16:16:49.440634    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:49 ctl01 kube-proxy[9844]: E0828 16:16:49.441381    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:49 ctl01 salt-minion[4490]: New file
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-cm.yml] at time 16:16:49.551047 duration_in_ms=702.539
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-deploy.yml] at time 16:16:49.551382
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-deploy.yml]
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-deploy.yml'
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:49 ctl01 salt-minion[4490]: New file
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-deploy.yml] at time 16:16:49.919593 duration_in_ms=368.21
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-svc.yml] at time 16:16:49.920191
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-svc.yml]
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-svc.yml'
Aug 28 16:16:49 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:50 ctl01 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Aug 28 16:16:50 ctl01 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Aug 28 16:16:50 ctl01 systemd[1]: Stopped Kubernetes Kubelet Server.
Aug 28 16:16:50 ctl01 systemd[1]: Started Kubernetes Kubelet Server.
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:50 ctl01 kubelet[11491]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:50 ctl01 kubelet[11491]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:50 ctl01 kubelet[11491]: Flag --allow-privileged has been deprecated, will be removed in a future version
Aug 28 16:16:50 ctl01 kubelet[11491]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:50 ctl01 kubelet[11491]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:50 ctl01 kubelet[11491]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172134   11491 flags.go:33] FLAG: --address="172.16.10.36"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172249   11491 flags.go:33] FLAG: --allow-privileged="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172267   11491 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172287   11491 flags.go:33] FLAG: --alsologtostderr="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172303   11491 flags.go:33] FLAG: --anonymous-auth="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172316   11491 flags.go:33] FLAG: --application-metrics-count-limit="100"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172330   11491 flags.go:33] FLAG: --authentication-token-webhook="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172342   11491 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172360   11491 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172376   11491 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172389   11491 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172401   11491 flags.go:33] FLAG: --azure-container-registry-config=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172414   11491 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172428   11491 flags.go:33] FLAG: --bootstrap-checkpoint-path=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172440   11491 flags.go:33] FLAG: --bootstrap-kubeconfig=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172451   11491 flags.go:33] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172464   11491 flags.go:33] FLAG: --cgroup-driver="cgroupfs"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172477   11491 flags.go:33] FLAG: --cgroup-root=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172488   11491 flags.go:33] FLAG: --cgroups-per-qos="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172500   11491 flags.go:33] FLAG: --chaos-chance="0"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172518   11491 flags.go:33] FLAG: --client-ca-file=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172529   11491 flags.go:33] FLAG: --cloud-config=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172540   11491 flags.go:33] FLAG: --cloud-provider=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172552   11491 flags.go:33] FLAG: --cluster-dns="[10.254.0.10]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172582   11491 flags.go:33] FLAG: --cluster-domain="cluster.local"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172595   11491 flags.go:33] FLAG: --cni-bin-dir="/opt/cni/bin"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172608   11491 flags.go:33] FLAG: --cni-conf-dir="/etc/cni/net.d"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172620   11491 flags.go:33] FLAG: --config=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172633   11491 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172647   11491 flags.go:33] FLAG: --container-log-max-files="5"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172663   11491 flags.go:33] FLAG: --container-log-max-size="10Mi"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172676   11491 flags.go:33] FLAG: --container-runtime="remote"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172688   11491 flags.go:33] FLAG: --container-runtime-endpoint="unix:///run/containerd/containerd.sock"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172702   11491 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172716   11491 flags.go:33] FLAG: --containerized="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172729   11491 flags.go:33] FLAG: --contention-profiling="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172741   11491 flags.go:33] FLAG: --cpu-cfs-quota="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172753   11491 flags.go:33] FLAG: --cpu-cfs-quota-period="100ms"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172767   11491 flags.go:33] FLAG: --cpu-manager-policy="none"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172778   11491 flags.go:33] FLAG: --cpu-manager-reconcile-period="10s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172801   11491 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172816   11491 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172830   11491 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172848   11491 flags.go:33] FLAG: --docker-only="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172861   11491 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172875   11491 flags.go:33] FLAG: --docker-tls="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172887   11491 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172899   11491 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172912   11491 flags.go:33] FLAG: --docker-tls-key="key.pem"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172924   11491 flags.go:33] FLAG: --dynamic-config-dir=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172940   11491 flags.go:33] FLAG: --enable-controller-attach-detach="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172953   11491 flags.go:33] FLAG: --enable-debugging-handlers="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172965   11491 flags.go:33] FLAG: --enable-load-reader="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172977   11491 flags.go:33] FLAG: --enable-server="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.172989   11491 flags.go:33] FLAG: --enforce-node-allocatable="[pods]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173005   11491 flags.go:33] FLAG: --event-burst="10"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173018   11491 flags.go:33] FLAG: --event-qps="5"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173030   11491 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173043   11491 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173055   11491 flags.go:33] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173088   11491 flags.go:33] FLAG: --eviction-max-pod-grace-period="0"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173102   11491 flags.go:33] FLAG: --eviction-minimum-reclaim=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173118   11491 flags.go:33] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173131   11491 flags.go:33] FLAG: --eviction-soft=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173144   11491 flags.go:33] FLAG: --eviction-soft-grace-period=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173168   11491 flags.go:33] FLAG: --exit-on-lock-contention="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173181   11491 flags.go:33] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173193   11491 flags.go:33] FLAG: --experimental-bootstrap-kubeconfig=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173204   11491 flags.go:33] FLAG: --experimental-check-node-capabilities-before-mount="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173217   11491 flags.go:33] FLAG: --experimental-dockershim="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173229   11491 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173242   11491 flags.go:33] FLAG: --experimental-fail-swap-on="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173262   11491 flags.go:33] FLAG: --experimental-kernel-memcg-notification="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173274   11491 flags.go:33] FLAG: --experimental-mounter-path=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173286   11491 flags.go:33] FLAG: --fail-swap-on="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173298   11491 flags.go:33] FLAG: --feature-gates=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173314   11491 flags.go:33] FLAG: --file-check-frequency="20s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173327   11491 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173339   11491 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173352   11491 flags.go:33] FLAG: --healthz-bind-address="127.0.0.1"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173365   11491 flags.go:33] FLAG: --healthz-port="10248"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173377   11491 flags.go:33] FLAG: --help="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173389   11491 flags.go:33] FLAG: --host-ipc-sources="[*]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173420   11491 flags.go:33] FLAG: --host-network-sources="[*]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173435   11491 flags.go:33] FLAG: --host-pid-sources="[*]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173453   11491 flags.go:33] FLAG: --hostname-override="ctl01"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173466   11491 flags.go:33] FLAG: --housekeeping-interval="10s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173478   11491 flags.go:33] FLAG: --http-check-frequency="20s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173491   11491 flags.go:33] FLAG: --image-gc-high-threshold="85"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173503   11491 flags.go:33] FLAG: --image-gc-low-threshold="80"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173515   11491 flags.go:33] FLAG: --image-pull-progress-deadline="1m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173527   11491 flags.go:33] FLAG: --image-service-endpoint=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173538   11491 flags.go:33] FLAG: --iptables-drop-bit="15"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173549   11491 flags.go:33] FLAG: --iptables-masquerade-bit="14"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173561   11491 flags.go:33] FLAG: --keep-terminated-pod-volumes="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173573   11491 flags.go:33] FLAG: --kube-api-burst="10"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173585   11491 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173608   11491 flags.go:33] FLAG: --kube-api-qps="5"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173621   11491 flags.go:33] FLAG: --kube-reserved=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173635   11491 flags.go:33] FLAG: --kube-reserved-cgroup=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173646   11491 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/kubelet.kubeconfig"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173659   11491 flags.go:33] FLAG: --kubelet-cgroups=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173671   11491 flags.go:33] FLAG: --lock-file=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173682   11491 flags.go:33] FLAG: --log-backtrace-at=":0"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173706   11491 flags.go:33] FLAG: --log-cadvisor-usage="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173719   11491 flags.go:33] FLAG: --log-dir=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173731   11491 flags.go:33] FLAG: --log-file=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173742   11491 flags.go:33] FLAG: --log-flush-frequency="5s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173755   11491 flags.go:33] FLAG: --logtostderr="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173767   11491 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173780   11491 flags.go:33] FLAG: --make-iptables-util-chains="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173792   11491 flags.go:33] FLAG: --manifest-url=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173804   11491 flags.go:33] FLAG: --manifest-url-header=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173822   11491 flags.go:33] FLAG: --master-service-namespace="default"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173835   11491 flags.go:33] FLAG: --max-open-files="1000000"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173851   11491 flags.go:33] FLAG: --max-pods="110"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173863   11491 flags.go:33] FLAG: --maximum-dead-containers="-1"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173876   11491 flags.go:33] FLAG: --maximum-dead-containers-per-container="1"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173888   11491 flags.go:33] FLAG: --minimum-container-ttl-duration="0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173900   11491 flags.go:33] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173912   11491 flags.go:33] FLAG: --network-plugin="cni"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173924   11491 flags.go:33] FLAG: --network-plugin-mtu="0"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173936   11491 flags.go:33] FLAG: --node-ip="172.16.10.36"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173948   11491 flags.go:33] FLAG: --node-labels="node-role.kubernetes.io/master=true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173969   11491 flags.go:33] FLAG: --node-status-max-images="50"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173982   11491 flags.go:33] FLAG: --node-status-update-frequency="10s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.173994   11491 flags.go:33] FLAG: --non-masquerade-cidr="10.0.0.0/8"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174006   11491 flags.go:33] FLAG: --oom-score-adj="-999"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174018   11491 flags.go:33] FLAG: --pod-cidr=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174040   11491 flags.go:33] FLAG: --pod-infra-container-image="docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174057   11491 flags.go:33] FLAG: --pod-manifest-path="/etc/kubernetes/manifests"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174070   11491 flags.go:33] FLAG: --pod-max-pids="-1"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174083   11491 flags.go:33] FLAG: --pods-per-core="0"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174095   11491 flags.go:33] FLAG: --port="10250"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174108   11491 flags.go:33] FLAG: --protect-kernel-defaults="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174120   11491 flags.go:33] FLAG: --provider-id=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174140   11491 flags.go:33] FLAG: --qos-reserved=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174154   11491 flags.go:33] FLAG: --read-only-port="10255"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174167   11491 flags.go:33] FLAG: --really-crash-for-testing="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174179   11491 flags.go:33] FLAG: --redirect-container-streaming="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174191   11491 flags.go:33] FLAG: --register-node="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174203   11491 flags.go:33] FLAG: --register-schedulable="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174214   11491 flags.go:33] FLAG: --register-with-taints=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174229   11491 flags.go:33] FLAG: --registry-burst="10"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174242   11491 flags.go:33] FLAG: --registry-qps="5"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174253   11491 flags.go:33] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174266   11491 flags.go:33] FLAG: --root-dir="/var/lib/kubelet"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174278   11491 flags.go:33] FLAG: --rotate-certificates="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174290   11491 flags.go:33] FLAG: --rotate-server-certificates="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174302   11491 flags.go:33] FLAG: --runonce="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174314   11491 flags.go:33] FLAG: --runtime-cgroups=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174325   11491 flags.go:33] FLAG: --runtime-request-timeout="2m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174338   11491 flags.go:33] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174350   11491 flags.go:33] FLAG: --serialize-image-pulls="true"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174362   11491 flags.go:33] FLAG: --stderrthreshold="2"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174374   11491 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174387   11491 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174400   11491 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174412   11491 flags.go:33] FLAG: --storage-driver-password="root"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174424   11491 flags.go:33] FLAG: --storage-driver-secure="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174435   11491 flags.go:33] FLAG: --storage-driver-table="stats"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174457   11491 flags.go:33] FLAG: --storage-driver-user="root"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174470   11491 flags.go:33] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174482   11491 flags.go:33] FLAG: --sync-frequency="1m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174495   11491 flags.go:33] FLAG: --system-cgroups=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174507   11491 flags.go:33] FLAG: --system-reserved=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174520   11491 flags.go:33] FLAG: --system-reserved-cgroup=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174531   11491 flags.go:33] FLAG: --tls-cert-file=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174550   11491 flags.go:33] FLAG: --tls-cipher-suites="[]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174565   11491 flags.go:33] FLAG: --tls-min-version=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174577   11491 flags.go:33] FLAG: --tls-private-key-file=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174589   11491 flags.go:33] FLAG: --v="2"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174601   11491 flags.go:33] FLAG: --version="false"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174619   11491 flags.go:33] FLAG: --vmodule=""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174632   11491 flags.go:33] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174646   11491 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174715   11491 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:50 ctl01 kubelet[11491]: W0828 16:16:50.174753   11491 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
Aug 28 16:16:50 ctl01 kubelet[11491]: W0828 16:16:50.174778   11491 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
Aug 28 16:16:50 ctl01 kubelet[11491]: W0828 16:16:50.174800   11491 server.go:182] Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.174908   11491 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:50 ctl01 systemd[1]: Started Kubernetes systemd probe.
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.189140   11491 mount_linux.go:180] Detected OS with systemd
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.189199   11491 server.go:407] Version: v1.13.5-3+98374c02d2d8c1
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.189250   11491 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.189309   11491 feature_gate.go:206] feature gates: &{map[]}
Aug 28 16:16:50 ctl01 kubelet[11491]: W0828 16:16:50.189379   11491 options.go:265] unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: [node-role.kubernetes.io/master]
Aug 28 16:16:50 ctl01 kubelet[11491]: W0828 16:16:50.189398   11491 options.go:266] in 1.15, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.189597   11491 plugins.go:103] No cloud provider specified.
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.189645   11491 server.go:523] No cloud provider specified: "" from the config file: ""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.197766   11491 manager.go:155] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.199327   11491 fs.go:142] Filesystem UUIDs: map[2019-08-28-18-06-57-00:/dev/sr0 9430be73-33d9-4f4e-beff-4a37ae3601c5:/dev/vda1 C738-8EF9:/dev/vda15]
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.199362   11491 fs.go:143] Filesystem partitions: map[tmpfs:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /dev/vda1:{mountpoint:/ major:252 minor:1 fsType:ext4 blockSize:0}]
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.206037   11491 manager.go:229] Machine: {NumCores:8 CpuFrequency:2799994 MemoryCapacity:14704947200 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:41672cdd030f48a18c2eb70f309172a9 SystemUUID:41672CDD-030F-48A1-8C2E-B70F309172A9 BootID:fefbdff1-5a6b-4a85-9190-6007d0de5d83 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:24 Capacity:1470496768 Type:vfs Inodes:1795037 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:103880232960 Type:vfs Inodes:12902400 HasInodes:true}] DiskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:br-mgmt MacAddress:52:54:00:2c:cd:bc Speed:0 Mtu:9000} {Name:ens3 MacAddress:52:54:00:3f:9b:0c Speed:-1 Mtu:9000} {Name:ens4 MacAddress:52:54:00:2c:cd:bc Speed:-1 Mtu:9000} {Name:ens5 MacAddress:52:54:00:9e:c5:f6 Speed:-1 Mtu:9000} {Name:ens5.1000 MacAddress:52:54:00:9e:c5:f6 Speed:-1 Mtu:9000} {Name:ens6 MacAddress:52:54:00:14:c4:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:14704947200 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:2 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:3 Memory:0 Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:4 Memory:0 Cores:[{Id:0 Threads:[4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:5 Memory:0 Cores:[{Id:0 Threads:[5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:6 Memory:0 Cores:[{Id:0 Threads:[6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:7 Memory:0 Cores:[{Id:0 Threads:[7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.206486   11491 manager.go:235] Version: {KernelVersion:4.15.0-58-generic ContainerOsVersion:Ubuntu 18.04.3 LTS DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:}
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.206609   11491 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.206907   11491 container_manager_linux.go:248] container manager verified user specified cgroup-root exists: []
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.206931   11491 container_manager_linux.go:253] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms}
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207064   11491 container_manager_linux.go:272] Creating device plugin manager: true
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207076   11491 manager.go:109] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207101   11491 state_mem.go:36] [cpumanager] initializing new in-memory state store
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207195   11491 state_mem.go:84] [cpumanager] updated default cpuset: ""
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207209   11491 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207218   11491 state_checkpoint.go:100] [cpumanager] state checkpoint: restored state from checkpoint
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207225   11491 state_checkpoint.go:101] [cpumanager] state checkpoint: defaultCPUSet:
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207310   11491 server.go:941] Using root directory: /var/lib/kubelet
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207332   11491 kubelet.go:281] Adding pod path: /etc/kubernetes/manifests
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207359   11491 file.go:68] Watching path "/etc/kubernetes/manifests"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.207373   11491 kubelet.go:306] Watching apiserver
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.208149   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.210566   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.210596   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.216730   11491 kuberuntime_manager.go:198] Container runtime containerd initialized, version: 1.2.6-0ubuntu1~18.04.2, apiVersion: v1alpha2
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.217072   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.217101   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/empty-dir"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.217112   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.217122   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/git-repo"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.217132   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/host-path"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218020   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/nfs"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218041   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/secret"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218052   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/iscsi"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218064   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218079   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218090   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218137   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/quobyte"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218150   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/cephfs"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218162   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/downward-api"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218205   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218216   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/flocker"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218256   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218267   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/configmap"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218306   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218318   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218328   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218369   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/projected"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218419   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218433   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218474   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/local-volume"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218514   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.218559   11491 plugins.go:547] Loaded volume plugin "kubernetes.io/csi"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.219369   11491 server.go:999] Started kubelet
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.219790   11491 server.go:157] Starting to listen read-only on 172.16.10.36:10255
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.224160   11491 server.go:137] Starting to listen on 172.16.10.36:10250
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.226020   11491 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.226504   11491 cri_stats_provider.go:320] Failed to get the info of the filesystem with mountpoint "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.227531   11491 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.227550   11491 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.227723   11491 status_manager.go:152] Starting to sync pod status with apiserver
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.227767   11491 kubelet.go:1829] Starting kubelet main sync loop.
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.227791   11491 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg has yet to be successful]
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.228190   11491 server.go:333] Adding debug handlers to kubelet server.
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.228293   11491 volume_manager.go:246] The desired_state_of_world populator starts
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.228679   11491 volume_manager.go:248] Starting Kubelet Volume Manager
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.228886   11491 desired_state_of_world_populator.go:130] Desired state populator starts to run
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.230684   11491 factory.go:136] Registering containerd factory
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.232506   11491 factory.go:54] Registering systemd factory
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.232959   11491 factory.go:97] Registering Raw factory
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.233310   11491 manager.go:1222] Started watching for new ooms in manager
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.235529   11491 manager.go:365] Starting recovery of all containers
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.286001   11491 manager.go:370] Recovery completed
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:50 ctl01 salt-minion[4490]: New file
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-svc.yml] at time 16:16:50.309126 duration_in_ms=388.933
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/coredns/coredns-rbac.yml] at time 16:16:50.309507
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/coredns/coredns-rbac.yml]
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.328014   11491 kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet]
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.328633   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addons/coredns/coredns-rbac.yml'
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.328870   11491 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.329360   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.330675   11491 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.330721   11491 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.330741   11491 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.330819   11491 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.331468   11491 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.351519   11491 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.351766   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.352903   11491 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.352969   11491 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.352998   11491 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.353228   11491 cpu_manager.go:155] [cpumanager] starting with none policy
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.353265   11491 cpu_manager.go:156] [cpumanager] reconciling every 10s
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.353370   11491 policy_none.go:42] [cpumanager] none policy: Start
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.355086   11491 manager.go:196] Starting Device Plugin manager
Aug 28 16:16:50 ctl01 kubelet[11491]: W0828 16:16:50.355139   11491 manager.go:537] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.355709   11491 manager.go:231] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.355969   11491 plugin_watcher.go:90] Plugin Watcher Start at /var/lib/kubelet/plugins_registry
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.357789   11491 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:50 ctl01 kube-proxy[9844]: I0828 16:16:50.416462    9844 proxier.go:645] Not syncing iptables until Services and Endpoints have been received from master
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.428826   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:50 ctl01 kube-proxy[9844]: E0828 16:16:50.441654    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 kube-proxy[9844]: E0828 16:16:50.442367    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.528304   11491 kubelet.go:1908] SyncLoop (ADD, "file"): ""
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.529050   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.531766   11491 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.532321   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.533615   11491 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.533672   11491 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.533697   11491 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.533744   11491 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.534312   11491 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.629259   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:50 ctl01 salt-minion[4490]: New file
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/coredns/coredns-rbac.yml] at time 16:16:50.679998 duration_in_ms=370.491
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/metrics-server] at time 16:16:50.680358
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/kubernetes/addons/metrics-server]
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/addons/metrics-server is not present
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/metrics-server] at time 16:16:50.681404 duration_in_ms=1.046
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Running state [/srv/kubernetes/known_tokens.csv] at time 16:16:50.681687
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/srv/kubernetes/known_tokens.csv]
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/known_tokens.csv'
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.729478   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.829716   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.929927   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.934600   11491 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.935202   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.937155   11491 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.937240   11491 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.937270   11491 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: I0828 16:16:50.937327   11491 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:50 ctl01 kubelet[11491]: E0828 16:16:50.937850   11491 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:50 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.030157   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.130365   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.209194   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.211481   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.212189   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.230760   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.331023   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:51 ctl01 salt-minion[4490]: New file
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Completed state [/srv/kubernetes/known_tokens.csv] at time 16:16:51.398712 duration_in_ms=717.025
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Running state [/srv/kubernetes/basic_auth.csv] at time 16:16:51.399032
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/srv/kubernetes/basic_auth.csv]
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/basic_auth.csv'
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.431327   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 kube-proxy[9844]: E0828 16:16:51.442306    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:51 ctl01 kube-proxy[9844]: E0828 16:16:51.443319    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.531545   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.631726   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.731972   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 kubelet[11491]: I0828 16:16:51.738108   11491 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:51 ctl01 kubelet[11491]: I0828 16:16:51.738599   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:51 ctl01 kubelet[11491]: I0828 16:16:51.739887   11491 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:51 ctl01 kubelet[11491]: I0828 16:16:51.739985   11491 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:51 ctl01 kubelet[11491]: I0828 16:16:51.740012   11491 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:51 ctl01 kubelet[11491]: I0828 16:16:51.740082   11491 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.740676   11491 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:51 ctl01 salt-minion[4490]: New file
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Completed state [/srv/kubernetes/basic_auth.csv] at time 16:16:51.822739 duration_in_ms=423.705
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kube-apiserver] at time 16:16:51.823376
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/kube-apiserver]
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:51 ctl01 salt-minion[4490]: New file
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kube-apiserver] at time 16:16:51.827299 duration_in_ms=3.923
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/scheduler.kubeconfig] at time 16:16:51.827990
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/scheduler.kubeconfig]
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.832207   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-scheduler/scheduler.kubeconfig'
Aug 28 16:16:51 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:51 ctl01 kubelet[11491]: E0828 16:16:51.932432   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.032712   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.132962   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.210049   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.212457   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.213057   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.240050   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.341098   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.441382   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 kube-proxy[9844]: E0828 16:16:52.443125    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:52 ctl01 kube-proxy[9844]: E0828 16:16:52.443962    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:52 ctl01 salt-minion[4490]: New file
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/scheduler.kubeconfig] at time 16:16:52.485205 duration_in_ms=657.214
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/controller-manager.kubeconfig] at time 16:16:52.485529
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/controller-manager.kubeconfig]
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-controller-manager/controller-manager.kubeconfig'
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.541596   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.641737   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.742001   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.842231   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:52 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:16:52 ctl01 kubelet[11491]: E0828 16:16:52.942514   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.042744   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.142994   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/controller-manager.kubeconfig] at time 16:16:53.175866 duration_in_ms=690.336
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kube-controller-manager] at time 16:16:53.176454
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/kube-controller-manager]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kube-controller-manager] at time 16:16:53.179565 duration_in_ms=3.112
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kube-scheduler] at time 16:16:53.180235
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/kube-scheduler]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kube-scheduler] at time 16:16:53.183139 duration_in_ms=2.905
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kube-apiserver.service] at time 16:16:53.184108
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-apiserver.service]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-apiserver.service'
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.210751   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.212950   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.214040   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kube-apiserver.service] at time 16:16:53.220214 duration_in_ms=36.106
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kube-scheduler.service] at time 16:16:53.220637
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-scheduler.service]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-scheduler.service'
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.243167   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kube-scheduler.service] at time 16:16:53.248955 duration_in_ms=28.317
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kube-controller-manager.service] at time 16:16:53.249305
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-controller-manager.service]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/systemd/kube-controller-manager.service'
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kube-controller-manager.service] at time 16:16:53.276778 duration_in_ms=27.472
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/ssl/kubernetes-server.crt] at time 16:16:53.277314
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/ssl/kubernetes-server.crt]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** '_certs/kubernetes/kubernetes-server.crt'
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/ssl/kubernetes-server.crt] at time 16:16:53.299059 duration_in_ms=21.744
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/ssl/kubernetes-server.key] at time 16:16:53.299575
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/ssl/kubernetes-server.key]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** '_certs/kubernetes/kubernetes-server.key'
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/ssl/kubernetes-server.key] at time 16:16:53.322889 duration_in_ms=23.314
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/ssl/kubernetes-server.pem] at time 16:16:53.323459
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/ssl/kubernetes-server.pem]
Aug 28 16:16:53 ctl01 kubelet[11491]: I0828 16:16:53.340917   11491 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:53 ctl01 kubelet[11491]: I0828 16:16:53.341225   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:53 ctl01 kubelet[11491]: I0828 16:16:53.342201   11491 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:53 ctl01 kubelet[11491]: I0828 16:16:53.342246   11491 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:53 ctl01 kubelet[11491]: I0828 16:16:53.342262   11491 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:53 ctl01 kubelet[11491]: I0828 16:16:53.342288   11491 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.342660   11491 kubelet_node_status.go:94] Unable to register node "ctl01" with API server: Post https://172.16.10.36:443/api/v1/nodes: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.343319   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** '_certs/kubernetes/kubernetes-server.pem'
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:16:53 ctl01 salt-minion[4490]: New file
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/ssl/kubernetes-server.pem] at time 16:16:53.346330 duration_in_ms=22.871
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.352449   11491 event.go:212] Unable to write event: 'Post https://172.16.10.36:443/api/v1/namespaces/default/events: dial tcp 172.16.10.36:443: connect: connection refused' (may retry after sleeping)
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Running state [kube-apiserver] at time 16:16:53.355008
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [kube-apiserver]
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kube-apiserver.service', '-n', '0'] in directory '/root'
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.443501   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 kube-proxy[9844]: E0828 16:16:53.444073    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:53 ctl01 kube-proxy[9844]: E0828 16:16:53.444913    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.16.10.36:443: connect: connection refused
Aug 28 16:16:53 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:16:53 ctl01 systemd[1]: Started /bin/systemctl start kube-apiserver.service.
Aug 28 16:16:53 ctl01 systemd[1]: Starting Kubernetes API Server...
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.543695   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611494   11892 flags.go:33] FLAG: --address="0.0.0.0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611561   11892 flags.go:33] FLAG: --admission-control="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611580   11892 flags.go:33] FLAG: --admission-control-config-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611588   11892 flags.go:33] FLAG: --advertise-address="172.16.10.36"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611595   11892 flags.go:33] FLAG: --allow-privileged="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611602   11892 flags.go:33] FLAG: --alsologtostderr="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611619   11892 flags.go:33] FLAG: --anonymous-auth="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611625   11892 flags.go:33] FLAG: --api-audiences="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611637   11892 flags.go:33] FLAG: --apiserver-count="1"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611645   11892 flags.go:33] FLAG: --application-metrics-count-limit="100"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611652   11892 flags.go:33] FLAG: --audit-dynamic-configuration="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611658   11892 flags.go:33] FLAG: --audit-log-batch-buffer-size="10000"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611669   11892 flags.go:33] FLAG: --audit-log-batch-max-size="1"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611675   11892 flags.go:33] FLAG: --audit-log-batch-max-wait="0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611683   11892 flags.go:33] FLAG: --audit-log-batch-throttle-burst="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611688   11892 flags.go:33] FLAG: --audit-log-batch-throttle-enable="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611694   11892 flags.go:33] FLAG: --audit-log-batch-throttle-qps="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611707   11892 flags.go:33] FLAG: --audit-log-format="json"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611713   11892 flags.go:33] FLAG: --audit-log-maxage="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611718   11892 flags.go:33] FLAG: --audit-log-maxbackup="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611724   11892 flags.go:33] FLAG: --audit-log-maxsize="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611730   11892 flags.go:33] FLAG: --audit-log-mode="blocking"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611736   11892 flags.go:33] FLAG: --audit-log-path=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611746   11892 flags.go:33] FLAG: --audit-log-truncate-enabled="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611752   11892 flags.go:33] FLAG: --audit-log-truncate-max-batch-size="10485760"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611760   11892 flags.go:33] FLAG: --audit-log-truncate-max-event-size="102400"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611767   11892 flags.go:33] FLAG: --audit-log-version="audit.k8s.io/v1"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611773   11892 flags.go:33] FLAG: --audit-policy-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611782   11892 flags.go:33] FLAG: --audit-webhook-batch-buffer-size="10000"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611789   11892 flags.go:33] FLAG: --audit-webhook-batch-initial-backoff="10s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611808   11892 flags.go:33] FLAG: --audit-webhook-batch-max-size="400"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611816   11892 flags.go:33] FLAG: --audit-webhook-batch-max-wait="30s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611822   11892 flags.go:33] FLAG: --audit-webhook-batch-throttle-burst="15"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611838   11892 flags.go:33] FLAG: --audit-webhook-batch-throttle-enable="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611848   11892 flags.go:33] FLAG: --audit-webhook-batch-throttle-qps="10"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611855   11892 flags.go:33] FLAG: --audit-webhook-config-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611861   11892 flags.go:33] FLAG: --audit-webhook-initial-backoff="10s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611867   11892 flags.go:33] FLAG: --audit-webhook-mode="batch"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611873   11892 flags.go:33] FLAG: --audit-webhook-truncate-enabled="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611879   11892 flags.go:33] FLAG: --audit-webhook-truncate-max-batch-size="10485760"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611889   11892 flags.go:33] FLAG: --audit-webhook-truncate-max-event-size="102400"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611895   11892 flags.go:33] FLAG: --audit-webhook-version="audit.k8s.io/v1"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611901   11892 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611930   11892 flags.go:33] FLAG: --authentication-token-webhook-config-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611940   11892 flags.go:33] FLAG: --authorization-mode="[Node,RBAC]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611960   11892 flags.go:33] FLAG: --authorization-policy-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611967   11892 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611973   11892 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611979   11892 flags.go:33] FLAG: --authorization-webhook-config-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611985   11892 flags.go:33] FLAG: --azure-container-registry-config=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.611991   11892 flags.go:33] FLAG: --basic-auth-file="/srv/kubernetes/basic_auth.csv"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612001   11892 flags.go:33] FLAG: --bind-address="0.0.0.0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612007   11892 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612014   11892 flags.go:33] FLAG: --cert-dir="/var/run/kubernetes"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612020   11892 flags.go:33] FLAG: --client-ca-file="/etc/kubernetes/ssl/ca-kubernetes.crt"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612027   11892 flags.go:33] FLAG: --cloud-config=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612032   11892 flags.go:33] FLAG: --cloud-provider=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612041   11892 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612052   11892 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612059   11892 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612065   11892 flags.go:33] FLAG: --contention-profiling="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612071   11892 flags.go:33] FLAG: --cors-allowed-origins="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612086   11892 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612093   11892 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612099   11892 flags.go:33] FLAG: --default-watch-cache-size="100"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612115   11892 flags.go:33] FLAG: --delete-collection-workers="1"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612121   11892 flags.go:33] FLAG: --deserialization-cache-size="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612127   11892 flags.go:33] FLAG: --disable-admission-plugins="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612143   11892 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612150   11892 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612155   11892 flags.go:33] FLAG: --docker-only="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612161   11892 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612167   11892 flags.go:33] FLAG: --docker-tls="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612177   11892 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612183   11892 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612189   11892 flags.go:33] FLAG: --docker-tls-key="key.pem"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612195   11892 flags.go:33] FLAG: --enable-admission-plugins="[NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,DefaultStorageClass]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612215   11892 flags.go:33] FLAG: --enable-aggregator-routing="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612226   11892 flags.go:33] FLAG: --enable-bootstrap-token-auth="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612232   11892 flags.go:33] FLAG: --enable-garbage-collector="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612238   11892 flags.go:33] FLAG: --enable-load-reader="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612245   11892 flags.go:33] FLAG: --enable-logs-handler="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612250   11892 flags.go:33] FLAG: --enable-swagger-ui="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612256   11892 flags.go:33] FLAG: --encryption-provider-config=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612265   11892 flags.go:33] FLAG: --endpoint-reconciler-type="lease"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612271   11892 flags.go:33] FLAG: --etcd-cafile="/var/lib/etcd/ca.pem"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612277   11892 flags.go:33] FLAG: --etcd-certfile="/var/lib/etcd/etcd-client.crt"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612284   11892 flags.go:33] FLAG: --etcd-compaction-interval="5m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612290   11892 flags.go:33] FLAG: --etcd-count-metric-poll-period="1m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612296   11892 flags.go:33] FLAG: --etcd-keyfile="/var/lib/etcd/etcd-client.key"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612306   11892 flags.go:33] FLAG: --etcd-prefix="/registry"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612311   11892 flags.go:33] FLAG: --etcd-servers="[https://172.16.10.36:4001]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612324   11892 flags.go:33] FLAG: --etcd-servers-overrides="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612334   11892 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612340   11892 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612350   11892 flags.go:33] FLAG: --event-ttl="1h0m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612357   11892 flags.go:33] FLAG: --experimental-encryption-provider-config=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612372   11892 flags.go:33] FLAG: --external-hostname=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612378   11892 flags.go:33] FLAG: --feature-gates=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612388   11892 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612394   11892 flags.go:33] FLAG: --help="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612404   11892 flags.go:33] FLAG: --housekeeping-interval="10s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612410   11892 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612416   11892 flags.go:33] FLAG: --insecure-bind-address="0.0.0.0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612422   11892 flags.go:33] FLAG: --insecure-port="8080"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612428   11892 flags.go:33] FLAG: --kubelet-certificate-authority=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612434   11892 flags.go:33] FLAG: --kubelet-client-certificate=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612443   11892 flags.go:33] FLAG: --kubelet-client-key=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612449   11892 flags.go:33] FLAG: --kubelet-https="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612455   11892 flags.go:33] FLAG: --kubelet-port="10250"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612462   11892 flags.go:33] FLAG: --kubelet-preferred-address-types="[Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612477   11892 flags.go:33] FLAG: --kubelet-read-only-port="10255"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612488   11892 flags.go:33] FLAG: --kubelet-timeout="5s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612494   11892 flags.go:33] FLAG: --kubernetes-service-node-port="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612500   11892 flags.go:33] FLAG: --log-backtrace-at=":0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612509   11892 flags.go:33] FLAG: --log-cadvisor-usage="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612515   11892 flags.go:33] FLAG: --log-dir=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612526   11892 flags.go:33] FLAG: --log-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612531   11892 flags.go:33] FLAG: --log-flush-frequency="5s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612537   11892 flags.go:33] FLAG: --logtostderr="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612543   11892 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612550   11892 flags.go:33] FLAG: --master-service-namespace="default"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612556   11892 flags.go:33] FLAG: --max-connection-bytes-per-sec="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612566   11892 flags.go:33] FLAG: --max-mutating-requests-inflight="200"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612572   11892 flags.go:33] FLAG: --max-requests-inflight="400"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612578   11892 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612584   11892 flags.go:33] FLAG: --mesos-agent-timeout="10s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612590   11892 flags.go:33] FLAG: --min-request-timeout="1800"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612596   11892 flags.go:33] FLAG: --oidc-ca-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612616   11892 flags.go:33] FLAG: --oidc-client-id=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612621   11892 flags.go:33] FLAG: --oidc-groups-claim=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612627   11892 flags.go:33] FLAG: --oidc-groups-prefix=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612632   11892 flags.go:33] FLAG: --oidc-issuer-url=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612638   11892 flags.go:33] FLAG: --oidc-required-claim=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612651   11892 flags.go:33] FLAG: --oidc-signing-algs="[RS256]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612664   11892 flags.go:33] FLAG: --oidc-username-claim="sub"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612670   11892 flags.go:33] FLAG: --oidc-username-prefix=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612676   11892 flags.go:33] FLAG: --port="8080"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612682   11892 flags.go:33] FLAG: --profiling="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612688   11892 flags.go:33] FLAG: --proxy-client-cert-file="/etc/kubernetes/ssl/kube-aggregator-proxy-client.crt"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612699   11892 flags.go:33] FLAG: --proxy-client-key-file="/etc/kubernetes/ssl/kube-aggregator-proxy-client.key"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612705   11892 flags.go:33] FLAG: --repair-malformed-updates="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612711   11892 flags.go:33] FLAG: --request-timeout="1m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612717   11892 flags.go:33] FLAG: --requestheader-allowed-names="[system:kube-controller-manager]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612731   11892 flags.go:33] FLAG: --requestheader-client-ca-file="/etc/kubernetes/ssl/ca-kubernetes.crt"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612742   11892 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[X-Remote-Extra-]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612753   11892 flags.go:33] FLAG: --requestheader-group-headers="[X-Remote-Group]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612765   11892 flags.go:33] FLAG: --requestheader-username-headers="[X-Remote-User]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612775   11892 flags.go:33] FLAG: --runtime-config=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612793   11892 flags.go:33] FLAG: --secure-port="443"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612800   11892 flags.go:33] FLAG: --service-account-api-audiences="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612816   11892 flags.go:33] FLAG: --service-account-issuer=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612822   11892 flags.go:33] FLAG: --service-account-key-file="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612834   11892 flags.go:33] FLAG: --service-account-lookup="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612840   11892 flags.go:33] FLAG: --service-account-max-token-expiration="0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612846   11892 flags.go:33] FLAG: --service-account-signing-key-file=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612856   11892 flags.go:33] FLAG: --service-cluster-ip-range="10.254.0.0/16"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612864   11892 flags.go:33] FLAG: --service-node-port-range="30000-32767"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612873   11892 flags.go:33] FLAG: --skip-headers="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612880   11892 flags.go:33] FLAG: --ssh-keyfile=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612885   11892 flags.go:33] FLAG: --ssh-user=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612905   11892 flags.go:33] FLAG: --stderrthreshold="2"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612911   11892 flags.go:33] FLAG: --storage-backend=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612917   11892 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612923   11892 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612929   11892 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612935   11892 flags.go:33] FLAG: --storage-driver-password="root"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612945   11892 flags.go:33] FLAG: --storage-driver-secure="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612951   11892 flags.go:33] FLAG: --storage-driver-table="stats"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612957   11892 flags.go:33] FLAG: --storage-driver-user="root"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612963   11892 flags.go:33] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.612969   11892 flags.go:33] FLAG: --storage-versions="admission.k8s.io/v1beta1,admissionregistration.k8s.io/v1beta1,apps/v1,auditregistration.k8s.io/v1alpha1,authentication.k8s.io/v1,authorization.k8s.io/v1,autoscaling/v1,batch/v1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,events.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,scheduling.k8s.io/v1beta1,settings.k8s.io/v1alpha1,storage.k8s.io/v1,v1"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613039   11892 flags.go:33] FLAG: --target-ram-mb="0"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613050   11892 flags.go:33] FLAG: --tls-cert-file="/etc/kubernetes/ssl/kubernetes-server.crt"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613057   11892 flags.go:33] FLAG: --tls-cipher-suites="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613071   11892 flags.go:33] FLAG: --tls-min-version=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613077   11892 flags.go:33] FLAG: --tls-private-key-file="/etc/kubernetes/ssl/kubernetes-server.key"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613084   11892 flags.go:33] FLAG: --tls-sni-cert-key="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613097   11892 flags.go:33] FLAG: --token-auth-file="/srv/kubernetes/known_tokens.csv"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613103   11892 flags.go:33] FLAG: --v="2"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613109   11892 flags.go:33] FLAG: --version="false"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613118   11892 flags.go:33] FLAG: --vmodule=""
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613125   11892 flags.go:33] FLAG: --watch-cache="true"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613135   11892 flags.go:33] FLAG: --watch-cache-sizes="[]"
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613157   11892 server.go:557] external host was not specified, using 172.16.10.36
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613436   11892 server.go:600] Initializing cache sizes based on 0MB limit
Aug 28 16:16:53 ctl01 kube-apiserver[11892]: I0828 16:16:53.613565   11892 server.go:146] Version: v1.13.5-3+98374c02d2d8c1
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.643877   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.744062   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.844213   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:53 ctl01 kubelet[11491]: E0828 16:16:53.944334   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.044651   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.144942   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.185092   11892 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.185161   11892 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.186857   11892 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.186888   11892 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.240599   11892 store.go:1414] Monitoring customresourcedefinitions.apiextensions.k8s.io count at <storage-prefix>//apiextensions.k8s.io/customresourcedefinitions
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.245091   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.261221   11892 master.go:228] Using reconciler: lease
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.345321   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.386970   11892 store.go:1414] Monitoring podtemplates count at <storage-prefix>//podtemplates
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.434914   11892 store.go:1414] Monitoring events count at <storage-prefix>//events
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.445766   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.491588   11892 store.go:1414] Monitoring limitranges count at <storage-prefix>//limitranges
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.541085   11892 store.go:1414] Monitoring resourcequotas count at <storage-prefix>//resourcequotas
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.545955   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.597663   11892 store.go:1414] Monitoring secrets count at <storage-prefix>//secrets
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.646176   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.650801   11892 store.go:1414] Monitoring persistentvolumes count at <storage-prefix>//persistentvolumes
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.705381   11892 store.go:1414] Monitoring persistentvolumeclaims count at <storage-prefix>//persistentvolumeclaims
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.756014   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.757115   11892 store.go:1414] Monitoring configmaps count at <storage-prefix>//configmaps
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.803081   11892 store.go:1414] Monitoring namespaces count at <storage-prefix>//namespaces
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.856175   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.868682   11892 store.go:1414] Monitoring endpoints count at <storage-prefix>//services/endpoints
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.925010   11892 store.go:1414] Monitoring nodes count at <storage-prefix>//minions
Aug 28 16:16:54 ctl01 kubelet[11491]: E0828 16:16:54.956323   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:54 ctl01 kube-apiserver[11892]: I0828 16:16:54.979052   11892 store.go:1414] Monitoring pods count at <storage-prefix>//pods
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.033828   11892 store.go:1414] Monitoring serviceaccounts count at <storage-prefix>//serviceaccounts
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.056382   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.088431   11892 store.go:1414] Monitoring services count at <storage-prefix>//services/specs
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.156511   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.247769   11892 store.go:1414] Monitoring replicationcontrollers count at <storage-prefix>//controllers
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.256641   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.356789   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.372683   11892 master.go:407] Skipping disabled API group "auditregistration.k8s.io".
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.372720   11892 master.go:415] Enabling API group "authentication.k8s.io".
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.372737   11892 master.go:415] Enabling API group "authorization.k8s.io".
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.425989   11892 store.go:1414] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.456941   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.476421   11892 store.go:1414] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.527753   11892 store.go:1414] Monitoring horizontalpodautoscalers.autoscaling count at <storage-prefix>//horizontalpodautoscalers
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.527786   11892 master.go:415] Enabling API group "autoscaling".
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.557026   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.583862   11892 store.go:1414] Monitoring jobs.batch count at <storage-prefix>//jobs
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.639837   11892 store.go:1414] Monitoring cronjobs.batch count at <storage-prefix>//cronjobs
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.639867   11892 master.go:415] Enabling API group "batch".
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.657168   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.692943   11892 store.go:1414] Monitoring certificatesigningrequests.certificates.k8s.io count at <storage-prefix>//certificatesigningrequests
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.692986   11892 master.go:415] Enabling API group "certificates.k8s.io".
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.746839   11892 store.go:1414] Monitoring leases.coordination.k8s.io count at <storage-prefix>//leases
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.746864   11892 master.go:415] Enabling API group "coordination.k8s.io".
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.757310   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.797670   11892 store.go:1414] Monitoring replicationcontrollers count at <storage-prefix>//controllers
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.847375   11892 store.go:1414] Monitoring daemonsets.apps count at <storage-prefix>//daemonsets
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.857480   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.904868   11892 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
Aug 28 16:16:55 ctl01 kubelet[11491]: E0828 16:16:55.957628   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:55 ctl01 kube-apiserver[11892]: I0828 16:16:55.969843   11892 store.go:1414] Monitoring ingresses.extensions count at <storage-prefix>//ingress
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.018929   11892 store.go:1414] Monitoring podsecuritypolicies.policy count at <storage-prefix>//podsecuritypolicy
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.057808   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.085134   11892 store.go:1414] Monitoring replicasets.apps count at <storage-prefix>//replicasets
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.136150   11892 store.go:1414] Monitoring networkpolicies.networking.k8s.io count at <storage-prefix>//networkpolicies
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.136188   11892 master.go:415] Enabling API group "extensions".
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.157913   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.185534   11892 store.go:1414] Monitoring networkpolicies.networking.k8s.io count at <storage-prefix>//networkpolicies
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.185557   11892 master.go:415] Enabling API group "networking.k8s.io".
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.238915   11892 store.go:1414] Monitoring poddisruptionbudgets.policy count at <storage-prefix>//poddisruptionbudgets
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.258062   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.288850   11892 store.go:1414] Monitoring podsecuritypolicies.policy count at <storage-prefix>//podsecuritypolicy
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.288867   11892 master.go:415] Enabling API group "policy".
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.350017   11892 store.go:1414] Monitoring roles.rbac.authorization.k8s.io count at <storage-prefix>//roles
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.358266   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.407245   11892 store.go:1414] Monitoring rolebindings.rbac.authorization.k8s.io count at <storage-prefix>//rolebindings
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.458025   11892 store.go:1414] Monitoring clusterroles.rbac.authorization.k8s.io count at <storage-prefix>//clusterroles
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.458408   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.508491   11892 store.go:1414] Monitoring clusterrolebindings.rbac.authorization.k8s.io count at <storage-prefix>//clusterrolebindings
Aug 28 16:16:56 ctl01 kubelet[11491]: I0828 16:16:56.542858   11491 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Aug 28 16:16:56 ctl01 kubelet[11491]: I0828 16:16:56.543199   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:16:56 ctl01 kubelet[11491]: I0828 16:16:56.544197   11491 kubelet_node_status.go:447] Recording NodeHasSufficientMemory event message for node ctl01
Aug 28 16:16:56 ctl01 kubelet[11491]: I0828 16:16:56.544239   11491 kubelet_node_status.go:447] Recording NodeHasNoDiskPressure event message for node ctl01
Aug 28 16:16:56 ctl01 kubelet[11491]: I0828 16:16:56.544254   11491 kubelet_node_status.go:447] Recording NodeHasSufficientPID event message for node ctl01
Aug 28 16:16:56 ctl01 kubelet[11491]: I0828 16:16:56.544277   11491 kubelet_node_status.go:72] Attempting to register node ctl01
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.558929   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.561911   11892 store.go:1414] Monitoring roles.rbac.authorization.k8s.io count at <storage-prefix>//roles
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.611338   11892 store.go:1414] Monitoring rolebindings.rbac.authorization.k8s.io count at <storage-prefix>//rolebindings
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.659101   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.659318   11892 store.go:1414] Monitoring clusterroles.rbac.authorization.k8s.io count at <storage-prefix>//clusterroles
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.715446   11892 store.go:1414] Monitoring clusterrolebindings.rbac.authorization.k8s.io count at <storage-prefix>//clusterrolebindings
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.715492   11892 master.go:415] Enabling API group "rbac.authorization.k8s.io".
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.759231   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.770204   11892 store.go:1414] Monitoring priorityclasses.scheduling.k8s.io count at <storage-prefix>//priorityclasses
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.770240   11892 master.go:415] Enabling API group "scheduling.k8s.io".
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.770265   11892 master.go:407] Skipping disabled API group "settings.k8s.io".
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.835781   11892 store.go:1414] Monitoring storageclasses.storage.k8s.io count at <storage-prefix>//storageclasses
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.859447   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.886844   11892 store.go:1414] Monitoring volumeattachments.storage.k8s.io count at <storage-prefix>//volumeattachments
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.949673   11892 store.go:1414] Monitoring storageclasses.storage.k8s.io count at <storage-prefix>//storageclasses
Aug 28 16:16:56 ctl01 kubelet[11491]: E0828 16:16:56.959680   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:56 ctl01 kube-apiserver[11892]: I0828 16:16:56.998770   11892 store.go:1414] Monitoring volumeattachments.storage.k8s.io count at <storage-prefix>//volumeattachments
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:56.998793   11892 master.go:415] Enabling API group "storage.k8s.io".
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.051668   11892 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.059902   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.107738   11892 store.go:1414] Monitoring statefulsets.apps count at <storage-prefix>//statefulsets
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.159783   11892 store.go:1414] Monitoring controllerrevisions.apps count at <storage-prefix>//controllerrevisions
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.160065   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.217969   11892 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.260208   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.292625   11892 store.go:1414] Monitoring statefulsets.apps count at <storage-prefix>//statefulsets
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.360335   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.367226   11892 store.go:1414] Monitoring daemonsets.apps count at <storage-prefix>//daemonsets
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.418241   11892 store.go:1414] Monitoring replicasets.apps count at <storage-prefix>//replicasets
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.460467   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.473421   11892 store.go:1414] Monitoring controllerrevisions.apps count at <storage-prefix>//controllerrevisions
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.524670   11892 store.go:1414] Monitoring deployments.apps count at <storage-prefix>//deployments
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.560736   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.577007   11892 store.go:1414] Monitoring statefulsets.apps count at <storage-prefix>//statefulsets
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.625285   11892 store.go:1414] Monitoring daemonsets.apps count at <storage-prefix>//daemonsets
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.660940   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.679738   11892 store.go:1414] Monitoring replicasets.apps count at <storage-prefix>//replicasets
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.742883   11892 store.go:1414] Monitoring controllerrevisions.apps count at <storage-prefix>//controllerrevisions
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.742936   11892 master.go:415] Enabling API group "apps".
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.761044   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.796075   11892 store.go:1414] Monitoring validatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//validatingwebhookconfigurations
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.853018   11892 store.go:1414] Monitoring mutatingwebhookconfigurations.admissionregistration.k8s.io count at <storage-prefix>//mutatingwebhookconfigurations
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.853086   11892 master.go:415] Enabling API group "admissionregistration.k8s.io".
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.861173   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.904371   11892 store.go:1414] Monitoring events count at <storage-prefix>//events
Aug 28 16:16:57 ctl01 kube-apiserver[11892]: I0828 16:16:57.904395   11892 master.go:415] Enabling API group "events.k8s.io".
Aug 28 16:16:57 ctl01 kubelet[11491]: E0828 16:16:57.961467   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kube-apiserver[11892]: W0828 16:16:58.045402   11892 genericapiserver.go:338] Skipping API batch/v2alpha1 because it has no resources.
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.061648   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.161899   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.262202   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kube-apiserver[11892]: W0828 16:16:58.276070   11892 genericapiserver.go:338] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Aug 28 16:16:58 ctl01 kube-apiserver[11892]: W0828 16:16:58.301485   11892 genericapiserver.go:338] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Aug 28 16:16:58 ctl01 kube-apiserver[11892]: W0828 16:16:58.337748   11892 genericapiserver.go:338] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.362346   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kube-apiserver[11892]: W0828 16:16:58.428183   11892 genericapiserver.go:338] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.462676   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kube-apiserver[11892]: [restful] 2019/08/28 16:16:58 log.go:33: [restful/swagger] listing is available at https://172.16.10.36:443/swaggerapi
Aug 28 16:16:58 ctl01 kube-apiserver[11892]: [restful] 2019/08/28 16:16:58 log.go:33: [restful/swagger] https://172.16.10.36:443/swaggerui/ is mapped to folder /swagger-ui/
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.562814   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.663061   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.763349   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.863546   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:58 ctl01 kubelet[11491]: E0828 16:16:58.963847   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.064069   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.164202   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.264778   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.365061   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.465216   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.565404   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.665610   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.765852   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.866068   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:16:59 ctl01 kubelet[11491]: E0828 16:16:59.966321   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.066643   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.166970   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.267220   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.358032   11491 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.367378   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.467718   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.568006   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.668205   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.768386   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.868632   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kubelet[11491]: E0828 16:17:00.969030   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:00 ctl01 kube-apiserver[11892]: [restful] 2019/08/28 16:17:00 log.go:33: [restful/swagger] listing is available at https://172.16.10.36:443/swaggerapi
Aug 28 16:17:00 ctl01 kube-apiserver[11892]: [restful] 2019/08/28 16:17:00 log.go:33: [restful/swagger] https://172.16.10.36:443/swaggerui/ is mapped to folder /swagger-ui/
Aug 28 16:17:01 ctl01 kube-apiserver[11892]: I0828 16:17:01.053991   11892 plugins.go:158] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
Aug 28 16:17:01 ctl01 kube-apiserver[11892]: I0828 16:17:01.054238   11892 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.069183   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kube-apiserver[11892]: I0828 16:17:01.101786   11892 store.go:1414] Monitoring apiservices.apiregistration.k8s.io count at <storage-prefix>//apiregistration.k8s.io/apiservices
Aug 28 16:17:01 ctl01 kube-apiserver[11892]: I0828 16:17:01.159230   11892 store.go:1414] Monitoring apiservices.apiregistration.k8s.io count at <storage-prefix>//apiregistration.k8s.io/apiservices
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.169377   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.269531   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.369680   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.469811   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.569929   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.670067   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.770225   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.870463   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:01 ctl01 CRON[11963]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 28 16:17:01 ctl01 kubelet[11491]: E0828 16:17:01.970688   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.070900   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.171138   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.271556   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.371953   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.472165   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.572394   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.672679   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.772909   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.873101   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:02 ctl01 kubelet[11491]: E0828 16:17:02.973326   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.073526   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.173680   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.273855   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.374001   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.474137   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.574259   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.674357   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.774553   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.874754   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:03 ctl01 kubelet[11491]: E0828 16:17:03.974975   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.075147   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.175429   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.211865   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.16.10.36:443/api/v1/pods?fieldSelector=spec.nodeName%3Dctl01&limit=500&resourceVersion=0: net/http: TLS handshake timeout
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.213787   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://172.16.10.36:443/api/v1/nodes?fieldSelector=metadata.name%3Dctl01&limit=500&resourceVersion=0: net/http: TLS handshake timeout
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.214852   11491 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.275712   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.375960   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kube-proxy[9844]: E0828 16:17:04.445299    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: Get https://172.16.10.36:443/api/v1/endpoints?limit=500&resourceVersion=0: net/http: TLS handshake timeout
Aug 28 16:17:04 ctl01 kube-proxy[9844]: E0828 16:17:04.446144    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://172.16.10.36:443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.476201   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.576387   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.676698   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.776916   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.877260   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:04 ctl01 kubelet[11491]: E0828 16:17:04.977508   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.078736   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.178913   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.279092   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.380135   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.480295   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.580451   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.680633   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.780827   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.880991   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:05 ctl01 kubelet[11491]: E0828 16:17:05.981223   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.008241   11892 deprecated_insecure_serving.go:51] Serving insecurely on [::]:8080
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.010339   11892 secure_serving.go:116] Serving securely on [::]:443
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.010581   11892 apiservice_controller.go:90] Starting APIServiceRegistrationController
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.010604   11892 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
Aug 28 16:17:06 ctl01 systemd[1]: Started Kubernetes API Server.
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.011453   11892 controller.go:84] Starting OpenAPI AggregationController
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.012776   11892 crd_finalizer.go:242] Starting CRDFinalizer
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.012852   11892 customresource_discovery_controller.go:203] Starting DiscoveryController
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.012877   11892 naming_controller.go:284] Starting NamingConditionController
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.012898   11892 establishing_controller.go:73] Starting EstablishingController
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.013153   11892 available_controller.go:283] Starting AvailableConditionController
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.013176   11892 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.022769   11892 autoregister_controller.go:136] Starting autoregister controller
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.022932   11892 cache.go:32] Waiting for caches to sync for autoregister controller
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.036764   11892 log.go:172] http: TLS handshake error from 172.16.10.56:48322: EOF
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.052033   11892 log.go:172] http: TLS handshake error from 172.16.10.55:33084: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.066548   11892 log.go:172] http: TLS handshake error from 172.16.10.55:33082: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.069587   11892 log.go:172] http: TLS handshake error from 172.16.10.55:33086: EOF
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.079555   11892 crdregistration_controller.go:112] Starting crd-autoregister controller
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.079589   11892 controller_utils.go:1027] Waiting for caches to sync for crd-autoregister controller
Aug 28 16:17:06 ctl01 kubelet[11491]: E0828 16:17:06.081578   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:06 ctl01 kube-proxy[9844]: E0828 16:17:06.082537    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.084313   11892 log.go:172] http: TLS handshake error from 172.16.10.56:48318: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.089493   11892 log.go:172] http: TLS handshake error from 172.16.10.56:48324: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.096239   11892 log.go:172] http: TLS handshake error from 172.16.10.56:48320: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.099005   11892 log.go:172] http: TLS handshake error from 172.16.10.56:48326: EOF
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.113887   11892 cache.go:39] Caches are synced for APIServiceRegistrationController controller
Aug 28 16:17:06 ctl01 systemd[1]: Started /bin/systemctl enable kube-apiserver.service.
Aug 28 16:17:06 ctl01 systemd[1]: Reloading.
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.125299   11892 log.go:172] http: TLS handshake error from 172.16.10.55:33088: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.133802   11892 log.go:172] http: TLS handshake error from 172.16.10.55:33090: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.139983   11892 cache.go:39] Caches are synced for autoregister controller
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.149600   11892 log.go:172] http: TLS handshake error from 172.16.10.55:33092: EOF
Aug 28 16:17:06 ctl01 kube-proxy[9844]: E0828 16:17:06.174951    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-proxy" cannot list resource "endpoints" in API group "" at the cluster scope
Aug 28 16:17:06 ctl01 kubelet[11491]: E0828 16:17:06.188029   11491 kubelet.go:2266] node "ctl01" not found
Aug 28 16:17:06 ctl01 kubelet[11491]: I0828 16:17:06.188210   11491 kubelet.go:1908] SyncLoop (ADD, "api"): ""
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.190033   11892 log.go:172] http: TLS handshake error from 172.16.10.36:43464: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.194247   11892 log.go:172] http: TLS handshake error from 172.16.10.36:43448: EOF
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.195993   11892 controller_utils.go:1034] Caches are synced for crd-autoregister controller
Aug 28 16:17:06 ctl01 kube-proxy[9844]: E0828 16:17:06.200666    9844 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"ctl01.15bf225d858e9471", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ctl01", UID:"ctl01", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kube-proxy.", Source:v1.EventSource{Component:"kube-proxy", Host:"ctl01"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf51c81518cf4c71, ext:499732733, loc:(*time.Location)(0xaf74780)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf51c81518cf4c71, ext:499732733, loc:(*time.Location)(0xaf74780)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:kube-proxy" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!)
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.204045   11892 log.go:172] http: TLS handshake error from 172.16.10.36:43462: EOF
Aug 28 16:17:06 ctl01 kubelet[11491]: I0828 16:17:06.205862   11491 kubelet_node_status.go:75] Successfully registered node ctl01
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.206203   11892 log.go:172] http: TLS handshake error from 172.16.10.36:43450: EOF
Aug 28 16:17:06 ctl01 kubelet[11491]: I0828 16:17:06.208156   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.213337   11892 cache.go:39] Caches are synced for AvailableConditionController controller
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.219605   11892 log.go:172] http: TLS handshake error from 172.16.10.36:43452: EOF
Aug 28 16:17:06 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:17:06 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.245719   11892 cacher.go:605] cacher (*apiregistration.APIService): 1 objects queued in incoming channel.
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.245757   11892 cacher.go:605] cacher (*apiregistration.APIService): 2 objects queued in incoming channel.
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.245774   11892 cacher.go:605] cacher (*apiregistration.APIService): 3 objects queued in incoming channel.
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.246304   11892 cacher.go:605] cacher (*apiregistration.APIService): 1 objects queued in incoming channel.
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.246342   11892 cacher.go:605] cacher (*apiregistration.APIService): 2 objects queued in incoming channel.
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-apiserver.service'] in directory '/root'
Aug 28 16:17:06 ctl01 kubelet[11491]: I0828 16:17:06.288190   11491 reconciler.go:154] Reconciler: start to sync state
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: W0828 16:17:06.293657   11892 lease.go:222] Resetting endpoints for master service "kubernetes" to [172.16.10.36]
Aug 28 16:17:06 ctl01 kube-apiserver[11892]: I0828 16:17:06.294409   11892 controller.go:608] quota admission added evaluator for: endpoints
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] {'kube-apiserver': True}
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Completed state [kube-apiserver] at time 16:17:06.300593 duration_in_ms=12945.584
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Running state [kube-scheduler] at time 16:17:06.307354
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [kube-scheduler]
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kube-scheduler.service', '-n', '0'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 systemd[1]: Started /bin/systemctl start kube-scheduler.service.
Aug 28 16:17:06 ctl01 systemd[1]: Started Kubernetes Scheduler Plugin.
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 systemd[1]: Started /bin/systemctl enable kube-scheduler.service.
Aug 28 16:17:06 ctl01 systemd[1]: Reloading.
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.538647   12041 flags.go:33] FLAG: --address="0.0.0.0"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539426   12041 flags.go:33] FLAG: --algorithm-provider=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539450   12041 flags.go:33] FLAG: --alsologtostderr="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539461   12041 flags.go:33] FLAG: --application-metrics-count-limit="100"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539468   12041 flags.go:33] FLAG: --authentication-kubeconfig=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539473   12041 flags.go:33] FLAG: --authentication-skip-lookup="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539481   12041 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539489   12041 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="true"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539495   12041 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539509   12041 flags.go:33] FLAG: --authorization-kubeconfig=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539515   12041 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539521   12041 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539528   12041 flags.go:33] FLAG: --azure-container-registry-config=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539533   12041 flags.go:33] FLAG: --bind-address="0.0.0.0"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539541   12041 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539547   12041 flags.go:33] FLAG: --cert-dir=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539553   12041 flags.go:33] FLAG: --client-ca-file=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539558   12041 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539568   12041 flags.go:33] FLAG: --config=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539574   12041 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539586   12041 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539592   12041 flags.go:33] FLAG: --contention-profiling="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539597   12041 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539603   12041 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539610   12041 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539617   12041 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539623   12041 flags.go:33] FLAG: --docker-only="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539629   12041 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539635   12041 flags.go:33] FLAG: --docker-tls="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539641   12041 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539646   12041 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539652   12041 flags.go:33] FLAG: --docker-tls-key="key.pem"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539658   12041 flags.go:33] FLAG: --enable-load-reader="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539663   12041 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539670   12041 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539676   12041 flags.go:33] FLAG: --failure-domains="kubernetes.io/hostname,failure-domain.beta.kubernetes.io/zone,failure-domain.beta.kubernetes.io/region"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539686   12041 flags.go:33] FLAG: --feature-gates=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539695   12041 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539701   12041 flags.go:33] FLAG: --hard-pod-affinity-symmetric-weight="1"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539709   12041 flags.go:33] FLAG: --help="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539715   12041 flags.go:33] FLAG: --housekeeping-interval="10s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539720   12041 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539728   12041 flags.go:33] FLAG: --kube-api-burst="100"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539735   12041 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539741   12041 flags.go:33] FLAG: --kube-api-qps="50"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539749   12041 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/scheduler.kubeconfig"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539756   12041 flags.go:33] FLAG: --leader-elect="true"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539786   12041 flags.go:33] FLAG: --leader-elect-lease-duration="15s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539793   12041 flags.go:33] FLAG: --leader-elect-renew-deadline="10s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539799   12041 flags.go:33] FLAG: --leader-elect-resource-lock="endpoints"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539805   12041 flags.go:33] FLAG: --leader-elect-retry-period="2s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539811   12041 flags.go:33] FLAG: --lock-object-name="kube-scheduler"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539817   12041 flags.go:33] FLAG: --lock-object-namespace="kube-system"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539823   12041 flags.go:33] FLAG: --log-backtrace-at=":0"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539832   12041 flags.go:33] FLAG: --log-cadvisor-usage="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539838   12041 flags.go:33] FLAG: --log-dir=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539844   12041 flags.go:33] FLAG: --log-file=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539859   12041 flags.go:33] FLAG: --log-flush-frequency="5s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539866   12041 flags.go:33] FLAG: --logtostderr="true"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539871   12041 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539878   12041 flags.go:33] FLAG: --master=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539883   12041 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539889   12041 flags.go:33] FLAG: --mesos-agent-timeout="10s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539895   12041 flags.go:33] FLAG: --policy-config-file=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539901   12041 flags.go:33] FLAG: --policy-configmap=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539906   12041 flags.go:33] FLAG: --policy-configmap-namespace="kube-system"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539936   12041 flags.go:33] FLAG: --port="10251"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539944   12041 flags.go:33] FLAG: --profiling="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539949   12041 flags.go:33] FLAG: --requestheader-allowed-names="[]"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539988   12041 flags.go:33] FLAG: --requestheader-client-ca-file=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.539998   12041 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540010   12041 flags.go:33] FLAG: --requestheader-group-headers="[x-remote-group]"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540022   12041 flags.go:33] FLAG: --requestheader-username-headers="[x-remote-user]"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540032   12041 flags.go:33] FLAG: --scheduler-name="default-scheduler"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540039   12041 flags.go:33] FLAG: --secure-port="10259"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540045   12041 flags.go:33] FLAG: --skip-headers="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540072   12041 flags.go:33] FLAG: --stderrthreshold="2"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540080   12041 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540086   12041 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540095   12041 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540102   12041 flags.go:33] FLAG: --storage-driver-password="root"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540107   12041 flags.go:33] FLAG: --storage-driver-secure="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540113   12041 flags.go:33] FLAG: --storage-driver-table="stats"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540118   12041 flags.go:33] FLAG: --storage-driver-user="root"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540124   12041 flags.go:33] FLAG: --tls-cert-file=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540130   12041 flags.go:33] FLAG: --tls-cipher-suites="[]"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540143   12041 flags.go:33] FLAG: --tls-min-version=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540148   12041 flags.go:33] FLAG: --tls-private-key-file=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540154   12041 flags.go:33] FLAG: --tls-sni-cert-key="[]"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540171   12041 flags.go:33] FLAG: --use-legacy-policy-config="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540177   12041 flags.go:33] FLAG: --v="2"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540183   12041 flags.go:33] FLAG: --version="false"
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540191   12041 flags.go:33] FLAG: --vmodule=""
Aug 28 16:17:06 ctl01 kube-scheduler[12041]: I0828 16:17:06.540197   12041 flags.go:33] FLAG: --write-config-to=""
Aug 28 16:17:06 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:17:06 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-scheduler.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] {'kube-scheduler': True}
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Completed state [kube-scheduler] at time 16:17:06.701546 duration_in_ms=394.191
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Running state [kube-controller-manager] at time 16:17:06.706450
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [kube-controller-manager]
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kube-controller-manager.service', '-n', '0'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:06 ctl01 systemd[1]: Started /bin/systemctl start kube-controller-manager.service.
Aug 28 16:17:06 ctl01 systemd[1]: Started Kubernetes Controller Manager.
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:06 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemd-run', '--scope', 'systemctl', 'enable', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:06 ctl01 systemd[1]: Started /bin/systemctl enable kube-controller-manager.service.
Aug 28 16:17:06 ctl01 systemd[1]: Reloading.
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931284   12121 flags.go:33] FLAG: --address="0.0.0.0"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931458   12121 flags.go:33] FLAG: --allocate-node-cidrs="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931481   12121 flags.go:33] FLAG: --allow-untagged-cloud="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931487   12121 flags.go:33] FLAG: --alsologtostderr="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931496   12121 flags.go:33] FLAG: --application-metrics-count-limit="100"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931502   12121 flags.go:33] FLAG: --attach-detach-reconcile-sync-period="1m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931510   12121 flags.go:33] FLAG: --authentication-kubeconfig=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931517   12121 flags.go:33] FLAG: --authentication-skip-lookup="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931523   12121 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931529   12121 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931535   12121 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931548   12121 flags.go:33] FLAG: --authorization-kubeconfig=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931554   12121 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931560   12121 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931566   12121 flags.go:33] FLAG: --azure-container-registry-config=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931571   12121 flags.go:33] FLAG: --bind-address="0.0.0.0"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931578   12121 flags.go:33] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931584   12121 flags.go:33] FLAG: --cert-dir=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931590   12121 flags.go:33] FLAG: --cidr-allocator-type="RangeAllocator"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931596   12121 flags.go:33] FLAG: --client-ca-file=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931601   12121 flags.go:33] FLAG: --cloud-config=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931607   12121 flags.go:33] FLAG: --cloud-provider=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931612   12121 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931622   12121 flags.go:33] FLAG: --cluster-cidr=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931628   12121 flags.go:33] FLAG: --cluster-name="kubernetes"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931634   12121 flags.go:33] FLAG: --cluster-signing-cert-file="/etc/kubernetes/ca/ca.pem"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931654   12121 flags.go:33] FLAG: --cluster-signing-key-file="/etc/kubernetes/ca/ca.key"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931661   12121 flags.go:33] FLAG: --concurrent-deployment-syncs="5"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931671   12121 flags.go:33] FLAG: --concurrent-endpoint-syncs="5"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931677   12121 flags.go:33] FLAG: --concurrent-gc-syncs="20"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931683   12121 flags.go:33] FLAG: --concurrent-namespace-syncs="10"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931690   12121 flags.go:33] FLAG: --concurrent-replicaset-syncs="5"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931696   12121 flags.go:33] FLAG: --concurrent-resource-quota-syncs="5"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931702   12121 flags.go:33] FLAG: --concurrent-service-syncs="1"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931719   12121 flags.go:33] FLAG: --concurrent-serviceaccount-token-syncs="5"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931725   12121 flags.go:33] FLAG: --concurrent-ttl-after-finished-syncs="5"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931732   12121 flags.go:33] FLAG: --concurrent_rc_syncs="5"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931738   12121 flags.go:33] FLAG: --configure-cloud-routes="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931744   12121 flags.go:33] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931751   12121 flags.go:33] FLAG: --containerd="unix:///var/run/containerd.sock"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931773   12121 flags.go:33] FLAG: --contention-profiling="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931781   12121 flags.go:33] FLAG: --controller-start-interval="0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931788   12121 flags.go:33] FLAG: --controllers="[*]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931802   12121 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931810   12121 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931816   12121 flags.go:33] FLAG: --deleting-pods-burst="0"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931823   12121 flags.go:33] FLAG: --deleting-pods-qps="0.1"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931834   12121 flags.go:33] FLAG: --deployment-controller-sync-period="30s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931841   12121 flags.go:33] FLAG: --disable-attach-detach-reconcile-sync="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931847   12121 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931854   12121 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931860   12121 flags.go:33] FLAG: --docker-only="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931866   12121 flags.go:33] FLAG: --docker-root="/var/lib/docker"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931873   12121 flags.go:33] FLAG: --docker-tls="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931879   12121 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931886   12121 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931892   12121 flags.go:33] FLAG: --docker-tls-key="key.pem"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931898   12121 flags.go:33] FLAG: --enable-dynamic-provisioning="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931904   12121 flags.go:33] FLAG: --enable-garbage-collector="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931932   12121 flags.go:33] FLAG: --enable-hostpath-provisioner="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931941   12121 flags.go:33] FLAG: --enable-load-reader="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931947   12121 flags.go:33] FLAG: --enable-taint-manager="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931954   12121 flags.go:33] FLAG: --event-storage-age-limit="default=0"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931960   12121 flags.go:33] FLAG: --event-storage-event-limit="default=0"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931967   12121 flags.go:33] FLAG: --experimental-cluster-signing-duration="8760h0m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.931974   12121 flags.go:33] FLAG: --external-cloud-volume-plugin=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932004   12121 flags.go:33] FLAG: --feature-gates=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932016   12121 flags.go:33] FLAG: --flex-volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932023   12121 flags.go:33] FLAG: --global-housekeeping-interval="1m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932030   12121 flags.go:33] FLAG: --help="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932035   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-cpu-initialization-period="5m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932041   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-downscale-delay="5m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932047   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-downscale-stabilization="5m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932053   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-initial-readiness-delay="30s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932059   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-sync-period="15s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932064   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-tolerance="0.1"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932073   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-upscale-delay="3m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932079   12121 flags.go:33] FLAG: --horizontal-pod-autoscaler-use-rest-clients="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932084   12121 flags.go:33] FLAG: --housekeeping-interval="10s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932090   12121 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932098   12121 flags.go:33] FLAG: --kube-api-burst="30"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932104   12121 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932110   12121 flags.go:33] FLAG: --kube-api-qps="20"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932116   12121 flags.go:33] FLAG: --kubeconfig="/etc/kubernetes/controller-manager.kubeconfig"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932123   12121 flags.go:33] FLAG: --large-cluster-size-threshold="50"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932128   12121 flags.go:33] FLAG: --leader-elect="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932134   12121 flags.go:33] FLAG: --leader-elect-lease-duration="15s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932140   12121 flags.go:33] FLAG: --leader-elect-renew-deadline="10s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932145   12121 flags.go:33] FLAG: --leader-elect-resource-lock="endpoints"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932151   12121 flags.go:33] FLAG: --leader-elect-retry-period="2s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932157   12121 flags.go:33] FLAG: --log-backtrace-at=":0"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932165   12121 flags.go:33] FLAG: --log-cadvisor-usage="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932172   12121 flags.go:33] FLAG: --log-dir=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932177   12121 flags.go:33] FLAG: --log-file=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932182   12121 flags.go:33] FLAG: --log-flush-frequency="5s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932188   12121 flags.go:33] FLAG: --logtostderr="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932194   12121 flags.go:33] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932201   12121 flags.go:33] FLAG: --master=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932217   12121 flags.go:33] FLAG: --mesos-agent="127.0.0.1:5051"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932224   12121 flags.go:33] FLAG: --mesos-agent-timeout="10s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932244   12121 flags.go:33] FLAG: --min-resync-period="12h0m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932251   12121 flags.go:33] FLAG: --namespace-sync-period="5m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932258   12121 flags.go:33] FLAG: --node-cidr-mask-size="24"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932265   12121 flags.go:33] FLAG: --node-eviction-rate="0.1"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932272   12121 flags.go:33] FLAG: --node-monitor-grace-period="40s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932278   12121 flags.go:33] FLAG: --node-monitor-period="5s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932285   12121 flags.go:33] FLAG: --node-startup-grace-period="1m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932291   12121 flags.go:33] FLAG: --node-sync-period="0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932298   12121 flags.go:33] FLAG: --pod-eviction-timeout="5m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932304   12121 flags.go:33] FLAG: --port="10252"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932311   12121 flags.go:33] FLAG: --profiling="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932317   12121 flags.go:33] FLAG: --pv-recycler-increment-timeout-nfs="30"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932324   12121 flags.go:33] FLAG: --pv-recycler-minimum-timeout-hostpath="60"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932330   12121 flags.go:33] FLAG: --pv-recycler-minimum-timeout-nfs="300"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932337   12121 flags.go:33] FLAG: --pv-recycler-pod-template-filepath-hostpath=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932343   12121 flags.go:33] FLAG: --pv-recycler-pod-template-filepath-nfs=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932349   12121 flags.go:33] FLAG: --pv-recycler-timeout-increment-hostpath="30"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932355   12121 flags.go:33] FLAG: --pvclaimbinder-sync-period="15s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932362   12121 flags.go:33] FLAG: --register-retry-count="10"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932368   12121 flags.go:33] FLAG: --requestheader-allowed-names="[]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932380   12121 flags.go:33] FLAG: --requestheader-client-ca-file=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932386   12121 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932395   12121 flags.go:33] FLAG: --requestheader-group-headers="[x-remote-group]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932406   12121 flags.go:33] FLAG: --requestheader-username-headers="[x-remote-user]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932414   12121 flags.go:33] FLAG: --resource-quota-sync-period="5m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932421   12121 flags.go:33] FLAG: --root-ca-file="/etc/kubernetes/ssl/ca-kubernetes.crt"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932428   12121 flags.go:33] FLAG: --route-reconciliation-period="10s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932435   12121 flags.go:33] FLAG: --secondary-node-eviction-rate="0.01"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932442   12121 flags.go:33] FLAG: --secure-port="10257"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932449   12121 flags.go:33] FLAG: --service-account-private-key-file="/etc/kubernetes/ssl/kubernetes-server.key"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932470   12121 flags.go:33] FLAG: --service-cluster-ip-range=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932477   12121 flags.go:33] FLAG: --skip-headers="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932483   12121 flags.go:33] FLAG: --stderrthreshold="2"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932490   12121 flags.go:33] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932496   12121 flags.go:33] FLAG: --storage-driver-db="cadvisor"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932503   12121 flags.go:33] FLAG: --storage-driver-host="localhost:8086"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932510   12121 flags.go:33] FLAG: --storage-driver-password="root"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932516   12121 flags.go:33] FLAG: --storage-driver-secure="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932523   12121 flags.go:33] FLAG: --storage-driver-table="stats"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932529   12121 flags.go:33] FLAG: --storage-driver-user="root"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932536   12121 flags.go:33] FLAG: --terminated-pod-gc-threshold="12500"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932543   12121 flags.go:33] FLAG: --tls-cert-file=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932549   12121 flags.go:33] FLAG: --tls-cipher-suites="[]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932560   12121 flags.go:33] FLAG: --tls-min-version=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932566   12121 flags.go:33] FLAG: --tls-private-key-file=""
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932572   12121 flags.go:33] FLAG: --tls-sni-cert-key="[]"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932582   12121 flags.go:33] FLAG: --unhealthy-zone-threshold="0.55"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932590   12121 flags.go:33] FLAG: --use-service-account-credentials="true"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932597   12121 flags.go:33] FLAG: --v="2"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932604   12121 flags.go:33] FLAG: --version="false"
Aug 28 16:17:06 ctl01 kube-controller-manager[12121]: I0828 16:17:06.932613   12121 flags.go:33] FLAG: --vmodule=""
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: I0828 16:17:07.004419   12041 serving.go:318] Generated self-signed cert in-memory
Aug 28 16:17:07 ctl01 systemd[1]: kubelet.service: Dependency Conflicts=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:17:07 ctl01 systemd[1]: kubelet.service: Dependency ConflictedBy=cadvisor.service dropped, merged into kubelet.service
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.027602   11892 storage_scheduling.go:91] created PriorityClass system-node-critical with value 2000001000
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.029895   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/cluster-admin
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.031873   11892 storage_scheduling.go:91] created PriorityClass system-cluster-critical with value 2000000000
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.031894   11892 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.035185   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:discovery
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.038275   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:basic-user
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.042598   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/admin
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.051082   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/edit
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.053911   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/view
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.057557   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-admin
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-controller-manager.service'] in directory '/root'
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.066370   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-edit
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.070111   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-view
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.073037   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:heapster
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.076065   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.078921   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-problem-detector
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.081877   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-proxier
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] {'kube-controller-manager': True}
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Completed state [kube-controller-manager] at time 16:17:07.083301 duration_in_ms=376.85
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Running state [kubectl create ns "netchecker"] at time 16:17:07.083670
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing state cmd.run for [kubectl create ns "netchecker"]
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing command 'kubectl get ns -o=custom-columns=NAME:.metadata.name | grep -v NAME | grep "netchecker"' in directory '/root'
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.084540   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kubelet-api-admin
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.087722   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:node-bootstrapper
Aug 28 16:17:07 ctl01 kube-proxy[9844]: E0828 16:17:07.088549    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.090446   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:auth-delegator
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.094402   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-aggregator
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.098175   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-controller-manager
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.104130   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-scheduler
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.107296   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:kube-dns
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.110416   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:persistent-volume-provisioner
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.113324   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:csi-external-attacher
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.117262   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:aws-cloud-provider
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.121884   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:nodeclient
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.124923   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.128378   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:volume-scheduler
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.132485   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:csi-external-provisioner
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.135418   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:attachdetach-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.138362   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.141385   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:cronjob-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.144828   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:daemon-set-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.148317   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:deployment-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.151813   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:disruption-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.155680   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:endpoint-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.159622   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:expand-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.162693   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.165766   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.170943   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:job-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.173671   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:namespace-controller
Aug 28 16:17:07 ctl01 kube-proxy[9844]: E0828 16:17:07.176252    9844 reflector.go:125] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:kube-proxy" cannot list resource "endpoints" in API group "" at the cluster scope
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.176559   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:node-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.179504   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.182568   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.186222   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:replicaset-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.189240   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:replication-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.192286   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:resourcequota-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.195518   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:route-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.198116   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:service-account-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.201359   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:service-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.204750   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:statefulset-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.208007   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:ttl-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.213519   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:certificate-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.253455   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.293884   11892 storage_rbac.go:187] created clusterrole.rbac.authorization.k8s.io/system:controller:pv-protection-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.333716   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/cluster-admin
Aug 28 16:17:07 ctl01 kube-controller-manager[12121]: I0828 16:17:07.349715   12121 serving.go:318] Generated self-signed cert in-memory
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing command 'kubectl create ns "netchecker"' in directory '/root'
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.374680   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:discovery
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.413456   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:basic-user
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.454362   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node-proxier
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.493855   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-controller-manager
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] {'pid': 12216, 'retcode': 0, 'stderr': '', 'stdout': 'namespace/netchecker created'}
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Completed state [kubectl create ns "netchecker"] at time 16:17:07.521234 duration_in_ms=437.563
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/calicoctl] at time 16:17:07.522017
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/bin/calicoctl]
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.533319   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-dns
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.573964   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:kube-scheduler
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.613855   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:aws-cloud-provider
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.653618   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:node
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] File /usr/bin/calicoctl is in the correct state
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/calicoctl] at time 16:17:07.666894 duration_in_ms=144.876
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/birdcl] at time 16:17:07.667319
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/usr/bin/birdcl]
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] File /usr/bin/birdcl is in the correct state
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/birdcl] at time 16:17:07.671025 duration_in_ms=3.706
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Running state [/opt/cni/bin/calico] at time 16:17:07.671403
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico]
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.693611   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:volume-scheduler
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.733702   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:attachdetach-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.773472   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] File /opt/cni/bin/calico is in the correct state
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Completed state [/opt/cni/bin/calico] at time 16:17:07.798872 duration_in_ms=127.468
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Running state [/opt/cni/bin/calico-ipam] at time 16:17:07.799288
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/opt/cni/bin/calico-ipam]
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.813547   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:cronjob-controller
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: W0828 16:17:07.824307   12041 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: W0828 16:17:07.824422   12041 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: W0828 16:17:07.824450   12041 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: I0828 16:17:07.832496   12041 server.go:150] Version: v1.13.5-3+98374c02d2d8c1
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: I0828 16:17:07.832541   12041 defaults.go:210] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: I0828 16:17:07.832776   12041 factory.go:1157] Creating scheduler from algorithm provider 'DefaultProvider'
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: I0828 16:17:07.832804   12041 factory.go:1256] Creating scheduler with fit predicates 'map[CheckVolumeBinding:{} CheckNodeUnschedulable:{} NoDiskConflict:{} GeneralPredicates:{} PodToleratesNodeTaints:{} NoVolumeZoneConflict:{} MaxCSIVolumeCountPred:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxGCEPDVolumeCount:{} MaxEBSVolumeCount:{}]' and priority functions 'map[TaintTolerationPriority:{} ImageLocalityPriority:{} SelectorSpreadPriority:{} InterPodAffinityPriority:{} LeastRequestedPriority:{} BalancedResourceAllocation:{} NodePreferAvoidPodsPriority:{} NodeAffinityPriority:{}]'
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: W0828 16:17:07.839128   12041 authorization.go:47] Authorization is disabled
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: W0828 16:17:07.839256   12041 authentication.go:55] Authentication is disabled
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: I0828 16:17:07.839365   12041 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251
Aug 28 16:17:07 ctl01 kube-scheduler[12041]: I0828 16:17:07.840045   12041 secure_serving.go:116] Serving securely on [::]:10259
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.873071   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:daemon-set-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.893545   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:deployment-controller
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.933907   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:disruption-controller
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] File /opt/cni/bin/calico-ipam is in the correct state
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Completed state [/opt/cni/bin/calico-ipam] at time 16:17:07.938706 duration_in_ms=139.418
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/cni/net.d/10-calico.conf] at time 16:17:07.939049
Aug 28 16:17:07 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/cni/net.d/10-calico.conf]
Aug 28 16:17:07 ctl01 kube-apiserver[11892]: I0828 16:17:07.974763   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpoint-controller
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.014461   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.053639   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.094188   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: W0828 16:17:08.102417   12121 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: W0828 16:17:08.102514   12121 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: W0828 16:17:08.102539   12121 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: I0828 16:17:08.102566   12121 controllermanager.go:151] Version: v1.13.5-3+98374c02d2d8c1
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: I0828 16:17:08.103000   12121 secure_serving.go:116] Serving securely on [::]:10257
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: I0828 16:17:08.103365   12121 deprecated_insecure_serving.go:51] Serving insecurely on [::]:10252
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: I0828 16:17:08.103515   12121 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-controller-manager...
Aug 28 16:17:08 ctl01 kube-proxy[9844]: I0828 16:17:08.116576    9844 controller_utils.go:1034] Caches are synced for service config controller
Aug 28 16:17:08 ctl01 kube-proxy[9844]: I0828 16:17:08.116671    9844 proxier.go:645] Not syncing iptables until Services and Endpoints have been received from master
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.133645   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: I0828 16:17:08.151978   12121 leaderelection.go:214] successfully acquired lease kube-system/kube-controller-manager
Aug 28 16:17:08 ctl01 kube-controller-manager[12121]: I0828 16:17:08.155226   12121 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"485515ad-c9af-11e9-b58a-5254003f9b0c", APIVersion:"v1", ResourceVersion:"189", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' ctl01_484de4dc-c9af-11e9-b09c-5254003f9b0c became leader
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.173538   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.214187   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller
Aug 28 16:17:08 ctl01 kube-proxy[9844]: I0828 16:17:08.216484    9844 controller_utils.go:1034] Caches are synced for endpoints config controller
Aug 28 16:17:08 ctl01 kube-proxy[9844]: I0828 16:17:08.216643    9844 service.go:309] Adding new service port "default/kubernetes:https" at 10.254.0.1:443/TCP
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.253458   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.294686   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pod-garbage-collector
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.334605   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replicaset-controller
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.373876   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:replication-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.414277   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:resourcequota-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.453778   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:route-controller
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.495018   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-account-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.534295   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:service-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.574936   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:statefulset-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.614168   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:ttl-controller
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] File /etc/cni/net.d/10-calico.conf is in the correct state
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/cni/net.d/10-calico.conf] at time 16:17:08.633682 duration_in_ms=694.632
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/calico/network-environment] at time 16:17:08.634167
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/calico/network-environment]
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.654444   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:certificate-controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.694823   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pvc-protection-controller
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.734581   11892 storage_rbac.go:215] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:pv-protection-controller
Aug 28 16:17:08 ctl01 kube-scheduler[12041]: I0828 16:17:08.741463   12041 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.772730   11892 controller.go:608] quota admission added evaluator for: roles.rbac.authorization.k8s.io
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.774731   11892 storage_rbac.go:246] created role.rbac.authorization.k8s.io/extension-apiserver-authentication-reader in kube-system
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.814747   11892 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
Aug 28 16:17:08 ctl01 kube-scheduler[12041]: I0828 16:17:08.841651   12041 controller_utils.go:1034] Caches are synced for scheduler controller
Aug 28 16:17:08 ctl01 kube-scheduler[12041]: I0828 16:17:08.841717   12041 leaderelection.go:205] attempting to acquire leader lease  kube-system/kube-scheduler...
Aug 28 16:17:08 ctl01 kube-scheduler[12041]: I0828 16:17:08.846711   12041 leaderelection.go:214] successfully acquired lease kube-system/kube-scheduler
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.854172   11892 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
Aug 28 16:17:08 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.893633   11892 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.934865   11892 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
Aug 28 16:17:08 ctl01 kube-apiserver[11892]: I0828 16:17:08.974102   11892 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.015986   11892 storage_rbac.go:246] created role.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] File /etc/calico/network-environment is in the correct state
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/calico/network-environment] at time 16:17:09.034939 duration_in_ms=400.769
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/calico/calicoctl.cfg] at time 16:17:09.036490
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/calico/calicoctl.cfg]
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.052736   11892 controller.go:608] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.054996   11892 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-controller-manager in kube-system
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.094253   11892 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system::leader-locking-kube-scheduler in kube-system
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.134467   11892 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-system
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.174252   11892 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:cloud-provider in kube-system
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.217696   11892 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system
Aug 28 16:17:09 ctl01 kube-apiserver[11892]: I0828 16:17:09.253681   11892 storage_rbac.go:276] created rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] File /etc/calico/calicoctl.cfg is in the correct state
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/calico/calicoctl.cfg] at time 16:17:09.498998 duration_in_ms=462.508
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/calico-node.service] at time 16:17:09.500533
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/calico-node.service]
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] File /etc/systemd/system/calico-node.service is in the correct state
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/calico-node.service] at time 16:17:09.538218 duration_in_ms=37.685
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/var/lib/calico] at time 16:17:09.538599
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/var/lib/calico]
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Directory /var/lib/calico is in the correct state
Aug 28 16:17:09 ctl01 salt-minion[4490]: Directory /var/lib/calico updated
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/var/lib/calico] at time 16:17:09.540410 duration_in_ms=1.81
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/var/log/calico] at time 16:17:09.540767
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/var/log/calico]
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Directory /var/log/calico is in the correct state
Aug 28 16:17:09 ctl01 salt-minion[4490]: Directory /var/log/calico updated
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [/var/log/calico] at time 16:17:09.542542 duration_in_ms=1.775
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Running state [calico-node] at time 16:17:09.545686
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [calico-node]
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'calico-node.service', '-n', '0'] in directory '/root'
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'calico-node.service'] in directory '/root'
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'calico-node.service'] in directory '/root'
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] The service calico-node is already running
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Completed state [calico-node] at time 16:17:09.626267 duration_in_ms=80.58
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/proxy.kubeconfig] at time 16:17:09.627123
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/proxy.kubeconfig]
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:09 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.179397   12121 plugins.go:103] No cloud provider specified.
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.181951   12121 controllermanager.go:501] Starting "garbagecollector"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.182393   12121 controller_utils.go:1027] Waiting for caches to sync for tokens controller
Aug 28 16:17:10 ctl01 kube-apiserver[11892]: I0828 16:17:10.189509   11892 controller.go:608] quota admission added evaluator for: serviceaccounts
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.282636   12121 controller_utils.go:1034] Caches are synced for tokens controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: W0828 16:17:10.330743   12121 garbagecollector.go:649] failed to discover preferred resources: the cache has not been filled yet
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.331116   12121 garbagecollector.go:133] Starting garbage collector controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.331133   12121 controllermanager.go:516] Started "garbagecollector"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.331137   12121 controller_utils.go:1027] Waiting for caches to sync for garbage collector controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.331182   12121 graph_builder.go:308] GraphBuilder running
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.331156   12121 controllermanager.go:501] Starting "nodeipam"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: W0828 16:17:10.331199   12121 controllermanager.go:508] Skipping "nodeipam"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.331210   12121 controllermanager.go:501] Starting "nodelifecycle"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.373979   12121 node_lifecycle_controller.go:272] Sending events to api server.
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.374382   12121 node_lifecycle_controller.go:312] Controller is using taint based evictions.
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.374571   12121 taint_manager.go:175] Sending events to api server.
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.374729   12121 node_lifecycle_controller.go:378] Controller will taint node by condition.
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.374784   12121 controllermanager.go:516] Started "nodelifecycle"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.374806   12121 controllermanager.go:501] Starting "persistentvolume-binder"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.374951   12121 node_lifecycle_controller.go:423] Starting node controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.375002   12121 controller_utils.go:1027] Waiting for caches to sync for taint controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.383997   12121 garbagecollector.go:204] syncing garbage collector with updated resources from discovery (attempt 1): added: [/v1, Resource=configmaps /v1, Resource=endpoints /v1, Resource=events /v1, Resource=limitranges /v1, Resource=namespaces /v1, Resource=nodes /v1, Resource=persistentvolumeclaims /v1, Resource=persistentvolumes /v1, Resource=pods /v1, Resource=podtemplates /v1, Resource=replicationcontrollers /v1, Resource=resourcequotas /v1, Resource=secrets /v1, Resource=serviceaccounts /v1, Resource=services admissionregistration.k8s.io/v1beta1, Resource=mutatingwebhookconfigurations admissionregistration.k8s.io/v1beta1, Resource=validatingwebhookconfigurations apiextensions.k8s.io/v1beta1, Resource=customresourcedefinitions apiregistration.k8s.io/v1, Resource=apiservices apps/v1, Resource=controllerrevisions apps/v1, Resource=daemonsets apps/v1, Resource=deployments apps/v1, Resource=replicasets apps/v1, Resource=statefulsets autoscaling/v1, Resource=horizontalpodautoscalers batch/v1, Resource=jobs batch/v1beta1, Resource=cronjobs certificates.k8s.io/v1beta1, Resource=certificatesigningrequests coordination.k8s.io/v1beta1, Resource=leases events.k8s.io/v1beta1, Resource=events extensions/v1beta1, Resource=daemonsets extensions/v1beta1, Resource=deployments extensions/v1beta1, Resource=ingresses extensions/v1beta1, Resource=networkpolicies extensions/v1beta1, Resource=podsecuritypolicies extensions/v1beta1, Resource=replicasets networking.k8s.io/v1, Resource=networkpolicies policy/v1beta1, Resource=poddisruptionbudgets policy/v1beta1, Resource=podsecuritypolicies rbac.authorization.k8s.io/v1, Resource=clusterrolebindings rbac.authorization.k8s.io/v1, Resource=clusterroles rbac.authorization.k8s.io/v1, Resource=rolebindings rbac.authorization.k8s.io/v1, Resource=roles scheduling.k8s.io/v1beta1, Resource=priorityclasses storage.k8s.io/v1, Resource=storageclasses storage.k8s.io/v1, Resource=volumeattachments], removed: []
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426124   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/host-path"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426149   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/nfs"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426161   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426175   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426186   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/quobyte"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426195   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426204   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/flocker"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426224   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426237   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426248   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/local-volume"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426258   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426267   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426276   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426288   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426299   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426308   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426318   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426380   12121 controllermanager.go:516] Started "persistentvolume-binder"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426392   12121 controllermanager.go:501] Starting "persistentvolume-expander"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426497   12121 pv_controller_base.go:271] Starting persistent volume controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.426508   12121 controller_utils.go:1027] Waiting for caches to sync for persistent volume controller
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] File /etc/kubernetes/proxy.kubeconfig is in the correct state
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/proxy.kubeconfig] at time 16:17:10.445190 duration_in_ms=818.068
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kube-proxy.service] at time 16:17:10.445615
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/systemd/system/kube-proxy.service]
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466208   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466233   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466245   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466261   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466271   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466280   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/glusterfs"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466289   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466298   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466325   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-file"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466337   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466347   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466357   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466366   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466475   12121 controllermanager.go:516] Started "persistentvolume-expander"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466489   12121 controllermanager.go:501] Starting "podgc"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466595   12121 expand_controller.go:153] Starting expand controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.466606   12121 controller_utils.go:1027] Waiting for caches to sync for expand controller
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] File /etc/systemd/system/kube-proxy.service is in the correct state
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kube-proxy.service] at time 16:17:10.473405 duration_in_ms=27.79
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kube-proxy] at time 16:17:10.473722
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/default/kube-proxy]
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] File /etc/default/kube-proxy is in the correct state
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kube-proxy] at time 16:17:10.477761 duration_in_ms=4.039
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Running state [kube-proxy] at time 16:17:10.481041
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing state service.running for [kube-proxy]
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kube-proxy.service', '-n', '0'] in directory '/root'
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-active', 'kube-proxy.service'] in directory '/root'
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.513082   12121 controllermanager.go:516] Started "podgc"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.513235   12121 controllermanager.go:501] Starting "horizontalpodautoscaling"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.513161   12121 gc_controller.go:76] Starting GC controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.513522   12121 controller_utils.go:1027] Waiting for caches to sync for GC controller
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'is-enabled', 'kube-proxy.service'] in directory '/root'
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] The service kube-proxy is already running
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [kube-proxy] at time 16:17:10.545538 duration_in_ms=64.496
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Running state [/srv/kubernetes] at time 16:17:10.550334
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing state file.directory for [/srv/kubernetes]
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Directory /srv/kubernetes is in the correct state
Aug 28 16:17:10 ctl01 salt-minion[4490]: Directory /srv/kubernetes updated
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [/srv/kubernetes] at time 16:17:10.551968 duration_in_ms=1.634
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Running state [/srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:17:10.552842
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml]
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/rolebinding.yml'
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:17:10 ctl01 salt-minion[4490]: New file
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [/srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:17:10.597540 duration_in_ms=44.697
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Running state [kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:17:10.598926
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing state cmd.run for [kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml]
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Executing command 'kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml' in directory '/root'
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.785344   12121 controllermanager.go:516] Started "horizontalpodautoscaling"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.785386   12121 controllermanager.go:501] Starting "ttl"
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.785389   12121 horizontal.go:156] Starting HPA controller
Aug 28 16:17:10 ctl01 kube-controller-manager[12121]: I0828 16:17:10.785411   12121 controller_utils.go:1027] Waiting for caches to sync for HPA controller
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] {'pid': 12518, 'retcode': 0, 'stderr': '', 'stdout': 'clusterrolebinding.rbac.authorization.k8s.io/root-cluster-admin-binding created'}
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Completed state [kubectl apply -f /srv/kubernetes/roles/cluster-admin/root-cluster-admin-binding-rolebinding.yml] at time 16:17:10.914633 duration_in_ms=315.706
Aug 28 16:17:10 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161630039595
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.041744   12121 controllermanager.go:516] Started "ttl"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: W0828 16:17:11.041778   12121 controllermanager.go:495] "tokencleaner" is disabled
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.041792   12121 controllermanager.go:501] Starting "replicationcontroller"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.041814   12121 ttl_controller.go:116] Starting TTL controller
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.041837   12121 controller_utils.go:1027] Waiting for caches to sync for TTL controller
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.288967   12121 controllermanager.go:516] Started "replicationcontroller"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.288992   12121 controllermanager.go:501] Starting "route"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.289000   12121 core.go:151] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: W0828 16:17:11.289007   12121 controllermanager.go:508] Skipping "route"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.289014   12121 controllermanager.go:501] Starting "ttl-after-finished"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: W0828 16:17:11.289022   12121 controllermanager.go:508] Skipping "ttl-after-finished"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.289028   12121 controllermanager.go:501] Starting "csrcleaner"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.289064   12121 replica_set.go:182] Starting replicationcontroller controller
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.289075   12121 controller_utils.go:1027] Waiting for caches to sync for ReplicationController controller
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.538616   12121 controllermanager.go:516] Started "csrcleaner"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.538916   12121 controllermanager.go:501] Starting "cronjob"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.538679   12121 cleaner.go:81] Starting CSR cleaner controller
Aug 28 16:17:11 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command state.sls with jid 20190828161711578885
Aug 28 16:17:11 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 12543
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.792350   12121 controllermanager.go:516] Started "cronjob"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: W0828 16:17:11.792376   12121 controllermanager.go:495] "bootstrapsigner" is disabled
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.792386   12121 controllermanager.go:501] Starting "service"
Aug 28 16:17:11 ctl01 kube-controller-manager[12121]: I0828 16:17:11.792515   12121 cronjob_controller.go:92] Starting CronJob Manager
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: E0828 16:17:12.039283   12121 core.go:76] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: W0828 16:17:12.039307   12121 controllermanager.go:508] Skipping "service"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.039319   12121 controllermanager.go:501] Starting "pv-protection"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.290414   12121 controllermanager.go:516] Started "pv-protection"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.290474   12121 controllermanager.go:501] Starting "serviceaccount"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.290481   12121 pv_protection_controller.go:81] Starting PV protection controller
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.290515   12121 controller_utils.go:1027] Waiting for caches to sync for PV protection controller
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.539139   12121 controllermanager.go:516] Started "serviceaccount"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.539440   12121 controllermanager.go:501] Starting "job"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.539183   12121 serviceaccounts_controller.go:115] Starting service account controller
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.540272   12121 controller_utils.go:1027] Waiting for caches to sync for service account controller
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.790724   12121 controllermanager.go:516] Started "job"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.790940   12121 controllermanager.go:501] Starting "replicaset"
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.790758   12121 job_controller.go:143] Starting job controller
Aug 28 16:17:12 ctl01 kube-controller-manager[12121]: I0828 16:17:12.791170   12121 controller_utils.go:1027] Waiting for caches to sync for job controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.044604   12121 controllermanager.go:516] Started "replicaset"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.044928   12121 controllermanager.go:501] Starting "disruption"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.044911   12121 replica_set.go:182] Starting replicaset controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.045268   12121 controller_utils.go:1027] Waiting for caches to sync for ReplicaSet controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.293598   12121 controllermanager.go:516] Started "disruption"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.294070   12121 controllermanager.go:501] Starting "attachdetach"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.293685   12121 disruption.go:288] Starting disruption controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.294384   12121 controller_utils.go:1027] Waiting for caches to sync for disruption controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.688956   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/aws-ebs"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.688998   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/gce-pd"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689011   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/cinder"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689029   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/portworx-volume"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689040   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/vsphere-volume"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689050   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/azure-disk"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689060   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/photon-pd"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689070   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/scaleio"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689083   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/storageos"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689093   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/fc"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689103   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/iscsi"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689113   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/rbd"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689124   12121 plugins.go:547] Loaded volume plugin "kubernetes.io/csi"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689299   12121 controllermanager.go:516] Started "attachdetach"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689313   12121 controllermanager.go:501] Starting "endpoint"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689350   12121 attach_detach_controller.go:315] Starting attach detach controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.689359   12121 controller_utils.go:1027] Waiting for caches to sync for attach detach controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.940701   12121 controllermanager.go:516] Started "endpoint"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.940726   12121 controllermanager.go:501] Starting "namespace"
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.940726   12121 endpoints_controller.go:160] Starting endpoint controller
Aug 28 16:17:13 ctl01 kube-controller-manager[12121]: I0828 16:17:13.940747   12121 controller_utils.go:1027] Waiting for caches to sync for endpoint controller
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.201261   12121 controllermanager.go:516] Started "namespace"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.201285   12121 controllermanager.go:501] Starting "csrapproving"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.201323   12121 namespace_controller.go:186] Starting namespace controller
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.201333   12121 controller_utils.go:1027] Waiting for caches to sync for namespace controller
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.337336   12121 controllermanager.go:516] Started "csrapproving"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.337372   12121 controllermanager.go:501] Starting "clusterrole-aggregation"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.337447   12121 certificate_controller.go:113] Starting certificate controller
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.337467   12121 controller_utils.go:1027] Waiting for caches to sync for certificate controller
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.592441   12121 controllermanager.go:516] Started "clusterrole-aggregation"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.592471   12121 controllermanager.go:501] Starting "pvc-protection"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.592513   12121 clusterroleaggregation_controller.go:148] Starting ClusterRoleAggregator
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.592523   12121 controller_utils.go:1027] Waiting for caches to sync for ClusterRoleAggregator controller
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.841180   12121 controllermanager.go:516] Started "pvc-protection"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.841227   12121 controllermanager.go:501] Starting "resourcequota"
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.841271   12121 pvc_protection_controller.go:99] Starting PVC protection controller
Aug 28 16:17:14 ctl01 kube-controller-manager[12121]: I0828 16:17:14.841282   12121 controller_utils.go:1027] Waiting for caches to sync for PVC protection controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103263   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103343   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103388   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for statefulsets.apps
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103428   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for events.events.k8s.io
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103476   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.extensions
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103510   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.apps
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103549   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103606   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpoints
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103673   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.apps
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103710   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for jobs.batch
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103801   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for serviceaccounts
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103845   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for podtemplates
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103891   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.103966   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104034   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.extensions
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104070   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.apps
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104109   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104223   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for controllerrevisions.apps
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104263   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for cronjobs.batch
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104315   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for limitranges
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104388   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.extensions
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104475   12121 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.extensions
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: E0828 16:17:15.104498   12121 resource_quota_controller.go:171] initial monitor sync has error: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104518   12121 controllermanager.go:516] Started "resourcequota"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104529   12121 controllermanager.go:501] Starting "root-ca-cert-publisher"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.104538   12121 controllermanager.go:508] Skipping "root-ca-cert-publisher"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104544   12121 controllermanager.go:501] Starting "deployment"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104625   12121 resource_quota_controller.go:276] Starting resource quota controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104637   12121 controller_utils.go:1027] Waiting for caches to sync for resource quota controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.104652   12121 resource_quota_monitor.go:301] QuotaMonitor running
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.343830   12121 controllermanager.go:516] Started "deployment"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.343854   12121 controllermanager.go:501] Starting "statefulset"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.343895   12121 deployment_controller.go:152] Starting deployment controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.343906   12121 controller_utils.go:1027] Waiting for caches to sync for deployment controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.591766   12121 controllermanager.go:516] Started "statefulset"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.592296   12121 controllermanager.go:501] Starting "csrsigning"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.592236   12121 stateful_set.go:151] Starting stateful set controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.593014   12121 controller_utils.go:1027] Waiting for caches to sync for stateful set controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.593309   12121 controllermanager.go:508] Skipping "csrsigning"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.593581   12121 controllermanager.go:501] Starting "daemonset"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.839550   12121 controllermanager.go:516] Started "daemonset"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.839720   12121 daemon_controller.go:269] Starting daemon sets controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.840204   12121 controller_utils.go:1027] Waiting for caches to sync for daemon sets controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.860449   12121 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="ctl01" does not exist
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.862208   12121 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="cmp002" does not exist
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.862361   12121 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="cmp001" does not exist
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.862717   12121 controller_utils.go:1027] Waiting for caches to sync for garbage collector controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.866753   12121 controller_utils.go:1034] Caches are synced for expand controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.875283   12121 controller_utils.go:1034] Caches are synced for taint controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.875459   12121 node_lifecycle_controller.go:665] Controller observed a new Node: "cmp002"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.875599   12121 controller_utils.go:212] Recording Registered Node cmp002 in Controller event message for node cmp002
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.875753   12121 node_lifecycle_controller.go:1222] Initializing eviction metric for zone:
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.875928   12121 node_lifecycle_controller.go:665] Controller observed a new Node: "cmp001"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.876050   12121 controller_utils.go:212] Recording Registered Node cmp001 in Controller event message for node cmp001
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.876190   12121 node_lifecycle_controller.go:665] Controller observed a new Node: "ctl01"
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.876334   12121 controller_utils.go:212] Recording Registered Node ctl01 in Controller event message for node ctl01
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.876390   12121 taint_manager.go:198] Starting NoExecuteTaintManager
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.876672   12121 event.go:221] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"cmp002", UID:"472c5f85-c9af-11e9-b58a-5254003f9b0c", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node cmp002 event: Registered Node cmp002 in Controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.878191   12121 event.go:221] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"cmp001", UID:"49d44939-c9af-11e9-b58a-5254003f9b0c", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node cmp001 event: Registered Node cmp001 in Controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.878349   12121 event.go:221] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ctl01", UID:"472bf958-c9af-11e9-b58a-5254003f9b0c", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ctl01 event: Registered Node ctl01 in Controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.878503   12121 node_lifecycle_controller.go:895] Missing timestamp for Node cmp002. Assuming now as a timestamp.
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.879164   12121 node_lifecycle_controller.go:895] Missing timestamp for Node cmp001. Assuming now as a timestamp.
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: W0828 16:17:15.879333   12121 node_lifecycle_controller.go:895] Missing timestamp for Node ctl01. Assuming now as a timestamp.
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.879659   12121 node_lifecycle_controller.go:1122] Controller detected that zone  is now in state Normal.
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.885501   12121 controller_utils.go:1034] Caches are synced for HPA controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.889378   12121 controller_utils.go:1034] Caches are synced for ReplicationController controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.889511   12121 controller_utils.go:1034] Caches are synced for attach detach controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.891831   12121 controller_utils.go:1034] Caches are synced for PV protection controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.892074   12121 controller_utils.go:1034] Caches are synced for job controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.893155   12121 controller_utils.go:1034] Caches are synced for ClusterRoleAggregator controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.893361   12121 controller_utils.go:1034] Caches are synced for stateful set controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.894630   12121 controller_utils.go:1034] Caches are synced for disruption controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.894784   12121 disruption.go:296] Sending events to api server.
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.901669   12121 controller_utils.go:1034] Caches are synced for namespace controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.913688   12121 controller_utils.go:1034] Caches are synced for GC controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.926657   12121 controller_utils.go:1034] Caches are synced for persistent volume controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.937640   12121 controller_utils.go:1034] Caches are synced for certificate controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.940341   12121 controller_utils.go:1034] Caches are synced for daemon sets controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.942052   12121 controller_utils.go:1034] Caches are synced for TTL controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.942234   12121 controller_utils.go:1034] Caches are synced for endpoint controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.942245   12121 controller_utils.go:1034] Caches are synced for service account controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.943017   12121 controller_utils.go:1034] Caches are synced for PVC protection controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.944180   12121 controller_utils.go:1034] Caches are synced for deployment controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.945602   12121 controller_utils.go:1034] Caches are synced for ReplicaSet controller
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.946051   12121 ttl_controller.go:271] Changed ttl annotation for node ctl01 to 0 seconds
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.948253   12121 ttl_controller.go:271] Changed ttl annotation for node cmp002 to 0 seconds
Aug 28 16:17:15 ctl01 kube-controller-manager[12121]: I0828 16:17:15.948820   12121 ttl_controller.go:271] Changed ttl annotation for node cmp001 to 0 seconds
Aug 28 16:17:16 ctl01 kube-controller-manager[12121]: I0828 16:17:16.104808   12121 controller_utils.go:1034] Caches are synced for resource quota controller
Aug 28 16:17:16 ctl01 kubelet[11491]: I0828 16:17:16.219803   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:17:16 ctl01 kube-controller-manager[12121]: I0828 16:17:16.431487   12121 controller_utils.go:1034] Caches are synced for garbage collector controller
Aug 28 16:17:16 ctl01 kube-controller-manager[12121]: I0828 16:17:16.431513   12121 garbagecollector.go:142] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
Aug 28 16:17:16 ctl01 kube-controller-manager[12121]: I0828 16:17:16.464652   12121 controller_utils.go:1034] Caches are synced for garbage collector controller
Aug 28 16:17:16 ctl01 kube-controller-manager[12121]: I0828 16:17:16.464779   12121 garbagecollector.go:245] synced garbage collector
Aug 28 16:17:16 ctl01 kube-controller-manager[12121]: I0828 16:17:16.596680   12121 resource_quota_controller.go:427] syncing resource quota controller with updated resources from discovery: map[apps/v1, Resource=controllerrevisions:{} networking.k8s.io/v1, Resource=networkpolicies:{} /v1, Resource=configmaps:{} /v1, Resource=persistentvolumeclaims:{} extensions/v1beta1, Resource=daemonsets:{} extensions/v1beta1, Resource=deployments:{} extensions/v1beta1, Resource=replicasets:{} apps/v1, Resource=replicasets:{} batch/v1beta1, Resource=cronjobs:{} rbac.authorization.k8s.io/v1, Resource=roles:{} /v1, Resource=pods:{} /v1, Resource=resourcequotas:{} /v1, Resource=services:{} coordination.k8s.io/v1beta1, Resource=leases:{} /v1, Resource=endpoints:{} extensions/v1beta1, Resource=networkpolicies:{} rbac.authorization.k8s.io/v1, Resource=rolebindings:{} apps/v1, Resource=deployments:{} batch/v1, Resource=jobs:{} /v1, Resource=limitranges:{} /v1, Resource=podtemplates:{} /v1, Resource=serviceaccounts:{} apps/v1, Resource=statefulsets:{} events.k8s.io/v1beta1, Resource=events:{} /v1, Resource=replicationcontrollers:{} extensions/v1beta1, Resource=ingresses:{} apps/v1, Resource=daemonsets:{} policy/v1beta1, Resource=poddisruptionbudgets:{} /v1, Resource=events:{} /v1, Resource=secrets:{} autoscaling/v1, Resource=horizontalpodautoscalers:{}]
Aug 28 16:17:16 ctl01 kube-controller-manager[12121]: E0828 16:17:16.597418   12121 resource_quota_controller.go:437] failed to sync resource monitors: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies"
Aug 28 16:17:17 ctl01 salt-minion[4490]: [INFO    ] Loading fresh modules for state activity
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/kubeconfig.sh] at time 16:17:18.629683
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/kubeconfig.sh]
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kubeconfig.sh'
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:18 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:17:19 ctl01 salt-minion[4490]: New file
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/kubeconfig.sh] at time 16:17:19.070610 duration_in_ms=440.926
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config] at time 16:17:19.073100
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing state cmd.run for [/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config]
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing command '/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config' in directory '/root'
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] {'pid': 12682, 'retcode': 0, 'stderr': '', 'stdout': ''}
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/kubeconfig.sh > /etc/kubernetes/admin-kube-config] at time 16:17:19.126612 duration_in_ms=53.511
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/addons/namespace.yaml] at time 16:17:19.127271
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/addons/namespace.yaml]
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/kube-addon-manager/namespace.yaml'
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:17:19 ctl01 salt-minion[4490]: New file
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/addons/namespace.yaml] at time 16:17:19.157881 duration_in_ms=30.611
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/kubernetes/manifests/kube-addon-manager.yml] at time 16:17:19.158245
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/etc/kubernetes/manifests/kube-addon-manager.yml]
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/manifest/kube-addon-manager.yml'
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:17:19 ctl01 salt-minion[4490]: New file
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/kubernetes/manifests/kube-addon-manager.yml] at time 16:17:19.580111 duration_in_ms=421.865
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/default/kube-addon-manager] at time 16:17:19.580623
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/default/kube-addon-manager]
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] File /etc/default/kube-addon-manager is not present
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/default/kube-addon-manager] at time 16:17:19.582016 duration_in_ms=1.393
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Running state [/usr/bin/kube-addons.sh] at time 16:17:19.582383
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/usr/bin/kube-addons.sh]
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] File /usr/bin/kube-addons.sh is not present
Aug 28 16:17:19 ctl01 salt-minion[4490]: [INFO    ] Completed state [/usr/bin/kube-addons.sh] at time 16:17:19.583622 duration_in_ms=1.24
Aug 28 16:17:19 ctl01 kubelet[11491]: E0828 16:17:19.592964   11491 file.go:108] Unable to process watch event: can't process config file "/etc/kubernetes/manifests/kube-addon-manager.ymllMaT1T": open /etc/kubernetes/manifests/kube-addon-manager.ymllMaT1T: no such file or directory
Aug 28 16:17:19 ctl01 kubelet[11491]: E0828 16:17:19.593000   11491 file.go:108] Unable to process watch event: can't process config file "/etc/kubernetes/manifests/kube-addon-manager.ymllMaT1T": open /etc/kubernetes/manifests/kube-addon-manager.ymllMaT1T: no such file or directory
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.593494   11491 kubelet.go:1908] SyncLoop (ADD, "file"): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.595482   11491 kubelet.go:1918] SyncLoop (REMOVE, "file"): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.595523   11491 kubelet.go:2120] Failed to delete pod "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)", err: pod not found
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.595551   11491 kubelet.go:1908] SyncLoop (ADD, "file"): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.606781   11491 kubelet.go:1908] SyncLoop (ADD, "api"): "kube-addon-manager-ctl01_kube-system(4f28d62d-c9af-11e9-b58a-5254003f9b0c)"
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.714388   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "varlog" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-varlog") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.714446   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-addons") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.814681   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "varlog" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-varlog") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.814730   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "addons" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-addons") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.814801   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "addons" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-addons") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.814919   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "varlog" (UniqueName: "kubernetes.io/host-path/b6e9bf37167122649376fb470187a295-varlog") pod "kube-addon-manager-ctl01" (UID: "b6e9bf37167122649376fb470187a295")
Aug 28 16:17:19 ctl01 kubelet[11491]: I0828 16:17:19.906959   11491 kuberuntime_manager.go:397] No sandbox for pod "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)" can be found. Need to start a new one
Aug 28 16:17:19 ctl01 containerd[6590]: time="2019-08-28T16:17:19.907765103Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-addon-manager-ctl01,Uid:b6e9bf37167122649376fb470187a295,Namespace:kube-system,Attempt:0,}"
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Running state [kube-addon-manager] at time 16:17:20.011490
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing state service.dead for [kube-addon-manager]
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing command ['systemctl', 'status', 'kube-addon-manager.service', '-n', '0'] in directory '/root'
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] The named service kube-addon-manager is not available
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Completed state [kube-addon-manager] at time 16:17:20.046826 duration_in_ms=35.336
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Running state [/etc/systemd/system/kube-addon-manager.service] at time 16:17:20.047333
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing state file.absent for [/etc/systemd/system/kube-addon-manager.service]
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] File /etc/systemd/system/kube-addon-manager.service is not present
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Completed state [/etc/systemd/system/kube-addon-manager.service] at time 16:17:20.048709 duration_in_ms=1.376
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Running state [/srv/kubernetes/conformance.yml] at time 16:17:20.049039
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing state file.managed for [/srv/kubernetes/conformance.yml]
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Fetching file from saltenv 'base', ** done ** 'kubernetes/files/conformance/conformance.yml'
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/-.*//g' -e 's/v//g' -e 's/Kubernetes //g' | awk -F'.' '{print $1 "." $2}'' in directory '/root'
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Executing command '(hyperkube --version kubelet 2> /dev/null || echo '0.0') | sed -e 's/+.*//g' -e 's/v//g' -e 's/Kubernetes //g'' in directory '/root'
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] File changed:
Aug 28 16:17:20 ctl01 salt-minion[4490]: New file
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Completed state [/srv/kubernetes/conformance.yml] at time 16:17:20.726070 duration_in_ms=677.031
Aug 28 16:17:20 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161711578885
Aug 28 16:17:20 ctl01 containerd[6590]: time="2019-08-28T16:17:20.813258748Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3,Labels:map[string]string{},}"
Aug 28 16:17:20 ctl01 containerd[6590]: time="2019-08-28T16:17:20.826097893Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:478b5c5586708a45bfb7aab46b7a298c4eb65c3442311e7868f49ba269f358a4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:20 ctl01 containerd[6590]: time="2019-08-28T16:17:20.827047427Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:20 ctl01 containerd[6590]: time="2019-08-28T16:17:20.880667086Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:478b5c5586708a45bfb7aab46b7a298c4eb65c3442311e7868f49ba269f358a4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:20 ctl01 containerd[6590]: time="2019-08-28T16:17:20.881944914Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64:v1.13.5-3,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:20 ctl01 containerd[6590]: time="2019-08-28T16:17:20.883057853Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/kubernetes/pause-amd64@sha256:9e2b9d9d64bab9f4d80790ec6c6fe09cdb5714d43bf23357e0ed0d0ab512fffd,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:20 ctl01 containerd[6590]: time="2019-08-28T16:17:20.892110070Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6/shim.sock" debug=false pid=12804
Aug 28 16:17:21 ctl01 containerd[6590]: time="2019-08-28T16:17:21.268113103Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-addon-manager-ctl01,Uid:b6e9bf37167122649376fb470187a295,Namespace:kube-system,Attempt:0,} returns sandbox id "0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6""
Aug 28 16:17:21 ctl01 kubelet[11491]: I0828 16:17:21.269780   11491 provider.go:116] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 28 16:17:21 ctl01 containerd[6590]: time="2019-08-28T16:17:21.270042288Z" level=info msg="PullImage "k8s.gcr.io/kube-addon-manager:v8.9""
Aug 28 16:17:21 ctl01 kubelet[11491]: I0828 16:17:21.275666   11491 kubelet.go:1953] SyncLoop (PLEG): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)", event: &pleg.PodLifecycleEvent{ID:"b6e9bf37167122649376fb470187a295", Type:"ContainerStarted", Data:"0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6"}
Aug 28 16:17:21 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command cp.push with jid 20190828161721272209
Aug 28 16:17:21 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 12863
Aug 28 16:17:21 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161721272209
Aug 28 16:17:22 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command cp.push_dir with jid 20190828161722060706
Aug 28 16:17:22 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 12870
Aug 28 16:17:22 ctl01 containerd[6590]: time="2019-08-28T16:17:22.827801915Z" level=info msg="ImageCreate event &ImageCreate{Name:k8s.gcr.io/kube-addon-manager:v8.9,Labels:map[string]string{},}"
Aug 28 16:17:22 ctl01 containerd[6590]: time="2019-08-28T16:17:22.838923595Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:13da3ecfc10d338d91f3fecf494fac162c704fc284006ec289d660ec7cbba3e4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:22 ctl01 containerd[6590]: time="2019-08-28T16:17:22.839446719Z" level=info msg="ImageUpdate event &ImageUpdate{Name:k8s.gcr.io/kube-addon-manager:v8.9,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:22 ctl01 salt-minion[4490]: [INFO    ] Returning information for job: 20190828161722060706
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.693840730Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:13da3ecfc10d338d91f3fecf494fac162c704fc284006ec289d660ec7cbba3e4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.695876168Z" level=info msg="ImageUpdate event &ImageUpdate{Name:k8s.gcr.io/kube-addon-manager:v8.9,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.697848073Z" level=info msg="ImageCreate event &ImageCreate{Name:k8s.gcr.io/kube-addon-manager@sha256:2fd1daf3d3cf0e94a753f2263b60dbb0d42b107b5cde0c75ee3fc5c830e016e4,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.698314281Z" level=info msg="PullImage "k8s.gcr.io/kube-addon-manager:v8.9" returns image reference "sha256:13da3ecfc10d338d91f3fecf494fac162c704fc284006ec289d660ec7cbba3e4""
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.701861674Z" level=info msg="CreateContainer within sandbox "0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6" for container &ContainerMetadata{Name:kube-addon-manager,Attempt:0,}"
Aug 28 16:17:25 ctl01 kernel: [  335.597909] audit: type=1400 audit(1567009045.846:18): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cri-containerd.apparmor.d" pid=12947 comm="apparmor_parser"
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.869547697Z" level=info msg="CreateContainer within sandbox "0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6" for &ContainerMetadata{Name:kube-addon-manager,Attempt:0,} returns container id "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345""
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.870767415Z" level=info msg="StartContainer for "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345""
Aug 28 16:17:25 ctl01 containerd[6590]: time="2019-08-28T16:17:25.872086168Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345/shim.sock" debug=false pid=12948
Aug 28 16:17:26 ctl01 kubelet[11491]: I0828 16:17:26.227613   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:17:26 ctl01 containerd[6590]: time="2019-08-28T16:17:26.264804244Z" level=info msg="Finish piping stderr of container "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345""
Aug 28 16:17:26 ctl01 containerd[6590]: time="2019-08-28T16:17:26.264932102Z" level=info msg="Finish piping stdout of container "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345""
Aug 28 16:17:26 ctl01 containerd[6590]: time="2019-08-28T16:17:26.288229103Z" level=info msg="StartContainer for "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345" returns successfully"
Aug 28 16:17:26 ctl01 containerd[6590]: time="2019-08-28T16:17:26.290874291Z" level=info msg="CreateContainer within sandbox "0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6" for container &ContainerMetadata{Name:kube-addon-manager,Attempt:0,}"
Aug 28 16:17:26 ctl01 kubelet[11491]: I0828 16:17:26.293562   11491 kubelet.go:1953] SyncLoop (PLEG): "kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)", event: &pleg.PodLifecycleEvent{ID:"b6e9bf37167122649376fb470187a295", Type:"ContainerStarted", Data:"2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345"}
Aug 28 16:17:26 ctl01 containerd[6590]: time="2019-08-28T16:17:26.300729470Z" level=error msg="CreateContainer within sandbox "0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6" for &ContainerMetadata{Name:kube-addon-manager,Attempt:0,} failed" error="failed to reserve container name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0": name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0" is reserved for "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345""
Aug 28 16:17:26 ctl01 kubelet[11491]: E0828 16:17:26.301004   11491 remote_runtime.go:191] CreateContainer in sandbox "0385e2ebbbe74b2c4220d778c59f39527fd69316bd35c875d98d2d414babb3e6" from runtime service failed: rpc error: code = Unknown desc = failed to reserve container name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0": name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0" is reserved for "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345"
Aug 28 16:17:26 ctl01 kubelet[11491]: E0828 16:17:26.301070   11491 kuberuntime_manager.go:749] container start failed: CreateContainerError: failed to reserve container name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0": name "kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0" is reserved for "2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345"
Aug 28 16:17:26 ctl01 kubelet[11491]: E0828 16:17:26.301186   11491 pod_workers.go:190] Error syncing pod b6e9bf37167122649376fb470187a295 ("kube-addon-manager-ctl01_kube-system(b6e9bf37167122649376fb470187a295)"), skipping: failed to "StartContainer" for "kube-addon-manager" with CreateContainerError: "failed to reserve container name \"kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0\": name \"kube-addon-manager_kube-addon-manager-ctl01_kube-system_b6e9bf37167122649376fb470187a295_0\" is reserved for \"2c020a9da83637bd924a892321e280223b7f567e4367df57e3359f7616809345\""
Aug 28 16:17:26 ctl01 kube-apiserver[11892]: I0828 16:17:26.970732   11892 controller.go:608] quota admission added evaluator for: namespaces
Aug 28 16:17:27 ctl01 kube-apiserver[11892]: I0828 16:17:27.861499   11892 controller.go:608] quota admission added evaluator for: deployments.extensions
Aug 28 16:17:27 ctl01 kube-apiserver[11892]: I0828 16:17:27.870991   11892 controller.go:608] quota admission added evaluator for: replicasets.apps
Aug 28 16:17:27 ctl01 kube-controller-manager[12121]: I0828 16:17:27.872601   12121 replica_set.go:477] Too few replicas for ReplicaSet kube-system/calico-kube-controllers-996f9b774, need 1, creating 1
Aug 28 16:17:27 ctl01 kube-controller-manager[12121]: I0828 16:17:27.874023   12121 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"calico-kube-controllers", UID:"5414af85-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"374", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set calico-kube-controllers-996f9b774 to 1
Aug 28 16:17:27 ctl01 kube-controller-manager[12121]: I0828 16:17:27.878144   12121 deployment_controller.go:484] Error syncing deployment kube-system/calico-kube-controllers: Operation cannot be fulfilled on deployments.apps "calico-kube-controllers": the object has been modified; please apply your changes to the latest version and try again
Aug 28 16:17:27 ctl01 kube-controller-manager[12121]: I0828 16:17:27.938528   12121 replica_set.go:477] Too few replicas for ReplicaSet kube-system/coredns-7f8f94c97b, need 2, creating 2
Aug 28 16:17:27 ctl01 kube-controller-manager[12121]: I0828 16:17:27.939625   12121 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"541f5449-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"385", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-7f8f94c97b to 2
Aug 28 16:17:27 ctl01 kube-controller-manager[12121]: I0828 16:17:27.949577   12121 deployment_controller.go:484] Error syncing deployment kube-system/coredns: Operation cannot be fulfilled on deployments.apps "coredns": the object has been modified; please apply your changes to the latest version and try again
Aug 28 16:17:27 ctl01 kube-controller-manager[12121]: I0828 16:17:27.961912   12121 deployment_controller.go:484] Error syncing deployment kube-system/coredns: Operation cannot be fulfilled on deployments.apps "coredns": the object has been modified; please apply your changes to the latest version and try again
Aug 28 16:17:27 ctl01 kube-proxy[9844]: I0828 16:17:27.974417    9844 service.go:309] Adding new service port "kube-system/coredns:dns" at 10.254.0.10:53/UDP
Aug 28 16:17:27 ctl01 kube-proxy[9844]: I0828 16:17:27.974472    9844 service.go:309] Adding new service port "kube-system/coredns:dns-tcp" at 10.254.0.10:53/TCP
Aug 28 16:17:27 ctl01 kube-apiserver[11892]: I0828 16:17:27.989401   11892 controller.go:608] quota admission added evaluator for: daemonsets.extensions
Aug 28 16:17:28 ctl01 kube-apiserver[11892]: I0828 16:17:28.004017   11892 controller.go:608] quota admission added evaluator for: controllerrevisions.apps
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.012272   12121 replica_set.go:477] Too few replicas for ReplicaSet netchecker/netchecker-server-7876fb46d4, need 1, creating 1
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.012913   12121 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"netchecker", Name:"netchecker-server", UID:"542ae43a-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"404", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set netchecker-server-7876fb46d4 to 1
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.016353   12121 event.go:221] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"netchecker", Name:"netchecker-agent", UID:"54283355-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"399", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-agent-ggnhh
Aug 28 16:17:28 ctl01 kubelet[11491]: I0828 16:17:28.017923   11491 kubelet.go:1908] SyncLoop (ADD, "api"): "netchecker-agent-ggnhh_netchecker(542be8fa-c9af-11e9-b58a-5254003f9b0c)"
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.037400   12121 deployment_controller.go:484] Error syncing deployment netchecker/netchecker-server: Operation cannot be fulfilled on deployments.apps "netchecker-server": the object has been modified; please apply your changes to the latest version and try again
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.037507   12121 event.go:221] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"netchecker", Name:"netchecker-agent", UID:"54283355-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"399", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-agent-pk6cn
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.037699   12121 event.go:221] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"netchecker", Name:"netchecker-agent", UID:"54283355-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"399", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-agent-42t6m
Aug 28 16:17:28 ctl01 kube-proxy[9844]: I0828 16:17:28.074686    9844 service.go:309] Adding new service port "netchecker/netchecker:" at 10.254.56.89:80/TCP
Aug 28 16:17:28 ctl01 kube-proxy[9844]: I0828 16:17:28.092607    9844 proxier.go:1427] Opened local port "nodePort for netchecker/netchecker:" (:32653/tcp)
Aug 28 16:17:28 ctl01 kubelet[11491]: I0828 16:17:28.169339   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-4b6sg" (UniqueName: "kubernetes.io/secret/542be8fa-c9af-11e9-b58a-5254003f9b0c-default-token-4b6sg") pod "netchecker-agent-ggnhh" (UID: "542be8fa-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:28 ctl01 kubelet[11491]: I0828 16:17:28.269945   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "default-token-4b6sg" (UniqueName: "kubernetes.io/secret/542be8fa-c9af-11e9-b58a-5254003f9b0c-default-token-4b6sg") pod "netchecker-agent-ggnhh" (UID: "542be8fa-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:28 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/542be8fa-c9af-11e9-b58a-5254003f9b0c/volumes/kubernetes.io~secret/default-token-4b6sg.
Aug 28 16:17:28 ctl01 kubelet[11491]: I0828 16:17:28.286484   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "default-token-4b6sg" (UniqueName: "kubernetes.io/secret/542be8fa-c9af-11e9-b58a-5254003f9b0c-default-token-4b6sg") pod "netchecker-agent-ggnhh" (UID: "542be8fa-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:28 ctl01 kubelet[11491]: I0828 16:17:28.332341   11491 kuberuntime_manager.go:397] No sandbox for pod "netchecker-agent-ggnhh_netchecker(542be8fa-c9af-11e9-b58a-5254003f9b0c)" can be found. Need to start a new one
Aug 28 16:17:28 ctl01 containerd[6590]: time="2019-08-28T16:17:28.333214542Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:netchecker-agent-ggnhh,Uid:542be8fa-c9af-11e9-b58a-5254003f9b0c,Namespace:netchecker,Attempt:0,}"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.639 [INFO][13200] calico.go 75: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"netchecker", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"ctl01", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"netchecker-agent-ggnhh", ContainerID:"7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af"}}
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.703 [INFO][13200] k8s.go 60: Extracted identifiers for CmdAddK8s ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Namespace="netchecker" Pod="netchecker-agent-ggnhh" WorkloadEndpoint="ctl01-k8s-netchecker--agent--ggnhh-eth0"
Aug 28 16:17:28 ctl01 containerd[6590]: Calico CNI IPAM request count IPv4=1 IPv6=0
Aug 28 16:17:28 ctl01 containerd[6590]: Calico CNI IPAM handle=calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.787 [INFO][13215] calico-ipam.go 186: Auto assigning IP ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" HandleID="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Workload="ctl01-k8s-netchecker--agent--ggnhh-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0xc4202ef270), Attrs:map[string]string(nil), Hostname:"ctl01", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}}
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.788 [INFO][13215] ipam.go 70: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'ctl01'
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.789 [INFO][13215] ipam.go 254: Looking up existing affinities for host handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.790 [INFO][13215] ipam.go 265: Ran out of existing affine blocks for host handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.791 [INFO][13215] ipam.go 324: No more affine blocks, but need to allocate 1 more addresses - allocate another block handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.791 [INFO][13215] ipam.go 328: Looking for an unclaimed block handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.792 [INFO][13215] ipam_block_reader_writer.go 106: Found free block: 192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.792 [INFO][13215] ipam.go 340: Found unclaimed block host="ctl01" subnet=192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.792 [INFO][13215] ipam_block_reader_writer.go 122: Trying to create affinity in pending state host="ctl01" subnet=192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.795 [INFO][13215] ipam_block_reader_writer.go 152: Successfully created pending affinity for block host="ctl01" subnet=192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.795 [INFO][13215] ipam.go 118: Attempting to load block cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.795 [INFO][13215] ipam.go 123: The referenced block doesn't exist, trying to create it cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.798 [INFO][13215] ipam.go 130: Wrote affinity as pending cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.799 [INFO][13215] ipam.go 139: Attempting to claim the block cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.799 [INFO][13215] ipam_block_reader_writer.go 175: Attempting to create a new block host="ctl01" subnet=192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.800 [INFO][13215] ipam_block_reader_writer.go 217: Successfully created block
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.800 [INFO][13215] ipam_block_reader_writer.go 228: Confirming affinity host="ctl01" subnet=192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.801 [INFO][13215] ipam_block_reader_writer.go 243: Successfully confirmed affinity host="ctl01" subnet=192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.801 [INFO][13215] ipam.go 372: Claimed new block &{BlockKey(cidr=192.168.237.0/26) 0xc4200867e0 437 0s} - assigning 1 addresses host="ctl01" subnet=192.168.237.0/26
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.801 [INFO][13215] ipam.go 677: Attempting to assign 1 addresses from block block=192.168.237.0/26 handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.802 [INFO][13215] ipam.go 1110: Creating new handle: calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.803 [INFO][13215] ipam.go 700: Writing block in order to claim IPs block=192.168.237.0/26 handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.804 [INFO][13215] ipam.go 710: Successfully claimed IPs: [192.168.237.0] block=192.168.237.0/26 handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.804 [INFO][13215] ipam.go 456: Auto-assigned 1 out of 1 IPv4s: [192.168.237.0] handle="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" host="ctl01"
Aug 28 16:17:28 ctl01 containerd[6590]: Calico CNI IPAM assigned addresses IPv4=[192.168.237.0] IPv6=[]
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.804 [INFO][13215] calico-ipam.go 214: IPAM Result ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" HandleID="calico-k8s-network.7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Workload="ctl01-k8s-netchecker--agent--ggnhh-eth0" result.IPs=[]*current.IPConfig{(*current.IPConfig)(0xc42011c540)}
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.807 [INFO][13200] k8s.go 365: Populated endpoint ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Namespace="netchecker" Pod="netchecker-agent-ggnhh" WorkloadEndpoint="ctl01-k8s-netchecker--agent--ggnhh-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--agent--ggnhh-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"", Pod:"netchecker-agent-ggnhh", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.0/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
Aug 28 16:17:28 ctl01 containerd[6590]: Calico CNI using IPs: [192.168.237.0/32]
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.807 [INFO][13200] network.go 75: Setting the host side veth name to cali8d00ca21f5c ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Namespace="netchecker" Pod="netchecker-agent-ggnhh" WorkloadEndpoint="ctl01-k8s-netchecker--agent--ggnhh-eth0"
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.814 [INFO][13200] network.go 380: Disabling IPv4 forwarding ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Namespace="netchecker" Pod="netchecker-agent-ggnhh" WorkloadEndpoint="ctl01-k8s-netchecker--agent--ggnhh-eth0"
Aug 28 16:17:28 ctl01 kernel: [  338.561695] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 28 16:17:28 ctl01 kernel: [  338.561984] IPv6: ADDRCONF(NETDEV_UP): cali8d00ca21f5c: link is not ready
Aug 28 16:17:28 ctl01 kernel: [  338.561999] IPv6: ADDRCONF(NETDEV_CHANGE): cali8d00ca21f5c: link becomes ready
Aug 28 16:17:28 ctl01 kernel: [  338.562040] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 28 16:17:28 ctl01 systemd-udevd[13235]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.844 [INFO][13200] k8s.go 392: Added Mac, interface name, and active container ID to endpoint ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Namespace="netchecker" Pod="netchecker-agent-ggnhh" WorkloadEndpoint="ctl01-k8s-netchecker--agent--ggnhh-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--agent--ggnhh-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af", Pod:"netchecker-agent-ggnhh", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.0/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"cali8d00ca21f5c", MAC:"62:86:da:fd:c8:37", Ports:[]v3.EndpointPort(nil)}}
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.848 [INFO][13200] k8s.go 424: Wrote updated endpoint to datastore ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Namespace="netchecker" Pod="netchecker-agent-ggnhh" WorkloadEndpoint="ctl01-k8s-netchecker--agent--ggnhh-eth0"
Aug 28 16:17:28 ctl01 containerd[6590]: Calico CNI creating profile: calico-k8s-network
Aug 28 16:17:28 ctl01 containerd[6590]: 2019-08-28 16:17:28.850 [INFO][13200] calico.go 366: Creating profile ContainerID="7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" Namespace="netchecker" Pod="netchecker-agent-ggnhh" WorkloadEndpoint="ctl01-k8s-netchecker--agent--ggnhh-" profile=&v3.Profile{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"calico-k8s-network", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.ProfileSpec{Ingress:[]v3.Rule{v3.Rule{Action:"Allow", IPVersion:(*int)(nil), Protocol:(*numorstring.Protocol)(nil), ICMP:(*v3.ICMPFields)(nil), NotProtocol:(*numorstring.Protocol)(nil), NotICMP:(*v3.ICMPFields)(nil), Source:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Ports:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, Destination:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Ports:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, HTTP:(*v3.HTTPMatch)(nil)}}, Egress:[]v3.Rule{v3.Rule{Action:"Allow", IPVersion:(*int)(nil), Protocol:(*numorstring.Protocol)(nil), ICMP:(*v3.ICMPFields)(nil), NotProtocol:(*numorstring.Protocol)(nil), NotICMP:(*v3.ICMPFields)(nil), Source:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Ports:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, Destination:v3.EntityRule{Nets:[]string(nil), Selector:"", NamespaceSelector:"", Ports:[]numorstring.Port(nil), NotNets:[]string(nil), NotSelector:"", NotPorts:[]numorstring.Port(nil), ServiceAccounts:(*v3.ServiceAccountMatch)(nil)}, HTTP:(*v3.HTTPMatch)(nil)}}, LabelsToApply:map[string]string{"calico-k8s-network":""}}}
Aug 28 16:17:28 ctl01 containerd[6590]: time="2019-08-28T16:17:28.877145555Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af/shim.sock" debug=false pid=13303
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.885769   12121 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"calico-kube-controllers-996f9b774", UID:"54162303-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"375", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: calico-kube-controllers-996f9b774-j7lpq
Aug 28 16:17:28 ctl01 kubelet[11491]: I0828 16:17:28.888294   11491 kubelet.go:1908] SyncLoop (ADD, "api"): "calico-kube-controllers-996f9b774-j7lpq_kube-system(54b09258-c9af-11e9-b58a-5254003f9b0c)"
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.949909   12121 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-7f8f94c97b", UID:"541fc3ea-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"386", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-7f8f94c97b-lwlss
Aug 28 16:17:28 ctl01 kube-controller-manager[12121]: I0828 16:17:28.956895   12121 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-7f8f94c97b", UID:"541fc3ea-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"386", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-7f8f94c97b-6cm5z
Aug 28 16:17:28 ctl01 kubelet[11491]: I0828 16:17:28.958339   11491 kubelet.go:1908] SyncLoop (ADD, "api"): "coredns-7f8f94c97b-6cm5z_kube-system(54bb05f5-c9af-11e9-b58a-5254003f9b0c)"
Aug 28 16:17:29 ctl01 kube-controller-manager[12121]: I0828 16:17:29.025490   12121 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"netchecker", Name:"netchecker-server-7876fb46d4", UID:"542b7b00-c9af-11e9-b58a-5254003f9b0c", APIVersion:"apps/v1", ResourceVersion:"405", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: netchecker-server-7876fb46d4-lq5jp
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.056058463Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:netchecker-agent-ggnhh,Uid:542be8fa-c9af-11e9-b58a-5254003f9b0c,Namespace:netchecker,Attempt:0,} returns sandbox id "7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af""
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.058420357Z" level=info msg="PullImage "mirantis/k8s-netchecker-agent:stable""
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.072069   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/54b09258-c9af-11e9-b58a-5254003f9b0c-etcd-certs") pod "calico-kube-controllers-996f9b774-j7lpq" (UID: "54b09258-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.072143   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-kube-controllers-token-8vmnr" (UniqueName: "kubernetes.io/secret/54b09258-c9af-11e9-b58a-5254003f9b0c-calico-kube-controllers-token-8vmnr") pod "calico-kube-controllers-996f9b774-j7lpq" (UID: "54b09258-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.072234   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/54bb05f5-c9af-11e9-b58a-5254003f9b0c-config-volume") pod "coredns-7f8f94c97b-6cm5z" (UID: "54bb05f5-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.072315   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-sl4w9" (UniqueName: "kubernetes.io/secret/54bb05f5-c9af-11e9-b58a-5254003f9b0c-coredns-token-sl4w9") pod "coredns-7f8f94c97b-6cm5z" (UID: "54bb05f5-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.105630   11491 kubelet.go:1908] SyncLoop (ADD, "api"): "netchecker-server-7876fb46d4-lq5jp_netchecker(54c5749e-c9af-11e9-b58a-5254003f9b0c)"
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.172600   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/54bb05f5-c9af-11e9-b58a-5254003f9b0c-config-volume") pod "coredns-7f8f94c97b-6cm5z" (UID: "54bb05f5-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.172886   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "coredns-token-sl4w9" (UniqueName: "kubernetes.io/secret/54bb05f5-c9af-11e9-b58a-5254003f9b0c-coredns-token-sl4w9") pod "coredns-7f8f94c97b-6cm5z" (UID: "54bb05f5-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.173496   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/54b09258-c9af-11e9-b58a-5254003f9b0c-etcd-certs") pod "calico-kube-controllers-996f9b774-j7lpq" (UID: "54b09258-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.173737   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/54b09258-c9af-11e9-b58a-5254003f9b0c-etcd-certs") pod "calico-kube-controllers-996f9b774-j7lpq" (UID: "54b09258-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.174066   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "calico-kube-controllers-token-8vmnr" (UniqueName: "kubernetes.io/secret/54b09258-c9af-11e9-b58a-5254003f9b0c-calico-kube-controllers-token-8vmnr") pod "calico-kube-controllers-996f9b774-j7lpq" (UID: "54b09258-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.179079   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "config-volume" (UniqueName: "kubernetes.io/configmap/54bb05f5-c9af-11e9-b58a-5254003f9b0c-config-volume") pod "coredns-7f8f94c97b-6cm5z" (UID: "54bb05f5-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/54bb05f5-c9af-11e9-b58a-5254003f9b0c/volumes/kubernetes.io~secret/coredns-token-sl4w9.
Aug 28 16:17:29 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/54b09258-c9af-11e9-b58a-5254003f9b0c/volumes/kubernetes.io~secret/calico-kube-controllers-token-8vmnr.
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.190522   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "coredns-token-sl4w9" (UniqueName: "kubernetes.io/secret/54bb05f5-c9af-11e9-b58a-5254003f9b0c-coredns-token-sl4w9") pod "coredns-7f8f94c97b-6cm5z" (UID: "54bb05f5-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.191845   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "calico-kube-controllers-token-8vmnr" (UniqueName: "kubernetes.io/secret/54b09258-c9af-11e9-b58a-5254003f9b0c-calico-kube-controllers-token-8vmnr") pod "calico-kube-controllers-996f9b774-j7lpq" (UID: "54b09258-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.202231   11491 kuberuntime_manager.go:397] No sandbox for pod "calico-kube-controllers-996f9b774-j7lpq_kube-system(54b09258-c9af-11e9-b58a-5254003f9b0c)" can be found. Need to start a new one
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.202764023Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:calico-kube-controllers-996f9b774-j7lpq,Uid:54b09258-c9af-11e9-b58a-5254003f9b0c,Namespace:kube-system,Attempt:0,}"
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.210059078Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/4c3fbb4a6ecbb17a79c9ebe5b4a65a2983369c0e5ac2a442f6ec35bc2f84a30f/shim.sock" debug=false pid=13374
Aug 28 16:17:29 ctl01 kubelet[11491]: W0828 16:17:29.216382   11491 container.go:409] Failed to create summary reader for "/system.slice/run-r4ae4879fb5104743af77d263cc0efd7c.scope": none of the resources are being tracked.
Aug 28 16:17:29 ctl01 kubelet[11491]: W0828 16:17:29.216789   11491 container.go:409] Failed to create summary reader for "/system.slice/run-rd3439fb7b1ab498b959e56a1d747892f.scope": none of the resources are being tracked.
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.266257   11491 kuberuntime_manager.go:397] No sandbox for pod "coredns-7f8f94c97b-6cm5z_kube-system(54bb05f5-c9af-11e9-b58a-5254003f9b0c)" can be found. Need to start a new one
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.266894086Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:coredns-7f8f94c97b-6cm5z,Uid:54bb05f5-c9af-11e9-b58a-5254003f9b0c,Namespace:kube-system,Attempt:0,}"
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.274519   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/54c5749e-c9af-11e9-b58a-5254003f9b0c-etcd-certs") pod "netchecker-server-7876fb46d4-lq5jp" (UID: "54c5749e-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.274568   11491 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "netchecker-token-6xxjj" (UniqueName: "kubernetes.io/secret/54c5749e-c9af-11e9-b58a-5254003f9b0c-netchecker-token-6xxjj") pod "netchecker-server-7876fb46d4-lq5jp" (UID: "54c5749e-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.374804   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/54c5749e-c9af-11e9-b58a-5254003f9b0c-etcd-certs") pod "netchecker-server-7876fb46d4-lq5jp" (UID: "54c5749e-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.374951   11491 reconciler.go:252] operationExecutor.MountVolume started for volume "netchecker-token-6xxjj" (UniqueName: "kubernetes.io/secret/54c5749e-c9af-11e9-b58a-5254003f9b0c-netchecker-token-6xxjj") pod "netchecker-server-7876fb46d4-lq5jp" (UID: "54c5749e-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.374895   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/54c5749e-c9af-11e9-b58a-5254003f9b0c-etcd-certs") pod "netchecker-server-7876fb46d4-lq5jp" (UID: "54c5749e-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/54c5749e-c9af-11e9-b58a-5254003f9b0c/volumes/kubernetes.io~secret/netchecker-token-6xxjj.
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.385732   11491 operation_generator.go:571] MountVolume.SetUp succeeded for volume "netchecker-token-6xxjj" (UniqueName: "kubernetes.io/secret/54c5749e-c9af-11e9-b58a-5254003f9b0c-netchecker-token-6xxjj") pod "netchecker-server-7876fb46d4-lq5jp" (UID: "54c5749e-c9af-11e9-b58a-5254003f9b0c")
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.389597650Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-kube-controllers-996f9b774-j7lpq,Uid:54b09258-c9af-11e9-b58a-5254003f9b0c,Namespace:kube-system,Attempt:0,} returns sandbox id "4c3fbb4a6ecbb17a79c9ebe5b4a65a2983369c0e5ac2a442f6ec35bc2f84a30f""
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.394478   11491 kubelet.go:1953] SyncLoop (PLEG): "netchecker-agent-ggnhh_netchecker(542be8fa-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"542be8fa-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af"}
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.395401   11491 kubelet.go:1953] SyncLoop (PLEG): "calico-kube-controllers-996f9b774-j7lpq_kube-system(54b09258-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"54b09258-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"4c3fbb4a6ecbb17a79c9ebe5b4a65a2983369c0e5ac2a442f6ec35bc2f84a30f"}
Aug 28 16:17:29 ctl01 kubelet[11491]: I0828 16:17:29.417395   11491 kuberuntime_manager.go:397] No sandbox for pod "netchecker-server-7876fb46d4-lq5jp_netchecker(54c5749e-c9af-11e9-b58a-5254003f9b0c)" can be found. Need to start a new one
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.418255514Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:netchecker-server-7876fb46d4-lq5jp,Uid:54c5749e-c9af-11e9-b58a-5254003f9b0c,Namespace:netchecker,Attempt:0,}"
Aug 28 16:17:29 ctl01 kubelet[11491]: W0828 16:17:29.535007   11491 container.go:409] Failed to create summary reader for "/system.slice/run-rf7a1772a837e4d9f985d3674fbda787f.scope": none of the resources are being tracked.
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.578 [INFO][13453] calico.go 75: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"kube-system", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"ctl01", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"coredns-7f8f94c97b-6cm5z", ContainerID:"5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb"}}
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.633 [INFO][13453] k8s.go 60: Extracted identifiers for CmdAddK8s ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Namespace="kube-system" Pod="coredns-7f8f94c97b-6cm5z" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0"
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI IPAM request count IPv4=1 IPv6=0
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI IPAM handle=calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.725 [INFO][13468] calico-ipam.go 186: Auto assigning IP ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" HandleID="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Workload="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0xc42020cee0), Attrs:map[string]string(nil), Hostname:"ctl01", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}}
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.725 [INFO][13468] ipam.go 70: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'ctl01'
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.726 [INFO][13468] ipam.go 254: Looking up existing affinities for host handle="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.727 [INFO][13468] ipam.go 275: Trying affinity for 192.168.237.0/26 handle="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.728 [INFO][13468] ipam.go 118: Attempting to load block cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.729 [INFO][13468] ipam.go 195: Affinity is confirmed and block has been loaded cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.729 [INFO][13468] ipam.go 677: Attempting to assign 1 addresses from block block=192.168.237.0/26 handle="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.730 [INFO][13468] ipam.go 1110: Creating new handle: calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.732 [INFO][13468] ipam.go 700: Writing block in order to claim IPs block=192.168.237.0/26 handle="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.733 [INFO][13468] ipam.go 710: Successfully claimed IPs: [192.168.237.1] block=192.168.237.0/26 handle="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.733 [INFO][13468] ipam.go 307: Block '192.168.237.0/26' provided addresses: [192.168.237.1] handle="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.734 [INFO][13468] ipam.go 456: Auto-assigned 1 out of 1 IPv4s: [192.168.237.1] handle="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI IPAM assigned addresses IPv4=[192.168.237.1] IPv6=[]
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.734 [INFO][13468] calico-ipam.go 214: IPAM Result ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" HandleID="calico-k8s-network.5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Workload="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0" result.IPs=[]*current.IPConfig{(*current.IPConfig)(0xc4200d0660)}
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.735 [INFO][13453] k8s.go 365: Populated endpoint ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Namespace="kube-system" Pod="coredns-7f8f94c97b-6cm5z" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"", Pod:"coredns-7f8f94c97b-6cm5z", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.1/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI using IPs: [192.168.237.1/32]
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.736 [INFO][13453] network.go 75: Setting the host side veth name to cali074dcc11f9e ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Namespace="kube-system" Pod="coredns-7f8f94c97b-6cm5z" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.737 [INFO][13453] network.go 380: Disabling IPv4 forwarding ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Namespace="kube-system" Pod="coredns-7f8f94c97b-6cm5z" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0"
Aug 28 16:17:29 ctl01 kernel: [  339.484602] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 28 16:17:29 ctl01 kernel: [  339.484847] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.757 [INFO][13453] k8s.go 392: Added Mac, interface name, and active container ID to endpoint ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Namespace="kube-system" Pod="coredns-7f8f94c97b-6cm5z" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb", Pod:"coredns-7f8f94c97b-6cm5z", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.1/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"cali074dcc11f9e", MAC:"b2:54:5c:f8:94:6d", Ports:[]v3.EndpointPort(nil)}}
Aug 28 16:17:29 ctl01 systemd-udevd[13483]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.768 [INFO][13453] k8s.go 424: Wrote updated endpoint to datastore ContainerID="5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" Namespace="kube-system" Pod="coredns-7f8f94c97b-6cm5z" WorkloadEndpoint="ctl01-k8s-coredns--7f8f94c97b--6cm5z-eth0"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.785 [INFO][13505] calico.go 75: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"netchecker", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"ctl01", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"netchecker-server-7876fb46d4-lq5jp", ContainerID:"af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112"}}
Aug 28 16:17:29 ctl01 containerd[6590]: time="2019-08-28T16:17:29.786452477Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb/shim.sock" debug=false pid=13539
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.858 [INFO][13505] k8s.go 60: Extracted identifiers for CmdAddK8s ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-lq5jp" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0"
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI IPAM request count IPv4=1 IPv6=0
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI IPAM handle=calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.948 [INFO][13601] calico-ipam.go 186: Auto assigning IP ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" HandleID="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Workload="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0" assignArgs=ipam.AutoAssignArgs{Num4:1, Num6:0, HandleID:(*string)(0xc42027cee0), Attrs:map[string]string(nil), Hostname:"ctl01", IPv4Pools:[]net.IPNet{}, IPv6Pools:[]net.IPNet{}}
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.948 [INFO][13601] ipam.go 70: Auto-assign 1 ipv4, 0 ipv6 addrs for host 'ctl01'
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.949 [INFO][13601] ipam.go 254: Looking up existing affinities for host handle="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.949 [INFO][13601] ipam.go 275: Trying affinity for 192.168.237.0/26 handle="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.950 [INFO][13601] ipam.go 118: Attempting to load block cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.951 [INFO][13601] ipam.go 195: Affinity is confirmed and block has been loaded cidr=192.168.237.0/26 host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.951 [INFO][13601] ipam.go 677: Attempting to assign 1 addresses from block block=192.168.237.0/26 handle="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.951 [INFO][13601] ipam.go 1110: Creating new handle: calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.952 [INFO][13601] ipam.go 700: Writing block in order to claim IPs block=192.168.237.0/26 handle="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.953 [INFO][13601] ipam.go 710: Successfully claimed IPs: [192.168.237.2] block=192.168.237.0/26 handle="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.953 [INFO][13601] ipam.go 307: Block '192.168.237.0/26' provided addresses: [192.168.237.2] handle="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.954 [INFO][13601] ipam.go 456: Auto-assigned 1 out of 1 IPv4s: [192.168.237.2] handle="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" host="ctl01"
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI IPAM assigned addresses IPv4=[192.168.237.2] IPv6=[]
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.954 [INFO][13601] calico-ipam.go 214: IPAM Result ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" HandleID="calico-k8s-network.af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Workload="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0" result.IPs=[]*current.IPConfig{(*current.IPConfig)(0xc420092420)}
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.955 [INFO][13505] k8s.go 365: Populated endpoint ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-lq5jp" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"", Pod:"netchecker-server-7876fb46d4-lq5jp", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.2/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"", MAC:"", Ports:[]v3.EndpointPort(nil)}}
Aug 28 16:17:29 ctl01 containerd[6590]: Calico CNI using IPs: [192.168.237.2/32]
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.955 [INFO][13505] network.go 75: Setting the host side veth name to calib4eb019900f ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-lq5jp" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0"
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.956 [INFO][13505] network.go 380: Disabling IPv4 forwarding ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-lq5jp" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0"
Aug 28 16:17:29 ctl01 kernel: [  339.703846] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 28 16:17:29 ctl01 kernel: [  339.704047] IPv6: ADDRCONF(NETDEV_UP): calib4eb019900f: link is not ready
Aug 28 16:17:29 ctl01 systemd-udevd[13629]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 28 16:17:29 ctl01 kernel: [  339.733221] IPv6: ADDRCONF(NETDEV_UP): calib4eb019900f: link is not ready
Aug 28 16:17:29 ctl01 kernel: [  339.733236] IPv6: ADDRCONF(NETDEV_CHANGE): calib4eb019900f: link becomes ready
Aug 28 16:17:29 ctl01 kernel: [  339.733322] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.989 [INFO][13505] k8s.go 392: Added Mac, interface name, and active container ID to endpoint ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-lq5jp" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0" endpoint=&v3.WorkloadEndpoint{TypeMeta:v1.TypeMeta{Kind:"WorkloadEndpoint", APIVersion:"projectcalico.org/v3"}, ObjectMeta:v1.ObjectMeta{Name:"ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0", GenerateName:"", Namespace:"netchecker", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v3.WorkloadEndpointSpec{Orchestrator:"k8s", Workload:"", Node:"ctl01", ContainerID:"af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112", Pod:"netchecker-server-7876fb46d4-lq5jp", Endpoint:"eth0", IPNetworks:[]string{"192.168.237.2/32"}, IPNATs:[]v3.IPNAT(nil), IPv4Gateway:"", IPv6Gateway:"", Profiles:[]string{"calico-k8s-network"}, InterfaceName:"calib4eb019900f", MAC:"42:69:35:2d:67:55", Ports:[]v3.EndpointPort(nil)}}
Aug 28 16:17:29 ctl01 containerd[6590]: 2019-08-28 16:17:29.993 [INFO][13505] k8s.go 424: Wrote updated endpoint to datastore ContainerID="af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" Namespace="netchecker" Pod="netchecker-server-7876fb46d4-lq5jp" WorkloadEndpoint="ctl01-k8s-netchecker--server--7876fb46d4--lq5jp-eth0"
Aug 28 16:17:30 ctl01 containerd[6590]: time="2019-08-28T16:17:30.009544011Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112/shim.sock" debug=false pid=13672
Aug 28 16:17:30 ctl01 containerd[6590]: time="2019-08-28T16:17:30.019489325Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-7f8f94c97b-6cm5z,Uid:54bb05f5-c9af-11e9-b58a-5254003f9b0c,Namespace:kube-system,Attempt:0,} returns sandbox id "5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb""
Aug 28 16:17:30 ctl01 containerd[6590]: time="2019-08-28T16:17:30.189556577Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:netchecker-server-7876fb46d4-lq5jp,Uid:54c5749e-c9af-11e9-b58a-5254003f9b0c,Namespace:netchecker,Attempt:0,} returns sandbox id "af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112""
Aug 28 16:17:30 ctl01 kubelet[11491]: I0828 16:17:30.398268   11491 kubelet.go:1953] SyncLoop (PLEG): "coredns-7f8f94c97b-6cm5z_kube-system(54bb05f5-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"54bb05f5-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb"}
Aug 28 16:17:30 ctl01 kubelet[11491]: I0828 16:17:30.400385   11491 kubelet.go:1953] SyncLoop (PLEG): "netchecker-server-7876fb46d4-lq5jp_netchecker(54c5749e-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"54c5749e-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112"}
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.028654555Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-agent:stable,Labels:map[string]string{},}"
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.036527854Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:d16c8d7f4d5bdaecda098c54c655027871da52f292683759f5a32c8ecd43b3f0,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.037535511Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-agent:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.334891483Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:d16c8d7f4d5bdaecda098c54c655027871da52f292683759f5a32c8ecd43b3f0,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.337308975Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-agent:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.339771235Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-agent@sha256:4ac49ebef7eaeaa5a8a19f56faa73740bf4861979aa067d0125867a72846720a,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.340505863Z" level=info msg="PullImage "mirantis/k8s-netchecker-agent:stable" returns image reference "sha256:d16c8d7f4d5bdaecda098c54c655027871da52f292683759f5a32c8ecd43b3f0""
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.342164849Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2""
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.346806296Z" level=info msg="CreateContainer within sandbox "7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" for container &ContainerMetadata{Name:netchecker-agent,Attempt:0,}"
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.448015850Z" level=info msg="CreateContainer within sandbox "7a5164ff7b42d3339c4c24115e981216ca147b3b48f5a319ccf6c631178188af" for &ContainerMetadata{Name:netchecker-agent,Attempt:0,} returns container id "7e20df48f5fcb45b1b06685fef1603bf4349b9ba3624c759985e4dc6067658eb""
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.449185702Z" level=info msg="StartContainer for "7e20df48f5fcb45b1b06685fef1603bf4349b9ba3624c759985e4dc6067658eb""
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.450284725Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/7e20df48f5fcb45b1b06685fef1603bf4349b9ba3624c759985e4dc6067658eb/shim.sock" debug=false pid=13776
Aug 28 16:17:31 ctl01 containerd[6590]: time="2019-08-28T16:17:31.658116492Z" level=info msg="StartContainer for "7e20df48f5fcb45b1b06685fef1603bf4349b9ba3624c759985e4dc6067658eb" returns successfully"
Aug 28 16:17:32 ctl01 kubelet[11491]: I0828 16:17:32.406613   11491 kubelet.go:1953] SyncLoop (PLEG): "netchecker-agent-ggnhh_netchecker(542be8fa-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"542be8fa-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"7e20df48f5fcb45b1b06685fef1603bf4349b9ba3624c759985e4dc6067658eb"}
Aug 28 16:17:35 ctl01 containerd[6590]: time="2019-08-28T16:17:35.899635410Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2,Labels:map[string]string{},}"
Aug 28 16:17:35 ctl01 containerd[6590]: time="2019-08-28T16:17:35.907923271Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:22c16a9aecce95ea9631683dd66ee0329e399e07bb86950239d4a6db1c6d201b,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:35 ctl01 containerd[6590]: time="2019-08-28T16:17:35.908423852Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:36 ctl01 kubelet[11491]: I0828 16:17:36.239174   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.437211303Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:22c16a9aecce95ea9631683dd66ee0329e399e07bb86950239d4a6db1c6d201b,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.439377239Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.441815897Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers@sha256:59296255e44bcb3ab00dae468a795dd669c41d6d1f16b8b5a3f2a1271e88e810,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.442229499Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/projectcalico/calico/kube-controllers:v3.3.2" returns image reference "sha256:22c16a9aecce95ea9631683dd66ee0329e399e07bb86950239d4a6db1c6d201b""
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.443904658Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96""
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.445686271Z" level=info msg="CreateContainer within sandbox "4c3fbb4a6ecbb17a79c9ebe5b4a65a2983369c0e5ac2a442f6ec35bc2f84a30f" for container &ContainerMetadata{Name:calico-kube-controllers,Attempt:0,}"
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.529851794Z" level=info msg="CreateContainer within sandbox "4c3fbb4a6ecbb17a79c9ebe5b4a65a2983369c0e5ac2a442f6ec35bc2f84a30f" for &ContainerMetadata{Name:calico-kube-controllers,Attempt:0,} returns container id "4a4d16fd044d3b5b617e551aab3e4f3b8597beb1ac6a88282912e4e4b8cb2c49""
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.530711800Z" level=info msg="StartContainer for "4a4d16fd044d3b5b617e551aab3e4f3b8597beb1ac6a88282912e4e4b8cb2c49""
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.531389105Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/4a4d16fd044d3b5b617e551aab3e4f3b8597beb1ac6a88282912e4e4b8cb2c49/shim.sock" debug=false pid=13858
Aug 28 16:17:37 ctl01 kube-proxy[9844]: I0828 16:17:37.874601    9844 proxier.go:659] Stale udp service kube-system/coredns:dns -> 10.254.0.10
Aug 28 16:17:37 ctl01 containerd[6590]: time="2019-08-28T16:17:37.892513722Z" level=info msg="StartContainer for "4a4d16fd044d3b5b617e551aab3e4f3b8597beb1ac6a88282912e4e4b8cb2c49" returns successfully"
Aug 28 16:17:37 ctl01 kernel: [  347.653239] ctnetlink v0.93: registering with nfnetlink.
Aug 28 16:17:38 ctl01 kubelet[11491]: I0828 16:17:38.417664   11491 kubelet.go:1953] SyncLoop (PLEG): "calico-kube-controllers-996f9b774-j7lpq_kube-system(54b09258-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"54b09258-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"4a4d16fd044d3b5b617e551aab3e4f3b8597beb1ac6a88282912e4e4b8cb2c49"}
Aug 28 16:17:38 ctl01 etcd[5283]: read-only range request "key:\"/calico/resources/v3/projectcalico.org/clusterinformations/default\" " with result "range_response_count:1 size:399" took too long (100.288115ms) to execute
Aug 28 16:17:39 ctl01 containerd[6590]: time="2019-08-28T16:17:39.201196909Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96,Labels:map[string]string{},}"
Aug 28 16:17:39 ctl01 containerd[6590]: time="2019-08-28T16:17:39.210652860Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:31b943183ba2eec561c44c1fdda2a39ab03c0cb72f90a24c1bea4adf0a44bfbc,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:39 ctl01 containerd[6590]: time="2019-08-28T16:17:39.211462856Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.322287853Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:31b943183ba2eec561c44c1fdda2a39ab03c0cb72f90a24c1bea4adf0a44bfbc,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.324437815Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.326729686Z" level=info msg="ImageCreate event &ImageCreate{Name:docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns@sha256:30f28dcd8a8c9a97c206bba187bd9b1e8fbc1cf52ce38f7e937c85a191709376,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.327323135Z" level=info msg="PullImage "docker-prod-local.artifactory.mirantis.com/mirantis/coredns/coredns:v1.4.0-96" returns image reference "sha256:31b943183ba2eec561c44c1fdda2a39ab03c0cb72f90a24c1bea4adf0a44bfbc""
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.332582281Z" level=info msg="PullImage "mirantis/k8s-netchecker-server:stable""
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.336449797Z" level=info msg="CreateContainer within sandbox "5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" for container &ContainerMetadata{Name:coredns,Attempt:0,}"
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.440863619Z" level=info msg="CreateContainer within sandbox "5ee2fbc8b5f8800e53b09636c78fbde18987afe08397ed1bab12cd2ac267c4fb" for &ContainerMetadata{Name:coredns,Attempt:0,} returns container id "6f984c3a5ab01b48299daad772ec89c5f40f9c30955c8cec96f9a87ea7f7e17a""
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.442029064Z" level=info msg="StartContainer for "6f984c3a5ab01b48299daad772ec89c5f40f9c30955c8cec96f9a87ea7f7e17a""
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.443199911Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/6f984c3a5ab01b48299daad772ec89c5f40f9c30955c8cec96f9a87ea7f7e17a/shim.sock" debug=false pid=13965
Aug 28 16:17:40 ctl01 containerd[6590]: time="2019-08-28T16:17:40.605880777Z" level=info msg="StartContainer for "6f984c3a5ab01b48299daad772ec89c5f40f9c30955c8cec96f9a87ea7f7e17a" returns successfully"
Aug 28 16:17:41 ctl01 kubelet[11491]: I0828 16:17:41.426758   11491 kubelet.go:1953] SyncLoop (PLEG): "coredns-7f8f94c97b-6cm5z_kube-system(54bb05f5-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"54bb05f5-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"6f984c3a5ab01b48299daad772ec89c5f40f9c30955c8cec96f9a87ea7f7e17a"}
Aug 28 16:17:41 ctl01 etcd[5283]: read-only range request "key:\"/calico/resources/v3/projectcalico.org/workloadendpoints/netchecker/cmp002-k8s-netchecker--agent--pk6cn-eth0\" " with result "range_response_count:1 size:906" took too long (144.408571ms) to execute
Aug 28 16:17:42 ctl01 containerd[6590]: time="2019-08-28T16:17:42.586372403Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-server:stable,Labels:map[string]string{},}"
Aug 28 16:17:42 ctl01 containerd[6590]: time="2019-08-28T16:17:42.595806459Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:22d27d4fe076561ad51f5e7a0228ab7b1035773a492c95fd4a246b40ba4f9b58,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:42 ctl01 containerd[6590]: time="2019-08-28T16:17:42.596735095Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-server:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.447829647Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:22d27d4fe076561ad51f5e7a0228ab7b1035773a492c95fd4a246b40ba4f9b58,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.449863677Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/mirantis/k8s-netchecker-server:stable,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.451712241Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/mirantis/k8s-netchecker-server@sha256:87064b3cb72a8fb419eb714ae892ebd5929649e84499c08111e46e466acc8407,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.452473430Z" level=info msg="PullImage "mirantis/k8s-netchecker-server:stable" returns image reference "sha256:22d27d4fe076561ad51f5e7a0228ab7b1035773a492c95fd4a246b40ba4f9b58""
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.455117349Z" level=info msg="CreateContainer within sandbox "af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" for container &ContainerMetadata{Name:netchecker-server,Attempt:0,}"
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.535609341Z" level=info msg="CreateContainer within sandbox "af6bdcc8acb881d8ab8aab0851ec07aff441d13a287a601d7b28e1804f191112" for &ContainerMetadata{Name:netchecker-server,Attempt:0,} returns container id "9a70faa9c56fc04025a62fdc5dd90f2f03afe40690df0924c5a6b4f496a92739""
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.536410063Z" level=info msg="StartContainer for "9a70faa9c56fc04025a62fdc5dd90f2f03afe40690df0924c5a6b4f496a92739""
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.537264703Z" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/9a70faa9c56fc04025a62fdc5dd90f2f03afe40690df0924c5a6b4f496a92739/shim.sock" debug=false pid=14089
Aug 28 16:17:45 ctl01 containerd[6590]: time="2019-08-28T16:17:45.732202501Z" level=info msg="StartContainer for "9a70faa9c56fc04025a62fdc5dd90f2f03afe40690df0924c5a6b4f496a92739" returns successfully"
Aug 28 16:17:46 ctl01 kubelet[11491]: I0828 16:17:46.247085   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:17:46 ctl01 kubelet[11491]: I0828 16:17:46.438006   11491 kubelet.go:1953] SyncLoop (PLEG): "netchecker-server-7876fb46d4-lq5jp_netchecker(54c5749e-c9af-11e9-b58a-5254003f9b0c)", event: &pleg.PodLifecycleEvent{ID:"54c5749e-c9af-11e9-b58a-5254003f9b0c", Type:"ContainerStarted", Data:"9a70faa9c56fc04025a62fdc5dd90f2f03afe40690df0924c5a6b4f496a92739"}
Aug 28 16:17:56 ctl01 kubelet[11491]: I0828 16:17:56.266667   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:18:06 ctl01 kubelet[11491]: I0828 16:18:06.276283   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:18:16 ctl01 kubelet[11491]: I0828 16:18:16.288107   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:18:26 ctl01 kubelet[11491]: I0828 16:18:26.302703   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:18:36 ctl01 kubelet[11491]: I0828 16:18:36.318263   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:18:46 ctl01 kubelet[11491]: I0828 16:18:46.329299   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:18:56 ctl01 kubelet[11491]: I0828 16:18:56.340440   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:19:06 ctl01 kubelet[11491]: I0828 16:19:06.348883   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:19:16 ctl01 kubelet[11491]: I0828 16:19:16.359156   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:19:26 ctl01 kubelet[11491]: I0828 16:19:26.370286   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:19:36 ctl01 kubelet[11491]: I0828 16:19:36.382550   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:19:46 ctl01 kubelet[11491]: I0828 16:19:46.408464   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:19:56 ctl01 kubelet[11491]: I0828 16:19:56.420545   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:20:05 ctl01 kube-apiserver[11892]: I0828 16:20:05.576933   11892 node_authorizer.go:176] NODE DENY: ctl01.mcp-k8s-calico-noha.local &authorizer.AttributesRecord{User:(*user.DefaultInfo)(0xc00b0d2040), Verb:"create", Namespace:"default", APIGroup:"", APIVersion:"v1", Resource:"configmaps", Subresource:"", Name:"", ResourceRequest:true, Path:"/api/v1/namespaces/default/configmaps"}
Aug 28 16:20:05 ctl01 kube-apiserver[11892]: I0828 16:20:05.659720   11892 node_authorizer.go:176] NODE DENY: ctl01.mcp-k8s-calico-noha.local &authorizer.AttributesRecord{User:(*user.DefaultInfo)(0xc00b09e580), Verb:"delete", Namespace:"default", APIGroup:"", APIVersion:"v1", Resource:"configmaps", Subresource:"", Name:"k8s-b576531c-key", ResourceRequest:true, Path:"/api/v1/namespaces/default/configmaps/k8s-b576531c-key"}
Aug 28 16:20:05 ctl01 kube-apiserver[11892]: I0828 16:20:05.804593   11892 node_authorizer.go:176] NODE DENY: ctl01.mcp-k8s-calico-noha.local &authorizer.AttributesRecord{User:(*user.DefaultInfo)(0xc00bf84900), Verb:"delete", Namespace:"default", APIGroup:"", APIVersion:"v1", Resource:"configmaps", Subresource:"", Name:"k8s-b576531c-key", ResourceRequest:true, Path:"/api/v1/namespaces/default/configmaps/k8s-b576531c-key"}
Aug 28 16:20:06 ctl01 kubelet[11491]: I0828 16:20:06.429223   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:20:16 ctl01 kubelet[11491]: I0828 16:20:16.445340   11491 setters.go:72] Using node IP: "172.16.10.36"
Aug 28 16:20:18 ctl01 salt-minion[4490]: [INFO    ] User sudo_ubuntu Executing command cp.push_dir with jid 20190828162018279276
Aug 28 16:20:18 ctl01 salt-minion[4490]: [INFO    ] Starting a new job with PID 14872
