6. API and Feature Testing requirements

6.1. Introduction

The CNCF has defined a Testing Special Interest Group to make it easier for the community to write and run tests, and to contribute, analyse and act upon test results. This chapter maps the requirements written in the previous chapters as mandatory Special Interest Group Features. It enforces the overall requirements traceability to testing, especially those offered for End-to-End Testing. The Anuket Reference Conformance (RC2) testing then matches the following Features tabs defined here.

6.1.1. Kubernetes feature gate policy

Feature gates are a set of key-value pairs that describe Kubernetes features. The components of the control plane of Kubernetes Clusters can be run with different Feature Gate settings.

A feature can be in Alpha, Beta or GA stage:

  • Alpha features are disabled by default, may be buggy, and support may be dropped

  • Beta features are enabled by default, are well tested, and support will not be dropped (although breaking API changes may happen)

  • GA features are stable, always enabled and cannot be disabled.

The policy for RA2 to include Kubernetes features as mandatory is:

Only features that are either in Beta or GA stage can be made mandatory, subject to RA2 requirements.

A list of feature gates is available here.

6.1.2. Kubernetes API policy

The Kubernetes API supports all operations and communications between components, and external user commands. Everything in the Kubernetes platform is treated as an API object. Different API versions indicate different levels of stability and support. An API can have Alpha, Beta or Stable versions. The version of APIs that are backed by a feature will match the stage of the feature itself (i.e. Alpha, Beta or GA/Stable).

The policy for RA2 to include Kubernetes APIs as mandatory is:

Only APIs that are either in Beta or Stable stage can be made mandatory, subject to RA2 requirements.

The Kubernetes API reference is available here.

The list of API groups that are mandatory is:

Table 6.1 Mandatory API Groups

Group

Version

admissionregistration.k8s.io

v1

apiextensions.k8s.io

v1

apiregistration.k8s.io

v1

apps

v1

authentication.k8s.io

v1

authorization.k8s.io

v1

autoscaling

v1, v2, v2beta2, v2beta1

batch

v1, v1beta1

certificates.k8s.io

v1

coordination.k8s.io

v1

core

v1

discovery.k8s.io

v1, v1beta1

events.k8s.io

v1, v1beta1

flowcontrol.apiserver.k8s.io

v1beta2, v1beta1

networking.k8s.io

v1

node.k8s.io

v1, v1beta1

policy

v1, v1beta1

rbac.authorization.k8s.io

v1

scheduling.k8s.io

v1

storage.k8s.io

v1, v1beta1

6.2. API Machinery Special Interest Group

Table 6.2 API Machinery Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:ComprehensiveNamespaceDraining

X

Namespaces should always delete fast (ALL of 100 namespaces in 150 seconds)

Feature:CrossNamespacePodAffinity

Should verify ResourceQuota with cross namespace pod affinity scope using scope-selectors

Feature:Feature:PodPriority

X

Verify ResourceQuota’s priority class scope against a pod with different priority class

Feature:ScopeSelectors

X

Verify ResourceQuota with terminating scopes through scope selectors

Feature:StorageVersionAPI

6.3. Apps Special Interest Group

Table 6.3 Apps Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:DaemonSetUpdateSurge

Daemon set should surge pods onto nodes when spec was updated and update strategy is RollingUpdate

Feature:IndexedJob

Should create pods for an Indexed job with completion indexes

Feature:StatefulSet

Should creating a working zookeeper cluster

Feature:StatefulUpgrade

Stateful upgrade should maintain a functioning cluster

Feature:SuspendJob

Should not create pods when created in suspend state

Feature:TaintEviction

All pods on the unreachable node should be marked as NotReady upon the node turn NotReady AND all pods should be evicted after eviction timeout passes

Feature:TTLAfterFinished

X

Job should be deleted once it finishes after TTL seconds

6.4. Auth Special Interest Group

Table 6.4 Auth Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:BoundServiceAccountTokenVolume

ServiceAccount admission controller migration master upgrade should maintain a functioning cluster

Feature:NodeAuthenticator

X

The kubelet’s main port 10250 should reject requests with no credentials

Feature:NodeAuthorizer

X

Setting existing and non-existent attributes should exit with the Forbidden error, not a NotFound error

Feature:PodSecurityPolicy

Should enforce the restricted policy.PodSecurityPolicy

NodeFeature:FSGroup

X

ServiceAccounts should set ownership and permission when RunAsUser or FsGroup is present

6.5. Cluster Lifecycle Special Interest Group

Table 6.5 Cluster Lifecycle Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:BootstrapTokens

X

Should delete the token secret when the secret expired

6.6. Instrumentation Special Interest Group

Table 6.6 Instrumentation Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:Elasticsearch

Should check that the Kibana logging instance is alive

Feature: StackdriverAcceleratorMonitoring

Stackdriver Monitoring should have accelerator metrics

Feature:StackdriverCustomMetrics

Stackdriver Monitoring should run Custom Metrics - Stackdriver Adapter for new resource model

Feature:StackdriverExternalMetrics

Stackdriver Monitoring should run Custom Metrics - Stackdriver Adapter for external metrics

Feature:StackdriverMetadataAgent

Stackdriver Monitoring should run Stackdriver Metadata Agent

Feature:StackdriverMonitoring

6.7. Network Special Interest Group

Table 6.7 Network Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:Example

Should create pod that uses DNS

Feature:Ingress

Should prevent Ingress creation if more than 1 IngressClass marked as default

Feature:IPv6DualStack

IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to Pods and Services. IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in 1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses.

Feature:kubemci

Should create ingress with pre-shared certificate

Feature:KubeProxyDaemonSetMigration

Upgrade kube-proxy from static pods to a DaemonSet should maintain a functioning cluster

Feature:KubeProxyDaemonSetUpgrade

Upgrade kube-proxy from static pods to a DaemonSet should maintain a functioning cluster

Feature:NEG

Should sync endpoints to NEG

Feature:NoSNAT

X

Should be able to send traffic between Pods without SNAT

Feature:Networking-IPv4

X

Networking should provide Internet connection for containers

Feature:Networking-IPv6

Networking should provide Internet connection for containers

Feature:Networking-Performance

X

run iperf2

Feature:NetworkPolicy

NetworkPolicy between server and client should enforce policy to allow traffic only from a different namespace, based on NamespaceSelector

Feature:PerformanceDNS

Should answer DNS query for maximum number of services per cluster

Feature:SCTP

should allow creating a basic SCTP service with pod and endpoints

Feature:SCTPConnectivity

Pods should function for intra-pod communication: sctp

6.8. Node Special Interest Group

Table 6.8 Node Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:Example

X

Liveness pods should be automatically restarted

Feature: ExperimentalResourceUsageTracking

Resource tracking for 100 pods per node

Feature:GPUUpgrade

Master upgrade should NOT disrupt GPU Pod

Feature:PodGarbageCollector

Should handle the creation of 1000 pods

Feature:RegularResourceUsageTracking

Resource tracking for 0 pods per node

Feature:ProbeTerminationGracePeriod

X

Probing container should override timeoutGracePeriodSeconds when LivenessProbe field is set

NodeFeature:DownwardAPIHugePages

Downward API tests for huge pages should provide container’s limits.hugepages-pagesize; and requests.hugepages-pagesize& as env vars

NodeFeature:PodReadinessGate

X

Pods should support pod readiness gates

NodeFeature:RuntimeHandler

RuntimeClass should run a Pod requesting a RuntimeClass with a configured handler

NodeFeature:Sysctls

X

Should not launch unsafe, but not explicitly enabled sysctls on the node

6.9. Scheduling Special Interest Group

Table 6.9 Scheduling Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:GPUDevicePlugin

Run Nvidia GPU Device Plugin tests

Feature:LocalStorageCapacityIsolation

X

Validates local ephemeral storage resource limits of pods that are allowed to run

Feature:Recreate

Run Nvidia GPU Device Plugin tests with a recreation

6.10. Storage Special Interest Group

Table 6.10 API Machinery Special Interest Group

Labels

Mandatory

Description

Conformance

X

Kubernetes conformance test

None

X

Kubernetes mainstream features

Feature:ExpandInUsePersistentVolumes

Feature:Flexvolumes

Feature:GKELocalSSD

Feature:VolumeSnapshotDataSource

Feature:Volumes

X

Feature:vsphere

Feature:Windows

NodeFeature:EphemeralStorage

X

NodeFeature:FSGroup

X