==== start of pod log for container csi-mockplugin-0/csi-provisioner ==== W0418 18:13:04.609550 1 feature_gate.go:237] Setting GA feature gate Topology=true. It will be removed in a future release. I0418 18:13:04.609617 1 feature_gate.go:245] feature gates: &{map[Topology:true]} I0418 18:13:04.609650 1 csi-provisioner.go:150] Version: v3.3.0 I0418 18:13:04.609663 1 csi-provisioner.go:173] Building kube configs for running in cluster... I0418 18:13:04.610503 1 connection.go:154] Connecting to unix:///csi/csi.sock I0418 18:13:05.613986 1 common.go:111] Probing CSI driver for readiness I0418 18:13:05.614025 1 connection.go:183] GRPC call: /csi.v1.Identity/Probe I0418 18:13:05.614033 1 connection.go:184] GRPC request: {} I0418 18:13:05.625005 1 connection.go:186] GRPC response: {} I0418 18:13:05.625135 1 connection.go:187] GRPC error: I0418 18:13:05.625153 1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo I0418 18:13:05.625160 1 connection.go:184] GRPC request: {} I0418 18:13:05.626081 1 connection.go:186] GRPC response: {"name":"csi-mock-csi-mock-volumes-7259","vendor_version":"v1.9.0"} I0418 18:13:05.626158 1 connection.go:187] GRPC error: I0418 18:13:05.626175 1 csi-provisioner.go:217] Detected CSI driver csi-mock-csi-mock-volumes-7259 I0418 18:13:05.626190 1 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginCapabilities I0418 18:13:05.626197 1 connection.go:184] GRPC request: {} I0418 18:13:05.627458 1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]} I0418 18:13:05.627676 1 connection.go:187] GRPC error: I0418 18:13:05.627700 1 connection.go:183] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0418 18:13:05.627709 1 connection.go:184] GRPC request: {} I0418 18:13:05.629212 1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":12}}},{"Type":{"Rpc":{"type":4}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":11}}},{"Type":{"Rpc":{"type":13}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":2}}}]} I0418 18:13:05.629543 1 connection.go:187] GRPC error: I0418 18:13:05.629655 1 csi-provisioner.go:286] CSI driver supports PUBLISH_UNPUBLISH_VOLUME, watching VolumeAttachments I0418 18:13:05.631292 1 controller.go:732] Using saving PVs to API server in background I0418 18:13:05.633076 1 reflector.go:221] Starting reflector *v1.StorageClass (1h0m0s) from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633081 1 reflector.go:221] Starting reflector *v1.Node (1h0m0s) from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633117 1 reflector.go:257] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633127 1 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633089 1 reflector.go:221] Starting reflector *v1.CSINode (1h0m0s) from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633152 1 reflector.go:257] Listing and watching *v1.CSINode from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633132 1 reflector.go:221] Starting reflector *v1.VolumeAttachment (1h0m0s) from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633198 1 reflector.go:257] Listing and watching *v1.VolumeAttachment from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633182 1 reflector.go:221] Starting reflector *v1.PersistentVolumeClaim (15m0s) from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.633229 1 reflector.go:257] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:134 I0418 18:13:05.732159 1 shared_informer.go:285] caches populated I0418 18:13:05.732207 1 shared_informer.go:285] caches populated I0418 18:13:05.732216 1 shared_informer.go:285] caches populated I0418 18:13:05.732222 1 shared_informer.go:285] caches populated I0418 18:13:05.732229 1 shared_informer.go:285] caches populated I0418 18:13:05.732242 1 controller.go:811] Starting provisioner controller csi-mock-csi-mock-volumes-7259_csi-mockplugin-0_aff0ebcb-2ce1-4fb8-8549-7c9964bd4f11! I0418 18:13:05.732287 1 volume_store.go:97] Starting save volume queue I0418 18:13:05.732279 1 clone_controller.go:66] Starting CloningProtection controller I0418 18:13:05.732467 1 clone_controller.go:82] Started CloningProtection controller I0418 18:13:05.732668 1 reflector.go:221] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:848 I0418 18:13:05.732685 1 reflector.go:257] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:848 I0418 18:13:05.732811 1 reflector.go:221] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:845 I0418 18:13:05.732835 1 reflector.go:257] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/v8/controller/controller.go:845 I0418 18:13:05.833272 1 shared_informer.go:285] caches populated I0418 18:13:05.833342 1 controller.go:860] Started provisioner controller csi-mock-csi-mock-volumes-7259_csi-mockplugin-0_aff0ebcb-2ce1-4fb8-8549-7c9964bd4f11! E0418 18:13:05.833637 1 controller.go:1026] claim "6ed3df6b-49f8-4c74-ac26-49d2aa9409db" in work queue no longer exists E0418 18:13:05.833701 1 controller.go:1026] claim "ce064b0c-3b64-400c-84be-98029e4f7990" in work queue no longer exists I0418 18:13:11.445520 1 controller.go:1337] provision "csi-mock-volumes-7259/pvc-volume-tester-km4kk-my-volume" class "csi-mock-volumes-7259-scmxzfz": started I0418 18:13:11.445886 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume I0418 18:13:11.445754 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"csi-mock-volumes-7259", Name:"pvc-volume-tester-km4kk-my-volume", UID:"f1ad1d59-8484-4a1f-9c8d-69b86771463a", APIVersion:"v1", ResourceVersion:"90829", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "csi-mock-volumes-7259/pvc-volume-tester-km4kk-my-volume" I0418 18:13:11.445905 1 connection.go:184] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.hostpath.csi/node":"v126-worker"}},{"segments":{"topology.hostpath.csi/node":"v126-worker2"}}],"requisite":[{"segments":{"topology.hostpath.csi/node":"v126-worker"}},{"segments":{"topology.hostpath.csi/node":"v126-worker2"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}}]} I0418 18:13:11.451208 1 connection.go:186] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.hostpath.csi/node":"v126-worker2"}}],"capacity_bytes":1073741824,"volume_id":"50f66235-fdaf-11ee-b1d9-8ac63d14c32c"}} I0418 18:13:11.451386 1 connection.go:187] GRPC error: I0418 18:13:11.451403 1 controller.go:812] create volume rep: {CapacityBytes:1073741824 VolumeId:50f66235-fdaf-11ee-b1d9-8ac63d14c32c VolumeContext:map[] ContentSource: AccessibleTopology:[segments: ] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} I0418 18:13:11.451482 1 controller.go:881] createVolumeOperation: set annotation [volume.kubernetes.io/provisioner-deletion-secret-namespace/volume.kubernetes.io/provisioner-deletion-secret-name] on pv [pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a]. I0418 18:13:11.451513 1 controller.go:906] successfully created PV pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a for PVC pvc-volume-tester-km4kk-my-volume and csi volume name 50f66235-fdaf-11ee-b1d9-8ac63d14c32c I0418 18:13:11.451533 1 controller.go:922] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:csi-mock-csi-mock-volumes-7259,VolumeHandle:50f66235-fdaf-11ee-b1d9-8ac63d14c32c,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{storage.kubernetes.io/csiProvisionerIdentity: 1713463985629-8081-csi-mock-csi-mock-volumes-7259,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,}} I0418 18:13:11.451755 1 controller.go:1442] provision "csi-mock-volumes-7259/pvc-volume-tester-km4kk-my-volume" class "csi-mock-volumes-7259-scmxzfz": volume "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a" provisioned I0418 18:13:11.451786 1 controller.go:1455] provision "csi-mock-volumes-7259/pvc-volume-tester-km4kk-my-volume" class "csi-mock-volumes-7259-scmxzfz": succeeded I0418 18:13:11.451805 1 volume_store.go:154] Saving volume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a I0418 18:13:11.458694 1 volume_store.go:157] Volume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a saved I0418 18:13:11.458817 1 controller.go:1069] Claim processing succeeded, removing PVC f1ad1d59-8484-4a1f-9c8d-69b86771463a from claims in progress I0418 18:13:11.458874 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"csi-mock-volumes-7259", Name:"pvc-volume-tester-km4kk-my-volume", UID:"f1ad1d59-8484-4a1f-9c8d-69b86771463a", APIVersion:"v1", ResourceVersion:"90829", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a I0418 18:13:33.300134 1 controller.go:1471] delete "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": started I0418 18:13:33.300214 1 controller.go:1261] volume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a does not need any deletion secrets E0418 18:13:33.300247 1 controller.go:1481] delete "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": volume deletion failed: persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 W0418 18:13:33.300313 1 controller.go:989] Retrying syncing volume "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a", failure 0 E0418 18:13:33.300339 1 controller.go:1007] error syncing volume "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 I0418 18:13:33.300423 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a", UID:"d477a2ff-b021-401b-88cb-b6ebf61a6c4b", APIVersion:"v1", ResourceVersion:"91406", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 I0418 18:13:34.300802 1 controller.go:1471] delete "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": started I0418 18:13:34.300913 1 controller.go:1261] volume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a does not need any deletion secrets E0418 18:13:34.300962 1 controller.go:1481] delete "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": volume deletion failed: persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 W0418 18:13:34.301015 1 controller.go:989] Retrying syncing volume "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a", failure 1 E0418 18:13:34.301048 1 controller.go:1007] error syncing volume "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 I0418 18:13:34.301129 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a", UID:"d477a2ff-b021-401b-88cb-b6ebf61a6c4b", APIVersion:"v1", ResourceVersion:"91406", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 I0418 18:13:36.301887 1 controller.go:1471] delete "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": started I0418 18:13:36.301983 1 controller.go:1261] volume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a does not need any deletion secrets E0418 18:13:36.302042 1 controller.go:1481] delete "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": volume deletion failed: persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 W0418 18:13:36.302096 1 controller.go:989] Retrying syncing volume "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a", failure 2 E0418 18:13:36.302130 1 controller.go:1007] error syncing volume "pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a": persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 I0418 18:13:36.302161 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a", UID:"d477a2ff-b021-401b-88cb-b6ebf61a6c4b", APIVersion:"v1", ResourceVersion:"91406", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' persistentvolume pvc-f1ad1d59-8484-4a1f-9c8d-69b86771463a is still attached to node v126-worker2 ==== end of pod log for container csi-mockplugin-0/csi-provisioner ====