==== start of pod log for container csi-mockplugin-attacher-0/csi-attacher ==== I0525 11:57:48.402071 1 main.go:91] Version: v2.2.0-0-g97411fa7 I0525 11:57:48.403598 1 connection.go:153] Connecting to unix:///csi/csi.sock I0525 11:57:49.405411 1 common.go:111] Probing CSI driver for readiness I0525 11:57:49.405463 1 connection.go:182] GRPC call: /csi.v1.Identity/Probe I0525 11:57:49.405474 1 connection.go:183] GRPC request: {} I0525 11:57:49.414898 1 connection.go:185] GRPC response: {"ready":{"value":true}} I0525 11:57:49.416118 1 connection.go:186] GRPC error: I0525 11:57:49.416212 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo I0525 11:57:49.416224 1 connection.go:183] GRPC request: {} I0525 11:57:49.417647 1 connection.go:185] GRPC response: {"manifest":{"url":"https://github.com/kubernetes-csi/csi-test/mock"},"name":"csi-mock-csi-mock-volumes-6755","vendor_version":"0.3.0"} I0525 11:57:49.418563 1 connection.go:186] GRPC error: I0525 11:57:49.418580 1 main.go:136] CSI driver name: "csi-mock-csi-mock-volumes-6755" W0525 11:57:49.418603 1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified. I0525 11:57:49.418614 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities I0525 11:57:49.418623 1 connection.go:183] GRPC request: {} I0525 11:57:49.420264 1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}}]} I0525 11:57:49.423292 1 connection.go:186] GRPC error: I0525 11:57:49.423308 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0525 11:57:49.423317 1 connection.go:183] GRPC request: {} I0525 11:57:49.425882 1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":4}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":8}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":12}}},{"Type":{"Rpc":{"type":11}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":9}}}]} I0525 11:57:49.435007 1 connection.go:186] GRPC error: I0525 11:57:49.435107 1 main.go:162] CSI driver supports ControllerPublishUnpublish, using real CSI handler I0525 11:57:49.435119 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0525 11:57:49.435134 1 connection.go:183] GRPC request: {} I0525 11:57:49.442201 1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":4}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":8}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":12}}},{"Type":{"Rpc":{"type":11}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":9}}}]} I0525 11:57:49.450829 1 connection.go:186] GRPC error: I0525 11:57:49.450860 1 main.go:175] CSI driver supports list volumes published nodes. Using capability to reconcile volume attachment objects with actual backend state I0525 11:57:49.451049 1 controller.go:121] Starting CSI attacher I0525 11:57:49.451391 1 reflector.go:153] Starting reflector *v1.PersistentVolume (10m0s) from k8s.io/client-go/informers/factory.go:135 I0525 11:57:49.451424 1 reflector.go:188] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:135 I0525 11:57:49.451436 1 reflector.go:153] Starting reflector *v1beta1.CSINode (10m0s) from k8s.io/client-go/informers/factory.go:135 I0525 11:57:49.451456 1 reflector.go:188] Listing and watching *v1beta1.CSINode from k8s.io/client-go/informers/factory.go:135 I0525 11:57:49.451473 1 reflector.go:153] Starting reflector *v1beta1.VolumeAttachment (10m0s) from k8s.io/client-go/informers/factory.go:135 I0525 11:57:49.451493 1 reflector.go:188] Listing and watching *v1beta1.VolumeAttachment from k8s.io/client-go/informers/factory.go:135 I0525 11:57:49.551350 1 shared_informer.go:227] caches populated I0525 11:57:49.551443 1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state I0525 11:57:49.551465 1 controller.go:198] Started VA processing "csi-9be1a89999d615993da7da4bb4ae63e1453e5cfb34c514e44f140a682958ed5d" I0525 11:57:49.551486 1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes I0525 11:57:49.551493 1 controller.go:213] Skipping VolumeAttachment csi-9be1a89999d615993da7da4bb4ae63e1453e5cfb34c514e44f140a682958ed5d for attacher csi-mock-csi-mock-volumes-6581 I0525 11:57:49.551500 1 connection.go:183] GRPC request: {} I0525 11:57:49.553977 1 connection.go:185] GRPC response: {"entries":[{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 1"},"volume_id":"1"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 2"},"volume_id":"2"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 3"},"volume_id":"3"}}]} I0525 11:57:49.562143 1 connection.go:186] GRPC error: W0525 11:57:49.562181 1 csi_handler.go:159] Failed to repair volume handle for driver csi-mock-csi-mock-volumes-6581: could not find In-Tree driver name for CSI plugin csi-mock-csi-mock-volumes-6581 I0525 11:57:58.477910 1 controller.go:198] Started VA processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.477946 1 csi_handler.go:209] CSIHandler: processing VA "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.477960 1 csi_handler.go:236] Attaching "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.477974 1 csi_handler.go:396] Starting attach operation for "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.478064 1 csi_handler.go:335] Adding finalizer to PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 11:57:58.489764 1 csi_handler.go:344] PV finalizer added to "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 11:57:58.489809 1 csi_handler.go:689] Found NodeID csi-mock-csi-mock-volumes-6755 in CSINode v1.21-worker2 I0525 11:57:58.489894 1 csi_handler.go:297] VA finalizer added to "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.489919 1 csi_handler.go:311] NodeID annotation added to "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.495313 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0525 11:57:58.495339 1 connection.go:183] GRPC request: {"node_id":"csi-mock-csi-mock-volumes-6755","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"name":"pvc-48803b22-1f1c-4980-84b6-1de5139d6f33","storage.kubernetes.io/csiProvisionerIdentity":"1621943869419-8081-csi-mock-csi-mock-volumes-6755"},"volume_id":"4"} I0525 11:57:58.500306 1 connection.go:185] GRPC response: {"publish_context":{"device":"/dev/mock","readonly":"false"}} I0525 11:57:58.503079 1 connection.go:186] GRPC error: I0525 11:57:58.503104 1 csi_handler.go:249] Attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.503123 1 util.go:35] Marking as attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508683 1 util.go:48] Marked as attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508719 1 csi_handler.go:255] Fully attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508734 1 csi_handler.go:225] CSIHandler: finished processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508786 1 controller.go:198] Started VA processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508851 1 csi_handler.go:209] CSIHandler: processing VA "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508885 1 csi_handler.go:231] "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" is already attached I0525 11:57:58.508903 1 csi_handler.go:225] CSIHandler: finished processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508959 1 controller.go:198] Started VA processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508981 1 csi_handler.go:209] CSIHandler: processing VA "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:57:58.508996 1 csi_handler.go:231] "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" is already attached I0525 11:57:58.509008 1 csi_handler.go:225] CSIHandler: finished processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:02.811773 1 controller.go:198] Started VA processing "csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302" I0525 11:58:02.811820 1 controller.go:213] Skipping VolumeAttachment csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302 for attacher csi-mock-csi-mock-volumes-4911 I0525 11:58:02.828740 1 controller.go:198] Started VA processing "csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302" I0525 11:58:02.828770 1 controller.go:213] Skipping VolumeAttachment csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302 for attacher csi-mock-csi-mock-volumes-4911 I0525 11:58:02.839372 1 controller.go:198] Started VA processing "csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302" I0525 11:58:02.839407 1 controller.go:213] Skipping VolumeAttachment csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302 for attacher csi-mock-csi-mock-volumes-4911 I0525 11:58:06.548577 1 controller.go:198] Started VA processing "csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16" I0525 11:58:06.548615 1 controller.go:213] Skipping VolumeAttachment csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16 for attacher csi-mock-csi-mock-volumes-519 I0525 11:58:06.562075 1 controller.go:198] Started VA processing "csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16" I0525 11:58:06.562106 1 controller.go:213] Skipping VolumeAttachment csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16 for attacher csi-mock-csi-mock-volumes-519 I0525 11:58:06.572928 1 controller.go:198] Started VA processing "csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16" I0525 11:58:06.572957 1 controller.go:213] Skipping VolumeAttachment csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16 for attacher csi-mock-csi-mock-volumes-519 I0525 11:58:11.492363 1 controller.go:198] Started VA processing "csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565" I0525 11:58:11.492404 1 controller.go:213] Skipping VolumeAttachment csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565 for attacher csi-mock-csi-mock-volumes-9550 I0525 11:58:11.505562 1 controller.go:198] Started VA processing "csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565" I0525 11:58:11.505611 1 controller.go:213] Skipping VolumeAttachment csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565 for attacher csi-mock-csi-mock-volumes-9550 I0525 11:58:11.517215 1 controller.go:198] Started VA processing "csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565" I0525 11:58:11.517263 1 controller.go:213] Skipping VolumeAttachment csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565 for attacher csi-mock-csi-mock-volumes-9550 I0525 11:58:22.588804 1 controller.go:198] Started VA processing "csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16" I0525 11:58:22.588869 1 controller.go:213] Skipping VolumeAttachment csi-9c4700d9cf306c833c179bde43f5e108357862ce4922bd0947ec5b419d44ac16 for attacher csi-mock-csi-mock-volumes-519 I0525 11:58:22.598475 1 controller.go:198] Started VA processing "csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565" I0525 11:58:22.598514 1 controller.go:213] Skipping VolumeAttachment csi-a2bdc6ae94dcbd2d1119495f8c5d9808d26e767ca06fd3e8772750af7ba79565 for attacher csi-mock-csi-mock-volumes-9550 I0525 11:58:22.601705 1 controller.go:232] Started PV processing "pvc-83650bc8-06ba-4849-bcff-52f7a584b4d1" I0525 11:58:22.601762 1 csi_handler.go:591] CSIHandler: processing PV "pvc-83650bc8-06ba-4849-bcff-52f7a584b4d1" I0525 11:58:22.601778 1 csi_handler.go:595] CSIHandler: processing PV "pvc-83650bc8-06ba-4849-bcff-52f7a584b4d1": no deletion timestamp, ignoring I0525 11:58:22.607353 1 controller.go:198] Started VA processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.607391 1 csi_handler.go:209] CSIHandler: processing VA "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.607405 1 csi_handler.go:260] Starting detach operation for "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.607468 1 csi_handler.go:267] Detaching "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.607554 1 csi_handler.go:689] Found NodeID csi-mock-csi-mock-volumes-6755 in CSINode v1.21-worker2 I0525 11:58:22.607629 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0525 11:58:22.607647 1 connection.go:183] GRPC request: {"node_id":"csi-mock-csi-mock-volumes-6755","volume_id":"4"} I0525 11:58:22.610014 1 connection.go:185] GRPC response: {} I0525 11:58:22.611143 1 connection.go:186] GRPC error: I0525 11:58:22.611157 1 csi_handler.go:549] Detached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.611219 1 controller.go:232] Started PV processing "pvc-de4d3b79-92bf-4748-9457-1219c0232f8e" I0525 11:58:22.611304 1 csi_handler.go:591] CSIHandler: processing PV "pvc-de4d3b79-92bf-4748-9457-1219c0232f8e" I0525 11:58:22.611344 1 csi_handler.go:595] CSIHandler: processing PV "pvc-de4d3b79-92bf-4748-9457-1219c0232f8e": no deletion timestamp, ignoring I0525 11:58:22.611221 1 util.go:76] Marking as detached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.621025 1 controller.go:232] Started PV processing "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 11:58:22.621072 1 csi_handler.go:591] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 11:58:22.621096 1 csi_handler.go:595] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33": no deletion timestamp, ignoring I0525 11:58:22.621351 1 util.go:90] Finalizer removed from "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.621402 1 csi_handler.go:280] Fully detached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:22.621436 1 csi_handler.go:225] CSIHandler: finished processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.531835 1 controller.go:198] Started VA processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.531871 1 csi_handler.go:209] CSIHandler: processing VA "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.531882 1 csi_handler.go:236] Attaching "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.531894 1 csi_handler.go:396] Starting attach operation for "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.532004 1 csi_handler.go:329] PV finalizer is already set on "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 11:58:26.532057 1 csi_handler.go:689] Found NodeID csi-mock-csi-mock-volumes-6755 in CSINode v1.21-worker2 I0525 11:58:26.532099 1 csi_handler.go:297] VA finalizer added to "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.532117 1 csi_handler.go:311] NodeID annotation added to "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.538530 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerPublishVolume I0525 11:58:26.538565 1 connection.go:183] GRPC request: {"node_id":"csi-mock-csi-mock-volumes-6755","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"name":"pvc-48803b22-1f1c-4980-84b6-1de5139d6f33","storage.kubernetes.io/csiProvisionerIdentity":"1621943869419-8081-csi-mock-csi-mock-volumes-6755"},"volume_id":"4"} I0525 11:58:26.542795 1 connection.go:185] GRPC response: {"publish_context":{"device":"/dev/mock","readonly":"false"}} I0525 11:58:26.543538 1 connection.go:186] GRPC error: I0525 11:58:26.543549 1 csi_handler.go:249] Attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.543560 1 util.go:35] Marking as attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.548758 1 util.go:48] Marked as attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.548782 1 csi_handler.go:255] Fully attached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.548793 1 csi_handler.go:225] CSIHandler: finished processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.548825 1 controller.go:198] Started VA processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.548836 1 csi_handler.go:209] CSIHandler: processing VA "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:26.548849 1 csi_handler.go:231] "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" is already attached I0525 11:58:26.548859 1 csi_handler.go:225] CSIHandler: finished processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 11:58:32.594565 1 controller.go:198] Started VA processing "csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302" I0525 11:58:32.594627 1 controller.go:213] Skipping VolumeAttachment csi-a9943b9f361ed62c6fbf90d86ae96df60c0c67325c0009bf9a55c27085554302 for attacher csi-mock-csi-mock-volumes-4911 I0525 11:58:32.606887 1 controller.go:232] Started PV processing "pvc-26e64179-c99e-4ee0-ba37-51ac9288c0bf" I0525 11:58:32.606936 1 csi_handler.go:591] CSIHandler: processing PV "pvc-26e64179-c99e-4ee0-ba37-51ac9288c0bf" I0525 11:58:32.606956 1 csi_handler.go:595] CSIHandler: processing PV "pvc-26e64179-c99e-4ee0-ba37-51ac9288c0bf": no deletion timestamp, ignoring I0525 11:58:36.813526 1 controller.go:198] Started VA processing "csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c" I0525 11:58:36.813567 1 controller.go:213] Skipping VolumeAttachment csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c for attacher csi-mock-csi-mock-volumes-2982 I0525 11:58:36.831866 1 controller.go:198] Started VA processing "csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c" I0525 11:58:36.831934 1 controller.go:213] Skipping VolumeAttachment csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c for attacher csi-mock-csi-mock-volumes-2982 I0525 11:58:36.843778 1 controller.go:198] Started VA processing "csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c" I0525 11:58:36.843812 1 controller.go:213] Skipping VolumeAttachment csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c for attacher csi-mock-csi-mock-volumes-2982 I0525 11:58:49.562417 1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state I0525 11:58:49.562499 1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes I0525 11:58:49.562511 1 connection.go:183] GRPC request: {} I0525 11:58:49.564908 1 connection.go:185] GRPC response: {"entries":[{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 1"},"volume_id":"1"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 2"},"volume_id":"2"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 3"},"volume_id":"3"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":6442450944,"volume_context":{"csi-mock-csi-mock-volumes-6755/dev":"/dev/mock","csi-mock-csi-mock-volumes-6755/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/globalmount":"/dev/mock","csi-mock-csi-mock-volumes-6755/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/globalmount/size":"6442450944","csi-mock-csi-mock-volumes-6755/var/lib/kubelet/pods/5900517b-a1f2-4754-8400-26e7d6c16542/volumes/kubernetes.io~csi/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/mount":"/var/lib/kubelet/pods/5900517b-a1f2-4754-8400-26e7d6c16542/volumes/kubernetes.io~csi/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/mount","name":"pvc-48803b22-1f1c-4980-84b6-1de5139d6f33","readonly":"false"},"volume_id":"4"}}]} I0525 11:58:49.575779 1 connection.go:186] GRPC error: W0525 11:58:49.575820 1 csi_handler.go:159] Failed to repair volume handle for driver csi-mock-csi-mock-volumes-6581: could not find In-Tree driver name for CSI plugin csi-mock-csi-mock-volumes-6581 W0525 11:58:49.575848 1 csi_handler.go:159] Failed to repair volume handle for driver csi-mock-csi-mock-volumes-6755: could not find In-Tree driver name for CSI plugin csi-mock-csi-mock-volumes-6755 W0525 11:58:49.575862 1 csi_handler.go:159] Failed to repair volume handle for driver csi-mock-csi-mock-volumes-2982: could not find In-Tree driver name for CSI plugin csi-mock-csi-mock-volumes-2982 I0525 11:59:02.896005 1 controller.go:198] Started VA processing "csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c" I0525 11:59:02.896048 1 controller.go:213] Skipping VolumeAttachment csi-3136da6543b3febae852b37caad09dde07a8c2121633bb3930c6e186619e254c for attacher csi-mock-csi-mock-volumes-2982 I0525 11:59:02.909725 1 controller.go:232] Started PV processing "pvc-9bbe0f07-a69c-4365-a7a5-b47244a95bfa" I0525 11:59:02.909757 1 csi_handler.go:591] CSIHandler: processing PV "pvc-9bbe0f07-a69c-4365-a7a5-b47244a95bfa" I0525 11:59:02.909770 1 csi_handler.go:595] CSIHandler: processing PV "pvc-9bbe0f07-a69c-4365-a7a5-b47244a95bfa": no deletion timestamp, ignoring I0525 11:59:08.720478 1 controller.go:198] Started VA processing "csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847" I0525 11:59:08.720536 1 controller.go:213] Skipping VolumeAttachment csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847 for attacher csi-mock-csi-mock-volumes-844 I0525 11:59:08.738341 1 controller.go:198] Started VA processing "csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847" I0525 11:59:08.738378 1 controller.go:213] Skipping VolumeAttachment csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847 for attacher csi-mock-csi-mock-volumes-844 I0525 11:59:08.750439 1 controller.go:198] Started VA processing "csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847" I0525 11:59:08.750491 1 controller.go:213] Skipping VolumeAttachment csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847 for attacher csi-mock-csi-mock-volumes-844 I0525 11:59:32.900004 1 controller.go:198] Started VA processing "csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847" I0525 11:59:32.900049 1 controller.go:213] Skipping VolumeAttachment csi-d7b588a91c3ea00299e475f306105561217058b15b02c0dcbb8552f05f5e0847 for attacher csi-mock-csi-mock-volumes-844 I0525 11:59:32.913135 1 controller.go:232] Started PV processing "pvc-40448b9e-4081-4224-bf86-4bf315c9dc1b" I0525 11:59:32.913181 1 csi_handler.go:591] CSIHandler: processing PV "pvc-40448b9e-4081-4224-bf86-4bf315c9dc1b" I0525 11:59:32.913208 1 csi_handler.go:595] CSIHandler: processing PV "pvc-40448b9e-4081-4224-bf86-4bf315c9dc1b": no deletion timestamp, ignoring I0525 11:59:33.701869 1 controller.go:198] Started VA processing "csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f" I0525 11:59:33.701913 1 controller.go:213] Skipping VolumeAttachment csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f for attacher csi-mock-csi-mock-volumes-8008 I0525 11:59:33.716374 1 controller.go:198] Started VA processing "csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f" I0525 11:59:33.716402 1 controller.go:213] Skipping VolumeAttachment csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f for attacher csi-mock-csi-mock-volumes-8008 I0525 11:59:33.738502 1 controller.go:198] Started VA processing "csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f" I0525 11:59:33.738546 1 controller.go:213] Skipping VolumeAttachment csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f for attacher csi-mock-csi-mock-volumes-8008 I0525 11:59:38.138752 1 controller.go:198] Started VA processing "csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee" I0525 11:59:38.138786 1 controller.go:213] Skipping VolumeAttachment csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee for attacher csi-mock-csi-mock-volumes-2353 I0525 11:59:38.148739 1 controller.go:198] Started VA processing "csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee" I0525 11:59:38.148770 1 controller.go:213] Skipping VolumeAttachment csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee for attacher csi-mock-csi-mock-volumes-2353 I0525 11:59:38.156552 1 controller.go:198] Started VA processing "csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee" I0525 11:59:38.156576 1 controller.go:213] Skipping VolumeAttachment csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee for attacher csi-mock-csi-mock-volumes-2353 I0525 11:59:42.929119 1 controller.go:198] Started VA processing "csi-9be1a89999d615993da7da4bb4ae63e1453e5cfb34c514e44f140a682958ed5d" I0525 11:59:42.929148 1 controller.go:213] Skipping VolumeAttachment csi-9be1a89999d615993da7da4bb4ae63e1453e5cfb34c514e44f140a682958ed5d for attacher csi-mock-csi-mock-volumes-6581 I0525 11:59:42.943996 1 controller.go:232] Started PV processing "pvc-6d7ced8c-15ee-46db-90fa-92e5f1e10d96" I0525 11:59:42.944046 1 csi_handler.go:591] CSIHandler: processing PV "pvc-6d7ced8c-15ee-46db-90fa-92e5f1e10d96" I0525 11:59:42.944063 1 csi_handler.go:595] CSIHandler: processing PV "pvc-6d7ced8c-15ee-46db-90fa-92e5f1e10d96": no deletion timestamp, ignoring I0525 11:59:49.576127 1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state I0525 11:59:49.576261 1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes I0525 11:59:49.576274 1 connection.go:183] GRPC request: {} I0525 11:59:49.579596 1 connection.go:185] GRPC response: {"entries":[{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 1"},"volume_id":"1"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 2"},"volume_id":"2"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":107374182400,"volume_context":{"name":"Mock Volume 3"},"volume_id":"3"}},{"status":{"published_node_ids":["csi-mock-csi-mock-volumes-6755"]},"volume":{"capacity_bytes":6442450944,"volume_context":{"csi-mock-csi-mock-volumes-6755/dev":"/dev/mock","csi-mock-csi-mock-volumes-6755/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/globalmount":"/dev/mock","csi-mock-csi-mock-volumes-6755/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/globalmount/size":"6442450944","csi-mock-csi-mock-volumes-6755/var/lib/kubelet/pods/5900517b-a1f2-4754-8400-26e7d6c16542/volumes/kubernetes.io~csi/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/mount":"/var/lib/kubelet/pods/5900517b-a1f2-4754-8400-26e7d6c16542/volumes/kubernetes.io~csi/pvc-48803b22-1f1c-4980-84b6-1de5139d6f33/mount","name":"pvc-48803b22-1f1c-4980-84b6-1de5139d6f33","readonly":"false"},"volume_id":"4"}}]} I0525 11:59:49.592641 1 connection.go:186] GRPC error: W0525 11:59:49.592703 1 csi_handler.go:159] Failed to repair volume handle for driver csi-mock-csi-mock-volumes-6755: could not find In-Tree driver name for CSI plugin csi-mock-csi-mock-volumes-6755 W0525 11:59:49.593002 1 csi_handler.go:159] Failed to repair volume handle for driver csi-mock-csi-mock-volumes-8008: could not find In-Tree driver name for CSI plugin csi-mock-csi-mock-volumes-8008 W0525 11:59:49.593028 1 csi_handler.go:159] Failed to repair volume handle for driver csi-mock-csi-mock-volumes-2353: could not find In-Tree driver name for CSI plugin csi-mock-csi-mock-volumes-2353 I0525 11:59:53.107986 1 controller.go:198] Started VA processing "csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f" I0525 11:59:53.108020 1 controller.go:213] Skipping VolumeAttachment csi-21f3aae2a77dc10ce88f93c8a3852b1a9968ffc003dad9cf4a3dbdb8000bc40f for attacher csi-mock-csi-mock-volumes-8008 I0525 11:59:53.120948 1 controller.go:232] Started PV processing "pvc-9488e0f5-94d1-49ae-83d8-863749bb99a9" I0525 11:59:53.120988 1 csi_handler.go:591] CSIHandler: processing PV "pvc-9488e0f5-94d1-49ae-83d8-863749bb99a9" I0525 11:59:53.120999 1 csi_handler.go:595] CSIHandler: processing PV "pvc-9488e0f5-94d1-49ae-83d8-863749bb99a9": no deletion timestamp, ignoring I0525 11:59:59.236778 1 controller.go:198] Started VA processing "csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b" I0525 11:59:59.236818 1 controller.go:213] Skipping VolumeAttachment csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b for attacher csi-mock-csi-mock-volumes-8210 I0525 11:59:59.252821 1 controller.go:198] Started VA processing "csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b" I0525 11:59:59.252862 1 controller.go:213] Skipping VolumeAttachment csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b for attacher csi-mock-csi-mock-volumes-8210 I0525 11:59:59.262804 1 controller.go:198] Started VA processing "csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b" I0525 11:59:59.262839 1 controller.go:213] Skipping VolumeAttachment csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b for attacher csi-mock-csi-mock-volumes-8210 I0525 12:00:03.086949 1 controller.go:198] Started VA processing "csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee" I0525 12:00:03.087026 1 controller.go:213] Skipping VolumeAttachment csi-c6873c47915015c92b08c69a121c19f2aad63eece8cd7b4f31fecef1614e2dee for attacher csi-mock-csi-mock-volumes-2353 I0525 12:00:03.100261 1 controller.go:232] Started PV processing "pvc-81a1cdbe-ba92-4948-bfe9-a8a3be06bb3e" I0525 12:00:03.100287 1 csi_handler.go:591] CSIHandler: processing PV "pvc-81a1cdbe-ba92-4948-bfe9-a8a3be06bb3e" I0525 12:00:03.100299 1 csi_handler.go:595] CSIHandler: processing PV "pvc-81a1cdbe-ba92-4948-bfe9-a8a3be06bb3e": no deletion timestamp, ignoring I0525 12:00:13.057869 1 controller.go:198] Started VA processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.057947 1 csi_handler.go:209] CSIHandler: processing VA "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.057961 1 csi_handler.go:260] Starting detach operation for "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.058049 1 csi_handler.go:267] Detaching "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.058082 1 csi_handler.go:689] Found NodeID csi-mock-csi-mock-volumes-6755 in CSINode v1.21-worker2 I0525 12:00:13.058173 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume I0525 12:00:13.058196 1 connection.go:183] GRPC request: {"node_id":"csi-mock-csi-mock-volumes-6755","volume_id":"4"} I0525 12:00:13.059923 1 connection.go:185] GRPC response: {} I0525 12:00:13.060873 1 connection.go:186] GRPC error: I0525 12:00:13.060893 1 csi_handler.go:549] Detached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.061040 1 util.go:76] Marking as detached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.068454 1 controller.go:198] Started VA processing "csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b" I0525 12:00:13.068488 1 controller.go:213] Skipping VolumeAttachment csi-1011171de0b01fb7fdf7d5f759d25637ed0143fd6d81e3727a19d5e915eb944b for attacher csi-mock-csi-mock-volumes-8210 I0525 12:00:13.070793 1 controller.go:232] Started PV processing "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:13.070830 1 csi_handler.go:591] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:13.070848 1 csi_handler.go:595] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33": no deletion timestamp, ignoring I0525 12:00:13.070875 1 util.go:90] Finalizer removed from "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.070923 1 csi_handler.go:280] Fully detached "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.070937 1 csi_handler.go:225] CSIHandler: finished processing "csi-f866cc7cad364f671f57deb7c32cddd3d86a84506df2a8df0d066b64bee70e12" I0525 12:00:13.081698 1 controller.go:232] Started PV processing "pvc-3f08190f-33d6-40b0-b90a-40e5249598a4" I0525 12:00:13.081735 1 csi_handler.go:591] CSIHandler: processing PV "pvc-3f08190f-33d6-40b0-b90a-40e5249598a4" I0525 12:00:13.081753 1 csi_handler.go:595] CSIHandler: processing PV "pvc-3f08190f-33d6-40b0-b90a-40e5249598a4": no deletion timestamp, ignoring I0525 12:00:16.544256 1 controller.go:232] Started PV processing "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:16.544303 1 csi_handler.go:591] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:16.544328 1 csi_handler.go:633] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33": no VA found, removing finalizer E0525 12:00:16.551635 1 csi_handler.go:650] Failed to remove finalizer from PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33": PersistentVolume "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{"kubernetes.io/pv-protection"} I0525 12:00:16.551701 1 controller.go:232] Started PV processing "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:16.551719 1 csi_handler.go:591] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:16.551751 1 csi_handler.go:633] CSIHandler: processing PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33": no VA found, removing finalizer I0525 12:00:16.559915 1 csi_handler.go:655] Removed finalizer from PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:17.551952 1 controller.go:232] Started PV processing "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" I0525 12:00:17.552006 1 controller.go:239] PV "pvc-48803b22-1f1c-4980-84b6-1de5139d6f33" deleted, ignoring ==== end of pod log for container csi-mockplugin-attacher-0/csi-attacher ====