本次发现的问题是踩到了新版本Kubernetes的坑，查找资料发现zhangsi-lzq大佬已经分析的很清楚了，此处转载过来仅做学习记录，并新增永久解决方案。

作者：zhangsi-lzq

本文出处：https://www.cnblogs.com/zhangsi-lzq/p/14292628.html

问题背景

Kubernetes升级为1.20版本后，原有的后端nfs存储storageclass无法自动创建PV。查看PVC状态一直为pending状态。

分析过程

查看nfs-provisioner日志：

kubectl logs nfs-client-provisioner-5f696dc8bb-qhmsnnE0118 03:01:07.352670 1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=3737, ErrCode=NO_ERROR, debug=""nE0118 03:01:07.353951 1 reflector.go:322] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:668: Failed to watch *v1.StorageClass: Get https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=604432&timeoutSeconds=387&watch=true: dial tcp 10.96.0.1:443: connect: connection refusednW0118 03:01:07.366606 1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:665: watch of *v1.PersistentVolume ended with: too old resource version: 11565 (605173)nW0118 03:01:07.367679 1 reflector.go:341] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:662: watch of *v1.PersistentVolumeClaim ended with: too old resource version: 11565 (605173)nI0118 03:08:28.340240 1 controller.go:987] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": startednE0118 03:08:28.343582 1 controller.go:1004] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make referencenI0118 03:16:08.373590 1 controller.go:987] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": startednE0118 03:16:08.382178 1 controller.go:1004] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make referencenI0118 03:30:41.647626 1 controller.go:987] provision "default/test-pvc" class "course-nfs-storage": startednE0118 03:30:41.658419 1 controller.go:1004] provision "default/test-pvc" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make referencenI0118 03:31:08.373713 1 controller.go:987] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": startednI0118 03:31:08.373969 1 controller.go:987] provision "default/test-pvc" class "course-nfs-storage": startednE0118 03:31:08.382279 1 controller.go:1004] provision "default/test-pvc" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make referencenE0118 03:31:08.382791 1 controller.go:1004] provision "default/auth-platorm-redis-data-auth-platorm-redis-cluster-0" class "course-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

报错信息selfLink was empty，于是上网查询相关内容，在官方1.20的变更说明中[1]看到其中一条说明为：

Stop propagating SelfLink (deprecated in 1.16) in kube-apiserver

selfLink在1.16版本以后已经弃用，在1.20版本停用。而由于nfs-provisioner的实现是基于selfLink功能（同时也会影响其他用到selfLink这个功能的第三方软件），需要等nfs-provisioner的制作方重新提供新的解决方案。

解决方案

目前可用的临时方案是：修改/etc/kubernetes/manifests/kube-apiserver.yaml文件，找到如下内容后，在最后添加一项参数：

spec:n containers:n - command:n - kube-apiserver n - --advertise-address=192.168.210.20 n - --....... n - --feature-gates=RemoveSelfLink=false #添加此行

如果是高可用的k8s集群，则需要在所有master节点上进行此操作。

添加后需要删除apiserver的所有pod进行重启

kubectl delete pod kube-apiserver-master01 -n kube-systemnkubectl delete pod kube-apiserver-master02 -n kube-systemnkubectl delete pod kube-apiserver-master03 -n kube-system

三台apiserver被kubelet重启拉起后，再次查询PVC，可以看到PVC状态都为Bound，可以正常被PV绑定了。

（新增）永久解决方案：查资料发现，nfs-provisioner项目实际上已经放弃[2,3]，转而提供了下面3个provisioner，可以根据需要适配更新：

nfs-subdir-external-provisioner

nfs-ganesha-server-and-external-provisioner

external-provisioner

参考资料

1. https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md
2. https://github.com/kubernetes-retired/nfs-provisioner
3. https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner