kops 升级 kubernetes cluster

kops 升级 kubernetes cluster

升级(update)

首先准备一些环境变量

export AWS_ACCESS_KEY_ID=AKIADFJIAFAIFJAISFWRCexport AWS_SECRET_ACCESS_KEY=YlIAJSDIFASF2REIJASDhENexport AWS_REGION=cn-northwest-1export NAME=mycluster.k8s.localexport KOPS_STATE_STORE=s3://my-k8s-cluster-bucketexport ASSET_BUCKET=my-kops-assetsexport KUBERNETES_VERSION=v1.15.5

然后更新集群

kops update cluster --name $NAME --yes[ec2-user@ip-172-24-1-133 .bin]$ kops update cluster --name $NAME --yes*********************************************************************************A new kubernetes version is available: 1.15.10Upgrading is recommended (try kops upgrade cluster)More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_k8s.md#1.15.10*********************************************************************************I0227 18:05:26.203484 5824 apply_cluster.go:556] Gossip DNS: skipping DNS validationW0227 18:05:26.290994 5824 firewall.go:250] Opening etcd port on masters for access from the nodes, for calico. This is unsafe in untrusted environments.I0227 18:05:26.655305 5824 executor.go:103] Tasks: 0 done 103 total; 48 can runI0227 18:05:26.848420 5824 executor.go:103] Tasks: 48 done 103 total; 26 can runI0227 18:05:27.013770 5824 executor.go:103] Tasks: 74 done 103 total; 21 can runI0227 18:05:27.464892 5824 executor.go:103] Tasks: 95 done 103 total; 5 can runI0227 18:05:27.617109 5824 executor.go:103] Tasks: 100 done 103 total; 3 can runI0227 18:05:27.716468 5824 executor.go:103] Tasks: 103 done 103 total; 0 can runI0227 18:05:28.077045 5824 update_cluster.go:294] Exporting kubecfg for clusterkops has set your kubectl context to mycluster.k8s.localCluster changes have been applied to the cloud.Changes may require instances to restart: kops rolling-update cluster

从输出可见，当前版本是 1.15.5，发现新版本 1.15.10。

下面进行到 1.15.10 版本的升级。

滚动更新(rolling update)

升级集群版本 kubectl rolling update cluster --yes。

[ec2-user@ip-172-24-1-133 .bin]$ kops rolling-update cluster --yesUsing cluster from kubectl context: mycluster.k8s.localNAME STATUS NEEDUPDATE READY MIN MAX NODESmaster-cn-northwest-1a NeedsUpdate 1 0 1 1 1master-cn-northwest-1b NeedsUpdate 1 0 1 1 1master-cn-northwest-1c NeedsUpdate 1 0 1 1 1nodes NeedsUpdate 2 0 2 2 2I0227 18:13:39.414923 6106 instancegroups.go:163] Draining the node: "ip-172-24-1-191.cn-northwest-1.compute.internal".WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-nsrg4evicting pod "calico-kube-controllers-54c96b97b9-tzjzm"I0227 18:13:43.513293 6106 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.I0227 18:13:48.513499 6106 instancegroups.go:183] deleting node "ip-172-24-1-191.cn-northwest-1.compute.internal" from kubernetesI0227 18:13:48.527485 6106 instancegroups.go:301] Stopping instance "i-0044d5656b22ee6d", node "ip-172-24-1-191.cn-northwest-1.compute.internal", in group "master-cn-northwest-1a.masters.mycluster.k8s.local" (this may take a while).I0227 18:13:48.732988 6106 instancegroups.go:196] waiting for 15s after terminating instanceI0227 18:14:03.733259 6106 instancegroups.go:207] Validating the cluster.W0227 18:14:03.985663 6106 aws_cloud.go:666] ignoring instance as it is terminating: i-0044d5656b22ee6d in autoscaling group: master-cn-northwest-1a.masters.mycluster.k8s.localI0227 18:14:04.028145 6106 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: InstanceGroup "master-cn-northwest-1a" did not have enough nodes 0 vs 1.W0227 18:14:34.246498 6106 aws_cloud.go:666] ignoring instance as it is terminating: i-0044d5656b22ee6d in autoscaling group: master-cn-northwest-1a.masters.mycluster.k8s.localI0227 18:14:34.285779 6106 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-095aaaa2323ccaaa" has not yet joined cluster.I0227 18:15:04.298013 6106 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: machine "i-095aaaa2323ccaaa" has not yet joined cluster.

升级过程先升级 mater nodes，再升级 worker nodes。在 3AZ 的集群，会依次升级 master a, master b, master c 三台 ec2 instance。

我们也可以 aws console 进行查看：

3AZ master nodes 升级完毕。

天有不测风云，三个 master nodes 升级完之后，当对 worker nodes 的升级过程时发现如下日志：

error when evicting pod "kube-dns-5fdb85bb5b-zlj8w" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.evicting pod "kube-dns-5fdb85bb5b-zlj8w"error when evicting pod "kube-dns-5fdb85bb5b-zlj8w" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.evicting pod "kube-dns-5fdb85bb5b-zlj8w"

这是什么原因呢？

kube-dns 无法启动导致升级不成功问题

kube-dns 是 kubernetes 的基础组件，升级过程会确保这些组件被 evict 到正常 node 才可以对 worker node 进行升级。

在通过 kops 升级集群的过程中，由于 kube-dns 无法重新 schedule 到新的机器，导致在运行 kops rolling-update cluster 的过程中持续输出这样的日志：

error when evicting pod "kube-dns-5fdb85bb5b-zlj8w" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.evicting pod "kube-dns-5fdb85bb5b-zlj8w"error when evicting pod "kube-dns-5fdb85bb5b-zlj8w" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.evicting pod "kube-dns-5fdb85bb5b-zlj8w"

通过 kubectl -n kube-system describe pod kube-dns-5fdb85bb5b-zlj8w  追查出问题的 pod 之后，发现可能是因资源不够 FailedScheduling 10s (x3 over 83s) default-scheduler 0/5 nodes are available: 1 Insufficient cpu, 1 node(s) were unschedulable, 3 node(s) had taints that the pod didn't tolerate 导致 kube-dns 无法被调度。

admin@ip-172-24-5-200:~$ k -n kube-system describe pod kube-dns-5fdb85bb5b-6str2Name: kube-dns-5fdb85bb5b-6str2Namespace: kube-systemPriority: 0Node: Labels: k8s-app=kube-dns pod-template-hash=5fdb85bb5bAnnotations: prometheus.io/port: 10055 prometheus.io/scrape: true scheduler.alpha.kubernetes.io/critical-pod:Status: PendingIP:Controlled By: ReplicaSet/kube-dns-5fdb85bb5bContainers: kubedns: Image: k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.13 Ports: 10053/UDP, 10053/TCP, 10055/TCP Host Ports: 0/UDP, 0/TCP, 0/TCP Args: --config-dir=/kube-dns-config --dns-port=10053 --domain=cluster.local. --v=2 Limits: memory: 170Mi Requests: cpu: 100m memory: 70Mi Liveness: http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5 Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3 Environment: PROMETHEUS_PORT: 10055 Mounts: kube-dns-config from kube-dns-config (rw) var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-45tn8 (ro) dnsmasq: Image: k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.13 Ports: 53/UDP, 53/TCP Host Ports: 0/UDP, 0/TCP Args: -v=2 -logtostderr -configDir=/etc/k8s/dns/dnsmasq-nanny -restartDnsmasq=true -- -k --cache-size=1000 --dns-forward-max=150 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/in6.arpa/127.0.0.1#10053 --min-port=1024 Requests: cpu: 150m memory: 20Mi Liveness: http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: Mounts: etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw) var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-45tn8 (ro) sidecar: Image: k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.13 Port: 10054/TCP Host Port: 0/TCP Args: --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A Requests: cpu: 10m memory: 20Mi Liveness: http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5 Environment: Mounts: var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-45tn8 (ro)Conditions: Type Status PodScheduled FalseVolumes: kube-dns-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: kube-dns Optional: true kube-dns-token-45tn8: Type: Secret (a volume populated by a Secret) SecretName: kube-dns-token-45tn8 Optional: falseQoS Class: BurstableNode-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300sEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 10s (x3 over 83s) default-scheduler 0/5 nodes are available: 1 Insufficient cpu, 1 node(s) were unschedulable, 3 node(s) had taints that the pod didn't tolerate.

解决方案

根据 describe 的输出提示，可以强制降低可被调度 worker node 的负载，或是在业务非高峰期降低 deployment replicas 的数量来解决。

kube-dns 被调度成功之后，upgrade 得以继续。

此时查看 upgrade 日志，可见旧版本 ec2 instance 正在被停止 Stopping instance

error when evicting pod "kube-dns-5fdb85bb5b-zlj8w" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.evicting pod "kube-dns-5fdb85bb5b-zlj8w"I0227 20:35:49.977319 7003 instancegroups.go:352] Waiting for 5s for pods to stabilize after draining.I0227 20:35:54.977615 7003 instancegroups.go:183] deleting node "ip-172-24-1-222.cn-northwest-1.compute.internal" from kubernetesI0227 20:35:54.990034 7003 instancegroups.go:301] Stopping instance "i-0f122233445566ac", node "ip-172-24-1-222.cn-northwest-1.compute.internal", in group "nodes.mycluster.k8s.local" (this may take a while).I0227 20:35:55.185653 7003 instancegroups.go:196] waiting for 15s after terminating instanceI0227 20:36:10.185967 7003 instancegroups.go:207] Validating the cluster.W0227 20:36:10.390480 7003 aws_cloud.go:666] ignoring instance as it is terminating: i-0f122233445566ac in autoscaling group: nodes.mycluster.k8s.localI0227 20:41:40.660688 7003 instancegroups.go:275] Cluster did not pass validation, will try again in "30s" until duration "15m0s" expires: kube-system pod "calico-node-67f6k" is pending.

此时再通过 kubectl get nodes 查看 nodes 状态，发现包括 master 和 worker 全部 nodes 升级到最新版本。此时对 cluster 的升级结束。

此时旧 worker node 依旧是旧版本 1.15.5，并且无法被调度 SchedulingDisabled。

admin@ip-172-24-1-111:~$ k get nodesNAME STATUS ROLES AGE VERSIONip-172-24-1-222.cn-northwest-1.compute.internal Ready,SchedulingDisabled node 51d v1.15.5ip-172-24-1-111.cn-northwest-1.compute.internal Ready master 134m v1.15.10ip-172-24-1-51.cn-northwest-1.compute.internal Ready node 105m v1.15.10ip-172-24-3-145.cn-northwest-1.compute.internal Ready master 124m v1.15.10ip-172-24-5-200.cn-northwest-1.compute.internal Ready master 115m v1.15.10

此时旧版 worker node 从集群中被摘下。

admin@ip-172-24-1-111:~$ k get nodesNAME STATUS ROLES AGE VERSIONip-172-24-1-111.cn-northwest-1.compute.internal Ready master 136m v1.15.10ip-172-24-1-51.cn-northwest-1.compute.internal Ready node 108m v1.15.10ip-172-24-3-145.cn-northwest-1.compute.internal Ready master 127m v1.15.10ip-172-24-5-200.cn-northwest-1.compute.internal Ready master 118m v1.15.10

此时旧版 worker node 成功升级到新版 1.15.10，但状态为 NotReady。

admin@ip-172-24-1-111:~$ k get nodesNAME STATUS ROLES AGE VERSIONip-172-24-1-113.cn-northwest-1.compute.internal NotReady node 16s v1.15.10ip-172-24-1-111.cn-northwest-1.compute.internal Ready master 139m v1.15.10ip-172-24-1-51.cn-northwest-1.compute.internal Ready node 110m v1.15.10ip-172-24-3-145.cn-northwest-1.compute.internal Ready master 129m v1.15.10ip-172-24-5-200.cn-northwest-1.compute.internal Ready master 120m v1.15.10

此时旧版 worker node 成功升级到新版 1.15.10，状态为 Ready。

admin@ip-172-24-1-111:~$ k get nodesNAME STATUS ROLES AGE VERSIONip-172-24-1-113.cn-northwest-1.compute.internal Ready node 111m v1.15.10ip-172-24-1-111.cn-northwest-1.compute.internal Ready master 4h10m v1.15.10ip-172-24-1-51.cn-northwest-1.compute.internal Ready node 3h41m v1.15.10ip-172-24-3-145.cn-northwest-1.compute.internal Ready master 4h v1.15.10ip-172-24-5-200.cn-northwest-1.compute.internal Ready master 3h51m v1.15.10