How do you manage Kubernetes cluster upgrades on Amazon EKS?

Upgrading a Kubernetes cluster on Amazon EKS (Elastic Kubernetes Service) involves updating both the control plane and the worker nodes to a new Kubernetes version. Managing upgrades carefully is crucial to ensuring minimal downtime and maintaining the stability of your applications. Here's a detailed guide on the process and best practices for upgrading an EKS cluster:

Step-by-Step Process for Upgrading an EKS Cluster

1. Preparation and Planning

Review Release Notes: Check the Kubernetes release notes and EKS documentation for the target version to understand new features, deprecations, and changes that might impact your workloads.
Check Compatibility: Ensure that your add-ons, tools (e.g., Helm, kubectl), and custom resources are compatible with the new Kubernetes version.
Back Up Cluster Data:
- Take a backup of your application data, including any stateful data stored in persistent volumes.
- Export current cluster configurations using kubectl get <resource> -o yaml > <resource>.yaml.

2. Upgrade the EKS Control Plane

Use AWS Management Console, AWS CLI, or eksctl:
- Using AWS Management Console:
  1. Go to the Amazon EKS console.
  2. Select your cluster and click on "Update now" in the Kubernetes version section.
  3. Choose the target Kubernetes version and start the update process.
- Using AWS CLI:
```
  aws eks update-cluster-version --name <cluster-name> --kubernetes-version <new-version>
```
- Using eksctl:
```
  eksctl upgrade cluster --name <cluster-name> --version <new-version>
```
Verify the Control Plane Upgrade:
- Confirm that the control plane is upgraded by checking the cluster version:
```
  kubectl get nodes
```

3. Upgrade Managed Node Groups or Self-Managed Nodes

Upgrade Managed Node Groups:
1. Update Node Group Version:
  - In the EKS console, navigate to the "Compute" tab, select your managed node group, and click "Update version."
  - Choose the Kubernetes version that matches your control plane.
2. Rolling Update:
  - EKS performs a rolling update, creating new nodes with the updated version and draining old nodes one by one.
  - Verify that new nodes are ready and joined the cluster before old nodes are terminated.
Upgrade Self-Managed Nodes:
1. Launch New AMIs:
  - Use the latest EKS optimized AMI for the target Kubernetes version.
  - Update your autoscaling group to use the new AMI ID and set the desired capacity to a higher number to start creating new nodes.
2. Drain Old Nodes:
  - Once new nodes are available, gracefully drain old nodes to ensure that workloads are shifted properly:
```
  kubectl drain <old-node-name> --ignore-daemonsets --delete-emptydir-data
```
3. Terminate Old Nodes:
  - After draining, you can terminate the old nodes manually or reduce the autoscaling group's desired capacity.

4. Upgrade Add-ons and Custom Resources

Upgrade EKS Add-ons:

Upgrade EKS add-ons like CoreDNS, kube-proxy, and the AWS VPC CNI plugin to versions compatible with the new Kubernetes version.

Use AWS CLI or eksctl to update add-ons:

  eksctl utils update-kube-proxy --cluster <cluster-name> --approve
  eksctl utils update-coredns --cluster <cluster-name> --approve
  eksctl utils update-aws-node --cluster <cluster-name> --approve

Upgrade Custom Add-ons:
- Update custom add-ons like Prometheus, Grafana, or Ingress controllers using Helm or kubectl to ensure compatibility.

5. Test and Validate

Run Application Tests:
- Deploy test workloads or run smoke tests to validate that applications are functioning correctly after the upgrade.
Monitor Logs and Metrics:
- Use monitoring tools (e.g., CloudWatch, Prometheus) to check for any errors or performance issues during and after the upgrade.

6. Roll Back if Necessary

Revert Control Plane:
- If there are major issues, consider rolling back by restoring from backups or re-creating the cluster with the previous Kubernetes version.
Revert Node Groups:
- For managed node groups, downgrade the version or roll back to a previous AMI version for self-managed nodes.

Best Practices for Minimal Downtime

Use Managed Node Groups: Leveraging managed node groups simplifies the upgrade process with automated rolling updates.
Perform Upgrades During Maintenance Windows: Schedule upgrades during low-traffic periods to minimize the impact on production workloads.
Use Blue-Green or Canary Deployments: For critical services, consider using Blue-Green or Canary deployments to test the upgraded environment before fully committing.
Monitor Health and Roll Back Quickly if Needed: Continuously monitor the health of the cluster and be prepared to roll back if there are critical issues.

You can upgrade your EKS cluster with minimal downtime, ensuring a smooth transition to the new Kubernetes version.

Reference:

https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html