GCP GKE Auto Scaling
GCP GKE Auto Scaling
- Decrease number of replicas for a Deployment with Horizontal Pod Autoscaler
- Decrease CPU request of a Deployment with Vertical Pod Autoscaler
- Decrease number of nodes used in cluster with Cluster Autoscaler
- Automatically create an optimized node pool for workload with Node Auto Provisioning
- Test the autoscaling behavior against a spike in demand
- Overprovision your cluster with Pause Pods
Welcome to Cloud Shell! Type "help" to get started.
Your Cloud Platform project in this session is set to qwiklabs-gcp-01-696b4c0e98b8.
Use “gcloud config set project [PROJECT_ID]” to change to a different project.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ gcloud config set compute/zone us-west1-a
Updated property [compute/zone].
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ gcloud container clusters create scaling-demo --num-nodes=3 --enable-vertical-pod-autoscaling
Default change: VPC-native is the default mode during cluster creation for versions greater than 1.21.0-gke.1500. To create advanced routes based clusters, please pass the `--no-enable-ip-alias` flag
Note: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).
Creating cluster scaling-demo in us-west1-a... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/qwiklabs-gcp-01-696b4c0e98b8/zones/us-west1-a/clusters/scaling-demo].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1-a/scaling-demo?project=qwiklabs-gcp-01-696b4c0e98b8
kubeconfig entry generated for scaling-demo.
NAME: scaling-demo
LOCATION: us-west1-a
MASTER_VERSION: 1.27.8-gke.1067004
MASTER_IP: 35.203.170.65
MACHINE_TYPE: e2-medium
NODE_VERSION: 1.27.8-gke.1067004
NUM_NODES: 3
STATUS: RUNNING
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ cat php-apache.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
spec:
selector:
matchLabels:
run: php-apache
replicas: 3
template:
metadata:
labels:
run: php-apache
spec:
containers:
- name: php-apache
image: k8s.gcr.io/hpa-example
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
labels:
run: php-apache
spec:
ports:
- port: 80
selector:
run: php-apache
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl apply -f php-apache.yaml
deployment.apps/php-apache created
service/php-apache created
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Scale pods with Horizontal Pod Autoscaling
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
php-apache 3/3 3 3 72s
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
horizontalpodautoscaler.autoscaling/php-apache autoscaled
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
This autoscale command will configure a Horizontal Pod Autoscaler that will maintain between 1 and 10 replicas of the pods controlled by the php-apache deployment. The cpu-percent flag specifies 50% as the target average CPU utilization of requested CPU over all the pods. HPA will adjust the number of replicas (via the deployment) to maintain an average CPU utilization of 50% across all pods.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 1%/50% 1 10 3 114s
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Under the Targets column you see 1%/50%.
This means that the pods within your deployment are currently at 1% of their target average CPU utilization. This is to be expected as the php-apache app is receiving no traffic right now.
Also, take note of the Replicas column. To start with, the value will be 3. This number will be changed by the autoscaler as the number of required pods changes.
In this case, the autoscaler will scale the deployment down to the minimum number of pods indicated when you run the autoscale command. Horizontal Pod Autoscaling takes 5-10 minutes and will require shutting down or starting new pods depending on which way it’s scaling.
Scale size of pods with Vertical Pod Autoscaling
Vertical Pod Autoscaling frees you from having to think about what values to specify for a container’s CPU and memory requests. The autoscaler can recommend values for CPU and memory requests and limits, or it can automatically update the values.
Vertical Pod Autoscaling should not be used alongside Horizontal Pod Autoscaling on CPU or memory. Both autoscalers will try to respond to changes in demand on the same metrics and conflict. However, VPA on CPU or memory can be used with HPA on custom metrics to avoid overlap.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ gcloud container clusters describe scaling-demo | grep ^verticalPodAutoscaling -A 1
verticalPodAutoscaling:
enabled: true
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Vertical Pod Autoscaling can be enabled on an existing cluster with
gcloud container clusters update scaling-demo --enable-vertical-pod-autoscaling
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0
deployment.apps/hello-server created
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get deployment hello-server
NAME READY UP-TO-DATE AVAILABLE AGE
hello-server 1/1 1 1 9s
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Assign a CPU resource request of 450m to the deployment:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl set resources deployment hello-server --requests=cpu=450m
deployment.apps/hello-server resource requirements updated
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl describe pod hello-server | sed -n "/Containers:$/,/Conditions:/p"
Containers:
hello-app:
Container ID: containerd://b1acb5451e2d957019664b6025aff416b3a7d7e4dc6d28ff6d5cb5a1bbeb38d6
Image: gcr.io/google-samples/hello-app:1.0
Image ID: gcr.io/google-samples/hello-app@sha256:b1455e1c4fcc5ea1023c9e3b584cd84b64eb920e332feff690a2829696e379e7
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 11 Apr 2024 04:09:17 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 450m
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g4v99 (ro)
Conditions:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
create a manifest for you Vertical Pod Autoscaler:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ cat hello-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: hello-server-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: hello-server
updatePolicy:
updateMode: "Off"
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
The above generates a manifest for a Vertical Pod Autoscaler targeting the hello-server deployment with an Update Policy of Off. A VPA can have one of three different update policies which can be useful depending on your application:
- Off: this policy means VPA will generate recommendations based on historical data which you can manually apply.
- Initial: VPA recommendations will be used to create new pods once and then won’t change the pod size after.
- Auto: pods will regularly be deleted and recreated to match the size of the recommendations.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl apply -f hello-vpa.yaml
verticalpodautoscaler.autoscaling.k8s.io/hello-server-vpa created
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get vpa
NAME MODE CPU MEM PROVIDED AGE
hello-server-vpa Off 6s
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl describe vpa hello-server-vpa
Name: hello-server-vpa
Namespace: default
Labels: <none>
Annotations: <none>
API Version: autoscaling.k8s.io/v1
Kind: VerticalPodAutoscaler
Metadata:
Creation Timestamp: 2024-04-11T04:11:32Z
Generation: 1
Resource Version: 7499
UID: 56e63a5e-f38c-477b-b48a-002fa7504cea
Spec:
Target Ref:
API Version: apps/v1
Kind: Deployment
Name: hello-server
Update Policy:
Update Mode: Off
Events: <none>
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
After some time
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl describe vpa hello-server-vpa
Name: hello-server-vpa
Namespace: default
Labels: <none>
Annotations: <none>
API Version: autoscaling.k8s.io/v1
Kind: VerticalPodAutoscaler
Metadata:
Creation Timestamp: 2024-04-11T04:11:32Z
Generation: 2
Resource Version: 7862
UID: 56e63a5e-f38c-477b-b48a-002fa7504cea
Spec:
Target Ref:
API Version: apps/v1
Kind: Deployment
Name: hello-server
Update Policy:
Update Mode: Off
Status:
Conditions:
Last Transition Time: 2024-04-11T04:12:12Z
Message: Some containers have a small number of samples
Reason: hello-app
Status: True
Type: LowConfidence
Last Transition Time: 2024-04-11T04:12:12Z
Status: True
Type: RecommendationProvided
Recommendation:
Container Recommendations:
Container Name: hello-app
Lower Bound:
Cpu: 1m
Memory: 2097152
Target:
Cpu: 2m
Memory: 3145728
Uncapped Target:
Cpu: 2m
Memory: 3145728
Upper Bound:
Cpu: 1180m
Memory: 1595932672
Events: <none>
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
- Lower Bound: this is the lower bound number VPA looks at for triggering a resize. If your pod utilization goes below this, VPA will delete the pod and scale it down.
- Target: this is the value VPA will use when resizing the pod.
- Uncapped Target: if no minimum or maximum capacity is assigned to the VPA, this will be the target utilization for VPA.
- Upper Bound: this is the upper bound number VPA looks at for triggering a resize. If your pod utilization goes above this, VPA will delete the pod and scale it up.
You’ll notice VPA is recommending the CPU request for this container be set to 25m instead of the previous 100m as well as giving you a suggested number for how much memory should be requested. At this point, these recommendations can be manually applied to the hello-server deployment.
Vertical Pod Autoscaling bases its recommendations on historical data from the container. In practice, it’s recommended to wait at least 24 hours to collect recommendation data before applying any changes.
Update the manifest to set the policy to Auto and apply the configuration:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ sed -i 's/Off/Auto/g' hello-vpa.yaml
kubectl apply -f hello-vpa.yaml
verticalpodautoscaler.autoscaling.k8s.io/hello-server-vpa configured
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ cat hello-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: hello-server-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: hello-server
updatePolicy:
updateMode: "Auto"
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
In order to resize a pod, Vertical Pod Autoscaler will need to delete that pod and recreate it with the new size. By default, to avoid downtime, VPA will not delete and resize the last active pod. Because of this, you will need at least 2 replicas to see VPA make any changes.
Scale hello-server deployment to 2 replicas:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl scale deployment hello-server --replicas=2
deployment.apps/hello-server scaled
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
hello-server 2/2 2 2 11m
php-apache 1/1 1 1 17m
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get vpa
NAME MODE CPU MEM PROVIDED AGE
hello-server-vpa Auto 2m 3145728 True 8m17s
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
hello-server-7c7f99c596-n9mj5 1/1 Running 0 11m
hello-server-7c7f99c596-r7wvg 1/1 Running 0 70s
php-apache-69f9bc5fd5-2d6h6 1/1 Running 0 18m
hello-server-7c7f99c596-n9mj5 1/1 Running 0 11m
hello-server-7c7f99c596-n9mj5 1/1 Terminating 0 11m
hello-server-7c7f99c596-n9mj5 1/1 Terminating 0 11m
hello-server-7c7f99c596-cvkmx 0/1 Pending 0 0s
hello-server-7c7f99c596-cvkmx 0/1 Pending 0 0s
hello-server-7c7f99c596-cvkmx 0/1 ContainerCreating 0 0s
hello-server-7c7f99c596-n9mj5 0/1 Terminating 0 11m
hello-server-7c7f99c596-n9mj5 0/1 Terminating 0 11m
hello-server-7c7f99c596-n9mj5 0/1 Terminating 0 11m
hello-server-7c7f99c596-n9mj5 0/1 Terminating 0 11m
hello-server-7c7f99c596-cvkmx 1/1 Running 0 2s
This is a sign that your VPA is deleting and resizing your pods.
HPA results
By this point, your Horizontal Pod Autoscaler will have most likely scaled your php-apache deployment down.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 1%/50% 1 10 1 19m
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
You’ll see that your php-apache deployment has been scaled down to 1 pod.
VPA results
Now, the VPA should have resized your pods in the hello-server deployment.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl describe pod hello-server | sed -n "/Containers:$/,/Conditions:/p"
Containers:
hello-app:
Container ID: containerd://15adff5ee96b6b39332baa723b1653fe9dc0eccafa86156d7bd10868749e3ce3
Image: gcr.io/google-samples/hello-app:1.0
Image ID: gcr.io/google-samples/hello-app@sha256:b1455e1c4fcc5ea1023c9e3b584cd84b64eb920e332feff690a2829696e379e7
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 11 Apr 2024 04:21:03 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 2m
memory: 3145728
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2f7zw (ro)
Conditions:
Containers:
hello-app:
Container ID: containerd://95b0eb44f3c159a4df4a96fc80d016c36b66fc90f54db068497412905182a130
Image: gcr.io/google-samples/hello-app:1.0
Image ID: gcr.io/google-samples/hello-app@sha256:b1455e1c4fcc5ea1023c9e3b584cd84b64eb920e332feff690a2829696e379e7
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 11 Apr 2024 04:19:30 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 2m
memory: 3145728
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-q449t (ro)
Conditions:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Cluster autoscaler
The Cluster Autoscaler is designed to add or remove nodes based on demand. When demand is high, cluster autoscaler will add nodes to the node pool to accommodate that demand. When demand is low, cluster autoscaler will scale your cluster back down by removing nodes. This allows you to maintain high availability of your cluster while minimizing superfluous costs associated with additional machines.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ gcloud beta container clusters update scaling-demo --enable-autoscaling --min-nodes 1 --max-nodes 5
Updating scaling-demo...done.
Updated [https://container.googleapis.com/v1beta1/projects/qwiklabs-gcp-01-696b4c0e98b8/zones/us-west1-a/clusters/scaling-demo].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1-a/scaling-demo?project=qwiklabs-gcp-01-696b4c0e98b8
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
When scaling a cluster, the decision of when to remove a node is a trade-off between optimizing for utilization or the availability of resources. Removing underutilized nodes improves cluster utilization, but new workloads might have to wait for resources to be provisioned again before they can run.
You can specify which autoscaling profile to use when making such decisions. The currently available profiles are:
-
Balanced: The default profile.
-
Optimize-utilization: Prioritize optimizing utilization over keeping spare resources in the cluster. When selected, the cluster autoscaler scales down the cluster more aggressively. It can remove more nodes, and remove nodes faster. This profile has been optimized for use with batch workloads that are not sensitive to start-up latency.
Switch to the
optimize-utilizationautoscaling profile so that the full effects of scaling can be observed:tudent_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ gcloud beta container clusters update scaling-demo \ --autoscaling-profile optimize-utilization Updating scaling-demo...working Updating scaling-demo...done. Updated [https://container.googleapis.com/v1beta1/projects/qwiklabs-gcp-01-696b4c0e98b8/zones/us-west1-a/clusters/scaling-demo]. To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1-a/scaling-demo?project=qwiklabs-gcp-01-696b4c0e98b8 student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
By default, most of the system pods from these deployments will prevent cluster autoscaler from taking them completely offline to reschedule them. Generally, this is desired because many of these pods collect data used in other deployments and services. For example, metrics-agent being temporarily down would cause a gap in data collected for VPA and HPA, or the fluentd pod being down could create a gap in your cloud logs.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get deployment -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
event-exporter-gke 1/1 1 1 37m
konnectivity-agent 3/3 3 3 37m
konnectivity-agent-autoscaler 1/1 1 1 37m
kube-dns 2/2 2 2 37m
kube-dns-autoscaler 1/1 1 1 37m
l7-default-backend 1/1 1 1 37m
metrics-server-v0.5.2 1/1 1 1 37m
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
For the purpose of this lab, you will apply Pod Disruption Budgets to your kube-system pods which will allow cluster autoscaler to safely reschedule them on another node. This will give enough room to scale your cluster down.
Pod Disruption Budgets (PDB) define how Kubernetes should handle disruptions like upgrades, pod removals, running out of resources, etc. In PDBs, you can specify the max-unavailable and/or the min-available number of pods a deployment should have.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl create poddisruptionbudget kube-dns-pdb --namespace=kube-system --selector k8s-app=kube-dns --max-unavailable 1
kubectl create poddisruptionbudget prometheus-pdb --namespace=kube-system --selector k8s-app=prometheus-to-sd --max-unavailable 1
kubectl create poddisruptionbudget kube-proxy-pdb --namespace=kube-system --selector component=kube-proxy --max-unavailable 1
kubectl create poddisruptionbudget metrics-agent-pdb --namespace=kube-system --selector k8s-app=gke-metrics-agent --max-unavailable 1
kubectl create poddisruptionbudget metrics-server-pdb --namespace=kube-system --selector k8s-app=metrics-server --max-unavailable 1
kubectl create poddisruptionbudget fluentd-pdb --namespace=kube-system --selector k8s-app=fluentd-gke --max-unavailable 1
kubectl create poddisruptionbudget backend-pdb --namespace=kube-system --selector k8s-app=glbc --max-unavailable 1
kubectl create poddisruptionbudget kube-dns-autoscaler-pdb --namespace=kube-system --selector k8s-app=kube-dns-autoscaler --max-unavailable 1
kubectl create poddisruptionbudget stackdriver-pdb --namespace=kube-system --selector app=stackdriver-metadata-agent --max-unavailable 1
kubectl create poddisruptionbudget event-pdb --namespace=kube-system --selector k8s-app=event-exporter --max-unavailable 1
poddisruptionbudget.policy/kube-dns-pdb created
poddisruptionbudget.policy/prometheus-pdb created
poddisruptionbudget.policy/kube-proxy-pdb created
poddisruptionbudget.policy/metrics-agent-pdb created
poddisruptionbudget.policy/metrics-server-pdb created
poddisruptionbudget.policy/fluentd-pdb created
poddisruptionbudget.policy/backend-pdb created
poddisruptionbudget.policy/kube-dns-autoscaler-pdb created
poddisruptionbudget.policy/stackdriver-pdb created
poddisruptionbudget.policy/event-pdb created
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
In each of these commands, you are selecting a different kube-system deployment pod based on a label defined in its creation and specifying that there can be 1 unavailable pod for each of these deployments. This will allow the autoscaler to reschedule the system pods.
With the PDBs in place, your cluster should scale down from three nodes to two nodes in a minute or two.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-scaling-demo-default-pool-0e73e0ba-3s7q NotReady <none> 42m v1.27.8-gke.1067004
gke-scaling-demo-default-pool-0e73e0ba-hd9j Ready <none> 42m v1.27.8-gke.1067004
gke-scaling-demo-default-pool-0e73e0ba-q83h Ready <none> 42m v1.27.8-gke.1067004
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-scaling-demo-default-pool-0e73e0ba-3s7q NotReady <none> 42m v1.27.8-gke.1067004
gke-scaling-demo-default-pool-0e73e0ba-hd9j Ready <none> 42m v1.27.8-gke.1067004
gke-scaling-demo-default-pool-0e73e0ba-q83h Ready <none> 42m v1.27.8-gke.1067004
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
You set up automation that scaled your cluster down from three nodes to two nodes!
Thinking about the costs, as a result of scaling down your nodepool, you will be billed for less machines during periods of low demand on your cluster. This scaling could be even more dramatic if you were fluctuating from high demand to low demand periods during the day.
It’s important to note that, while Cluster Autoscaler removed an unnecessary node, Vertical Pod Autoscaling and Horizontal Pod Autoscaling helped reduce enough CPU demand so that the node was no longer needed. Combining these tools is a great way to optimize your overall costs and resource usage.
So, the cluster autoscaler helps add and remove nodes in response to pods needing to be scheduled. However, GKE specifically has another feature to scale vertically, called node auto-provisioning.
Node Auto Provisioning
Node Auto Provisioning (NAP) actually adds new node pools that are sized to meet demand. Without node auto provisioning, the cluster autoscaler will only be creating new nodes in the node pools you’ve specified, meaning the new nodes will be the same machine type as the other nodes in that pool. This is perfect for helping optimize resource usage for batch workloads and other apps that don’t need extreme scaling, since creating a node pool that is specifically optimized for your use case might take more time than just adding more nodes to an existing pool.
-
Enable Node Auto Provisioning:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ gcloud container clusters update scaling-demo \ --enable-autoprovisioning \ --min-cpu 1 \ --min-memory 2 \ --max-cpu 45 \ --max-memory 160 Updating scaling-demo...done. Updated [https://container.googleapis.com/v1/projects/qwiklabs-gcp-01-696b4c0e98b8/zones/us-west1-a/clusters/scaling-demo]. To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1-a/scaling-demo?project=qwiklabs-gcp-01-696b4c0e98b8 student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
In the command, you specify a minimum and maximum number for your CPU and memory resources. This is for the entire cluster.
NAP can take a little bit of time and it’s also highly likely it won’t create a new node pool for the scaling-demo cluster at its current state.
Test with larger demand
So far, you’ve analyzed how HPA, VPA, and cluster autoscaler can help save resources and costs while your application has low demand. Now, you’ll look at how these tools handle availability for increased demand.
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
If you don't see a command prompt, try pressing enter.
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 1%/50% 1 10 1 48m
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 173%/50% 1 10 1 48m
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Wait and rerun the command until you see your target above 100%.
Now, monitor how your cluster handles the increased load by periodically running this command:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get deployment php-apache
NAME READY UP-TO-DATE AVAILABLE AGE
php-apache 2/4 4 2 50m
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
After a few minutes, you will see a few things happen.
- First, your
php-apachedeployment will automatically be scaled up by HPA to handle the increased load. - Then, cluster autoscaler will need to provision new nodes to handle the increased demand.
- Finally, node auto provisioning will create a node pool optimized for the CPU and memory requests of your cluster’s workloads. In this case, it should be a high cpu, low memory node pool because the load test is pushing the cpu limits.
Wait until your php-apache deployment is scaled up to 7 replicas and your nodes tab looks similar to this:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
hello-server 2/2 2 2 45m
php-apache 6/6 6 6 52m
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-scaling-demo-default-pool-0e73e0ba-ctx4 Ready <none> 116s v1.27.8-gke.1067004
gke-scaling-demo-default-pool-0e73e0ba-hd9j Ready <none> 53m v1.27.8-gke.1067004
gke-scaling-demo-default-pool-0e73e0ba-pwnr Ready <none> 54s v1.27.8-gke.1067004
gke-scaling-demo-default-pool-0e73e0ba-q83h Ready <none> 53m v1.27.8-gke.1067004
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Your cluster efficiently scaled up to meet a higher demand! However, take note of the amount of time it took to handle this spike in demand. For many applications, losing availability while provisioning new resources can be an issue.
Optimize larger loads
When scaling up for larger loads, horizontal pod autoscaling will add new pods while vertical pod autoscaling will resize them depending on your settings. If there’s room on an existing node, it might be able to skip pulling the image and immediately start running the application on a new pod. If you’re working with a node that hasn’t deployed your application before, a bit of time might be added if it needs to download the container images before running it.
So, if you don’t have enough room on your existing nodes and you’re using the cluster autoscaler, it could take even longer. Now it needs to provision a new node, set it up, then download the image and start up pods. If the node auto-provisioner is going to create a new node pool like it did in your cluster, there will be even more time as you provision the new node pool first, and then go through all the same steps for the new node.
In order to handle these different latencies for autoscaling, you’ll probably want to over-provision a little bit so there’s less pressure on your apps when autoscaling-up. This is really important for cost-optimization, because you don’t want to pay for more resources than you need, but you also don’t want your apps’ performance to suffer.
An efficient strategy to overprovision a cluster with Cluster Autoscaling is to use Pause Pods.
Pause Pods are low priority deployments which are able to be removed and replaced by high priority deployments. This means you can create low priority pods which don’t actually do anything except reserve buffer space. When the higher-priority pod needs room, the pause pods will be removed and rescheduled to another node, or a new node, and the higher-priority pod has the room it needs to be scheduled quickly.
Create a manifest for a pause pod:
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ cat pause-pod.yaml
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: overprovisioning
value: -1
globalDefault: false
description: "Priority class used by overprovisioning."
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: overprovisioning
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
run: overprovisioning
template:
metadata:
labels:
run: overprovisioning
spec:
priorityClassName: overprovisioning
containers:
- name: reserve-resources
image: k8s.gcr.io/pause
resources:
requests:
cpu: 1
memory: 4Gi
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ kubectl apply -f pause-pod.yaml
priorityclass.scheduling.k8s.io/overprovisioning created
deployment.apps/overprovisioning created
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$
Observe how a new node is created, most likely in a new node pool, to fit your newly created pause pod. Now, if you were to run the load test again, when you needed an extra node for your php-apache deployment, it could be scheduled on the node with your pause pod while your pause pod is instead put on a new node. This is excellent because your dummy pause pods allow your cluster to provision a new node in advance so that your actual application can scale up much faster. If you were expecting higher amounts of traffic, you could add more pause pods, but it’s considered best practice to not add more than one pause pod per node.
In this lab, you configured a cluster to automatically and efficiently scale up or down based on its demand. Horizontal Pod Autoscaling and Vertical Pod Autoscaling provided solutions for automatically scaling your cluster’s deployments while Cluster Autoscaler and Node Auto Provisioning provided solutions for automatically scaling your cluster’s infrastructure.
As always, knowing which of these tools to use will depend on your workload. Careful use of these autoscalers can mean maximizing availability when you need it while only paying for what you need during times of low demand. When thinking about costs, this means you are optimizing your resource usage and saving money.
## Summary
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$ history
1 gcloud config set compute/zone us-west1-a
2 gcloud container clusters create scaling-demo --num-nodes=3 --enable-vertical-pod-autoscaling
3 cat << EOF > php-apache.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
spec:
selector:
matchLabels:
run: php-apache
replicas: 3
template:
metadata:
labels:
run: php-apache
spec:
containers:
- name: php-apache
image: k8s.gcr.io/hpa-example
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
labels:
run: php-apache
spec:
ports:
- port: 80
selector:
run: php-apache
EOF
4 cat php-apache.yaml
5 kubectl apply -f php-apache.yaml
6 kubectl get deployment
7 kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
8 kubectl get deployment
9 kubectl get deployment
10 kubectl get hpa
11 kubectl get hpa
12 kubectl get deployment
13 gcloud container clusters describe scaling-demo | grep ^verticalPodAutoscaling -A 1
14 kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0
15 kubectl get deployment hello-server
16 kubectl set resources deployment hello-server --requests=cpu=450m
17 kubectl describe pod hello-server | sed -n "/Containers:$/,/Conditions:/p"
18 cat << EOF > hello-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: hello-server-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: hello-server
updatePolicy:
updateMode: "Off"
EOF
19 cat hello-vpa.yaml
20 kubectl apply -f hello-vpa.yaml
21 kubectl get vpa
22 kubectl describe vpa hello-server-vpa
23 kubectl describe vpa hello-server-vpa
24 sed -i 's/Off/Auto/g' hello-vpa.yaml
25 kubectl apply -f hello-vpa.yaml
26 cat hello-vpa.yaml
27 kubectl scale deployment hello-server --replicas=2
28 kubectl get deployment
29 kubectl get pva
30 kubectl get vpa
31 kubectl get pods -w
32 kubectl get hpa
33 kubectl describe pod hello-server | sed -n "/Containers:$/,/Conditions:/p"
34 gcloud beta container clusters update scaling-demo --enable-autoscaling --min-nodes 1 --max-nodes 5
35 gcloud beta container clusters update scaling-demo --autoscaling-profile optimize-utilization
36 kubectl get deployment -n kube-system
37 kubectl create poddisruptionbudget kube-dns-pdb --namespace=kube-system --selector k8s-app=kube-dns --max-unavailable 1
38 kubectl create poddisruptionbudget prometheus-pdb --namespace=kube-system --selector k8s-app=prometheus-to-sd --max-unavailable 1
39 kubectl create poddisruptionbudget kube-proxy-pdb --namespace=kube-system --selector component=kube-proxy --max-unavailable 1
40 kubectl create poddisruptionbudget metrics-agent-pdb --namespace=kube-system --selector k8s-app=gke-metrics-agent --max-unavailable 1
41 kubectl create poddisruptionbudget metrics-server-pdb --namespace=kube-system --selector k8s-app=metrics-server --max-unavailable 1
42 kubectl create poddisruptionbudget fluentd-pdb --namespace=kube-system --selector k8s-app=fluentd-gke --max-unavailable 1
43 kubectl create poddisruptionbudget backend-pdb --namespace=kube-system --selector k8s-app=glbc --max-unavailable 1
44 kubectl create poddisruptionbudget kube-dns-autoscaler-pdb --namespace=kube-system --selector k8s-app=kube-dns-autoscaler --max-unavailable 1
45 kubectl create poddisruptionbudget stackdriver-pdb --namespace=kube-system --selector app=stackdriver-metadata-agent --max-unavailable 1
46 kubectl create poddisruptionbudget event-pdb --namespace=kube-system --selector k8s-app=event-exporter --max-unavailable 1
47 kubectl get nodes
48 kubectl get nodes -w
49 kubectl get nodes
50 kubectl get nodes
51 kubectl get nodes
52 kubectl get nodes
53 kubectl get nodes
54 kubectl get nodes
55 kubectl get nodes
56 kubectl get nodes
57 kubectl get nodes
58 kubectl get nodes
59 kubectl get nodes
60 kubectl get nodes
61 kubectl get nodes
62 kubectl get nodes
63 kubectl get nodes
64 kubectl get nodes
65 gcloud container clusters update scaling-demo --enable-autoprovisioning --min-cpu 1 --min-memory 2 --max-cpu 45 --max-memory 160
66 kubectl get hpa
67 kubectl get hpa
68 kubectl get deployment php-apache
69 kubectl get deployment php-apache
70 kubectl get deployment
71 kubectl get nodes
72 kubectl get deployment
73 kubectl get deployment
74 kubectl get deployment
75 kubectl get nodes
76 cat << EOF > pause-pod.yaml
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: overprovisioning
value: -1
globalDefault: false
description: "Priority class used by overprovisioning."
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: overprovisioning
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
run: overprovisioning
template:
metadata:
labels:
run: overprovisioning
spec:
priorityClassName: overprovisioning
containers:
- name: reserve-resources
image: k8s.gcr.io/pause
resources:
requests:
cpu: 1
memory: 4Gi
EOF
77 cat pause-pod.yaml
78 kubectl apply -f pause-pod.yaml
79 kubectl get nodes
80 history
student_01_fa9e4f85d2dc@cloudshell:~ (qwiklabs-gcp-01-696b4c0e98b8)$