Kubernetes Multiple Schedulers

Kubernetes Multiple Schedulers

Multiple Schedulers

Kubernetes ships with a default scheduler (kube-scheduler) that we discussed earlier. If the default scheduler does not suit your needs you can implement your own scheduler. Moreover, you can even run multiple schedulers simultaneously alongside the default scheduler and instruct Kubernetes what scheduler to use for each of your pods.

We can use the staticPod manifest file located at /etc/kubernetes/manifests/kube-scheduler.yaml to create our own scheduler.

First, let us take a look at the definition of the default scheduler.

pradeep@learnk8s$ minikube ssh -p k8s
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ cat /etc/kubernetes/manifests/
etcd.yaml                     kube-apiserver.yaml           kube-controller-manager.yaml  kube-scheduler.yaml

$ sudo cat /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=false
    image: k8s.gcr.io/kube-scheduler:v1.23.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}
$

The default scheduler listens on port 10259, so for our custom scheduler, we have to choose another port, for example 10282.

Here is the definition of the new scheduler called my-scheduler, which is listening on the secure-port 10282.

pradeep@learnk8s$ minikube ssh -p k8s
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ sudo cat /etc/kubernetes/manifests/my-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: my-scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=false
    - --secure-port=10282
    image: k8s.gcr.io/kube-scheduler:v1.23.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10282
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: my-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10282
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}
$

Verify that both schedulers kube-scheduler-k8s and my-scheduler-k8s are in running state.

pradeep@learnk8s$ kubectl get pods  -n kube-system
NAME                          READY   STATUS    RESTARTS        AGE
coredns-64897985d-r9tzv       1/1     Running   8 (3h2m ago)    6d3h
etcd-k8s                      1/1     Running   2 (3h2m ago)    6d3h
kindnet-jpxdd                 1/1     Running   5 (9h ago)      6d3h
kindnet-p77mb                 1/1     Running   50 (3h6m ago)   6d3h
kube-apiserver-k8s            1/1     Running   3 (9h ago)      6d3h
kube-controller-manager-k8s   1/1     Running   4 (3h1m ago)    6d3h
kube-proxy-fszkr              1/1     Running   1 (2d16h ago)   6d3h
kube-proxy-m747v              1/1     Running   1               6d3h
kube-scheduler-k8s            1/1     Running   0               14m
my-scheduler-k8s              1/1     Running   0               2m14s
storage-provisioner           1/1     Running   49 (149m ago)   6d3h

Now, it is time to deploy a Pod that makes use of this new scheduler. We just need to add one line for the schedulerName in the Pod spec.

pradeep@learnk8s$ cat pod-my-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: nginx-my-scheduler
  name: nginx-my-scheduler
spec:
  schedulerName: my-scheduler
  containers:
  - image: nginx
    name: nginx-my-scheduler
    resources: {}

pradeep@learnk8s$ kubectl create -f pod-my-scheduler.yaml
pod/nginx-my-scheduler created

pradeep@learnk8s$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
demo-6c54f77c95-mgz7f   1/1     Running   1          6d
demo-6c54f77c95-q679r   1/1     Running   1          5d
demo-6c54f77c95-qqzbf   1/1     Running   1          5d
demo-6c54f77c95-vjgc2   1/1     Running   1          5d
demo-6c54f77c95-wv78b   1/1     Running   1          6d
demo-ds-gtcf7           1/1     Running   0          16h
demo-ds-kkw4g           1/1     Running   0          16h
nginx-k8s-m02           1/1     Running   0          137m
nginx-manual            1/1     Running   1          4d23h
nginx-my-scheduler      0/1     Pending   0          2s
nginx-no-tolerate       1/1     Running   1          4d17h
nginx-node-selector     1/1     Running   0          2d17h
nginx-taint-demo        1/1     Running   1          4d17h
with-node-affinity      1/1     Running   0          2d17h

NOTE: There seems to be some issue with this, the Pod is in Pending state, I could not find any events. In the recent Kubernetes version 1.23, there seems to be some changes related to Scheduler definition. I will have to park this aside for some time and continue investigating.