Skip to main content

Node Remediation

If a worker node consistently exhibits abnormal behaviour – such as the NotReady status, hung pods or recurring container runtime errors – restart it individually rather than taking action across the entire cluster.

Prerequisites​

  • Access to the centron Control Panel
  • A Kubernetes cluster with the status Running and a verified connection via kubectl

Check the condition​

First, get an overview of the affected node:

# Status aller Nodes
kubectl get nodes

# Details, Conditions und Events eines Nodes
kubectl describe node <node-name>

# Pods, die auf dem Node laufen
kubectl get pods --all-namespaces --field-selector spec.nodeName=<node-name>

In the Conditions section of the describe output, you will find entries such as MemoryPressure, DiskPressure or NetworkUnavailable.

In Control Panel, you can view the status in the Resources tab: expand the node pool using the + icon to display the individual nodes and their statuses.

Restart the node (Recycle)​

Selecting Recycle completely restarts the node. The node pool replaces it with a newly provisioned node – whilst maintaining the target size of the pool. This is the preferred course of action in the event of a faulty node.

  1. Open the cluster and go to the Resources tab.
  2. Expand the relevant node pool using the + icon.
  3. Open the Kebab menu (three dots) on the desired node.
  4. Select Recycle and confirm the action.

The node changes to status Deleting, after which the replacement node appears with status Provisioning and finally Running.

Remove node (Delete)​

Selecting Delete removes the node without a replacement node taking its place. The capacity of the pool is reduced accordingly.

  1. In the Resources tab, open the node’s drop-down menu.
  2. Select Delete and confirm the action.
Warnung

If autoscaling is enabled for the pool and the number of nodes falls below the configured minimum after deletion, the cluster autoscaler automatically provisions a new node.

Protect workloads in advance​

Both Recycle and Delete terminate all pods running on the node. Make sure beforehand that your applications can cope with this:

  • Run production services with multiple replicas distributed across different nodes.

  • Set a PodDisruptionBudget value to ensure that a minimum number of pods remain accessible at all times:

    apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: web-pdb
    spec:
      minAvailable: 2
      selector:
        matchLabels:
          app: web

  • For stateful workloads, check whether the volumes being used are bound to the node.

Optionally, you can empty the node in a controlled manner beforehand:

# Keine neuen Pods mehr auf dem Node platzieren
kubectl cordon <node-name>

# Vorhandene Pods verlagern
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
Hinweis

kubectl drain permanently deletes data from the node’s emptyDir volumes. Only use this option if this data is expendable.

If the problem persists​

If the error reoccurs after a restart, the cause is usually not down to the individual node:

  • Resource shortage: Check the Analytics tab to see whether the nodes in the pool are consistently operating at full capacity. See View metrics .
  • Missing limits: Without limits set, a single pod can utilise a node to full capacity.
  • Insufficient provisioning: See Select plan .

If you are unable to pinpoint the cause, please contact centron Support, providing the cluster name, node name and the time at which the issue occurred.

Last updated on