Efazati

Kubernetes

Kubernetes at the edge, on x86 and ARM

Mohammad Efazati · · 2 min read

At Cainthus I helped build a hybrid platform: part cloud, part edge, one Kubernetes cluster of more than a hundred nodes that mixed x86 servers with ARM boards like the NVIDIA Jetson and the Raspberry Pi. The goal was simple to say and hard to do: every team should be able to ship to any of those machines on their own, the same way.

Here is what that taught me.

Multi-arch is a build problem, not a cluster problem

Kubernetes is happy to schedule onto ARM. The trouble starts earlier, in the image. If a team builds only for amd64, the pod lands on a Jetson and fails with an exec format error that nobody understands at first sight.

The fix belongs in CI. Every image is built as a multi-arch manifest, so the same tag works everywhere and the node pulls the variant it can run:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t registry.example/app:1.4.2 --push .

Where a workload genuinely needs one architecture (a GPU model built for the Jetson, say), say so explicitly with a node selector on kubernetes.io/arch rather than hoping the scheduler guesses.

The network will go away. Plan for it.

Edge nodes live on links you do not control. They drop, they get slow, they come back with a different address. A design that assumes the control plane is always reachable will spend its life in alert storms.

  • Keep workloads at the edge able to run on their own for a while: local buffering, retries with backoff, nothing that blocks on a round trip to the cloud.
  • Tune node failure timeouts so a short drop does not evict and reschedule everything.
  • Treat the VPN as production infrastructure. We automated the OpenVPN deployment, its monitoring, central user management and high availability, because when it broke, everything broke.

Installs as code, all the way down

With a few nodes you can get away with a runbook. With a hundred, across hardware types, every hand-run step becomes a node that is subtly different from its neighbours. We built the whole Kubernetes installation pipeline as code, from a bare board to a ready node, with Ansible and Terraform, so a new device was a pipeline run rather than an afternoon.

The payoff is not speed. It is that when something is wrong, you can rebuild the node instead of debugging it.

Give teams a wrapper, not a cluster

Most product engineers did not want to learn Kubernetes and AWS in depth, and they should not have to. We wrote a thin wrapper, an API and a CLI, with access control and CI/CD built in. Teams described what they wanted to run; the wrapper turned it into the right objects in the right place.

That wrapper was the real product of the platform team. The cluster was just where it ran.

What I would tell someone starting today

  1. Make multi-arch builds the default before the first ARM node joins.
  2. Decide what happens when the edge is offline, and test it by pulling the cable.
  3. Never configure a node by hand, not even once.
  4. Build the paved road for teams early; it is what makes the platform worth having.

If you are working on something similar, I would enjoy hearing about it.