Thirty seconds is enough
Spot instances are the cheapest compute in the cloud and the least polite. Sampleless Spot turns a preemption notice into an orderly handover instead of an outage.
What happens when the notice arrives
The whole sequence fits inside the eviction window, with the expensive work started first.
T+0s
Notice observed
The agent polls the cloud scheduled-events endpoint every second, so a preemption, termination, or redeploy notice is seen almost as soon as it is posted.
T+1s
Node cordoned
The node is marked unschedulable immediately, so nothing new lands on a machine that is about to disappear.
T+1s
Replacements pre-warmed
For any replica set with several pods on this node, one is deleted right away so its replacement starts pulling images on a healthy node while there is still time.
T+11s
Leaders handed over
Pods with a pre-failover hook get an orderly handover first: a Patroni switchover, or a Sentinel failover for Valkey and Redis, before anything is torn down.
T+12s
Drained and acknowledged
Remaining pods are evicted according to their annotations, then the event is acknowledged so the reclaim proceeds instead of waiting out the window.
Timings are illustrative; the settle delay before draining is configurable.
Decide once, on one page
A single cluster-wide profile describes how eviction and image mirroring behave. Everything below is the platform’s problem, not yours.
Per-pod eviction policy
Annotate a pod to choose graceful delete, force delete, or skip. Unannotated workloads get defaults that leave DaemonSet, bare, and node-owned pods alone.
Database-aware handover
Pre-failover hooks for Patroni-managed Postgres and for Valkey and Redis via Sentinel, so a leader moves deliberately rather than being discovered dead.
Candidate awareness
Failover targets that are themselves on cordoned nodes are filtered out, and a synchronous standby is preferred over a plain replica.
Manual fast roll
An endpoint that gracefully rolls selected pods on a node using the same failover-aware path, for planned maintenance rather than preemption.
Prometheus metrics
Counters for events by type, pods evicted and pre-warmed, each distinct skip reason, failover outcomes, and acknowledgement success, so you can see how reclaims actually went.
In-cluster image mirror
Configures the container runtime on each node to pull public images through an in-cluster cache, with fallback to the public registry on a miss.
One handler, every cloud’s notice
Each provider announces a reclaim differently and gives you a different amount of warning. What you configure stays the same.
Azure / AKS
~30s notice
Scheduled events polled every second, with acknowledgement so the reclaim proceeds as soon as the node is drained.
AWS / EKS
~2min notice
Instance metadata interruption notices and rebalance recommendations. The longer window means more of the drain happens gracefully, and rebalance signals arrive before the reclaim does.
GCP / GKE
~30s notice
Preemption notice via the metadata server, on the same handler and the same per-pod policy model.
Per-pod eviction policy, failover hooks, and the image mirror are provider independent. Only the notice source differs.
Frequently asked questions
- Why do I need anything to run on spot instances?
- Because the default behavior when a spot node is reclaimed is bad. The node disappears and the kubelet stops; pods are noticed as gone only after node-monitor timeouts, and stateful workloads lose their leader without warning. Spot watches the cloud preemption notice, so you get an orderly cordon, failover, and drain inside the notice window instead of an abrupt loss.
- How much warning does it actually get?
- Azure gives roughly 30 seconds of notice for a spot eviction. Spot polls the scheduled events endpoint every second so it sees the notice almost immediately, and it acknowledges the event as soon as it has drained the node. Acknowledging early lets the reclaim proceed rather than burning the rest of the window waiting.
- What happens to a database whose leader is on the reclaimed node?
- A pod can declare a pre-failover hook. For Patroni-managed Postgres, Spot confirms this pod really is the leader, picks a candidate that is not itself on a cordoned node, prefers a synchronous standby, and triggers a switchover before the drain. For Valkey and Redis it issues a Sentinel failover, which skips the several-second failure detection window. Both are best effort: if the handoff fails, it is logged and the drain proceeds.
- Can I control what happens to individual pods?
- Yes, with a per-pod annotation choosing graceful delete, force delete, or skip. Unannotated pods get sensible defaults: DaemonSet pods, bare pods, and node-owned static pods are skipped, and everything else gets a graceful delete.
- What is the registry mirror for?
- Spot nodes come and go, which means image pulls happen constantly. The mirror configures the container runtime on each node to pull public images through an in-cluster cache first, falling back to the public registry on a miss. That keeps replacement pods starting quickly and keeps repeated pulls off the public internet.
- Does this work on AWS and GCP?
- Yes, on all three. Each cloud announces a reclaim differently and gives you a different amount of warning — roughly two minutes on AWS, about thirty seconds on Azure and GCP — so the notice source is provider specific. Everything above it is not: per-pod eviction policy, failover hooks, and the image mirror behave identically wherever you run.