KostrixBeta, home
Log in Start free trial

EC2 Spot Instances Done Right, Without the Drama

Run EC2 Spot Instances safely: interruption notices, price-capacity-optimized, diversification, Karpenter, EMR and Batch, plus the workloads to avoid.

Kostrix Team10 min read

Spot has a reputation problem, and it’s mostly self-inflicted. Somebody ran a stateful service on a single instance type in a single Availability Zone, capacity in that one pool dried up, and the post-mortem concluded that “Spot is unreliable.” Nobody wrote down that the setup had exactly one way to get capacity and no plan for losing it.

Spot Instances are spare EC2 capacity sold at a discount that AWS puts at up to 90% off on-demand. The deal is that AWS can take the capacity back with a short warning. Design for that as a routine event and Spot is the cheapest compute you’ll find on AWS. Treat it as a rare accident and it’ll eventually ruin someone’s weekend.

This is how we set it up so it stays boring. Where we mention prices, they’re us-east-1 list prices at the time of writing.

How Spot pricing works now

There’s no bidding any more. AWS sets the Spot price for each instance type in each Availability Zone, and it moves gradually with long-term supply and demand rather than jumping around minute by minute. You can set a maximum price, but if you don’t, the cap is the on-demand price. AWS notes that setting a maximum price makes interruptions more frequent, so our advice is simply not to set one.

Billing is per second on Linux and Windows. If EC2 interrupts an instance in its first hour, you aren’t charged for that hour at all. After that you pay for the seconds used.

To see what you’d pay today, ask for the current price across AZs. This is read-only:

aws ec2 describe-spot-price-history \
  --instance-types m7i.large m6i.large m5.large \
  --product-descriptions "Linux/UNIX" \
  --start-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --query 'SpotPriceHistory[].[AvailabilityZone,InstanceType,SpotPrice]' \
  --output table

The Spot Instance Advisor adds the other half of the picture. It shows each instance type’s typical savings and an interruption frequency bucket (under 5%, 5-10% and so on, up to over 20%) for the trailing month. The savings figures are averaged across AZs, so your own AZ can differ. Containers have their own option too: Fargate Spot runs interruption-tolerant ECS tasks for up to 70% off the regular Fargate price.

What an interruption actually looks like

You get up to two signals, and you should handle both.

  1. Rebalance recommendation. A heads-up that the instance is at elevated risk. It usually arrives before the interruption notice, but AWS is clear that it’s best effort and can arrive at the same moment.
  2. Interruption notice. A two-minute warning before EC2 stops, hibernates or terminates the instance. Terminate is the default. Stop and hibernate suit time-flexible work that can resume when capacity returns.

Both show up in EventBridge (as EC2 Instance Rebalance Recommendation and EC2 Spot Instance Interruption Warning) and in instance metadata. AWS suggests polling metadata every five seconds:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
# 404 means no interruption is scheduled
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/spot/instance-action
# 404 means no rebalance signal yet
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/events/recommendations/rebalance

Two minutes is enough to deregister from a load balancer, finish an in-flight request, flush logs and write a small checkpoint. It isn’t enough to drain a 20 GB cache or wait out a long database transaction. If your shutdown path takes longer than about ninety seconds, fix that before you put the workload on Spot.

Diversify, or don’t bother

A Spot capacity pool is one instance type in one Availability Zone. Ask for a single pool and you’re betting everything on spare capacity in that one slot. AWS’s own rule of thumb is to be flexible across at least ten instance types per workload, with every AZ in the VPC enabled.

Say a group only asks for c7i.2xlarge in one AZ. That’s one pool. Allow c5, c6i, c7i, c6a, c7a, m5, m6i, m7i, m6a and m7a at 2xlarge, across three AZs, and you have 30. Older generations help here because fewer on-demand customers want them.

Before you commit to a region or AZ, check the Spot placement score. It rates how likely a request is to be fulfilled on a scale of 1 to 10, costs nothing, and needs at least three instance types to return a useful answer:

aws ec2 get-spot-placement-scores \
  --instance-types c6i.2xlarge c7i.2xlarge m6i.2xlarge m7i.2xlarge c6a.2xlarge \
  --target-capacity 64 --target-capacity-unit-type vcpu \
  --single-availability-zone --region-names us-east-1 us-east-2 us-west-2

It’s a point-in-time recommendation. It doesn’t promise capacity and it doesn’t predict interruptions.

Use price-capacity-optimized

The allocation strategy decides which pools your instances come from. This is the single setting we change most often in other people’s accounts.

Strategy What it does Use it?
price-capacity-optimized Picks pools with deep capacity, then the lowest price among them Yes, the default choice
capacity-optimized Picks the deepest pools, ignoring price When interruptions are very costly
capacity-optimized-prioritized Capacity first, then your preferred type order When you need some types favoured
lowest-price Picks the cheapest pools No, it has the highest interruption risk

The lowest-price strategy looks cheapest on paper. In practice the cheapest pools tend to be the ones everybody else is draining too, so you pay for it in churn. AWS Batch calls the recommended option SPOT_PRICE_CAPACITY_OPTIMIZED, and EMR instance fleets use price-capacity-optimized by default from release 6.10.0.

Auto Scaling groups with mixed instances

For EC2 services, an Auto Scaling group with a mixed instances policy is the workhorse. You keep a small on-demand base for the capacity you can’t lose, then split the rest between on-demand and Spot. This is a complete input for aws autoscaling create-auto-scaling-group --cli-input-json file://asg.json, with placeholder subnet IDs:

{
  "AutoScalingGroupName": "web",
  "MinSize": 2,
  "MaxSize": 20,
  "VPCZoneIdentifier": "subnet-0123456789abcdef0,subnet-0123456789abcdef1,subnet-0123456789abcdef2",
  "CapacityRebalance": true,
  "MixedInstancesPolicy": {
    "InstancesDistribution": {
      "OnDemandBaseCapacity": 2,
      "OnDemandPercentageAboveBaseCapacity": 20,
      "SpotAllocationStrategy": "price-capacity-optimized"
    },
    "LaunchTemplate": {
      "LaunchTemplateSpecification": { "LaunchTemplateName": "web", "Version": "$Default" },
      "Overrides": [{
        "InstanceRequirements": {
          "VCpuCount": { "Min": 4, "Max": 8 },
          "MemoryMiB": { "Min": 16384 }
        }
      }]
    }
  }
}

That InstanceRequirements block is attribute-based instance selection. Instead of maintaining a list of types, you describe vCPUs and memory and the group picks every matching type, including new ones as AWS releases them. Price protection is on by default and filters out types priced well above the cheapest current-generation C, M or R match. For on-demand the default cut-off is 20% above that price. Spot gets a much looser default (the EC2 API reference lists 100%), so set SpotMaxPricePercentageOverLowestPrice yourself if you want a tighter one. Preview what you’d get before creating anything:

aws ec2 get-instance-types-from-instance-requirements \
  --architecture-types x86_64 --virtualization-types hvm \
  --instance-requirements 'VCpuCount={Min=4,Max=8},MemoryMiB={Min=16384}' \
  --query 'InstanceTypes[].InstanceType' --output text

Keep Capacity Rebalancing on, as CapacityRebalance is in the example. When a rebalance recommendation arrives, the group launches a replacement, waits for it to pass health checks, then terminates the at-risk instance. Two side effects: the group can briefly go over its maximum size by up to 10% of desired capacity, and it will sometimes replace an instance that wouldn’t have been interrupted. Both are a fair price for the extra lead time. If you add a lifecycle hook for graceful shutdown, keep the work under two minutes.

To find groups that need attention, run this read-only check and look for lowest-price or False:

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].[AutoScalingGroupName,MixedInstancesPolicy.InstancesDistribution.SpotAllocationStrategy,CapacityRebalance]' \
  --output table

Karpenter and EKS

On EKS, Karpenter makes Spot fairly painless. Allow both capacity types in a NodePool with karpenter.sh/capacity-type set to ["spot", "on-demand"], and Karpenter tries Spot before falling back to on-demand. Keep the requirements wide, for example instance categories c, m and r rather than one pinned type.

The parts people miss:

  • Interruption handling isn’t automatic. Karpenter needs an SQS queue fed by EventBridge rules, passed in through its interruption queue setting. With that in place it cordons and drains nodes when an interruption warning arrives. Without it, pods just die.
  • Rebalance recommendations are only logged. Karpenter records an event but doesn’t drain the node.
  • Spot-to-spot consolidation is behind the SpotToSpotConsolidation feature flag, and single-node replacements need at least 15 candidate instance types. That’s deliberate, to stop it consolidating everything onto the cheapest and most interrupted type.
  • Pods need to cooperate. Set PodDisruptionBudgets and keep terminationGracePeriodSeconds comfortably under two minutes.

EMR and AWS Batch

For long-running EMR clusters, AWS recommends keeping the primary and core nodes on-demand and putting task nodes on Spot, since task nodes hold no HDFS data. Losing a task node slows the job down without losing data. Transient, cost-driven clusters can go all-Spot. Instance fleets accept up to 30 instance types per fleet through the CLI or API when you set an allocation strategy, and you can tell a fleet to switch to on-demand if Spot isn’t fulfilled within a timeout.

For Batch, use a Spot compute environment with SPOT_PRICE_CAPACITY_OPTIMIZED and give job definitions a retry strategy so interrupted jobs come back on their own. Retry on host failures rather than on every non-zero exit, or a genuine bug will happily burn through its attempts.

Workloads that fit, and ones that don’t

Good fit Poor fit
CI/CD runners and build agents Single-instance databases
Batch, ETL and data processing Stateful services without replication
Stateless web and API tiers above an on-demand base Long jobs with no checkpointing
Containers on EKS or ECS Tightly coupled HPC that needs every node at once
Rendering and HPC with checkpoints Software licensed to a specific host
Dev and test environments Anything that can’t run at reduced capacity

AWS specifically discourages putting inflexible workloads on Spot and planning to fail over to on-demand during interruptions. When capacity is tight, that fallback is the thing most likely to let you down.

Checkpointing

The rule we use: the largest unit of work should be cheap to redo. Split ETL into small partitions. Save ML training state to S3 every few hundred steps rather than once per epoch. Make queue consumers idempotent so a redelivered message does no harm. Use the two-minute notice for a final checkpoint only if writing one takes seconds, not minutes.

Mistakes we keep seeing

  • One type, one AZ. Covered above, and still the most common cause of “Spot doesn’t work for us”.
  • Load balancer deregistration delay left at 300 seconds. That’s the ALB default, and it’s longer than the notice. Spot targets should drain well inside two minutes.
  • Setting a maximum price to “save money”. It only adds interruptions.
  • No handler at all. No EventBridge rule, no metadata polling, no lifecycle hook. Instances vanish mid-request.
  • Running the EMR primary node on Spot for a cluster that’s meant to live for months.
  • Forgetting the Spot quota. Spot limits are counted in vCPUs, separately from on-demand, and a scale-out can stall on them.
  • Using the legacy APIs. RequestSpotInstances and Spot Fleet get no new investment from AWS. Use Auto Scaling groups or EC2 Fleet.
  • Counting Spot in a Savings Plan baseline. Savings Plans don’t apply to Spot usage, so leave that usage out when you size commitments. Our Reserved Instances vs Savings Plans guide explains the baseline maths.

FAQ

How much cheaper are EC2 Spot Instances than On-Demand?

AWS advertises up to 90% off on-demand. The real figure depends on instance type, AZ and time, so check the Spot Instance Advisor or the price history command above for the types you actually use.

How much warning do you get before a Spot Instance is interrupted?

Two minutes, through EventBridge and instance metadata. A rebalance recommendation often arrives earlier, but it’s best effort and can come with the notice.

What is the best allocation strategy for Spot Instances?

AWS recommends price-capacity-optimized for Auto Scaling groups and EC2 Fleet, and its equivalent in Batch and EMR. Avoid lowest-price.

Can I use Spot Instances for production?

Yes, for stateless or fault-tolerant tiers that are diversified across many instance types and AZs, with an on-demand base for capacity you can’t lose. Keep stateful systems on on-demand capacity.

Do Savings Plans apply to Spot Instances?

No. Savings Plans cover on-demand usage. Spot already carries its own discount.

Pick one boring workload

Choose something that already restarts cleanly, like a CI runner fleet or a queue worker group. Run the Auto Scaling check above, move it to price-capacity-optimized with at least ten instance types, keep Capacity Rebalancing on and wire up an interruption handler. Then leave it alone for two weeks and read the interruption events before moving anything else. If the fleet underneath is still oversized, fix that first with our rightsizing and Graviton guide, because a smaller Spot instance is cheaper than a big one.

Picking the second workload is where Kostrix can help: its Spot check lists on-demand instances that already sit in Auto Scaling groups or carry a dev or test environment tag, priced at the current Spot rate for their type.

All articles