Ask GitHub to retry an action because runner is going away

[gautamg795]

We run our Actions runners on AWS Spot instances, which are subject to termination with a 2-minute notice. Although infrequent, termination like this usually means the job just suddenly fails and the developer triggering the job often has no idea why.
It'd be great if runners could, upon notification from AWS of impending termination, run some command to notify GitHub that the job is going to fail and needs to be restarted.

Anyone have existing solutions for runners on Spot instances?

[booleanbetrayal]

@gautamg795 - Did you ever find some graceful out in this spot termination scenario?

[gautamg795]

Unfortunately no. I've just had to increase my CI spend to pick instances that are less likely to be terminated, which has been an imperfect solution.

For starters I'm guessing it'd help to have a daemon checking for interruption notices and at least killing the runner process — I bet this would, at the very least, cause the job to show up as "failed" rather than timing out after some time of inactivity. But I don't think there's a good way to handle this yet

[crohr]

RunsOn properly detects spot instance termination since v2.2, so at least the job status is seen as failed in the GitHub UI. I plan on implementing an (opt-in) retry mechanism soon as well.

[gautamg795]

@crohr any pointers on how to notify GitHub the job has failed if we're doing "bespoke" self-hosted runners? Something polling for a spot interruption notice on the machine?

[crohr]

@gautamg795 Yes you need to poll http://169.254.169.254/latest/meta-data/spot/instance-action for the termination action. When this happens, send a signal to the runner agent to kill it, and then add other steps if needed like auto-retrying the job.

[booleanbetrayal]

@gautamg795 - We have also started observing these centrally via an EventBridge Rule, and are seeing them occur more frequently than anticipated. This is modeled off the same Event stream that AWS uses in a EKS Managed Node Groups + Spot Instances default wiring:

resource "aws_cloudwatch_event_rule" "spot_termination" {
  name        = "spot-termination"
  description = "Capture SPOT termination events and publish to SNS"

  event_pattern = jsonencode({
    detail-type = [
      "EC2 Instance Rebalance Recommendation",
      "EC2 Spot Instance Interruption Warning"
    ]
    source = ["aws.ec2"]
  })
}

resource "aws_cloudwatch_event_target" "spot_termination" {
  rule      = aws_cloudwatch_event_rule.spot_termination.name
  target_id = "SendToSNS"
  arn       = var.sns_topic_arn
}

I can't tell you how nice some sort of Retry-Scheduling GitHub Actions API support / awareness from runner through controller (ARC) would be, as this would otherwise be an ideal use case for Spot Instances.