Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure the Autoscaling Group to use Spot Fleet to enable CloudWatch Spot metrics #1053

Open
RemyDeWolf opened this issue Oct 11, 2022 · 2 comments

Comments

@RemyDeWolf
Copy link

Is your feature request related to a problem? Please describe.
We would like to have CloudWatch metrics about our Spot instances such as FulfilledCapacity and TargetCapacity we can automatically detect Spot outages. (AWS reference for these metrics)

Currently, the limitation with this stack, is the Autoscaling Group is configured to launch "Spot Instance", and AWS doesn't provide any Spot metrics for these.
If we want the metrics, the Autoscaling Group should be configured to leverages a "Spot Fleet"
This is entirely different from creating an Autoscaling Group that leverages a Spot Fleet.

Describe the solution you'd like
Expose a parameter UseSpotFleet, default false, to configure the load balancer to use a Spot Fleet.
Here is the documentation on how to configure Spot Fleet on an autoscaler.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-automatic-scaling.html

Describe alternatives you've considered
If we had a fallback to use OnDemand instances when Spot is not available, we would not need to implement this.
#851

Additional context
We set OnDemandPercentage=0 to save on cost, and it works most of the time. But we experience some spot outage every few days and we would like to automatically fall back to OnDemandPercentage=100` when we detect the spot outage. We would write a custom lambda to update the OnDemandPercentage when we detect an outage. Having no CloudWatch metrics make it hard for us to automate this process.

@sj26
Copy link
Member

sj26 commented Oct 11, 2022

It looks like there was some earlier work investigating using spot fleets in #269, but it was never finished.

@triarius
Copy link
Contributor

Hi @RemyDeWolf, we've had a look at what we would need to make this change, and it looks like enabling the use of a spot fleet instead of an ASG is not as straight forward a change for us to take on at this time. Furthermore, we would also want to update https://github.com/buildkite/buildkite-agent-scaler/ to work with spot fleets at the same time as well.

However, this is a feature we would very much support the contribution of PRs for. So please don't hesitate to have a shot at it yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants