An open source tool to autoscale Memorystore Cluster instances
Home
·
Poller component
·
Scaler component
·
Forwarder component
·
Terraform configuration
·
Monitoring
The Cloud Memorystore Cluster Autoscaler is a companion tool that allows you to automatically increase or reduce the number of nodes/shards in one or more Memorystore Cluster instances, based on their utilization.
When you create a Memorystore Cluster instance, you choose the number of shards/nodes that provide compute resources for the instance.
The Autoscaler monitors your instances and automatically adds or removes capacity to ensure that the memory, CPU utilization, and other metrics remain within recommend limits.
If you would like to get started quickly with a test deployment of the Autoscaler, you can deploy to Cloud Run functions in a single project.
The diagram above shows the high level components of the Autoscaler and the interaction flow:
-
The Autoscaler consists of two main decoupled components:
These can be deployed to Cloud Run functions and configured so that the Autoscaler runs according to a user-defined schedule.
-
At the specified time and frequency, the Poller component queries the Cloud Monitoring API to retrieve the utilization metrics for each Memorystore Cluster instance.
-
For each instance, the Poller component pushes one message to the Scaler component. The payload contains the utilization metrics for the specific Memorystore Cluster instance, and some of its corresponding configuration parameters.
-
Using the chosen scaling method, the Scaler compares the cluster instance metrics against the recommended thresholds, (plus or minus an allowed margin), and determines if the instance should be scaled, and the number of shards/nodes that it should be scaled to. If the configured cooldown period has passed, then the Scaler component requests the cluster to scale out or in.
Throughout the flow, the Autoscaler writes a step by step summary of its recommendations and actions to Cloud Logging for tracking and auditing.
To deploy the Autoscaler, decide which of the following strategies is best adjusted to fulfill your technical and operational needs:
In both of the above cases, the Google Cloud Platform resources are deployed using Terraform. Please see the Terraform instructions for more information on the deployment options available.
You can find some recommendations for productionizing deployment of the Autoscaler in the Productionization section of the Terraform documentation.
The parameters for configuring the Autoscaler are identical regardless of the chosen deployment type, but the mechanism for configuration differs slightly:
In the case of the Cloud Run functions deployment, the parameters are defined using the JSON payload of the PubSub message that is published by the Cloud Scheduler job.
In the case of the Kubernetes deployment, the parameters are defined using a Kubernetes ConfigMap that is loaded by the Cron job.
You can find the details about the parameters and their default values in the Poller component page.
There is also a browser-based configuration file editor and a command line configuration file validator.
Copyright 2024 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
The Autoscaler project is based on open source contributions (see Contributing).
Please note that this is not an officially supported Google product.