From d0eefdd06355c1588e542c1b9522fde8716c9627 Mon Sep 17 00:00:00 2001 From: Georgiana Dolocan Date: Tue, 26 Nov 2024 14:41:16 +0200 Subject: [PATCH 1/6] Enable relevant tags --- terraform/aws/projects/template.tfvars | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/terraform/aws/projects/template.tfvars b/terraform/aws/projects/template.tfvars index 055c983c1..3243f37cb 100644 --- a/terraform/aws/projects/template.tfvars +++ b/terraform/aws/projects/template.tfvars @@ -40,3 +40,14 @@ filestores = { # }, # }, {% endfor %} + +# Enable cost allocation tags +# This will work for for standalone AWS accounts, +# but fail for member accounts part of an organization that we don't manage. +active_cost_allocation_tags = [ + "2i2c:hub-name", + "2i2c.org/cluster-name", + "alpha.eksctl.io/cluster-name", + "kubernetes.io/cluster/{var_cluster_name}", + "kubernetes.io/created-for/pvc/namespace", +] \ No newline at end of file From b6d53109b0e0202ea7e18f657ec27305e8ce916c Mon Sep 17 00:00:00 2001 From: Georgiana Dolocan Date: Tue, 26 Nov 2024 14:41:46 +0200 Subject: [PATCH 2/6] Add topic about cost attribution system --- docs/topic/billing/cost-attribution-system.md | 89 +++++++++++++++++++ docs/topic/billing/index.md | 1 + 2 files changed, 90 insertions(+) create mode 100644 docs/topic/billing/cost-attribution-system.md diff --git a/docs/topic/billing/cost-attribution-system.md b/docs/topic/billing/cost-attribution-system.md new file mode 100644 index 000000000..fafb9b239 --- /dev/null +++ b/docs/topic/billing/cost-attribution-system.md @@ -0,0 +1,89 @@ +# Cost Attribution System + +The Cost Attribution System is designed to monitor and attribute cloud +infrastructure costs to specific hub deployments managed. This system +integrates with the [AWS Cost Explorer +API](https://docs.aws.amazon.com/cost-management/latest/userguide/ce-api.html) +to provide detailed cost insights within a hub's Grafana dashboard. + +{{% callout note %}} Note that this feature is currently available to AWS +hosted hubs only and will be rolled out to other cloud providers in the +future. {{% /callout %}} + +## Components + +1. AWS IAM Role Configuration + +A [dedicated IAM +role](https://github.com/2i2c-org/infrastructure/blob/main/terraform/aws/aws-ce-grafana-backend-iam.tf) +is created to grant the necessary permissions for accessing the Cost Explorer +API. + +2. Python Web Server + +A [Python-based web +server](https://github.com/2i2c-org/infrastructure/tree/main/helm-charts/images/aws-ce-grafana-backend) +is deployed to interact with the Cost Explorer API. It retrieves cost data +and serves it as JSON, making it accessible for Grafana. + +3. Grafana Integration + +A custom Helm chart, +[`aws-ce-grafana-backend`](https://github.com/2i2c-org/infrastructure/tree/main/helm-charts/aws-ce-grafana-backend), +is introduced to facilitate the deployment of the Python web server alongside +Grafana. + +This enables Grafana to query the web server for cost data, allowing users to +visualize and analyze cloud expenses directly within the Grafana interface. + +It +[uses](https://github.com/2i2c-org/infrastructure/blob/48e06a02a411e31b03db2f30fd6a090b5f6eeeb5/helm-charts/support/values.yaml#L405-L406) +the [Infinity Grafana +plugin](https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/) +to serve JSON from AWS Cost Explorer API, for use by Grafana dashboard +panels. + +## Technical implementation details + +The system relies on _at least one of these tags_ to be on any cloud infra to +attribute cost to. + +- `2i2c.org/cluster-name` +- `alpha.eksctl.io/cluster-name` +- `kubernetes.io/cluster/` + +```{important} +Currently, on clusters that have a k8s version greater or equal with 1.30, +terraform managed resources already have the `2i2c.org/cluster-name` +tag configured via the `default_tags` variable, and eksctl managed resources +already have the tag configured for node groups via `nodegroup.libsonnet`. + +On clusters that have a k8s version less than 1.30, eksctl managed resources, +the `alpha.eksctl.io/cluster-name` and `kubernetes.io/cluster/` +tags are present and used instead. + +New clusters have _all_ eksctl managed resources configured to be tagged, not +just the node groups. This isn't important to ensure for existing clusters' +cost attribution though. +``` + +The system also relies on the tag `2i2c:hub-name` to be specified in addition to +the tags above for any cloud infra tied to specific hubs. + +We only need to ensure the `2i2c.org/cluster-name` and `2i2c:hub-name` tags are +declared, the others are applied by `eksctl` and Kubernetes controllers that can +create cloud resources to represent k8s resources (block storage volumes for k8s +PV resources referencing certain storage classes, and load balancers for k8s +Service's of type LoadBalancer). + +The following resources are known to be hub specific in some cases and known +to incur costs. + +- **S3 buckets** in terraform +- **EFS storage** in terraform +- **EBS volumes** in terraform +- **Node groups** in eksctl + +```{important} +There are still some clusters that don't have separate EFS storage per hub yet. +``` diff --git a/docs/topic/billing/index.md b/docs/topic/billing/index.md index c4e13941d..52052a4a8 100644 --- a/docs/topic/billing/index.md +++ b/docs/topic/billing/index.md @@ -10,4 +10,5 @@ accounts reports tools budget-alerts +cost-attribution-system ``` From 5095b2725cfa473aae8023a03609b58ee7bd4d1d Mon Sep 17 00:00:00 2001 From: Georgiana Dolocan Date: Tue, 26 Nov 2024 14:55:30 +0200 Subject: [PATCH 3/6] Turn setup into a comment as this is not general config --- terraform/aws/projects/template.tfvars | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/terraform/aws/projects/template.tfvars b/terraform/aws/projects/template.tfvars index 3243f37cb..6f12f067b 100644 --- a/terraform/aws/projects/template.tfvars +++ b/terraform/aws/projects/template.tfvars @@ -41,13 +41,13 @@ filestores = { # }, {% endfor %} -# Enable cost allocation tags -# This will work for for standalone AWS accounts, -# but fail for member accounts part of an organization that we don't manage. -active_cost_allocation_tags = [ - "2i2c:hub-name", - "2i2c.org/cluster-name", - "alpha.eksctl.io/cluster-name", - "kubernetes.io/cluster/{var_cluster_name}", - "kubernetes.io/created-for/pvc/namespace", -] \ No newline at end of file +# Uncomment the lines below to Enable cost allocation tags +# for standalone AWS accounts + +# active_cost_allocation_tags = [ +# "2i2c:hub-name", +# "2i2c.org/cluster-name", +# "alpha.eksctl.io/cluster-name", +# "kubernetes.io/cluster/{var_cluster_name}", +# "kubernetes.io/created-for/pvc/namespace", +# ] \ No newline at end of file From 57d0e71d7b8de0cf0e5373275d8f5b7e0c6f5666 Mon Sep 17 00:00:00 2001 From: Georgiana Dolocan Date: Tue, 26 Nov 2024 14:55:53 +0200 Subject: [PATCH 4/6] Slim dowmn the how to enable cost allocation section --- docs/howto/cost-attribution/aws.md | 103 ++---------------- docs/topic/billing/cost-attribution-system.md | 1 + 2 files changed, 11 insertions(+), 93 deletions(-) diff --git a/docs/howto/cost-attribution/aws.md b/docs/howto/cost-attribution/aws.md index 18fe77fb2..4a94aec99 100644 --- a/docs/howto/cost-attribution/aws.md +++ b/docs/howto/cost-attribution/aws.md @@ -1,85 +1,14 @@ (howto:cost-attribution:enable-aws)= # Enable AWS cost attribution system -```{important} -Until the [epic to roll out a cost attribution system in all AWS -clusters](https://github.com/2i2c-org/infrastructure/issues/4872) is finalized, -this documentation is subject to active change. -``` - The AWS cost attribution system, referenced as the system in this document, consists of the `aws-ce-grafana-backend` Helm chart and a Grafana dashboard using the backend as a data source. -## Practical steps - -### 1. Tag cloud infra - -The system relies on _at least one of these tags_ to be on any cloud infra to -attribute cost to. - -- `2i2c.org/cluster-name` -- `alpha.eksctl.io/cluster-name` -- `kubernetes.io/cluster/` - -```{important} -Currently, on clusters that have a k8s version greater or equal with 1.30, -terraform managed resources already have the `2i2c.org/cluster-name` -tag configured via the `default_tags` variable, and eksctl managed resources -already have the tag configured for node groups via `nodegroup.libsonnet`. - -On clusters that have a k8s version less than 1.30, eksctl managed resources, -the `alpha.eksctl.io/cluster-name` and `kubernetes.io/cluster/` -tags are present and used instead. - -New clusters have _all_ eksctl managed resources configured to be tagged, not -just the node groups. This isn't important to ensure for existing clusters' -cost attribution though. -``` - -The system also relies on the tag `2i2c:hub-name` to be specified in addition to -the tags above for any cloud infra tied to specific hubs. - -We only need to ensure the `2i2c.org/cluster-name` and `2i2c:hub-name` tags are -declared, the others are applied by `eksctl` and Kubernetes controllers that can -create cloud resources to represent k8s resources (block storage volumes for k8s -PV resources referencing certain storage classes, and load balancers for k8s -Service's of type LoadBalancer). - -1. _Configure `2i2c:hub-name` tags_ +Checkout the docs at [](topic:billing:cost-attribution) for more information on +the system. - For any resource _specific to a hub_, declare an additional tag - `2i2c:hub-name=`. If this isn't done, they will be listed under a - `shared` instead. - - The following resources are known to be hub specific in some cases and known - to incur costs. - - - **S3 buckets** in terraform - - **EFS storage** in terraform - - **EBS volumes** in terraform - - **Node groups** in eksctl - - ```{important} - If EFS, EBS or nodegroups are not split based on the hub they're deployed to - and instead they are shared by the entire cluster, then splitting it in order - to add the `2i2c:hub-name` tag is an opt-in feature because the split incurs - additional cloud costs and startup times for communities. - - See the following GitHub issue for additional context - https://github.com/2i2c-org/infrastructure/issues/4928#issuecomment-2417091407 - ``` - - Search and mimic configuration of other clusters to understand how to - configure the `2i2c:hub-name` tags for specific cloud infra types. - -2. _Apply changes_ - - 1. If you changed anything in terraform, apply those changes. - 2. If you changed anything in eksctl, apply those changed by re-creating - those resources. - -### 2. Enable cost allocation tags +### 1. Enable cost allocation tags Enabling cost allocation tags via terraform can be done for standalone AWS accounts, but not for member accounts part of an organization that we don't manage. @@ -90,22 +19,10 @@ Due to this, we'll provide separate ways of doing this depending on the situatio ````{tab-item} Standalone account :sync: standalone -In standalone accounts, we should have relevant permissions. Use terraform to -enable relevant tags to also function as a cost allocation tags. - -Configure the following terraform variable like below and apply the changes. - -``` -active_cost_allocation_tags = [ - "2i2c:hub-name", - "2i2c.org/cluster-name", - "alpha.eksctl.io/cluster-name", - "kubernetes.io/cluster/{var_cluster_name}", - "kubernetes.io/created-for/pvc/namespace", -] -``` +The relevant tags are already present in the terraform template used to generate +the new cluster and will be applied when creating the cluster. -The apply operation could fail with the following errors: +If the apply operation fails with the following errors: 1. _Tag keys not found_ @@ -178,7 +95,7 @@ not used by `aws-ce-grafana-backend`, as it could help us attribute cost for storage disks dynamically provisioned in case that's relevant in the future. ``` -### 3. (optional) Backfill billing data +### 2. (optional) Backfill billing data You can optionally backfill billing data to tags having been around for a while but not enabled as cost allocation tags. @@ -188,15 +105,15 @@ process the request. Make a request through the AWS web console by navigating to "Cost allocation tags" under "Billing and Cost Management", then from there click the "Backfill tags" button. -### 4. Install `aws-ce-grafana-backend` +### 3. Install `aws-ce-grafana-backend` -In the cluster's terraform variables, declare and apply the following: +In the cluster's terraform variables, make sure the following is present: ``` enable_aws_ce_grafana_backend_iam = true ``` -Look at the terraform output named `aws_ce_grafana_backend_k8s_sa_annotation`: +After applying this, look at the terraform output named `aws_ce_grafana_backend_k8s_sa_annotation`: ``` terraform output -raw aws_ce_grafana_backend_k8s_sa_annotation diff --git a/docs/topic/billing/cost-attribution-system.md b/docs/topic/billing/cost-attribution-system.md index fafb9b239..8d8c517db 100644 --- a/docs/topic/billing/cost-attribution-system.md +++ b/docs/topic/billing/cost-attribution-system.md @@ -1,3 +1,4 @@ +(topic:billing:cost-attribution)= # Cost Attribution System The Cost Attribution System is designed to monitor and attribute cloud From 53486d32562eff66376cd0ffa0734c2c37a71226 Mon Sep 17 00:00:00 2001 From: Georgiana Dolocan Date: Tue, 26 Nov 2024 15:02:57 +0200 Subject: [PATCH 5/6] Small adjustements to content --- docs/topic/billing/cost-attribution-system.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/topic/billing/cost-attribution-system.md b/docs/topic/billing/cost-attribution-system.md index 8d8c517db..1518d6da8 100644 --- a/docs/topic/billing/cost-attribution-system.md +++ b/docs/topic/billing/cost-attribution-system.md @@ -2,10 +2,10 @@ # Cost Attribution System The Cost Attribution System is designed to monitor and attribute cloud -infrastructure costs to specific hub deployments managed. This system -integrates with the [AWS Cost Explorer +infrastructure costs to 2i2c hub deployments. This system integrates with the +[AWS Cost Explorer API](https://docs.aws.amazon.com/cost-management/latest/userguide/ce-api.html) -to provide detailed cost insights within a hub's Grafana dashboard. +to provide detailed cost insights from a hub's Grafana dashboard. {{% callout note %}} Note that this feature is currently available to AWS hosted hubs only and will be rolled out to other cloud providers in the From cf315571d1a5815c9b0cdfb1ebf9235c42ca8b27 Mon Sep 17 00:00:00 2001 From: Georgiana Dolocan Date: Tue, 26 Nov 2024 15:10:03 +0200 Subject: [PATCH 6/6] Rm steps headline --- docs/howto/cost-attribution/aws.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/howto/cost-attribution/aws.md b/docs/howto/cost-attribution/aws.md index 4a94aec99..b9dd11d09 100644 --- a/docs/howto/cost-attribution/aws.md +++ b/docs/howto/cost-attribution/aws.md @@ -8,6 +8,8 @@ using the backend as a data source. Checkout the docs at [](topic:billing:cost-attribution) for more information on the system. +## Steps + ### 1. Enable cost allocation tags Enabling cost allocation tags via terraform can be done for standalone AWS