Skip to content

Commit

Permalink
Merge branch 'http2-lading' of github.com:DataDog/datadog-agent into …
Browse files Browse the repository at this point in the history
…http2-lading
  • Loading branch information
DDuongNguyen committed Dec 3, 2024
2 parents 78f00bc + c0c0ef8 commit 0610542
Show file tree
Hide file tree
Showing 929 changed files with 24,892 additions and 13,664 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ experimental:
templates:
job_template: &job_template
docker:
- image: gcr.io/datadoghq/agent-circleci-runner:v48372186-ff395e52
- image: gcr.io/datadoghq/agent-circleci-runner:v50094600-7b86dd80
environment:
USE_SYSTEM_LIBS: "1"
working_directory: /go/src/github.com/DataDog/datadog-agent
Expand Down
7 changes: 6 additions & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
/Makefile.trace @DataDog/agent-delivery

/mkdocs.yml @DataDog/agent-devx-infra
/release.json @DataDog/agent-delivery @DataDog/agent-metrics-logs @DataDog/windows-kernel-integrations @DataDog/agent-release-management @DataDog/agent-security
/release.json @DataDog/agent-delivery @DataDog/agent-metrics-logs @DataDog/windows-kernel-integrations @DataDog/agent-security
/requirements.txt @DataDog/agent-devx-infra
/pyproject.toml @DataDog/agent-devx-infra @DataDog/agent-devx-loops
/repository.datadog.yml @DataDog/agent-devx-infra
Expand All @@ -45,6 +45,8 @@
/static-analysis.datadog.yml @DataDog/software-integrity-and-trust @DataDog/agent-devx-infra

/modules.yml @DataDog/agent-shared-components
# if go.work changes then either .go-version or modules.yml changed too, so ASC might as well own it
/go.work @DataDog/agent-shared-components

/.circleci/ @DataDog/agent-devx-infra

Expand Down Expand Up @@ -465,6 +467,8 @@
/pkg/util/pdhutil/ @DataDog/windows-agent
/pkg/util/winutil/ @DataDog/windows-agent
/pkg/util/testutil/flake @DataDog/agent-devx-loops
/pkg/util/testutil/patternscanner.go @DataDog/universal-service-monitoring @DataDog/ebpf-platform
/pkg/util/testutil/docker @DataDog/universal-service-monitoring @DataDog/ebpf-platform
/pkg/util/trie @DataDog/container-integrations
/pkg/languagedetection @DataDog/processes @DataDog/universal-service-monitoring
/pkg/linters/ @DataDog/agent-devx-loops
Expand Down Expand Up @@ -609,6 +613,7 @@
/test/new-e2e/tests/agent-subcommands @DataDog/agent-shared-components
/test/new-e2e/tests/containers @DataDog/container-integrations @DataDog/container-platform
/test/new-e2e/tests/discovery @DataDog/universal-service-monitoring
/test/new-e2e/tests/ha-agent @DataDog/ndm-core
/test/new-e2e/tests/language-detection @DataDog/processes
/test/new-e2e/tests/ndm @DataDog/ndm-core
/test/new-e2e/tests/ndm/netflow @DataDog/ndm-integrations
Expand Down
12 changes: 11 additions & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,17 @@

### Motivation

### Describe how to test/QA your changes
### Describe how you validated your changes
<!--
Validate your changes before merge, ensuring that:
* Your PR is tested by static / unit / integrations / e2e tests
* Your PR description details which e2e tests cover your changes, if any
* The PR description contains details of how you validated your changes. If you validated changes manually and not through automated tests, add context on why automated tests did not fit your changes validation.
If you want additional validation by a second person, you can ask reviewers to do it. Describe how to set up an environment for manual tests in the PR description. Manual validation is expected to happen on every commit before merge.
Any manual validation step should then map to an automated test. Manual validation should not substitute automation, minus exceptions not supported by test tooling yet.
-->

### Possible Drawbacks / Trade-offs

Expand Down
30 changes: 30 additions & 0 deletions .github/workflows/assign_issue.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
name: "Assign issue to a team"

on:
issues:
types: [opened, reopened]

jobs:
auto_assign_issue:
runs-on: ubuntu-latest
container:
image: ghcr.io/datadog/agent-issue-auto-assign:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
permissions:
packages: read
issues: write
steps:
- name: Checkout repository
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4.1.4
- name: Install dependencies
# Dependencies are installed at runtime. Otherwise it would create a huge image see https://hub.docker.com/r/pytorch/pytorch/tags
run: |
pip install --upgrade pip && pip install --no-compile --no-cache-dir torch transformers invoke codeowners slack-sdk PyGithub python-gitlab semver
- name: Assign issue
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
inv -e issue.assign-owner --issue ${{ github.event.issue.number }}
4 changes: 2 additions & 2 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
go-version-file: ".go-version"

- name: Initialize CodeQL
uses: github/codeql-action/init@4f3212b61783c3c68e8309a0f18a699764811cda # v3.27.1
uses: github/codeql-action/init@f09c1c0a94de965c15400f5634aa42fac8fb8f88 # v3.27.5
with:
languages: ${{ matrix.language }}
config: |
Expand All @@ -67,4 +67,4 @@ jobs:
invoke agent.build --build-exclude=systemd
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@4f3212b61783c3c68e8309a0f18a699764811cda # v3.27.1
uses: github/codeql-action/analyze@f09c1c0a94de965c15400f5634aa42fac8fb8f88 # v3.27.5
72 changes: 72 additions & 0 deletions .github/workflows/collector-generate-and-update.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
name: Update OTel Collector Dependencies and Generate OTel Agent

on:
schedule:
- cron: '0 12 * * 3'
workflow_dispatch:

jobs:
update-and-generate:
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- name: Set date
id: date
run: echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT

- name: Checkout repository
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4.1.4

- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: '3.12.6'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run update task
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: inv -e collector.update

- name: Run generate task
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: inv -e collector.generate

- name: Check for changes
id: check_changes
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git add .
if git diff-index --quiet HEAD; then
echo "No changes detected"
echo "changes_detected=false" >> $GITHUB_OUTPUT
else
echo "Changes detected"
echo "changes_detected=true" >> $GITHUB_OUTPUT
fi
- name: Commit changes
if: steps.check_changes.outputs.changes_detected == 'true'
run: |
git switch -c update-otel-collector-dependencies-${{ steps.date.outputs.date }}
git commit -m "Update OTel Collector dependencies and generate OTel Agent"
git push -u origin update-otel-collector-dependencies-${{ steps.date.outputs.date }}
- name: Install GitHub CLI
if: steps.check_changes.outputs.changes_detected == 'true'
run: |
sudo apt-get update
sudo apt-get install gh
- name: Create draft pull request
if: steps.check_changes.outputs.changes_detected == 'true'
run: |
gh auth login --with-token <<< ${{ secrets.GITHUB_TOKEN }}
gh pr create --title "Update OTel collector dependencies" --body "This PR updates the OTel Collector dependencies to the latest version. Please ensure that all tests pass before marking ready for review." --base main --head update-otel-collector-dependencies-${{ steps.date.outputs.date }} --draft
31 changes: 15 additions & 16 deletions .github/workflows/label-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,25 +102,24 @@ jobs:
if: github.triggering_actor != 'dd-devflow[bot]'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4.1.4
with:
persist-credentials: false
- name: Setup Python3
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.12.6"
cache: "pip"
cache-dependency-path: '**/requirements*.txt'
- name: Install python dependencies
run: pip3 install -r requirements.txt
- name: Check qa/[done|no-code-change] labels are not set together
run: |
is_qa_done=1
is_qa_no_code_change=1
for label in $LABELS; do
if [[ "$label" == "qa/done" ]]; then
is_qa_done=0
fi
if [[ "$label" == "qa/no-code-change" ]]; then
is_qa_no_code_change=0
fi
done
if [ $is_qa_done -eq 0 ] && [ $is_qa_no_code_change -eq 0 ]; then
echo "Both 'qa/done' and 'qa/no-code-change' labels are set -- only one of them should be set"
exit 1
fi
echo "No issue with 'qa/done' and 'qa/no-code-change' labels"
env:
LABELS: ${{ needs.fetch-labels.outputs.LABELS}}
run: |
inv -e github.check-qa-labels --labels "${LABELS[@]}"
agenttelemetry-list-change-ack-check:
if: github.triggering_actor != 'dd-devflow[bot]'
runs-on: ubuntu-latest
Expand Down
7 changes: 4 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -186,9 +186,6 @@ pkg/network/netlink/testdata/message_dump*
# cscope
cscope.out

# Go workspace until we have proper support in the tooling
go.work*

tools/windows/DatadogAgentInstaller/.vs/
tools/windows/DatadogAgentInstaller/packages/
tools/windows/DatadogAgentInstaller/WixSetup/cabcache/
Expand Down Expand Up @@ -220,3 +217,7 @@ ebpf-calculator

# File generated by job creating flake finder pipeline
flake-finder-gitlab-ci.yml


# go.work.sum is always changing, we should not need it since we tidy each module individually
go.work.sum
97 changes: 33 additions & 64 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -173,49 +173,49 @@ variables:
# To use images from datadog-agent-buildimages dev branches, set the corresponding
# SUFFIX variable to _test_only
DATADOG_AGENT_BUILDIMAGES_SUFFIX: ""
DATADOG_AGENT_BUILDIMAGES: v48372186-ff395e52
DATADOG_AGENT_BUILDIMAGES: v50094600-7b86dd80
DATADOG_AGENT_WINBUILDIMAGES_SUFFIX: ""
DATADOG_AGENT_WINBUILDIMAGES: v48372186-ff395e52
DATADOG_AGENT_WINBUILDIMAGES: v50094600-7b86dd80
DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX: ""
DATADOG_AGENT_ARMBUILDIMAGES: v48372186-ff395e52
DATADOG_AGENT_ARMBUILDIMAGES: v50094600-7b86dd80
DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX: ""
DATADOG_AGENT_SYSPROBE_BUILDIMAGES: v48372186-ff395e52
DATADOG_AGENT_SYSPROBE_BUILDIMAGES: v50094600-7b86dd80
DATADOG_AGENT_BTF_GEN_BUILDIMAGES_SUFFIX: ""
DATADOG_AGENT_BTF_GEN_BUILDIMAGES: v48372186-ff395e52
DATADOG_AGENT_BTF_GEN_BUILDIMAGES: v50094600-7b86dd80
# New images to enable different version per image - not used yet
CI_IMAGE_BTF_GEN: v48372186-ff395e52
CI_IMAGE_BTF_GEN: v50094600-7b86dd80
CI_IMAGE_BTF_GEN_SUFFIX: ""
CI_IMAGE_DEB_X64: v48372186-ff395e52
CI_IMAGE_DEB_X64: v50094600-7b86dd80
CI_IMAGE_DEB_X64_SUFFIX: ""
CI_IMAGE_DEB_ARM64: v48372186-ff395e52
CI_IMAGE_DEB_ARM64: v50094600-7b86dd80
CI_IMAGE_DEB_ARM64_SUFFIX: ""
CI_IMAGE_DEB_ARMHF: v48372186-ff395e52
CI_IMAGE_DEB_ARMHF: v50094600-7b86dd80
CI_IMAGE_DEB_ARMHF_SUFFIX: ""
CI_IMAGE_DD_AGENT_TESTING: v48372186-ff395e52
CI_IMAGE_DD_AGENT_TESTING: v50094600-7b86dd80
CI_IMAGE_DD_AGENT_TESTING_SUFFIX: ""
CI_IMAGE_DOCKER_X64: v48372186-ff395e52
CI_IMAGE_DOCKER_X64: v50094600-7b86dd80
CI_IMAGE_DOCKER_X64_SUFFIX: ""
CI_IMAGE_DOCKER_ARM64: v48372186-ff395e52
CI_IMAGE_DOCKER_ARM64: v50094600-7b86dd80
CI_IMAGE_DOCKER_ARM64_SUFFIX: ""
CI_IMAGE_GITLAB_AGENT_DEPLOY: v48372186-ff395e52
CI_IMAGE_GITLAB_AGENT_DEPLOY: v50094600-7b86dd80
CI_IMAGE_GITLAB_AGENT_DEPLOY_SUFFIX: ""
CI_IMAGE_LINUX_GLIBC_2_17_X64: v48372186-ff395e52
CI_IMAGE_LINUX_GLIBC_2_17_X64: v50094600-7b86dd80
CI_IMAGE_LINUX_GLIBC_2_17_X64_SUFFIX: ""
CI_IMAGE_LINUX_GLIBC_2_23_ARM64: v48372186-ff395e52
CI_IMAGE_LINUX_GLIBC_2_23_ARM64: v50094600-7b86dd80
CI_IMAGE_LINUX_GLIBC_2_23_ARM64_SUFFIX: ""
CI_IMAGE_SYSTEM_PROBE_X64: v48372186-ff395e52
CI_IMAGE_SYSTEM_PROBE_X64: v50094600-7b86dd80
CI_IMAGE_SYSTEM_PROBE_X64_SUFFIX: ""
CI_IMAGE_SYSTEM_PROBE_ARM64: v48372186-ff395e52
CI_IMAGE_SYSTEM_PROBE_ARM64: v50094600-7b86dd80
CI_IMAGE_SYSTEM_PROBE_ARM64_SUFFIX: ""
CI_IMAGE_RPM_X64: v48372186-ff395e52
CI_IMAGE_RPM_X64: v50094600-7b86dd80
CI_IMAGE_RPM_X64_SUFFIX: ""
CI_IMAGE_RPM_ARM64: v48372186-ff395e52
CI_IMAGE_RPM_ARM64: v50094600-7b86dd80
CI_IMAGE_RPM_ARM64_SUFFIX: ""
CI_IMAGE_RPM_ARMHF: v48372186-ff395e52
CI_IMAGE_RPM_ARMHF: v50094600-7b86dd80
CI_IMAGE_RPM_ARMHF_SUFFIX: ""
CI_IMAGE_WIN_1809_X64: v48372186-ff395e52
CI_IMAGE_WIN_1809_X64: v50094600-7b86dd80
CI_IMAGE_WIN_1809_X64_SUFFIX: ""
CI_IMAGE_WIN_LTSC2022_X64: v48372186-ff395e52
CI_IMAGE_WIN_LTSC2022_X64: v50094600-7b86dd80
CI_IMAGE_WIN_LTSC2022_X64_SUFFIX: ""

DATADOG_AGENT_EMBEDDED_PATH: /opt/datadog-agent/embedded
Expand All @@ -236,58 +236,16 @@ variables:

# Start aws ssm variables
# They must be defined as environment variables in the GitLab CI/CD settings, to ease rotation if needed
AGENT_QA_PROFILE: ci.datadog-agent.agent-qa-profile # agent-devx-infra
API_KEY_ORG2: ci.datadog-agent.datadog_api_key_org2 # agent-devx-infra
API_KEY_DDDEV: ci.datadog-agent.datadog_api_key # agent-devx-infra
APP_KEY_ORG2: ci.datadog-agent.datadog_app_key_org2 # agent-devx-infra
CHANGELOG_COMMIT_SHA: ci.datadog-agent.gitlab_changelog_commit_sha # agent-devx-infra
CHOCOLATEY_API_KEY: ci.datadog-agent.chocolatey_api_key # windows-agent
CODECOV_TOKEN: ci.datadog-agent.codecov_token # agent-devx-infra
DEB_GPG_KEY: ci.datadog-agent.deb_signing_private_key_${DEB_GPG_KEY_ID} # agent-delivery
DEB_SIGNING_PASSPHRASE: ci.datadog-agent.deb_signing_key_passphrase_${DEB_GPG_KEY_ID} # agent-delivery
DOCKER_REGISTRY_LOGIN: ci.datadog-agent.docker_hub_login # container-integrations
DOCKER_REGISTRY_PWD: ci.datadog-agent.docker_hub_pwd # container-integrations
E2E_TESTS_API_KEY: ci.datadog-agent.e2e_tests_api_key # agent-devx-loops
E2E_TESTS_APP_KEY: ci.datadog-agent.e2e_tests_app_key # agent-devx-loops
E2E_TESTS_RC_KEY: ci.datadog-agent.e2e_tests_rc_key # agent-devx-loops
E2E_TESTS_AZURE_CLIENT_ID: ci.datadog-agent.e2e_tests_azure_client_id # agent-devx-loops
E2E_TESTS_AZURE_CLIENT_SECRET: ci.datadog-agent.e2e_tests_azure_client_secret # agent-devx-loops
E2E_TESTS_AZURE_TENANT_ID: ci.datadog-agent.e2e_tests_azure_tenant_id # agent-devx-loops
E2E_TESTS_AZURE_SUBSCRIPTION_ID: ci.datadog-agent.e2e_tests_azure_subscription_id # agent-devx-loops
E2E_TESTS_GCP_CREDENTIALS: ci.datadog-agent.e2e_tests_gcp_credentials # agent-devx-loops
KITCHEN_EC2_SSH_KEY: ci.datadog-agent.aws_ec2_kitchen_ssh_key # agent-devx-loops
KITCHEN_AZURE_CLIENT_ID: ci.datadog-agent.azure_kitchen_client_id # agent-devx-loops
KITCHEN_AZURE_CLIENT_SECRET: ci.datadog-agent.azure_kitchen_client_secret # agent-devx-loops
KITCHEN_AZURE_SUBSCRIPTION_ID: ci.datadog-agent.azure_kitchen_subscription_id # agent-devx-loops
KITCHEN_AZURE_TENANT_ID: ci.datadog-agent.azure_kitchen_tenant_id # agent-devx-loops
GITHUB_PR_COMMENTER_APP_KEY: pr-commenter.github_app_key # agent-devx-infra
GITHUB_PR_COMMENTER_INTEGRATION_ID: pr-commenter.github_integration_id # agent-devx-infra
GITHUB_PR_COMMENTER_INSTALLATION_ID: pr-commenter.github_installation_id # agent-devx-infra
GITLAB_SCHEDULER_TOKEN: ci.datadog-agent.gitlab_pipelines_scheduler_token # ci-cd
GITLAB_READ_API_TOKEN: ci.datadog-agent.gitlab_read_api_token # ci-cd
GITLAB_FULL_API_TOKEN: ci.datadog-agent.gitlab_full_api_token # ci-cd
INSTALL_SCRIPT_API_KEY: ci.agent-linux-install-script.datadog_api_key_2 # agent-delivery
JIRA_READ_API_TOKEN: ci.datadog-agent.jira_read_api_token # agent-devx-infra
AGENT_GITHUB_APP_ID: ci.datadog-agent.platform-github-app-id # agent-devx-infra
AGENT_GITHUB_INSTALLATION_ID: ci.datadog-agent.platform-github-app-installation-id # agent-devx-infra
AGENT_GITHUB_KEY: ci.datadog-agent.platform-github-app-key # agent-devx-infra
MACOS_GITHUB_APP_ID: ci.datadog-agent.macos_github_app_id # agent-devx-infra
MACOS_GITHUB_INSTALLATION_ID: ci.datadog-agent.macos_github_installation_id # agent-devx-infra
MACOS_GITHUB_KEY: ci.datadog-agent.macos_github_key_b64 # agent-devx-infra
MACOS_GITHUB_APP_ID_2: ci.datadog-agent.macos_github_app_id_2 # agent-devx-infra
MACOS_GITHUB_INSTALLATION_ID_2: ci.datadog-agent.macos_github_installation_id_2 # agent-devx-infra
MACOS_GITHUB_KEY_2: ci.datadog-agent.macos_github_key_b64_2 # agent-devx-infra
RPM_GPG_KEY: ci.datadog-agent.rpm_signing_private_key_${RPM_GPG_KEY_ID} # agent-delivery
RPM_SIGNING_PASSPHRASE: ci.datadog-agent.rpm_signing_key_passphrase_${RPM_GPG_KEY_ID} # agent-delivery
SLACK_AGENT_CI_TOKEN: ci.datadog-agent.slack_agent_ci_token # agent-devx-infra
SMP_ACCOUNT_ID: ci.datadog-agent.single-machine-performance-account-id # single-machine-performance
SMP_AGENT_TEAM_ID: ci.datadog-agent.single-machine-performance-agent-team-id # single-machine-performance
SMP_API: ci.datadog-agent.single-machine-performance-api # single-machine-performance
SMP_BOT_ACCESS_KEY: ci.datadog-agent.single-machine-performance-bot-access-key # single-machine-performance
SMP_BOT_ACCESS_KEY_ID: ci.datadog-agent.single-machine-performance-bot-access-key-id # single-machine-performance
SSH_KEY: ci.datadog-agent.ssh_key # system-probe
SSH_KEY_RSA: ci.datadog-agent.ssh_key_rsa # agent-devx-loops
SSH_PUBLIC_KEY_RSA: ci.datadog-agent.ssh_public_key_rsa # agent-devx-loops
VCPKG_BLOB_SAS_URL: ci.datadog-agent-buildimages.vcpkg_blob_sas_url # windows-agent
WINGET_PAT: ci.datadog-agent.winget_pat # windows-agent
# End aws ssm variables
Expand Down Expand Up @@ -1040,6 +998,17 @@ workflow:
- when: manual
allow_failure: true

.on_ha_agent_or_e2e_changes:
- !reference [.on_e2e_main_release_or_rc]
- changes:
paths:
- comp/haagent/**/*
- pkg/aggregator/**/*
- test/new-e2e/tests/ha-agent/**/*
- test/new-e2e/go.mod
compare_to: main # TODO: use a variable, when this is supported https://gitlab.com/gitlab-org/gitlab/-/issues/369916
when: on_success

.on_otel_or_e2e_changes:
- !reference [.on_e2e_main_release_or_rc]
- changes:
Expand Down
2 changes: 1 addition & 1 deletion .gitlab/common/test_infra_version.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ variables:
# and check the job creating the image to make sure you have the right SHA prefix
TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX: ""
# Make sure to update test-infra-definitions version in go.mod as well
TEST_INFRA_DEFINITIONS_BUILDIMAGES: 7cd5e8a62570
TEST_INFRA_DEFINITIONS_BUILDIMAGES: 1f340fd450e3
Loading

0 comments on commit 0610542

Please sign in to comment.