Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi platform images #2273

Draft
wants to merge 54 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
4a9dfec
Build flavors workflow
bdovaz Jun 23, 2022
e876649
Add arm_compliant field to env descriptor
bdovaz Jun 24, 2022
afaccf4
Empty Dockerfile_arm
bdovaz Jun 24, 2022
806a17f
build.py WIP
bdovaz Jun 24, 2022
760dd4c
Add linter supported_architectures with CPU architecture-specific ins…
echoix Jul 4, 2022
d50b5ef
Apply formatting of json schema
echoix Jul 4, 2022
8051514
Use proper Docker architectures names
echoix Jul 4, 2022
00f4c10
Change to supported_platforms to match the built-in build variable TA…
echoix Jul 4, 2022
dbbf788
Add riscv into cspell ignored words
echoix Jul 4, 2022
6d289d3
Remove changes
bdovaz Jul 10, 2022
a02001c
WIP
bdovaz Jul 10, 2022
fbc6f85
WIP
bdovaz Jul 10, 2022
34cca97
Update megalinter-descriptor.jsonschema for supported platforms at th…
echoix Jul 10, 2022
90d7369
Update examples to remove parent property from being included
echoix Jul 10, 2022
7cd155f
Add supported_platforms to bash descriptor
echoix Jul 4, 2022
f96a1a7
First iteration on multi-platform support in descriptors
bdovaz Jul 12, 2022
5435aa1
Add cspell exclusions for TARGETPLATFORM, aarch64, and armv6
echoix Sep 3, 2022
7d93ba4
Update jsonschema
bdovaz Dec 29, 2022
0962074
Add TARGETPLATFORM arg to shellcheck
bdovaz Dec 29, 2022
c54c36c
Add conditions to kubeval and misspell
bdovaz Jan 5, 2023
5411a2f
Powershell cross platform support
bdovaz Jan 11, 2023
ddb912a
Revert build.py changes
bdovaz Feb 26, 2023
bb159fe
Add linux/arm64 target
bdovaz Feb 26, 2023
13f3774
Add arm64 arch where possible to descriptors
bdovaz Feb 26, 2023
7a5fb66
Fix linters
bdovaz Feb 26, 2023
0d4a3d7
[MegaLinter] Apply linters fixes
bdovaz Feb 26, 2023
16d4bfb
qemu fixes
bdovaz Feb 26, 2023
cb90ee9
qemu fixes
bdovaz Feb 26, 2023
b8c7440
Fix buildx command
bdovaz Feb 26, 2023
4cbdeef
Remove --platform from run
bdovaz Feb 26, 2023
f614e6c
Split Run Test Cases step by platform
bdovaz Feb 26, 2023
01b03f5
Fix linters
bdovaz Mar 26, 2023
c07208e
Add supported platforms to all_linters.md
bdovaz Mar 26, 2023
fb7f91d
Run update.sh --doc
bdovaz Mar 26, 2023
2de2494
Run build.sh --doc
bdominguez Apr 13, 2023
fa75c38
Add additional platforms
bdominguez Apr 13, 2023
8dadab4
[MegaLinter] Apply linters fixes
bdovaz Apr 13, 2023
d69d9c8
[MegaLinter] Apply linters fixes
nvuillam Apr 13, 2023
2658829
Run build.sh --doc
bdovaz May 14, 2023
21a6b61
[MegaLinter] Apply linters fixes
bdovaz May 14, 2023
2979f08
Squashed commit of the following:
bdovaz Jul 18, 2023
49ab2ad
Merge branch 'main' into dev/multi-platform-images
bdovaz Jul 18, 2023
7348c0a
remove duplicated entry
bdovaz Jul 18, 2023
7e4bdba
Run build.sh
bdovaz Jul 18, 2023
ac63bda
Merge branch 'main' into dev/multi-platform-images
bdovaz Jan 4, 2024
8840e1a
Merge branch 'main' into dev/multi-platform-images
bdovaz Jan 4, 2024
fe466b2
Restore action
bdovaz Jan 5, 2024
0873468
Fix arg usage
bdovaz Jan 5, 2024
5754c46
Fix workflow
bdovaz Jan 5, 2024
87ab3eb
Fix workflow
bdovaz Jan 5, 2024
84f0eed
Merge branch 'main' into dev/multi-platform-images
bdovaz Jun 1, 2024
d5cae30
[MegaLinter] Apply linters fixes
bdovaz Jun 1, 2024
0a83116
Merge branch 'main' into dev/multi-platform-images
echoix Jun 10, 2024
aa664d3
[build-command] Update generated files
github-actions[bot] Jun 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions .automation/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -2805,6 +2805,7 @@ def generate_documentation_all_linters():
linters.sort(key=lambda x: x.linter_name)
table_header = [
"Linter",
"Supported Platforms",
"Version",
"License",
"Popularity",
Expand Down Expand Up @@ -2990,9 +2991,17 @@ def generate_documentation_all_linters():
f"[![GitHub stars](https://img.shields.io/github/stars/{repo}?cacheSeconds=3600)]"
f"(https://github.com/{repo}){{target=_blank}}"
)
supported_platforms = []
# supported platforms
if (
hasattr(linter, "supported_platforms")
and "platform" in linter.supported_platforms
):
supported_platforms += linter.supported_platforms["platform"]
# line
table_line = [
linter.linter_name,
", ".join(supported_platforms),
linter_version,
license,
"N/A",
Expand All @@ -3009,6 +3018,7 @@ def generate_documentation_all_linters():
linter_doc_links += [link]
md_table_line = [
md_linter_name,
"<br/> ".join(supported_platforms),
linter_version,
md_license,
md_popularity,
Expand Down Expand Up @@ -3049,10 +3059,10 @@ def generate_documentation_all_linters():
outfile.write("<!-- markdownlint-disable -->\n\n")
outfile.write("# References\n\n")
outfile.write(
"| Linter | Version | License | Popularity | Descriptors | Ref | URL |\n"
"| Linter | Supported Platforms | Version | License | Popularity | Descriptors | Ref | URL |\n"
)
outfile.write(
"| :---- | :-----: | :-----: | :-----: | :--------- | :--------------: | :-: |\n"
"| :---- | :-----: | :-----: | :-----: | :-----: | :--------- | :--------------: | :-: |\n"
)
for md_table_line in md_table_lines:
outfile.write("| %s |\n" % " | ".join(md_table_line))
Expand Down
1 change: 1 addition & 0 deletions .codecov.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
---
# https://docs.codecov.io/docs/commit-status#disabling-a-status
# Disable blocking patch status to avoid to block PRs without good reason !
coverage:
Expand Down
5 changes: 5 additions & 0 deletions .cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,7 @@
"Syft",
"Symlinks",
"Syntastic",
"TARGETPLATFORM",
"TEAMPROJECT",
"TEKTON",
"TERMIOS",
Expand Down Expand Up @@ -492,6 +493,7 @@
"YMLs",
"YOURBRANCH",
"YOURUSERNAME",
"aarch64",
"abhith",
"absolutized",
"abstractproperty",
Expand Down Expand Up @@ -524,6 +526,7 @@
"arctan",
"aren",
"argparse",
"armv6",
"artefacts",
"asdict",
"ashokm",
Expand Down Expand Up @@ -1017,6 +1020,7 @@
"msrest",
"mstruebing",
"msvs",
"multiarch",
"muandane",
"multiline",
"multimatch",
Expand Down Expand Up @@ -1250,6 +1254,7 @@
"returnrules",
"rexec",
"risd",
"riscv",
"rmfamily",
"rockspec",
"rockspecs",
Expand Down
2 changes: 1 addition & 1 deletion .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
# These are supported funding model platforms

github: [nvuillam]
1 change: 1 addition & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
---
#################################
# GitHub Dependabot Config info #
#################################
Expand Down
1 change: 1 addition & 0 deletions .github/linters/.cfnlintrc.yml
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
---
include_checks:
- I
1 change: 1 addition & 0 deletions .github/linters/.checkov.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
---
# You can see all available properties here: https://github.com/bridgecrewio/checkov#configuration-using-a-config-file
quiet: true
skip-check:
Expand Down
1 change: 1 addition & 0 deletions .github/linters/.hadolint.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
---
ignored:
- DL3007
- DL3016
Expand Down
1 change: 1 addition & 0 deletions .github/linters/.protolintrc.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
---
# Lint directives.
lint:
# Linter rules.
Expand Down
1 change: 1 addition & 0 deletions .github/release-drafter.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
---
name-template: "v$RESOLVED_VERSION"
tag-template: "v$RESOLVED_VERSION"
template: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/auto-update-linters.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ jobs:
with:
context: .
file: Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ env.BUILD_DATE }}
BUILD_REVISION=auto_update_${{ github.sha }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-ALPHA-flavors.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ jobs:
with:
context: .
file: flavors/${{ matrix.flavor }}/Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ env.BUILD_DATE }}
BUILD_REVISION=${{ github.sha }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-ALPHA.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ jobs:
with:
context: .
file: Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ env.BUILD_DATE }}
BUILD_REVISION=${{ github.sha }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-BETA-flavors.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ jobs:
with:
context: .
file: flavors/${{ matrix.flavor }}/Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ env.BUILD_DATE }}
BUILD_REVISION=${{ github.sha }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-BETA-linters.yml
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ jobs:
with:
context: .
file: linters/${{ matrix.linter }}/Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ env.BUILD_DATE }}
BUILD_REVISION=${{ github.sha }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-BETA.yml
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ jobs:
with:
context: .
file: Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ env.BUILD_DATE }}
BUILD_REVISION=${{ github.sha }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-DEV-linters.yml
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ jobs:
"yaml_v8r",
]
# linters-end
platform: ['linux/amd64']
platform: ['linux/amd64', 'linux/arm64']
# Only run this on the main repo
if: |
(
Expand Down
27 changes: 17 additions & 10 deletions .github/workflows/deploy-DEV.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,14 @@ jobs:
# Set the agent to run on
runs-on: ubuntu-latest
permissions: read-all
services:
registry:
image: registry:2
ports:
- 5000:5000
env:
# If using Docker Hub, trivy would need to have docker.io as prefix
REGISTRY_PREFIX: localhost:5000
# Prevent duplicate run from happening when a forked push is committed
if: (github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository) && !contains(github.event.head_commit.message, 'skip deploy')
# Set max build time for the job
Expand Down Expand Up @@ -80,13 +88,15 @@ jobs:
id: meta
with:
images: |
${{ github.repository }}
${{ env.REGISTRY_PREFIX }}/${{ github.repository }}

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver-opts: network=host

###################################
# Build image locally for testing #
Expand All @@ -98,14 +108,13 @@ jobs:
with:
context: .
file: Dockerfile-quick
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.created'] }}
BUILD_VERSION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.version'] }}
BUILD_REVISION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.revision'] }}
MEGA_LINTER_BASE_IMAGE="oxsecurity/megalinter:beta"
load: true
push: false
push: true
secrets: |
GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
tags: ${{ steps.meta.outputs.tags }}
Expand All @@ -127,13 +136,12 @@ jobs:
with:
context: .
file: Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.created'] }}
BUILD_VERSION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.version'] }}
BUILD_REVISION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.revision'] }}
load: true
push: false
push: true
secrets: |
GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
tags: ${{ steps.meta.outputs.tags }}
Expand All @@ -154,13 +162,12 @@ jobs:
with:
context: .
file: Dockerfile
platforms: linux/amd64
platforms: linux/amd64,linux/arm64
build-args: |
BUILD_DATE=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.created'] }}
BUILD_VERSION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.version'] }}
BUILD_REVISION=${{ fromJSON(steps.meta.outputs.json).labels['org.opencontainers.image.revision'] }}
load: true
push: false
push: true
secrets: |
GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
tags: ${{ steps.meta.outputs.tags }}
Expand Down
1 change: 1 addition & 0 deletions .mega-linter.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
---
# Configuration file for MegaLinter
# See all available variables at https://megalinter.io/configuration/ and in linters documentation

Expand Down
62 changes: 44 additions & 18 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -62,17 +62,27 @@ FROM alpine/terragrunt:latest as terragrunt
##################
# Get base image #
##################
# https://stackoverflow.com/a/73711302/699056
FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu

FROM python:3.12.3-alpine3.19

# https://stackoverflow.com/a/73711302/699056
COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin/
# https://stackoverflow.com/a/73711302/699056
RUN apk add --update --no-cache libc6-compat \
gcompat \
qemu-x86_64

#############################################################################################
## @generated by .automation/build.py using descriptor files, please do not update manually ##
#############################################################################################
#ARG__START
ARG TARGETPLATFORM
ARG ARM_TTK_NAME='master.zip'
ARG ARM_TTK_URI='https://github.com/Azure/arm-ttk/archive/master.zip'
ARG ARM_TTK_DIRECTORY='/opt/microsoft'
ARG BICEP_EXE='bicep'
ARG BICEP_URI='https://github.com/Azure/bicep/releases/latest/download/bicep-linux-musl-x64'
ARG BICEP_DIR='/usr/local/bin'
ARG DART_VERSION='2.8.4'
# renovate: datasource=github-tags depName=pmd/pmd extractVersion=^pmd_releases/(?<version>.*)$
Expand Down Expand Up @@ -351,11 +361,15 @@ COPY --link --from=terragrunt /bin/terraform /usr/bin/
#OTHER__START
RUN rc-update add docker boot && rc-service docker start || true \
# ARM installation
&& curl -L https://github.com/PowerShell/PowerShell/releases/download/v7.4.2/powershell-7.4.2-linux-musl-x64.tar.gz -o /tmp/powershell.tar.gz \
&& mkdir -p /opt/microsoft/powershell/7 \
&& tar zxf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \
&& chmod +x /opt/microsoft/powershell/7/pwsh \
&& ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh
&& case ${TARGETPLATFORM} in \
"linux/amd64") POWERSHELL_ARCH=musl-x64 ;; \
"linux/arm64") POWERSHELL_ARCH=arm64 ;; \
esac \
&& curl -L https://github.com/PowerShell/PowerShell/releases/download/v7.4.2/powershell-7.4.2-linux-${POWERSHELL_ARCH}.tar.gz -o /tmp/powershell.tar.gz \
&& mkdir -p /opt/microsoft/powershell/7 \
&& tar zxf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \
&& chmod +x /opt/microsoft/powershell/7/pwsh \
&& ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh


# CLOJURE installation
Expand Down Expand Up @@ -479,11 +493,15 @@ ENV PATH="/root/.composer/vendor/bin:${PATH}"

# POWERSHELL installation
# Next line commented because already managed by another linter
# RUN curl -L https://github.com/PowerShell/PowerShell/releases/download/v7.4.2/powershell-7.4.2-linux-musl-x64.tar.gz -o /tmp/powershell.tar.gz \
# && mkdir -p /opt/microsoft/powershell/7 \
# && tar zxf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \
# && chmod +x /opt/microsoft/powershell/7/pwsh \
# && ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh
# RUN case ${TARGETPLATFORM} in \
# "linux/amd64") POWERSHELL_ARCH=musl-x64 ;; \
# "linux/arm64") POWERSHELL_ARCH=arm64 ;; \
# esac \
# && curl -L https://github.com/PowerShell/PowerShell/releases/download/v7.4.2/powershell-7.4.2-linux-${POWERSHELL_ARCH}.tar.gz -o /tmp/powershell.tar.gz \
# && mkdir -p /opt/microsoft/powershell/7 \
# && tar zxf /tmp/powershell.tar.gz -C /opt/microsoft/powershell/7 \
# && chmod +x /opt/microsoft/powershell/7/pwsh \
# && ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh

# SALESFORCE installation
# Next line commented because already managed by another linter
Expand Down Expand Up @@ -535,9 +553,13 @@ RUN curl --retry 5 --retry-delay 5 -sLO "${ARM_TTK_URI}" \
# Managed with COPY --link --from=shfmt /bin/shfmt /usr/bin/

# bicep_linter installation
&& curl --retry 5 --retry-delay 5 -sLo ${BICEP_EXE} "${BICEP_URI}" \
&& chmod +x "${BICEP_EXE}" \
&& mv "${BICEP_EXE}" "${BICEP_DIR}" \
&& case ${TARGETPLATFORM} in \
"linux/amd64") POWERSHELL_ARCH=musl-x64 ;; \
"linux/arm64") POWERSHELL_ARCH=arm64 ;; \
esac \
&& curl --retry 5 --retry-delay 5 -sLo ${BICEP_EXE} "https://github.com/Azure/bicep/releases/latest/download/bicep-linux-${POWERSHELL_ARCH}" \
&& chmod +x "${BICEP_EXE}" \
&& mv "${BICEP_EXE}" "${BICEP_DIR}" \

# clj-kondo installation
&& curl --retry 5 --retry-delay 5 -sLO https://raw.githubusercontent.com/clj-kondo/clj-kondo/master/script/install-clj-kondo \
Expand All @@ -551,10 +573,14 @@ RUN curl --retry 5 --retry-delay 5 -sLO "${ARM_TTK_URI}" \
&& dotnet tool install -g roslynator.dotnet.cli \

# dartanalyzer installation
&& wget --tries=5 https://storage.googleapis.com/dart-archive/channels/stable/release/${DART_VERSION}/sdk/dartsdk-linux-x64-release.zip -O - -q | unzip -q - \
&& chmod +x dart-sdk/bin/dart* \
&& mv dart-sdk/bin/* /usr/bin/ && mv dart-sdk/lib/* /usr/lib/ && mv dart-sdk/include/* /usr/include/ \
&& rm -r dart-sdk/ \
&& case ${TARGETPLATFORM} in \
"linux/amd64") DART_ARCH=x64 ;; \
"linux/arm64") DART_ARCH=arm64 ;; \
esac \
&& wget --tries=5 https://storage.googleapis.com/dart-archive/channels/stable/release/${DART_VERSION}/sdk/dartsdk-linux-${DART_ARCH}-release.zip -O - -q | unzip -q - \
&& chmod +x dart-sdk/bin/dart* \
&& mv dart-sdk/bin/* /usr/bin/ && mv dart-sdk/lib/* /usr/lib/ && mv dart-sdk/include/* /usr/include/ \
&& rm -r dart-sdk/ \

# hadolint installation
# Managed with COPY --link --from=hadolint /bin/hadolint /usr/bin/hadolint
Expand Down
Loading