Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBZ-3226: Add step to verify the Debezium-server distribution #140

Merged
merged 1 commit into from
Dec 18, 2024
+97 −0
Merged

DBZ-3226: Add step to verify the Debezium-server distribution #140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/cross-maven.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,63 @@ jobs:
run: ./server/mvnw clean install -f core/pom.xml -DskipTests -DskipITs -Dformat.formatter.goal=validate -Dformat.imports.goal=check -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120
- name: Maven build Debezium Server
run: ./server/mvnw clean install -fae -f server/pom.xml -Passembly -Dformat.formatter.goal=validate -Dformat.imports.goal=check -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 -DskipNonCore
- name: Archive Debezium distribution
uses: actions/upload-artifact@v4
with:
name: debezium-dist-artifact
path: ./server/debezium-server-dist/target/debezium-server-dist*.tar.gz

validate-distribution:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this job needs to be added also to the maven.yml, so I'm not sure why there are two files for building, so I prefer not duplicating it for now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sgitario One is executed on pull requests. That job migh need to build core not from the main branch but from a different branch (same name as this one). This is used for cases when PR depends on changes in core repo.
maven.yml is used exclusively for main/branch builds and always build from main of the core repo

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if a different job is needed as it executes yet another build. Can't the test be part of `build step?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to have a different job to use the strategy.matrix, so we can execute the same job to cover other different sinks / configurations with little effort.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the built artifact can be shared between jobs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Mario, if we can reuse the build across jobs since there is this build dependency defined, I think that's reasonable to reduce the overall execution time on CI.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mfvitale @Naros PR updated to reuse the distribution from the Build job.
I confirmed that the validation job now takes a few seconds when before it used to take some minutes.

name: Verify debezium-server distribution
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
sink: [ redis ]
needs: build
steps:
- name: Set up JDK
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: 21
- name: Checkout Debezium Server repository
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is no more required right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, to check out the sink specific test resources which is used to verify the server

uses: actions/checkout@v4
with:
path: server
- name: Download Debezium distribution
uses: actions/download-artifact@v4
with:
name: debezium-dist-artifact
- name: Unzip Debezium distribution
run: tar xf debezium-server-dist-*.tar.gz
- name: Validate sink '${{matrix.sink}}'
run: |
# Creating the distribution archives
DISTRIBUTION_FOLDER=./debezium-server
SINK=${{matrix.sink}}
CONFIG_FOLDER=./server/debezium-server-dist/src/test/resources/$SINK

# Start Up dependant services
docker compose -f $CONFIG_FOLDER/docker-compose.yml up -d

# Copy configuration file
cp $CONFIG_FOLDER/application.properties $DISTRIBUTION_FOLDER/config/application.properties

# Run debezium server at background
docker run --name server-$SINK -d -v $DISTRIBUTION_FOLDER:/opt:z -p 8080:8080 --network debezium-backend -w /opt registry.access.redhat.com/ubi8/openjdk-21 ./run.sh

# Verify
timeout 60 bash -c 'while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' http://localhost:8080/q/health)" != "200" ]]; do sleep 5; done' || false
- name: Print logs
if: failure()
run: |
echo "Logs from dependant services:"
docker compose -f ./server/debezium-server-dist/src/test/resources/${{matrix.sink}}/docker-compose.yml logs
echo "Logs from Debezium server:"
docker logs server-${{matrix.sink}}
- name: Clean Up
if: always()
run: |
docker rm -f server-${{matrix.sink}}
docker compose -f ./server/debezium-server-dist/src/test/resources/${{matrix.sink}}/docker-compose.yml down
14 changes: 14 additions & 0 deletions debezium-server-dist/src/test/resources/redis/application.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Docker file used by GitHub actions to verify the debezium-server-dist works.
debezium.sink.type=redis
debezium.sink.redis.address=redis:6379
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.database.hostname=postgres
debezium.source.database.port=5432
debezium.source.database.user=postgres
debezium.source.database.password=postgres
debezium.source.database.dbname=postgres
debezium.source.topic.prefix=tutorial
debezium.source.schema.include.list=inventory
quarkus.log.console.json=false
23 changes: 23 additions & 0 deletions debezium-server-dist/src/test/resources/redis/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Docker file used by GitHub actions to verify the debezium-server-dist works.
services:
postgres:
image: quay.io/debezium/example-postgres:3.0
ports:
- "5432:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
networks:
- backend

redis:
image: bitnami/redis:7.0
ports:
- "6379:6379"
environment:
- ALLOW_EMPTY_PASSWORD=yes
networks:
- backend
networks:
backend:
name: debezium-backend
Loading