Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added 2 separate restic containers one for S3 other for GC. #50

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 64 additions & 9 deletions compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ services:
restart: unless-stopped
environment:
- MYSQL_ROOT_HOST=%
- MYSQL_ROOT_PASSWORD=$MW_DB_INSTALLDB_PASS
- MYSQL_ROOT_PASSWORD=/run/secrets/db_root_password
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not this be MYSQL_ROOT_PASSWORD_FILE ?

- MYSQL_DATABASE=$MW_DB_NAME
volumes:
- ./__initdb:/docker-entrypoint-initdb.d
Expand Down Expand Up @@ -37,7 +37,7 @@ services:
- MW_DB_INSTALLDB_USER=root
- MW_DB_INSTALLDB_PASS
- MW_DB_USER=root
- MW_DB_PASS=$MW_DB_INSTALLDB_PASS
- MW_DB_PASS=/run/secrets/db_root_password
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work by itself, please see how it's done here https://github.com/WikiWorks/docker-wikicity.com/blob/main/config/LocalSettings.php#L96

- MW_SECRET_KEY
# - XDEBUG_CONFIG=remote_host=0.0.0.0 remote_enable=1 remote_autostart=Off remote_connect_back=On
- MW_SITE_SERVER=${MW_SITE_SERVER:-https://bugsigdb.org}
Expand Down Expand Up @@ -118,7 +118,9 @@ services:
- MATOMO_DATABASE_HOST=db
- MATOMO_DATABASE_DBNAME=matomo
- MATOMO_DATABASE_USERNAME=root
- MATOMO_DATABASE_PASSWORD=$MW_DB_INSTALLDB_PASS
- MATOMO_DATABASE_PASSWORD=/run/secrets/db_root_password
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also won't work. The variable is expected to contain a password string, not a path to the password file. We may need to invent something here, I welcome @pastakhov for input

secrets:
- db_root_password
volumes:
- ./_data/matomo:/var/www/html
- ./_resources/matomo-bootstrap.php:/var/www/html/bootstrap.php
Expand All @@ -141,7 +143,6 @@ services:
- traefik.http.middlewares.${COMPOSE_PROJECT_NAME}-matomo-headers.headers.customrequestheaders.X-Forwarded-Uri=/${MATOMO_FOLDER:-matomo}
- traefik.http.routers.${COMPOSE_PROJECT_NAME}-matomo.middlewares=${COMPOSE_PROJECT_NAME}-matomo-stripprefix,${COMPOSE_PROJECT_NAME}-matomo-headers


varnish:
image: pastakhov/varnish:7.0
restart: unless-stopped
Expand Down Expand Up @@ -197,21 +198,21 @@ services:
entrypoint:
- /updateEFO/updateEFO.sh

restic:
restic_s3:
image: ghcr.io/wikiteq/restic-backup-docker:stream
env_file:
- ./.env
- ./.env.secret.restic
profiles:
- production
restart: unless-stopped
links:
- db
environment:
- MW_DB_USER=root
- MW_DB_PASS=$MW_DB_INSTALLDB_PASS
- MW_DB_PASS=/run/secrets/db_root_password
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above - this won't work like that, the variable is expected to contain a password string, not a path to a file containing a password

- MW_DB_NAME=$MW_DB_NAME
- RESTIC_REPOSITORY_FILE=/run/secrets/restic_s3_repository
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think repository should be made secret

- RESTIC_PASSWORD_FILE=/run/secrets/restic_password
- RESTIC_TAG=backup_full
- AWS_SHARED_CREDENTIALS_FILE=/run/secrets/aws_credentials
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY instead https://github.com/WikiTeq/restic-backup-docker?tab=readme-ov-file#environment-variables

- BACKUP_CRON=0 1 * * 6
- CHECK_CRON=0 23 * * 5
- RESTIC_FORGET_ARGS=--prune --keep-weekly 1 --keep-monthly 2 --group-by paths
Expand All @@ -222,11 +223,53 @@ services:
- RESTIC_STREAM=yes
- RESTIC_STREAM_CMD=tar -cf - -C /data_mediawiki/ /data/database.sql.gz images
- RESTIC_STREAM_FILENAME=full.images.database.tar
secrets:
- aws_credentials
- db_root_password
- restic_password
- restic_s3_repository
volumes:
# read-only mount of mediawiki data directory for images access
- ./_data/mediawiki:/data_mediawiki:ro
# logs
- ./_logs/restic:/var/log

restic_gc:
image: ghcr.io/wikiteq/restic-backup-docker:stream
profiles:
- production
restart: unless-stopped
links:
- db
environment:
- MW_DB_USER=root
- MW_DB_PASS=/run/secrets/db_root_password
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

- MW_DB_NAME=$MW_DB_NAME
- RESTIC_REPOSITORY_FILE=/run/secrets/restic_gc_repository
- RESTIC_PASSWORD_FILE=/run/secrets/restic_password
- RESTIC_TAG=backup_full
- GOOGLE_APPLICATION_CREDENTIALS=/run/secrets/gcloud_credentials
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#google-cloud-storage there should be both GOOGLE_PROJECT_ID and GOOGLE_APPLICATION_CREDENTIALS set

- BACKUP_CRON=30 1 * * *
- CHECK_CRON=0 23 * * 5
- RESTIC_FORGET_ARGS=--prune --keep-weekly 1 --keep-monthly 2 --group-by paths
# pre-backup script
- RESTIC_PRE_BACKUP=if [ ! -d "/data" ]; then mkdir /data; fi && mysqldump -hdb -u$$MW_DB_USER -p$$MW_DB_PASS $$MW_DB_NAME | gzip -c > /data/database.sql.gz
# post-backup script
- RESTIC_POST_BACKUP=rm /data/database.sql.gz
- RESTIC_STREAM=yes
- RESTIC_STREAM_CMD=tar -cf - -C /data_mediawiki/ /data/database.sql.gz images
- RESTIC_STREAM_FILENAME=full.images.database.tar
secrets:
- gcloud_credentials
- db_root_password
- restic_password
- restic_gc_repository
volumes:
# read-only mount of mediawiki data directory for images access
- ./_data/mediawiki:/data_mediawiki:ro
# logs
- ./_logs/restic:/var/log


# https://github.com/mcuadros/ofelia
ofelia:
Expand All @@ -250,6 +293,18 @@ services:
secrets:
matomo_password:
file: ./secrets/matomo_password.txt
aws_credentials:
file: ./secrets/aws_credentials
db_root_password:
file: ./secrets/db_root_password
restic_password:
file: ./secrets/restic_password
restic_s3_repository:
file: ./secrets/restic_s3_repository
restic_gc_repository:
file: ./secrets/restic_gc_repository
gcloud_credentials:
file: ./secrets/gcloud_credentials

networks:
traefik-public:
Expand Down