Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pgo storage fills up too quick / different psql operator? #245

Open
durandom opened this issue May 9, 2023 · 4 comments
Open

pgo storage fills up too quick / different psql operator? #245

durandom opened this issue May 9, 2023 · 4 comments
Labels
priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@durandom
Copy link
Member

durandom commented May 9, 2023 •
edited
Loading

because backups are eating up our storage: CrunchyData/postgres-operator#2531

Consider migration to https://cloudnative-pg.io/

https://blog.palark.com/cloudnativepg-and-other-kubernetes-operators-for-postgresql/
@durandom
Copy link
Member Author

durandom commented May 9, 2023

either increasing the backup frequency might help as in CrunchyData/postgres-operator#2531 (comment)

or setting archive_mode to off as in CrunchyData/postgres-operator#2531 (comment)

@durandom durandom mentioned this issue May 9, 2023
Merged
@goern
Copy link
Member

goern commented May 25, 2023

/priority backlog

@op1st-prow op1st-prow bot added the priority/backlog Higher priority than priority/awaiting-more-evidence. label May 25, 2023
@goern goern mentioned this issue Sep 7, 2023
Closed
@durandom
Copy link
Member Author

durandom commented Sep 7, 2023

Extend the PVC for psql by +2Gb and wait until the DB recovers.
Then trigger a one off backup either by:

oc annotate postgrescluster db --overwrite postgres-operator.crunchydata.com/pgbackrest-backup="$(date)"

or oc create job --from=cronjob/db-repo1-full db-backup-full

Long term we need either

  • monitoring of the psql PVC (but this happens fast)
  • a setup that can tolerate failing backups
  • less WAL for psql

@goern
Copy link
Member

goern commented Sep 13, 2023

implemented some altering, see https://github.com/b4mad/racing/blob/main/manifests/env/phobos/postgresql/alerting.yaml

alertmanager will send to https://discord.com/channels/850841654327640074/1106466539273732136

@durandom durandom changed the title pgo storage fills up too quick pgo storage fills up too quick / different psql operator? Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@durandom @goern