Add zstd compression support to Barman Cloud #898
Conversation
|
Tested backup and restore with basebackup and Wal files. |
|
Ping @martinmarques |
A little patience, we have some deadlines with products, but this is on our radar and we do want to fully analyse this with the CNPG team. More later. |
|
Is there any plan for this to be merged? |
gcalacoci
force-pushed
the
feature/zstd
branch
from
06056fd to
0536907
Compare
| zstd = _try_import_zstd() | ||
| self.compressor = zstd.ZstdCompressor() | ||
| self.decompressor = zstd.ZstdDecompressor().decompressobj( | ||
| read_across_frames=True |
There was a problem hiding this comment.
Writing this here but is valid in other places of the code...
decompressobj doesn't support read_across_frames for versions of the zstandard lib that are older than 0.22, as you can see here: https://github.com/indygreg/python-zstandard/blob/0.21.0/zstandard/backend_cffi.py#L3903
Unfortunately, this means that this implementation is not compatible with Python 3.6 and 3.7, as you can also see in the unit tests run for this PR.
@Jellyfrog we cannot accept and merge this implementation as it is now, but would be nice if you could be able to make this patch compatible with older python versions that barman still needs to support.
There was a problem hiding this comment.
Hi @gcalacoci I'm interested in finish this. zstd library doesn't seem to have a breaking change with 3.6 and 3.7. Just no longer build the binaries for 3.6 and 3.7. To solve this, we may need a fork of zstd, with some new releases. Does this work with barman? Is EDB interested in holding the fork as well?
There was a problem hiding this comment.
Hi @gcalacoci I'm interested in finish this. zstd library doesn't seem to have a breaking change with 3.6 and 3.7. Just no longer build the binaries for 3.6 and 3.7. To solve this, we may need a fork of zstd, with some new releases. Does this work with barman? Is EDB interested in holding the fork as well?
Feel free to take over. I’m not going to work on this (not using Barman anymore, and especially not for EOL python versions)
There was a problem hiding this comment.
I don't think that's something we want to do or encourage people to do so. There are other solutions to this. Giulio is trying to encourage @Jellyfrog to address those issues.
In short, Barman has requirements, and we have to be careful about those requirements because there are users running Barman on systems that still have python 3.6. This PR would require the deprecation of python 3.6 and 3.7, and that's not something we want to do without some deep thinking.
|
I'm closing this PR. We plan to add zstd and lz4 support for WAL compression, but we feel the implementation has to cover the whole Barman ecosystem, not just the cloud scripts. |
|
@martinmarques: I'm looking to roll out CNPG in production, and we'd really prefer to use I'm assuming the biggest pain here would be including the native dependencies for the Python packages and extending any test harnesses to support it? |
|
@jmealo You have that now, as of yesterday |
This implements zstandard support to Barman cloud
Quick benchmark (from my laptop over internet to S3) shows it's the fastest and second smallest size:
Fixes #850