Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

23 update s3 util to allow for untarred uploading #24

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

Veldhoen
Copy link

@Veldhoen Veldhoen commented Jul 26, 2024

Updated the s3_util to allow non-tarred output transfer, to enable (for instance) syncing keyframes without having to (un)tar everything. Also makes it easier to link to proper provenance directly.
For this, created a method to parse the incoming file_list and create proper keys for the contents.

Breaking change for release: changed the return type of s3_download, to obtain provenance of download step from DANE module directly (rather than have each worker implement this). Please check if this makes sense to you.

Added some docstrings here and there, and put the newly created code under test

For review, please:

Sara Veldhoen added 2 commits July 26, 2024 10:52
…y. Also, empty prefix in bucket before uploading, and improved download code. Added some docstrings as wel
@Veldhoen Veldhoen requested a review from jblom July 26, 2024 08:59
@Veldhoen Veldhoen linked an issue Jul 26, 2024 that may be closed by this pull request
@Veldhoen Veldhoen marked this pull request as draft July 30, 2024 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update S3_util to allow for untarred uploading
1 participant