23 update s3 util to allow for untarred uploading #24
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updated the s3_util to allow non-tarred output transfer, to enable (for instance) syncing keyframes without having to (un)tar everything. Also makes it easier to link to proper provenance directly.
For this, created a method to parse the incoming file_list and create proper keys for the contents.
Breaking change for release: changed the return type of s3_download, to obtain provenance of download step from DANE module directly (rather than have each worker implement this). Please check if this makes sense to you.
Added some docstrings here and there, and put the newly created code under test
For review, please:
poetry install
and thenpoetry run pytest
)s3_download
? I haven't thoroughly tested this (will be used in Use DANE version with no-tar option (WIP) beeldengeluid/dane-visual-feature-extraction-worker#48)