Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Object Storage v1: Add upload and download helper functions #47

Merged

Conversation

jtopjian
Copy link
Contributor

@jtopjian jtopjian commented Jun 2, 2018

This commit adds helper functions for uploading and downloading objects.
It includes support for streaming and file-based sources, segmented
objects via SLO and DLO.

For #46

@jtopjian
Copy link
Contributor Author

jtopjian commented Jun 2, 2018

This is a large commit. I began this by working on the Upload function, and when I finished, it was only a few hundred more lines of code for Download. Coupled with the difficulty of reviewing only Upload without the corresponding Download, I included everything here.

Unit tests are included where possible. The majority of the testing is from the acceptance tests, though. Since these functions involve several individual API calls to the Object Storage service, it would have been really difficult to write unit tests for what the acceptance tests are doing.

No doubt there's some refactoring that can be done here, but I've been working on this for almost six weeks now and want to publish what I have.

The acceptance tests show what's possible:

  • Upload/Download streaming
  • Upload/Download file
  • Upload/Download SLO
  • Upload/Download DLO
  • Download full directory structures (support for uploading a full directory is not included since I didn't see that detailed in the python-swiftclient files I looked at. It should be easy to add this support later, though)

@jtopjian
Copy link
Contributor Author

jtopjian commented Jun 6, 2018

@jrperritt if you're feeling up to reading a pile of code, let me know your opinion on this. It's pretty much a straight Python-to-Go conversion so there's probably better ways of doing some things. However, I could easily spend several more weeks tweaking this and really want to scratch it off my list.

This does work. I originally had numerous debug lines to ensure the right places in the code were being hit while I was writing the acceptance tests. It has about 90% parity with python-swiftclient. In addition, it's compatible with python-swiftclient so a file could be uploaded with this and downloaded via python-swiftclient and vice versa.

@jrperritt
Copy link
Contributor

That is indeed a pile of code :) I'll try to find some time to go over it

This commit adds helper functions for uploading and downloading objects.
It includes support for streaming and file-based sources, segmented
objects via SLO and DLO.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants