Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: integrate opendal storage #11508

Merged
merged 30 commits into from
Dec 11, 2024
Merged

feat: integrate opendal storage #11508

merged 30 commits into from
Dec 11, 2024

Conversation

laipz8200
Copy link
Member

@laipz8200 laipz8200 commented Dec 9, 2024 •
edited
Loading

Summary

  1. Integrate OpenDAL Storage.
  2. Remove LocalFSStorage.
  3. Add basic S3 support.
  4. Add Cloudflare R2 support.

tested on local fs and r2.

part of #9315

Tip

Close issue syntax: Fixes #<issue number> or Resolves #<issue number>, see documentation for more details.

Screenshots

Before After
... ...

Checklist

Important

Please review the checklist below before submitting your pull request.

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

@laipz8200 laipz8200 changed the title Feat/integrate opendal storage feat: integrate opendal storage Dec 9, 2024
@laipz8200 laipz8200 force-pushed the feat/integrate-opendal-storage branch from c2a0076 to a60469f Compare December 10, 2024 07:28
@laipz8200 laipz8200 marked this pull request as ready for review December 10, 2024 07:41
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Dec 10, 2024
@laipz8200 laipz8200 requested a review from crazywoola December 10, 2024 07:41
@dosubot dosubot bot added 💪 enhancement New feature or request 📚 documentation Improvements or additions to documentation labels Dec 10, 2024
@laipz8200 laipz8200 removed the request for review from crazywoola December 10, 2024 08:32
@laipz8200 laipz8200 marked this pull request as draft December 10, 2024 08:33
@laipz8200
test(test_opendal): fix tests
a1a3567 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
feat(opendal_storage): add basic s3 support
87a4a69 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
chore(.env.example): add opendal s3 settings
b06357f 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
feat(opendal_storage): add support for aws sse with aws managed key
21f50f3 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
fix(storage): compatible with local
af55eb7 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
feat(storage): use opendal instead of boto3
470fdda 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
fix(storage): set default endpoint for s3
d8d98d5 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
fix(storage): check if region exists
c90dcd0 
Signed-off-by: -LAN- <[email protected]>
@laipz8200
fix(storage): set default s3 region to us-east-1
36a10ae 
Signed-off-by: -LAN- <[email protected]>
@laipz8200 laipz8200 marked this pull request as ready for review December 11, 2024 06:48
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Dec 11, 2024
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 11, 2024
@laipz8200 laipz8200 merged commit 8d4bb9b into main Dec 11, 2024
15 checks passed
@laipz8200 laipz8200 deleted the feat/integrate-opendal-storage branch December 11, 2024 06:50
@marioalexandreantunes marioalexandreantunes mentioned this pull request Dec 12, 2024
Closed
iamjoel pushed a commit that referenced this pull request Dec 16, 2024
@laipz8200 @iamjoel
feat: integrate opendal storage (#11508)
4762d9b 
Signed-off-by: -LAN- <[email protected]>
@laipz8200 laipz8200 mentioned this pull request Dec 16, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zxhlyh zxhlyh zxhlyh approved these changes

Assignees
No one assigned
Labels
📚 documentation Improvements or additions to documentation 💪 enhancement New feature or request lgtm This PR has been approved by a maintainer size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@laipz8200 @zxhlyh