feat: support elasticsearch vector database

[miendinh]

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Add Elasticsearch vector database for RAG, this feature requires package elasticsearch==7.15.2

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• TODO

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• optional I have made corresponding changes to the documentation
• optional I have added tests that prove my fix is effective or that my feature works
• optional New and existing unit tests pass locally with my changes

[miendinh]

hi @JohnJyong
do you have any comments?

[crazywoola]

Hi there, please run ./dev/reformat in root dir. The CI build didn't pass.

[miendinh]

Could you add `Setup ElasticSearch' to the CI like Milvus or Quant? @crazywoola

[JohnJyong]

please fix the pytest and add vdb integration test for it ， thanks @miendinh
intergration test folder ： ‘tests.integration_tests.vdb’

[miendinh]

please fix the pytest and add vdb integration test for it ， thanks @miendinh intergration test folder ： ‘tests.integration_tests.vdb’

added, but need ElasticSearch on CI to run ES tests, plz check @JohnJyong

[JohnJyong]

CI workflow is set up in the .github/workflows/api-tests.yml @miendinh

[bowenliang123]

Please test on your local development environment first.
Do not keep pushing untested code to run all the CI jobs, which is meaningless and costly.

[mboo2005]

@miendinh I tried your branch. Celery run failed because the config is missed.
error message:

Config is not interpreted

[bowenliang123]

LGTM overall. But several changes are needed for response and actions.

[miendinh]

plz check @JohnJyong @crazywoola, thanks.

[crazywoola]

We will check it before v0.6.17