feat: support tencent vector db

[quicksandznzn]

Description

Support Tencent Vector DB

dependencies

tcvectordb==1.3.2

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• TODO

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• optional I have made corresponding changes to the documentation
• optional I have added tests that prove my fix is effective or that my feature works
• optional New and existing unit tests pass locally with my changes

[bowenliang123]

-1 for supporting tencent vdb

1. tencent vdb , or Tencent Cloud VectorDB (https://cloud.tencent.com/product/vdb), is not an open-sourced vector db , which leads to less testability to test against the target vdb instance. The code will easily come to an idle status.
2. missing required tcvectordb python package in requirements.txt.
3. the package tcvectordb provides no information for usage and requirements on Pypl public repo, according to https://pypi.org/project/tcvectordb/
4. never put .env file to the PR

[crazywoola]

The rest looks good to me.

[wade30822]

how is it going?

[quicksandznzn]

how is it going?

wait review ~

[bowenliang123]

1. please resolve the sytle violation in Python code by running dev/reformat.
2. move the tests to api/tests/integration_tests/vdb/tcvectordb

[quicksandznzn]

1. please resolve the sytle violation in Python code by running dev/reformat.
2. move the tests to api/tests/integration_tests/vdb/tcvectordb

done~

[bowenliang123]

ok, thx.

[JohnJyong]

the socre is not returned and we have the score thresold check.
pls refer to below code:

        for doc, score in docs_and_scores:
            score_threshold = kwargs.get("score_threshold", .0) if kwargs.get('score_threshold', .0) else 0.0
            # check score threshold
            if score > score_threshold:
                doc.metadata['score'] = score
                docs.append(doc)

[JohnJyong]

@quicksandznzn

[zeroameli]

@quicksandznzn In the method def search_by_vector(), it return the results without score_threshold filtering, and tcvectordb won't return the score. Score_threshold won't work as expect. For example, you want the agent to ask for more information instead of giving an irrelevant reply. It seems a problem of the sdk.

[JohnJyong]

@quicksandznzn In the method def search_by_vector(), it return the results without score_threshold filtering, and tcvectordb won't return the score. Score_threshold won't work as expect. For example, you want the agent to ask for more information instead of giving an irrelevant reply. It seems a problem of the sdk.

@quicksandznzn In the method def search_by_vector(), it return the results without score_threshold filtering, and tcvectordb won't return the score. Score_threshold won't work as expect. For example, you want the agent to ask for more information instead of giving an irrelevant reply. It seems a problem of the sdk.

the sdk is fine , it has returned the score

[JohnJyong]

        score_threshold = kwargs.get("score_threshold", .0) if kwargs.get('score_threshold', .0) else 0.0
        return self._get_search_res(res, score_threshold)

    def _get_search_res(self, res, score_threshold):
        docs = []
        if res is None or len(res) == 0:
            return docs

        for result in res[0]:
            meta = result.get(self.field_metadata)
            if meta is not None:
                meta = json.loads(meta)
            score = 1 - result.get("score")
            if score > score_threshold:
                meta['score'] = score
                doc = Document(page_content=result.get(self.field_text), metadata=meta)
                docs.append(doc)
        return docs

[quicksandznzn]

        score_threshold = kwargs.get("score_threshold", .0) if kwargs.get('score_threshold', .0) else 0.0
        return self._get_search_res(res, score_threshold)

    def _get_search_res(self, res, score_threshold):
        docs = []
        if res is None or len(res) == 0:
            return docs

        for result in res[0]:
            meta = result.get(self.field_metadata)
            if meta is not None:
                meta = json.loads(meta)
            score = 1 - result.get("score")
            if score > score_threshold:
                meta['score'] = score
                doc = Document(page_content=result.get(self.field_text), metadata=meta)
                docs.append(doc)
        return docs

thanks , optimized

[JohnJyong]

the self.collection not initialized @quicksandznzn

[quicksandznzn]

fix

[zeroameli]

@quicksandznzn I found some problems:

dify/api/core/rag/datasource/vdb/tencent/tencent_vector.py

Lines 158 to 161 in a591366

	def delete_by_metadata_field(self, key: str, value: str) -> None:
	docs = self._db.collection(self._collection_name).query(filter=Filter(Filter.In(key, [value])))
	if docs and len(docs) > 0:
	self.collection.delete(document_ids=[doc['id'] for doc in docs])

• param limit is needed when query with filter, why not use delete with filter.
• fields in metadata should be indexed if we need to filter by them.

The self.collection won't be initialized in multithread because of redis lock (For example, create a dataset), why not use self._db.collection(self._collection_name)

[wade30822]

it takes so long~~ 😭