Metadata is not returned in query response when using LanceDB vector store

[mballav]

After searching for days for a solution, I am posting it here. Here is the code that reads PDFs, converts them into nodes with all metadata information. The issue is that after building the index with storage context backed by LanceDB vector store, when I query the index, the response returned does not contain any metadata!

        self.storage_folder = os.path.join(os.environ.get('DOCSEARCH__STORAGE_FOLDER'), 'lancedb')
        self.lancedb_vector_store = LanceDBUtil.get_lancedb_vector_store(self.storage_folder, table_name=topic_name)
        self.storage_context = StorageContext.from_defaults(vector_store=self.lancedb_vector_store)

        dir_reader = SimpleDirectoryReader(folder, required_exts=[".pdf"])
        documents = dir_reader.load_data()
        parser = SimpleNodeParser.from_defaults(chunk_size=1024, chunk_overlap=20)
        nodes: list[BaseNode] = parser.get_nodes_from_documents(documents)
        index = VectorStoreIndex(nodes=nodes, storage_context=self.storage_context)
        query_engine = index.as_query_engine(service_context=self.service_context)
        response = query_engine.query(query)

Response for a query looks like:

    "response": {
        "response": "The document is a ...",
        "source_nodes": [
            {
                "node": {
                    "id_": "49356020-1bc9-4b35-a985-b6dc417bee3f",
                    "embedding": null,
                    "extra_info": {},
                    "excluded_embed_metadata_keys": [],
                    "excluded_llm_metadata_keys": [],
                    "relationships": {
                        "1": {
                            "node_id": "84dc2ec9-7311-4220-804d-ab71da097f08",
                            "node_type": null,
                            "metadata": {},
                            "hash": null,
                            "class_name": "RelatedNodeInfo"
                        }
                    },
                    "hash": "7e39ec0be4324a59d7793a5f6f77df5889653226ff35d77231bef5e3945c595c",
                    "text": "...",
                    "start_char_idx": null,
                    "end_char_idx": null,
                    "text_template": "{metadata_str}\n\n{content}",
                    "metadata_template": "{key}: {value}",
                    "metadata_seperator": "\n",
                    "class_name": "TextNode"
                },
                "score": 0.5972621440887451,
                "class_name": "NodeWithScore"
            },
        ],
        "metadata": {
            "49356020-1bc9-4b35-a985-b6dc417bee3f": {},
        }
    }

Could anyone please help and tell me if there is a way to get metadata in the query response?

[dosubot[bot]]

🤖

Hello @mballav! I'm here to assist you with your issue while we wait for a human maintainer. I'm a bot, and I can help you solve bugs, answer questions, and guide you on becoming a contributor. Let's work on this together!

Based on the information you've provided and the similar issues I found in the LlamaIndex repository, it seems like the metadata is not being returned in the query response because it might not be properly loaded or stored in the LanceDB vector store.

From the issue #7250, it was suggested that the problem might occur when loading an existing index. The solution provided was to use the following code to properly load an existing index:

# load from disk
db2 = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = db2.get_or_create_collection("quickstart")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_vector_store(
    vector_store,
    service_context=service_context,
)

However, this solution was provided for ChromaDB, not LanceDB. You might need to adapt it to your use case.

In the LlamaIndex framework, LanceDB vector store handles metadata by converting it into a dictionary format using the node_to_metadata_dict function. This function takes a node and a flag indicating whether to remove text and whether to flatten the metadata. The metadata is then appended to the data that is being added to the LanceDB. Here is the relevant code snippet from llama_index/vector_stores/lancedb.py:

for node in nodes:
    metadata = node_to_metadata_dict(
        node, remove_text=True, flat_metadata=self.flat_metadata
    )
    append_data = {
        "id": node.node_id,
        "doc_id": node.ref_doc_id,
        "vector": node.get_embedding(),
        "text": node.get_content(metadata_mode=MetadataMode.NONE),
    }
    append_data.update(metadata)
    data.append(append_data)
    ids.append(node.node_id)

To retrieve metadata in the query response, you can use the VectorStoreQueryResult object returned by the vector_store.query() or vector_store.aquery() methods. This object contains the results of the query, including the nodes (data points) and their associated metadata. Here is a simplified example of how you can retrieve metadata:

def retrieve_metadata(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
    # Build the vector store query
    query = self._build_vector_store_query(query_bundle, self._similarity_top_k)

    # Execute the query
    query_result = self._vector_store.query(query, **self._kwargs)

    # Retrieve nodes and their metadata
    nodes_with_metadata = []
    for node in query_result.nodes:
        metadata = node.metadata  # Access the metadata of the node
        nodes_with_metadata.append((node, metadata))

    return nodes_with_metadata

Please note that the actual structure and content of the metadata depend on how the data was indexed into the LanceDB vector store. If no metadata was indexed with the data points, the metadata attribute will be None or empty.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• [Question]: Can I persist metadata in node without using it in the embedding?
• Support for Document Metadata in Query Responses
• Support for Document Metadata in Query Responses
• llama_index/indices/multi_modal/retriever.py
• llama_index/vector_stores/lancedb.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mballav]

@dosu-bot,

You say "If no metadata was indexed with the data points, the metadata attribute will be None or empty."

Is there a way to make sure that metadata gets indexed with the data points using LanceDB? Does anyone have experience of that happening automatically using ChromaDB? I am willing to switch to ChromaDB or any other vector DB that provides persistence to local disk.

[dosubot[bot]]

🤖

Based on the issues I found in the LlamaIndex repository, it seems that others have encountered similar problems with metadata not being returned in the query response when using LanceDB as a vector store.

One of the solutions provided in the issue [Bug]: metadata is lost when using LanceDB as vector store suggests that the problem might occur when you load an existing index. The user logan-markewich provided a solution where they used ChromaDB to load the existing index. Here is the code they used:

# load from disk
db2 = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = db2.get_or_create_collection("quickstart")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_vector_store(
    vector_store,
    service_context=service_context,
)

Another issue, Support for Document Metadata in Query Responses , suggests referring to the following documentation for customization of documents and document management in LlamaIndex:

1. https://gpt-index.readthedocs.io/en/latest/how_to/customization/custom_documents.html
2. https://gpt-index.readthedocs.io/en/latest/how_to/index/document_management.html

As for your question about how the LanceDBUtil.get_lancedb_vector_store method works in the LlamaIndex framework, I wasn't able to find specific information about this method in the repository. It might be helpful to check the documentation or source code of the LanceDBUtil class for more details.

Regarding the use of an alternative vector DB like ChromaDB, I found no specific information in the provided context. However, the solution provided in the first issue I mentioned suggests that ChromaDB might be a viable alternative.

I hope this information helps you resolve the issue. If you have further questions or need more clarification, feel free to ask.

Sources

• [Bug]: metadata is lost when using LanceDB as vector store
• [Question]: Can I persist metadata in node without using it in the embedding?
• Support for Document Metadata in Query Responses
• llama_index/vector_stores/cassandra.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mballav]

Just FYI to those who are in the same boat. I switched to ChromaDB and now I get all the metadata in the query response!