[Question]:

[ddealwis09]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

For the SimpleDirectoryReader, how do you exclude metadata, using 'exclude' does not appear to work? Thanks!

``

doc_metadata = ["2023-03-21", "Poland Pricing Research", "BgM", "Sam Smith"]
def add_file_metadata(self) -> Dict:
    return {'pubdate': doc_metadata[0], 
            'filename': doc_metadata[1], 
            'Agency': doc_metadata[2],
            'Coauthor': doc_metadata[3]}
document = SimpleDirectoryReader(input_files=[temp_file_path], 
                                           filename_as_id=True,
                                           file_metadata=add_file_metadata,
                                           exclude=["Agency"] # metadata to make invisible to LLM response
                                           ).load_data()
query_engine.query("Which agency was used?")

``

Model response still includes "Agency":

responseResponseResponse(response='The agency used for the Poland Pricing Research was BgM.', source_nodes=[NodeWithScore(node=TextNode(id_='0a17f4c4-e080-4a27-8d21-9df1eb18d729', embedding=None, metadata={'pubdate': '2023-03-21', 'filename': 'Winston Poland Pricing Research', 'Agency': 'Gfk', 'Coauthor': '...

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out with your question. The 'exclude' parameter in the SimpleDirectoryReader in LlamaIndex is used to specify a list of glob patterns for Python file paths that should be excluded from the reading process. It is not used for excluding metadata.

In your case, you are trying to exclude certain metadata from being visible to the LLM response. Unfortunately, the current version of LlamaIndex does not support this feature directly. The 'exclude' parameter won't work for this purpose as it is designed to exclude files, not metadata.

However, you can control what metadata is added to the document by modifying the add_file_metadata function. If you don't want the 'Agency' metadata to be included, you can simply not add it in this function. Here's an example:

def add_file_metadata(self) -> Dict:
    return {'pubdate': doc_metadata[0], 
            'filename': doc_metadata[1], 
            'Coauthor': doc_metadata[3]}

In this example, I've removed the 'Agency' line. Now, the 'Agency' metadata won't be included in the document, and thus won't be visible to the LLM response.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• llama_index/readers/file/base.py
• tests/readers/test_file.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@ddealwis09 I think the proper solution here is setting the excluded metadata keys on the document objects, I don't think simple directory reader sets this

for doc in documents:
  doc.excluded_llm_metadata_keys=[".."]