How to change completion_to_prompt dynamically after being initialized

[semoal]

We have an use case that we start the LLM class

    llm = Replicate(
        model=SELECTED_MODEL,
        api_key="x",
        stream=True,
        temperature=0.1,
        num_output=800,
        completion_to_prompt=qa_expert_wrapper,
        messages_to_prompt=messages_to_prompt,
    )
    service_context = ServiceContext.from_defaults(
        context_window=CONTEXT_WINDOW,
        llm=llm,
    )
    set_global_service_context(service_context)

And runs fine, but then we want to change it dynamically given an input that comes from a websocket, we have tried to reinitialize this LLM class and do ServiceContext.from_defaults() and run again the set_global_service_context but when trying to run the engine, it's still with the old prompt.

We run the application with uvicorn if this is useful for diagnosing what happens

We use by the way this way of querying:

     index = VectorStoreIndex.from_documents(
            show_progress=True,
            storage_context=storage_ctx,
            documents=documents,
        )
        chat_engine = index.as_chat_engine(
            chat_mode=ChatMode.CONTEXT,
            memory=self.memory,
        )
        stream = chat_engine.stream_chat(prompt)

I would expect that could be possible to pass completion_to_prompt on as_chat_engine function call, but is not possible.. Should be?
Is something we can do for handling dynamic prompts easily?

[dosubot[bot]]

🤖

Hello @semoal! I'm Dosu, a friendly bot here to assist you with your LlamaIndex queries or issues while we're waiting for a human maintainer. I'm here to help you navigate through bug fixes, answer your questions, and even guide you on becoming a contributor. Let's get started!

Based on the information you've provided, it seems like you're trying to change the completion_to_prompt attribute of the Replicate class after it has been initialized. This attribute is used to process the prompt before it is passed to the replicate.run function in the stream_complete method.

If you want to change this attribute dynamically, you can do so by directly setting the attribute to the new value. Here's how you can do it:

llm._completion_to_prompt = new_function

Where new_function is the new function you want to use to process the prompt. This will change the completion_to_prompt for the Replicate instance, and the new function will be used in subsequent predictions.

However, please note that _completion_to_prompt is a private attribute, which means it's not intended to be accessed directly outside of the class. Changing it directly could lead to unexpected behavior. It would be better to add a method in the Replicate class to change it, like this:

class Replicate:
    ...
    def set_completion_to_prompt(self, new_function):
        self._completion_to_prompt = new_function
    ...

Then you can use this method to change completion_to_prompt:

llm.set_completion_to_prompt(new_function)

This is a safer and more Pythonic way to change the attribute. However, this would require modifying the Replicate class in the LlamaIndex codebase.

If you're still having issues after trying these solutions, please provide more details about your use case and the exact error message you're getting, and I'll be happy to help further.

Sources

• llama_index/llm_predictor/base.py
• llama_index/llms/replicate.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.