What's the recommended way for caching in llama_index?

[ibicdev]

It seems cache in LLMPredictor (or LLMPredictor itself) has been deprecated, and GPTCache isn't fully integrated with llama_index, so is Portkey the only recommended way of doing caching in llama_index? In this case, it would be helpful to have local/open-source caching support that doesn't require 3rd party services. Thanks.

[dosubot[bot]]

🤖

Yes, you are correct. With the deprecation of cache in LLMPredictor and the incomplete integration of GPTCache, Portkey is currently the recommended method for caching in LlamaIndex.

The Portkey class integrates with LlamaIndex and allows for enhanced monitoring. It also provides a method to add LLM parameters, which can be used for fallbacks or load-balancing as specified in the mode.

Regarding local or open-source caching options, the Portkey class constructor (__init__) accepts a base_url parameter. This parameter can be used to specify the URL of a self-hosted server, such as Rubeus, which is the open-source version of Portkey. This suggests that you can use local or open-source caching options that don't require third-party services.

Here is the relevant code:

def __init__(
    self,
    *,
    mode: Union["Modes", "ModesLiteral"],
    api_key: Optional[str] = None,
    base_url: Optional[str] = None,
    system_prompt: Optional[str] = None,
    messages_to_prompt: Optional[Callable[[Sequence[ChatMessage]], str]] = None,
    completion_to_prompt: Optional[Callable[[str], str]] = None,
    pydantic_program_mode: PydanticProgramMode = PydanticProgramMode.DEFAULT,
    output_parser: Optional[BaseOutputParser] = None,
) -> None:
    """
    Initialize a Portkey instance.

    Args:
        mode (Optional[Modes]): The mode for using the Portkey integration
        (default: Modes.SINGLE).
        api_key (Optional[str]): The API key to authenticate with Portkey.
        base_url (Optional[str]): The Base url to the self hosted rubeus \
            (the opensource version of portkey) or any other self hosted server.
    """
    ...
    if base_url is not None:
        portkey.base_url = base_url
    ...

This code shows that you can initialize a Portkey instance with a base_url pointing to a self-hosted server, allowing for local or open-source caching options. You can find this in the llama_index/llms/portkey.py file.

As for the LLMPredictor class, it has been deprecated to encourage the use of a new method llm.structured_predict() instead of using the StructuredLLMPredictor class directly. This suggests a shift towards a more streamlined or updated approach for structured predictions within the LlamaIndex framework.

I hope this helps! If you have any more questions, feel free to ask.

Sources

• llama_index/llms/portkey.py
• llama_index/llm_predictor/base.py
• llama_index/llm_predictor/structured.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.