How to use Azure OpenAI together with Azure embedding for a query engine?

[aiwalter]

If I am correct, I can give a service_context to an index only once and the query engine does get the index. So how can I still specify the embedding separately to the LLM using both from Azure? Please provide example :) I am instantiating the query engine via from_args

[dosubot[bot]]

🤖

Hey @aiwalter! Good to see you back. Hope you're cutting through code like a hot knife through butter. Let's dive into this new challenge together.

To specify the embedding separately to the LLM using Azure OpenAI and Azure embedding for a query engine, you can instantiate the AzureOpenAI and AzureOpenAIEmbedding separately with their respective model and deployment names. Then, you can create a ServiceContext with these two separate instances and set it as the global service context. Here is an example:

from llama_index import AzureOpenAI, AzureOpenAIEmbedding, ServiceContext, set_global_service_context

llm = AzureOpenAI(
    model="gpt-35-turbo-16k",
    deployment_name="my-custom-llm",
    api_key=api_key,
    azure_endpoint=azure_endpoint,
    api_version=api_version,
)
embed_model = AzureOpenAIEmbedding(
    model="text-embedding-ada-002",
    deployment_name="my-custom-embedding",
    api_key=api_key,
    azure_endpoint=azure_endpoint,
    api_version=api_version,
)

service_context = ServiceContext.from_defaults(
    llm=llm,
    embed_model=embed_model,
)

set_global_service_context(service_context)

In this code, llm is the instance of AzureOpenAI for text generation and embed_model is the instance of AzureOpenAIEmbedding for retrieval. Both are passed to the ServiceContext.from_defaults method to create a ServiceContext instance. This instance is then set as the global service context using set_global_service_context(service_context).

The service_context can only be given to an index once and the query engine obtains the index. However, since the service_context contains both the llm and embed_model, the query engine can use both of them separately. You can find more details in the AzureOpenAI notebook in the LlamaIndex repository.

Regarding your question about the from_args method, I wasn't able to find specific information about this method in the LlamaIndex repository. Could you please provide more details or context about this method? For example, in which class or module is this method located? This would help me provide a more accurate answer.

Sources

• docs/examples/customization/llms/AzureOpenAI.ipynb
• llama_index/service_context.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[aiwalter]

whats difference between llm param and llm_predictor @dosubot

[dosubot[bot]]

🤖

The llm and llm_predictor parameters in the LlamaIndex data framework for LLM applications serve different purposes.

The llm parameter is used to specify the type of Language Learning Model (LLM) to be used. It is used in the load_llm function to load a specific LLM by its name. If llm is not "default", it will resolve the 'llm' to an instance of LLM and create an LLMPredictor with the resolved 'llm'. If llm is "default", it will use the 'llm_predictor' provided or the one from the global service context. If both 'llm' and 'llm_predictor' are specified, a ValueError will be raised. You can refer to the llm versus llm_predict issue for more details.

On the other hand, the llm_predictor parameter is used to specify the LLMPredictor to be used. The 'llm_predictor' is essentially a wrapper that formats prompts before calling the LLM, and it is automatically created when you pass in the 'llm' object to the service context. You generally only need to worry about the LLM object itself.

In the context of the VellumPredictor class, the llm property raises a NotImplementedError with the message "Vellum does not expose the LLM." This suggests that the underlying LLM used by the VellumPredictor is not directly accessible or modifiable by the user. You can refer to the vellum/predictor.py for more details.

I hope this clarifies the difference between the llm and llm_predictor parameters. If you have any more questions, feel free to ask.

Sources

• llm versus llm_predict
• llama_index/llm_predictor/vellum/predictor.py
• llama_index/llms/loading.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

llm and llm_predictor are essentially the same now. LLMPredictor is actually deprecated now, since llm has the same interface