More Model Providers support 更多模型服务商支持

[arvinxx]

LobeChat currently supports many model providers, and considering the development of the AI ecosystem, we have specifically opened this discussion to record everyone's needs for new model service providers.

LobeChat 目前已经支持了诸多模型供应商，考虑到 AI 生态的发展，我们专门开放此贴用于记录大家对于新模型服务商的需求。

---

Currently, the service providers currently supported by LobeChat are as follows:

目前 LobeChat 已支持的服务商如下：

• OpenAI
• Azure OpenAI
  • ✨ feat: support Azure OpenAI #177
• AWS Bedrock
  • ✨ feat: support Google / Zhipu / AWS Bedrock model providers #1173
  • Claude3: ✨ feat: add claude 3 to bedrock provider #1551
• Google Model （Gemini Pro 、Gemini Pro Vision）
  • ✨ feat: support Google / Zhipu / AWS Bedrock model providers #1173
  • Gemini 1.5 pro: 🐛 fix: fix Google Gemini pro 1.5 and system role issue #1801
• Zhipu AI (智谱AI)
  • ✨ feat: support Google / Zhipu / AWS Bedrock model providers #1173
• Moonshot AI (月之暗面) by @arvinxx
  • ✨ feat: support Moonshot AI #1232
• Ollama (Local LLM) Thanks to @sjy
  • ✨ feat(llm): support Ollama AI Provider (local llm) #1265
• Perplexity AI Thanks to @gijigae
  • ✨ feat: support the perplexity provider #1339
• Anthropic Thanks to @danielglh
  • ✨ feat: support anthropic as model provider #1409
• Mistral Thanks to @danielglh
  • ✨ feat: support mistral as model provider #1455
• Groq Thanks to @jcdiv47 and @pzcn
  • ✨ feat: add support for model provider Groq #1562
  • Add Groq localization #1570
  • ✨ feat: support model provider groq #1569
• OpenRouter: Thanks to @danielglh
  • ✨ feat: support openrouter as a new model provider #1572
• 01.AI（零一万物）: Thanks to @MapleEve
  • ✨ feat: support 01.AI as a new provider #1627
• TogetherAI: Thanks to @phsontung
  • ✨ feat: Support TogetherAI as new model provider #1709
• MiniMax Thanks to @lwshen
  • ✨ feat: support minimax as provider #2087
• DeepSeek: Thanks to @RubuJam
  • ✨ feat: Support DeepSeek as new model provider #2446
  • 🐛 fix: fix DeepSeek using wrong model ID #2484
• Qwen: Thanks to @justmywyw
  • ✨ feat: Support Qwen as new model provider #2715
• Stepfun(阶跃星辰)： by @arvinxx
  • 💄 style: add stepfun as a new provider #2803
• Baichuan (百川）： Thanks to @hezhijie0327
  • ✨ feat: Add Baichuan model provider #3097
• Taichu(太初): Thanks to @hezhijie0327
  • ✨ feat: Add Taichu model provider #3129
• 360 AI（360智脑） : Thanks to @hezhijie0327
  • ✨ feat: Add 360AI model provider #3130
• Novita AI: Thanks to @jasonhp
  • 💄 style: Add new model provider Novita AI #3177
• SiliconCloud: Thanks to @sxjeru
  • ✨ feat: Add SiliconCloud model provider #3092
• Fireworks AI Thanks to @hezhijie0327
  • ✨ feat: Add Fireworks AI Model Provider #3392
• Spark (讯飞星火): Thanks to @hezhijie0327
  • ✨ feat: Add Spark model provider #3098
• Github Models: Thanks to @CloudPassenger
  • ✨ feat: add Github Models provider #3869
• AI21 Labs: Thanks to @hezhijie0327
  • ✨ feat: add Ai21Labs model provider #3727
• Tencent Hunyuan(腾讯混元): Thanks to @hezhijie0327
  • ✨ feat: add Hunyuan(Tencent) model provider #4147
• Wenxin(文心一言): by @arvinxx
  • ✨ feat: add wenxin as a new provider #2802
  • ✨ feat: add wenxin model provider #4018
• Hugging Face Inference API: by @arvinxx
  • ✨ feat: add HuggingFace Model Provider #4225
• SenseNova: Thanks to @hezhijie0327
• ✨ feat: add SenseNova (商汤) model provider #4162
• Cloudflare AI: Thanks to @BrandonStudio , @sxjeru
  • ✨ feat: Support Cloudflare Workers AI #3402
  • ✨ feat: Support Cloudflare Workers AI #2966
• xAI : Thanks to @hezhijie0327
  • ✨ feat: add support xAI provider #4627
• InternLM: Thanks to @hezhijie0327
  • ✨ feat: add support InternLM (书生浦语) provider #4711
• Gitee AI: Thanks to @hezhijie0327
  • ✨ feat: add Gitee AI model provider #4716

WIP：

• Cohere : Thanks to @ystoneman
  • Add initial implementation for Cohere provider support #2539
• Doubao: Thanks to @sxjeru
  • ✨ feat: Support Doubao Models #3055
• LM Studio: by @arvinxx
  • ✨ feat: add lm studio #4483
• Google Vertex AI: by @arvinxx
  • ✨ feat: support vertex ai #4487

Plan：

• replicate

Important

If you have any requests for new model suppliers, feel free to comment below. When the number of upvotes exceeds 15, we will consider implementing it.

如果大家对于新模型供应商有诉求，欢迎在下方跟帖，当up数超过 15 时，我们会考虑将其实现

---

PRs Welcome 💞

• A demo PR of add OpenAI Compatible Provider: 📑 [DEMO] This is a demo PR of how to add an OpenAI Compatible Provider #2804

Tip

We also welcome PRs. We have received many contributions regarding model Providers, so we have maintained a library of icons for LLMs and Provider vendors, facilitating everyone's future development. https://github.com/lobehub/lobe-icons

同时欢迎 PR，我们已经收获到许多关于模型 Provider 的贡献, 因此我们维护了一个 LLMs 和 Provider 供应商的图标库，方便大家后续开发 https://github.com/lobehub/lobe-icons

[lesonky]

申请增加GCP的Vertex AI大模型服务，此类服务用服务账号身份认证。可以满足企业级数据合规要求。🙏

[arvinxx]

the upvote is more than 8, we will add Google Vertex AI Soon.

[AiharaMahiru]

There are still fewer API providers that offer the Claude family of models, I've used AWS before as a Claude API, but asking me to provide sensitive data such as credit card statements is unacceptable!

[CXwudi]

Beside Azure ML Studio, there is also an Azure AI Studio at https://ai.azure.com/

Here I deployed the LLaMa 3.1 405B and this is the sample code from Azure code snippet:

# pip install azure-ai-inference
import os
from azure.ai.inference import ChatCompletionsClient
from azure.core.credentials import AzureKeyCredential

api_key = os.getenv("AZURE_INFERENCE_CREDENTIAL", '')
if not api_key:
  raise Exception("A key should be provided to invoke the endpoint")

client = ChatCompletionsClient(
    endpoint='https://Meta-Llama-3-1-405B-Instruct-bft.eastus.models.ai.azure.com',
    credential=AzureKeyCredential(api_key)
)

model_info = client.get_model_info()
print("Model name:", model_info.model_name)
print("Model type:", model_info.model_type)
print("Model provider name:", model_info.model_provider_name)

payload = {
  "messages": [
    {
      "role": "user",
      "content": "I am going to Paris, what should I see?"
    },
    {
      "role": "assistant",
      "content": "Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world."
    },
    {
      "role": "user",
      "content": "What is so great about #1?"
    }
  ],
  "max_tokens": 4096,
  "temperature": 0.8,
  "top_p": 0.1,
  "presence_penalty": 0
}
response = client.complete(payload)

print("Response:", response.choices[0].message.content)
print("Model:", response.model)
print("Usage:")
print("	Prompt tokens:", response.usage.prompt_tokens)
print("	Total tokens:", response.usage.total_tokens)
print("	Completion tokens:", response.usage.completion_tokens)

[BrandonStudio]

Is this compatible with azureOpenAI?

[razertory]

Cerebras AI

[lyc0826]

建议增加对“无问芯穹”的支持。
网址：infini-ai.com

[lyfhq]

LM studio +

[e1100x]

AIMLAPI.com

[BrandonStudio]

什么时候能更新一下模板pr呢？原来那个已经不适用了吧

[arvinxx]

我最近加一个 LM Studio，到时候以那个为新的好了

[CrazyGoudanli]

期望尽快能将豆包合并到主分支。

[CrazyGoudanli]

@arvinxx We hope to merge Doubao into the main branch as soon as possible.

[yordyi]

麻烦添加英伟达云大模型,新晋王者:https://build.nvidia.com/nvidia/llama-3_1-nemotron-70b-instruct?snippet_tab=Python

[534002646]

@arvinxx 希望可以增加个按钮，增加openai api兼容的模型

[Vulcanraven91]

Some days ago I discovered Portkey which is awesome, what about adding it to LobeChat?

What is Portkey?"

Portkey AI is a comprehensive platform designed to streamline and enhance AI integration for developers and organizations. It serves as a unified interface for interacting with over 250 AI models, offering advanced tools for control, visibility, and security in your Generative AI apps.

It takes 2 mins to integrate and with that, it starts monitoring all of your LLM requests and makes your app resilient, secure, performant, and more accurate at the same time.

Integrate in 3 Lines of Code for Python, NodeJS, OpenAI NodeJS SDK, RestAPI

NodeJS:

import Portkey from 'portkey-ai';

const portkey = new Portkey({
  apiKey: "YOUR_PORTKEY_API_KEY",
  virtualKey: "YOUR_VIRTUAL_KEY"
});

async function createChatCompletion() {
  const chat_complete = await portkey.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: "Hello!" }
    ]
  });
  console.log(chat_complete.choices[0].message.content);
}

createChatCompletion();

FAQs

Will Portkey increase the latency of my API requests?

Portkey is hosted on edge workers throughout the world, ensuring minimal latency. Our benchmarks estimate a total latency addition between 20-40ms compared to direct API calls. This slight increase is often offset by the benefits of our caching and routing optimizations.

Is my data secure?

Portkey AI is ISO:27001 and SOC 2 certified, and GDPR & HIPAA compliant. We maintain best practices for service security, data storage, and retrieval. All data is encrypted in transit and at rest using industry-standard AES-256 encryption.

For enhanced security, we offer:

1. On request, we can enable a feature that does NOT store any of your request and response body objects in Portkey datastores or our logs.
2. For enterprises, we offer managed hosting to deploy Portkey inside private clouds.

For more information on these options, contact us at [email protected].

Will Portkey scale if my app explodes?

Portkey is built on scalable infrastructure and can handle millions of requests per minute with very high concurrency. We currently serve over 25M requests daily with a 99.99% uptime. Our edge architecture & scaling capabilities ensure we can accommodate sudden spikes in traffic without performance degradation.

View our Status Page

Does Portkey impose timeouts on requests?

We DO NOT impose any explicit timeout for our free OR paid plans currently. While we don't time out requests on our end, we recommend implementing client-side timeouts appropriate for your use case to handle potential network issues or upstream API delays.

Do you support SSO?

Yes! We support SSO with any custom OIDC provider.

What are the pricing options? Is there a free trial?

Portkey's Gateway is open source and free to use.
On managed version, Portkey offers a free plan with 10k requests per month. We also offer paid plans with more requests and additional features.
`

[arvinxx]

If portkey can use baseURL, it's possible. Or it's not reality to integrate their sdk

[Vulcanraven91]

Yes you can use baseURL but you must append 2 headers.
Example curl:

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: openai" \
  -d '{
    "model": "gpt-4-turbo",
    "messages": [{
        "role": "system",
        "content": "You are a helpful assistant."
      },{
        "role": "user",
        "content": "Hello!"
      }]
  }'

[sxjeru]

It looks like a project similar to songquanpeng/one-api , with gateway to accelerate.

ref docs.

import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai';

const openai = new OpenAI({
  apiKey: 'YOUR_OPENAI_API_KEY',
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: "openai",
    apiKey: "YOUR_PORTKEY_API_KEY"
  })
});

async function createChatCompletion() {
  const chat_complete = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: "Hello!" }
    ]
  });
  console.log(chat_complete.choices[0].message.content);
}

createChatCompletion();

[darkelfzdy]

希望能够提供自定义模型设置（openai api标准），有时候需要增加很多相同api标准的模型，提高了灵活度

[arvinxx]

如果我们在付费镜像里做这个特性，你会考虑购买么？