Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for Groq's Hosted Models via Groq's OpenAI Compatibility API #682

Open
jackspirou opened this issue Mar 14, 2024 · 0 comments · May be fixed by #683
Open

Add Support for Groq's Hosted Models via Groq's OpenAI Compatibility API #682

jackspirou opened this issue Mar 14, 2024 · 0 comments · May be fixed by #683
Labels
enhancement New feature or request

Comments

@jackspirou
Copy link

jackspirou commented Mar 14, 2024

Is your feature request related to a problem? Please describe.
I want the option to use Go to access Groq hosted models, due to their greater inference speed.

Describe the solution you'd like
I would like to suggest this project do two three things:

  1. Use Groq's OpenAI Compatibility for minimal changes
  2. Provide a list of Groq models for easy usage
  3. Add basic documentation since this more of a side-feature with huge value than a main case.

Describe alternatives you've considered
This could be a new/different library, but I think it makes sense more sense to build on the work done by this project.

Additional context

OpenAI Compatibility
Groq's APIs are designed to be compatible with OpenAI's, with the goal of making it easy to leverage Groq in applications you may have already built.

Supported Models
GroqCloud currently supports the following models:
LLaMA2-70b
Developer: Meta
Model Name: LLaMA2-70b-chat
Context Window: 4,096 tokens
API String: llama2-70b-4096
Mixtral-8x7b
Developer: Mistral
Model Name: Mixtral-8x7b-Instruct-v0.1
Context Window: 32,768 tokens
API String: mixtral-8x7b-32768
Gemma-7b-it
Developer: Google
Model Name: Gemma-7b-it
Context Window: 8,192 tokens
API String: gemma-7b-it

I'll submit a PR that outlines this change shortly.

EDIT FOR PR LINK: #683

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant