Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new integration with Opik Tracking tool #11501

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

Lothiraldan
Copy link

Summary

This PR integrates Opik (https://github.com/comet-ml/opik), an open-source platform for evaluating, testing, and monitoring LLM applications. It features built-in evaluation metrics and data annotation. Closes #11407, requested by a community user.

Screenshots

Before After
main branch

Checklist

Important

Please review the checklist below before submitting your pull request.

  • This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs/pulls)
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. 💪 enhancement New feature or request 📚 documentation Improvements or additions to documentation labels Dec 9, 2024
@ZhouhaoJiang
Copy link
Collaborator

ZhouhaoJiang commented Dec 12, 2024

CleanShot 2024-12-12 at 15 43 28@2x
CleanShot 2024-12-12 at 15 53 23@2x
The elements have a position problem whenever I add another ops trace provider.

@ZhouhaoJiang
Copy link
Collaborator

ZhouhaoJiang commented Dec 12, 2024

CleanShot 2024-12-12 at 15 49 36@2x
CleanShot 2024-12-12 at 15 50 49@2x
When I add a credential, I don't fill in the Url field, because I see that it has a default value, but the actual request interface does not include this value, in order to better user experience should set it to the default value.

I will remove the api key after I finish test it.

@ZhouhaoJiang
Copy link
Collaborator

ZhouhaoJiang commented Dec 12, 2024

I have tested basic chat, agent chat, and workflow. There have no problems with data tracing.

Also add better documentation link for Opik configuration in the UI
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Dec 12, 2024
@Lothiraldan
Copy link
Author

@ZhouhaoJiang Thanks for your review. I fixed the issue for the default url, it was actually on the backend. I'm guessing that the frontend was sending an empty string for it.

I'm not sure to understand your feedback about the position of the Opik Tracing Provider. I added Opik at the end of both lists, the first one with configured tracing providers and the second one with non-configured tracing providers. Do you prefer a different behavior?

@ZhouhaoJiang
Copy link
Collaborator

@ZhouhaoJiang Thanks for your review. I fixed the issue for the default url, it was actually on the backend. I'm guessing that the frontend was sending an empty string for it.

I'm not sure to understand your feedback about the position of the Opik Tracing Provider. I added Opik at the end of both lists, the first one with configured tracing providers and the second one with non-configured tracing providers. Do you prefer a different behavior?

I think my explanation is not clear. What I mean is that after one provider is configured, there is no gap between the components of the other two providers, and there is a problem with the ui

@Lothiraldan
Copy link
Author

@ZhouhaoJiang I was able to fix the spacing issue but I realized that the Opik provider is missing the "IN USE" tag, I will check it out and fix the merge conflict on api/poetry.lock

Screenshot 2024-12-13 at 10-30-36 Long Story Generator (Iteration) - Dify
Screenshot 2024-12-13 at 10-30-19 Long Story Generator (Iteration) - Dify

@Lothiraldan
Copy link
Author

I fixed the merged conflict on poetry.lock and finally understood that you can have only a single provider enabled per project. That's why I only saw only one provider with the "IN USE" tag. I think I fixed everything, let me know otherwise

Copy link
Collaborator

@ZhouhaoJiang ZhouhaoJiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have updated the workflow run id as workflow trace id and move the workflow app log id into metadata, So please update the workflow trace.

   def workflow_trace(self, trace_info: WorkflowTraceInfo):
        dify_trace_id = trace_info.message_id or trace_info.workflow_app_log_id or trace_info.workflow_run_id
        opik_trace_id = uuid4_to_uuid7(trace_info.start_time, dify_trace_id)

        if trace_info.message_id:
            trace_data = {
                "id": opik_trace_id,
                "name": TraceTaskName.MESSAGE_TRACE.value,
                "start_time": trace_info.start_time,
                "end_time": trace_info.end_time,
                "metadata": wrap_metadata(trace_info.metadata, message_id=trace_info.message_id),
                "input": wrap_dict("input", trace_info.workflow_run_inputs),
                "output": wrap_dict("output", trace_info.workflow_run_outputs),
                "tags": ["message", "workflow"],
                "project_name": self.project,
            }
            self.add_trace(trace_data)

You can refer to this pr #11585

@Lothiraldan
Copy link
Author

@ZhouhaoJiang Thanks, I will update the workflow trace method accordingly. Can you remind me in which case the message_if field in the WorkflowTraceInfo could be None? I want to double-check that everything works nicely after the update.

iamjoel
iamjoel previously approved these changes Dec 17, 2024
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 17, 2024
@Lothiraldan
Copy link
Author

@ZhouhaoJiang I've updated the Opik tracer to use workflow_run_id or message_id when present.

@laipz8200
Copy link
Member

Thank you for this work, we'll test this feature in our development environment and may need to do some work on documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📚 documentation Improvements or additions to documentation 💪 enhancement New feature or request lgtm This PR has been approved by a maintainer size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for new Ops Tool, Opik
4 participants