Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Control concurrency of evals #98

Open
filip-michalsky opened this issue Oct 7, 2024 · 0 comments
Open

Control concurrency of evals #98

filip-michalsky opened this issue Oct 7, 2024 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers
Milestone

Comments

@filip-michalsky
Copy link
Collaborator

running 10+ evals on different browsers in parallel locally can become computationally challenging.

Braintrust offers a concurrency semaphor:
https://www.braintrust.dev/docs/guides/evals/run

import { Factuality, Levenshtein } from "autoevals";
import { Eval } from "braintrust";
 
Eval("Say Hi Bot", {
  data: () =>
    Array.from({ length: 100 }, (_, i) => ({
      input: `${i}`,
      expected: `${i + 1}`,
    })),
  task: (input) => {
    return input + 1;
  },
  scores: [Factuality, Levenshtein],
  maxConcurrency: 5, // Run 5 tests concurrently
});
@filip-michalsky filip-michalsky added enhancement New feature or request good first issue Good for newcomers labels Nov 4, 2024
@kamath kamath added this to Stagehand Nov 29, 2024
@kamath kamath added this to the Evaluation milestone Nov 29, 2024
@kamath kamath moved this to Todo in Stagehand Nov 29, 2024
@kamath kamath removed the status in Stagehand Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
Status: No status
Development

No branches or pull requests

2 participants