Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch evaluation inference scripts #155

Open
1 of 4 tasks
yqzhishen opened this issue Nov 18, 2023 · 1 comment
Open
1 of 4 tasks

Batch evaluation inference scripts #155

yqzhishen opened this issue Nov 18, 2023 · 1 comment

Comments

@yqzhishen
Copy link
Member

yqzhishen commented Nov 18, 2023

This issue (and its corresponding PR) will add evaluate.py, which applies a model on all items from an existing dataset.

Motivation

Sometimes people just want to hear what if songs of one person are sung in the voice of another person, like what most SVC pipelines actually do.

Also, this may be useful finding edge cases or labeling mistakes (so that the most severe mistakes can be fixed quickly), especially with the help of SVC pipelines.

Possible workflow to fully evaluate a dataset might be:

  1. Run evaluate.py with a model trained from accurately labeled datasets (the reference model) on the target dataset and get the reference audio samples.
  2. (Optional but recommended) Train a SVC model on the target dataset and apply it to the evaluation results to convert their timbre to that of the target dataset.
  3. Calculate the mel spectrogram likelihood between the reference samples and the original recordings in the target dataset.
  4. Sort the likelihood values in ascending order and look into the items with the lowest likelihood.

TODO

  • Support mel likelihood metrics during acoustic model training.
  • Make binarizers reusable, for extracting parameters without saving them.
  • Implement testing-related methods in task classes. Should be able to generate both audio samples and likelihood values.
  • Add the main CLI entry, which is evaluate.py, with proper control options.
@yqzhishen yqzhishen converted this from a draft issue Nov 18, 2023
@yqzhishen yqzhishen moved this from Todo to In Progress in DiffSinger Nov 22, 2023
@blueyred
Copy link

This would be amazing - being able to get an evaluation on a dataset would save so much time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

No branches or pull requests

2 participants