-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improved outputs to analyze integration test results #445
base: master
Are you sure you want to change the base?
Conversation
1e36329
to
9e38c4e
Compare
96bfccb
to
f5ba137
Compare
3dcff8c
to
7e81ca3
Compare
…f baseline and new results for CI comparisons.
7e81ca3
to
a568072
Compare
|
||
return key_diffs | ||
|
||
def compare_dict_values(self, dict1, dict2, percent_threshold=10, abs_threshold=1000): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What should the threshold be when deciding to report percent changes of json values? Should the thresholds for agg_results be different from ecm_results?
note - percent threshold means that only differences >= to that will be reported, absolute threshold only reports differences if the original values exceed that number to prevent outputting large percent diffs due to small numbers.
|
||
!tests/integration_testing/results/plots/tech_potential/*.xlsx | ||
!tests/integration_testing/results/plots/max_adopt_potential/*.xlsx |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To overwrite the ignored .xlsx files specified above
Fixes #415
Introduces a class to compare integration test results on a branch with the results stored on master. A previous PR (#440) added all integration test results to master. This PR provides a way of evaluating the differences between the working branch and master, which include:
compare_results.py
to:agg_results_key_diffs.csv
andecm_results_key_diffs.csv
)agg_results_value_diffs.csv
andecm_results_value_diffs.csv
)Summary_Data-MAP.xlsx
andSummary_Data-TP.xlsx
(output toSummary_Data-MAP_percent_diffs.csv
andSummary_Data-TP_percent_diffs.csv
)agg_results.json
orecm_results.json
, then:tests/integration_tests/results_base
tests/integration_tests/compare_results.py
Example Outputs
Example CI artifacts are found at https://github.com/trynthink/scout/actions/runs/11943859504
Example
*_results_key_diffs.csv
:Example
*_results_value_diffs.csv
:Example
Summary_Data-*_percent_diffs.xlsx
:Same format as original xlsx files, but values are the percent differences