Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passing --measurement-time works but returns error 101 #59

Open
winksaville opened this issue Jan 12, 2023 · 0 comments
Open

Passing --measurement-time works but returns error 101 #59

winksaville opened this issue Jan 12, 2023 · 0 comments

Comments

@winksaville
Copy link

When I use --measurement-time criterion it works as expected, but has an exit status: 101 and echo $? returns 1. But without itecho $? returns 0. I've used time so with you can see that "measurement-time" increased. See below for a little more info.

$ time cargo criterion -- --measurement-time=10
    Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs                                                                             
                        time:   [48.865 ns 48.942 ns 49.027 ns]
                        change: [-4.7610% -4.3953% -4.0512%] (p = 0.00 < 0.05)
                        Performance has improved.

sm_enum_msgs_any/sm_enum_msgs_any                                                                             
                        time:   [62.483 ns 62.589 ns 62.693 ns]
                        change: [-5.8203% -5.0130% -4.3315%] (p = 0.00 < 0.05)
                        Performance has improved.

sm_individual_msgs_any/sm_individual_msgs_any                                                                             
                        time:   [45.092 ns 45.393 ns 45.704 ns]
                        change: [-3.6137% -2.9714% -2.2768%] (p = 0.00 < 0.05)
                        Performance has improved.

error: Unrecognized option: 'measurement-time'
Error: Non-Criterion.rs benchmark target exper_message_trait exited with exit status: 101

real    0m53.730s
user    1m14.721s
sys     0m0.818s

$ echo $?
1

$ time cargo criterion
    Finished bench [optimized] target(s) in 0.03s
sm_enum_msgs/sm_enum_msgs                                                                             
                        time:   [50.109 ns 50.196 ns 50.283 ns]
                        change: [+2.1998% +2.4297% +2.6668%] (p = 0.00 < 0.05)
                        Performance has regressed.

sm_enum_msgs_any/sm_enum_msgs_any                                                                            
                        time:   [63.337 ns 63.442 ns 63.566 ns]
                        change: [+1.6191% +2.0968% +2.5335%] (p = 0.00 < 0.05)
                        Performance has regressed.

sm_individual_msgs_any/sm_individual_msgs_any                                                                             
                        time:   [48.363 ns 48.574 ns 48.759 ns]
                        change: [+4.3939% +5.0563% +5.7090%] (p = 0.00 < 0.05)
                        Performance has regressed.


running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


real    0m39.131s
user    1m1.939s
sys     0m0.796s

$ echo $?
0

If you interrupt the "interactive output" you can see "estimated 10.000 s (195M iterations)":

$ time cargo criterion -- --measurement-time=10
    Finished bench [optimized] target(s) in 0.03s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 10.000 s (195M iterations)^C

real    0m5.631s
user    0m0.177s
sys     0m0.051s

Where as without --measurement-time we see "estimated 5.0001 s (97M iterations)":

$ time cargo criterion
    Finished bench [optimized] target(s) in 0.02s
Benchmarking sm_enum_msgs/sm_enum_msgs: Collecting 100 samples in estimated 5.0001 s (97M iterations)^C

real    0m4.583s
user    0m0.152s
sys     0m0.058s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant