Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reliablility report is inaccurate #647

Open
MoLow opened this issue Sep 29, 2022 · 2 comments
Open

Reliablility report is inaccurate #647

MoLow opened this issue Sep 29, 2022 · 2 comments

Comments

@MoLow
Copy link
Member

MoLow commented Sep 29, 2022

I started investigating nodejs/node#44805
when taking a look at nodejs/reliability#388:

Reason sequential/test-watch-mode-inspect
Type JS_TEST_FAILURE
Failed PR 9 (nodejs/node#44787, nodejs/node#44741, nodejs/node#44767, nodejs/node#44770, nodejs/node#44376, nodejs/node#44693, nodejs/node#44686, nodejs/node#44771, nodejs/node#44711)
Appeared test-nearform-macos10.15-x64-2, test-nearform-macos10.15-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46819/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46857/
Example
not ok 3789 sequential/test-watch-mode-inspect
  ---
  duration_ms: 120.67
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    TAP version 13
    (node:37761) ExperimentalWarning: The test runner is an experimental feature. This feature could change at any time
    (Use `node --trace-warnings ...` to show where the warning was created)
  ...


the first PR in the Failed PR row has two CI runs prior to this report: 46819, 46808

the first CI run reports a failure for this based on the previous osx1015 run https://ci.nodejs.org/job/node-test-commit-osx/47578/ that did not actually run in this build (due to java.net.ConnectException: Connection timed out) but Jenkins still considers the child of a previous run to be the child of this run.

this obviously happened in other PRS/CI runs as well, wich makes the report links and count very hard to rely on when investigating flaky test issues

@richardlau
Copy link
Member

The reports here are generated using the ncu-ci walk ... command from node-core-utils.

@MoLow
Copy link
Member Author

MoLow commented Sep 30, 2022

So this issue should be transferred, but I don't have write accees on the ncu repository so I cannot transfer it

@richardlau richardlau transferred this issue from nodejs/reliability Sep 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants