Test order

[IceflowRE]

Hi guys i couldnt find something about it so:

Can the test order be different from run to run?
or may the test order different under Linux and Windows, (this is what i discovered so far, afaik).

WIth test order i mean the order how test classes are executed not the tests inside a class.

[sirosen]

The current test loader can't guarantee test ordering will be consistent in all situations.
The loader uses python stdlib APIs like os.listdir, and doesn't sort the results. The order of results could change in a few circumstances.

We could try to change this if it's a big deal, but I'm not sure how easy or hard it would be.

Since this is the only time I think this has been asked, I'm going to close without trying to add this to the docs.

[mzient]

@sirosen
Bumping up the issue. We introduced timing into our CI so we can see how long it takes for given point in the tests to be reached (this way we can identify misbehaving nodes in the CI cluster), but since we switched from nosetests to nose2 this system no longer gives us meaningful results.

[sirosen]

It's been quite a while at this point since I was able (and, TBH, strongly motivated 😢 ) to devote serious time to nose2.
I ought to cut a release just to ensure that the latest version is one which has been tested against py3.12 and do some other maintenance stuff.

In that context, what I can say is:

• I'd definitely be open to a PR to change this behavior
• I would probably not want to include it in the next release (because it has been too long since the last one already)
• I'm unlikely to work on this in the near term

That said, this is definitely a good enough reason to reopen this issue.