Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flaky fails with pytest==8.1.0 #198

Closed
skshetry opened this issue Mar 4, 2024 · 11 comments
Closed

flaky fails with pytest==8.1.0 #198

skshetry opened this issue Mar 4, 2024 · 11 comments

Comments

@skshetry
Copy link

skshetry commented Mar 4, 2024 

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.2/x64/bin/pytest", line 8, in <module>
    sys.exit(console_main())
             ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 195, in console_main
    code = main()
           ^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 153, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 335, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 138, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/helpconfig.py", line 105, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1141, in pytest_cmdline_parse
    self.parse(args)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1490, in parse
    self._preparse(args, addopts=addopts)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1377, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 414, in load_setuptools_entrypoints
    plugin = ep.load()
             ^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/assertion/rewrite.py", line 178, in exec_module
    exec(co, module.__dict__)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/flaky/flaky_pytest_plugin.py", line 5, in <module>
    from _pytest.runner import call_runtest_hook  # pylint:disable=import-error
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'call_runtest_hook' from '_pytest.runner' (/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py)

call_runtest_hook no longer exists and has been inlined. See pytest-dev/pytest#11854.
skshetry added a commit to iterative/dvc that referenced this issue Mar 4, 2024
@skshetry
limit pytest version to <8.1
d76efa6 
Change in pytest==8.1.0 breaks `flaky` plugin.

See box/flaky#198.
@skshetry skshetry mentioned this issue Mar 4, 2024
Merged
skshetry added a commit to iterative/dvc that referenced this issue Mar 4, 2024
@skshetry
limit pytest version to <8.1 (#10334)
bc95b65 
Change in pytest==8.1.0 breaks `flaky` plugin.

See box/flaky#198.
yngve-sk pushed a commit to yngve-sk/ert that referenced this issue Mar 4, 2024
Pin pytest due to flaky
3e41e21 
box/flaky#198
This was referenced Mar 4, 2024
Merged
Closed
yngve-sk added a commit to equinor/ert that referenced this issue Mar 4, 2024
@yngve-sk
Pin pytest due to flaky
0a4b5c5 
box/flaky#198
@FlorentClarret FlorentClarret mentioned this issue Mar 4, 2024
Merged
3 tasks
jonathan-eq pushed a commit to jonathan-eq/ert that referenced this issue Mar 4, 2024
@yngve-sk @jonathan-eq
Pin pytest due to flaky
37b64d3 
box/flaky#198
@azgabur azgabur mentioned this issue Mar 4, 2024
Merged
@jonathan-eq jonathan-eq mentioned this issue Mar 4, 2024
Merged
9 tasks
@anakin87 anakin87 mentioned this issue Mar 4, 2024
Merged
jonathan-eq added a commit to equinor/ert that referenced this issue Mar 4, 2024
@jonathan-eq @yngve-sk
Pin pytest due to flaky (#7337)
4999dbd 
box/flaky#198

Co-authored-by: Yngve S. Kristiansen <[email protected]>
@The-Compiler The-Compiler mentioned this issue Mar 4, 2024
Closed
4 tasks
@The-Compiler
Copy link

This project seems to be pretty dead (last release in 2020, last commit half a year ago, #192 with no reaction).

You might want to look into migrating to pytest-rerunfailures which is actively maintained and under the pytest-dev community namespace.

@shreyasvinaya shreyasvinaya mentioned this issue Mar 4, 2024
Closed
@anakin87 anakin87 mentioned this issue Mar 4, 2024
Merged
@jonathan-eq jonathan-eq mentioned this issue Mar 4, 2024
Closed
@The-Compiler The-Compiler mentioned this issue Mar 4, 2024
Closed
@qci-amos
Copy link

qci-amos commented Mar 4, 2024

flaky should either be fixed or it should be removed from the list here:
https://docs.pytest.org/en/stable/explanation/flaky.html#plugins

This was referenced Mar 4, 2024
Merged
Closed
@ananth102 ananth102 mentioned this issue Mar 10, 2024
Merged
ack-prow bot pushed a commit to aws-controllers-k8s/sagemaker-controller that referenced this issue Mar 10, 2024
@ananth102
Lock Pytest to 8.0.2 (#264)
8a77430 
The minor version bump seems to cause issues with flakey which might not be maintained anymore (box/flaky#198)

Issue #, if available: N/A

Description of changes: Lock pytest to 8.0.2

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@bwoodsend bwoodsend mentioned this issue Mar 10, 2024
Closed
nicoddemus added a commit to nicoddemus/pytest that referenced this issue Mar 10, 2024
@nicoddemus
Remove flaky plugin from the docs
958a3a3 
As discussed in box/flaky#198, `flaky` does not seem to be maintained anymore, so no longer recommend it in the documentation.
@nicoddemus nicoddemus mentioned this issue Mar 10, 2024
Closed
Jeff-Meadows added a commit that referenced this issue Mar 10, 2024
@Jeff-Meadows @jonathan-eq
Fix incompatibilities with newer dependencies (#201)
2bb1514 
This commit fixes an incompatibility with pytest 8.1.1,
and  some with python 3.12.

This commit also fixes up some pycodestyle and
pylint problems that snuck in with some earlier commits.

Fixes #199 and #198
---------

Co-authored-by: Jonathan Karlsen <[email protected]>
ananth102 added a commit to ananth102/ack-sagemaker-controller that referenced this issue Mar 10, 2024
@ananth102
!Lock Pytest to 8.0.2 (aws-controllers-k8s#264)
d31a5b7 
The minor version bump seems to cause issues with flakey which might not be maintained anymore (box/flaky#198)

Issue #, if available: N/A

Description of changes: Lock pytest to 8.0.2

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
ananth102 added a commit to ananth102/ack-sagemaker-controller that referenced this issue Mar 10, 2024
@ananth102
Lock Pytest to 8.0.2 (aws-controllers-k8s#264)
cf98fba 
The minor version bump seems to cause issues with flakey which might not be maintained anymore (box/flaky#198)

Issue #, if available: N/A

Description of changes: Lock pytest to 8.0.2

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@kloczek
Copy link

kloczek commented Mar 11, 2024

Just retested 3.8.0 with pytest 8.1.1 and still I see 5 failing units

Here is pytest output: 
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages
+ /usr/bin/pytest -ra -m 'not network'
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0
plugins: flaky-3.7.0
collected 77 items

test/test_flaky_decorator.py ...                                         [  3%]
test/test_flaky_plugin.py FF.                                            [  7%]
test/test_multiprocess_string_io.py FF                                   [ 10%]
test/test_pytest/pytest_generate_example/test_pytest_generate_example.py  [ 11%]
..                                                                       [ 12%]
test/test_pytest/test_flaky_pytest_plugin.py ........................... [ 48%]
.....................                                                    [ 75%]
test/test_pytest/test_pytest_example.py .xx.x...s......s                 [ 94%]
test/test_pytest/test_pytest_options_example.py F..                      [ 98%]
test/test_pytest/test_pytester_plugin.py .                               [100%]

=================================== FAILURES ===================================
_ TestFlakyPlugin.test_flaky_plugin_handles_non_ascii_byte_string_in_exception _

self = <test.test_flaky_plugin.TestFlakyPlugin testMethod=test_flaky_plugin_handles_non_ascii_byte_string_in_exception>

    def test_flaky_plugin_handles_non_ascii_byte_string_in_exception(self):
        mock_method_name = 'my_method'
        mock_exception = 'ńőń ȁŝćȉȉ ŝƭȕƒƒ'.encode('utf-16')
        mock_message = 'information about retries'
        # pylint:disable=protected-access
>       self._flaky_plugin._log_test_failure(
            mock_method_name,
            (ValueError.__name__, mock_exception, ''),
            mock_message,
        )

test/test_flaky_plugin.py:38:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/_flaky_plugin.py:38: in _log_test_failure
    formatted_exception_info = ''.join(format_exception(*err)).replace('\n', '\n\t').rstrip()
/usr/lib64/python3.9/traceback.py:120: in format_exception
    return list(TracebackException(
/usr/lib/python3.9/site-packages/exceptiongroup/_formatting.py:96: in __init__
    self.stack = traceback.StackSummary.extract(
/usr/lib64/python3.9/traceback.py:347: in extract
    for f, lineno in frame_gen:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

tb = ''

    def walk_tb(tb):
        """Walk a traceback yielding the frame and line number for each frame.

        This will follow tb.tb_next (and thus is in the opposite order to
        walk_stack). Usually used with StackSummary.extract.
        """
        while tb is not None:
>           yield tb.tb_frame, tb.tb_lineno
E           AttributeError: 'str' object has no attribute 'tb_frame'

/usr/lib64/python3.9/traceback.py:312: AttributeError
_____________ TestFlakyPlugin.test_flaky_plugin_identifies_failure _____________

self = <test.test_flaky_plugin.TestFlakyPlugin testMethod=test_flaky_plugin_identifies_failure>

    def test_flaky_plugin_identifies_failure(self):
>       for name, test in self._test_dataset:
E       ValueError: too many values to unpack (expected 2)

test/test_flaky_plugin.py:45: ValueError
________________ TestMultiprocessStringIO.test_write_then_read _________________

self = <test.test_multiprocess_string_io.TestMultiprocessStringIO testMethod=test_write_then_read>

    def test_write_then_read(self):
        for name, value in self._test_values.items():
            with self.subTest(name):
                for string_io in self._string_ios:
                    for item in value[0]:
                        string_io.write(item)
>                   self.assertEqual(string_io.getvalue(), value[1])
E                   AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ'
E                   - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ
E                   ?      -----------
E                   + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ

test/test_multiprocess_string_io.py:31: AssertionError
______________ TestMultiprocessStringIO.test_writelines_then_read ______________

self = <test.test_multiprocess_string_io.TestMultiprocessStringIO testMethod=test_writelines_then_read>

    def test_writelines_then_read(self):
        for name, value in self._test_values.items():
            with self.subTest(name):
                for string_io in self._string_ios:
                    string_io.writelines(value[0])
>               self.assertEqual(string_io.getvalue(), value[1])
E               AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ'
E               - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ
E               ?      -----------
E               + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ

test/test_multiprocess_string_io.py:38: AssertionError
_____________________________ test_something_flaky _____________________________

dummy_list = [0]

    def test_something_flaky(dummy_list=[]):
        # pylint:disable=dangerous-default-value
        dummy_list.append(0)
>       assert len(dummy_list) > 1
E       assert 1 > 1
E        +  where 1 = len([0])

test/test_pytest/test_pytest_options_example.py:11: AssertionError
================================== XFAILURES ===================================
________________ test_something_good_with_failing_setup_fixture ________________

    @pytest.fixture(scope='function')
    def failing_setup_fixture():
>       assert False
E       assert False

test/test_pytest/test_pytest_example.py:21: AssertionError
________________ test_something_good_with_failing_setup_fixture ________________

    @pytest.fixture(scope='function')
    def failing_setup_fixture():
>       assert False
E       assert False

test/test_pytest/test_pytest_example.py:21: AssertionError
___________________ TestExample.test_non_flaky_failing_thing ___________________

self = <test.test_pytest.test_pytest_example.TestExample object at 0x7f6bf65e6d60>

    @pytest.mark.xfail
    def test_non_flaky_failing_thing(self):
        """Flaky will also not interact with this test"""
>       assert self == 1
E       assert <test.test_pytest.test_pytest_example.TestExample object at 0x7f6bf65e6d60> == 1

test/test_pytest/test_pytest_example.py:40: AssertionError
=============================== warnings summary ===============================
test/test_flaky_plugin.py:0
  /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_flaky_plugin.py:0: PytestCollectionWarning: cannot collect test class 'TestCaseDataset' because it has a __new__ constructor (from: test/test_flaky_plugin.py)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===Flaky Test Report===

Flaky report texttest_something_flaky failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 1 > 1
 +  where 1 = len([0])
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:16>]
test_something_flaky passed 1 out of the required 1 times. Success!
test_something_good_with_failing_setup_fixture failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:85>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:843>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:878>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:82>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:158>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:515>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:512>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1780>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:687>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:541>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:570>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:647>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1079>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/setuponly.py:36>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1132>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:913>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:21>]
test_something_good_with_failing_setup_fixture failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:85>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:843>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:878>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:82>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:158>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:515>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:512>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1780>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:687>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:541>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:570>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:647>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1079>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/setuponly.py:36>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1132>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:913>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:21>]
test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExample._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:50>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times.
test_flaky_thing_that_fails_then_succeeds passed 2 out of the required 2 times. Success!
test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times.
test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 2 out of the required 2 times. Success!
test_flaky_thing_that_always_passes passed 1 out of the required 2 times. Running test again until it passes 2 times.
test_flaky_thing_that_always_passes passed 2 out of the required 2 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTests._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:87>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTestCase._threshold
        [<TracebackEntry /usr/lib64/python3.9/unittest/case.py:59>, <TracebackEntry /usr/lib64/python3.9/unittest/case.py:592>, <TracebackEntry /usr/lib64/python3.9/unittest/case.py:550>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:101>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestMarkedClass._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:119>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_requiring_my_fixture failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 1 > 1
 +  where 1 = len([0])
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:140>]
test_requiring_my_fixture passed 1 out of the required 1 times. Success!
test_something_flaky failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleRerunFilter._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:155>]
test_something_flaky passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert -1 >= 1
 +  where -1 = TestExample._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:27>]
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExample._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:27>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert -1 >= 1
 +  where -1 = TestExampleFlakyTests._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:43>]
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTests._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:43>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
=========================== short test summary info ============================
SKIPPED [1] test/test_pytest/test_pytest_example.py:66: This really fails! Remove skipif to see the test failure.
SKIPPED [1] ../../../../../usr/lib/python3.9/site-packages/_pytest/unittest.py:357: This test always fails
XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture
XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture
XFAIL test/test_pytest/test_pytest_example.py::TestExample::test_non_flaky_failing_thing
FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_handles_non_ascii_byte_string_in_exception
FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_identifies_failure
FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_write_then_read
FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_writelines_then_read
FAILED test/test_pytest/test_pytest_options_example.py::test_something_flaky
======== 5 failed, 68 passed, 2 skipped, 3 xfailed, 1 warning in 0.44s =========

@legoktm legoktm mentioned this issue Mar 11, 2024
Open
@mtelka
Copy link

mtelka commented Mar 11, 2024 •
edited
Loading

FYI, I tested flaky 3.8.0 on OpenIndiana and all tests passed here. Tested with pytest 8.1.1.

@kloczek
Copy link

kloczek commented Mar 11, 2024

One se c I'll try to retest thah ton 8.1.1 😋

@kloczek
Copy link

kloczek commented Mar 11, 2024

Nope it still fails

Here is pytest output: 
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0
plugins: flaky-3.7.0
collected 77 items

test/test_flaky_decorator.py ...                                                                                                                                                      [  3%]
test/test_flaky_plugin.py FF.                                                                                                                                                         [  7%]
test/test_multiprocess_string_io.py FF                                                                                                                                                [ 10%]
test/test_pytest/pytest_generate_example/test_pytest_generate_example.py ..                                                                                                           [ 12%]
test/test_pytest/test_flaky_pytest_plugin.py ................................................                                                                                         [ 75%]
test/test_pytest/test_pytest_example.py .xx.x...s......s                                                                                                                              [ 94%]
test/test_pytest/test_pytest_options_example.py F..                                                                                                                                   [ 98%]
test/test_pytest/test_pytester_plugin.py .                                                                                                                                            [100%]

========================================================================================= FAILURES ==========================================================================================
_______________________________________________________ TestFlakyPlugin.test_flaky_plugin_handles_non_ascii_byte_string_in_exception ________________________________________________________

self = <test.test_flaky_plugin.TestFlakyPlugin testMethod=test_flaky_plugin_handles_non_ascii_byte_string_in_exception>

    def test_flaky_plugin_handles_non_ascii_byte_string_in_exception(self):
        mock_method_name = 'my_method'
        mock_exception = 'ńőń ȁŝćȉȉ ŝƭȕƒƒ'.encode('utf-16')
        mock_message = 'information about retries'
        # pylint:disable=protected-access
>       self._flaky_plugin._log_test_failure(
            mock_method_name,
            (ValueError.__name__, mock_exception, ''),
            mock_message,
        )

test/test_flaky_plugin.py:38:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/_flaky_plugin.py:38: in _log_test_failure
    formatted_exception_info = ''.join(format_exception(*err)).replace('\n', '\n\t').rstrip()
/usr/lib64/python3.9/traceback.py:120: in format_exception
    return list(TracebackException(
/usr/lib/python3.9/site-packages/exceptiongroup/_formatting.py:96: in __init__
    self.stack = traceback.StackSummary.extract(
/usr/lib64/python3.9/traceback.py:347: in extract
    for f, lineno in frame_gen:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

tb = ''

    def walk_tb(tb):
        """Walk a traceback yielding the frame and line number for each frame.

        This will follow tb.tb_next (and thus is in the opposite order to
        walk_stack). Usually used with StackSummary.extract.
        """
        while tb is not None:
>           yield tb.tb_frame, tb.tb_lineno
E           AttributeError: 'str' object has no attribute 'tb_frame'

/usr/lib64/python3.9/traceback.py:312: AttributeError
___________________________________________________________________ TestFlakyPlugin.test_flaky_plugin_identifies_failure ____________________________________________________________________

self = <test.test_flaky_plugin.TestFlakyPlugin testMethod=test_flaky_plugin_identifies_failure>

    def test_flaky_plugin_identifies_failure(self):
>       for name, test in self._test_dataset:
E       ValueError: too many values to unpack (expected 2)

test/test_flaky_plugin.py:45: ValueError
_______________________________________________________________________ TestMultiprocessStringIO.test_write_then_read _______________________________________________________________________

self = <test.test_multiprocess_string_io.TestMultiprocessStringIO testMethod=test_write_then_read>

    def test_write_then_read(self):
        for name, value in self._test_values.items():
            with self.subTest(name):
                for string_io in self._string_ios:
                    for item in value[0]:
                        string_io.write(item)
>                   self.assertEqual(string_io.getvalue(), value[1])
E                   AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ'
E                   - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ
E                   ?      -----------
E                   + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ

test/test_multiprocess_string_io.py:31: AssertionError
____________________________________________________________________ TestMultiprocessStringIO.test_writelines_then_read _____________________________________________________________________

self = <test.test_multiprocess_string_io.TestMultiprocessStringIO testMethod=test_writelines_then_read>

    def test_writelines_then_read(self):
        for name, value in self._test_values.items():
            with self.subTest(name):
                for string_io in self._string_ios:
                    string_io.writelines(value[0])
>               self.assertEqual(string_io.getvalue(), value[1])
E               AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ'
E               - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ
E               ?      -----------
E               + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ

test/test_multiprocess_string_io.py:38: AssertionError
___________________________________________________________________________________ test_something_flaky ____________________________________________________________________________________

dummy_list = [0]

    def test_something_flaky(dummy_list=[]):
        # pylint:disable=dangerous-default-value
        dummy_list.append(0)
>       assert len(dummy_list) > 1
E       assert 1 > 1
E        +  where 1 = len([0])

test/test_pytest/test_pytest_options_example.py:11: AssertionError
========================================================================================= XFAILURES =========================================================================================
______________________________________________________________________ test_something_good_with_failing_setup_fixture _______________________________________________________________________

    @pytest.fixture(scope='function')
    def failing_setup_fixture():
>       assert False
E       assert False

test/test_pytest/test_pytest_example.py:21: AssertionError
______________________________________________________________________ test_something_good_with_failing_setup_fixture _______________________________________________________________________

    @pytest.fixture(scope='function')
    def failing_setup_fixture():
>       assert False
E       assert False

test/test_pytest/test_pytest_example.py:21: AssertionError
_________________________________________________________________________ TestExample.test_non_flaky_failing_thing __________________________________________________________________________

self = <test.test_pytest.test_pytest_example.TestExample object at 0x7fb0b6f35d30>

    @pytest.mark.xfail
    def test_non_flaky_failing_thing(self):
        """Flaky will also not interact with this test"""
>       assert self == 1
E       assert <test.test_pytest.test_pytest_example.TestExample object at 0x7fb0b6f35d30> == 1

test/test_pytest/test_pytest_example.py:40: AssertionError
===================================================================================== warnings summary ======================================================================================
test/test_flaky_plugin.py:0
  /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_flaky_plugin.py:0: PytestCollectionWarning: cannot collect test class 'TestCaseDataset' because it has a __new__ constructor (from: test/test_flaky_plugin.py)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===Flaky Test Report===

Flaky report texttest_something_flaky failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 1 > 1
 +  where 1 = len([0])
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:16>]
test_something_flaky passed 1 out of the required 1 times. Success!
test_something_good_with_failing_setup_fixture failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:85>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:843>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:878>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:82>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:158>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:515>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:512>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1780>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:687>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:541>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:570>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:647>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1079>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/setuponly.py:36>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1132>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:913>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:21>]
test_something_good_with_failing_setup_fixture failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:85>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:843>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:878>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:82>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:158>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:515>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:512>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1780>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:687>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:541>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:570>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:647>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1079>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/setuponly.py:36>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:1132>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/fixtures.py:913>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:21>]
test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExample._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:50>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times.
test_flaky_thing_that_fails_then_succeeds passed 2 out of the required 2 times. Success!
test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times.
test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 2 out of the required 2 times. Success!
test_flaky_thing_that_always_passes passed 1 out of the required 2 times. Running test again until it passes 2 times.
test_flaky_thing_that_always_passes passed 2 out of the required 2 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTests._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:87>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTestCase._threshold
        [<TracebackEntry /usr/lib64/python3.9/unittest/case.py:59>, <TracebackEntry /usr/lib64/python3.9/unittest/case.py:592>, <TracebackEntry /usr/lib64/python3.9/unittest/case.py:550>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:101>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestMarkedClass._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:119>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_requiring_my_fixture failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 1 > 1
 +  where 1 = len([0])
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:140>]
test_requiring_my_fixture passed 1 out of the required 1 times. Success!
test_something_flaky failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleRerunFilter._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_example.py:155>]
test_something_flaky passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert -1 >= 1
 +  where -1 = TestExample._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:27>]
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExample._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:27>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!
test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert -1 >= 1
 +  where -1 = TestExampleFlakyTests._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:43>]
test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTests._threshold
        [<TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:340>, <TracebackEntry /home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:849>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/logging.py:832>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/capture.py:883>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:121>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/skipping.py:256>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:182>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/runner.py:172>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:1777>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_hooks.py:501>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_manager.py:119>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:138>, <TracebackEntry /usr/lib/python3.9/site-packages/pluggy/_callers.py:102>, <TracebackEntry /usr/lib/python3.9/site-packages/_pytest/python.py:200>, <TracebackEntry /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_pytest/test_pytest_options_example.py:43>]
test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
================================================================================== short test summary info ==================================================================================
SKIPPED [1] test/test_pytest/test_pytest_example.py:66: This really fails! Remove skipif to see the test failure.
SKIPPED [1] ../../../../../usr/lib/python3.9/site-packages/_pytest/unittest.py:357: This test always fails
XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture
XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture
XFAIL test/test_pytest/test_pytest_example.py::TestExample::test_non_flaky_failing_thing
FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_handles_non_ascii_byte_string_in_exception - AttributeError: 'str' object has no attribute 'tb_frame'
FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_identifies_failure - ValueError: too many values to unpack (expected 2)
FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_write_then_read - AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ'
FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_writelines_then_read - AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ'
FAILED test/test_pytest/test_pytest_options_example.py::test_something_flaky - assert 1 > 1
=============================================================== 5 failed, 68 passed, 2 skipped, 3 xfailed, 1 warning in 0.47s ===============================================================```
</details>

<details>
<summary>List of installed modules in build env:</summary>

```console
Package            Version
------------------ -----------
build              1.1.1
distro             1.9.0
exceptiongroup     1.1.3
importlib_metadata 7.0.1
iniconfig          2.0.0
installer          0.7.0
packaging          24.0
pluggy             1.4.0
pyproject_hooks    1.0.0
pytest             8.1.1
python-dateutil    2.9.0.post0
setuptools         69.1.1
tokenize_rt        5.2.0
tomli              2.0.1
wheel              0.42.0
zipp               3.17.0

@kloczek
Copy link

kloczek commented Mar 11, 2024 •
edited
Loading

Just spotted odd detail in my output:

+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0           <<< so here is root directory with version 3.8.0 
plugins: flaky-3.7.0                                         <<< but here pytest reports version 3.7.0

@kloczek
Copy link

kloczek commented Mar 11, 2024

and indeed ..

[tkloczko@pers-jacek flaky-3.8.0]$ grep -r 3.7.0
HISTORY.rst:3.7.0 (2020-07-07)
setup.py:        version='3.7.0',

@kloczek
Copy link

kloczek commented Mar 11, 2024

https://github.com/box/flaky/blob/v3.8.0/setup.py#L52-L54

@mtelka
Copy link

mtelka commented Mar 11, 2024

and indeed ..

[tkloczko@pers-jacek flaky-3.8.0]$ grep -r 3.7.0
HISTORY.rst:3.7.0 (2020-07-07)
setup.py:        version='3.7.0',

This is #202.

@mtelka
Copy link

mtelka commented Mar 11, 2024

I believe the problem reported by OP is already fixed in flaky 3.8.0 via #201.

@mtelka mtelka mentioned this issue Mar 12, 2024
Merged
This was referenced Mar 20, 2024
Closed
Merged
BradyJ27 pushed a commit to BradyJ27/dvc that referenced this issue Apr 22, 2024
@skshetry @BradyJ27
limit pytest version to <8.1 (iterative#10334)
21c8d71 
Change in pytest==8.1.0 breaks `flaky` plugin.

See box/flaky#198.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix incompatibility with pytest == 8.1.0 jonathan-eq/flaky
5 participants
@The-Compiler @mtelka @skshetry @kloczek @qci-amos