Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve testing infrastructure #102

Open
8 of 11 tasks
masklinn opened this issue Apr 26, 2022 · 0 comments
Open
8 of 11 tasks

Improve testing infrastructure #102

masklinn opened this issue Apr 26, 2022 · 0 comments

Comments

@masklinn
Copy link
Contributor

masklinn commented Apr 26, 2022 •
edited
Loading

  • improve test matrix and configuration
    • add 3.9 and 3.10
    • add pypy
    • maybe add 3.11 alpha? Might be useful as a smoke test at least
    • reorganise in order to have a single CI for all stable implementations to avoid configuration churn (wrt branch protection), in the current state any addition or removal of a python target has to be impacted there
  • investigate ymyzk/tox-gh-actions (or possibly tox's homegrown but that's only for the yet-to-be-released tox 4), a gh action has been defined for CI but it duplicates the tox file (which I haven't updated and may be broken in part or whole), this duplication seems like a shame
    cancelling this, probably less useful now that the wheel / release actions are wedded so hard to GHA
  • test both regex and yaml implementations, otherwise yaml bitrots (cf Open UA_PARSER_YAML as binary #99)
  • add benches? needs a benchmarking dataset (cf ParseUserAgent, ParseOS and ParseDevice are not using the cache #97), also not sure it's acceptable to run on GHA, it's technically possible (free plans allow 20 concurrent jobs and up to 6h per job) but it might quickly stray into abuse, look further into this
  • fuzzing? not sure there's really enough actual code execution for this to be useful, though it might be able to uncover redos issues in the base set('s interaction with Python's standard regex engine), the two tools I could find for coverage-guided fuzzing are pythonfuzz (by gitlab) and atheris (by google)
  • look at coverage, this may be useless for the same reasons as above
  • enable merge queues
    it's not super urgent, but from time to time I have two PRs I want to merge (generally because I extracted one from the other) and in that case github's auto-merge is just stupid: with "require branches to be up to date before merging" once the first PR has passed its checks and been merged the second will just wait forever with no notification sent to anyone
@masklinn masklinn changed the title Run tests with UA_PARSER_YAML Run tests in yaml mode as well Apr 26, 2022
@masklinn masklinn changed the title Run tests in yaml mode as well Improve testing infrastructure Apr 30, 2022
masklinn added a commit that referenced this issue May 1, 2022
@masklinn
Add a bunch of new python versions (and pypy) to the CI
f637a86 
Also to the local tox, and try to rewrite tox.ini to make it more
reliable (wip):

- don't sdist into the repository, just develop into it matching the
  CI (and in a way the old expectations, given the old `setup.py
  develop` stanza)
- don't install the package at all into the checker environments, it's
  entirely unnecessay
- add doctesting of the readme to tox, in order to match the CI
  action, opt 2.7 out for the same reasons: changes in pprint's
  formatting between 2.7 and 3.5 make them incompatible, ignore 2.7
- requirements.txt was removed previously, use the dev requirements

Related to: #102
Closes: #111
@masklinn masklinn mentioned this issue May 2, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@masklinn