The TestRail CLI (trcli) is a command line tool for interacting with TestRail. It integrates directly with the TestRail API and provides abstractions to easily create test cases and upload automated test results.
The TestRail CLI currently supports:
- Uploading automated test results from JUnit reports
- Uploading automated test results from Robot Framework reports
- Auto-generating test cases from OpenAPI specifications
- Creating new test runs for results to be uploaded to.
To see further documentation about the TestRail CLI, please refer to the TestRail CLI documentation pages on the TestRail help center.
If you already have Python and pip installed, you can simply run the command below in your terminal. We recommend using Python 3.10 or newer.
pip install trcli
To verify the installation was successful, you can run the trcli command.
trcliYou should get something like this:
TestRail CLI v1.9.0
Copyright 2024 Gurock Software GmbH - www.gurock.com
Supported and loaded modules:
- parse_junit: JUnit XML Files (& Similar)
- parse_robot: Robot Framework XML Files
- parse_openapi: OpenAPI YML Files
- add_run: Create a new empty test run
- close_run: Close a test run
$ trcli --help
TestRail CLI v1.9.0
Copyright 2024 Gurock Software GmbH - www.gurock.com
Usage: trcli [OPTIONS] COMMAND [ARGS]...
TestRail CLI
Options:
-c, --config Optional path definition for testrail-credentials file or
CF file.
-h, --host Hostname of instance.
--project Name of project the Test Run should be created under.
--project-id Project id. Will be only used in case project name will
be duplicated in TestRail [x>=1]
-u, --username Username.
-p, --password Password.
-k, --key API key.
-v, --verbose Output all API calls and their results.
--verify Verify the data was added correctly.
--insecure Allow insecure requests.
-b, --batch-size Configurable batch size. [default: (50); x>=2]
-t, --timeout Batch timeout duration. [default: (30); x>=0]
-y, --yes answer 'yes' to all prompts around auto-creation
-n, --no answer 'no' to all prompts around auto-creation
-s, --silent Silence stdout
--help Show this message and exit.
Commands:
parse_junit Parse JUnit reports and upload results to TestRail
parse_openapi Parse OpenAPI spec and create cases in TestRail
parse_robot Parse Robot Framework report and upload results to TestRail
add_run Create a new test run (useful for CI/CD flows prior to parsing results)
close_run Close an open test run with the provided run IDThe parse_junit command allows you to upload automated test results, provided that you are using
a framework that supports generating JUnit XML report files, such as Cypress, Playwright, JUnit5, TestNG, and Pytest.
In case you are using Robot Framework, you can use the parse_robot command using the same parameters.
In the next sections you will find information on how to use the TestRail CLI for code-first and specification-first approaches to test automation.
trcli parse_junit --helpUsage: trcli parse_junit [OPTIONS]
Parse report files and upload results to TestRail
Options:
-f, --file Filenames with paths. Multiple files can be separated by comma. Globbing with * is allowed.
--close-run Close the newly created run
--title Title of Test Run to be created in TestRail.
--case-matcher Mechanism to match cases between the report and
TestRail.
--suite-id Suite ID to submit results to. [x>=1]
--suite-name Suite name to submit results to.
--run-id Run ID for the results they are reporting (otherwise the
tool will attempt to create a new run). [x>=1]
--plan-id Plan ID with which the Test Run will be associated.
[x>=1]
--config-ids Comma-separated configuration IDs to use along with Test
Plans (i.e.: 34,52).
--milestone-id Milestone ID to which the Test Run should be associated
to. [x>=1]
--section-id Section ID to create new sections with test cases under
(optional). [x>=1]
--run-description Summary text to be added to the test run.
--case-fields List of case fields and values for new test cases
creation. Usage: --case-fields type_id:1 --case-fields
priority_id:3
--result-fields List of result fields and values for test results
creation. Usage: --result-fields custom_field_a:value1
--result-fields custom_field_b:3
--allow-ms Allows using milliseconds for elapsed times.
--special-parser Optional special parser option for specialized JUnit
reports.
--result-status-mapping Set status mapping like '<tag_name>:<status_id>'.
Statuses in your xml must be parsable by the junit
parser. Usage: --result-status-mapping skipped:6
--result-status-mapping custom1:7
--help Show this message and exit.<testsuites name="test suites root">
<testsuite failures="0" errors="0" skipped="1" tests="1" time="3049" name="tests.LoginTests">
<properties>
<property name="setting1" value="True"/>
<property name="setting2" value="value2"/>
</properties>
<testcase classname="tests.LoginTests" name="test_case_1" time="159">
<skipped type="pytest.skip" message="Please skip">skipped by user</skipped>
</testcase>
<testcase classname="tests.LoginTests" name="test_case_2" time="650">
</testcase>
<testcase classname="tests.LoginTests" name="test_case_3" time="121">
<failure type="pytest.failure" message="Fail due to...">failed due to...</failure>
<properties>
<property name="testrail_attachment" value="path_to/screenshot.jpg"/>
</properties>
</testcase>
</testsuite>
</testsuites>Mapping JUnit elements to TestRail entities:
| XML junit file tag | TestRail entity |
|---|---|
<testsuites> |
suite |
<testsuite> |
section |
<testcase> |
case |
For further detail, please refer to the JUnit to TestRail mapping documentation.
To submit test case results, the TestRail CLI will attempt to match the test cases in your automation suite to test cases in TestRail. There are 2 mechanisms to match test cases:
- Using Automation ID
- Using Case ID (in test case
nameorproperty)
The first mechanism allows to automatically match test cases, meaning you can take a code-first approach, while the second one is suited for a specification-first approach, where you write your test cases in TestRail and add the case ID to your automated tests.
Notes:
- The TestRail CLI has a prompt mechanism that allows you to choose whether you want test cases to be automatically created:
- If you enter
yes(or use the-yoption), the TestRail CLI will automatically create any test case it can't match in TestRail- If you enter
no(or use the-noption), the TestRail CLI will not create any new test cases
- If you are using a multi-suite project in TestRail, you should provide the ID of the test suite you want the cases to be created in using the
--suite-idcommand line option, otherwise the CLI tool will attempt to find the suite on TestRail or create it.
To use this mechanism, you must first add a new custom field
of type Text with system name automation_id.
The TestRail CLI will use the unique combination of your automation test case’s classname and name
(expressed as classname.name) to compare against values of the automation_id field in your TestRail test case repository.
If a match is found, this test case will be included in the auto-generated test run for this upload.
Example:
| Test Result from Automation Results File | Automation ID in TestRail |
|---|---|
<testcase classname="tests.LoginTests" name="test_case_1" time="159"></testcase> |
tests.LoginTests.test_case_1 |
Important usage notes:
- If you would like to upload automation results for test cases that already exist in TestRail, be sure to update the
automation_idfor those test cases before uploading your automation results- If you change the test name in your automation suite later, that will create a new test case in TestRail, unless you also update the
automation_idfield for the test case in TestRail- If you are using the CLI tool in a CI context, we recommend using the
-yoption to automatically accept test case creation prompts
For more detail, please refer to the Automation workflows - Code-first documentation.
You can use the Case ID mechanism if you want to manually match your automated test cases to case IDs in TestRail. From an implementation perspective, you can do this in one of two ways:
- Map by setting the case ID in the test name, using the case-matcher
name:
<testsuites name="test suites root">
<testsuite failures="0" errors="0" skipped="1" tests="1" time="3049" name="tests.LoginTests">
<testcase classname="tests.LoginTests" name="[C123] test_case_1" time="650" />
</testsuite>
</testsuites>- Map by setting the case ID in a test case property, using case-matcher
property:
<testsuites name="test suites root">
<testsuite failures="0" errors="0" skipped="1" tests="1" time="3049" name="tests.LoginTests">
<testcase classname="tests.LoginTests" name="test_case_1" time="650">
<properties>
<property name="test_id" value="C123"/>
</properties>
</testcase>
</testsuite>
</testsuites>Important usage notes:
- We recommend using the
-noption to skip creating new test cases due to the potential risk of duplication
For more details, please refer to the Automation workflows - Specification-first documentation.
Please refer to the Usage examples documentation page to see how you can leverage all the functionalities provided by the TestRail CLI.
If you are using saucectl from SauceLabs to execute your automation projects, the TestRail CLI has an enhanced parser
that fetches session information and adds it to your test runs. You can enable this functionality by using
the --special-parser saucectl command line option.
Please refer to the SauceLabs and saucectl reports documentation for further information.
When a test run MUST created before using one of the parse commands, use the add_run command. For example, if
tests are run across parallel, independent test nodes, all nodes should report their results into the same test run.
First, use the add_run command to create a new run; then, pass the run title and id to each of the test nodes, which
will be used to upload all results into the same test run.
$ trcli add_run --help
TestRail CLI v1.9.5
Copyright 2024 Gurock Software GmbH - www.gurock.com
Usage: trcli add_run [OPTIONS]
Options:
--title Title of Test Run to be created or updated in
TestRail.
--suite-id Suite ID to submit results to. [x>=1]
--run-description Summary text to be added to the test run.
--milestone-id Milestone ID to which the Test Run should be
associated to. [x>=1]
--run-assigned-to-id The ID of the user the test run should be assigned
to. [x>=1]
--include-all
--case-ids Comma separated list of test case IDs to include in
the test run.
--run-refs A comma-separated list of references/requirements
-f, --file Write run title and id to file.
--help Show this message and exit.If the file parameter is used, the run title and id are written to the file in yaml format. Example:
title: Run Title
run_id: 1
This file can be used as the config file (or appended to an existing config file) in a later run.
When a test run must be closed after test results have been uploaded, use the close_run command. For example, tests
may be uploaded into an open test run using one or more CI/CD processes and nodes. After all processes have completed,
use the close_run command to close the test run.
$ trcli close_run --help
TestRail CLI v1.9.5
Copyright 2024 Gurock Software GmbH - www.gurock.com
Usage: trcli close_run [OPTIONS]
Options:
--suite-id Suite ID of the test run. [x>=1]
--run-id Run ID for the run to be closed. [x>=1]
--help Show this message and exit.The parse_openapi command allows you to automatically generate and upload test cases to TestRail based on an
OpenAPI specification. This feature is intended to be used once to quickly bootstrap your test case design,
providing you with a solid base of test cases, which you can further expand on TestRail.
$ trcli parse_openapi --help
TestRail CLI v1.9.0
Copyright 2024 Gurock Software GmbH - www.gurock.com
Usage: trcli parse_openapi [OPTIONS]
Parse OpenAPI spec and create cases in TestRail
Options:
-f, --file Filename and path.
--suite-id Suite ID to create the tests in (if project is multi-suite).
[x>=1]
--case-fields List of case fields and values for new test cases creation.
Usage: --case-fields type_id:1 --case-fields priority_id:3
--help Show this message and exit.openapi: 3.0.0
info:
description: This is a sample API.
version: 1.0.0
title: My API
paths:
/pet:
post:
summary: Add a new pet to the store
description: Add new pet to the store inventory.
operationId: addPet
responses:
'200':
description: Pet created
content:
application/json:
schema:
$ref: '#/components/schemas/Pet'
'400':
description: Invalid request
requestBody:
$ref: '#/components/schemas/Pet'
'/pet/{petId}':
get:
summary: Find pet by ID
description: Returns a single pet
operationId: getPetById
parameters:
- name: petId
in: path
description: ID of pet to return
required: true
schema:
type: integer
format: int64
responses:
'200':
description: Successful operation
content:
application/json:
schema:
$ref: '#/components/schemas/Pet'
'400':
description: Invalid request
'404':
description: Pet not found
components:
schemas:
Pet:
type: object
required:
- name
properties:
id:
type: integer
format: int64
readOnly: true
name:
description: The name given to a pet
type: string
example: GuruThe test cases are generated based on the OpenAPI specification paths, operation verbs and possible response status codes, which provides a good basic test coverage for an API, although we recommend you further expand your test cases to cover specific business logic and workflows.
| Pattern | Test case title example |
|---|---|
VERB /path -> status_code (summary) |
GET /pet/{petId} -> 200 (Successful operation) |
You can choose to set parameters from different sources, like a default config file, environment variables, custom config file, cli parameters or in some cases use default values. The priority of setting parameters from different sources is as per the table below, where 1 is the highest priority.
| priority | source |
|---|---|
| 1 | cli parameters |
| 2 | custom config file |
| 3 | environment variables |
| 4 | default config file |
| 5 | default value |
For more details, please refer to the Parameter sources documentation.
trcli tool will return 0 to the console in case of success and value greater than 1 (usually 1 or 2) in other cases.
Messages that are being printed on the console are being redirected to sys.stdout or sys.stderr.
trcli allows users to upload test cases and results using multithreading. This is enabled by default and set to MAX_WORKERS_ADD_CASE = 5 and
MAX_WORKERS_ADD_RESULTS = 10 in trcli/settings.py. To disable multithreading, set those to 1.
During performance tests we discovered that using more than 10 workers didn't improve time of upload and could cause errors. Please set it accordingly to your machine specs. Average time for uploading:
- 2000 test cases was around 460 seconds
- 5000 test cases was around 1000 seconds
Interested in contributing and helping improve the TestRail CLI client? Please start by looking into CONTRIBUTING.md and creating an issue.
The TestRail CLI client is licensed under the Mozilla Public License 2.0.