I cannot use selective tests run

Before jumping to any conclusions, that there is a bug in Allure TestOps, pipeline or whatever tool you are using, please read this article on how to check if you actually can use the selective tests run feature.

Tests selective run (tests filtering)

The Selective test run feature allows you triggering your build from Allure TestOps UI by selecting some test cases (not 100% of tests cases in your project).

The expected result is only the tests selected via Allure TestOps UI are executed.

How it works

This is the brief description without getting into the nuts and bolts, just to understand the basic mechanics and be able to check the mechanics.

1. Allure TestOps triggers your pipeline. 
   1. This is done via API.
   2. One of the parameters in the API request clearly states, the execution is triggered from Allure TestOps.
2. In the pipeline there is either a plug-in or allurectl. Let’s call them agent to address together.
3. Agent checks the parameters of the pipeline and understands that the pipeline is triggered from Allure TestOps
4. Agent checks if there is a test plan to be generated.
   1. Environment variable ALLURE_TESTPLAN_PATH must be defined (done automatically by the agent).
   2. Environment variable ALLURE_TESTPLAN_PATH must contain the path for testplan.json file.
5. Agent generates the testplan.json using path defined in ALLURE_TESTPLAN_PATH environment variable.
6. The Allure Report adaptor added to you tests checks if there is environment variable ALLURE_TESTPLAN_PATH
7. The Allure Report adaptor added to you tests checks if there is testplan.json file.
8. If the file exists, then adaptor reads it and instructs the test framework to execute the tests described in testplan.json file.

How to check the selective test run on your local machine.

This needs to be done before you try to use selective tests run in your pipelines.

Create testplan.json

File has the following format:

{ "version": "1.0", "tests": [{ "id": 11111, "selector": "my.company.SimpleTest.simpleTestOne" }] }

- "id": 11111 is the AllureID of the test

- "selector": "my.company.SimpleTest.simpleTestOne" is the full path of the test.

First, the adaptor looks for the ID, if the ID is not found in the code, then adaptor tries (sic!) using the full path. Some of the test frameworks cannot work with the full path, so only selection via ID is possible.

You can generate testplan.json using allurectl as follows:

export ALLURE_TOKEN=<TOKEN>
export ALLURE_ENDPOINT=<URL>
export ALLURE_PROJECT_ID=100
export TESTOPS_TESTPLAN_ID=1000
export TESTPLAN_JSON_PATH="./testplan.json"

./allurectl test-case plan --output-file ${TESTPLAN_JSON_PATH} -q "testPlan=${TESTOPS_TESTPLAN_ID}"

The file must be in the project’s root folder.

Declare ALLURE_TESTPLAN_PATH

Go to your tests project folder.

In the terminal execute the command:

export ALLURE_TESTPLAN_PATH: "./testplan.json"

Check if the variable is in fact created:

echo ${ALLURE_TESTPLAN_PATH}

The creation of the variables and the commands on Windows OS family will look differently.

Execute tests

Now execute your tests. Tests must be executed in the same context, where you've defined the path to testplan.json file.

If only the tests from testplan.json have been executed, then your integration works, otherwise either 1) the test framework hasn’t this capability, or 2) your project (code) is not configured properly, or 3) the adaptor for your framework hasn’t this capability.

What do you need to do if local test of selective test run failed?

1. Check the documentation of Allure Report adaptor for your test framework: https://github.com/orgs/allure-framework/repositories
   1. Repository usually has README.md file with the features added to a particular adaptor.
   2. If the feature hasn’t been added, then you can create an issue with the feature request content in appropriate repository of Allure Framework. It’ll be reviewed, researched and the answer will be provided in the scope of the created issue.
2. Ask the community, if the test framework allows usage of the selective test run: https://github.com/orgs/allure-framework/discussions

Setting temporary (session) variables

allurectl