I cannot use selective tests run

Before jumping to any conclusions, that there is a bug in Allure TestOps, pipeline or whatever tool you are using, please read this article on how to check if you actually can use the selective tests run feature.

TABLE OF CONTENTS

• Tests selective run (tests filtering).
• How it works
• How to check the selective test run on your local machine.
  • Create testplan.json
  • Declare ALLURE_TESTPLAN_PATH
• What do you need to do if local test of selective test run failed?

Tests selective run (tests filtering).

The Selective test run feature allows you triggering your build from Allure TestOps UI by selecting some test cases (not all test cases in your project).

The expected result is, only the tests selected via Allure TestOps UI are executed.

How it works

This is the brief description without getting into the nuts and bolts, just to understand the basic mechanics and be able to check the mechanics.

1. Allure TestOps triggers your pipeline. 
   1. This is done via API.
   2. One of the parameters in the API request clearly states, the execution is triggered from Allure TestOps.
2. In the pipeline there is either a plug-in or allurectl. Let’s call them agent to address together.
3. Agent checks the parameters of the pipeline and understands that the pipeline is triggered from Allure TestOps
4. Agent checks if there is a test plan to be generated.
   1. Environment variable ALLURE_TESTPLAN_PATH must be defined.
   2. Environment variable ALLURE_TESTPLAN_PATH must contain the path for testplan.json file.
5. Agent generates the testplan.json using path defined in ALLURE_TESTPLAN_PATH environment variable.
6. The Allure Report adaptor added to you tests checks if there is environment variable ALLURE_TESTPLAN_PATH
7. The Allure Report adaptor added to you tests checks if there is testplan.json file.
8. If the file exists, then adaptor reads it and instructs the test framework to execute the tests described in testplan.json file.

How to check the selective test run on your local machine.

This needs to be done before you try to use selective tests run in your pipelines.

Create testplan.json

File has the following format:

{ "version": "1.0", "tests": [{ "id": 11111, "selector": "my.company.SimpleTest.simpleTestOne" }] }

- "id": 11111 is the AllureID of the test

- "selector": "my.company.SimpleTest.simpleTestOne" is the full path of the test.

First, the adaptor looks for the ID, if the ID is not found in the code, then adaptor tries (sic!) using the full path. Some of the test frameworks cannot work with the full path, so only selection via ID is possible.

As an option, you can generate an example of testplan.json in your pipeline using allurectl as follows:

ALLURE_TESTPLAN_PATH: "./testplan.json"
./allurectl job-run plan --output-file ${ALLURE_TESTPLAN_PATH}
cat ${ALLURE_TESTPLAN_PATH}

Then copy the result of cat to local testplan.json.

The file must be in the project’s root folder.

Declare ALLURE_TESTPLAN_PATH

Go to your tests project folder.

In the terminal execute the command:

export ALLURE_TESTPLAN_PATH: "./testplan.json"

Check if the variable is in fact created:

echo ${ALLURE_TESTPLAN_PATH}

Of course, the creation of the variables and the commands on Windows OS family will look differently and we hope you can manage that with or without help of the internet.

Execute tests

Now, execute your tests.

If only the tests from testplan.json has been executed, then your test framework can do this, otherwise either 1) the test framework hasn’t this capability, or 2) your project is not configured properly, or 3) the adaptor for your framework hasn’t this capability.

What do you need to do if local test of selective test run failed?

1. Check the documentation of Allure Report adaptor for your test framework: https://github.com/orgs/allure-framework/repositories
   1. Repository usually has README.md file with the features added to a particular adaptor.
   2. If the feature hasn’t been added, then you can create an issue with the feature request content in appropriate repository of Allure Framework. It’ll be reviewed, researched and the answer will be provided in the scope of the created issue.
2. Ask the community, if the test framework allows usage of the selective test run: https://github.com/orgs/allure-framework/discussions