Problem

When you trigger test cases on a CI system from Allure Testops or you request a rerun of tests for a created launch, tests are hanging in In progress status and new results are added to a launch.

What causes the duplication

There could be two reasons for test result duplication.

• test parameters are different from the ones we expect
• the environment on which the tests are run is different from the one we expect

These two attributes of a test result (parameters, environments) are used in the calculation of test result's atribute called History ID, which is used by Allure Testops to match expected test result with the one we receive from CI.

History ID

The HistoryID in result file is used when gluing the expected vs actual results in TestOps launch. At this point, you need to understand at what point your work with it breaks.

What we need to know about History ID

historyId aka historyKey = md5(hash, sort(values(parameters)), sort(environment)))

To match re-runs, Allure Testops calculates the hash of the arguments passed to the method + the hash of the environment values. How this hash is calculated is not important, the main thing is that the algorithm is stable and for the same set of attributes provides the same hash.

If parameter's values or environment's values change from run to run (dynamic values like hashes, date/time values), this is treated as a new result, not as a re-run of the initial result.

Consequences

• if during a re-run the restarted test has N as a parameter's value instead of X, this will be treated as a new result, not a re-run, and the result we are waiting for will hang in In progressand a new one will be created, which will not be attached to it due to the difference of their HistoryIDs. 
  • There is also a possibility that the test simply did not start (rare, but it happens).

Same with environments

• we always trigger a re-run on the same set of parameters that the initial pipeline had. If some environment parameter was changed on rerun (say, it's a dynamic one), the result will be treated as if it's a new one, not a retry and not the expected result.

Examples

Example 1: Initial run or re-run

• There are child pipelines in GL
• Allure TestOps triggers the main Pipeline and passes the environment variables
• the main Pipeline initiates the start of the child Pipelines and does NOT pass all values for the environment to them.
• the requested result expects environment variable OS=iOS, VER=123
• but OS=iOS result arrives and that's all.
• the test hangs In progress (OS=iOS, VER=123) with HistoryID1
• so in launch, one more result (OS=iOS) with HistoryID2 appears.

Example 2: Dynamic parameters

• In initial run a parameterized test is executed with parameter AABBCCDDEEFF HistoryID1 is calculated.
• we request its re-run
• a new result with the same parameter  but value BBAACCDCDDEEFF arrives HistoryID2 is calculated.
• the expected (with HistoryID1) test hangs ‘In progress'
• new result (with HistoryID2) is added to the launch.

Solutions

Changing environments

Such environments need to be removed from the project configuration.

if you still need to see this information about your tests, it's better to add this as an attachment (not as a parameter) to a test result.

Dynamic parameters

Dynamic parameters can be moved to test attachments, so they won't affect HistoryID calculations. 

Some integrations with the test frameworks allow to mark a parameter as "excluded" and this parameter won't affect the calculation of the HistoryIID

Examples of such test frameworks integrations:

• allure-junit5: https://github.com/allure-framework/allure-java/blob/master/allure-junit5/src/main/java/io/qameta/allure/junit5/AllureJunit5.java#L88
• allure-playwright: https://github.com/allure-framework/allure-js/blob/master/packages/allure-playwright/README.md#parameters-usage
• Check your framework here: https://github.com/orgs/allure-framework/repositories
  • check if it supports excluded parameters
  • request the functionality in appropriate repository it it's absent