What are test execution statuses, can we add new ones?

Understanding Test Statuses in Software Testing

Test statuses provide a straightforward way of indicating the outcome of each test case executed during the testing phase. Here is a detailed look at the possible test statuses, which are considered industry standards, and their implications for software development and quality assurance teams.

PASSED

The ‘PASSED’ status indicates that the test has been successfully executed and that the application logic performed exactly as expected according to the test logic. This status is the goal for any test case, as it confirms that part of the system under test meets the specified requirements and behaves as intended under the tested scenarios.

FAILED

The ‘FAILED’ status is assigned when the test has been executed successfully in terms of its operational mechanics, but the application logic does not behave as expected based on the test logic. This outcome signals a discrepancy between the expected and actual behaviour, which typically prompts a bug or issue report and necessitates further investigation and potential adjustments in the code or design.

BROKEN

The ‘BROKEN’ status is used when a test could not be executed as planned. This failure to execute could be due to issues within the test code itself, problems in the test environment, or other external factors that prevent the test logic from running. Unlike a ‘FAILED’ test, which indicates a problem with the application logic, a ‘BROKEN’ test suggests complications in the testing setup or infrastructure. Identifying and resolving these issues is essential to ensure reliable test results.

SKIPPED

Tests marked as ‘SKIPPED’ are those intentionally not executed. This decision can be due to various reasons, such as the test being deemed irrelevant for the current testing cycle, conditions for running the test not being met, or resource constraints. Skipped tests are typically recorded for tracking purposes and to maintain transparency in the testing process.

UNKNOWN

The ‘UNKNOWN’ status is used when the system lacks information on the test execution. This situation usually occurs when automated tests are expected to run, but results have not been received. This can be due to issues in the test dispatch system, communication failures, or errors in result collection. It is crucial to investigate such occurrences to ensure test coverage and system integrity.

Extended statuses

For the time being, there is no way to introduce new statuses in Allure TestOps.