Skip to main content

Unique Error Analysis

You can analyze test failures from two perspectives: quantitative (How many tests have failed?) and qualitative (Why did they fail?). For example, if 1,000 test cases fail, then:

  1. They could all fail for the same reason.

  2. They could fail for various reasons.

While failed tests allow you to understand what is broken, Unique Error analysis functionality implemented in version 5.7 will show you why it broke. Moreover, the system automatically groups tests by the same errors: when you expand some error, you see a list of tests where it occurred.

important

The main advantage of this solution is that a list of all unique errors of the launch is presented in one place.

Thanks to the Unique Error analysis functionality, ReportPortal got the following benefits:

  1. A list of unique errors for the launch, with grouping.
  2. Easier test results analysis.
  3. Machine learning (ML) suggestions for groups of tests.
  4. The ‘New Errors’ Quality Gates rule.

How to run Unique Error analysis​

You can run Unique Error analysis automatically or manually.

To run Unique Error analysis automatically:

  1. Go to the Project Settings.
  2. Open Analyzer tab.
  3. Open ‘Unique Errors’ settings.
  4. Check the "Auto-Unique Error" checkbox.
  1. Run a launch.
  2. After launch finish, open any item level in the launch.
  3. Click ‘Unique errors’ tab.
note

Automatic Unique Error analysis is activated by default.

In case automatic Unique Error analysis is turned off, there are two ways to run Unique Error analysis manually:

  1. Run Unique error analysis manually from any item level.
  1. Run Unique Error analysis from the menu next to a particular launch.

After Unique Error analysis is finished, open any item level in the launch and click the ‘Unique errors’ tab.

How to see Unique Errors​

Finally, you can see the list of Unique Errors.

There you can see a list of unique logs with number of failed tests related to them. The number of matches found per Unique Errors is displayed in the column ‘MATCHED TESTS’. By default, Unique Errors are displayed from the one with the highest number of failed tests. By clicking on the column name, the sorting is updated each time.

You can expand unique error to check what tests belong to the same one and it could give you a hint during error analysis and defects assigning. The groups are formed based on unique error logs, some small error logs can be merged and displayed as one error.

How to get ML suggestions for the group of tests​

In addition, there is a possibility to get ML suggestions for the group of tests. It speeds up the process of analyzing failed tests and assigning defect types by several times.

For that,

  1. Open any item level in the launch.
  2. Click ‘Unique Errors’ tab.
  3. Expand a cluster with unique error.
  4. Select some failed tests.
  1. Open ‘Make decision’ modal.
  2. Check Analyzer suggestions.

'Include/exclude numbers' settings​

There are 2 settings to decide whether numbers in error logs have significant value for analysis or not:

  1. Include numbers to analyzed logs.

  2. Exclude numbers from analyzed logs.

Let’s consider some examples with the same Unique Errors but with different include/exclude numbers settings.

We have 2 errors with identical text, but the difference is numeric value in the first error.

  1. An example with 'Include numbers to analyzed logs' setting. Error with numeric value is displayed:
  1. An example with 'Exclude numbers from analyzed logs' setting. As you can see, error with numeric value is not displayed:

How to extend Unique Error analysis​

Unique Error analysis functionality can be extended by the ReportPortal premium feature – Quality Gates.

If you have Quality Gate plugin installed, you can create Quality Gate for the launch with the 'New Errors' rule. This rule helps identify any new unique errors in the current launch by comparing them to another specified launch.

If there is Quality Gate for the launch with the 'New Errors' rule, the 'ALL UNIQUE ERRORS' dropdown is visible and enabled. The available options are: All, New Errors and Known Errors.

If there is no Quality Gate for the launch with the 'New Errors' rule, the 'ALL UNIQUE ERRORS' dropdown is visible but disabled. A tooltip appears on hover for the disabled state: 'No Quality Gate with 'New Error' rule for the launch.'

If Quality Gate plugin is not installed or disabled, the 'ALL UNIQUE ERRORS' dropdown is invisible.