Back to blog
ReportPortal has several features to support effective test failure analysis. This article provides actionable strategies for setting up Pattern Analysis rules to maximize the potential of this powerful analytical method and drive better business outcomes.
What is Pattern Analysis?
Pattern Analysis is a feature used in software testing to help identify and analyze recurring patterns in error logs. It is commonly used in automation testing reporting tools, such as ReportPortal, to streamline the process of debugging and resolving test failures.
The following sections will guide you through the best practices and practical examples that demonstrate how Pattern Analysis can be implemented in various contexts.
1. Create rules for standard errors
If you don’t know where to start with Pattern Analysis, create rules for common errors:
400 Bad Request – this could indicate an issue with the tests.
401 Unauthorized – something may not be properly specified in the tests.
403 Forbidden – this could be a product issue, or certain details like passwords might not be provided in the tests.
500 Internal Server Error – this is likely not an issue with the automated tests; it suggests that your servers or some external services are unavailable.
Create rules to search for these errors, run Pattern Analysis on a launch, and it will be easier to sort test failures by defect types.
For example, consider the following: error codes in the 400-409 range are often caused by a Product Bug or an Automation Bug. A 500 Internal Server Error most often corresponds to a Product Bug or a System Issue. However, it's essential to review the error log to assign the correct defect type.
2. Study the documentation for your project
Ask developers about errors that can be observed in backend responses, which ones are critical, and which are less serious. Based on this information, create rules for Pattern Analysis to address these errors.
3. Create rules using regular expressions
In ReportPortal, you can create a Pattern Analysis rule with the help of regular expressions. This allows you to create fewer rules. For instance, if you know that various numbers appear in the tests, you could create several string rules for each number, or you could create a single RegEx rule to cover all numbers. Pattern analysis example:
Alternatively, if you know that some error logs end with the phrase "not found," you can create a rule to identify such tests.
4. Create Most popular pattern table and disable unnecessary rules
If you have many rules, you can create “Most popular pattern table” widget to view the Top-20 patterns that frequently appear in test automation results. The widget sorts patterns by how often they occur, allowing teams to easily spot common issues and plan how to fix them effectively.
You can set some of Pattern Analysis rules as inactive if they are irrelevant for specific launches. You can also enable or disable Auto Pattern Analysis.
Don’t worry if you forget to enable Auto Pattern Analysis – you can always run Pattern Analysis manually.
If we have Auto-Analysis enabled together with Pattern Analysis, Pattern Analysis starts before Auto-Analysis: in this case, ReportPortal first identifies known patterns, and then the Analyzer assigns defect types and links issues in the BTS, if available. As a result, when both Auto-Analysis and Auto Pattern Analysis occur, a test item will have two labels: AA and PA.
By setting up and customizing rules that match specific error patterns, teams can automate the identification and categorization of failures, leading to faster problem resolution and better overall software quality.