Fintech

Improving test automation stability

Back to Case StudiesDownload PDF version

Challenges

  • Complex, manual test runs

  • Low stability of regression for unclear reasons (60% passing rate)

  • Unclear reporting for non-technical stakeholders, leading to the lack of transparency in test automation results and progress

  • Test automation feedback is unclear, unreliable, incomprehensible, and insufficient to decide to push the app to production

Test automation stability before and after using our test automation dashboard

Highlights

By integrating the test framework with ReportPortal, EPAM's team provided:

  • Simplified test runs

  • Key info for the manual root cause analysis of test failures, such as logs, screenshots, attachments

  • Clear reporting for non-technical stakeholders

Benefits & Results

  • Improved automation stability from 60% to 77% in one sprint

  • Discovered that most failures were caused by environment issues and reduced their number from 20% to 2%

  • Reduced test automation results analysis effort by 45%

  • Provided clear and comprehensive test automation reporting: number of test cases, regression passing rate, reasons for failures, product status