Skip to main content

Improving test automation stability

Challenges

  • Complex, manual test runs
  • Low stability of regression for unclear reasons (60% passing rate)
  • Unclear reporting for non-technical stakeholders, leading to the lack of transparency in test automation results and progress
  • Test automation feedback is unclear, unreliable, incomprehensible, and insufficient to decide to push the app to production

Highlights

By integrating the test framework with ReportPortal.io, EPAM's team provided:

  • Simplified test runs
  • Key info for the manual root cause analysis of test failures, such as logs, screenshots, attachments
  • A possibility to triage failed items (AI-based and manual)
  • Clear reporting for non-technical stakeholders

Results

  • Improved automation stability from 60% to 77% in one sprint
  • Discovered that most failures were caused by environment issues and reduced the number of such failures from 20% to 2%
  • Reduced test automation results analysis effort by 45%
  • Provided clear and comprehensive test automation reporting dashboard: number of test cases, regression passing rate, reasons for failures, product status