Test Reports

RocketQA generates comprehensive, professional-grade test reports that provide deep insights into your test execution. These reports are essential for understanding test performance, debugging failures, and maintaining high-quality test suites.

Why Test Reports Matter

Test reports are more than just pass/fail indicators—they’re your window into test health and application quality:

🔍 Debugging: Quickly identify what went wrong with detailed step-by-step execution logs
📊 Analytics: Track test performance trends and identify flaky tests
👥 Collaboration: Share results with stakeholders in an easy-to-understand format
🚀 CI/CD Integration: Automate decision-making in your deployment pipeline
📈 Continuous Improvement: Use metrics to optimize your testing strategy

Report Formats

RocketQA generates reports in multiple formats to suit different needs:

HTML Report

The HTML report is RocketQA’s flagship reporting feature, providing a rich, interactive dashboard for analyzing test results. It’s designed for both technical teams and stakeholders who need to understand test outcomes.

Location: reports/report.html

Key Features

📊 Executive Dashboard

High-level test execution summary with pass/fail ratios
Test duration trends and performance metrics
Feature-wise breakdown of test results
Historical comparison with previous runs

📝 Detailed Test Results

Complete scenario execution logs with step-by-step details
Before/after state information for each test step
Element interaction logs with locator information
Test data and parameter values used

📸 Visual Debugging

Automatic screenshots captured on test failures
Screenshots at key interaction points (configurable)
Visual diff capabilities for UI changes
Full-page screenshots for comprehensive context

⏱️ Performance Analytics

Individual step execution times
Feature execution duration breakdown
Performance regression detection
Slow test identification and optimization suggestions

🏷️ Advanced Filtering

Filter by tags, features, scenarios, or steps
Search functionality across all test content
Status-based filtering (passed, failed, skipped)
Time-range filtering for historical analysis

🔍 Interactive Exploration

Expandable/collapsible test sections
Click-to-expand step details
Direct links to source code for each step
Export capabilities for specific test results

Opening the Report

# macOS
open reports/report.html

# Linux
xdg-open reports/report.html

# Windows
start reports/report.html

# Or simply navigate to the file in your browser
# File path: ./reports/report.html

The HTML report is organized into several key sections:

Summary Tab: Overview of test execution with key metrics
Features Tab: Detailed breakdown by feature files
Scenarios Tab: Individual scenario results with step details
Tags Tab: Results organized by test tags
Timeline Tab: Chronological view of test execution
Screenshots Tab: Visual evidence from test runs

JSON Report

The JSON report provides machine-readable test results perfect for automation, integration, and custom analysis. It follows the Cucumber JSON format standard, ensuring compatibility with a wide range of tools and platforms.

Location: reports/cucumber-report.json

Key Use Cases

🚀 CI/CD Pipeline Integration

Automated test result processing in build pipelines
Conditional deployment based on test outcomes
Integration with notification systems (Slack, Teams, email)
Automated rollback triggers on test failures

📊 Custom Reporting Dashboards

Building custom analytics dashboards
Real-time test monitoring systems
Business intelligence integration
Custom metrics and KPIs calculation

🔧 Test Metrics Collection

Test execution trends analysis
Performance regression tracking
Test coverage analysis
Quality gate enforcement

🤖 Automated Result Processing

Automated bug creation on test failures
Test result archiving and historical analysis
Custom alerting based on specific failure patterns
Integration with project management tools

JSON Structure Overview

The JSON report follows a hierarchical structure:

[
  {
    "description": "Feature description",
    "elements": [
      {
        "description": "Scenario description",
        "id": "unique-scenario-id",
        "keyword": "Scenario",
        "name": "Scenario name",
        "steps": [
          {
            "keyword": "Given|When|Then|And|But",
            "name": "Step description",
            "result": {
              "duration": 1234567890,
              "status": "passed|failed|skipped|undefined",
              "error_message": "Error details if failed"
            }
          }
        ],
        "type": "scenario",
        "tags": [
          {
            "name": "@tag-name",
            "line": 5
          }
        ]
      }
    ],
    "keyword": "Feature",
    "name": "Feature name",
    "tags": [],
    "uri": "path/to/feature.feature"
  }
]

Detailed Example

Here’s a comprehensive example showing various test scenarios:

[
  {
    "description": "User authentication and session management",
    "elements": [
      {
        "description": "Successful login with valid credentials",
        "id": "user-authentication;successful-login",
        "keyword": "Scenario",
        "name": "Successful login with valid credentials",
        "steps": [
          {
            "keyword": "Given ",
            "name": "navigate to \"https://your-app.com/login\"",
            "result": {
              "duration": 1234567890,
              "status": "passed"
            }
          },
          {
            "keyword": "When ",
            "name": "type \"john.doe@example.com\" into $loginPage.email",
            "result": {
              "duration": 987654321,
              "status": "passed"
            }
          },
          {
            "keyword": "And ",
            "name": "type \"securePassword123\" into $loginPage.password",
            "result": {
              "duration": 876543210,
              "status": "passed"
            }
          },
          {
            "keyword": "And ",
            "name": "click $loginPage.submitButton",
            "result": {
              "duration": 765432109,
              "status": "passed"
            }
          },
          {
            "keyword": "Then ",
            "name": "$dashboard.welcomeMessage is visible",
            "result": {
              "duration": 654321098,
              "status": "passed"
            }
          }
        ],
        "type": "scenario",
        "tags": [
          {
            "name": "@smoke",
            "line": 3
          },
          {
            "name": "@authentication",
            "line": 4
          }
        ]
      },
      {
        "description": "Login failure with invalid credentials",
        "id": "user-authentication;login-failure",
        "keyword": "Scenario",
        "name": "Login failure with invalid credentials",
        "steps": [
          {
            "keyword": "Given ",
            "name": "navigate to \"https://your-app.com/login\"",
            "result": {
              "duration": 1234567890,
              "status": "passed"
            }
          },
          {
            "keyword": "When ",
            "name": "type \"invalid@example.com\" into $loginPage.email",
            "result": {
              "duration": 987654321,
              "status": "passed"
            }
          },
          {
            "keyword": "And ",
            "name": "type \"wrongPassword\" into $loginPage.password",
            "result": {
              "duration": 876543210,
              "status": "passed"
            }
          },
          {
            "keyword": "And ",
            "name": "click $loginPage.submitButton",
            "result": {
              "duration": 765432109,
              "status": "passed"
            }
          },
          {
            "keyword": "Then ",
            "name": "$loginPage.errorMessage contains \"Invalid credentials\"",
            "result": {
              "duration": 654321098,
              "status": "failed",
              "error_message": "Expected error message 'Invalid credentials' but found 'Login failed'"
            }
          }
        ],
        "type": "scenario",
        "tags": [
          {
            "name": "@negative",
            "line": 15
          },
          {
            "name": "@authentication",
            "line": 16
          }
        ]
      }
    ],
    "keyword": "Feature",
    "name": "User Authentication",
    "tags": [],
    "uri": "features/authentication.feature"
  }
]