diff --git a/docs/accessibility/changelog.mdx b/docs/accessibility/changelog.mdx index 1ee1019fed..82b4abd1a6 100644 --- a/docs/accessibility/changelog.mdx +++ b/docs/accessibility/changelog.mdx @@ -9,6 +9,15 @@ sidebar_position: 200 # Changelog +## Week of 12/12/2025 + +- You can now create Jira issues directly from accessibility violations in the detail view. This allows you to quickly track and prioritize accessibility issues within your existing Jira workflow. The created Jira issue will include a link back to the accessibility violation in Cypress Cloud. [Learn more about creating Jira issues](/accessibility/core-concepts/inspecting-violation-details#create-jira-issue). + +## Week of 12/1/2025 + +- The `profiles` configuration property allows you to use different configuration settings for different runs based on [run tags](/app/references/command-line#cypress-run-tag-lt-tag-gt). This enables you to customize accessibility testing behavior for different environments, branches, or test scenarios. See the [Profiles](/accessibility/configuration/profiles) guide for more details. +- All configuration objects now support an optional `comment` property that you can use to provide context and explanations for why certain values are set. This helps make your configuration easier to understand and maintain, especially when working in teams or revisiting configuration after some time. + ## Week of 10/13/2025 - Individual rules, or set of rules, can now be ignored for specific elements using [a `data-a11y-ignore` attribute](/accessibility/configuration/ignoring-rules-per-element). diff --git a/docs/accessibility/configuration/overview.mdx b/docs/accessibility/configuration/overview.mdx index eac51fe0bf..853e3fbde1 100644 --- a/docs/accessibility/configuration/overview.mdx +++ b/docs/accessibility/configuration/overview.mdx @@ -22,50 +22,88 @@ To add or modify the configuration for your project, navigate to the "App Qualit alt="The Cypress Cloud UI showing the configuration editor" /> -You can use the provided editor to write configuration in JSON format. A complete configuration with all available options looks as follows: +You can use the provided editor to write configuration in JSON format. -```typescript +After new configuration changes have been saved, you can reprocess any historical run using the "regenerate" button on the properties tab where the configuration values that were used for that run are displayed. + +This allows you to iterate on the config quickly without having to execute test runs and wait for the results. You can try out different configurations and see what effects your changes have to make sure everything works the way you intend. + +### Viewing Configuration for a Run + +You can view configuration information for each run in the Properties tab, as shown below. This configuration is determined when your run begins. + + + +### Comments + +All configuration objects support an optional `comment` property that you can use to provide context and explanations for why certain values are set. This helps make your configuration easier to understand and maintain, especially when working in teams or revisiting configuration after some time. + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false, + "comment": "Exclude temporary test elements from accessibility reports" + } + ] +} +``` + +### Profiles + +The `profiles` property allows you to use different configuration settings for different runs based on [run tags](/app/references/command-line#cypress-run-tag-lt-tag-gt). See the [Profiles](/accessibility/configuration/profiles) guide for more details. + +### Complete configuration example + +A complete configuration with all available options looks as follows: + +```json { "views": [ { "pattern": string, "groupBy": [ string - ] + ], + "comment": string } ], "viewFilters": [ { "pattern": string, - "include": boolean + "include": boolean, + "comment": string } ], "elementFilters": [ { "selector": string, - "include": boolean + "include": boolean, + "comment": string } ], "significantAttributes": [ string - ] + ], "attributeFilters": [ { "attribute": string, "value": string, - "include": boolean + "include": boolean, + "comment": string + } + ], + "profiles": [ + { + "name": string, + "config": { + // Any App Quality configuration options + } } ] } ``` - -Note that these root-level App Quality configuration properties (`elementFilters`, `views`, and `viewFilters`) impact both UI Coverage and Accessibility. - -### Viewing Configuration for a Run - -You can view configuration information for each run in the Properties tab, as shown below. This is the configuration set for the project at the start of the run. - - diff --git a/docs/accessibility/configuration/profiles.mdx b/docs/accessibility/configuration/profiles.mdx new file mode 100644 index 0000000000..43cf2bf252 --- /dev/null +++ b/docs/accessibility/configuration/profiles.mdx @@ -0,0 +1,12 @@ +--- +sidebar_label: profiles +title: 'Profiles | Cypress Accessibility' +description: 'The `profiles` configuration property allows you to create configuration overrides that are applied based on run tags.' +sidebar_position: 100 +--- + + + +# profiles + + diff --git a/docs/accessibility/core-concepts/compare-reports.mdx b/docs/accessibility/core-concepts/compare-reports.mdx index bbffe3f60a..344238e669 100644 --- a/docs/accessibility/core-concepts/compare-reports.mdx +++ b/docs/accessibility/core-concepts/compare-reports.mdx @@ -164,3 +164,17 @@ It's likely that your tests have never been used for always-on accessibility tes For example, sometimes the test may spend enough time on a page to see a confirmation message after a form is submitted, unless the server response is slow, then the test may finish without rendering the state. There are different solutions depending on the nature of the issue. If you want to ensure the state is always picked up in Cypress Accessibility, you can add some assertion about that state to your tests. On the other hand, if you do not want to account for this state at all, it can be ignored with [`elementFilters`](/accessibility/configuration/elementfilters) configuration. Either approach will lead to a more stable branch comparison. + +## Optimizing for Pull Request reviews + +If you plan to use branch comparison for investigating Pull Requests that have been blocked through the Results API for not meeting your standard, it's useful to implement this in a gradual way, prioritizing the developer experience at first. + +Choose a scope of rules, page, and elements that reduces ambiguity. A good starting point is something like "We will fail a build if it introduces new critical issues on the login or checkout page." + +If your application is unstable between runs in terms of content, pages, or different states reached, fine tune your configuration to ignore the unstable parts of the DOM that have issues already. These can be documented and dealt with on their own. + +When everything is dialled in correctly, the Branch Review diff will be 100% clean, unless something has actually changed in an area you didn't already know about. This gives you the highest confidence that if an issue is blocking a merge, it is genuinely a new problem introduced by the incoming Pull Request. + +If the team is not already familiar with accessibility, it's useful to consider which rules to start with. See the [starting from scratch guide](/accessibility/guides/improve-accessibility) for suggestions about this. + +If multiple teams are working on the same project and need accessibility results to be isolated from each other, consider splitting runs into dedicated Cypress projects or using [configuration profiles](/accessibility/configuration/profiles) so that each team can have a customized report for the areas they are responsible for. diff --git a/docs/accessibility/core-concepts/inspecting-violation-details.mdx b/docs/accessibility/core-concepts/inspecting-violation-details.mdx index ce75d4cbea..df4f3c0965 100644 --- a/docs/accessibility/core-concepts/inspecting-violation-details.mdx +++ b/docs/accessibility/core-concepts/inspecting-violation-details.mdx @@ -36,7 +36,7 @@ Clicking on an element in the expanded Rule section provides: - **Pinned Element**: Highlights the element in the UI for easier identification. - **Copy Selector**: A button to copy the CSS selector uniquely identifying the element. - **Print to Console**: Logs the element reference to the browser's console for debugging. In Chrome-based browsers, you can right-click the logged element and select "Reveal in elements panel" to navigate directly to the live DOM. -- **Share issue**: A button to copy markdown or plain text snippet to share the issue with your team or collaborators. +- **Actions**: Share details of the accessibility problem or create a Jira issue (if the Jira integration is installed). - **Solutions**: A list of potential solutions to address the issue. +## Actions + +When you click on an element in the expanded Rule section, you can copy details of the accessibility violation to share, or directly create a Jira issue if you have the [Cypress Cloud Jira integration](/cloud/integrations/jira) installed. + +### Share issue + +The **Share issue** button allows you to copy a markdown or plain text snippet containing key details and related links for a particular accessibility violation. This makes it easy to share the issue with your team members or add it to tickets in your issue tracking system. + + + +The shared snippet includes: + +- The rule name and description +- The element selector +- Links back to the violation in Cypress Cloud +- Context about the violation + +### Create Jira issue + +You can create Jira issues directly from accessibility violations in the detail view. This allows you to quickly track and prioritize accessibility issues within your existing Jira workflow. + + + +To create a Jira issue: + +1. Click on an element in the expanded Rule section to open the element details. +2. Click the **Create Jira issue** button to open the Jira issue creation form. + + + +3. Complete the form by selecting the Jira project, issue type, assignee, and any additional fields required by your Jira configuration. +4. Submit the form to create the issue in Jira. + +The created Jira issue will include a link back to the accessibility violation in Cypress Cloud, making it easy to navigate between your issue tracker and the detailed violation information. + +Additional context like related elements or comments for future triage can be added to the issue summary. + ## Snapshots Snapshots are fully hydrated HTML and CSS representations of your application's state during the test. Unlike screenshots or video, these snapshots allow you to: @@ -53,4 +102,4 @@ Snapshots are fully hydrated HTML and CSS representations of your application's This area also has the Test Replay button to provide access to any tests where this snapshot appeared, as well as the specific URL of the snapshot displayed at the bottom of the screen. -It's also possible to cycle through the available snapshots to see all the states of the application that were captured, but this is usually not necessary. Think of snapshots more as examples and evidence relate to elements that are listed on the left hand side of the screen. The most useful way to go through the report is this way. +It's also possible to cycle through the available snapshots to see all the states of the application that were captured, but this is usually not necessary. Think of snapshots more as examples and evidence related to elements that are listed on the left hand side of the screen. The most useful way to browse the report is to move through the elements on the left. diff --git a/docs/accessibility/core-concepts/run-level-reports.mdx b/docs/accessibility/core-concepts/run-level-reports.mdx index 8be03ada3f..3eac664f0f 100644 --- a/docs/accessibility/core-concepts/run-level-reports.mdx +++ b/docs/accessibility/core-concepts/run-level-reports.mdx @@ -85,7 +85,7 @@ Attributes for each rule include: - **Inapplicable**: No applicable elements detected for this rule. - **Ignored by config**: The rule was excluded at the project level. - **Description**: A summary of the rule. -- **Severity**: The Axe Core® severity level for the rule. +- **Severity**: The Axe Core® impact level for the rule. - **Counts**: The counts of failed, inconclusive, and ignored elements for the rule. ## Filtering diff --git a/docs/accessibility/get-started/introduction.mdx b/docs/accessibility/get-started/introduction.mdx index b257ab948a..4b0cd2f325 100644 --- a/docs/accessibility/get-started/introduction.mdx +++ b/docs/accessibility/get-started/introduction.mdx @@ -9,11 +9,11 @@ sidebar_position: 10 # Automated accessibility checks on every test -Cypress Accessibility adds detailed accessibility checks and workflows in Cypress Cloud, with zero impact on test execution. +Cypress Accessibility adds detailed accessibility checks to Cypress Cloud. Checks run against every step of every test, with zero impact on test execution. -- Instantly visualize, triage, and fix accessibility violations without any additional code or configuration. -- Dive deep into each violation with live, fully-rendered DOM snapshots of your application as it appeared during your tests. -- Filter out the noise to explore only newly-introduced issues related to specific commits. +- Visualize, triage, and fix accessibility violations without any additional code or configuration. +- Debug each violation with interactive DOM snapshots from. +- Filter to newly-introduced issues related to specific commits. - Track your team's progress over time with historical scores to monitor improvements and identify regressions. { ``` By examining the results and customizing your response, you gain maximum control over how to handle accessibility violations. Leverage CI environment context, such as tags, to fine-tune responses to specific accessibility outcomes. + +## Using Profiles for PR-specific configuration + +You can use [Profiles](/accessibility/configuration/profiles) to apply different configuration settings for pull request runs versus regression runs. This allows you to: + +- Use a narrow, focused configuration for PR runs that blocks merges based on critical accessibility issues +- Maintain a broader configuration for regression runs that tracks all issues for long-term monitoring +- Apply team-specific configurations when multiple teams share the same Cypress Cloud project + +For example, you might configure a profile named `aq-config-pr` that excludes non-critical pages and focuses only on the most important accessibility rules, while your base configuration includes all pages and rules for comprehensive regression tracking. + +```shell +cypress run --record --tag "aq-config-pr" +``` + +This approach ensures that PR checks are fast and focused, while still maintaining comprehensive reporting for your full test suite. + +## Comparing against a baseline {#comparing-against-a-baseline} + +Comparing current results against a stored baseline allows you to detect only new violations that have been introduced, while ignoring existing known issues. This approach is more sophisticated than simply maintaining a list of known failing rules, as it tracks violations at the view level and can detect both regressions (new violations) and improvements (resolved violations). + +This is particularly useful in CI/CD pipelines where you want to fail builds only when new accessibility violations are introduced, allowing you to address existing issues incrementally without blocking deployments. + +### Baseline structure + +You can use any format you like as baseline for comparing reports. In the example code below we generate a baseline in a simplified format, which captures the state of accessibility violations from a specific run. The Results API handler code logs this for every run so that it can be easily copied as a new reference point. + +It includes: + +- **runNumber**: The run number used as the baseline reference +- **severityLevels**: The severity levels to track (e.g., `['critical', 'serious', 'moderate', 'minor']`) +- **views**: An object mapping view display names to arrays of failed rule names for that view + +```javascript +{ + "runNumber": "111", + "severityLevels": [ + "critical", + "serious", + "moderate", + "minor" + ], + "views": { + "/": [ + "aria-dialog-name", + "heading-order", + "scrollable-region-focusable" + ], + "/authorizations": [ + "heading-order", + "listitem", + "region" + ] + } +} +``` + +### Complete example + +The following example demonstrates how to compare current results against a baseline, detect new violations, identify resolved violations, and generate a new baseline when changes are detected. + +```javascript title="scripts/compareAccessibilityBaseline.js" +require('dotenv').config() + +const { getAccessibilityResults } = require('@cypress/extract-cloud-results') +const fs = require('fs') + +const TARGET_SEVERITY_LEVELS = ['critical', 'serious', 'moderate', 'minor'] + +// Parse the run number from an accessibility report URL +const parseRunNumber = (url) => { + return url.split('runs/')[1].split('/accessibility')[0] +} + +// Define your baseline - beginning with no details +// will list all current violations as "new", and log a +// baseline that you can copy and save as your starting point + +const baseline = { + runNumber: '', + severityLevels: ['critical', 'serious', 'moderate', 'minor'], + views: {}, +} + +getAccessibilityResults().then((results) => { + // Create objects to store the results + const viewRules = {} + const viewsWithNewFailedRules = {} + const viewsWithMissingRules = {} + + // Iterate through each view in current results + results.views.forEach((view) => { + const displayName = view.displayName + const ruleNames = view.rules.map((rule) => rule.name) + + // Add to our results object + viewRules[displayName] = ruleNames + + // Check for new failing rules + if (view.rules.length) { + view.rules.forEach((rule) => { + if (!TARGET_SEVERITY_LEVELS.includes(rule.severity)) { + return + } + + // If the view exists in baseline and the rule is not in baseline's failed rules + if (!baseline.views?.[displayName]?.includes(rule.name)) { + if (viewsWithNewFailedRules[displayName]) { + viewsWithNewFailedRules[displayName].newFailedRules.push({ + name: rule.name, + url: rule.accessibilityReportUrl, + }) + } else { + viewsWithNewFailedRules[displayName] = { + newFailedRules: [ + { + name: rule.name, + url: rule.accessibilityReportUrl, + }, + ], + } + } + } + }) + } + }) + + // Check for rules in baseline that are not in current results (resolved rules) + Object.entries(baseline.views).forEach(([displayName, baselineRules]) => { + const currentRules = viewRules[displayName] || [] + const resolvedRules = baselineRules.filter( + (rule) => !currentRules.includes(rule) + ) + + if (resolvedRules.length > 0) { + viewsWithMissingRules[displayName] = { resolvedRules } + } + }) + + // Report any missing rules + const countOfViewsWithMissingRules = Object.keys(viewsWithMissingRules).length + const countOfViewsWithNewFailedRules = Object.keys( + viewsWithNewFailedRules + ).length + + if (countOfViewsWithMissingRules || countOfViewsWithNewFailedRules) { + // Generate and log the new baseline values if there has been a change + const newBaseline = generateBaseline(results) + console.log('\nTo use this run as the new baseline, copy these values:') + console.log(JSON.stringify(newBaseline, null, 2)) + fs.writeFileSync('new-baseline.json', JSON.stringify(newBaseline, null, 2)) + } + + if (countOfViewsWithMissingRules) { + console.log( + '\nThe following Views had rules in the baseline that are no longer failing. This may be due to improvements in these rules, or because you did not run as many tests in this run as the baseline run:' + ) + console.dir(viewsWithMissingRules, { depth: 3 }) + } else if (!countOfViewsWithNewFailedRules) { + console.log( + '\nNo new or resolved rules were detected. All violations match the baseline.\n' + ) + } + + if (countOfViewsWithNewFailedRules) { + // Report any new failing rules + console.error( + '\nThe following Views had rules violated that were previously passing:' + ) + console.dir(viewsWithNewFailedRules, { depth: 3 }) + throw new Error( + `${countOfViewsWithNewFailedRules} Views contained new failing accessibility rules.` + ) + } + + return viewRules +}) + +function generateBaseline(results) { + try { + // Create an object to store the results + const viewRules = {} + + // Iterate through each view + results.views.forEach((view) => { + // Get the displayName and extract the rule names + const displayName = view.displayName + const ruleNames = view.rules + .filter((rule) => TARGET_SEVERITY_LEVELS.includes(rule.severity)) + .map((rule) => rule.name) + + // Add to our results object + viewRules[displayName] = ruleNames + }) + + const runNumber = parseRunNumber(results.views[0].accessibilityReportUrl) + + return { + runNumber, + severityLevels: TARGET_SEVERITY_LEVELS, + views: viewRules, + } + } catch (error) { + console.error('Error parsing accessibility results:', error) + return null + } +} +``` + +### Key concepts in the example script + +#### New failing rules + +A **new failing rule** is a rule that has violations on a View in the current run that were not present in the baseline. These represent regressions that need to be addressed. The script will fail the build if any new failing rules are detected. + +#### Resolved rules + +A **resolved rule** is a rule that was present in the baseline but no longer has violations in the current run. These represent improvements in accessibility. The script reports these but does not fail the build, allowing you to track progress, and printing out an updated baseline to copy over when things are fixed. + +#### Severity filtering + +The baseline comparison only tracks rules at the severity levels you specify in `TARGET_SEVERITY_LEVELS`. This allows you to focus on the severity levels that matter most to your team. Rules at other severity levels are ignored during comparison. + +#### View-level comparison + +Violations are tracked per view (URL pattern or component), allowing you to see exactly which pages or components have regressed or improved. This granular tracking makes it easier to identify the source of changes and assign fixes to the appropriate teams. + +### Best practices + +#### When to update the baseline + +Update your baseline when: + +- You've fixed accessibility violations and want to prevent regressions +- You've accepted certain violations as known issues that won't block deployments + +We recommend storing the baseline in version control so it's versioned alongside your code and accessible in CI environments, but some users prefer updating the baseline automatically based on the newest results from a full passing test run on the main branch of the project. This means any PRs will be compared against the reference run that makes sense at the time the results are analyzed, and avoids needing to manually own the standard. + +#### Handling partial reports + +If a run is cancelled or incomplete, the Results API may return a partial report. Consider checking `summary.isPartialReport` before comparing against the baseline, as partial reports may not include all views and could produce false positives. + +#### Managing baseline across branches + +You may want different baselines for different branches (e.g., `main` vs feature branches). Consider storing baselines in branch-specific files or using environment variables to specify which baseline to use. + +#### Storing the baseline + +Common approaches for storing baselines: + +- **Version control**: Commit the baseline JSON file to your repository +- **CI artifacts**: Store baselines as build artifacts that can be retrieved in subsequent runs diff --git a/docs/accessibility/guides/maintain-accessibility.mdx b/docs/accessibility/guides/maintain-accessibility.mdx index d91ff5baab..57ee5bb370 100644 --- a/docs/accessibility/guides/maintain-accessibility.mdx +++ b/docs/accessibility/guides/maintain-accessibility.mdx @@ -9,7 +9,7 @@ sidebar_position: 30 # Maintain accessibility -Accessibility isn't a one-time project—it's a continuous process. This guide explains how to transition from addressing known issues to maintaining long-term accessibility in your projects. +Accessibility is never a one-time project that can be completed in a certain timeframe and considered done. For an actively developed application, accessibility is a continuous process like all other aspects of quality. This guide explains how to transition from addressing known issues to maintaining long-term accessibility in your projects. ## Switching from "improving" to "maintaining" @@ -20,6 +20,8 @@ When a specific accessibility rule reaches zero violations for a page, component Your ultimate goal is to achieve a fully passing state across all rules. Cypress provides you a lot of flexibility to manage large amounts of accessibility violations and make incremental progress towards well-defined targets. Through a combination of fixing accessibility problems, and configuring Cypress to focus on a tight scope of rules, standards, and application areas that matter most to you, you might be closer than you think to a "clean" main branch, from which you can expand your standards over time. +You can also use [Configuration Profiles](/accessibility/configuration/profiles) to manage different standards for different run types, such as stricter requirements for pull requests versus comprehensive monitoring for regression runs. + ## Maintaining accessibility standards A clean report marks a pivotal change in your workflow. Instead of working through backlogs, the focus shifts to preventing accessibility issues from merging into the main branch. diff --git a/docs/accessibility/results-api.mdx b/docs/accessibility/results-api.mdx index c230ac672c..d65d123685 100644 --- a/docs/accessibility/results-api.mdx +++ b/docs/accessibility/results-api.mdx @@ -257,6 +257,10 @@ The Accessibility results for the run are returned as an object containing the f } ``` +## Comparing against a baseline + +For comprehensive examples of comparing results against a baseline, including complete code examples, baseline structure, and best practices, see the [Block pull requests and set policies](/accessibility/guides/block-pull-requests#comparing-against-a-baseline) guide. + ### **2. Add to CI Workflow** In your CI workflow that runs your Cypress tests, @@ -269,7 +273,7 @@ In your CI workflow that runs your Cypress tests, If you record multiple runs in a single CI build, you must record these runs using the `--tag` parameter and then call `getAccessibilityResults` with the `runTags` argument for each run. -This is necessary to identify each unique run and return a corresponding set of results. The tags are how each run is uniquely identified. +This is necessary to identify each unique run and return a corresponding set of results. The tags are how each run is uniquely identified. Tags can also be used to activate [Profiles](/accessibility/configuration/profiles) that apply different configuration settings to your runs. **Example** diff --git a/docs/app/references/command-line.mdx b/docs/app/references/command-line.mdx index 68c073654e..7b48352031 100644 --- a/docs/app/references/command-line.mdx +++ b/docs/app/references/command-line.mdx @@ -594,6 +594,11 @@ Cypress Cloud will display any tags sent with the appropriate run. alt="Cypress run in Cypress Cloud displaying flags" /> +:::info + +**App Quality Profiles**: If you use Cypress Cloud's [accessibility](/accessibility/get-started/introduction) and [UI Coverage reporting](/accessibility/get-started/introduction), a run tag can be mapped to [App Quality Profiles](/accessibility/configuration/profiles) to automatically apply different configuration settings for different types of runs. +::: + #### Exit code Cypress supports two different exit code behaviors. The default behavior is similar to that of Mocha, but is not POSIX compliant: reserved exit codes can be returned for certain conditions. As well, the default behavior diff --git a/docs/partials/_attributefilters.mdx b/docs/partials/_attributefilters.mdx index 91474d0613..b601c11b2e 100644 --- a/docs/partials/_attributefilters.mdx +++ b/docs/partials/_attributefilters.mdx @@ -1,4 +1,4 @@ -Some attributes used for identification may be auto-generated, dynamic, or unrepresentative, leading to inaccurate identification or grouping. The `attributeFilters` configuration property allows you to **exclude** specific attributes or patterns that should not be used for these purposes. +Some attributes used for identification may be auto-generated, dynamic, or state-based, leading to inaccurate identification or grouping. The `attributeFilters` configuration property allows you to **exclude** specific attributes or patterns that should not be used for these purposes. By using `attributeFilters`, you can ensure Cypress selects more appropriate identifiers, leading to cleaner and more accurate reports, with better element de-deduplication across distinct states of the application being tested. @@ -6,7 +6,7 @@ By using `attributeFilters`, you can ensure Cypress selects more appropriate ide - **Handling library-specific attributes**: Attributes generated by libraries may not represent the element's purpose and should be ignored. - **Improving grouping accuracy**: By filtering out irrelevant attributes, you ensure similar elements are grouped correctly. -- **Streamlining reports**: Eliminating noisy attributes reduces clutter in Cypress Accessibility or UI Coverage reports, making them easier to interpret and act upon. +- **Avoiding state-based duplication**: Classes like `link--focused` can clutter report findings. ## Scope @@ -24,7 +24,8 @@ supported, if you need to split them up. { "attribute": string, "value": string, - "include": boolean + "include": boolean, + "comment": string } ] } @@ -32,29 +33,34 @@ supported, if you need to split them up. ### Options -For every attribute that an element has, the first `attributeFilters` rule for which the `attribute` property matches the attribute's name and the `value` property matches the attribute's value, the `include` value is used to determine whether or not the attribute will be used for element identification and grouping. Attributes that do not match any rules are included by default. +Whether Cypress is allowed to use a certain attribute to identify an element when processing it for UI Coverage or Cypress Accessibility reports will be controlled by the **first** filter that matches the name and value of that attribute. + +This means that catch-all rules can be added at the bottom of the list by setting `include` to `false`, and exceptions can be defined earlier in the list using `include: true`. For example, you could avoid all use of `aria-label` for identification of elements as a catch-call filter, but then define exceptions for certain values it may have where it is a good identifier. + +Attributes that do not match any rules are included by default and used if needed, so `include: true` is only required for defining exceptions. If you want to make sure an attribute is preferred as an identifier when available, add it to your [attributeFilters](/accessibility/configuration/attributefilters). | Option | Required | Default | Description | | ----------- | -------- | ------- | ---------------------------------------------------------------------- | | `attribute` | Required | | A regex string to match attribute names | | `value` | Optional | `.*` | A regex string to match attribute values | | `include` | Optional | `true` | A boolean to specify whether the matched attribute should be included. | +| `comment` | Optional | | A comment describing the purpose of this filter rule. | ## Examples -### Excluding common auto-generated id values +### Excluding common auto-generated ID values + +Some component libraries generate ```json { - "uiCoverage": { - "attributeFilters": [ - { - "attribute": "id", - "value": ":r.*:", - "include": false - } - ] - } + "attributeFilters": [ + { + "attribute": "id", + "value": ":r.*", + "include": false + } + ] } ``` @@ -67,7 +73,7 @@ For every attribute that an element has, the first `attributeFilters` rule for w ``` -#### Elements shown in UI Coverage +#### Element identifiers displayed ``` [name="my-button"] @@ -76,108 +82,102 @@ For every attribute that an element has, the first `attributeFilters` rule for w --- -### Excluding auto-generated attribute names +### Filtering related dynamic attributes + +When filtering dynamic `id` attributes, you should also filter attributes that reference those IDs to prevent elements from being identified by these related dynamic values. Common relationships include: + +- Form associations (`for` attributes on labels) +- ARIA relationships (`aria-labelledby`, `aria-describedby`, `aria-controls`, `aria-owns`, `aria-details`) +- Name attributes that may mirror IDs in certain frameworks ```json { - "uiCoverage": { - "attributeFilters": [ - { - "attribute": "ng-include-me", - "value": ".*", - "include": true - }, - { - "attribute": "ng-.*", - "value": ".*", - "include": false - } - ] - } + "attributeFilters": [ + { + "attribute": "id|for|name|aria-.*", + "value": "dynamic-.*", + "include": false + } + ] } ``` #### HTML ```xml - - - - +
+ + +

Enter your first name

+
``` -#### Elements shown in UI Coverage +#### Element identifiers displayed ``` -[ng-include-me="my-button"] -:nth-child(2) +label +input +p ``` -### Ignoring dynamic attributes for accurate grouping +### Excluding auto-generated attribute names ```json { - "uiCoverage": { - "attributeFilters": [ - { - "attribute": "data-cy", - "value": "user-\\d+", - "include": false - } - ] - } + "attributeFilters": [ + { + "attribute": "ng-include-me", + "value": ".*", + "include": true + }, + { + "attribute": "ng-.*|_ng.*", + "value": ".*", + "include": false + } + ] } ``` #### HTML ```xml - - + + + + ``` -#### Elements shown in UI Coverage +#### Element identifiers displayed ``` -.user (2 instances) +[ng-include-me="my-button"] +:nth-child(2) ``` -### Filtering related dynamic attributes - -When filtering dynamic `id` attributes, you should also filter attributes that reference those IDs to prevent elements from being identified by these related dynamic values. Common relationships include: - -- Form associations (`for` attributes on labels) -- ARIA relationships (`aria-labelledby`, `aria-describedby`, `aria-controls`, `aria-owns`, `aria-details`) -- Name attributes that may mirror IDs in certain frameworks +### Ignoring dynamic attributes for accurate grouping ```json { - "uiCoverage": { - "attributeFilters": [ - { - "attribute": "id|for|name|aria-.*", - "value": "dynamic-.*", - "include": false - } - ] - } + "attributeFilters": [ + { + "attribute": "data-cy", + "value": "user-\\d+", + "include": false + } + ] } ``` #### HTML ```xml -
- - -

Enter your first name

-
+ + ``` -#### Elements shown in UI Coverage +#### Element identifiers displayed ``` -label -input -p +.user (2 instances) ``` diff --git a/docs/partials/_elementfilters.mdx b/docs/partials/_elementfilters.mdx index a71b966a0a..ab15d81d35 100644 --- a/docs/partials/_elementfilters.mdx +++ b/docs/partials/_elementfilters.mdx @@ -18,7 +18,8 @@ supported, if you need to split them up. "elementFilters": [ { "selector": string, - "include": boolean + "include": boolean, + "comment": string } ] } @@ -32,6 +33,7 @@ The first `elementFilters` rule for which the selector property matches the elem | ---------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `include` | Optional | `true` | A boolean indicating whether the matched elements should be included in the report. | +| `comment` | Optional | | A comment describing the purpose of this filter rule. | ## Examples @@ -44,7 +46,8 @@ The first `elementFilters` rule for which the selector property matches the elem "elementFilters": [ { "selector": "#button-2", - "include": false + "include": false, + "comment": "Exclude test-only button from reports" } ] } diff --git a/docs/partials/_profiles.mdx b/docs/partials/_profiles.mdx new file mode 100644 index 0000000000..d58bdb723a --- /dev/null +++ b/docs/partials/_profiles.mdx @@ -0,0 +1,251 @@ +The `profiles` property allows you to create configuration overrides that are applied based on [run tags](/app/references/command-line#cypress-run-tag-lt-tag-gt). This enables you to use different configuration settings for different types of runs in the same Cypress Cloud project, such as regression tests compared to smoke tests environments, or providing scoped results relevant to specific teams. + +In many cases, the need for different profiles can be avoided completely by having distinct Cypress Cloud projects for different kinds of runs or different team ownership. This is the recommended first choice, because it usually simplifies reporting and tracking. But where distinct projects are not the right answer, profiles will help. + +## Why use profiles? + +- **Team specific reporting**: If multiple teams use the same Cypress Cloud project, they may have completely different areas of concern for which pages or DOM elements are included in reports about accessibility or UI Coverage. Profiles allow each team to see and track against only the results that matter to them, and remove all other findings. +- **Run-type customization**: Use different filters or settings for regression runs versus pull requests. It can be useful to have a narrow config for blocking a merge, optimized for the most critical areas of your app and high "solvability", while still running a wide config on a full regression suite to manage findings on a difference cadence. +- **Skip runs when needed**: If you know that certain kinds of runs are not going to be valuable for App Quality reporting, you can ignore all view on these runs so that no report is created. In some situations this can improve clarity about when to look at a report and which reports are considered significant. + +## How profiles work + +Profiles are selected by matching run tags to profile names. When you run Cypress with the [`--tag`](/app/references/command-line#cypress-run-tag-lt-tag-gt) flag, Cypress Cloud looks for a profile whose `name` matches one of the tags. If a match is found, properties defined in profile's `config` properties override the root configuration. Properties defined in the root that are not referenced in the profile will be inherited, meaning you do not need to repeat config values that you want to keep the same. + +If more than one tag provided to a run matches a profile in your App Quality profiles array, the first matching profile in the array will be used. The order in which the tags are passed to the run doesn't matter. + +## Best practices + +Use a naming convention like `aq-config-x` (e.g., `aq-config-regression`, `aq-config-staging`) to make it clear that a tag is used for configuration lookup purposes. + +While relying on existing tags works just fine, being explicit will help avoid unintentionally changing or removing a tag which is depended on for a profile. + +## Syntax + +```json +{ + "profiles": [ + { + "name": string, + "config": { + // Any App Quality configuration options + }, + "comment": string + } + ] +} +``` + +### Options + +| Option | Required | Description | +| --------- | -------- | --------------------------------------------------------------------------------------------------------- | +| `name` | Required | The profile name that must match a run tag to activate this profile. | +| `config` | Required | An object containing any App Quality configuration options. These values override the root configuration. | +| `comment` | Optional | A comment describing the purpose of this profile. | + +## Examples + +### Basic profile structure + +#### Config + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false, + "comment": "Exclude temporary test elements" + } + ], + "profiles": [ + { + "name": "aq-config-regression", + "config": { + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false, + "comment": "Exclude temporary test elements in regression runs" + }, + { + "selector": "[data-role='debug']", + "include": false, + "comment": "Exclude debug elements in regression runs" + } + ] + } + } + ] +} +``` + +#### Usage + +To use this profile, run Cypress with a matching tag: + +```shell +cypress run --record --tag "aq-config-regression" +``` + +When this tag is used, the profile's `elementFilters` configuration will override the base `elementFilters`, excluding both temporary test elements and debug elements. + +--- + +### Multiple profiles + +You can define multiple profiles for different scenarios: + +#### Config + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false + } + ], + "profiles": [ + { + "name": "aq-config-regression", + "config": { + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false + }, + { + "selector": "[data-role='debug']", + "include": false, + "comment": "Exclude debug elements in regression runs" + } + ] + } + }, + { + "name": "aq-config-staging", + "config": { + "viewFilters": [ + { + "pattern": "https://staging.example.com/admin/*", + "include": false, + "comment": "Exclude admin pages in staging runs" + } + ] + } + } + ] +} +``` + +#### Usage + +Use different tags to activate different profiles: + +```shell +# For regression runs +cypress run --record --tag "aq-config-regression" + +# For staging runs +cypress run --record --tag "aq-config-staging" +``` + +--- + +### Profile with nested configuration + +Profiles can override configuration at any level, including nested configuration specific to Cypress Accessibility or UI Coverage, if your project has both projects enabled. For example: + +#### Config + +```json +{ + "elementFilters": [ + { + "selector": "[data-testid*='temp']", + "include": false + } + ], + "uiCoverage": { + "attributeFilters": [ + { + "attribute": "id", + "value": ":r.*:", + "include": false, + "comment": "Filter out React auto-generated IDs" + } + ] + }, + "profiles": [ + { + "name": "aq-config-feature-branch", + "config": { + "uiCoverage": { + "elementGroups": [ + { + "selector": "[data-feature='new-checkout']", + "name": "New Checkout Flow", + "comment": "Group new checkout elements for feature branch testing" + } + ] + } + } + } + ] +} +``` + +#### Usage + +```shell +cypress run --record --tag "aq-config-feature-branch" +``` + +This profile adds element grouping for the new checkout flow while keeping the base configuration for element filters and attribute filters. + +--- + +### Profile selection with multiple matching tags + +If you pass multiple tags and more than one matches a profile name, the first matching profile in the `profiles` array is used: + +#### Config + +```json +{ + "profiles": [ + { + "name": "aq-config-regression", + "config": { + "elementFilters": [ + { + "selector": "[data-role='debug']", + "include": false + } + ] + } + }, + { + "name": "aq-config-staging", + "config": { + "viewFilters": [ + { + "pattern": "https://staging.example.com/admin/*", + "include": false + } + ] + } + } + ] +} +``` + +#### Usage + +```shell +cypress run --record --tag "aq-config-regression,aq-config-staging" +``` + +In this case, the `aq-config-regression` profile will be used because it appears first in the `profiles` array, even though both tags match profile names. diff --git a/docs/partials/_viewfilters.mdx b/docs/partials/_viewfilters.mdx index d677361e85..a6c22304d9 100644 --- a/docs/partials/_viewfilters.mdx +++ b/docs/partials/_viewfilters.mdx @@ -27,7 +27,8 @@ supported, if you need to split them up. "viewFilters": [ { "pattern": string, - "include": boolean + "include": boolean, + "comment": string } ] } @@ -41,6 +42,7 @@ For every URL visited and link element found, the first `viewFilters` rule for w | --------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------- | | `pattern` | Required | | A string that matches URLs using [URL Pattern API](https://developer.mozilla.org/en-US/docs/Web/API/URL_Pattern_API) syntax. | | `include` | Optional | `true` | A boolean that determines whether matching URLs should be included in the report. | +| `comment` | Optional | | A comment describing the purpose of this filter rule. | ## Examples diff --git a/docs/partials/_views.mdx b/docs/partials/_views.mdx index b458411835..bc453d4d6b 100644 --- a/docs/partials/_views.mdx +++ b/docs/partials/_views.mdx @@ -34,7 +34,8 @@ URLs with the same values for the specified parameters will be grouped together, "pattern": string, "groupBy": [ string - ] + ], + "comment": string } ] } @@ -48,6 +49,7 @@ The first pattern that a given URL matches is used as its view. If a URL doesn't | --------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pattern` | Required | | A URL pattern to group matching URLs into a single view. Uses [URL Pattern API](https://developer.mozilla.org/en-US/docs/Web/API/URL_Pattern_API) syntax. | | `groupBy` | Optional | | An array of named parameters from your pattern that should create separate views. | +| `comment` | Optional | | A comment describing the purpose of this view configuration. | ## Examples diff --git a/docs/ui-coverage/changelog.mdx b/docs/ui-coverage/changelog.mdx index 6df262263d..ed61e8e376 100644 --- a/docs/ui-coverage/changelog.mdx +++ b/docs/ui-coverage/changelog.mdx @@ -9,6 +9,12 @@ sidebar_position: 200 # Changelog +## Week of 12/12/2025 + +- The `profiles` configuration property allows you to use different configuration settings for different runs based on [run tags](/app/references/command-line#cypress-run-tag-lt-tag-gt). This enables you to customize UI Coverage behavior for different environments, branches, or test scenarios. See the [Profiles](/ui-coverage/configuration/profiles) guide for more details. +- All configuration objects now support an optional `comment` property that you can use to provide context and explanations for why certain values are set. This helps make your configuration easier to understand and maintain, especially when working in teams or revisiting configuration after some time. +- A new guide on [blocking pull requests and setting policies](/ui-coverage/guides/block-pull-requests) is now available. This guide demonstrates how to use the Results API to enforce test coverage standards, compare results against baselines, and automatically block pull requests when coverage thresholds are not met. + ## Week of 6/23/25 UI Coverage now supports defining custom commands that will count towards coverage scores, restricting which kinds of interactions are allowed for certain elements, and including assertions in the UI Coverage calculations. See the following new properties for more details: diff --git a/docs/ui-coverage/configuration/allowedinteractioncommands.mdx b/docs/ui-coverage/configuration/allowedinteractioncommands.mdx index 98264e2f36..ef7467a4ff 100644 --- a/docs/ui-coverage/configuration/allowedinteractioncommands.mdx +++ b/docs/ui-coverage/configuration/allowedinteractioncommands.mdx @@ -9,7 +9,7 @@ sidebar_position: 100 # allowedInteractionCommands -UI Coverage tracks all interaction commands by default for comprehensive coverage reporting. The `allowedInteractionCommands` configuration allows you to limit which interaction commands are tracked for specific elements by defining rules based on CSS selectors. +UI Coverage tracks [all interaction commands](/ui-coverage/core-concepts/interactivity#Interaction-Commands) by default for comprehensive coverage reporting. The `allowedInteractionCommands` configuration allows you to limit which interaction commands are tracked for specific elements by defining rules based on CSS selectors. This is particularly useful for filtering out irrelevant interactions or focusing coverage tracking on specific interaction patterns for different types of elements. @@ -26,7 +26,8 @@ This is particularly useful for filtering out irrelevant interactions or focusin "allowedInteractionCommands": [ { "selector": string, - "commands": [string] + "commands": [string], + "comment": string } ] } @@ -41,6 +42,7 @@ The `allowedInteractionCommands` property accepts an array of objects, where eac | ---------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `commands` | Required | | An array of command names (strings) that should be tracked as interactions for elements matching the selector. All other interaction commands will be ignored for these elements. | +| `comment` | Optional | | A comment describing the purpose of this allowed interaction commands configuration. | ## Examples diff --git a/docs/ui-coverage/configuration/elementgroups.mdx b/docs/ui-coverage/configuration/elementgroups.mdx index 812fde8ed6..980c6f7a62 100644 --- a/docs/ui-coverage/configuration/elementgroups.mdx +++ b/docs/ui-coverage/configuration/elementgroups.mdx @@ -26,7 +26,8 @@ UI Coverage provides logic to automatically [group](/ui-coverage/core-concepts/e "elementGroups": [ { "selector": string, - "name": string + "name": string, + "comment": string } ] } @@ -41,6 +42,7 @@ For every element considered by UI Coverage, the first `elementGroup` rule for w | ---------- | -------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `name` | Optional | `selector` | A human-readable name for the group, displayed in UI Coverage reports. | +| `comment` | Optional | | A comment describing the purpose of this element group configuration. | ## Examples diff --git a/docs/ui-coverage/configuration/elements.mdx b/docs/ui-coverage/configuration/elements.mdx index d768403c03..c6baa57543 100644 --- a/docs/ui-coverage/configuration/elements.mdx +++ b/docs/ui-coverage/configuration/elements.mdx @@ -27,7 +27,8 @@ The `elements` configuration is used as the element's identity if **only one ele "elements": [ { "selector": string, - "name": string + "name": string, + "comment": string } ] } @@ -44,6 +45,7 @@ If multiple elements within the same snapshot satisfy the same rule, the rule ca | ---------- | -------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------- | | `selector` | Required | | A CSS selector to identify elements. Supports standard CSS selector syntax, including IDs, classes, attributes, and combinators. | | `name` | Optional | `selector` | A human-readable name for the element, displayed in UI Coverage reports. | +| `comment` | Optional | | A comment describing the purpose of this element configuration. | ## Examples diff --git a/docs/ui-coverage/configuration/overview.mdx b/docs/ui-coverage/configuration/overview.mdx index c4fe1903f9..6d5884a3ba 100644 --- a/docs/ui-coverage/configuration/overview.mdx +++ b/docs/ui-coverage/configuration/overview.mdx @@ -29,7 +29,11 @@ UI Coverage now supports defining custom commands that will count towards covera To add or modify the configuration for your project: 1. Navigate to the **App Quality** tab in your project settings on Cypress Cloud. -2. Use the configuration editor to add or edit confiuration in JSON format. +2. Use the configuration editor to add or edit configuration in JSON format. + +After new configuration changes have been saved, you can reprocess any historical run using the "regenerate" button on the properties tab where the configuration values that were used for that run are displayed, or through the UI Coverage report where the "configuration updated" message appears if you are looking at a historical run using an old config. + +Regenerating reports in this way allows you to make config changes and see their effects without running your Cypress tests again. + +# profiles + + diff --git a/docs/ui-coverage/get-started/introduction.mdx b/docs/ui-coverage/get-started/introduction.mdx index d60786f473..67b708ff1f 100644 --- a/docs/ui-coverage/get-started/introduction.mdx +++ b/docs/ui-coverage/get-started/introduction.mdx @@ -9,7 +9,7 @@ sidebar_position: 10 # Identify testing gaps with UI Coverage -Easily track, monitor, and visualize the test coverage of your UI to prevent regressions by ensuring critical flows of your app are tested. Save CI resources by removing redundant tests and improve your team's productivity with a visual overview of UI coverage across every page and component and quickly close coverage with Test Generation. +Easily track, monitor, and visualize the test coverage of your UI to prevent regressions by ensuring critical flows of your app are tested. Quickly close coverage gaps with Test Generation, and react to changes you previously would have missed with API access and on-demand comparisons between any two test runs. + +# Block pull requests and set policies + +Cypress UI Coverage reports are generated server-side in Cypress Cloud, based on test artifacts uploaded during execution. This ensures there is no performance impact on your Cypress test runs, but also means that nothing in your Cypress pipeline will fail due to coverage issues that are detected. Failing a build is a fully opt-in step based on your handling of the results in your CI process. + +## Using the Results API + +The [Cypress UI Coverage Results API](/ui-coverage/results-api) allows you to access UI Coverage results post-test run, enabling workflows like blocking pull requests or triggering alerts based on specific coverage criteria. This involves adding a dedicated UI Coverage verification step to your CI pipeline. With a Cypress helper function, you can automatically fetch the report for the relevant test run within the CI build context. + +## Implementing a status check + +The Results API offers full flexibility to analyze results and take tailored actions. It can also integrate with status checks on pull requests, allowing you to block merges when coverage thresholds are not met. + +## Defining policies in the verification step + +The [Results API Documentation](/ui-coverage/results-api) provides detailed guidance on the API's capabilities. Here's a simplified example demonstrating how to enforce a minimum coverage threshold: + +```js +const { getUICoverageResults } = require('@cypress/extract-cloud-results') + +// Fetch UI Coverage results +getUICoverageResults().then((results) => { + const { summary, views } = results + + // Verify overall coverage + if (summary.coverage < 80) { + throw new Error( + `Project coverage is ${summary.coverage}%, below the minimum threshold of 80%.` + ) + } + + // Verify critical view coverage + const criticalViews = [/login/, /checkout/] + views.forEach((view) => { + if ( + criticalViews.some((pattern) => pattern.test(view.displayName)) && + view.coverage < 95 + ) { + throw new Error( + `Critical view "${view.displayName}" coverage is ${view.coverage}%, below the required 95%.` + ) + } + }) +}) +``` + +By examining the results and customizing your response, you gain maximum control over how to handle coverage gaps. Leverage CI environment context, such as tags, to fine-tune responses to specific coverage outcomes. + +## Using Profiles for PR-specific configuration + +You can use [Profiles](/ui-coverage/configuration/profiles) to apply different configuration settings for pull request runs versus regression runs. This allows you to: + +- Use a narrow, focused configuration for PR runs that blocks merges based on critical coverage thresholds +- Maintain a broader configuration for regression runs that tracks all coverage gaps for long-term monitoring +- Apply team-specific configurations when multiple teams share the same Cypress Cloud project + +For example, you might configure a profile named `aq-config-pr` that excludes non-critical pages and focuses only on the most important coverage areas, while your base configuration includes all pages for comprehensive regression tracking. + +```shell +cypress run --record --tag "aq-config-pr" +``` + +This approach ensures that PR checks are fast and focused, while still maintaining comprehensive reporting for your full test suite. + +## Comparing against a baseline {#comparing-against-a-baseline} + +Comparing current results against a stored baseline allows you to detect only new untested elements that have been introduced, while ignoring existing coverage gaps. This approach helps you focus on regressions in test coverage and track improvements over time. + +This is particularly useful in CI/CD pipelines where you want to fail builds only when new untested elements are introduced, allowing you to address existing coverage gaps incrementally without blocking deployments. + +### Baseline structure + +In our example the baseline is a JSON object that captures the state of untested elements from a specific run. It includes: + +- **runNumber**: The run number used as the baseline reference +- **views**: An object mapping view display names to arrays of view identifiers and their untested elements counts +- **runUrl**: The link to the cloud run that generated this report + +```javascript +{ + "runNumber": 68086, + "runUrl": "https://cloud.cypress.io/projects/ypt4pf/runs/68086", + "views": { + "/": { testedElements: 55 }, + "/login": { testedElements: 3 }, + "/products/*": { testedElements: 3 }, + } +} +``` + +#### Why use untested element counts instead of UI Coverage percentage scores? + +The [UI Coverage score](/ui-coverage/guides/identify-coverage-gaps#Overall-Score) is expressed as based on a ratio of what was and was not tested over time, within what was rendered in Cypress during the run. Since testing one new element might reveal many more elements that aren't tested yet, the score isn't useful for a fine-grained baseline comparison between runs. Comparing the number of tested elements gives a more accurate sense of whether one run has added or removed coverage when compared to another, and is a better predictor of what information would be in the report. + +### Complete example + +The following example demonstrates how to compare current results against a baseline, detect new views with untested elements, identify views where coverage has improved, and generate a new baseline on every run so that it's easy to copy and update if needed. + +```javascript title="scripts/compareUICoverageBaseline.js" +require('dotenv').config() + +const { getUICoverageResults } = require('@cypress/extract-cloud-results') +const fs = require('fs') + +// Define your baseline - this should be stored and updated as your application improves +// Running the script once with no baseline will generate a baseline for you +const baseline = { + runNumber: undefined, + runUrl: undefined, + views: [], +} + +// Function to compare coverage between current results and baseline +const compareTestedElementsWithBaseline = (currentResults, baselineData) => { + const testedElementsIssues = [] + const testedElementsImprovements = [] + const newPages = [] + + // Check if current results has views + if (!currentResults.views || !Array.isArray(currentResults.views)) { + console.log( + 'Warning: Current results do not contain a valid "views" array. Skipping comparison.' + ) + return { + testedElementsIssues, + testedElementsImprovements, + newPages, + hasChanges: false, + } + } + + const currentViews = currentResults.views + + // Check if baseline data has the expected structure + if (!baselineData.views) { + console.log( + 'Warning: Baseline data does not contain "views" property. Skipping comparison.' + ) + console.log('Current baseline structure:', Object.keys(baselineData)) + return { + testedElementsIssues, + testedElementsImprovements, + newPages, + hasChanges: true, + } + } + + const baselineViews = baselineData.views + + // Ensure baselineViews is an array + if (!Array.isArray(baselineViews)) { + console.log('Warning: Baseline views is not an array. Skipping comparison.') + return { + testedElementsIssues, + testedElementsImprovements, + newPages, + hasChanges: true, + } + } + + // Create a map of baseline testedElementsCount by displayName for quick lookup + const baselineTestedElementsMap = {} + baselineViews.forEach((view) => { + baselineTestedElementsMap[view.displayName] = view.testedElementsCount + }) + + // Create a map of current views by displayName to check for missing pages + const currentViewsMap = {} + currentViews.forEach((view) => { + currentViewsMap[view.displayName] = view.testedElementsCount + }) + + // Compare each current view with baseline + currentViews.forEach((currentView) => { + const pageName = currentView.displayName + const currentTestedElements = currentView.testedElementsCount + const pageExistsInBaseline = pageName in baselineTestedElementsMap + + if (pageExistsInBaseline) { + const baselineTestedElements = baselineTestedElementsMap[pageName] + if (currentTestedElements < baselineTestedElements) { + testedElementsIssues.push({ + page: pageName, + currentTestedElements: currentTestedElements, + baselineTestedElements: baselineTestedElements, + difference: baselineTestedElements - currentTestedElements, + }) + } else if (currentTestedElements > baselineTestedElements) { + testedElementsImprovements.push({ + page: pageName, + currentTestedElements: currentTestedElements, + baselineTestedElements: baselineTestedElements, + difference: currentTestedElements - baselineTestedElements, + }) + } + } else { + // New page not in baseline + newPages.push({ + page: pageName, + testedElementsCount: currentTestedElements, + }) + } + }) + + // Check for pages in baseline that are missing from current results + // These are treated as regressions (0 tested elements) + baselineViews.forEach((baselineView) => { + const pageName = baselineView.displayName + if (!(pageName in currentViewsMap)) { + testedElementsIssues.push({ + page: pageName, + currentTestedElements: 0, + baselineTestedElements: baselineView.testedElementsCount, + difference: baselineView.testedElementsCount, + isMissing: true, + }) + } + }) + + const hasChanges = + testedElementsIssues.length > 0 || + testedElementsImprovements.length > 0 || + newPages.length > 0 + + return { + testedElementsIssues, + testedElementsImprovements, + newPages, + hasChanges, + } +} + +function generateUICoverageBaseline(results) { + try { + // Generate simplified baseline with only runNumber, runUrl, and views with displayName and testedElementsCount + return { + runNumber: results.runNumber, + runUrl: results.runUrl, + views: results.views.map((view) => ({ + displayName: view.displayName, + testedElementsCount: view.testedElementsCount, + })), + } + } catch (error) { + console.error('Error generating UI Coverage baseline:', error) + return null + } +} + +getUICoverageResults() + .then((results) => { + // Compare tested elements count with baseline + const { + testedElementsIssues, + testedElementsImprovements, + newPages, + hasChanges, + } = compareTestedElementsWithBaseline(results, baseline) + + if (hasChanges) { + // Generate and log the new baseline values if there has been a change + const newBaseline = generateUICoverageBaseline(results) + console.log('\nTo use this run as the new baseline, copy these values:') + console.log(JSON.stringify(newBaseline, null, 2)) + fs.writeFileSync( + 'new-UICbaseline.json', + JSON.stringify(newBaseline, null, 2) + ) + } + + // Log tested elements regressions + if (testedElementsIssues.length > 0) { + console.error('\n❌ TESTED ELEMENTS REGRESSION DETECTED!') + console.error( + 'The following pages have fewer tested elements than the baseline:' + ) + console.error('') + + testedElementsIssues.forEach((issue) => { + console.error(` 📄 ${issue.page}`) + if (issue.isMissing) { + console.error( + ` ⚠️ MISSING FROM REPORT (treated as 0 tested elements)` + ) + console.error( + ` Current: 0 | Baseline: ${issue.baselineTestedElements} | Difference: -${issue.difference}` + ) + } else { + console.error( + ` Current: ${issue.currentTestedElements} | Baseline: ${issue.baselineTestedElements} | Difference: -${issue.difference}` + ) + } + console.error('') + }) + + console.error( + `Total pages with tested elements regression: ${testedElementsIssues.length}` + ) + } + + // Log tested elements improvements + if (testedElementsImprovements.length > 0) { + console.log('\n✅ TESTED ELEMENTS IMPROVEMENTS DETECTED!') + console.log( + 'The following pages have more tested elements than the baseline:' + ) + console.log('') + + testedElementsImprovements.forEach((improvement) => { + console.log(` 📄 ${improvement.page}`) + console.log( + ` Current: ${improvement.currentTestedElements} | Baseline: ${improvement.baselineTestedElements} | Difference: +${improvement.difference}` + ) + console.log('') + }) + + console.log( + `Total pages with tested elements improvement: ${testedElementsImprovements.length}` + ) + } + + // Log new pages + if (newPages.length > 0) { + console.log('\n📝 NEW PAGES DETECTED (not in baseline):') + newPages.forEach((page) => { + console.log( + ` 📄 ${page.page} - Tested Elements: ${page.testedElementsCount}` + ) + }) + console.log(`Total new pages: ${newPages.length}`) + } + + // Summary + if (!hasChanges) { + console.log( + '\n✅ All pages meet or exceed baseline tested elements count!' + ) + } else if (testedElementsIssues.length === 0) { + console.log('\n✅ No tested elements regressions detected!') + } + + // Exit with error code if there are regressions + if (testedElementsIssues.length > 0) { + process.exit(1) + } + }) + .catch((error) => { + console.error('Error getting UI coverage results:', error) + process.exit(1) + }) +``` + +### Key concepts + +#### New untested elements + +A **new untested element** situation occurs when a view has untested elements in the current run but did not have untested elements in the baseline. This represents a regression in test coverage that needs to be addressed. The script will fail the build if any views with new untested elements are detected. + +#### Resolved untested elements + +A **resolved untested element** situation occurs when a view had untested elements in the baseline but no longer has untested elements in the current run. This represents an improvement in test coverage. The script reports these but does not fail the build, allowing you to track progress. + +#### View-level comparison + +Untested elements are tracked per view (URL pattern or component), allowing you to see exactly which pages or components have coverage regressions or improvements. This granular tracking makes it easier to identify where new tests are needed or where coverage has improved. + +### Best practices + +#### When to update the baseline + +Update your baseline when: + +- You've added tests to cover previously untested elements and want to prevent regressions +- You've accepted certain coverage gaps as known issues that won't block deployments +- You want to track coverage improvements over time + +Store the baseline in version control so it's versioned alongside your code and accessible in CI environments. + +#### Handling partial reports + +If a run is cancelled or incomplete, the Results API may return a partial report. Consider checking `summary.isPartialReport` before comparing against the baseline, as partial reports may not include all views and could produce false positives. + +#### Managing baseline across branches + +You may want different baselines for different branches (e.g., `main` vs feature branches). Consider storing baselines in branch-specific files or using environment variables to specify which baseline to use. + +#### Storing the baseline + +Common approaches for storing baselines: + +- **Version control**: Commit the baseline JSON file to your repository +- **CI artifacts**: Store baselines as build artifacts that can be retrieved in subsequent runs +- **External storage**: Use cloud storage or a database for baselines if you need more sophisticated versioning + +:::info + +This baseline comparison approach complements the [Branch Review](/ui-coverage/guides/compare-reports) UI feature, which provides visual comparisons between runs. The programmatic approach is ideal for CI/CD automation, while Branch Review is better suited for manual investigation and code review workflows. + +::: diff --git a/docs/ui-coverage/results-api.mdx b/docs/ui-coverage/results-api.mdx index a789bc177c..50327cc947 100644 --- a/docs/ui-coverage/results-api.mdx +++ b/docs/ui-coverage/results-api.mdx @@ -47,6 +47,27 @@ If you check this in as a dependency, your installation will fail when we update Write a script to fetch UI Coverage results and assert test coverage criteria. This script will be executed in CI. +#### Basic example + +This snippet uses the `getUICoverageResults()` helper to log out the results. It assumes your Project ID and Record Key variable are set. The following should work in any of the supported CI Providers out of the box: + +```javascript title="scripts/verifyUICoverageResults.js" +// Assuming these environment variables are set: +// CYPRESS_PROJECT_ID=your-id +// CYPRESS_RECORD_KEY=your-record-key + +const { getUICoverageResults } = require('@cypress/extract-cloud-results') + +getUICoverageResults().then((results) => { + // use `console.dir` instead of `console.log` because the data is nested + console.dir(results, { depth: Infinity }) +}) +``` + +#### How to assert test coverage meets your requirements + +The following example demonstrates how to verify that test coverage meets minimum thresholds: + ```javascript title="scripts/verifyUICoverageResults.js" const { getUICoverageResults } = require('@cypress/extract-cloud-results') @@ -152,6 +173,10 @@ The `getUICoverageResults` utility returns the following data: } ``` +## Comparing against a baseline + +For comprehensive examples of comparing results against a baseline, including complete code examples, baseline structure, and best practices, see the [Block pull requests and set policies](/ui-coverage/guides/block-pull-requests#comparing-against-a-baseline) guide. + ### **2. Add to CI Workflow** In your CI workflow that runs your Cypress tests, @@ -164,7 +189,7 @@ In your CI workflow that runs your Cypress tests, If you record multiple runs in a single CI build, you must record these runs using the `--tag` parameter and then call `getUICoverageResults` with the `runTags` argument for each run. -This is necessary to identify each unique run and return a corresponding set of results. The tags are how each run is uniquely identified. +This is necessary to identify each unique run and return a corresponding set of results. The tags are how each run is uniquely identified. Tags can also be used to activate [Profiles](/ui-coverage/configuration/profiles) that apply different configuration settings to your runs. **Example** diff --git a/src/theme/MDXComponents.js b/src/theme/MDXComponents.js index 9620dad1a4..9d0e073a9a 100644 --- a/src/theme/MDXComponents.js +++ b/src/theme/MDXComponents.js @@ -28,6 +28,7 @@ import Icon from "@site/src/components/icon"; import ImportMountFunctions from "@site/docs/partials/_import-mount-functions.mdx"; import IntellisenseCodeCompletion from "@site/docs/partials/_intellisense-code-completion.mdx"; import ProductHeading from "@site/src/components/product-heading"; +import Profiles from "@site/docs/partials/_profiles.mdx"; import SignificantAttributes from "@site/docs/partials/_significantattributes.mdx"; import SourceMaps from "@site/docs/partials/_source-maps.mdx"; import SupportFileConfiguration from "@site/docs/partials/_support-file-configuration.mdx"; @@ -192,6 +193,7 @@ export default { ImportMountFunctions, IntellisenseCodeCompletion, ProductHeading, + Profiles, SignificantAttributes, SourceMaps, SupportFileConfiguration, diff --git a/static/img/accessibility/a11y-jira-create.png b/static/img/accessibility/a11y-jira-create.png new file mode 100644 index 0000000000..ffd71a1055 Binary files /dev/null and b/static/img/accessibility/a11y-jira-create.png differ diff --git a/static/img/accessibility/a11y-jira-modal.png b/static/img/accessibility/a11y-jira-modal.png new file mode 100644 index 0000000000..2533c449a6 Binary files /dev/null and b/static/img/accessibility/a11y-jira-modal.png differ