Parallel builds feature corrupts and grossly inflates coverage numbers if cobertura.xml has multiple entries per file with different path separators

[HebaruSan]

Hi!

I am attempting to move from Ubuntu-only coverage numbers to Ubuntu-and-Windows coverage numbers in KSP-CKAN/CKAN#4454. As far as I can tell, the Parallel Builds feature is supposed to be able to handle this. Toward that end, I set up a matrix-strategy job in our GitHub workflows that collects and uploads coverage data from both platforms, then finalizes them with the parallel-finished input (including the rerun_build workaround from #1629).

After struggling to get merged data that made sense, I set up a temporary test in which only the Windows numbers were "merged" (scare quotes because there's only one source build) into the final parallel build. If parallel builds were working as they're supposed to, I would expect to see the same numbers for all stats between the two builds, but that's not what happens.

Windows build: https://coveralls.io/jobs/173106504

The numbers are approximately accurate, comparing to the previous build on the main branch; they're a bit higher than I expected, but it's not inconceivable that the differences between platforms are significant enough to account for that. More investigation is called for once this parallel builds project is completed.

However, the "merged" data bears almost no resemblance to the only sole single build upon which it is supposedly based (and to which it links):

Merged: https://coveralls.io/builds/76114443

• 89.66% became 100.0%
• 84.46% became 99.05%
• 83.68% became 98.35%
• 88.92% became 98.93%
• 87.67% became 99.35%

These numbers are clearly not accurate or usable for the intended purpose of assessing the overall coverage.

The picture is no rosier at the individual file level. The accurate data in the Windows build shows the same "usual suspects" that show up as having double- and triple-digit missed lines in our main branch builds up to this point:

In the "combined" build, no file has more than 4 missed lines!

This data is not usable for the purpose of finding files that need more or better tests.

Coverage data that we are generating before the GitHub Action runs:

• coverage-windows-latest-1.zip

[afinetooth]

@HebaruSan Moving my comment from #1629 as we'll treat this as a new issue:

What we’ve found so far

It looks like we’ve identified the source of the duplicate files you’re seeing on your Job Pages: they’re already present in the original coverage report upload from your WINDOWS-LATEST job.

The uploaded coverage report itself (the JSON payload received by the Coveralls API) includes both versions of certain files—one with recorded hits and one with zero coverage. That explains why duplicates appear at the Job level.

Examples:

1. Coverage Report Upload for Job 18754394053.1 (SingleAssemblyResourceManager.cs):

2. Coverage Report Upload for Job 18754394053.1 (ModuleInstaller.cs):

To be clear:

• Coveralls does not modify or deduplicate coverage data in Job uploads.
  Each upload is treated as a canonical record of what your CI job produced.
• Deduplication and merging happen only later, when multiple Job uploads are combined into the final Build Page.
  That’s why your Build Page looks correct, even though the Job Page shows duplicates.

So, the duplication you’re seeing isn’t a bug in Coveralls processing—it reflects what’s in the incoming coverage data.

---

What I’m investigating next

The open question is why the duplication exists in the first place—specifically, why the coverage-reporter didn’t merge or deduplicate these files before uploading.

Normally, the coverage-reporter CLI should deduplicate repeated files across multiple coverage reports it discovers in your CI environment when combining them into a single Coveralls JSON upload.

That suggests one of two possibilities:

1. The merge step is not be happening as expected, or
2. The duplication may already exist within a single LCOV file generated in CI. In that case, since there’s only one report to process, coverage-reporter wouldn’t attempt any merge and would simply upload the file as-is.

---

How you can help

If possible, could you share (by saving as a CI artifact) one of your raw LCOV coverage reports from a run where this duplication occurs—ideally from the WINDOWS-LATEST job?

That will help confirm whether:

• the duplicates appear before coverage-reporter converts them, or
• something is slipping through during the conversion and upload process.

Once I can compare both versions (the raw LCOV and the converted JSON), I’ll be able to determine exactly where the duplication starts.

[HebaruSan]

Hi @afinetooth, here is the coverage data that we are generating before the GitHub Action runs:

• coverage-windows-latest-1.zip

I was not sure which files it decided to use, but the log says:

🔍 Detected coverage file: _build\test\coverage\cobertura.xml
🔍 Detected coverage file: _build\test\coverage\lcov.info

[HebaruSan]

I just realized that the previously shared links had a 1-day expiration time. Replaced with permanent reuploads.

[afinetooth]

@HebaruSan I think I may understand the issue and the solution from your sharing this from the log:

🔍 Detected coverage file: _build\test\coverage\cobertura.xml
🔍 Detected coverage file: _build\test\coverage\lcov.info

Issue:

• It appears AltCover was configured to emit coverage reports in two (2) formats;
• coverage-reporter is therefore identifying both coverage reports, of those two (2) different formats (lcov and (Cobertura) XML) in your environment; but
• Since both reports are covering the same coverage run, they are duplicates of one another, just in separate formats, which cannot be merged.

Solution:

• Eliminate duplicate reports in non-mergeable formats by specifying your preferred coverage report format using the format: input option
• You'll probably want to use lcov, which is most standard for C/C++/C#.

Modify these lines in test.yml to include the with: and format: keys, as follows:

# .github/workflows/test.yml

      - name: Report Coveralls
        uses: coverallsapp/github-action@v2
        env:
          COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
        with:
          format: lcov

It should be that simple to filter out the duplicate xml format report(s).

API Token

BTW, at this point you should be able to call the Coveralls github-action without passing the COVERALLS_REPO_TOKEN. So just this should work:

# .github/workflows/test.yml

      - name: Report Coveralls
        uses: coverallsapp/github-action@v2
        with:
          format: lcov

The Coveralls GitHub Action is one of the only Coveralls Integrations that can use the app token of its underlying GitHub App to identify your repo with the Coveralls API, rather than the COVERALLS_REPO_TOKEN.

In fact, the action is configured with a default setting equivalent to:

        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}

Which is nice because it keeps your github-action config less verbose. But if it's working for you with COVERALLS_REPO_TOKEN, feel free to leave it.

[HebaruSan]

Huh. With those suggested changes, it now says "Nothing to report". Maybe the other file is the one with the real data? I'll try that next...

[HebaruSan]

Hi @afinetooth, thanks for the response. I have tried your suggested changes (though I had to use format: cobertura because format: lcov resulted in "Nothing to report"), but they have had no impact on the reported problem. The windows-latest build still has an accurate-ish 84.95% overall coverage, but the parallel build that merges the window-latest build with no other data is at 99% for no reason that I can discern. (And drilling down into the per-file data, I do still see files listed multiple times.)

• https://coveralls.io/jobs/173303052
• https://coveralls.io/builds/76185922

[HebaruSan]

Inspecting the cobertura.xml file, I think the duplicates may be internal to just that one file, on Windows only, due to inconsistent file path separators (/ vs \):

        <class name="CKAN.IO.ModuleInstaller" filename="Core\IO\ModuleInstaller.cs" line-rate="0.86" branch-rate="0.8" complexity="9.96">

        <class name="CKAN.IO.ModuleInstaller" filename="/Core/IO/ModuleInstaller.cs" line-rate="0.87" branch-rate="0.76" complexity="7.06">

However, I do not see how either of those entries could be reported as 100.0% on Coveralls (since there is no 1.0 in line-rate="0.86" branch-rate="0.8" or line-rate="0.87" branch-rate="0.76"):

[HebaruSan]

Checking our previous main branch build (which would have built exclusively on Ubuntu, and doesn't show any duplicates), it also detects _build/test/coverage/coverage.xml:

• https://github.com/KSP-CKAN/CKAN/actions/runs/18723750756/job/53403095984

🔍 Detected coverage file: _build/test/coverage/cobertura.xml
🔍 Detected coverage file: _build/test/coverage/coverage.xml
🔍 Detected coverage file: _build/test/coverage/lcov.info

The Windows build targets three platforms, so it has three files instead of coverage.xml:

Any ideas why none of those three are detected on Windows? Maybe that would help somehow.

[HebaruSan]

To test whether the undetected coverage.xml file was a factor, I used AltCover's API to merge those three files into _build/test/coverage/coverage.xml, which is now detected by the Action, just like the previous working build from the main branch:

• https://github.com/KSP-CKAN/CKAN/actions/runs/18884853051/job/53897657138

🔍 Detected coverage file: _build\test\coverage\cobertura.xml
🔍 Detected coverage file: _build\test\coverage\coverage.xml
🔍 Detected coverage file: _build\test\coverage\lcov.info

There is no noticeable change to the data shown in Coveralls (still very different before/after "merging"):

• https://coveralls.io/jobs/173315254
• https://coveralls.io/builds/76190216

[HebaruSan]

Since cobertura.xml seems to have those duplicated file paths in it, and lcov.info seems to have nothing in it, I tried specifying just the merged coverage.xml in hopes of finding some data that works:

• https://github.com/KSP-CKAN/CKAN/actions/runs/18885891826/job/53901155797

But that takes us back to "Nothing to report":

Same result using just coverage.net481.xml:

It seems that all of the usable data that we've ever uploaded to Coveralls was in cobertura.xml all along, so investigating how AltCover handles path separators in that file on Windows is the main remaining lead...

[HebaruSan]

Just for fun, I tried running an Ubuntu-only "merged" build, and the numbers line up about as I would have expected:

• https://coveralls.io/jobs/173318316

• https://coveralls.io/builds/76191172

That's consistent with the / vs \ problem on Windows being the root cause. I should probably update the issue title and description to clarify that parallel builds are a necessary but not sufficient factor.

Comparing cobertura.xml from each platform side-by-side, it looks like the entries with \ are extra and redundant; the entries with / on Windows have very slightly different numbers but no missing lines as compared to the same entry from Ubuntu.

Parking a command that removes those elements while I look for a way to run it in a Windows container...

hxremove 'class[filename*="\\"]' < cobertura.xml | sed '/^[[:space:]]*$/d' > fixed-cobertura.xml

... easier to do it in Cake:

            // Remove spurious duplicate entries with '\' paths from cobertura.xml
            var cobertura = XDocument.Load(context.Paths.CoverageOutputFile("cobertura.xml").FullPath);
            cobertura.Descendants().Where(elt => elt.Attribute("filename")
                                                    ?.Value.Contains('\\')
                                                    ?? false)
                                   .Remove();
            cobertura.Save(context.Paths.CoverageOutputFile("cobertura.xml").FullPath);

[HebaruSan]

With those nodes removed, we still get multiple entries on Coveralls, but now they both have realistic-ish coverage percentages rather than 100.00%:

My next guess about the duplication was that one comes from the push GitHub event, and the other comes from the pull_request GitHub event. Having added the event name to our flag-names to make sure they don't collide seems to falsify that; only push shows up in Coveralls:

... ahh, interesting: the pull_request builds do exist, but they have a message indicating they've been (correctly) deduplicated! I'll make a note to look into suppressing the redundant uploads.

... the duplicate entries appear finally to have been resolved by putting format: cobertura back in, and the per-line coverage on Windows, Linux, and merged all make sense!

• https://coveralls.io/builds/76193851
• https://coveralls.io/jobs/173326639
• https://coveralls.io/jobs/173326640

[HebaruSan]

Looking closely at the Cobertura entries that confuse Coveralls, not only do they have paths with backslashes, but they also have multiple empty <lines /> elements like this, one under each <method /> and one under each <class />:

        <class name="CKAN.NetKAN.Validators.ModuleManagerDependsValidator" filename="Netkan\Validators\ModuleManagerDependsValidator.cs" line-rate="0.74" branch-rate="0.67" complexity="12">
          <methods>
            <method name="Validate" signature="System.Void (CKAN.NetKAN.Model.Metadata)" line-rate="0.74" branch-rate="0.63" complexity="12">
              <lines />
            </method>
          </methods>
          <lines />
        </class>

That probably calls for an issue to be submitted to AltCover, which I will plan to take on unless someone else would like to volunteer.

The duplicate entries displayed on Coveralls have coverage of 100.0% somehow, despite line-rate and branch-rate having lower values as shown above:

So one way this issue could resolve would be for Coveralls to:

• treat / and \ as equivalent when analyzing file paths and merge such entries together
• fall back to line-rate and/or branch-rate when the <lines /> element is empty, for file- and class-level statistics, rather than treating the class/file as 100% covered (which should not be necessary if, after merging \ entries with the equivalent / entries, <lines /> is no longer empty, as it would not be here)