From 90 to 12 Minutes: How We Accelerated a .NET Pipeline
Published · 5 min read · Tomasz Dłuski
The client’s code changes went through pipelines whose full check had a p99 of an hour and a half. After we reorganized the .NET build, that figure fell to 12 minutes. The median dropped from 40 minutes to 11 minutes and 30 seconds. The change was to build around 200 C# projects in one solution and move file copying to the end of the process.
The full check: from job start to result
| Percentile | Before | After |
|---|---|---|
| p99 | 90 min | 12 min |
| p50 · median | 40 min | 11 min 30 s |
| p1 | 25 min | 10 min 10 s |
p99 is the time by which roughly 99% of runs have finished. It is not the maximum: individual runs can take longer. p50 is the median, the middle of the distribution. p1 describes the fast end: roughly 1% of runs finish at or below that time. Together, these figures show the improvement in both the typical run and the slower end of the distribution.
The drop from 90 to 12 minutes makes the p99 figure 7.5 times faster. The median provides a more everyday reference: a typical check now takes about eleven and a half minutes. Every value in the table measures job execution. Waiting for an available agent is a separate matter and is not included in these numbers.
The disk was busy. More builds were waiting.
The repository contained around 200 .csproj files, but the pipeline built them through multiple separate .sln solutions, invoked one after another. Each invocation saw its own slice of the system. Shared dependencies appeared across successive runs, and the number of file-copy operations reached around 14,000.
Multiple Azure DevOps agents ran on a single virtual machine. Their jobs shared storage resources, so one build’s heavy file reads and writes slowed the others down too. This caused I/O congestion. Increasing parallelism alone did not remove the repeated copying that was putting pressure on the whole machine.
The effect compounded: running jobs competed for I/O, occupied agents for longer, and left subsequent jobs waiting in the queue. PR authors received results later even when their own builds were not the source of the excessive copying.
Same code, a different build flow
Simplified flow of a single build. Letters stand for example solutions and projects.
Before
One solution after another
Solution A
Post-build: copy DLLs
to a shared folder
Solution B
Post-build: copy DLLs
to a shared folder
Solution C
Post-build: copy DLLs
to a shared folder
Around 14,000 file-copy operations
Repeated copying across successive builds.
After
One shared dependency graph
One .sln file
around 200 .csproj projects
- Shared dependency
Independent projects in parallel
- ProjectA
- ProjectB
- ProjectC
Entire solution built
One final copy stage
only the required files
One solution, with dependencies visible to MSBuild
We brought the projects into one solution and let MSBuild handle their build order. When one project depends on another, the engine must respect that dependency. When several projects can be built independently, their builds can run in parallel. There is no longer a need to schedule entire solutions manually and restart the process for their overlapping parts.
Execution settings matter as well. MSBuild supports multiple worker processes through -maxcpucount, also written as -m. That lets it use multiple cores to build projects in parallel. Moving .csproj files into a single .sln does not automatically enable this capability. The dependency graph and appropriately configured multiprocessing need to work together.
Build everything first. Copy afterward.
The legacy projects had post-build actions that copied DLLs into a shared folder. Finishing one project therefore triggered another round of file operations while the rest of the solution was still building. Those operations repeated the same work and complicated parallel execution: several projects could attempt to write shared files.
We centrally disabled or overrode those steps through a shared .targets file. MSBuild allows target definitions to be overridden, and import order matters: the last definition with a given name is used. This makes it possible to configure unnecessary copying in one place instead of maintaining similar changes separately across hundreds of projects.
The required copying became a single step after the whole solution had built successfully. That distinction matters: an AfterBuild hook attached to a project belongs to that project. It is not a global signal that everything has finished. The final step must wait for the entire solution before collecting the required files. In our process, that meant doing this work once.
The build itself: under two minutes on average
Before: 20-40 min
Now: < 2 min on average
The same machine. Usually within 3 minutes.
We measured the build stage separately. On the same machine, it previously took between 20 and 40 minutes. After the change, the average is under two minutes, and the build usually finishes within three. The machine could use multiple cores, while the disk stopped handling the repeated copying. This measurement covers a single build. The full check in the opening table includes two builds, the second for the client’s analysis tool, as well as the remaining pipeline steps.
Less copying sped up the other jobs too
Speeding up a single build of the shared solution benefited all builds running on the same VM. Reducing the copying eased I/O contention. More jobs could make progress concurrently, and the other jobs stopped having runs stretch towards an hour and a half because of that congestion.
This is a second level of improvement: within a build, MSBuild uses multiple cores; across jobs, agents have less congested storage resources available. Optimizing one process also shortened the execution time of its neighbours on the machine.
We suspect antivirus scanning added to the cost of thousands of file-copy operations. That remains a hypothesis, not a confirmed cause. The observed benefit of reducing copying extended beyond a faster individual build: the whole shared VM ran better.
Two builds with full analysis, still twice as fast
The pipeline with analysis previously checked only selected projects. Each one was built separately, along with the dependency tree needed to check it. Checking a smaller scope did not make the run quick: the build steps themselves took too long, and shared parts of the dependency tree came up again for successive projects.
We used the same new solution covering the whole system. First, we build it once. Then we build it again for the analysis tool the client uses. The pipeline runs two builds of the shared .sln file and a full analysis, expanding coverage beyond the selected projects previously checked through separate build invocations.
Even with that second build, the entire pipeline with full analysis takes about half the time of the previous single pass that built selected projects separately with their dependencies. We check more and get the result sooner. Execution time also depends on how much work is repeated across projects and whether the build engine can use a shared dependency graph.
How did we deliver it? Dev and ops working together
We delivered the change because developers took ownership of the build as part of the codebase. The problem lay in project definitions, dependencies, and the company’s build logic. Agent configuration alone could not resolve it.
The DevOps team needed to work with developers who knew those projects. Combining knowledge of the code with knowledge of the CI environment made it possible to remove unnecessary work at its source and relieve pressure on the shared machine. This improvement came from dev and ops working together.