Microsoft Introduces Open-Source Agent for Polyglot Unit Test Generation
Microsoft has released an open-source testing agent that researches repositories to generate, build, and validate unit tests across multiple programming languages.
Microsoft's new polyglot agent researches existing repositories to ensure generated unit tests match local conventions and pass CI build checks.
- Microsoft released code-testing-generator, an open-source polyglot agent that automates unit test creation.
- The agent performs repository research to identify test frameworks, build commands, and local coding conventions.
- Benchmark data shows a 92.1% completion rate, significantly outperforming standard LLM prompts in reliability and validation.
Automated Unit Testing Through Repository Context
Microsoft has announced the release of code-testing-generator, an open-source, polyglot unit-test agent designed to bridge the gap between AI-generated code and production-ready test suites. Part of the dotnet-test plugin within the dotnet/skills repository, this specialized agent is engineered to handle the complexities of unit testing that generic Large Language Model (LLM) prompts often miss, such as identifying existing frameworks, mocking dependencies, and ensuring tests integrate with continuous integration (CI) pipelines.
The primary challenge addressed by this agent is the ambiguity of brief user prompts. When a developer asks an AI to "generate unit tests," several critical questions remain unanswered: the location of the tests, the specific testing framework in use, how the build system discovers tests, and which external dependencies must be mocked. The specialized agent addresses these by researching the repository before writing any code.
The Four-Stage Testing Workflow
The agent operates through a structured workflow that prioritizes repository context and verification over immediate code generation. This process ensures that the resulting tests are not only syntactically correct but also functional within the project's specific environment.
1. Repository Discovery and Research
Before implementing tests, the agent scans the repository to detect the programming language and the specific unit-testing framework employed by the team. It analyzes existing tests to understand local conventions, such as naming patterns and file structures. Crucially, it identifies the exact commands required to build the project and execute the test suite. This discovery phase prevents a common failure mode where new test projects are created but never actually run by the repository's CI/CD workflows.
2. Planning and Work Estimation
The agent does not apply a one-size-fits-all approach to test generation. Depending on the complexity of the request, it selects one of three execution paths: direct implementation for simple methods, a single-pass research-and-plan phase for standard tasks, or an iterative cycle for covering large modules or reaching specific coverage targets. This ensures that larger requests involve mapping behaviors to specific test files and prioritizing code with higher dependency counts.
3. Implementation and Isolation
During the writing phase, the agent isolates the code under test by mocking external services and outside dependencies. It is restricted from modifying production code during the test generation process. To maintain test reliability, the agent is programmed to avoid creating unit tests that depend on external URLs, open ports, or precise timing. If the generated tests fail to compile, the agent reads the compiler errors and self-corrects the code.
4. Verification and Mutation Testing
A central feature of the agent is its validation suite. Once tests are written, the agent checks for "weak" assertions—those that might pass even if the logic is flawed. It performs a lightweight form of mutation testing by considering small code changes that should cause the tests to fail. Finally, it builds the entire workspace and runs the full test suite to confirm that the new tests are discovered by the repository's standard test commands.
Benchmark Performance and Reliability
Microsoft conducted benchmarks using an internal unit-testing suite consisting of 152 tasks from real-world repositories. The results compared the specialized agent against "stock" versions of GitHub Copilot and Claude Code using the same underlying models. The specialized agent achieved a 92.1% completion rate (140 out of 152 tasks), compared to 78.9% for stock Copilot. This represents a 63% reduction in failures.
The performance gains were most pronounced in what the researchers termed "vague prompts"—requests that provided minimal detail. In these scenarios, the specialized agent passed 88.8% of tasks, while stock Copilot passed only 66.3%. This suggests that the agent's ability to research the repository compensates for lack of detail in user instructions. Additionally, in tasks involving specific code diffs, the agent maintained a 100% success rate, whereas stock Copilot failed to complete those specific tasks.
Efficiency and Language Support
While the agent generated approximately 2.3% fewer tests than the baseline, it maintained nearly identical line coverage (~72.4%) and branch coverage (~49.8%). Notably, the agent was 5.5% faster on average, completing tasks in 359 seconds compared to 380 seconds for stock Copilot. This indicates that the agent's value lies in its reliability and the utility of the tests it produces rather than the sheer volume of code generated.
The benchmark included 45 specific .NET tasks, but the agent's polyglot nature allows it to operate across various languages and frameworks supported by the underlying LLM. By automating the research, planning, and validation phases, the tool aims to turn short developer prompts into trusted, verified test suites that adhere to established repository standards.
Enjoyed this?
Get more posts like this delivered to your inbox.
