Sherry Shi's Research Explores AI Agents in Software Engineering

Sherry Shi, a researcher at the forefront of artificial intelligence and software engineering, is making significant contributions to the evaluation and integration of AI agents within software development processes. Her recent body of work, detailed in several new publications, centers on enhancing the reliability and collaborative potential of AI systems in complex software engineering tasks, particularly in the realm of automated program repair (APR).

Advancing Automated Program Repair with LLM-as-a-Judge

A key area of Shi's research addresses the critical need for reliable patch evaluation in automated program repair. Traditional methods, such as pass@k, often fall short of capturing the true quality of a software patch, creating a gap between automated metrics and real-world validity. To bridge this divide, Shi and her colleagues have developed an LLM-as-a-Judge framework. This innovative approach leverages large language models to assess patch validity at scale, incorporating a human-in-the-loop workflow. In this system, an LLM initially generates a rubric for evaluating a given bug, which a human expert then refines into a "golden" standard. This refined rubric is subsequently used by the LLM judge to evaluate candidate patches, demonstrating substantial agreement with human developer consensus on a set of 48 bugs and 115 patches. This methodology offers a scalable and high-fidelity signal for measuring progress in APR.

Reducing Noise in Agentic Program Repair

Furthering the application of AI in software development, another publication introduces a dual-LLM policy designed to reduce noise in agentic program repair. Agentic APR systems are increasingly employed for complex, repository-level bugs, but human review of generated changes remains essential. Presenting developers with unlikely or incorrect patches can lead to wasted time and diminished trust. To combat this, the proposed policies include bug abstention, which filters out bugs the agent is unlikely to fix, and a patch critic, which rejects patches deemed unlikely to be good fixes. Evaluations on bugs from Google's codebase showed that these policies can significantly raise filtered success rates, improving them by up to 13 percentage points for abstention and 15 percentage points for the critic, with a combined improvement of up to 29 percentage points. This work directly addresses the practical challenges of integrating AI-generated code into existing development pipelines.

Defining AI Agent Behavior in Software Engineering

Beyond specific repair tasks, Shi's research also delves into the broader integration of AI as a collaborative partner in software engineering. Two related publications present a taxonomy of desirable AI agent behaviors in software engineering. As LLMs transition from simple code generators to agentic partners, defining and measuring their success requires a new paradigm. This taxonomy, synthesized from developer-defined rules and validated through interviews with professional developers, identifies four core expectations: Adhere to Standards and Processes, Ensure Code Quality and Reliability, Solve Problems Effectively, and Collaborate with the Developer. This framework provides a concrete vocabulary for aligning AI agent behavior with developer preferences, moving beyond simple correctness metrics to encompass the socio-technical realities of professional software development.

Dynamic Cogeneration of Bug Reproduction Tests

In a related effort, Shi's team has explored the dynamic cogeneration of bug reproduction tests (BRTs) within agentic program repair. BRTs are crucial for validating fixes and aiding generation, but canonical APR systems often generate them separately from the fix. Recognizing that developers desire BRTs alongside AI-generated patches, this research investigates strategies where the APR agent generates both a fix and a BRT concurrently. Evaluations on 120 human-reported bugs at Google demonstrated that this cogeneration approach allows agents to generate BRTs for a substantial number of bugs without compromising the generation rate of plausible fixes. This integrated approach reduces engineering effort and enhances the confidence developers have in AI-generated code changes.

The Future of AI in Collaborative Software Development

Sherry Shi's research collectively paints a picture of a future where AI agents are not just tools but trusted collaborators in the software development lifecycle. By focusing on rigorous evaluation methodologies, noise reduction techniques, and clear behavioral expectations, her work aims to accelerate the adoption of AI in professional software engineering. The emphasis on human-centered design and practical integration challenges suggests a path toward more reliable, efficient, and trustworthy AI-assisted software development. The ongoing exploration of these themes promises to shape how software is built and maintained in the coming years, fostering a more synergistic relationship between human developers and artificial intelligence.