Yandex Accelerated QA by 30%: How LLM Agents Transformed Testing

Yandex implemented LLMs in its testing process, leading to a 30% acceleration in automated test writing and daily generation of hundreds of checklists. Now, QA engineers spend half the time on routine tasks, focusing on more complex and creative challenges.

If your QA team is drowning in routine and wasting time on repetitive tasks, message our manager — he will run a free analysis of your business and niche and show exactly how to get a real business result from an AI agent in your case, not a nice-looking picture. Message the manager

Where it all began: Testing as an ideal proving ground for AI

Yandex had long considered implementing generative neural networks. The QA domain, with its clear structure, repetitive patterns, and high granularity of tasks, proved to be an ideal candidate for experimentation. Early MVPs, created across various departments, demonstrated that LLM agents could successfully generate simple automated tests and decent checklists.

However, scaling revealed a problem: the quality of AI work plummeted when moving beyond narrow scenarios. What worked well for one engineer was unsuitable for a team of 15, let alone a thousand testers.

How they tackled the “zoo” of technologies and achieved scale

The rapid emergence of AI prototypes led to a “zoo” of technologies, where each team created its own AI agents. This created issues with support, standardization, and a lack of common quality metrics. To avoid administrative pressure and maintain team motivation, Yandex adopted a compromise solution:

• The central team took responsibility for data infrastructure and quality measurement.
• Local teams were given freedom to experiment with prompts and agents.

A key element was the Test Management System (TMS), integrated with AI tools. TMS became the central point for orchestrating all AI use cases in testing, ensuring seamless operation and control.

Results of AI Agent Implementation

Checklist and Test Case Generation

AI agents, integrated with TMS, task trackers, repositories, and internal Wikis, generate over 200 checklists daily. This reduces the time spent on their creation by an average of 50%.

To ensure quality, the LLM-As-A-Judge approach is used: one model checks the results of another against a set of criteria, comparing generated cases with reference ones created by experienced testers. This allowed for the creation of a flexible system where specialized models are used for complex scenarios, and a continuously evolving baseline model, maintained by the central team, handles the rest.

E2E Automated Test Generation

The integration of an AI code assistant (Yandex Code Assistant) significantly accelerated the writing of E2E automated tests. Following specialized training sessions conducted by the central team, the active use of AI in teams increased from 30% to 60%, and code writing speed rose by an average of 30%.

AI Agent for Manual Test Execution

This direction is the most knowledge-intensive and potentially profitable. Yandex uses its own AI agent, which is being fine-tuned for functional testing in web and mobile applications. Despite the current accuracy of 45% (with a target of 80%), in the future, this will allow for:

• running most regression tests without human involvement;
• shortening the feedback loop and development cycle for new features;
• freeing QA engineers from routine tasks to focus on incident analysis, exploratory testing, and improving quality strategy.

Source: habr.com

Want to learn how to integrate AI agents into your testing process and achieve similar results? Message the manager. We will analyze your case for free and show you where the growth points lie.