MIPT "Pusk" Shares Experience: How AI Agents Were Implemented and Mistakes Avoided, Saving 30% Testing Time

Implementing AI agents into business processes is no longer science fiction but a pragmatic reality. Denis Prilepsky, an expert from the MIPT "Pusk" Center for online master's programs, analyzed the experience of implementing such solutions in highly regulated industries, identifying key mistakes and best practices. For example, when testing time was reduced by 30%, and the proportion of tasks resolved on first contact increased from 60% to 85%, it not only broke management's skepticism but also demonstrated the correct approach to AI projects.

Many companies still approach AI agent implementation as a magic wand that will solve all problems on its own. However, without clear goals, data preparation, and an understanding of risks, such projects are doomed to failure, becoming an endless experiment that eats up budget and time. This is not "just trying AI"; it is an engineering task that can and must be solved systematically.

The Reality of the Problem: Why an AI Agent, Not Just Automation

Implementing new technologies is always a challenge, and with AI agents, it is particularly acute. Many executives in 2023–2024 mistakenly believed that it was enough to connect a powerful model, and it would "figure everything out on its own." They underestimated the need for data preparation, processes, and clear evaluation criteria. Without a clear business case and KPIs, the agent either solved the wrong problem or its value remained unclear to the business. This led to pilots becoming endless experiments without real implementation, and skepticism grew.

Unlike traditional automation, which operates on strictly defined scripts, an AI agent can adapt and make decisions in situations that go beyond rigid rules. But this requires not only technological readiness but also organizational maturity, an understanding that AI is not magic, but a tool that requires configuration and control.

The Path to the AI Agent: From Demonstrations to Engineering Responsibility

The early stages of AI agent development often resembled impressive demonstrations, far removed from real business problems. Companies, inspired by the possibilities, rushed into pilots without a clear understanding of what problem they wanted to solve and how they would measure success. This led to inflated expectations and disappointments. A shift was needed from "what's possible?" to "what can we actually implement and maintain?" It was at this point that it became clear that AI agents require an engineering approach, focusing on operationalization and integration into existing processes.

MIPT "Pusk" experts observed that successful projects start not with the model, but with goals and metrics. For example, instead of an abstract "try AI," the goal was to reduce incident processing time by N% or automate X% of operations of a certain type. This approach allowed not only to quickly prove the value of the solution but also to gain stakeholder support for further investment.

How the AI Agent Was Designed: Focus on Data, Security, and People

When designing the AI agent, MIPT "Pusk" identified several key aspects that formed the basis for successful implementation:

• Clear Goals and Metrics. Each project began with a specific goal, such as reducing testing time by 30% or increasing the proportion of tasks resolved on first contact to 85%. This allowed for ROI measurement and value demonstration.
• Data Quality. Since the agent is highly dependent on data, knowledge bases were reviewed, cleaned, and regularly updated. Input filtering mechanisms were implemented to prevent "context poisoning" and ensure the reliability of the agent's decisions.
• Multi-Level Security. Given that the agent can have access to dozens of systems, multi-level guardrails were developed. Agent permissions were minimized, and requests to external systems passed through a centralized gateway with whitelists of allowed actions and limits. All actions were logged and traceable.
• Human Factor Management. An important part of the design was involving end-users. The goals of implementation were explained, how the agent would make their work easier (removing routine, providing new opportunities) was shown, and it was emphasized that control always remains with humans.

Implementation: From Pilot to Scaling

The implementation of AI agents proceeded in stages, starting with the most massive and predictable tasks. This allowed for minimizing risks and quickly achieving initial positive results. For example, a pilot project to reduce testing time showed a 30% saving, which became a convincing argument for scaling.

Special attention was paid to monitoring and support after launch. A regime of continuous monitoring of quality metrics, analysis of agent logs, and regular updating of its knowledge and prompts was established. The team met to analyze errors and adjust settings, ensuring continuous improvement of the system.

Results and Prospects

MIPT "Pusk"'s experience shows that with the right approach, AI agents can bring tangible benefits. However, Gartner warns that over 40% of AI agent projects will be closed by 2027 due to rising costs, unclear business value, or lack of risk control. This highlights the importance of realistic expectations and competent project management.

Despite the difficulties of the first few years, the long-term potential is enormous. Gartner predicts that by 2028, about 15% of daily work decisions will be made by autonomous AI agents, and up to 1/3 of corporate software will include agent components. The concept of digital colleagues remains attractive, promising a revolution in productivity and a shift from automation to autonomy.

How to Implement This in Your Company

To successfully implement AI agents in your company, follow these recommendations:

• Define a specific business case. Choose a task with a repeatable process, a tangible pain point that can be eliminated, and a way to measure success.
• Ensure data quality. Review knowledge bases, clean "junk," and set up regular data updates and validation.
• Develop a multi-level security system. Minimize agent permissions, pass requests through a centralized gateway with whitelists and limits, and log all actions.
• Pay attention to the human factor. Clearly explain the goals of implementation to the team, show how the agent will make their work easier, and involve end-users in the process.
• Establish a regime of continuous monitoring and support. Assign responsibility for monitoring metrics, analyzing agent logs, and regularly updating the agent's knowledge.

If this case sounds like what's happening in your company, our manager can help: he'll analyze your business and niche for free and point out where an AI agent would bring a real result in your case. Message the manager