Amazon Payments Reduced Test Case Generation from Weeks to Hours: How AI Agents Transformed QA Processes

The Amazon Payments AMET (Africa, Middle East, and Türkiye) team serves approximately 10 million customers monthly. Each month, they launch an average of five new features, and each requires comprehensive test case generation. Previously, this process consumed up to one week of manual effort per project, equivalent to one full-time QA engineer annually just for test creation. After implementing a multi-agent AI solution, the time required for test case generation was reduced to mere hours, while significantly improving test coverage quality.

In large companies like Amazon, routine operations aren't just extra costs; they represent a colossal loss of strategic potential. Employees whose job is to ensure quality spend weeks on monotonous test preparation instead of focusing on complex, non-trivial tasks. This slows down new product releases, increases risks, and deprives the business of a competitive edge. Today, there's a tool that allows this burden to be shifted to AI, freeing people for more valuable work.

The Hidden Cost of Manual Test Generation at Amazon Payments

The AMET Payments team is responsible for payment functionality for millions of customers across five countries, with diverse regulatory environments and numerous payment methods. In such a complex environment, any new feature requires exhaustive testing. Traditionally, QA engineers spent an entire week on each project, analyzing Business Requirement Documents (BRDs), design documents, UI mocks, and historical test preparations. This was a labor-intensive process that not only slowed down the product release cycle but also diverted valuable engineering resources from more strategic tasks.

The problem wasn't just about lost time. Manual test generation was prone to human error: omissions, incomplete coverage, and subjectivity. In a context involving financial transactions, any error can cost millions and erode customer trust. A method was needed not just to accelerate, but also to systematize, standardize, and improve the quality of this process.

Why Traditional AI Approaches Failed

Initial attempts to use AI for test generation followed standard approaches: the entire BRD was fed to a single AI agent. The results were disappointing. The agent often produced generic statements like "verify payment works correctly," instead of specific, actionable test cases. For example, a test like "verify that when a UAE customer selects cash on delivery (COD) for an order above 1,000 AED with a saved credit card, the system displays the COD fee of 11 AED and processes the payment through the COD gateway with order state transitioning to ‘pending delivery’" was needed. A single agent could not handle such detail.

The limitations were clear: context length restrictions prevented effective processing of large documents, and the lack of specialized processing phases meant the AI couldn't understand testing priorities or risk-based approaches. "Hallucination" issues led to the creation of irrelevant scenarios. The team realized that AI was attempting to compress complex business logic without the iterative thought process that experienced testers employ.

The Breakthrough: A Human-Centric Approach to AI Agent Design

The pivotal moment came when the Amazon Payments team shifted their question: instead of "How should AI think about testing?" they asked, "How do experienced humans think about testing?" This led to a detailed study of the cognitive processes of senior QA professionals. It was discovered that experienced testers do not process documents holistically but work through specialized mental phases: first, they extract acceptance criteria, identify customer journeys, understand UX requirements, map product requirements, and analyze data. Then, they develop tests through a systematic process: journey analysis, scenario identification, data flow mapping, test case development, and finally, organization and prioritization.

This insight became the foundation for designing SAARAM (QA Lifecycle App) — a multi-agent AI solution. The agent was built as an orchestrator, where each specialized sub-agent mirrored a distinct phase of a human tester's thought process. This architecture allowed the AI not just to generate tests, but to "think" like an expert, decomposing complex tasks into sequential, manageable actions.

Implementation and Evolution of the Multi-Agent Architecture

The first iteration of SAARAM involved five specialized agents for end-to-end test generation. For example, the "Customer Segment Creator" agent had four sub-agents for segment discovery, decision matrix generation, end-to-end scenario creation, and detailed test step development. Other agents handled user journey mapping, combining segments, and analyzing product states.

However, during operation, limitations emerged: agent isolation led to limited context and increased hallucinations, inefficient data generation, narrow parsing capabilities (only Word documents), and a rigid architecture. This necessitated a complete redesign.

The second iteration of SAARAM was rebuilt from the ground up on principles of modularity, context-awareness, and extensibility. An "Intelligent Gateway" emerged, routing data from various sources (documents, Figma, code repositories) to specialized agents. The "Data Extractor" agent used six sub-agents for parallel, domain-specific data extraction. The "Visualizer" agent transformed extracted data into six types of diagrams (ER diagrams, flowcharts, requirement diagrams, etc.) that helped humans and other agents understand complex relationships. The "Data Condenser" agent synthesized information, providing each subsequent agent with only the necessary context.

This iterative approach, based on a deep understanding of the human workflow, allowed for the creation of a flexible and powerful solution that is continuously improving.

Results

The implementation of AI agents at Amazon Payments led to a reduction in test case generation time from one week to mere hours. This is equivalent to freeing up one full-time QA engineer per year, who was previously solely occupied with test creation. These resources can now be directed towards more complex and strategic aspects of quality assurance, such as risk analysis, exploratory testing, and the development of innovative approaches.

Furthermore, the quality and detail of test coverage significantly improved, which is critical for a sensitive area like payments. The SAARAM system does not replace humans; rather, it enhances their expertise, allowing specialists to focus on decision-making rather than routine data collection and formatting.

How to Implement This in Your Company

The Amazon Payments case demonstrates that for effective AI agent implementation, the key is not just technology, but also a deep understanding of human cognitive processes. If your company has routine yet multi-stage processes requiring the analysis of large volumes of information, this approach can be a breakthrough:

• Study how your best specialists work. Decompose their thought processes into distinct, sequential steps. These steps will form the foundation of your multi-agent architecture.
• Divide the AI agent into specialized modules. Instead of one "universal" agent, create a set of highly specialized sub-agents, each responsible for its stage of information processing or decision-making.
• Ensure transparent context exchange between agents. Solve the problem of agent "isolation" by creating mechanisms that allow each module to receive only the necessary and already processed information from previous stages.
• Start with a "smart gateway." Create an entry point that will classify and route incoming data to the appropriate specialized agents. This will improve efficiency and accuracy.
• Implement iteratively. Don't try to automate everything at once. Start with the least risky but most labor-intensive stages, gradually expanding functionality and improving the architecture based on feedback.

If this case sounds like what's happening in your company, our manager can help: he'll analyze your business and niche for free and point out where an AI agent would bring a real result in your case. Message the manager