Amazon AMET Payments Reduced Test Case Generation from Weeks to Hours: How an AI Agent Freed QA Engineers

The Amazon AMET Payments team, responsible for payment systems for 10 million customers across the Middle East and Africa, releases up to five new features monthly. Previously, testing each feature required a week of manual effort from a QA engineer. After implementing an AI agent, this process was reduced to a few hours, significantly freeing up team resources and improving test coverage quality.

In a large company where the speed of new feature releases is critical, manual test case generation becomes a bottleneck. A week of an engineer's work for each feature is not just a direct cost, but also a slowdown in product time-to-market. Moreover, it's monotonous work that doesn't fully utilize a specialist's qualifications and leads to errors. Today, this routine can and should be automated so that people can focus on strategic tasks.

The Pain of Manual Test Case Generation

The AMET Payments team at Amazon processes payments daily for millions of customers in various countries, each with its own regulatory specifics and payment methods. With five new features released monthly, each requiring thorough testing, manual test case generation became a full-time job for one QA engineer for a year. This specialist spent weeks analyzing Business Requirement Documents (BRDs), design documents, UI mocks, and historical data to create an adequate test plan. This not only slowed down the development cycle but also diverted valuable engineering resources from more strategic tasks.

The goal was not just to speed up the process, but to make it more qualitative, standardized, and less prone to human error. At the same time, it was necessary to retain the institutional knowledge of experienced testers and minimize problems associated with AI "hallucinations."

Why Traditional AI Approaches Failed

Initial attempts at automation using AI involved feeding entire BRD documentation to a single AI agent. The result was unsatisfactory: the agent produced overly general formulations, such as "verify payment works correctly." The team, however, required highly specific and detailed test cases, for example: "verify that when a UAE customer selects cash on delivery (COD) for an order above 1,000 AED with a saved credit card, the system displays the COD fee of 11 AED and processes the payment through the COD gateway with order state transitioning to 'pending delivery.'"

Context length limitations, the absence of specialized processing phases, and the propensity for hallucinations made this approach ineffective. The AI attempted to compress complex business logic without the iterative thinking process that experienced testers employ.

How a Human-Centric Approach Led to a Breakthrough

The breakthrough occurred when the team shifted its question: instead of "how should AI think about testing?", they asked "how do experienced humans think about testing?". Research into the cognitive processes of senior QA professionals revealed that testers do not process documents holistically. They go through specialized mental phases: first, they analyze by extracting acceptance criteria, identifying customer journeys, understanding UX requirements, and then develop tests through a systematic process: journey analysis, scenario identification, data flow mapping, test case development, and finally, organization and prioritization.

This led to the creation of SAARAM, a multi-agent AI solution that mimics these expert approaches. Each agent focuses on a specific aspect of the testing process, as if a human expert mentally compartmentalized different analysis phases.

AI Agent Design and Architecture

SAARAM was designed as a complex multi-agent system. Initially, the team tried to build agents from scratch, but then switched to using an off-the-shelf SDK for orchestration, which allowed them to coordinate complex, interdependent tasks and reduce development time.

The first iteration of SAARAM included five specialized agents:

• Agent 1: Customer Segment Creator. Focuses on customer segmentation analysis, using four subagents for segment identification, parameter matrix creation, end-to-end scenario development, and detailed test case generation.
• Agent 2: User Journey Mapper. Employs four subagents for comprehensive user journey mapping, including flow diagrams, sequence diagrams, end-to-end scenarios, and detailed test steps.
• Agent 3: Customer Segment x Journey Coverage. Combines inputs from Agents 1 and 2 to create detailed segment-specific analysis.
• Agent 4: State Transition Agent. Analyzes various product states in user journey flows, creating state diagrams and generating test scenarios.

The second iteration of SAARAM featured a completely reimagined architecture with an emphasis on modularity, context awareness, and extensibility:

• Agent 1: Intelligent Gateway. Determines file type (documentation, Figma, code repositories) and routes data to appropriate agents.
• Agent 2: Specialized Data Extractor. Uses six specialized subagents for parallel data extraction from various domains.
• Agent 3: Visualization Agent. Transforms extracted data into six different diagram types (ERD, flow diagrams, requirements, UX, process, mind maps), providing various perspectives for analysis.
• Agent 4: Data Condenser Agent. Performs intelligent context distillation, ensuring that each subsequent agent receives only the necessary information.

Implementation Results

The main outcome was the reduction in test case generation time from one week to a few hours. This allowed Amazon AMET Payments to significantly accelerate the release cycle of new features, free up valuable QA engineer resources for more complex and strategic tasks, and standardize testing approaches across all teams. The solution is already being scaled within AMET QA and is planned for implementation in other QA teams within the International Emerging Stores and Payments (IESP) Org.

How to Implement This in Your Company

The Amazon AMET Payments case demonstrates that even in the most complex and high-load systems, processes can be significantly improved with AI agents. If your company has routine but critically important tasks that consume a lot of time from highly qualified specialists, it's worth considering the implementation of AI agents. Here's where to start:

• Identify routine processes. Find tasks that are performed manually, require analysis of large volumes of data, and are repeated with high frequency.
• Study the cognitive patterns of your experts. Instead of trying to teach AI to "think," model its work based on how your best specialists solve problems.
• Break down complex tasks into subtasks for a multi-agent system. Create specialized agents, each responsible for a specific stage of the process, mimicking the work of an expert.
• Start small and iterate. Implement agents in stages, starting with the least risky and most obvious areas for automation, gradually expanding functionality and scaling the solution.

If this case sounds like what's happening in your company, our manager can help: he'll analyze your business and niche for free and point out where an AI agent would bring a real result in your case. Message the manager