Test Automation with AI and GPT: How to Avoid Wasting Weeks on Bugs

Over the last 11 years in this industry, I've learned that while you check each line of code & write a test plan to find bugs in your code, your competition has typically released two or three versions of their software and received feedback from real users. For example, in May 2023 when we needed to test a crypto platform prior to its listing on Binance, we had to run a battery of tests 48 hours before the listing date. Since 5 QA Engineers couldn’t test all of the test cases due to the physical limit of how much one person can do in 48 hours, we used an AI agent to generate test cases for such a large project for the first time. We ran 2800 tests overnight and found 47 major bugs that would not have been found until 1 week after releasing our product; therefore, automated testing has evolved beyond just an automation tool into a strategic asset.

The shift toward AI Test Automation

As the software testing industry is changing, companies putting AI into their QA processes are reducing release cycles by 40-60% and achieving greater than 85% test coverage. Traditional manual script writing for Selenium or Playwright cannot meet the demands of rapidly changing Agile technologies, and companies are rapidly embracing AI because of these two factors:

• The most valuable feature of machine learning is its ability to build adaptable auto-testing systems. By using self-modifying and self-monitoring techniques, it eliminates manual refactoring—the leading cause of UI test fragility.
• Operating full-time means that there will be many thousands of tests running simultaneously (human test professionals could never accomplish this).

In cryptocurrency development, where time is critical to project success; if there are delays or mistakes in completing all of the necessary testing for production, you will lose money. During the flash crash of the Solana DEX in October 2024, our AI completed regression testing of all major modules in 12 minutes and identified no defects related to our systems, where a manual check from a person would have taken 4 to 6 hours, at which point user confidence would have been lost.

AI differs from traditional test automation because it is not a set of "if X then Y" rules; AI utilizes neural networks to learn how to recognize and identify patterns and identify anomalies and recommend the proper action. We have transitioned from "robotic executors" to "robotic analysts," where robots can impactfully evaluate a given task using their electronic brains.

Automated Testing and Artificial Intelligence are fundamentally different

Automated testing is a type of automation that enables one software application to test another software application; these tests are typically performed with scripts, which include actions the automated test will take, such as starting a web application, entering the user's ID and password, clicking buttons, and ensuring that the user is taken to the home page after logging in. All of these types of tests save a considerable amount of time compared to doing it manually.

AI supplies an element of intelligence. The system adapts and does more than perform pre-written actions or operate with known actions. A standard test (which uses a UI component) will fail if the "Login" button on the application was changed to "Sign in", while an AI solution will be able to find the same element based on the context. For the autotest generation process, the neural network serves three functions:

• Test Case Creator: creates detailed scenarios for testing based on functional description and considers all cases: positive, negative and boundary cases.
• Code Scanner: GPT reviews all functions for writing unit tests to cover all branches and exceptions.
• Error Prediction: Uses previous history to determine where an error is likely to be in the next release.

There are two primary approaches to automating with AI:

• Rule-based automation: the traditional way of creating tests manually.
• AI-augmented automation: setting a goal of "test the payment page" and allowing the AI to automatically create both a test plan and results analysis.

Hybrid solutions incorporate both rule-based and AI-generated testing, where there will be a combination of manually written tests and then AI automatic tests for the balance.

The following example demonstrates the use of Playwright and GPT-4 AI at ASCN.AI to test blockchain APIs. The system generates both positive and negative data for requests made to an API endpoint, analyzes the response and captures logs. Testing new endpoints has a time requirement of less than 30 seconds, with developers only performing verification and execution after the test.

It is important to realize that the tester is not replaced by AI, but instead, AI removes mundane activities and assures consistent results. The focus of QA work is now on identifying a strategy, priorities and assessing risk to the business.

How do you automate testing?

We will first conduct a manual audit of what tests and processes are mostly performed manually, which would usually be regression tests — checking that all previously working functions still work as expected. Typically, this is where you will realize the greatest amount of benefit through automation.

The steps to be taken are as follows:

1. Identify the high-priority scenarios based on how frequently they occur and how stable and critical they are.
2. Install various test automation tools such as Selenium or Playwright for web apps; Postman and Newman for API testing; and Appium for mobile applications.
3. Create your initial automated test — a smoke test verifying the main functions of the application.
4. Integrate your automated test(s) into your Continuous Integration/Continuous Delivery pipelines.

With AI, you can have even more ease in achieving test automation. You simply write your requirements in everyday human terms and GPT will automatically generate the code needed for the test. Further, you can have 20 to 30 simple test cases generated in less than one hour, as opposed to several days to manually execute all of the tests.

As an example, at ASCN.AI we recently used this method to build an Arbitrage Trading Scanner, where the exchange inputs are changed every time due to frequent API updates by those exchanges. By using an AI agent to read the documentation, create an API request, and check the response from the API based on what the expectations were before making the request, we have managed to decrease the number of critical failures from 12 per month down to 2 per month.

It's also worth noting that exploratory testing is done manually to find hidden defects, which may include things like opening or closing 50 browser tabs or entering an emoji in a numeric entry field. While AI is able to identify common examples, humans will continue to test for failures outside of expected parameters.

According to Google's testing pyramid, recommended levels of test coverage include 60% unit, 30% integration and 10% end-to-end testing.

Selenium Test Automation

When it comes to browser automation frameworks, Selenium is probably the most popular and historical. It's been around a long time, works on all major browsers (Chrome, Firefox, Safari, Edge), and supports a number of programming languages. Selenium allows you to mimic user actions, such as clicking, typing text, checking text, etc.

One of the reasons that Selenium can be so fragile, i.e., tests can break easily, is that when a change is made to the HTML DOM, the "locators" used to identify objects on the page must be updated, which can become a chore to keep up with. The use of AI will help in the development of "intelligent locators". Instead of needing to rely on an exact link to a button's ID, you could let AI find an object by context; logically analyzing the type of object, where it's located on the page, and any text contained in or around it. The use of intelligent locators will decrease the fragility of Selenium tests and reduce the maintenance effort required. An example of how well Selenium and GPT (Generative Pre-training) can work together is generating code from a simple text description to verify that an email address is in a form; a task that can take approximately 5 – 7 times longer when done manually.

At ASCN.AI, we were able to complete smoke tests for over 200 pages of a crypto exchange admin panel in only 3 days, whereas manual testing would have taken a month to complete. Unfortunately, Selenium cannot automate the testing of certain web technologies; such as Canvas, WebGL, and complex single-page applications (SPAs). In those cases, using Playwright is recommended.

Automating Testing is Easier with Playwright and AI

Playwright is a new tool developed by Microsoft specifically for working with SPAs. It offers excellent stability, multi-tab sessions, and automatic waiting for elements. In terms of speed and reliability, in most situations, it is faster and more reliable than Selenium.

The combination of the Playwright framework that includes the features of the GPT platform allows for tests to be generated automatically. Following the study of the page, the AI agent creates the scenarios, such as adding to the Shopping Cart and placing an order in a test.

For example, using GPT, a Playwright test was generated to check for order placement on a website in 15 seconds. Over 150 tests are run daily for a React application with asynchronous components using Playwright.

If you have Swagger/OpenAPI documentation, you can request GPT to generate tests to verify the appropriate status codes and responses for all requests, which could save a week's worth of manual effort. Limitations: at this time, Playwright AI does not offer testing for native mobile applications. In some instances, selectors will need to be manually adjusted for use with custom components.

CI/CD Automated Testing

Automated testing is a working practice in which all tests are run automatically after each commit. If the tests fail, the code will not be deployed into production. The work being performed is enhanced with "intelligence" provided by AI.

• Automatic launching of tests via predictive launch: The use of machine learning technology analyzes the files that have changed and utilizes only the most relevant tests for execution, reducing your pipeline from 40 minutes to 8 minutes.
• Automatic generation of regression tests: The use of AI will write the required tests for newly created features on a real-time basis.
• Failure analytics: AI will analyze the logs and stack trace information and provide hypotheses as to the cause of the error, thus reducing the time required for debugging (3–5 times faster).

At ASCN.AI, GitLab CI is utilized to execute linters, run unit tests, and then launch AI regression agents with Playwright by configuring it to execute more than 200 tests over a six-month period. This has enabled ASCN.AI to prevent over 14 critical bugs. However, it is vital that the infrastructure is maintained properly in terms of performing parallel executions and providing retries for flaky tests. If not done, the risk is that tests will fail, expectations will not be met, and any time savings will be negated as you are expending resources to repair the tests.

Test Case Generation with Artificial Intelligence

AI recognizes the requirements quite well. AI will analyze code to automatically generate a set of test scenarios for each requirement. For example, AI would generate scenarios for testing an email password reset function based on the requirements for checking ‘correctness’ of an email, the length of time an email link will work, and reuse of an email link, etc.

When ASCN.AI was monitoring funding rates, AI was able to generate 80+ test cases in 2 hours compared to the 1 week it took a person to generate the same test cases manually! AI has the benefit of recognizing common bottlenecks and security motifs (e.g. SQL Injection and XSS), but does not have a complete understanding of the business logic, meaning you will still need to add specific edge cases manually.

Automation of the QA Process with Artificial Intelligence

AI automation can assist with many of the processes around planning, analyzing logs, creating reports, and managing test data in addition to the execution of tests:

• Log analysis and production monitoring: AI will self-discover artifacts and create the appropriate log entries through ticketing.
• Report and dashboard generation: To provide coverage, track bug trends, and provide average bug fix time, and provide recommendations.
• Test data management: The sheer amount of load data that can be created by AI cannot be performed manually.

Utilizing synthetic datasets over a representative sample size (10,000 tokens) with a distribution of patterns allowed us to identify potential performance issues prior to release. In order to facilitate productive use of AI technology, it is necessary to establish a historical reference that consists of approximately 3–6 months' worth of collaborative effort between humans and computers. While pre-trained machine learning models may be immediately employed to generate relatively simple test cases and achieve desired coverage.

ASCN.AI Core Technology Base and Tools supporting AI-Enabled Test Automation

Neural networks are capable of accomplishing three primary objectives with regards to automated testing:

• Code Creation: produces functional tests written in different programming languages and/or testing frameworks from high-level descriptions of desired functionality.
• Optimization: analyzes entire automated test suites searching for repetitions and/or obsolete tests while generating recommendations for updated versions of existing tests through automated refactoring techniques.
• Self-Healing Abilities: automatically address issues related to changes to the Document Object Model (DOM) by updating the associated identification selectors used by automated tests as well as modifying any underlying logic that has been changed as a result of modification to the DOM.

At ASCN.AI, neural network algorithms can perform the following functions: refactor over 300 legacy automated tests in under 3 days; remove 47 duplicate automated tests; eliminate one month of manual effort in performing maintenance on legacy automated tests. However, it is important to note that neural networks do not possess any understanding of business rules; therefore, it is essential that requirements are clear.

A Summary of New and Emerging AI Tools That Can Be Used for Testing Automation

Methods or Practices for Integrating AI into QA Processes

There are four phases within the process:

1. Collect data — includes all artifacts such as user stories, specifications, code, and API documentation.
2. Analyze — includes processing key entities and relationships, and building a formalized model for the system.
3. Scenario Synthesis — will include creating positive, negative, and boundary tests.
4. Validate — including verifying and modifying test cases by the QA Engineer.

Using some type of executed prompt to generate test cases through a two-factor authentication via SMS will generate every scenario that could be executed, including being blocked due to entering an incorrect code or timing out the process. Jira has built-in artificial intelligence (AI), and can therefore generate new test cases on the fly almost immediately using AI. The duration between indicating tasks and the production of test cases has been decreased from a few days to only a few minutes.

The next step is an AI-driven way of analyzing test efficiency and optimizing tests. AI can detect ineffective tests by:

• Redundant Test Detection: AI can discover duplicated tests, thereby allowing you to optimize the test suite.
• Failure Prediction: By analyzing past commits, AI can predict that there will likely be failures in certain module selections.
• Test Execution Order Optimization: Prepare to execute the most critical tests first to reduce feedback time.
• Flaky Test Detection: AI can identify unstable tests that will contribute to ensuring positions are stable, minimizing false-positive test results.

Since the use of AI in ASCN.AI testing tool, we successfully eliminated redundancies and obsolete tests from the test suite and reduced testing time from 45 minutes down to 18 minutes without sacrificing reliability.

The advantages and disadvantages of using AI within Automated Testing

The use of AI within QA processes is not merely a trend but also a necessity, particularly as companies grow rapidly and need to produce products more quickly. However, there are some pros and cons associated with employing AI.

Benefits of Using AI in Automated Testing:

1. Dramatic Decrease in Test Development Cycle. Leveraging AI technology leads to a decrease in development time for test cases of 60-80%. What was traditionally a week of labor to manually create test cases can be accomplished in just a few hours.
2. Increased Test Case Coverage. AI is capable of generating test cases for hundreds of different scenarios, including complex edge cases that may not normally come to mind for a tester.
3. Reduction of Maintenance Costs (Self-Healing). One of the major challenges with traditional automated testing is "fragility" of test scripts. AI has the ability to adapt when there are changes in the UI by updating element identifiers within automation code.
4. Freeing Human Resources. Testers are free from doing routine tasks and can perform analysis of business risk and complex exploratory testing.

Restrictions/Limitations:

1. “Hallucination” and false confidence with AI — AI can generate code that looks correct (syntax) but has logical errors. Also, high coverage reports do not mean safety of the actual product.
2. Input Data Quality is everything — If the requirements are vague (PRD) or have outdated documentation, AI will create related tests. The old saying of “garbage in, garbage out” holds true here.
3. Business Context Understanding is inadequate — The neural network understands the code as per its own logic but does not understand the business logic as to why a function should “work” from an end-users perspective. Complex business logic must still be checked manually or strictly controlled to create relevant AI prompt.
4. Infrastructure Cost — When using high-level models such as GPT-4, there is a need to pay for tokens and computing resources. When scaling the cost to test thousands of test cases, it becomes excessive.
5. Decision opacity (black box) — Sometimes, AI may predict a failure and not know or explain why it predicted that failure. Explainable AI methods must be used in order for the team and/or company to understand the product/system logic.

Case Studies/Something to Learn from

Case 1: Automated Regression Testing for an E-Commerce Site
A platform sells 50,000 SKUs with two releases per week. Using manual regression testing, it would take 3 days for the regression to be completely tested. While the city sleeps — at night, GPT and Playwright come in, review and analyze the code, and output test cases for the modified code. Consequently, the total coverage expanded from 30% to 82%, regression testing time reduced from 3 days to 4 hours, and we discovered 27 bugs prior to the launch.

Case 2: In the Financial Technology Microservices Unit Test Generation
20 microservices in Node.js without any testing. Used pre-commit hook with GPT for unit test generation, checking coverage, and blocking commits if testing is inadequate. After 3 months of using these processes, 78% coverage was achieved; production incidents fell 65%. Now the team spends 10% of time on tests compared to none previously.

Case 3: ASCN.AI Test During a 10% Bitcoin Crash in the Crypto Market
Bitcoin crashed by 10% on October 11, 2024. An AI agent generated 50 scenarios in 12 minutes to account for high volatility; parallel testing showed that the system continued to function normally, allowing users to have arbitrage earnings during the crisis.

Case 4: AI Testing in a Legacy Banking System
European Bank uses a COBOL legacy system and a modern web front-end. Regression test generation assignment given to the model to analyze current and potential dependencies; Selenium provided automation for execution. Upon initial testing, results showed testing time was reduced from 6 weeks to 2 weeks; release frequency increased threefold and 14 incidents prevented annually.

Case 5: AI Testing for Virtual Scenarios in Automotive Industry
Electric vehicle manufacturer produces models on extreme road conditions. GPT generates edge cases based on real sensor readings; a simulator evaluates how the autopilot reacts to the case. An estimated 89 critical defects were found before performing live testing with the vehicles. This will save $2 million and approximately 70% in time.

FAQs

How do I start using AI for test automation?

• Assess your current level of automation and how much of your daily tasks are repetitive.
• Choose your tools: GitHub Copilot and GPT API have a very low barrier to entry; closed platforms are more comprehensive.
• Start with a pilot project on two or three key testing modules.
• Integrate automated test execution into CI/CD.
• Train the team on AI workflows, CI/CD, and critical thinking.

What skill set should a tester have to work with AI?

• Basic programming skills in a programming language (Python or JavaScript) to comprehend generated code and change code if necessary.
• Basic understanding of machine learning to be familiar with AI capabilities.
• Experience with APIs and CI/CD.
• Be able to apply critical thinking to results for review and refinement.

What is the difference between traditional autotests and AI-based tests?

Traditional autotests are based on rigidly defined logic-driven scripts that will require manual modification anytime the elements tested are changed within the application. AI-based autotests are adaptive; they will modify themselves (if possible) to account for UI and will be auto-generated including edge cases (minimizing time spent writing and maintaining, increasing coverage, and decreasing fragility). In addition, basic and critical scenarios will generally continue to be written manually.

What tool should I use for AI-based automated testing?

• Startups: GitHub Copilot + ASCN.AI NoCode + Playwright — a good stack at reasonable pricing.
• Mid-Size Companies: Mabl + GPT API for unit tests — provides a good balance of convenience and price for mid-size companies.
• Large Enterprises: Testim, Functionize, Applitools — have a very well-rounded functionality suite.

Disclaimer

The information in this article is general in nature and does not replace investment, legal, or security advice. Using AI assistants requires a conscious approach and an understanding of the features of specific platforms.