Automated Software Testing with LLMs

Software testing has always been a bottleneck in the development lifecycle. Writing comprehensive test suites is time-consuming, maintaining them as code evolves is tedious, and edge cases are notoriously difficult to identify. Large language models offer a compelling solution to each of these problems — but only if we address the unique verification challenges that AI-generated tests introduce.

The Testing Gap

Consider a typical enterprise application with hundreds of API endpoints, complex business logic, and integrations with external services. A QA team might achieve 70–80% code coverage through manual test writing, but the remaining 20–30% often contains the bugs that cause production incidents.

The gaps typically fall into these categories:

• Edge cases that developers did not anticipate during initial implementation
• Integration scenarios involving multiple services and state transitions
• Regression tests that should be added after every bug fix but often are not
• Property-based tests that verify invariants across random inputs

LLMs can help fill these gaps by analyzing code structure, understanding business logic from comments and documentation, and generating tests that humans might overlook.

LLM-Powered Test Generation

Unit Test Generation

Given a function and its context, LLMs can generate unit tests covering:

• Happy path scenarios with typical inputs
• Edge cases (empty inputs, boundary values, null/undefined handling)
• Error conditions and exception handling
• Type-specific behaviors for generic functions

// Example: LLM-generated test for a discount calculation function
describe("calculateDiscount", () => {
  it("applies percentage discount correctly", () => {
    expect(calculateDiscount(100, 20)).toBe(80);
  });

  it("returns original price when discount is zero", () => {
    expect(calculateDiscount(100, 0)).toBe(100);
  });

  it("throws for negative prices", () => {
    expect(() => calculateDiscount(-10, 20)).toThrow("Price must be positive");
  });

  it("caps discount at 100%", () => {
    expect(calculateDiscount(100, 150)).toBe(0);
  });
});

The quality of generated tests depends heavily on the context provided. Supplying the function implementation, related types, existing tests, and business rules produces significantly better results than providing the function alone.

Integration Test Generation

For API endpoints, LLMs can generate integration tests that:

• Set up required database state
• Make HTTP requests with realistic payloads
• Assert on response status, body structure, and side effects
• Clean up test data after execution

This is particularly valuable for teams that prioritize unit tests but underinvest in integration testing.

Regression Test Generation

When a bug is fixed, LLMs can analyze the bug report, the fix diff, and the surrounding code to generate a regression test that would have caught the original bug. This closes the loop between bug fixes and test coverage — a practice that many teams intend to follow but rarely maintain consistently.

The Verification Problem

AI-generated tests introduce a meta-testing problem: how do we verify that the tests themselves are correct?

The oracle problem. A test is only as good as its assertions. LLMs sometimes generate tests that pass trivially — asserting that a function "returns a value" rather than asserting the correct value. We need automated checks for test quality:

• Mutation testing — intentionally introduce bugs and verify tests fail
• Assertion strength analysis — flag tests with weak or missing assertions
• Coverage analysis — verify generated tests actually exercise the target code paths

False confidence. A suite of AI-generated tests that all pass can create a false sense of security. We treat AI-generated tests as drafts that require review, not as finished artifacts. Our review checklist includes:

1. Do assertions test the right behavior, not just that code runs?
2. Are edge cases meaningful, not just syntactically varied?
3. Do tests clean up resources and avoid side effects on other tests?
4. Would these tests have caught known past bugs in this module?

A Practical Workflow

In projects at IdeoMetriX and Sawaine, we integrate LLM test generation into the development workflow:

1. Developer writes or modifies code and creates a pull request
2. CI pipeline triggers test generation for changed files without existing test coverage
3. Generated tests are added as a separate commit labeled "AI-generated tests — review required"
4. Reviewer evaluates test quality alongside code review
5. Approved tests merge; rejected tests are discarded with feedback logged for model improvement

This workflow treats AI as a test-writing assistant rather than an autonomous test author, maintaining human accountability for test quality.

Beyond Generation: AI for Test Maintenance

Test maintenance is often more costly than test creation. LLMs can help here too:

• Detect obsolete tests that no longer match updated code behavior
• Suggest test updates when API signatures or business logic changes
• Identify redundant tests that cover the same code paths
• Generate test documentation explaining what each test verifies and why

Looking Ahead

Automated testing with LLMs connects directly to several important areas in AI4SE and software verification:

• Can we develop formal methods to verify AI-generated test correctness?
• How do we measure the "bug-catching potential" of a generated test suite?
• What training data and fine-tuning approaches produce the highest-quality test generators?
• How do multi-agent systems compare to single-model approaches for comprehensive test generation?

The intersection of AI and software testing represents one of the most immediately impactful areas of AI4SE. Better automated testing directly translates to more reliable software — a goal that aligns with both industry needs and the broader push toward trustworthy AI systems.

As LLMs continue to improve, the question shifts from "can AI write tests?" to "how do we ensure AI-written tests are trustworthy?" That verification challenge is where the most important work lies.