Edge Case Testing Guide: How to Test AI Support Agents

You've tested the happy path until the results look reassuring. The agent answers common questions, retrieves approved content, and hands difficult conversations to a human. Then a customer sends a long message in an unsupported language, omits the account identifier, triggers a rate limit, and asks for an exception to a policy. The response sounds polished, but it invents an answer and never escalates.

That failure isn't unusual because conversational systems don't operate on inputs alone. They interpret meaning, select tools, apply policy, retrieve context, preserve conversation state, and decide whether to answer, refuse, or hand off. Edge case testing has to challenge all of those behaviors together, not just verify that a form rejects an oversized value.

Table of Contents

• Why Edge Cases Fail AI Support Agents Differently
  • The hidden dimensions behind a conversational failure
• Designing Edge Case Tests for Conversational Workflows
  • Define the task contract first
  • Generate controlled variants
  • Build a versioned test matrix
  • Evaluate degradation, not fluency
• How Figr Can Help
• Applying NIST Research to Prioritize Test Combinations
  • Choose dimensions by failure impact
  • Select combinations around the agent's decisions
  • Turn the matrix into an operating practice
• Running Tests Manually and Automating Regression Suites
  • Turn discoveries into structured assertions
  • Put regression checks in the delivery path
• Quarantining Flaky Tests and Measuring Coverage Effectiveness
  • Reproduce before you quarantine
  • Measure behavioral coverage
  • Keep the suite aligned with production

Why Edge Cases Fail AI Support Agents Differently

A support agent can pass every ordinary test and still fail at the exact moment a customer needs judgment. Consider an account-recovery workflow. A customer correctly provides a recognized email address, but the session has expired. The request contains a typo in the product name, the customer's message exceeds the preferred context length, and the identity service is temporarily unavailable. Each condition may be manageable in isolation. Together, they can push the agent into an unsafe assumption.

Traditional software edge cases often center on a boundary. A field accepts an empty value when it should not. A counter reaches its maximum. A dependency returns an unexpected status. Conversational agents add behavioral boundaries that are harder to express with a simple pass or fail assertion. The agent might preserve the topic but lose the user's intent, answer confidently without evidence, expose information outside the user's permissions, or continue chatting when it should escalate.

Research on how AI support holds up makes this distinction clear. An agent can sound professional while hallucinating, violating scope, or failing to escalate, so evaluation must include noisy, incomplete, adversarial, and out-of-distribution inputs, not just clean questions (research on validity and reliability in AI risk management).

The hidden dimensions behind a conversational failure

For an AI support workflow, the test input is only one part of the scenario. A useful case records the surrounding state:

• Language and phrasing: The user may switch languages, use dialect, omit punctuation, or describe the same issue indirectly.
• Account state: The account may be active, locked, unverified, suspended, or missing from the retrieved context.
• Authentication state: A valid conversation doesn't necessarily prove that the user is authorized for the requested action.
• Message shape: Empty, malformed, duplicated, fragmented, extremely long, or contradictory messages can expose different assumptions.
• Knowledge availability: The approved article may be missing, stale, inaccessible, or irrelevant to the question.
• Routing conditions: A policy exception, sensitive topic, failed tool call, or repeated misunderstanding may require a human handoff.

A normal test asks, “Does the agent answer this question?” An edge test asks, “Does the agent make the right decision when intent, evidence, identity, and policy no longer line up neatly?”

That distinction also changes what counts as a defect. A fluent response can be wrong. A refusal can be wrong if the request was safe and answerable. An escalation can be wrong if it creates unnecessary workload, but failing to escalate a high-impact request is usually more serious. Teams need to evaluate intent preservation, evidence use, policy compliance, safety, and graceful degradation as separate behaviors.

Practical rule: Treat every answer, refusal, tool call, and escalation as a product decision that needs an explicit expected outcome.

This is why grounding belongs inside edge-case design rather than in a separate quality checklist. If you're reviewing why grounding is important for AI support, focus on the uncomfortable cases where the source is absent, ambiguous, or in conflict with the user's request. Those cases reveal whether the agent knows its limits or merely produces convincing language.

Designing Edge Case Tests for Conversational Workflows

Start with the work the agent must perform, not with a random collection of strange prompts. A useful test suite begins with critical tasks and maps the consequences of failure. Refund decisions, account deletion, authentication, billing changes, privacy requests, and human handoff deserve stronger controls than a low-risk product-information query.

Define the task contract first

For each workflow, write a compact contract that describes what the agent may do and what it must never do. Include the required evidence, permitted tools, user state, escalation triggers, and acceptable fallback.

For an account-deletion flow, the contract might require verified identity, a current policy source, confirmation of the irreversible action, and escalation when verification fails. For a product question, the agent may answer from approved documentation but should acknowledge when no relevant source is available.

A contract gives reviewers something more precise than “the answer looked good.” It also makes it possible to compare the same workflow across model versions, prompt changes, retrieval changes, and tool failures.

Generate controlled variants

Create variants systematically so that a single underlying intent is tested under altered conditions. Change one dimension first, then combine the dimensions that carry the greatest risk.

Include:

• Paraphrases: Ask for the same outcome directly, indirectly, politely, angrily, or with unnecessary background.
• Typos and spacing changes: Remove spaces, repeat characters, misspell product names, and vary punctuation.
• Incomplete inputs: Omit the order number, account identifier, date, or required confirmation.
• Malformed inputs: Send broken markup, copied error logs, strange formatting, or contradictory fields.
• Language variation: Use supported languages, unsupported languages, dialect, mixed-language messages, and translated terminology.
• Adversarial intent: Ask the agent to bypass a policy, reveal hidden instructions, or treat an unverified claim as proof.
• Prompt injection: Place instructions inside retrieved content, pasted tickets, attachments, or quoted customer messages.
• Distribution shifts: Use vocabulary, product descriptions, user behavior, and conversation structures that weren't represented in the original evaluation set.

The point isn't to maximize weirdness. It's to preserve the business intent while changing the surface conditions, then observe whether the agent makes the same safe decision.

Build a versioned test matrix

Use rows for test cases and columns for dimensions that affect behavior. A practical matrix can include:

Don't generate every possible combination automatically. Use pairwise or risk-based sampling for ordinary workflows, then exhaustively test boundaries in high-impact paths. For example, combine missing identity data with escalation triggers, maximum message length with unsupported language, and ambiguous intent with a strict policy constraint.

Compare every variant with a baseline. The response doesn't need identical wording, but it should preserve the same intent interpretation, policy outcome, safety posture, and escalation decision. Record the reason for any acceptable difference, especially when retrieved context or external-service state changes.

Evaluate degradation, not fluency

A well-built agent doesn't need to answer every prompt. It needs to fail safely. Define assertions such as:

• Must escalate when identity verification fails.
• Must refuse requests outside the user's authorization.
• Must cite or rely on an approved source when policy requires it.
• Must not expose private account information.
• Must state uncertainty when the knowledge base has no answer.
• Must preserve the user's intent after a typo or paraphrase.
• Must recover after a failed tool call without inventing a result.

Use human review for semantic consistency and automated checks for structural requirements. Store the prompt, model version, retrieved context, tool responses, output, decision labels, and reviewer notes so a regression can be reproduced.

For teams building a repeatable library, realistic mock chat examples for AI support testing can help turn difficult conversations into versioned fixtures rather than one-off manual checks.

How Figr Can Help

Figr is an AI design agent for product teams that want to turn live product context into usable design and QA artifacts. It can capture an existing application through Chrome, import Figma systems and tokens, retain decisions across sessions, and generate user flows, PRDs, prototypes, UX reviews, edge cases, and test cases that reflect the product rather than a generic template.

For edge-case work, its value is upstream. QA teams often receive a clean happy-path flow after many product assumptions have already been made. Figr can help expose alternate paths during discovery, map states around a workflow, and convert design decisions into testable cases before handoff. Its accessibility checks and design-token enforcement are useful when a visual or interaction boundary could create a support issue later.

Figr also analyzes patterns from 200,000+ screens, connects to analytics, compares funnels with industry benchmarks, and supports one-click export to Figma. Those capabilities can help a product team identify where users drop out, explore alternatives, and give QA more context before the conversational workflow is implemented. The platform lists SOC 2 compliance, SSO, zero data retention, and use by 500+ teams, so security and collaboration requirements can be part of the buying decision rather than an afterthought.

The trade-off is scope. Figr is most useful when the problem begins in product discovery, flow design, or handoff quality. It won't replace runtime evaluation of an AI agent's retrieval behavior, refusal decisions, escalation logic, or model nondeterminism. Teams should pair design-level analysis with a dedicated conversational test harness.

For a focused explanation of mapping a product's edge case scenarios, Figr's use-case material is a useful starting point. Choose it when your team needs design and QA artifacts connected to the actual product. Choose a specialized agent-evaluation workflow when the primary risk sits in model behavior after deployment.

Applying NIST Research to Prioritize Test Combinations

Exhaustive testing appears complete, but a conversational support system can make it impractical almost immediately. Language, account state, authentication, message shape, source availability, tool health, and escalation rules each introduce multiple values. Combining them produces more cases than a team can execute, review, and maintain.

NIST's combinatorial testing research offers a disciplined way to choose where to spend that effort. In a study based on 15 years of U.S. Food and Drug Administration medical-device recall data, NIST found that 65% of faults were triggered by a single factor, while 96% were triggered by either a single factor or a two-factor interaction (NIST research on combinatorial testing). That finding does not predict every failure in an AI support agent. It does establish a useful order: test single-factor boundaries first, then prioritize pairs that could change the agent's decision.

Choose dimensions by failure impact

List the variables that can alter the workflow outcome. A support agent commonly needs coverage across:

• language and locale
• account status and permissions
• authentication state
• message length and completeness
• missing or conflicting fields
• rate-limit state
• retrieved-source availability
• tool success or failure
• escalation eligibility
• policy sensitivity

Give each value a category: ordinary, boundary, invalid, or exceptional. Then rank combinations by the consequence of an incorrect response. A missing order number with a routine product question may require clarification. A missing identity signal with an account-deletion request should usually produce a controlled refusal or a human handoff.

The dimensions are not equally independent. Authentication changes which tools are safe to call. Source availability changes whether the agent can answer with confidence. Escalation eligibility changes the acceptable response when a policy-sensitive request remains unresolved. Those dependencies are where conversational edge cases differ from a simple input matrix.

Select combinations around the agent's decisions

Start with each boundary on its own, such as an empty required field, maximum message length, unsupported language, unavailable source, or failed escalation service. Pair each high-impact boundary with the condition most likely to alter the decision. Examples include an unauthenticated deletion request, a valid account with conflicting personal details, or a policy question asked while retrieval is unavailable.

NIST case studies show the value of structured selection. In one reported case, four-way combinatorial tests found all known faults using less than 5% of the tests required for exhaustive testing, a reduction of at least 20 times (NIST case studies and examples). The result supports efficient test selection, not a guarantee that an AI agent will behave correctly.

The same source reports that selected pairs identified approximately 50% to more than 97% of faults across different investigations. The range depends on the system and fault population, so pairwise coverage should be treated as a prioritization method rather than proof of correctness.

Turn the matrix into an operating practice

Use the matrix to define expected outcomes before running large batches. For each combination, record the permitted answer type, required verification, tool-call restrictions, citation or source requirements, and handoff condition. A conversational test case should evaluate the resulting state, not only whether the response contains a particular phrase.

Review failures manually, then convert confirmed defects into deterministic regression cases. Reserve three-way and higher-order combinations for workflows where the interaction can cause material harm, create an incorrect account action, expose restricted information, or block a required handoff. Rebuild the priority list after incidents, policy changes, tool changes, retrieval updates, and model releases.

Before writing any combination, export the agent's last 50 escalations and tag which of the ten dimensions above were present. Use the tag frequency to create the first pairwise priority list, then add combinations involving severe outcomes even when they occur less often. For implementation details, connect that matrix to the runtime described in this guide to AI agent testing. A spreadsheet that never exercises retrieval, tools, authentication, or handoff state is only a catalog of intentions.

Running Tests Manually and Automating Regression Suites

A refund conversation can pass its first test and still fail under pressure. Remove the order identifier, introduce a conflicting date, add a policy-bypass instruction, and change the customer's tone. Manual exploration exposes these interactions before they become formal requirements. Automation protects the behaviors the team has examined and assigned clear expected outcomes.

Start each release cycle with human-led sessions for high-impact workflows. Give testers realistic goals rather than fixed scripts. Ask them to vary tone, ordering, language, completeness, persistence, and the timing of corrections. The reviewer should record not only the final answer, but also whether the agent requested the right evidence, preserved the conversation state, used the correct tool, and avoided an unsupported promise.

Manual review catches semantic defects that simple checks miss. An automated assertion may confirm that a response contains a citation, while a reviewer sees that the cited policy does not apply to the customer's account state. A regular expression may detect “escalate,” while the transcript shows that the agent promised a handoff without creating a ticket.

Turn discoveries into structured assertions

After a tester confirms the intended behavior, encode it at the strongest reliable level:

• State assertions: The case must end in escalation, refusal, clarification, or resolution.
• Safety assertions: The response must not disclose private information or authorize a restricted action.
• Evidence assertions: The answer must use an approved source when one is required.
• Tool assertions: The agent must call the identity, billing, or ticketing tool before making a decision.
• Semantic assertions: The response must preserve the customer's intent and avoid unsupported claims.
• Recovery assertions: A failed dependency must produce a safe fallback rather than a fabricated success.

Keep wording flexible where natural variation is acceptable. Exact matches work well for structured fields, status changes, and routing events, but they are brittle for generative prose. Combine them with semantic evaluation, approved-source checks, and explicit business rules. Test the resulting state, not just whether the answer contains a particular phrase.

Pairwise automation fits the regression layer because it exercises interactions systematically. Human exploration remains necessary for unknown combinations, ambiguous intent, unclear evaluation criteria, and failures that require judgment. A practical starting split is to reserve roughly 20% of suite runtime for exploratory sessions and 80% for automated regression, then rebalance after each release according to which layer found defects.

Put regression checks in the delivery path

Separate fast, deterministic checks from slower evaluation suites. Run authentication, deletion, refund, and escalation cases on every relevant change. Run broader language, paraphrase, retrieval, and adversarial suites on scheduled builds or before release, based on cost and risk.

Version every input that can alter the result:

• model and system-prompt version
• retrieval index or source snapshot
• tool and API behavior
• temperature and generation settings
• test data and account state
• expected policy version
• evaluator version

A model update can change an answer without changing application code. The pipeline should report behavioral differences, not only code-test status. Store the complete transcript and execution context for every failure, including retrieved passages, tool responses, account state, and timing. Reviewers can then distinguish a model regression from a stale expected answer or a dependency problem.

SupportGPT provides a playground for trying real support questions, including difficult, unsupported, and adversarial cases. Teams can also build a custom harness around their model provider, retrieval layer, and ticketing system. The tool matters less than traceability from exploratory finding to assertion, failed run, review decision, and production incident.

For the operating process that connects those activities, follow these AI quality assurance practices. A regression suite earns trust when it records what was tested, why the behavior was expected, which workflow state resulted, and whether a human confirmed the oracle.

Quarantining Flaky Tests and Measuring Coverage Effectiveness

A flaky edge-case test is dangerous because it creates two bad choices. Engineers either rerun it until the pipeline turns green, which can hide a real defect, or ignore the failure because they no longer trust the test. Neither response improves confidence.

Google reported that approximately 16% of its test inventory exhibited some flakiness, while rerunning flaky tests consumed roughly 2–16% of compute resources (Google research on test flakiness). The lesson applies strongly to generative agents. A low failure rate per execution can still create substantial operational noise when the suite runs repeatedly across many environments and versions.

Reproduce before you quarantine

Run new boundary and adversarial cases repeatedly in clean environments. Capture the variables that can change an agent's output:

• random seed, when available
• model and prompt version
• retrieved passages
• temperature and other generation settings
• tool responses and external-service state
• timing, retries, and timeouts
• account and authentication state
• evaluator and policy versions

Classify each failure as a product defect, oracle defect, environment defect, dependency defect, or nondeterministic result. Quarantine only after evidence shows that the test cannot currently provide a reliable signal. Keep the case visible, assign an owner, and define the condition for returning it to the active suite.

A retry is a diagnostic tool, not a fix. If the first run violates “must not expose private data” and the second run passes, the workflow still needs investigation.

A test that passes after a silent retry hasn't proved that the agent is safe. It has only proved that one execution passed.

Measure behavioral coverage

Count more than executed tests and pass rates. Useful measures include:

• Edge-case detection rate: How often the suite identifies a known defect or meaningful behavioral deviation.
• Production escape rate: How often an edge-case failure reaches users before detection.
• False-positive rate: How often reviewers overturn automated failures.
• Flaky-test rate: How often each test family produces inconsistent results.
• Critical workflow coverage: Whether high-impact paths have boundary and interaction cases.
• Escalation correctness: Whether the agent hands off when it must and answers when it can.
• Semantic consistency: Whether paraphrases and perturbations preserve the expected decision.
• Latency and error behavior: Whether degradation remains safe when dependencies slow or fail.

Group these metrics by test family. A language-variation suite may be stable while a retrieval-failure suite is noisy. A high overall pass rate can conceal poor coverage of account state or escalation behavior.

Keep the suite aligned with production

Every confirmed production escape should produce one of three outcomes: a new deterministic test, a revised expected behavior, or a documented accepted limitation with an explicit owner. Review the matrix after policy updates, product launches, source changes, tool changes, and model releases.

Coverage is effective when it represents risk, not when it produces an impressive case count. A small suite that exercises identity boundaries, source absence, policy exceptions, tool failure, and human handoff may protect a support workflow better than a large collection of duplicate paraphrases.

Performance benchmarking for support agents can complement functional edge-case testing, but performance numbers shouldn't replace behavioral checks. The agent must remain safe, grounded, and correctly routed when the system is slow, context is incomplete, or the user's request falls outside normal patterns.

Build your first versioned edge-case matrix around one high-impact workflow, such as authentication, refunds, account deletion, or human handoff. Define the expected decision for each boundary, run the cases manually, automate the confirmed assertions, and review every flaky result instead of hiding it with retries. That process will give your team a reliable foundation for expanding AI support with confidence.