AI assistants (MCP)

Run your AutoTestX suites and find out why a test failed by asking your AI assistant, without leaving your editor.

AutoTestX includes an MCP server. The Model Context Protocol is the open standard AI assistants use to call tools in other applications. Connect Claude Code, Cursor, VS Code (GitHub Copilot) or Claude Desktop to it once, then ask:

“Run the Checkout tests on Staging and tell me if anything broke.”

The assistant finds the suite, starts it, waits for the result and, for each failed test, reads the evidence and explains what went wrong:

Assistant: 11 of 12 Checkout tests passed on Staging. Apply discount code failed at step 4: it typed SAVE10 and clicked Apply, then expected “Discount applied” but the page said “Code expired”. The button and form worked, so this is the test data: the promo code has expired. The test passed until 29 Sep and has failed every run since. Renew the code in the staging seed data, or use one that does not expire. Run: https://app.autotestx.com/#/runs/run_…

In an editor, the assistant can also look at your code and fix it. It then re-runs the suite to confirm the fix.

How it works

 you ──"run checkout tests"──► AI assistant (Claude Code, Cursor, VS Code…)
                                 │  picks a tool from its description
                                 ▼
                     https://app.autotestx.com/api/mcp     ← Authorization: Bearer atx_pat_…
                                 │  the same REST API a CI pipeline uses
                                 ▼
                     AutoTestX runs the suite on its own browsers
                                 │
 you ◄── plain-language answer ◄─┘

Nothing runs on your computer. The tests run on the AutoTestX server, as with Run on the Test Suites page.
The AI is your assistant’s, not AutoTestX’s. The reasoning happens in your assistant, under your own subscription. AutoTestX supplies the facts: results, failing steps, history. It does not use your organization’s AI allowance.
The API token decides what the assistant can do. A Read results only token lets it look but not start runs. See Permissions.

The tools

Tool	What it does	Typical question
`list_suites`	The project’s suites with test counts and how each did last, plus the environment names	“What test suites do we have?” “How did the suites do?”
`run_suite`	Starts a suite on an environment. Option: Ready tests only.	“Run Smoke on Staging.”
`get_result`	Status, counts and failed tests of a suite run. Waits up to 50 s for it to finish. Without an id it returns the latest run.	“Did it pass?” “What failed overnight?”
`investigate_failures`	Every failed test of a suite run with its failure class, confidence, request and evidence links, grouped by likely cause; an Unclassified failure may carry an AI suggestion	“Why did the suite fail?”
`explain_failure`	The evidence for why a test failed (below), for one run or the first 3 failures of a suite run	“Why did Apply discount code fail?”
`release_report`	The release evidence report: every suite run of a release, regressions against the release before, and a Go / Go with risks / No-go recommendation with its reasons. Option: one environment.	“Is v2.14.0 ready to release?”
`component_maintenance`	The component maintenance queue: components with a failing step, a self-healing fix waiting, drift, unpublished changes, outdated callers or no callers, most urgent first, each with a recommended action. Options: one component, how many findings.	“Which components need maintenance?”

What `explain_failure` gives the assistant

Evidence	Why it helps
The failing step with expected vs actual and its error	“expected Discount applied, got Code expired” is a data or app problem, not a broken test
The 3 steps before it, with what each did	Shows whether the test even reached the right page
Self-heals in the same run, and whether they came before the failure	A heal that picked the wrong element often makes a later step fail
Steps found only by a fallback locator	The page is drifting from what the test recorded
For a data-driven test, the first failing row and its inputs	“fails only with qty = 2”
The test’s recent results on the same environment: last passed, failing since, flaky	Separates “broke with the last deploy” from “has always been flaky”
Hints: short rule-based conclusions from all of the above	A starting point for the assistant, and readable by a person too
Links to the run report and the failing step’s screenshot	So you can check it yourself

Secret values and data columns marked sensitive are redacted before any of this leaves the server. The same evidence is available without an AI assistant at GET /api/runs/:id/failure-context.

Connect your assistant

You need an API token for the project. A project Admin or Test Manager creates it under Project Members → API tokens for CI (API tokens):

Role Run tests (tester) lets the assistant start suites. Read results only (viewer) lets it only look.
Create one token per person, named after them (MCP - Dana), so it can be revoked without affecting anyone else, and suite runs show whose assistant started them.
Treat it like a password. It goes in your assistant’s configuration, never in a repository.

In the examples, replace https://app.autotestx.com with your AutoTestX address and atx_pat_… with the token.

Claude Code

claude mcp add --transport http autotestx https://app.autotestx.com/api/mcp --header "Authorization: Bearer atx_pat_…"

Check it with claude mcp list, or /mcp inside a session. Add --scope user to have it in every project on your computer.

Cursor

In Settings → MCP → Add new MCP server, or in ~/.cursor/mcp.json:

{
  "mcpServers": {
    "autotestx": {
      "url": "https://app.autotestx.com/api/mcp",
      "headers": { "Authorization": "Bearer atx_pat_…" }
    }
  }
}

VS Code (GitHub Copilot agent mode)

In .vscode/mcp.json. VS Code asks for the token once and stores it securely, so the file can be shared with your team:

{
  "inputs": [
    { "type": "promptString", "id": "autotestx-token", "description": "AutoTestX API token", "password": true }
  ],
  "servers": {
    "autotestx": {
      "type": "http",
      "url": "https://app.autotestx.com/api/mcp",
      "headers": { "Authorization": "Bearer ${input:autotestx-token}" }
    }
  }
}

Claude Desktop

Claude Desktop’s configuration file starts local servers. The small mcp-remote bridge connects it to AutoTestX (it needs Node.js). In Settings → Developer → Edit Config (claude_desktop_config.json):

{
  "mcpServers": {
    "autotestx": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://app.autotestx.com/api/mcp", "--header", "Authorization:${AUTOTESTX_AUTH}"],
      "env": { "AUTOTESTX_AUTH": "Bearer atx_pat_…" }
    }
  }
}

Restart Claude Desktop. AutoTestX then appears under the tools icon.

Other assistants

Any MCP client that supports Streamable HTTP with a custom header can connect. Give it:

Setting	Value
URL	`https://<your AutoTestX address>/api/mcp`
Transport	Streamable HTTP (stateless)
Header	`Authorization: Bearer atx_pat_…`

Connecting from claude.ai on the web, the Claude mobile apps and ChatGPT needs a sign-in flow (OAuth) instead of a token. That is planned.

Things to ask

You say	The assistant
“Run the AutoTestX Smoke suite on Staging.”	`list_suites` → `run_suite` → `get_result` until done
“What failed overnight, and why?”	`get_result` (latest) → `investigate_failures` → `explain_failure` for a test that needs a closer look
“Can we release v2.14.0? What are the risks?”	`release_report` → `investigate_failures` on a suite with a blocker
“Twenty tests fail at sign-in. Is it our Login component?”	`component_maintenance` → `explain_failure` on one of the failing runs
“Is the Login test flaky?”	`explain_failure` on its last failure and reads `history.flaky`
“I renamed the checkout button. Run Checkout and tell me whether self-healing coped.”	`run_suite` → `get_result` → `explain_failure` looks at `heals`
“Fix whatever broke the discount test, then re-run it.” (in an editor)	`explain_failure` → searches your code → edits → `run_suite` again

Say “AutoTestX” if your project also has unit tests. Otherwise “run the tests” could mean npm test to the assistant.

Permissions and safety

The token’s role is the limit. run_suite with a Read results only token is refused, and the assistant is told why. A token can never manage people, environments, secrets or other tokens, whatever the assistant tries.
Your assistant asks first. Claude Code, Cursor, VS Code and Claude Desktop ask your permission before calling a tool, unless you allow it permanently. list_suites, get_result, investigate_failures, explain_failure, release_report and component_maintenance are marked read-only. run_suite is not.
Runs started by an assistant are labelled. Recent suite runs shows them as AI assistant. They do not send deploy pipeline alerts: a developer trying a fix is not a broken release.
Limits apply as for pipelines. One token can have 5 suite runs in progress. Tests beyond your plan’s capacity wait in the queue.
Revoke at once under Project Members → API tokens for CI. The assistant’s next call is refused.

Troubleshooting

You see	Why	Fix
The assistant does not use AutoTestX	It is not connected, or the question sounded like unit tests	Check the connection (`claude mcp list`, Cursor’s MCP settings). Say “AutoTestX” in the question.
AutoTestX refused the API token	Token wrong, revoked or expired	Create a new token and update the configuration
Not allowed … “Read results only”	A viewer token	Use a Run tests token if the assistant should start runs
AutoTestX answered 404: The suite “…” has no shared tests	Wrong suite name, or its tests are private	Ask the assistant to call `list_suites` first
AutoTestX answered 429	5 suite runs from this token are still going	Wait, or cancel some on the Test Suites page
`405 Method not allowed` in a client log	The client tried to open an event stream	Normal: the endpoint is stateless and answers on POST only
`401` with Connect with an API token	The request carried a browser session, not a token	Configure the `Authorization: Bearer` header

To try the MCP server step by step in the MCP Inspector, see the exercises (section D).