Skip to content
AutoTestX Docs

Course: CI and releases — 5. Run after deployment, notifications and MCP

Hands-on exercises for three features that work together:

To use a real GitHub repository instead of a simulated pipeline, see GitHub Actions pipeline.

The exercises are grouped as follows:

Area What you test Where
A. Notifications card Who is emailed, editing the lists Test Suites page → Notifications
B. Run after deployment A pipeline starts a suite with a token; deploy details, waiting, limits, emails PowerShell (plays the pipeline) + Test Suites page
C. Self-healing review digest Email listing new fixes that need a review Test editor + Self-healing page + server terminal
D. MCP server An AI assistant runs suites and explains failures MCP Inspector (a web page) or Claude Code
E. Failure context The “why did it fail” evidence Browser address bar
F. Small screens The new UI at phone width Any of the above

Mark each test Pass / Fail in the Result column. Text in quotes under “Expected” is the exact wording you should see.

Why some steps use PowerShell. “Run after deployment” is started by a pipeline, not by a person, so there is no button for it. In section B your PowerShell window plays the pipeline, and you check the results in the browser. Emails are not really sent in this test: the server prints them in its terminal.

Time needed: about 75 minutes. Section C waits about 4 minutes for the digest.

0. Setup (once)

Section titled “0. Setup (once)”

0.1 Build and start the server with a separate data folder, and with the heal digest every 2 minutes instead of every hour, so section C does not take an hour. In PowerShell, from the project folder:

Terminal window
npm run build:web
$env:ATX_DATA_DIR = ".data-features-manual"; $env:ATX_HEAL_DIGEST_MINUTES = "2"; npm run start

Keep this terminal visible for the whole test. Emails appear in it as boxes starting with ┌─ email:.

0.2 Sign in as the built-in administrator.

  1. Open http://127.0.0.1:4100
  2. In the terminal running AutoTestX, find the link printed after set the system administrator’s password.
  3. Open it and set a password.
  4. You are signed in as the system administrator (Local User).

0.3 Organization and project. Create organization QA Org with project Shop.

0.4 Plan. Organization → Settings → plan card → Change plan → Team → save. API tokens need the Team plan.

0.5 Environment. Configurations → Environments → add Staging (any variable).

0.6 Suite. Testing → Test Suites → add Smoke.

0.7 Tests. Testing → Test Cases → create two shared tests in suite Smoke. Every Open step uses this page address (copy it exactly, as one line):

data:text/html,<h1>Checkout</h1><input id="promo"><button id="apply">Apply</button><p id="msg">Code expired</p>
Test Steps (type → setting) Status
Open checkout 1. Open → URL = the address above
2. Visible → target strategy CSS #apply		 Ready
Apply discount code 1. Open → URL = the address above
2. Type → target CSS #promo, value SAVE10
3. Click → target CSS #apply
4. Text → target CSS #msg, expected Discount applied		 Ready

Run each test once from the editor: Open checkout passes. Apply discount code fails at step 4 (expected “Discount applied” … “Code expired”). This failure is on purpose.

Changing a Ready test. A Ready test is locked. Several steps below ask you to change Apply discount code: open it, click New revision, make the change, set Status back to Ready, and Save. The run uses the new version.

0.8 API tokens. Configurations → Project Members → API tokens for CI → create two tokens. Copy each value when it is shown, because it is shown only once:

Name Role
Pipeline - test Run tests (Tester)
Read only - test Read results only (Viewer)

0.9 The “pipeline” window. Open a second PowerShell window, paste the following, and put in your two token values. This adds a small atx command that calls AutoTestX the way a pipeline does and prints the HTTP status.

Terminal window
$ATX    = 'http://127.0.0.1:4100'
$TOKEN  = 'atx_pat_PASTE_THE_TESTER_TOKEN'
$VIEWER = 'atx_pat_PASTE_THE_VIEWER_TOKEN'
function atx([string]$Method, [string]$Path, $Body = $null, [string]$Token = $TOKEN) {
  if (-not $ATX) { Write-Host 'Set $ATX first (step 0.9)' -ForegroundColor Red; return 'Set $ATX first (step 0.9)' }
  $p = @{ Method = $Method; Uri = "$ATX$Path"; Headers = @{ Authorization = "Bearer $Token" }; UseBasicParsing = $true }
  if ($Method -ne 'GET') { $p.ContentType = 'application/json'; $p.Body = if ($Body) { $Body | ConvertTo-Json -Depth 5 } else { '{}' } }
  try { $r = Invoke-WebRequest @p; Write-Host "HTTP $($r.StatusCode)" -ForegroundColor Green; $r.Content | ConvertFrom-Json | ForEach-Object { $_ } }
  catch {
    $resp = $_.Exception.Response
    if (-not $resp) { Write-Host "NO RESPONSE: $($_.Exception.Message)" -ForegroundColor Red; return "NO RESPONSE: $($_.Exception.Message)" }
    $code = [int]$resp.StatusCode
    Write-Host "HTTP $code" -ForegroundColor Red
    if ($_.ErrorDetails.Message) { $_.ErrorDetails.Message } else { "HTTP $code (no message in the response)" }
  }
}
atx GET /api/tokens/self

Expected: HTTP 200 and the token’s name, Pipeline - test, with project Shop.

The helper lives only in this PowerShell window. If you open a new window, paste it again. NO RESPONSE: Unable to connect to the remote server means the server is not running (or $ATX has the wrong address). Every result in section B needs it running.

A. Notifications card (Test Suites page)

Section titled “A. Notifications card (Test Suites page)”
ID Steps Expected Result
A1 Open Test Suites. Scroll below Schedules. A card Notifications with two rows: “Deploy pipeline runs” and “Self-healing fixes to review”, each with a one-line explanation. The second says “at most every 2 minutes”.
A2 Read who is emailed in each row. Deploy pipeline runs: Local User, with “Default: Admins and Test Managers”. Self-healing: Local User, with “Default: everyone who may approve heals (Test Developers and above)”.
A3 Click Edit on “Deploy pipeline runs”. A checklist of the project’s people, with Local User (you) ticked, and “1 person.”. Buttons Cancel and Save. No “Use default” yet.
A4 Untick Local User. The text changes to “Nobody will be emailed.”
A5 Click Save. Toast “Notifications saved”. The row now says “Nobody - turned off”.
A6 Click Edit again. A Use default button is now shown.
A7 Click Use default. Back to Local User with “Default: Admins and Test Managers”.
A8 Click Edit, then Cancel. Nothing changes.

Leave both rows on their default for the rest of the test. Local User must receive the emails.

B. Run after deployment (pipeline window + browser)

Section titled “B. Run after deployment (pipeline window + browser)”

B-Start and wait

Section titled “B-Start and wait”
ID Steps Expected Result
B1 In the pipeline window, start the suite as a deploy pipeline would:
$start = atx POST /api/executions @{ suite = 'smoke'; environment = 'staging'; scope = 'ready'; deployment = @{ source = 'github-actions'; commit = 'a1b2c3d4e5f60718'; branch = 'main'; deployId = '4711'; url = 'https://github.com/acme/shop/actions/runs/4711'; note = 'Release 2.14' } }; $start		 HTTP 202. Shows executionId (exec_…), tests = 2, status = running, url (…/#/suites/exec_…) and statusUrl. Lower-case smoke and staging are accepted.
B2 Wait for the result, as a pipeline does:
$r = atx GET "/api/executions/$($start.executionId)?wait=60"; $r.status; $r.counts		 Returns as soon as the suite ends (a few seconds, not 60). status = failed, counts: passed 1, failed 1.
B3 In the browser, reload Test Suites. Look at Recent suite runs. A new row: Smoke, FAIL, “1 / 2 passed · 1 failed”, Staging. Next to “Smoke” a small marker reads deploy.
B4 Hover over the deploy marker. Tooltip: “Started by a deploy pipeline (commit a1b2c3d4e5f60718)”.
B5 Click the row to open it. Below the counts, a line: “Deploy · github-actions · main @ a1b2c3d4e5f6 · #4711 · pipeline run”, and under it “Release 2.14”.
B6 Click pipeline run. A new browser tab opens to the GitHub address from B1. It does not exist, which is fine: the link is what we are testing. Close the tab.

B-Emails

Section titled “B-Emails”
ID Steps Expected Result
B7 Look at the server terminal. An email box: subject “Smoke is failing on Staging after a deploy (Shop)”, to the system administrator. The text contains: “Deploy: github-actions · main @ a1b2c3d4e5f6 · #4711”, “Note: Release 2.14”, “Pipeline: https://github.com/…/4711”, the failed test Apply discount code with why it failed, a link to its run, and “Open the run: http://…/#/suites/exec_…”.
B8 Run B1 and B2 again (same commands). Again FAIL. No new email in the terminal: the suite was already failing on Staging.
B9 Start the suite on no environment:
$s2 = atx POST /api/executions @{ suite = 'Smoke'; deployment = @{ commit = 'b2' } }; (atx GET "/api/executions/$($s2.executionId)?wait=60").status		 failed. A new email in the terminal, “Smoke is failing on no environment after a deploy (Shop)”. Each environment is tracked on its own.
B10 Fix the test: Apply discount code → New revision → step 4’s expected text = Code expired → Status Ready → Save. Then run B1 and B2 again. status = passed. Terminal: email “Smoke is passing again on Staging (Shop)”, with “It had been failing since … (2 runs).”
B11 In the browser, click Run on Smoke (a person, not a pipeline). The run shows in Recent suite runs without a deploy/CI marker, and no email appears in the terminal. Runs a person starts are never emailed: they are watching.
B12 Turn pipeline emails off (A3–A5). Break the test again (New revision → expected Discount applied → Ready → Save). Run B1 and B2. Then set the list back with Use default. FAIL, and no email, because the list was empty when the suite started failing.

After B12, leave Apply discount code failing (expected Discount applied). Sections D and E use the failure.

B-Ready only

Section titled “B-Ready only”
ID Steps Expected Result
B13 Create a shared test Work in progress in suite Smoke: one Open step with the checkout address. Leave it Draft. Run B1 and B2 again (B1 has scope = 'ready'). Opening the run in the browser: Work in progress is SKIP with “Not Ready (this run includes Ready tests only)”. The other two ran.
B14 Start without scope:
$s3 = atx POST /api/executions @{ suite = 'Smoke' }; (atx GET "/api/executions/$($s3.executionId)?wait=60").counts		 total = 3, skipped = 0: without scope every test runs, Drafts included. Then delete the Work in progress test (a Draft can be deleted).

B-Refusals (what a pipeline sees when it gets something wrong)

Section titled “B-Refusals (what a pipeline sees when it gets something wrong)”
ID Command Expected Result
B15 atx POST /api/executions @{ suite = 'Smoke'; scope = 'some' } HTTP 400, “scope must be "all" or "ready".”
B16 atx POST /api/executions @{ suite = 'Smoke'; deployment = @{ sha = 'x' } } HTTP 400, “Unknown deployment field "sha". Use: source, commit, branch, deployId, url, note.”
B17 atx POST /api/executions @{ suite = 'Smoke'; deployment = @{ url = 'ftp://x' } } HTTP 400, “deployment.url must start with http:// or https://.”
B18 atx POST /api/executions @{ suite = 'Smoke'; environment = 'Prod' } HTTP 404, “Environment "Prod" not found in this project.”
B19 atx POST /api/executions @{ suite = 'Nope' } HTTP 404, “The suite "Nope" has no shared tests to run.”
B20 atx POST /api/executions @{ suite = 'Smoke' } -Token $VIEWER HTTP 403, “Your role in this project does not allow running tests.”
B21 atx GET "/api/executions?suite=Smoke" -Token $VIEWER | Select-Object -First 5 status, trigger, startedAt, deployment HTTP 200: a viewer token can still read results. A list of the latest Smoke runs, newest first. Runs started in B1 show trigger = api and deployment = @{source=github-actions; commit=a1b2c3d4e5f60718; …}. Runs started without deploy details (B14, B23) have an empty deployment, and runs from the Run button have trigger = manual.
B22 atx POST /api/executions @{ suite = 'Smoke' } -Token 'atx_pat_wrong' HTTP 401, code invalid_token.
B23 Start 6 runs at once:
1..6 | % { atx POST /api/executions @{ suite = 'Smoke' } | Out-Null }		 The first 5 show HTTP 202. The 6th shows HTTP 429. In the browser, 5 new Smoke runs appear (some queued).
B24 Read the refusal itself:
1..6 | % { atx POST /api/executions @{ suite = 'Smoke' } } | Select-Object -Last 1		 The last line: “This API token already has 5 suite runs in progress. Wait for one to finish, or cancel one.” and "code":"too_many_running". If B23’s runs are still going, all six are HTTP 429, which is expected. If instead you see NO RESPONSE: …, the server is not running. If you see a run (executionId …), the earlier runs had already finished: run the command again straight away.

B-CLI (optional)

Section titled “B-CLI (optional)”
ID Steps Expected Result
B25 Pretend to be GitHub Actions. Run these as two separate commands: first
$env:ATX_SERVER = $ATX; $env:ATX_TOKEN = $TOKEN; $env:GITHUB_ACTIONS = 'true'; $env:GITHUB_SHA = 'cafe1234'; $env:GITHUB_REF_NAME = 'release/2.15'
and then
npm run atx '--' run --suite Smoke --env Staging --ready		 One line per test, then “FAILED suite Smoke”. In the browser, the run is marked deploy, and its Deploy line shows “github-actions · release/2.15 @ cafe1234”. Afterwards: Remove-Item Env:GITHUB_*.
B26 npm run atx '--' run --suite Smoke --env Staging --note "manual check" --no-deployment The run’s Deploy line shows only “from a pipeline” and “manual check”. The CI variables were not used.

C. Self-healing review digest

Section titled “C. Self-healing review digest”

The digest runs every 2 minutes in this test (step 0.1). It leaves alone any fix younger than 2 minutes, so a fix shows up 2–4 minutes after it was found.

C0 Create a test that will heal. Create a shared test Place order in suite Smoke and leave it Draft, so you can keep editing it:

  1. Open → URL data:text/html,<h1>Shop</h1><button id="place-order">Place order</button>
  2. Click → target CSS #place-order

Save, and run it once from the editor. It passes. AutoTestX now remembers what the button looks like.

ID Steps Expected Result
C1 Change step 1’s URL to data:text/html,<h1>Shop</h1><button id="submit-order">Place order</button> (only the id changed). Save and run again. The run passes. Step 2 is marked as a self-healing suggestion: the button was found by what it looks like, not by #place-order.
C2 Open Self-healing in the menu. A badge with 1. Under Pending, a card for Place order, step “Click…”, proposing the new locator (#submit-order).
C3 Wait 2–4 minutes and watch the server terminal. An email: subject “1 self-healing fix to review in Shop”. It lists “Place order: Click …” with its risk, confidence, whether the run used it or only suggested it, and a link …/#/healing/tc_…. It ends with “1 fix in total awaits review: …/#/healing”.
C4 Wait another 2–4 minutes. No second email: nothing new was found.
C5 Change step 1 back to id="place-order", run, then change it to id="order-now" and run again. Wait 2–4 minutes. A new digest about the new fix. The total line counts every pending fix.
C6 On Self-healing → Pending, approve or reject the fixes. Wait 2–4 minutes. No email: decided fixes are not mailed.
C7 Test Suites → Notifications → Edit “Self-healing fixes to review” → untick everyone → Save. Make another heal (C5) and wait. Then Use default. No email while the list is empty.

D. MCP server (an AI assistant using AutoTestX)

Section titled “D. MCP server (an AI assistant using AutoTestX)”

You can test this in a web page, without any AI. The MCP Inspector is the official test tool for MCP servers. It shows the same tools an assistant sees and lets you click them.

D0 Open the Inspector. In a third PowerShell window:

Terminal window
npx @modelcontextprotocol/inspector@2.8.0

It opens a page in your browser. On the left, set:

Setting Value
Transport Type Streamable HTTP
URL http://127.0.0.1:4100/api/mcp
Authentication / Custom headers Header name Authorization, value Bearer + your Tester token (the Inspector may have a Bearer Token field instead: paste just the token there)

Click Connect.

ID Steps Expected Result
D1 Look at the connection status and server info. Connected to autotestx 1.0.0. The server’s instructions mention “regression tests, grouped into suites”.
D2 Tools tab → List Tools. Exactly four: list_suites, run_suite, get_result, explain_failure, each with a description.
D3 Run list_suites (no arguments). First line: “1 suite, 1 environment.” Then data: suite Smoke with tests = 3 and lastRun (its status and environment), and environments: ["Staging"].
D4 Run run_suite with suite = smoke, environment = Staging. “Started smoke on Staging: 3 tests. Call get_result with executionId “exec_…” to wait for the outcome.” (3 = Open checkout, Apply discount code, Place order.)
D5 Run get_result with that executionId, waitSeconds = 40. Returns when the suite ends: “Smoke on Staging: FAILED - …, 1 failed. Call explain_failure with a runId (“run_…” for Apply discount code) to see why.” The data has trigger: "mcp" and failedTests listing Apply discount code.
D6 Run get_result with no arguments. The same latest run: what an assistant uses for “what failed overnight?”.
D7 Run explain_failure with the runId from D5. First line: “"Apply discount code" failed on Staging at step 4 (…): Expected … "Discount applied", but it was "Code expired"”. The data has failedStep (index 4, expected / actual, a screenshot link), previousSteps (steps 1–3), history and hints. The first hint starts “The step ran and the page answered, but with a different value”.
D8 Run explain_failure with the executionId instead. “1 failed test in Smoke:” followed by the same summary.
D9 Run run_suite with suite = Nope. Shown as an error result: “AutoTestX answered 404: The suite "Nope" has no shared tests to run.”
D10 In AutoTestX, reload Test Suites → Recent suite runs. The run from D4 is marked AI assistant (tooltip “Started by an AI assistant through the MCP server”).
D11a Make Smoke “passing” on Staging first. (After B12–B13 it is recorded as failing, and the email only goes out on a change from passing to failing, so without this step D11 proves nothing.) Fix Apply discount code: New revision → step 4 expected = Code expired → Ready → Save. In the pipeline window, run B1 and B2. status = passed. Terminal: one email, “Smoke is passing again on Staging (Shop)”.
D11b Break it again: New revision → step 4 expected = Discount applied → Ready → Save. —
D11c Fail it through the AI assistant. Before clicking, press Enter a few times in the server terminal to leave a gap you will recognize. In the Inspector run run_suite (suite = Smoke, environment = Staging), then get_result with its executionId until finished. get_result: FAILED. Terminal: no new email box below your gap. An assistant’s run is not a deploy.
D11d Control: fail it through a pipeline. In the pipeline window, run B1 and B2. status = failed, and a new email box: “Smoke is failing on Staging after a deploy (Shop)”. This proves an email would have been sent, so the silence in D11c is the rule working. It also shows the assistant’s failure was ignored entirely: Smoke was still “passing” until this run.
D12 Disconnect. Change the header to the Viewer token, connect, run list_suites, then run_suite Smoke. list_suites works. run_suite is an error: “Not allowed: Your role in this project does not allow running tests. The AutoTestX API token’s role may be "Read results only".”
D13 Disconnect. Change the token to atx_pat_wrong, connect. The connection is refused (401).

D14 (optional) With a real assistant: Claude Code. In a terminal:

Terminal window
claude mcp add --transport http autotestx-local http://127.0.0.1:4100/api/mcp --header "Authorization: Bearer atx_pat_PASTE_THE_TESTER_TOKEN"

Start claude and ask: “Run the AutoTestX Smoke suite on Staging and tell me why anything failed.”

Expected: Claude asks to use the autotestx tools, runs the suite, waits, and explains that Apply discount code expected “Discount applied” but the page said “Code expired”. Remove it afterwards with claude mcp remove autotestx-local.

E. Failure context in the browser

Section titled “E. Failure context in the browser”

Each address below is in its own box. Copy it exactly: nothing before or after it, and no full stop at the end. Replace RUN_ID with a real run id (run_…).

E1 Open the failed run of Apply discount code (Test Runs, or the suite panel). Copy the run id from the address bar: the part that starts with run_.

E2 Open this address, with your run id:

http://127.0.0.1:4100/api/runs/RUN_ID/failure-context

Expected: a JSON page with summary, failedStep (with expected and actual), previousSteps, heals, history and hints. Chrome and Edge show a Pretty-print checkbox at the top. Firefox formats it by itself.

E3 In that JSON, find failedStep → screenshot and open that address. Expected: the screenshot taken when step 4 failed.

E4 Do E2 with the run id of an Open checkout run that passed. Expected: summary ends with “passed on Staging.”, failedStep is null and hints is [].

E5 Do E2 with the run id of Place order’s healed run (C1). Expected: heals lists the healed click step. failedStep is null, because the run passed.

E6 Open an id that does not exist:

http://127.0.0.1:4100/api/runs/run_doesnotexist/failure-context

Expected: {"error":"Run not found"}.

ID Result
E1
E2
E3
E4
E5
E6

If you get No route for GET …/failure-context without a full stop at the end, the server was started before this feature was added. Restart it (step 0.1).

F. Small screens

Section titled “F. Small screens”

Make the browser window narrow, about phone width, or use the browser’s device toolbar (F12).

ID Steps Expected Result
F1 Test Suites → Notifications. Each row stacks: title and explanation, then who is emailed, then Edit. Nothing is cut off and there is no sideways scrolling.
F2 Edit a list. The checklist and the Cancel / Use default / Save buttons fit the width.
F3 Open a deploy run. The Deploy line wraps onto several lines. The long commit does not push the page wider.

Clean up

Section titled “Clean up”
  1. Stop the server (Ctrl+C) and the Inspector (Ctrl+C).
  2. Close the pipeline window, or clear its variables: Remove-Item Env:ATX_*.
  3. Delete the test data folder: Remove-Item -Recurse -Force .data-features-manual.
  4. If you did D14: claude mcp remove autotestx-local.

If something fails

Section titled “If something fails”

Note the test ID, what you did and what you saw, plus:

  • the server terminal output around that moment;
  • for B and D: the full command or tool arguments, and the response;
  • the run or suite run id (run_…, exec_…) if there is one.

Reference: Run after deployment · Notifications · AI assistants (MCP) · GitHub Actions pipeline · REST API → Suite executions