Course: Diagnosing failures — 3. Failure classification

Eleven hands-on exercises on the failure class every failed run gets, and the failure investigation of a suite run (Failure class):

Module 1: failures the application caused: a Maximo business rule, a server error, a crash, a refused session, an unreachable host, missing test data;
Module 2: failures on the test’s side: a missing element, a timeout, a wrong value, and what happens when no rule matches;
Module 3: reading the report: likely causes, the analysis, the evidence links;
Module 4: how the test’s history moves the confidence;
Module 5: the same report through the API, the command line, the failure email and an AI assistant.

Each exercise has:

Goal: what you will see;
Before you start: the preparation it needs;
Steps: Action, Expected result, and a box to tick;
What happened: the explanation;
Check yourself: a question to answer before moving on. The answers are at the end.

Allow about 1 hour. Do the exercises in order: each one starts from what the one before left.

How a class is decided

Fixed rules, not AI: the same evidence always gives the same class and the same confidence. Every rule that matches the evidence adds a signal with a weight, and the class with the highest score wins.

Evidence	Class	Weight
A Maximo message `BMXAA…E` (in the error, a failed call’s response or the page’s dialog)	Maximo business rule	90%
A Maximo warning `BMXAA…W`	Maximo business rule	55%
An API step answered 5xx	Server error	95%
The page’s own calls got a 5xx	Server error	85%
The page threw an uncaught exception	Server error	55%
An API step answered 401 or 403	Authentication / permission	90%
The page’s own calls got 401 or 403	Authentication / permission	80%
The data source or pool gave no rows	Test data	95%
A host could not be reached	Environment	80%
The step ran out of time	Timeout	70%
The element was not found	Locator	75%
A value differed from the expected one	Wrong value	70%
Nothing matched	Unclassified	20%

Then four adjustments:

each further signal for the same class adds 5% (at most 10%);
a missing element, a timeout or locator drift counts for half when the application itself reported a cause: it is the symptom, not the cause;
+5% for a cause in the application when the test passed last time; −15% when the test is flaky;
two classes within 10% of each other: the winner loses 10%, and the other is shown as Also possible.

Preparation

The exercises use a small Maximo-like demo application whose pages fail in known ways. The browser that runs your tests has to reach it, so do these exercises on your own installation (Deployment): a hosted server cannot reach an application running on your computer.

P1: Start the demo application

In a PowerShell window in the AutoTestX folder:

node examples/failure-classification/app.mjs

It prints demo application on http://127.0.0.1:4430/. Leave it running.

P2: Start AutoTestX with its own data

In a second PowerShell window, so your normal data is not touched:

npm run build
npm run build:web
$env:ATX_DATA_DIR = ".data-fclass-training"; npm run start

Open the link printed after set the system administrator’s password, choose a password and sign in as the system administrator. Keep this window visible: emails are printed in it (Exercise 10).

P3: Practice tests and environment

Test Cases → Import the 17 files in examples/failure-classification/, one at a time. Tests 01–16 land in the suite Failure classification, test 17 alone in Classification history. All are Ready.

File	Test	What goes wrong
`01-maximo-error.json`	Save work order	The save answers 400 with BMXAA4129E; the dialog shows it
`02-maximo-warning.json`	Update asset (API)	An API step gets 400 with BMXAA7901W
`03-maximo-info.json`	Save work order 1002	Only an information message, BMXAA6713I
`04-server-error.json`	Place order	The order call answers 503
`05-api-500.json`	Read order 7 (API)	An API step gets 500
`06-page-crash.json`	Asset list	The page throws while loading
`07-auth-401.json`	Open profile	The profile call answers 401
`08-api-403.json`	Admin settings (API)	An API step gets 403
`09-login-button-missing.json`	Log in button	A step called “Log in” finds nothing
`10-no-data-rows.json`	Create work orders from a table	Its data table has no rows
`11-unreachable-host.json`	Open the old server	Nothing listens on port 4439
`12-timeout.json`	Slow report (API)	The API takes 10 s; the step allows 2 s
`13-locator.json`	Start Center chart	The chart is not on the page
`14-wrong-value.json`	Apply discount code	The page says “Code expired”
`15-unclassified.json`	Unset variable	`{{workOrderNumber}}` has no value
`16-passes.json`	Start Center opens	Nothing: it passes
`17-flip.json`	Inventory loads	Fails only while you switch it to “bad” (Module 4)

Configurations → Environments → New environment: name Demo Maximo, Variables (JSON) { "baseUrl": "http://127.0.0.1:4430" }, Save environment.
Configurations → Project Members → API tokens for CI → create a token Pipeline - classification that can Run tests. Copy its value: it is shown only once.

Where you read a class

On Test Suites, in the panel of a suite run, not in a single run’s report:

Likely causes  [3 × Locator] [1 × Server error POST …/api/orders] …

Save work order                                              FAIL
  [Maximo business rule] 90%  POST …/maximo/oslc/os/mxwo  BMXAA4129E   ← the class line
    • Maximo business rule (maximo-error): BMXAA4129E - …               ← click the line: the analysis
    Screenshot  Video  Trace  Network log

Click the class line to open the analysis; click it again to close it. The test’s name opens its run report, as before.

Module 1: Failures the application caused

Exercise 1: A Maximo business rule

Goal: see a Maximo message decide the class, over the HTTP status it came with.

Before you start: P1–P3.

#	Action	Expected result
1	Test Suites → Run suites against = Demo Maximo → Run on Failure classification. Wait about a minute	The panel: FAIL, 16 of 16 finished, 1 passed, 14 failed, 1 error
2	Find Save work order and read its class line	Maximo business rule 90% `POST 127.0.0.1:4430/maximo/oslc/os/mxwo` `BMXAA4129E`
3	Click the class line	First bullet: Maximo business rule (maximo-error): BMXAA4129E - Record already exists for Site=BEDFORD, Work Order=1001.
4	Find Update asset (API)	Maximo business rule 55% `POST …/maximo/oslc/os/mxasset` `BMXAA7901W`
5	Find Save work order 1002	Locator 75%, and no BMXAA code
6	Click Network log under Save work order’s analysis, and find the `/mxwo` line	`"status":400` and `"messages":[{"code":"BMXAA4129E", … }]`, and nothing else from the response

What happened: the save answered 400, and the dialog and the response both said BMXAA4129E. The code says why far better than the status, so it decides: one fact seen twice counts once, and the 400 adds nothing. A warning (…W) counts for less: it may be a dialog the test did not expect to dismiss. An information message (…I) is not a failure at all, so Save work order 1002 is simply an element that was not found. The demo’s error response also holds a customer’s name: only the code and its sentence are ever kept.

Check yourself: two runs fail with BMXAA4129E on different days. What does that tell you that two “400” failures would not?

Exercise 2: A server error and a crash

Goal: see a server error outrank the missing element it caused.

#	Action	Expected result
1	Find Place order	Server error 85% `POST 127.0.0.1:4430/api/orders`
2	Open its analysis	Server error (http-5xx): POST 127.0.0.1:4430/api/orders answered 503. Later: The element for step 2 (“See the receipt”) was not found.
3	Find Read order 7 (API)	Server error 95% `GET 127.0.0.1:4430/api/orders/7` (rule http-step-5xx)
4	Find Asset list and open its analysis	Server error 55%, first bullet Server error (page-exception): TypeError: …, and below the bullets Also possible: locator 38%

What happened: the receipt is missing because the order call failed. When the application reports a cause, the missing element counts for half (75% → 38%): it stays listed, but never competes with its cause. An API step’s own status is the strongest evidence there is (95%). A crash weighs less (55%): pages sometimes throw harmlessly, so Also possible keeps the locator in view.

Check yourself: why is Place order 85% and not 85% minus the 10% for a close runner-up?

Exercise 3: A refused session

Goal: tell an authentication problem from a step that merely talks about logging in.

#	Action	Expected result
1	Find Open profile	Authentication / permission 80% `GET 127.0.0.1:4430/api/me`
2	Find Admin settings (API)	Authentication / permission 90% `GET 127.0.0.1:4430/api/admin`
3	Find Log in button	Locator 75%: not authentication

What happened: 401 and 403 are evidence about the session or the test user’s permissions. A step labelled “Log in” that finds nothing is not: the rules read what the application and the error said, never the step’s name.

Check yourself: a suite fails with five tests in Authentication / permission. What do you check first?

Exercise 4: The environment and the test data

Goal: see failures that are neither the application’s code nor the test.

#	Action	Expected result
1	Find Open the old server	Environment 85% `GET 127.0.0.1:4439/`
2	Find Create work orders from a table	ERR, the reason Inline table (0 rows) has no rows, so there is nothing to run., and Test data 95%. Its name is not a link
3	Open the test and click Run test (environment Demo Maximo)	An error message with the same reason. No run is created

What happened: the old server’s address refused the connection: two signals agree (the request and the step’s error), 80% + 5%. The data table had no rows, so the test never ran: it has no run and no report, and is classified from the reason it could not start. That is only possible in a suite run; a single run that cannot start is refused at once.

Check yourself: why is Test data almost certain (95%)?

Module 2: Failures on the test’s side

Exercise 5: Missing element, timeout, wrong value

Goal: recognise the three classes that point at the test or the page it expected.

#	Action	Expected result
1	Find Start Center chart	Locator 75%; analysis Locator (element-not-found): step 2 (“See the KPI chart”): No strategy resolved…
2	Find Slow report (API)	Timeout 70%
3	Find Apply discount code	Wrong value 70%; analysis expected “Discount applied”, got “Code expired”

What happened: nothing on the application’s side said anything was wrong. A missing element means the page changed, the test went somewhere else, or the page was slow. A timeout means a wait that never ended. A wrong value means the step worked and the page answered differently: the application or its data, not the test finding the page.

Check yourself: what makes Wrong value different from Locator, when you go to fix it?

Exercise 6: When no rule matches

Goal: see that the classifier says “I don’t know” rather than guess.

#	Action	Expected result
1	Find Unset variable	Unclassified 20% (a grey chip)
2	Open its analysis	No bullets: no rule matched
3	Find Start Center opens	PASS, and no class line

What happened: the step failed on {{workOrderNumber}}, a variable no step set. No rule describes it, so the run is Unclassified. A low, honest answer is more useful than a confident wrong one: read the run report. A passed run gets no class.

Check yourself: many failures in a suite are Unclassified. What does that suggest?

Module 3: Reading the report

Exercise 7: Likely causes and the evidence

Goal: use the report to see how many problems a red suite really has.

#	Action	Expected result
1	Read Likely causes, between the tally and the tests	Coloured chips, the first 3 × Locator. At most five
2	Hover over 3 × Locator	The three tests: Log in button, Save work order 1002, Start Center chart
3	Look at the colours	Red: server error, Maximo, environment. Amber: auth, test data, timeout. Purple: locator. Blue: wrong value. Grey: unclassified
4	Open Save work order’s analysis and click each link	Screenshot, Video, Trace (downloads `trace.zip`) and Network log open without signing in again
5	Make the window narrow (about 375 px)	The chips and class lines wrap; nothing scrolls sideways

What happened: failures are grouped by class and request. Ten tests failing as Server error on one endpoint are one chip and most likely one fix. Different endpoints are different chips. The evidence links are the same files the run report shows.

Check yourself: a suite shows 8 × Server error POST …/api/orders and 2 × Locator. Where do you start?

Module 4: History moves the confidence

Exercise 8: Regression and flaky

Goal: see the same failure scored differently depending on the test’s past.

Before you start: open these two addresses in a browser tab when asked. Each answers /flip is now bad or …good: http://127.0.0.1:4430/toggle?state=bad and http://127.0.0.1:4430/toggle?state=good.

Run Classification history from Test Suites each time, against Demo Maximo.

#	Action	Expected result
1	Run it (the demo starts “good”)	PASS, no class line
2	Open `toggle?state=bad`, run it	FAIL: Server error 90% `GET 127.0.0.1:4430/api/inventory`
3	Open the analysis	The last bullet starts It passed last time (…) and fails now
4	Open `toggle?state=good`, run it	PASS
5	Open `toggle?state=bad`, run it	FAIL: Server error 70%
6	Open the analysis	The last bullet: This test has passed and failed alternately in its recent runs on this environment: likely flaky
7	Open `toggle?state=good`	The demo is back to normal

What happened: the evidence was the same each time: a 503 from the inventory call (85%). After a pass, a failure points at something that changed: +5%. After pass, fail, pass, fail the result has changed three times: the test is flaky, and nothing about it is sure: −15%.

Check yourself: why does a flaky test lower the confidence of every class, not only the locator?

Module 5: The same report everywhere

Exercise 9: Through the API

Goal: read the report as a pipeline or a script would.

Stay signed in and type these in the browser’s address bar.

#	Action	Expected result
1	Open `http://127.0.0.1:4100/api/executions?suite=Failure%20classification`	The suite’s runs, newest first. Copy the `executionId` of the one with `"status":"failed"`
2	Open `http://127.0.0.1:4100/api/executions/<executionId>/investigation`	`"complete":true`, `"failed":15`, `groups` and 15 `entries`
3	Find the entry for Save work order	`"class":"maximo"`, `"confidence":0.9`, `"rule":"maximo-error"`, the `endpoint`, `"bmxaa":["BMXAA4129E"]`, `artifacts` and `analysis`
4	Copy its `runId`; open `http://127.0.0.1:4100/api/runs/<runId>/failure-context`	The usual failure context, with a `classification`: the same class, plus `signals` and `"rulesVersion":1`

What happened: the class is stored with the run when it finishes, with the version of the rules that decided it. A later change to the rules never quietly relabels old runs.

Check yourself: why store the class instead of working it out each time the report is opened?

Exercise 10: The command line and the failure email

Goal: see the causes where a pipeline and its team see them.

In a third PowerShell window, with the token from P3:

$env:ATX_TOKEN = 'atx_pat_PASTE_THE_TOKEN'
node --import tsx packages/cli/src/index.ts run --suite "Failure classification" --server http://127.0.0.1:4100 --env "Demo Maximo" --release v9.1

#	Action	Expected result
1	Read the end of the output	FAILED suite Failure classification, then Likely causes: (3 × Locator, 1 × Server error - POST 127.0.0.1:4430/api/orders, …) and one line per failed test with its class
2	`echo $LASTEXITCODE`	`1`: the suite failed. The report never changes the exit code
3	Look at the AutoTestX window (P2)	One email: Failure classification is failing on Demo Maximo after a deploy of v9.1 (Maximo)
4	Read it	Likely causes:, then each failed test with a class line such as Maximo business rule, 90% - POST …/mxwo (BMXAA4129E)
5	Run the command again	The same output, and no new email: the suite was already failing

What happened: a run started with an API token is a pipeline run, and its first failure mails the team. The email and the command line read the same report as the panel.

Check yourself: Create work orders from a table has no class line in the email. Why?

Exercise 11: An AI assistant

Goal: see what an assistant gets when asked “why did the suite fail?”.

In the third window:

npx @modelcontextprotocol/inspector

Set Transport Type Streamable HTTP, URL http://127.0.0.1:4100/api/mcp, and the header Authorization = Bearer + your token. Connect.

#	Action	Expected result
1	Tools → List Tools	Five tools, among them `investigate_failures` (read-only)
2	Run investigate_failures with the `executionId` from Exercise 9	15 failed tests in Failure classification. Likely causes: and the list, then the full report
3	Run it with the ID of a Classification history run that passed (`/api/executions?suite=Classification%20history`)	Classification history has no failed tests (status passed).

What happened: the assistant gets the grouped report first, then calls explain_failure for a test that needs a closer look. It reads the same rule-based classes, so its answer does not depend on how it guesses.

Check yourself: why start with investigate_failures rather than explain_failure for each failed test?

Clean up

Stop the windows (Ctrl+C) and delete the practice data:

Remove-Item -Recurse -Force .data-fclass-training

Also delete trace.zip from your Downloads folder.

Answers

That it is the same business rule both times: the code is stable across languages and versions, while many different problems answer 400.
The missing element is a symptom of the 503 and counts for half (38%), so it is not within 10% of the server error.
The environment’s credentials and the test user’s roles: a session that expires or a permission that was removed.
The run could not get its inputs at all: no step ran, so nothing else could explain it.
A locator is fixed in the test (re-pick the element); a wrong value is fixed in the application or its data, or the test’s expectation is out of date.
Failures the rules do not cover, often in the test itself (variables, expressions). Read their run reports.
The 8 server errors on one endpoint: one fix in the application may clear all eight. Then the locators.
A test that flips is unreliable evidence of anything: the failure may be timing or shared data, whatever the class says.
So history does not change: a run keeps the class it was given, and rulesVersion says which rules gave it.
It never got a run: the email shows the class stored with a run. The investigation report still classifies it, from the reason it could not start.
One call groups all the failures by likely cause, so the assistant explains one problem instead of fifteen symptoms.