Course: Diagnosing failures — 4. AI diagnosis

Eleven hands-on exercises on the AI suggestion a failure gets when the classification rules cannot name it (AI suggestion). They follow the failure classification exercises: do those first, or at least read their Exercise 6 (When no rule matches).

• Module 1: turning it on, and reading a suggestion for what it is;
• Module 2: what it costs: one answer for failures that look the same, the daily limit, and Diagnose with AI;
• Module 3: telling the AI it was wrong, and what that changes;
• Module 4: from suggestions to new rules: the Unclassified causes view;
• Module 5: the same suggestions in the command line, the failure email and an AI assistant;
• Module 6: what an organization administrator controls.

Each exercise has:

• Goal: what you will see;
• Before you start: the preparation it needs;
• Steps: Action, Expected result, and a box to tick;
• What happened: the explanation;
• Check yourself: a question to answer before moving on. The answers are at the end.

Allow about 1 hour. Do the exercises in order: each one starts from what the one before left.

How AI diagnosis works

When	Only for a failure the rules left Unclassified, and only when the project turned it on. A failure with a class never goes to the AI.
What it gives	A cause in one sentence, a confidence (low, medium, high), the nearest category, 1 to 5 quotes from the run’s evidence, up to 3 things to check, and a rule that would have caught it.
What it never does	Change the class. The failure stays Unclassified in Likely causes, the release report and anything a pipeline checks.
What is sent	The failed step and its error, the requests around it (without query strings), Maximo codes, the steps before it, heals, the failing data row and the test’s history. Never full logs, response bodies, screenshots, recordings or secret values.
What it costs	One AI generation from the organization’s monthly allowance per new answer. Failures that look the same share one answer for 7 days.

What the AI writes is not fixed. The wording of a cause, its quotes and its suggested rule change from run to run. When an exercise says what a suggestion is about, check that, not the words.

---

Preparation

The exercises use the demo application from the failure classification exercises and make four real AI calls (a few cents) on your own Anthropic API key. Do them on your own installation (Deployment).

P1: Start the demo application

In a PowerShell window in the AutoTestX folder:

node examples/failure-classification/app.mjs

It prints demo application on http://127.0.0.1:4430/. Leave it running.

P2: Start AutoTestX with its own data and your API key

The AutoTestX folder’s .env file must have your key: a line ANTHROPIC_API_KEY=… (and ANTHROPIC_WORKSPACE_ID=… if your key needs one). It is the same key the AI step writer uses. In a second PowerShell window:

npm run build
npm run build:web
$env:ATX_DATA_DIR = ".data-aidiag-training"; npm run start

Open the link printed after set the system administrator’s password, choose a password and sign in as the system administrator. Keep this window visible: emails are printed in it (Exercise 10).

P3: Practice tests and environment

1. Test Cases → Import examples/failure-classification/15-unclassified.json (Unset variable) and 13-locator.json (Start Center chart).
2. On Test Cases, click Duplicate on Unset variable. The copy is called Unset variable (copy). It fails the same way.
3. Duplicate Unset variable again. In the new copy, rename the test Unset asset, change step 2’s label to Type the asset number and its value to {{assetNum}}, and Save. It fails the same kind of way, but on a different step.
4. Configurations → Environments → New environment: name Demo Maximo, Variables (JSON) { "baseUrl": "http://127.0.0.1:4430" }, Save environment.

All four tests are in the suite Failure classification.

Test	What goes wrong	Class
Unset variable	Step 2 types {{workOrderNumber}}, which nothing sets	Unclassified 20%
Unset variable (copy)	The same	Unclassified 20%
Unset asset	Step 2 types {{assetNum}}, which nothing sets	Unclassified 20%
Start Center chart	The chart is not on the page	Locator 75%

Running the suite and counting AI calls

• Run the suite: Test Suites → Run suites against = Demo Maximo → Run on Failure classification. Leave the run panel open: suggestions arrive a few seconds after the tests finish.
• Count AI calls: Organization → Settings, card AI generations this month. The big number is how many were used.

---

Module 1: A suggestion, not a class

Exercise 1: Off until a project turns it on

Goal: see that nothing is sent to the AI until someone decides to.

#	Action	Expected result	✓
1	Open Test Suites	A card AI diagnosis of unclassified failures: Off. Off until the project turns it on.
2	Run the suite	FAIL, 4 failed. Likely causes: 3 × Unclassified, 1 × Locator
3	Read the line under Unset variable	The grey Unclassified 20% chip, then AI diagnosis is off for this project. and a Diagnose with AI button
4	Open Organization → Settings	AI generations this month: 0

What happened: every new answer costs part of the organization’s AI allowance, so automatic diagnosis is off until a project’s Admin or Test Manager turns it on. The line says why there is no suggestion instead of saying nothing.

Check yourself: why is automatic diagnosis off by default, when Write with AI is available as soon as a key is set?

Exercise 2: Turn it on

Goal: turn automatic diagnosis on and watch the suggestions arrive.

#	Action	Expected result	✓
1	On the Test Suites card, click Change	A box Diagnose a failure with AI when no rule classifies it and New diagnoses per day, at most (empty: the default, 20)
2	Tick the box, Save	AI diagnosis: On, at most 20 new diagnoses a day. This project’s own setting.
3	Run the suite and watch the three Unclassified tests	First AI suggestion coming…, then an AI suggestion under each
4	Look at Start Center chart	Locator 75%, no suggestion

What happened: each test ran and was classified as before. Then, only for the three the rules left Unclassified, the server sent the evidence to the AI in the background. The run did not wait for it.

Check yourself: why does Start Center chart never get an AI suggestion, even with diagnosis on?

Exercise 3: Read a suggestion

Goal: read what the AI says, and see what it does not change.

#	Action	Expected result	✓
1	Read the line under Unset variable	AI suggestion · medium confidence · test authoring (or another confidence and category), then one sentence about workOrderNumber having no value
2	Look at the chip above it	Still Unclassified 20%
3	Click the class line to open it	Evidence: short quotes in code text, such as Unresolved reference "workOrderNumber". Check: up to three things to do. Then Written by AI from this run’s evidence. It does not change the class.
4	Click the test name to open the run report and find step 2’s error	Each quote under Evidence appears in the run’s own evidence

What happened: the category Test authoring is one the rules don’t have: the mistake is in the test, not the application or the environment. Every quote is checked against what was sent before the suggestion is kept. An answer that cannot point at the evidence is thrown away and not charged.

Check yourself: a release report counts this failure as a risk. Does turning AI diagnosis on change that?

---

Module 2: What it costs

Exercise 4: One answer for failures that look the same

Goal: see that the same failure is diagnosed once, however often it happens.

#	Action	Expected result	✓
1	Open Organization → Settings	2 used, not 3
2	Back on the run, open Unset variable (copy)	The same cause as Unset variable, and …shared with an earlier failure that looked the same
3	Read Likely causes	3 × Unclassified followed by AI suggestion: and that cause: two of the three share it
4	Run the suite again	The suggestions appear at once
5	Check Organization → Settings again	Still 2

What happened: Unset variable and its copy fail on the same step (same type, same label) with the same error, so they share one answer. Unset asset fails on a different step and got its own. A later run that fails the same way reuses the answer for 7 days, for free. The Settings page names these calls AI diagnosis (Maximo): automatic diagnoses are charged to the project, not to whoever started the run.

Check yourself: twelve tests fail with the same Unclassified error on the same shared login step. How many AI calls is that?

Exercise 5: The daily limit, and Diagnose with AI

Goal: see what stops automatic diagnosis, and what does not.

#	Action	Expected result	✓
1	Test Suites card → Change, set New diagnoses per day to 1, Save	On, at most 1 new diagnosis a day.
2	On Unset asset’s suggestion, click Wrong	Wrong 1, muted. (Exercise 6 explains why.)
3	Run the suite	Unset asset: The project’s daily limit on AI diagnoses is reached. and Diagnose with AI. The other two: their suggestion, at once
4	Click Diagnose with AI on Unset asset	Diagnosing…, then a new suggestion
5	Open Organization → Settings	3 used; your name has 1 of them
6	Test Suites card → Change, clear the number, Save	Back to 20 a day

What happened: the daily limit only stops automatic diagnosis: today’s 2 were already over 1. Diagnose with AI is a person asking, so it ignores the project’s on/off and daily limit, and it is charged to that person. Reused answers never count toward the limit.

Check yourself: who should get the Diagnose with AI button: everyone who can see the run, or only people who can write steps with AI? Why?

---

Module 3: Telling the AI it was wrong

Exercise 6: Helpful and Wrong

Goal: rate suggestions, and see what a rejected one looks like.

#	Action	Expected result	✓
1	On Unset asset’s new suggestion, click Helpful	Helpful 1, highlighted
2	Click Helpful again	Back to Helpful: your vote is taken back
3	On Unset variable, click Helpful, then Wrong	Helpful, Wrong 1: one vote per person, and your second replaced the first
4	Look at Unset variable (copy)	Wrong 1 too: it is the same answer
5	Look at both, then at Likely causes	Both muted, Marked wrong by more people than found it helpful. Likely causes: 3 × Unclassified with no AI suggestion

What happened: votes belong to the answer, not the run, so a vote on any failure that shares it counts for all of them. When more people say Wrong than Helpful, the suggestion stays on screen but no longer speaks for its group, and the email and command line leave it out.

Check yourself: one person clicks Wrong and another clicks Helpful. Is the suggestion muted?

Exercise 7: Wrong asks again

Goal: see that an answer marked Wrong is not reused.

#	Action	Expected result	✓
1	Run the suite	Unset variable and its copy: AI suggestion coming…, then a new suggestion with no votes. Unset asset: the same as before, at once
2	Open Organization → Settings	4 used: only the answer marked Wrong was made again

What happened: Wrong drops the answer from the shared cache, so the next failure that looks the same is diagnosed from scratch. The new answer starts with no votes. Taking a Wrong vote back later does not bring the old answer back.

Check yourself: you click Wrong by mistake on a suggestion that was right. What does it cost?

Exercise 8 (optional): A second person’s vote

Before you start: Organization → Members → Invite: bob@example.test, name Bob Builder, organization role Member, project role Tester. Copy the invitation link from the server window and open it in a private window (Edge InPrivate, Chrome Incognito); set a password and sign in.

#	Action	Expected result	✓
1	Bob: Test Suites → Recent suite runs → the latest run	Suggestions with Helpful / Wrong. On the AI diagnosis card, no Change and no Unclassified causes
2	Bob: click Helpful on Unset asset	Helpful 1 (Bob’s)
3	You: reload	Helpful 1, not highlighted: it is Bob’s vote, not yours
4	You: project Members & roles → change Bob to Viewer. Bob: reload	The votes show, but Bob’s buttons are disabled

What happened: anyone who runs tests can vote: they are the people who triage failures. Viewers read. Changing the setting and reading the rule list take an Admin or Test Manager.

Check yourself: why is a person’s own vote highlighted, but not other people’s?

---

Module 4: From suggestions to new rules

Exercise 9: Unclassified causes

Goal: find the next rule worth adding to the classifier.

#	Action	Expected result	✓
1	Test Suites card → Unclassified causes	A dialog with 7 days, 30 days (selected), 90 days
2	Read the first line	How many failures were Unclassified in the last 30 days, how many got a suggestion, why the rest didn’t (diagnosis off for Exercise 1’s run) and how often people rated them helpful
3	Read the table	One row per suggested rule, most failures first: the rule; its category, the tests it came from and an example cause; how many failures and when last; the votes
4	Look for Unset asset’s first answer, which you marked Wrong in Exercise 5	Not listed. Below the table: Not listed: … with a suggestion people marked wrong
5	Click 7 days, then 90 days	The same numbers: everything happened today

What happened: each suggestion also proposes a rule that would have caught the failure. This view groups the period’s answers by that rule, after putting aside case, punctuation and spacing. It never guesses that two differently worded rules mean the same. A rule that comes back across tests, with people agreeing, is worth adding to the classification rules. Once it is added, those failures get a class (here, something like Test authoring) and stop costing AI calls.

Check yourself: a suggested rule has 40 failures from one test, another has 12 failures from 9 tests. Which is the better candidate for a new rule?

---

Module 5: The same suggestions elsewhere

Exercise 10: Command line, email and an AI assistant

Before you start: Configurations → Project Members → API tokens for CI → create a token Pipeline - diagnosis, role Run tests, and copy its value. In a third PowerShell window:

$env:ATX_TOKEN = 'PASTE-THE-WHOLE-TOKEN-HERE'   # the whole value, starting atx_pat_; $env:ATX_TOKEN.Length should say 51
node --import tsx packages/cli/src/index.ts run --suite "Failure classification" --server http://127.0.0.1:4100 --env "Demo Maximo" --release v9.3

#	Action	Expected result	✓
1	Read the end of the output	Likely causes:, one dimmed line per failed test, and under the Unclassified ones AI suggestion (medium): …
2	echo $LASTEXITCODE	1: a suggestion never changes the exit code. (2 with Invalid or revoked API token. means the token is wrong: set it again)
3	Look at the server window	A failure email. Under each Unclassified test: Unclassified, 20%, then AI suggestion (…): …
4	Start npx @modelcontextprotocol/inspector, connect to http://127.0.0.1:4100/api/mcp with the header Authorization = Bearer + your token, and run Investigate the failures of an AutoTestX suite run (the Inspector lists tools by title; it is investigate_failures) with the run’s executionId (from http://127.0.0.1:4100/api/executions?suite=Failure%20classification)	3 × Unclassified (AI suggestion: …) in the text; aiDiagnosis and aiFeedback in the JSON

What happened: wherever the investigation report goes, the suggestion goes with it, labelled as one. The command line prints what is there when the suite finishes; the email waits up to 20 seconds for suggestions still being made. An AI assistant reading the report is told to weigh a suggestion as a lead, not a class, and sees how people rated it.

Check yourself: a pipeline gates a deploy on atx run --suite. Can an AI suggestion make the gate pass or fail?

---

Module 6: The organization’s say

Exercise 11: Turn it off for every project

Goal: see what an organization administrator controls.

#	Action	Expected result	✓
1	Organization → Settings, card AI diagnosis of unclassified failures	Projects may turn it on… On in: Maximo (up to 20 a day).
2	Click Turn off for all projects	Off for every project.
3	Test Suites	The project card: Off. The organization turned AI diagnosis off. No Change
4	On a failure with Diagnose with AI, click it (mark one Wrong and run the suite to get one)	The organization turned AI diagnosis off.
5	Organization → Settings → Allow, then Test Suites	On, at most 20 new diagnoses a day. This project’s own setting.

What happened: the organization’s switch overrides every project, the button included, so an administrator can stop all AI spending on failures at once. Each project’s own setting is kept, and comes back when it is allowed again.

Check yourself: an organization is close to its monthly AI allowance. What are two ways to stop diagnosis using it, and how do they differ?

Clean up

Stop the windows (Ctrl+C), close the private window, and delete the practice data:

Remove-Item -Recurse -Force .data-aidiag-training

---

Answers

1. Nobody asks for an automatic diagnosis at the moment it is made: it spends the organization’s allowance on its own. Write with AI only runs when a person clicks it.
2. The rules gave it a class (Locator). Only failures no rule can name are sent to the AI.
3. No. The failure stays Unclassified everywhere a decision is made; the suggestion only describes it.
4. One: they fail on the same step with the same error, so they share one answer.
5. Only people who can write steps with AI: the click is charged to them, and Tester and Viewer seats do not carry an AI allowance.
6. No: one Wrong and one Helpful is not more Wrong than Helpful.
7. One more AI call: the next failure that looks the same is diagnosed again.
8. The highlight is how you see, and change, your own vote; the numbers already count everyone’s.
9. The one with 12 failures from 9 tests: a cause that recurs across tests is a pattern a rule can describe. 40 failures of one test is one test to fix.
10. No. Only the tests’ results decide the exit code; the suggestion is printed beside them.
11. An organization administrator turns it off for every project (the button too), or the project’s Admin or Test Manager turns its own off or lowers its daily limit (the button still works).