Course: CI and releases — 7. Release evidence report
Twelve hands-on exercises on the release evidence report: everything the suite runs of one release say, and its recommendation, Go, Go with risks, No-go or Not enough evidence (Release evidence report):
- Module 1: the decision: nothing to go on, everything passed, a regression, re-runs;
- Module 2: comparing with the release before: known failures, fixes, a label taken from the environment, two environments;
- Module 3: which failures hold a release: a suite that did not run, every failure class, skipped tests, runs in progress and stopped;
- Module 4: sharing the report: as text, through the API and an AI assistant.
Each exercise has:
- Goal: what you will see;
- Before you start: the preparation it needs;
- Steps: Action, Expected result, and a box to tick;
- What happened: the explanation;
- Check yourself: a question to answer before moving on. The answers are at the end.
Allow about 75 minutes. Do the exercises in order: each one starts from what the one
before left. Use each release name (v1.0, v1.1…) exactly as written.
How the recommendation is decided
Section titled “How the recommendation is decided”Fixed rules, not AI: the same runs always give the same recommendation, and every reason behind it is listed.
| Decision | When |
|---|---|
| Not enough evidence | No suite run is labelled with the release. |
| No-go | A failing test whose class says the application is wrong (Server error, Maximo business rule, Authentication / permission, Wrong value) and that did not fail in the release before. Each one is a blocker. |
| Go with risks | No blocker, but something to accept knowingly (a risk): a known failure, a failure where the test could not check the application (Locator, Test data, Environment, Timeout, Unclassified), a suite that passed only on a re-run, a suite with Ready tests that did not run, skipped tests, a stopped run, runs still in progress. |
| Go | None of the above. |
Two ideas make it work:
- The run that counts is, for each suite and environment, the latest run of the release. Earlier ones are attempts, and a failure followed by a pass is named, not hidden.
- The release before is, for each suite and environment, the latest finished run of another release that started before this release’s first run there. Each failing test is then a Regression (it passed there), Known (it failed there too) or New (no result there).
A recommendation is advice: the person releasing decides. The report gives them the reasons.
Preparation
Section titled “Preparation”The exercises use the Maximo-like demo application of the failure classification exercises. The browser that runs your tests has to reach it, so do these exercises on your own installation (Deployment).
P1: Start the demo application
Section titled “P1: Start the demo application”In a PowerShell window in the AutoTestX folder:
node examples/failure-classification/app.mjsIt prints demo application on http://127.0.0.1:4430/. Leave it running. Its page /flip
works or fails as you set it. Keep these two addresses open in a browser tab:
P2: Start AutoTestX with its own data
Section titled “P2: Start AutoTestX with its own data”In a second PowerShell window, so your normal data is not touched:
npm run buildnpm run build:web$env:ATX_DATA_DIR = ".data-release-training"; npm run startOpen the link printed after set the system administrator’s password, choose a password and sign in as the system administrator.
P3: One test, two environments, a token
Section titled “P3: One test, two environments, a token”- Test Cases → Import only
examples/failure-classification/17-flip.json. The test Inventory loads lands in the suite Classification history, Ready. The other 16 files come in Exercise 8. - Configurations → Environments → New environment: name
Demo Maximo, Variables (JSON){ "baseUrl": "http://127.0.0.1:4430" }, Save environment. Then a second one,Demo Maximo EU, with the same variables. - Configurations → Project Members → API tokens for CI → create a token
Pipeline - releasesthat can Run tests. Copy its value: it is shown only once.
Already have the suite Failure classification? If you did the failure classification exercises with the same data folder, or imported all 17 files, the project already has that suite with Ready tests. Every release then also gets the risk 1 suite with Ready tests did not run for this release: Failure classification. until Exercise 9 runs it. That is the rule of Exercise 8 working, not a fault: where an exercise expects Go, you see Go with risks with only that risk, and
"go"in the API reads"go-with-risks". Skip step 1 of Exercise 8. For results exactly as written, start again with a new data folder (P2) and import only17-flip.json.
P4: Play the pipeline
Section titled “P4: Play the pipeline”A pipeline tells AutoTestX which release it is testing. In a third PowerShell window, paste this with your token:
$env:ATX_TOKEN = 'atx_pat_PASTE_THE_TOKEN'function atxrun([string]$Suite, [string]$Release, [string]$Environment = 'Demo Maximo', [switch]$Ready) { $extra = @(); if ($Ready) { $extra += '--ready' } node --import tsx packages/cli/src/index.ts run --suite $Suite --server http://127.0.0.1:4100 --env $Environment --release $Release @extra}atxrun "Classification history" v1.0 runs that suite on Demo Maximo as release v1.0. A
failed suite ends with exit code 1: that is expected.
Where you read the report
Section titled “Where you read the report”Test Suites → scroll to Recent suite runs → the release filter (it starts on All releases) → choose the release. The report appears above the list:
GO WITH RISKS No new application failure was found, but 1 risk … [Copy as text]Risks to accept • 1 test also failed in the release before (known failures): "Inventory loads".0 of 1 suite passed · 1 test: 0 passed, 1 failed · compared with v1.2Likely causes [1 × Server error GET …/api/inventory]Suite Status Result EnvironmentClassification history FAIL 0 / 1 passed Demo MaximoFailing tests [Known] Inventory loads Classification history · Demo Maximo [Server error 85%]Reload the page (F5) after a run from PowerShell, so its release is in the filter.
Module 1: The decision
Section titled “Module 1: The decision”Exercise 1: Nothing to go on, then everything passed
Section titled “Exercise 1: Nothing to go on, then everything passed”Goal: see the two ends of the scale: no evidence at all, and a clean Go.
Before you start: P1–P4.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open http://127.0.0.1:4100/api/releases/v0.9/report |
"decision":"insufficient", No suite has run against this release yet, so there is nothing to recommend from. |
|
| 2 | Open good. In the pipeline window: atxrun "Classification history" v1.0 |
PASSED | |
| 3 | Reload Test Suites, choose v1.0 in Recent suite runs | Green GO: Every suite passed: 1 test in 1 suite, nothing skipped or missing. | |
| 4 | Read the totals line | 1 of 1 suite passed · 1 test: 1 passed · no earlier release to compare with | |
| 5 | Choose All releases | The report disappears |
What happened: with no run labelled v0.9 there is nothing to judge, and the report says so rather than guessing. For v1.0 every suite with Ready tests ran and passed. It is the first release, so there is nothing to compare with yet.
Check yourself: why is “not enough evidence” not the same as “go”?
Exercise 2: A regression holds the release
Section titled “Exercise 2: A regression holds the release”Goal: see a failure the release before did not have block the release.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open bad. Run atxrun "Classification history" v1.1 |
FAILED | |
| 2 | Reload, choose v1.1 | Red NO-GO: Hold the release: 1 problem in the application that the release before did not have. | |
| 3 | Read Blockers | 1 test failing with server error (passed in the release before): “Inventory loads”. | |
| 4 | Read the totals and Failing tests | … · compared with v1.0. A red Regression mark, the test, and a red Server error chip | |
| 5 | Hover over Regression | Passed in the release before, fails now |
What happened: the test passed in v1.0 and fails in v1.1 with a server error: the application answered 503. That is the clearest reason to hold a release, a problem in the application that this version brought in.
Check yourself: where would you look first to find what broke?
Exercise 3: Re-running a suite
Section titled “Exercise 3: Re-running a suite”Goal: see that the latest run counts, and that a pass after a failure is named.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Still bad. Run atxrun "Classification history" v1.1 again. Reload, choose v1.1 |
Still NO-GO. The suite row reads latest of 2 runs | |
| 2 | Open good. Run it a third time. Reload, choose v1.1 | Amber GO WITH RISKS | |
| 3 | Read Risks to accept and the suite row | Classification history on Demo Maximo failed first and passed on a later run: possibly flaky. The row: passed on run 3 |
What happened: each run of a suite on an environment replaces the one before as the run that counts, so a fix deployed under the same release name clears the blocker. But two failures and then a pass is also what a flaky test looks like, so the report does not let it pass quietly: it becomes a risk someone has to accept.
Check yourself: a team re-runs a red suite until it is green. What does the report show?
Module 2: Comparing with the release before
Section titled “Module 2: Comparing with the release before”Exercise 4: A known failure does not block
Section titled “Exercise 4: A known failure does not block”Goal: see the same failure block one release and only be a risk in the next.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open bad. Run atxrun "Classification history" v1.2. Choose v1.2 |
NO-GO, compared with v1.1, marked Regression | |
| 2 | Still bad. Run atxrun "Classification history" v1.3. Choose v1.3 |
GO WITH RISKS, compared with v1.2 | |
| 3 | Read Risks to accept and Failing tests | 1 test also failed in the release before (known failures): “Inventory loads”. A grey Known mark |
What happened: v1.1’s run that counts is its last one, which passed, so the failure in v1.2 is a regression. In v1.3 the same failure was already in v1.2: this release did not bring it in, and holding v1.3 would not remove it. It stays visible as a risk.
Check yourself: a server error has failed for three releases. Why is it a risk and not a blocker, and what should the team do about it?
Exercise 5: A fix
Section titled “Exercise 5: A fix”Goal: see a test that failed before and passes now.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open good. Run atxrun "Classification history" v1.4. Choose v1.4 |
Green GO | |
| 2 | Read below the suite table | Fixed since v1.3: Inventory loads. |
What happened: the report compares passes too: a release that fixes something says so, which is worth telling the people who asked for the fix.
Check yourself: where does a fixed test appear in the API’s answer?
Exercise 6: A label taken from the environment
Section titled “Exercise 6: A label taken from the environment”Goal: see a run started from the screen count for a release.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | In the pipeline window: node --import tsx packages/cli/src/index.ts deployed --env "Demo Maximo" --server http://127.0.0.1:4100 --release v1.5 |
It confirms v1.5 on Demo Maximo | |
| 2 | Still good. Test Suites → Run suites against = Demo Maximo → Run on Classification history | PASS. The run panel shows Deployed v1.5 · … | |
| 3 | Reload, choose v1.5 | GO, compared with v1.4 |
What happened: a deploy job recorded what it put on Demo Maximo. A run there that says nothing about a release, from a person or a schedule, is labelled with it, so a nightly schedule adds evidence to the release without any pipeline.
Check yourself: your nightly schedule runs on Staging. What does a deploy job have to do so those runs count for the release it deployed?
Exercise 7: Two environments
Section titled “Exercise 7: Two environments”Goal: see each environment judged on its own, and narrow the report to one.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Still good. Run atxrun "Classification history" v1.6 |
PASSED | |
| 2 | Open bad. Run atxrun "Classification history" v1.6 -Environment "Demo Maximo EU". Then open good again |
FAILED | |
| 3 | Reload, choose v1.6 | NO-GO. Two rows: Demo Maximo PASS, Demo Maximo EU FAIL | |
| 4 | Read Blockers and Failing tests | 1 test failing with server error (are new): “Inventory loads”., marked New, Classification history · Demo Maximo EU | |
| 5 | Open …/api/releases/v1.6/report?environment=Demo%20Maximo |
"decision":"go" |
What happened: each suite is compared with the same suite on the same environment. The
suite had never run on Demo Maximo EU, so its failure is New, not a regression. A new
failure caused by the application still blocks: nothing shows it was there before. The
environment filter answers “can we release to this environment?”.
Check yourself: why is a new failure of class Server error a blocker, and a new Locator failure only a risk?
Module 3: Which failures hold a release
Section titled “Module 3: Which failures hold a release”Exercise 8: A suite that did not run
Section titled “Exercise 8: A suite that did not run”Goal: see missing evidence become a risk.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Test Cases → Import files 01-… to 16-… from examples/failure-classification/ |
16 new tests in the suite Failure classification, all Ready | |
| 2 | Reload Test Suites, choose v1.6 | Still NO-GO, and now Risks to accept: 1 suite with Ready tests did not run for this release: Failure classification. | |
| 3 | Open …/api/releases/v1.6/report?environment=Demo%20Maximo |
"decision":"go-with-risks" and "notRun":["Failure classification"] |
What happened: every suite with Ready tests is expected to run for a release. One that did not is a part of the application nobody checked. That is not a failure, but nobody should release without knowing about it.
Check yourself: a suite has only Draft tests. Is it expected to run?
Exercise 9: Every failure class
Section titled “Exercise 9: Every failure class”Goal: see which classes hold a release, and which are named as risks.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Still good. Run atxrun "Classification history" v2.0, then atxrun "Failure classification" v2.0 (about a minute) |
PASSED, then FAILED | |
| 2 | Reload, choose v2.0 | NO-GO: Hold the release: 4 problems in the application that the release before did not have. | |
| 3 | Read Blockers | Four lines: authentication / permission (2 tests), wrong value (1), server error (3), maximo business rule (2), each (are new) with the tests named | |
| 4 | Read Risks to accept | Five lines … could not check the application (…): test data, locator (3 tests), environment, timeout, unclassified. Each ends Fix the test or its setup, or check by hand. | |
| 5 | Read Likely causes and Failing tests | The first chip 3 × Locator. 15 failing tests, all New. Create work orders from a table is not a link: it never got a run |
What happened: four classes say the application gave a wrong answer: an error from the server, a Maximo business rule, a refused session, a wrong value. Those block. The other five say the test could not tell: the page changed (locator), there was no data, the host was unreachable, a step was too slow, or nothing matched. The release may be fine, but nobody checked those parts, so each is a risk with what to do about it.
Check yourself: the only failures of a release are three Locator failures. Is it safe to release?
Exercise 10: Skipped tests
Section titled “Exercise 10: Skipped tests”Goal: see a Ready-only run skip a Draft, and old failures stay risks.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Test Cases → open Start Center opens → New revision. Leave it a Draft | Its status is Draft | |
| 2 | Run atxrun "Failure classification" v2.1 -Ready |
Start Center opens is skipped. FAILED | |
| 3 | Reload, choose v2.1 | GO WITH RISKS: … but 3 risks should be accepted before releasing. | |
| 4 | Read Risks to accept | 15 tests also failed in the release before (known failures): “Admin settings (API)”, “Apply discount code”, “Asset list” and 12 more. · 1 suite with Ready tests did not run for this release: Classification history. · 1 test skipped (not Ready, or deprecated). |
What happened: a Ready-only run, what a deploy gate usually runs, leaves out tests that are being changed, and the report counts them as unchecked. The same server errors that held v2.0 are now Known: v2.1 did not bring them in.
Check yourself: v2.1 has the same server errors as v2.0. Why does one get No-go and the other Go with risks?
Exercise 11: Still running, then stopped
Section titled “Exercise 11: Still running, then stopped”Goal: see the report on a release whose evidence is not complete.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Run atxrun "Failure classification" v2.2. While it runs, reload Test Suites and choose v2.2 |
The suite row is RUNNING. A risk: Suite runs are still in progress: this report covers the tests finished so far. | |
| 2 | Click the Failure classification row in the report. In the run panel above, click Stop | Stopping: queued tests are cancelled, running ones will finish. | |
| 3 | When the pipeline window has finished, reload and choose v2.2 | The row is CANCELLED. A risk: Failure classification was stopped before it finished. |
What happened: the report can be read at any time, but it never presents half the evidence as the whole: a run in progress and a stopped run are each named as a risk.
Check yourself: why is a stopped run a risk even when every test that ran passed?
Module 4: Sharing the report
Section titled “Module 4: Sharing the report”Exercise 12: As text, through the API, to an AI assistant
Section titled “Exercise 12: As text, through the API, to an AI assistant”Goal: take the report to where the release decision is made.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Choose v2.0. Click Copy as text and paste into Notepad | Release v2.0: No-go, the headline, the totals, Blockers:, Risks: and Suites: with a link for each suite | |
| 2 | Open …/api/releases/v1.1/report |
"previousRelease":"v1.0", and in suites "attempts":3, "passedOnRerun":true |
|
| 3 | Start the MCP Inspector (npx @modelcontextprotocol/inspector): Streamable HTTP, URL http://127.0.0.1:4100/api/mcp, header Authorization = Bearer + your token. Connect → List Tools |
release_report is listed, read-only |
|
| 4 | Run release_report with release = v2.0 |
No-go for v2.0: Hold the release: 4 problems …, then - Blocker: … and - Risk: … lines, then the full report | |
| 5 | Run it with release = v1.6 and environment = Demo Maximo EU |
No-go for v1.6 on Demo Maximo EU: … |
What happened: the same report is a paragraph for a release ticket, JSON for a pipeline that decides whether to promote a build, and a tool an assistant calls when someone asks “Is v2.0 ready to release?”. All three read the same rules, so they always agree.
Check yourself: a pipeline wants to stop a promotion on No-go only. Which field does it read?
Clean up
Section titled “Clean up”Stop the windows (Ctrl+C) and delete the practice data:
Remove-Item -Recurse -Force .data-release-trainingAnswers
Section titled “Answers”- Go means the evidence was checked and is clean; not enough evidence means nothing was checked at all. Releasing on no evidence is a guess.
- At what changed between v1.0 and v1.1 behind
GET /api/inventory: the request is on the failing test and in Likely causes. - The latest run, passed on run N, and the risk possibly flaky: the earlier failures stay on record.
- This release did not bring it in, so holding it would not remove the problem. The team should fix it or accept it on purpose: a known failure is not a harmless one.
- In
fixed, with"change":"fixed". - Record the deploy with
atx deployed --env Staging --release …(or run the suite with--release): later runs there are labelled with it. - A server error is the application giving a wrong answer. A locator failure means the test could not reach what it checks: the application may be fine, but nobody knows.
- Not without a check: those parts of the application were not tested. Fix the locators or check those pages by hand, then accept the risk.
- No: only suites with at least one Ready test are expected.
- v2.0 has no earlier run of Failure classification, so its failures are new; in v2.1 they also failed in v2.0, so they are known.
- The tests that never ran were not checked: the report would otherwise look cleaner than the evidence is.
recommendation.decision: it is"no-go"exactly when there is a blocker.