Course: CI and releases — 6. Release metadata

Fourteen hands-on exercises on labelling suite runs with the version they tested (Run after deployment):

• Module 1: a pipeline names its release, build and commit on a suite run;
• Module 2: what the CLI reads from the CI system on its own, and what it never guesses;
• Module 3: finding the runs of one release, in the UI and through the API;
• Module 4: an environment remembers what is deployed to it, and every run there says so;
• Module 5: the failure email names the release.

Each exercise has:

• Goal: what you will see;
• Before you start: the preparation it needs;
• Steps: Action, Expected result, and a box to tick;
• What happened: the explanation;
• Check yourself: a question to answer before moving on. The answers are at the end.

Allow about 1 hour. Do the exercises in order: each one starts from what the one before left.

The two labels you will meet

On a suite run	Means
Deploy v2.14.0 · …	The run itself said which deploy it tests: a pipeline sent it.
Deployed v2.15.0 · … · recorded 5m ago	The run said nothing, so it took what was last deployed to its environment.

---

Preparation

P1: Accounts and plan

You need to be Admin or Test Manager in a project on the Team plan (API tokens need it): Organization → Settings → the plan card → Change plan → Team → Save.

P2: Suites, environments and practice tests

1. Test Suites → add the suites Release and Checkout.
2. Configurations → Environments → add Staging, and a second one named QA EU (with the space).
3. Test Cases → Import the two files in examples/release-metadata/. They open a page built into the test itself, so they need no network and always give the same result:

File	Test	Suite	Result
01-home-page.json	Home page shows Save	Release	always passes
02-checkout-broken.json	Checkout shows Pay (always fails)	Checkout	always fails

P3: Two API tokens

Configurations → Project Members → API tokens → create two tokens and copy each value: it is shown only once. The It can field starts on Run tests: change it for the second token.

Name	It can
Pipeline - release	Run tests
Read only - release	Read results only

P4: The pipeline window

Open a PowerShell window in the AutoTestX folder and paste this, with your server address and the two token values. It adds atxcli, which runs the AutoTestX command line and prints its exit code, and atx, which calls the API the way a pipeline does.

$env:ATX_SERVER = 'https://app.autotestx.com'
$env:ATX_TOKEN  = 'atx_pat_PASTE_THE_RUN_TESTS_TOKEN'
$VIEWER         = 'atx_pat_PASTE_THE_READ_ONLY_TOKEN'
function atxcli { node --import tsx packages/cli/src/index.ts @args; Write-Host "exit code $LASTEXITCODE" -ForegroundColor Cyan }
function atx([string]$Method, [string]$Path, $Body = $null, [string]$Token = $env:ATX_TOKEN) {
  $p = @{ Method = $Method; Uri = "$env:ATX_SERVER$Path"; Headers = @{ Authorization = "Bearer $Token" }; UseBasicParsing = $true }
  if ($Method -ne 'GET') { $p.ContentType = 'application/json'; $p.Body = if ($Body) { $Body | ConvertTo-Json -Depth 5 } else { '{}' } }
  try { $r = Invoke-WebRequest @p; Write-Host "HTTP $($r.StatusCode)" -ForegroundColor Green; $r.Content | ConvertFrom-Json | ForEach-Object { $_ } }
  catch { Write-Host "HTTP $([int]$_.Exception.Response.StatusCode)" -ForegroundColor Red; $_.ErrorDetails.Message }
}
atx GET /api/tokens/self
atx GET /api/tokens/self -Token $VIEWER

The first answer must show Pipeline - release with role=tester, the second Read only - release with role=viewer. If the second says tester, create that token again with Read results only.

Two traps in PowerShell. Do not call the helper cli: PowerShell already has a cli alias (Clear-Item), and an alias wins over a function. And NAME=value command is bash, not PowerShell: set a variable with $env:NAME = 'value' on its own.

---

Module 1: A pipeline names its deploy

Exercise 1: Label a suite run with a release, build and commit

Goal: a suite run says exactly which version it tested.

Before you start: P1-P4.

#	Action	Expected result	✓
1	atxcli run --suite Release --release v2.14.0 --build 4812 --commit a1b2c3d4e5f6 --branch main	The second line reads deploy: v2.14.0 · main @ a1b2c3d4e5f6 · build 4812. The test passes; the last line says PASSED; exit code 0
2	Test Suites → Recent suite runs	The run is listed with a v2.14.0 badge beside the suite name, and marked deploy
3	Hover over deploy	Started by a deploy pipeline (v2.14.0, commit a1b2c3d4e5f6)
4	Click the run	The panel shows Deploy v2.14.0 · main @ a1b2c3d4e5f6 · build 4812

What happened: the CLI sent the details with the run, and they are kept on it for good. The release comes first because it is the name people use; the commit and build point at the exact code.

Check yourself: the CLI printed the deploy: line before any test ran. Why is that useful?

Exercise 2: The release from a variable

Goal: a pipeline names the release without changing its command.

#	Action	Expected result	✓
1	$env:ATX_RELEASE = 'v2.14.1'	Nothing is printed
2	atxcli run --suite Release	deploy: v2.14.1. Passes
3	Remove-Item Env:ATX_RELEASE	Nothing is printed
4	Recent suite runs	The newest run has a v2.14.1 badge

What happened: ATX_RELEASE does what --release does. A pipeline usually knows its version in a variable already (ATX_RELEASE: ${{ needs.build.outputs.version }} in GitHub Actions).

Check yourself: why remove the variable in step 3?

---

Module 2: What the CLI reads from CI

Each command below pretends to be a GitHub Actions job by setting GitHub’s own variables, and removes them again on the same line. Run each line whole.

Exercise 3: A tag build names its release

Goal: a build of a git tag is labelled with the tag, with nothing typed.

#	Action	Expected result	✓
1	$env:GITHUB_ACTIONS='true'; $env:GITHUB_REF_TYPE='tag'; $env:GITHUB_REF_NAME='v3.0.0'; $env:GITHUB_SHA='cafe1234567890'; atxcli run --suite Release; Remove-Item Env:GITHUB_*	deploy: v3.0.0 · github-actions · v3.0.0 @ cafe12345678. Passes
2	Open the run in Recent suite runs	Deploy v3.0.0 · github-actions · v3.0.0 @ cafe12345678

What happened: a tag is a deliberate, one-off name for one commit, so it is safe to use as the release. The commit and the source come from GitHub’s variables too.

Check yourself: which other CI systems are read the same way?

Exercise 4: A branch is not a release

Goal: see that the CLI never guesses a release.

#	Action	Expected result	✓
1	$env:GITHUB_ACTIONS='true'; $env:GITHUB_REF_TYPE='branch'; $env:GITHUB_REF_NAME='release/2.14'; $env:GITHUB_SHA='cafe1234567890'; atxcli run --suite Release; Remove-Item Env:GITHUB_*	deploy: github-actions · release/2.14 @ cafe12345678: no release
2	Recent suite runs	The run is marked deploy but has no release badge

What happened: a branch like release/2.14 builds many commits before 2.14 ships. Labelling them all “2.14” would make “it passed on 2.14” meaningless, so a branch is shown as a branch only.

Check yourself: this pipeline does want the label 2.14 on its runs. What is the one-line fix?

Exercise 5: Send nothing

Goal: turn the automatic details off.

#	Action	Expected result	✓
1	$env:GITHUB_ACTIONS='true'; $env:GITHUB_SHA='cafe1234567890'; $env:GITHUB_REF_NAME='main'; atxcli run --suite Release --no-deployment; Remove-Item Env:GITHUB_*	deploy: none sent (…). Passes
2	Get-ChildItem Env:GITHUB_*	Nothing is listed (or Cannot find path): no GitHub variable is left
3	Recent suite runs	The run has no badge and is marked CI, not deploy

What happened: --no-deployment ignores what the CI system says. Step 2 matters: a variable left over in the window would make every later run look like a GitHub build.

Check yourself: a run is marked CI instead of deploy. What does that tell you?

---

Module 3: Finding the runs of one release

Exercise 6: Filter the history by release

Goal: see only the runs that tested one release.

#	Action	Expected result	✓
1	Recent suite runs → open All releases	v3.0.0, v2.14.1, v2.14.0: newest first
2	Pick v2.14.0	Only the Exercise 1 run is listed
3	Pick All releases again	Every run is back

What happened: the filter lists only releases that have runs. Together with the suite filter, it answers how did Release do on v2.14.0?

Check yourself: why is there no v2.14 in the list after Exercise 4?

Exercise 7: Ask the API

Goal: a pipeline or a script finds the same runs.

#	Action	Expected result	✓
1	atx GET "/api/executions?release=V2.14.0" | Select-Object suite, status, deployment	HTTP 200. The Exercise 1 run: upper or lower case does not matter
2	atx GET "/api/executions?commit=a1b2c3d"	The same run: a 7-character short commit matches the full one
3	atx GET "/api/executions?commit=a1b2"	HTTP 200 and an empty list
4	atx GET "/api/executions?suite=Release&release=v3.0.0"	Only the Exercise 3 run: filters combine
5	atx GET "/api/executions?release=v2.14.0" -Token $VIEWER	HTTP 200, the same run: a read-only token can read

What happened: release and commit are the two details made to be searched. The others (source, branch, note, link) are shown, never searched.

Check yourself: why is a1b2 refused while a1b2c3d is accepted?

---

Module 4: What is deployed on an environment

Exercise 8: Record a deploy without running tests

Goal: a deploy job tells AutoTestX what it put on Staging.

#	Action	Expected result	✓
1	Configurations → Environments	Neither environment shows a release badge
2	atxcli deployed --env Staging --release v2.15.0 --commit 1234567abcdef --branch main	recorded v2.15.0 · commit 1234567abcde on Staging, exit code 0. No test runs
3	Refresh Environments	Staging shows v2.15.0 · 1234567 · just now and a Forget deploy button
4	Hover over the badge	The date, Branch: main, Commit: 1234567abcdef, and Suite runs here that name no deploy are labelled with it.
5	atxcli deployed --env "QA EU" --commit abcdef1	recorded commit abcdef1 on QA EU. On Environments, QA EU shows abcdef1 · just now

What happened: the environment now remembers its deploy. Nothing else changed yet: the effect shows on the next suite run there.

Check yourself: atxcli deployed --env Staging alone is refused. Why?

Exercise 9: Every run there is labelled

Goal: runs that do not name a deploy say which version they tested anyway.

#	Action	Expected result	✓
1	Test Suites → Run suites against Staging → Run on Release	The panel shows Deployed v2.15.0 · main @ 1234567abcde · recorded Nm ago
2	Look at that run in Recent suite runs	Badge v2.15.0, and no marker: you started it
3	atxcli run --suite Release --env Staging	deploy: none sent (a run on an environment is labelled with what was last deployed there). Passes
4	Recent suite runs	That run shows v2.15.0 and is marked CI, not deploy
5	Schedules → New schedule: Release on Staging, any time → Create schedule → Run now	Deployed v2.15.0 · …, marked scheduled
6	Run Release against QA EU, then with No environment	QA EU: Deployed · @ abcdef1 · recorded …. No environment: no Deploy line

What happened: the label is taken when the run starts, from the environment it runs on. It says Deployed (not Deploy) and recorded … ago, because AutoTestX only knows what a pipeline last said: if nobody reported a newer deploy, the label may be out of date.

Check yourself: the nightly run says recorded 3d ago, but you know Staging was redeployed yesterday. What went wrong, and where?

Exercise 10: A pipeline run moves the environment on

Goal: a pipeline that deploys and then tests needs no extra step.

#	Action	Expected result	✓
1	atxcli run --suite Release --env Staging --release v2.16.0	deploy: v2.16.0. The panel says Deploy v2.16.0 (it named its own deploy)
2	Refresh Environments	Staging shows v2.16.0 · just now
3	Run Release on Staging from the UI	Deployed v2.16.0 · recorded …
4	Edit Staging, add the variable {"baseUrl": "https://staging.example"}, Save	The badge stays v2.16.0

What happened: a pipeline run that names its deploy and an environment has just deployed there, so it is recorded too. Editing the environment’s variables never touches it.

Check yourself: a CI job only tests Staging, without deploying, and the CLI picks up its commit from CI. What will Staging show afterwards, and how do you avoid it?

Exercise 11: Who may record a deploy

Goal: a read-only token cannot change what an environment runs.

#	Action	Expected result	✓
1	atx PUT /api/environments/Staging/deployed @{ release = 'v9' } -Token $VIEWER	HTTP 403, Your role in this project does not allow recording deploys.
2	atx PUT /api/environments/Staging/deployed @{ note = 'just a note' }	HTTP 400, Say what was deployed: at least a release or a commit, …
3	atxcli deployed --env Nowhere --release v9	Environment “Nowhere” not found in this project., exit code 2
4	Refresh Environments	Staging still shows v2.16.0

What happened: recording a deploy needs the same right as running tests: a pipeline’s normal token can do it, a read-only one cannot.

Check yourself: a token named Read only returned HTTP 200 in step 1. What do you check first?

Exercise 12: Forget a deploy

Goal: stop labelling runs with a deploy that is no longer true.

#	Action	Expected result	✓
1	Environments → Forget deploy on Staging	The badge and the button disappear
2	Run Release on Staging from the UI	No Deploy line, no badge
3	Open the run from Exercise 9, step 1	It still shows Deployed v2.15.0: earlier runs keep their labels
4	atx DELETE /api/environments/QA%20EU/deployed	HTTP 200. QA EU’s badge is gone

What happened: forgetting changes only the runs that start afterwards. Use it when an environment was rebuilt, or when the pipeline that reported on it was switched off.

Check yourself: why does a run keep its label after the environment forgets the deploy?

---

Module 5: The failure email

Pipeline alerts go to the project’s Admins and Test Managers by default. On your own server without mail settings, the server prints each email in its console window, as a box starting with ┌─ email:.

Exercise 13: The email names the release

Goal: the first email about a failing deploy says which release broke it.

#	Action	Expected result	✓
1	atxcli run --suite Checkout --env Staging --release v2.16.1	Checkout shows Pay (always fails) fails; exit code 1
2	Look at your inbox (or the server window)	Checkout is failing on Staging after a deploy of v2.16.1 (your project), with the line Deploy: v2.16.1 and the failed test

What happened: the release is in the subject, so the person who reads it knows which deploy to look at before opening anything.

Check yourself: where else in the email can a commit or a pipeline link appear?

Exercise 14: One email, not one per run

Goal: a release that stays broken does not flood the inbox.

#	Action	Expected result	✓
1	atxcli run --suite Checkout --env Staging --release v2.16.2	Fails again. No new email
2	atxcli run --suite Checkout --env Staging	Fails. No new email. The run shows Deployed v2.16.2: step 1 recorded it on Staging

What happened: you are told once when a suite starts failing on an environment, and once when it passes again, not for every failing run in between.

Check yourself: what has to happen for the next email to be sent?

---

Answers

1. Most of it may come from CI variables nobody typed. Printing it shows what will be stored before anything runs.
2. Every later run in that window would be labelled v2.14.1 too.
3. GitLab CI (CI_COMMIT_TAG), Bitbucket Pipelines (BITBUCKET_TAG) and Jenkins (TAG_NAME).
4. Add --release 2.14, or set ATX_RELEASE, in that job.
5. The run sent no deploy details: it was started with a token, but did not say what it tested.
6. Exercise 4 sent no release: a branch name is never turned into one.
7. A few characters match many commits. Seven, as git abbreviates, is enough to name one.
8. It says nothing about the version: at least a release or a commit is needed.
9. The redeploy was not reported. Add atx deployed (or --release on its test run) to that deploy job.
10. The commit of the CI job, which may not be what is deployed. Give that job --no-deployment, or let only deploy jobs report.
11. Which token was sent and its role: atx GET /api/tokens/self -Token …. A token’s name says nothing about its access.
12. The label was copied onto the run when it started. A run’s record of what it tested never changes.
13. In the Deploy: line, and a Pipeline: line with the link when the pipeline sent one.
14. Checkout must pass on Staging. You then get one passing again email.