Course: CI and releases — 3. Suite runs, schedules and failure emails

Thirty hands-on exercises on running a whole suite at once:

Module 1: running a suite from the Test Suites page, following it live, stopping it, and reading Recent suite runs;
Module 2: schedules that run a suite on their own, and what your plan allows;
Module 3: the email you get when a scheduled suite starts failing, and when it passes again;
Module 4: running a suite from a pipeline with atx run --suite;
Module 5: schedule limits in depth: pausing to make room, who may manage schedules, cron timings on Business, changing to a smaller plan, and server restarts.

Each exercise has:

Goal: what you will see;
Before you start: the preparation it needs;
Steps: Action, Expected result, and a box to tick;
What happened: the explanation;
Check yourself: a question to answer before moving on. The answers are at the end.

Allow about 2 hours. Exercises 11 and 26 wait for a real scheduled run (up to 3 and 15 minutes).

Preparation

P1: Accounts and plan

Who	Why
You, Admin or Test Manager in a project	To run suites and to create schedules
A platform administrator	Only a platform administrator can change an organization’s plan. On your own server this is the account claimed at First sign-in. If you are both, you need only one account.
A second person, Tester in the same project	Exercise 24 only. Add them as in First sign-in → Add other people, and sign them in from a private window

Put the organization on the Team plan: Organization → Settings → the plan card → Change plan → Team → Save. Team runs 2 tests at a time, allows one daily schedule per suite, and includes API tokens (Module 4).

P2: Environment

Configurations → Environments → add an environment named Staging. It needs no variables for these exercises.

P3: Practice tests

Every test below is Shared: only shared tests run in a suite. Each test opens a page and checks its address, so the result is predictable. Edit asset also waits 15 seconds, so there is time to watch a suite run and to stop it.

Name	Suite	Steps	Status
`Create asset`	`Asset`	Open `https://example.com` → URL, contains, `example.com`	Ready
`Edit asset`	`Asset`	Open `https://example.com` → Wait, ms `15000` → URL, contains, `example.com`	Ready
`Retire asset`	`Asset`	Open `https://example.com` → URL, contains, `retired` (fails on purpose)	Draft
`Count stock`	`Inventory`	Open `https://example.com` → URL, contains, `example.com`	Ready

Test Suites → add the suites Asset and Inventory.
Test Cases → New test for each row. Set the Status in the toolbar, then Save.
Run each test once from its editor. Retire asset must fail and the others must pass.

P4: Where things are on the Test Suites page

From top to bottom:

Run suites against: the environment for the Run buttons.
The suite table, with a Run button on each suite that has tests.
The suite run panel, which appears when a suite run is open: its status, a Stop button, a progress bar and one line per test.
Schedules: the project’s schedules, with New schedule.
Recent suite runs: the latest suite runs, newest first.

P5: Where the emails arrive

On the hosted service, emails go to the inbox of each person on the list.
On your own server without mail settings, the server prints each email in its console window, as a box starting with ┌─ email:. Keep that window visible during Module 3.

Module 1: Running a suite

Exercise 1: Run a suite and follow it live

Goal: run every test in a suite with one click, and read the result.

Before you start: P1-P4. Nothing is running.

#	Action	Expected result
1	Test Suites → check that Run suites against shows Staging	Each suite row with tests has a Run button
2	Click Run on Asset	The button briefly shows Starting…. A message says Running 3 tests in Asset. The panel appears: Asset on Staging · RUNNING, with a Stop button
3	Watch the panel	0 of 3 finished · up to 2 at a time. Two tests run, and the third shows QUEUED until a slot frees up
4	Wait about 20 seconds	The header shows FAIL and the bar turns red: 2 passed, 1 failed. `Retire asset` is FAIL with 1 / 2 steps

What happened: each test ran as its own run, and the suite counts as one result. A single failed test fails the suite. The plan’s 2 slots are shared, so the third test waited for one.

Check yourself: Retire asset is a Draft. Why did it run?

Exercise 2: From the suite to one test’s report

Goal: go from a failed suite to the reason a test failed.

Before you start: Exercise 1.

#	Action	Expected result
1	In the panel, click Retire asset	The run report opens. The URL step failed, with a message saying why
2	Look at the passed steps	There are no screenshots after the passing steps
3	Go back to Test Suites	The same suite run is still listed under Recent suite runs

What happened: a suite run is made of ordinary test runs, each with its own report. Suite runs keep evidence only where it matters (failures), to stay fast and small.

Check yourself: where else can you find Retire asset’s run besides the suite panel?

Exercise 3: Leave and come back

Goal: a suite run keeps going without you watching it.

#	Action	Expected result
1	Click Run on Asset	The panel shows RUNNING
2	Within a few seconds, open Test Cases, then come back to Test Suites (or reload the page)	The panel is back, following the same run
3	When it finishes, click ✕ on the panel	The panel closes. The run stays in Recent suite runs

What happened: the run lives on the server, not in your browser. The page asks for a suite run that is still going and opens it again.

Check yourself: if you close the browser during a suite run, does the run stop?

Exercise 4: Stop a suite run

Goal: stop a suite safely.

#	Action	Expected result
1	Click Run on Asset, then immediately Stop	Message: Stopping: queued tests are cancelled, running ones will finish. The header shows STOPPING
2	Look at the tests	The queued test shows CANCELLED. The ones already running continue
3	Wait until `Edit asset` finishes (about 15 seconds)	The header shows CANCELLED and the bar is grey. No test is left QUEUED or RUNNING

What happened: stopping never cuts a running test in half, so every report is complete. Only tests that had not started are cancelled.

Check yourself: after you click Stop, why might you still wait 15 seconds?

Exercise 5: Recent suite runs

Goal: find and compare earlier suite runs.

#	Action	Expected result
1	Run on Inventory, and let it finish	PASS
2	Look at Recent suite runs	Newest first. Each row has the suite, the status, a result such as 2 / 3 passed · 1 failed, the environment, the duration and the start time. The stopped run shows CANCELLED
3	Click an older Asset row	It opens in the panel above, and the row is highlighted
4	In the card header, choose Inventory in the suite picker	Only Inventory runs are listed. Choose All suites to see everything again
5	Click Run on Asset and watch the list	The new run appears at the top as RUNNING and changes to its result by itself

What happened: the list is the suite’s history. It is the quickest way to see whether a suite has been stable.

Check yourself: what would the result column say for a suite with 3 tests where 1 was Deprecated and the other 2 passed?

Module 2: Schedules

Exercise 6: Create a schedule and read its preview

Goal: schedule a suite and check when it will run before saving.

Before you start: P1-P3. The Asset suite has no schedule yet.

#	Action	Expected result
1	In Schedules, click New schedule	The form Schedule a suite opens: Suite, Environment, Runs, At, Time zone (yours), a preview, Email on failure with you ticked, and the note Scheduled runs include Ready tests only
2	Suite Asset, Environment Staging, At 02:00	Preview: Every day at 02:00 (your zone) and First run in …, with the next dates in your own time
3	Change At to the same time PM	The countdown jumps by 12 hours
4	Set it back to 02:00 AM. Change Time zone to `Asia/Kolkata`	The description names Asia/Kolkata, and the dates move: they are always shown in your time. Set the time zone back to yours
5	Create schedule	The form closes, and a row appears: Every day at 02:00, Alerts: you, Staging, the next run with a countdown, Not run yet

What happened: the server works out the run times, including your time zone and daylight saving. The countdown is the quickest check that AM and PM are right.

Check yourself: it is 12:42 AM and you want a run in two minutes. The preview says First run in 12 h 2 min. What is wrong?

Exercise 7: What your plan allows

Goal: see the Team plan’s schedule limits.

Before you start: Exercise 6.

#	Action	Expected result
1	New schedule → look at Runs	Custom (cron) is greyed out with a Business badge
2	Choose suite Asset → Create schedule	The form stays open: This plan allows 1 schedule per suite, and “Asset” already has one. Change or pause the existing schedule, or upgrade for more.
3	Change the suite to Inventory → Create schedule	It is created: each suite has its own limit
4	Delete the Inventory schedule → confirm	It is removed

What happened: Team schedules a suite once a day. Cron timings, and more than one schedule per suite, come with Business.

Check yourself: what does Business add for schedules?

Exercise 8: Pause, resume and edit

#	Action	Expected result
1	Pause on the Asset schedule	Schedule paused. The next run shows Paused, and the row is greyed out
2	Resume	Schedule resumed. The next run and its countdown are back
3	Edit → At 03:30 → Save	Schedule saved. The row shows Every day at 03:30

What happened: pausing keeps the schedule and its history but stops it running. A paused schedule does not count against your plan’s limit.

Check yourself: why is resuming checked against the plan, just like creating?

Exercise 9: Run now, and a run that is still going

Goal: start a schedule’s run on demand, and see that runs never overlap.

#	Action	Expected result
1	Run now on the Asset schedule	Running Asset now. The panel opens on the new run
2	Click Run now again while it is still running	Refused: The last run of this schedule is still going. The panel shows the run that is going
3	Wait for it to finish	The schedule’s Last run shows the result and just now. Click it to open the run

What happened: one schedule never runs twice at the same time. If a run is still going when the next one is due (a slow nightly suite, say), the due run is skipped.

Check yourself: what does Recent suite runs show next to a run started by a schedule?

Exercise 10: Scheduled runs include Ready tests only

Goal: see why a Draft never fails a scheduled run.

Before you start: Exercise 9. Retire asset is still a Draft.

#	Action	Expected result
1	Open the run from Exercise 9	PASS: 2 passed, 1 skipped
2	Look at `Retire asset`	SKIP, with the reason Not Ready (this run includes Ready tests only)
3	Compare with Exercise 1	The same suite failed from the Run button, because manual runs include Drafts

What happened: a schedule is your safety net. Tests still being written stay out of it until someone marks them Ready, but they are listed, so nothing is hidden.

Check yourself: what does a scheduled run report when every test in the suite is a Draft?

Exercise 11: A real scheduled run

Goal: watch a schedule run on its own.

#	Action	Expected result
1	Note the time. Edit the Asset schedule: At = now + 2 minutes. Check AM/PM	The preview says First run in 2 min. If it says in 12 h, fix AM/PM
2	Save	The next run shows today, with in 2 min under it
3	Wait until that time plus up to 30 seconds, then reload the page	Recent suite runs has a new row marked scheduled, PASS. The schedule’s next run has moved to tomorrow

What happened: the server checks for due schedules every 30 seconds and starts them by itself. Nobody has to be signed in.

Check yourself: the server was restarted and was down at the scheduled time for 10 minutes. What happens?

Exercise 12: What a schedule protects

#	Action	Expected result
1	Configurations → Environments → Delete Staging → confirm	Refused: 1 scheduled run uses this environment (“Asset”). Change or delete it first. Staging is still there
2	Test Suites → Rename `Asset` to `Assets` → Save	The schedule now shows Assets
3	Rename it back to `Asset`	The schedule follows again

What happened: a schedule would fail every night without its environment, so the delete is refused rather than letting it break silently. A rename carries the schedules along.

Check yourself: you really need to delete Staging. What do you do first?

Module 3: Failure emails

A schedule emails its Email on failure list when it changes state: once when it starts failing, and once when it passes again. These exercises use Run now, which counts exactly like the schedule’s own runs.

Exercise 13: Who gets the email

#	Action	Expected result
1	Look at the Asset schedule row	Alerts: followed by your name
2	Edit → look at Email on failure	The project’s people, with you ticked, and the hint One email when the suite starts failing, one when it passes again - not one for every failing run.
3	Cancel	—

What happened: a new schedule alerts the person who creates it. Only people in the project can be on the list.

Check yourself: a colleague on the list leaves the project. Do they still get the emails?

Exercise 14: The schedule starts failing

Before you start: P5. Open Retire asset → set Status to Ready → Save.

#	Action	Expected result
1	Run now on the Asset schedule, and wait for the result	FAIL: 2 passed, 1 failed
2	Check your inbox (or the server console)	One email: Asset is failing on Staging (your project)
3	Read it	The run time in the schedule’s time zone. Result: 2 of 3 passed, 1 failed. Under Failed test:, Retire asset, the step that failed and why, and a link to its report. Open the run: a link to the whole suite run
4	Open the Open the run link	The Test Suites page opens with that run in the panel

What happened: the schedule went from passing to failing, which is news, so it emailed.

Check yourself: why does the email name the step and not only the test?

Exercise 15: It keeps failing: no more emails

#	Action	Expected result	✓
1	Run now again, and wait	FAIL again
2	Check for email	No new email

What happened: you already know it is broken. A suite failing every night for a week sends one email, not seven, so the emails stay worth reading.

Check yourself: how do you see that it is still failing, without an email?

Exercise 16: It passes again

#	Action	Expected result
1	Open `Retire asset` → New revision	It becomes a Draft again
2	Run now, and wait	PASS: 2 passed, 1 skipped
3	Check for email	One email: Asset is passing again on Staging, with It had been failing since … (2 runs)
4	Run now once more	PASS, and no email

What happened: recovering is also a change of state, so everyone on the list learns that the problem is over, and how long it lasted.

Check yourself: how many emails did Exercises 14-16 send in total, for 4 runs?

Exercise 17: Nothing to run is a failure too

#	Action	Expected result
1	Make `Create asset` and `Edit asset` Drafts (New revision on each)	All three Asset tests are Drafts
2	Run now, and wait	ERR. The last run reads No Ready tests in this suite
3	Check for email	One failing email, with the note No Ready tests in this suite
4	Mark `Create asset` and `Edit asset` Ready again, then Run now	PASS, and a passing again email

What happened: a schedule that checks nothing is broken, even if no test failed. The same applies when a schedule cannot start at all, for example because its project no longer exists.

Check yourself: which runs never change the state, and so never send email?

Exercise 18: Nobody on the list

#	Action	Expected result
1	Edit → untick everyone under Email on failure	The hint says Nobody is emailed about this schedule.
2	Save	The row shows No failure alerts
3	Mark `Retire asset` Ready, then Run now	FAIL, and no email
4	Edit → tick yourself again → Save. Then New revision on `Retire asset`	Alerts: your name. `Retire asset` is a Draft

What happened: the list decides who hears. The schedule still records every run.

Check yourself: in step 4 you were added back while the schedule was failing. When will you get your next email?

Module 4: Suites from a pipeline

atx run --suite asks the server to run a suite, prints each test as it finishes, and ends with an exit code that a CI pipeline can act on. See the CLI.

Exercise 19: A token and a passing suite

Before you start: P1 (Team includes API tokens).

#	Action	Expected result
1	Configurations → Project Members → API tokens → create a token with role Tester	The token is shown once. Copy it
2	In PowerShell, in the project folder: `$env:ATX_SERVER = "<your server address>"` and `$env:ATX_TOKEN = "<the token>"`	—
3	`npm run atx '--' run --suite inventory --env staging --json inv.json`	`[PASS] Count stock`, then PASSED suite Inventory
4	`$LASTEXITCODE`	`0`. The file `inv.json` holds the full result
5	Test Suites → Recent suite runs	The run is listed, marked CI

What happened: the pipeline needs no browser. The server runs the tests, with its own environments and secrets.

Check yourself: why is -- written in quotes in PowerShell?

Exercise 20: A failing suite fails the pipeline

Before you start: Retire asset is Ready.

#	Action	Expected result
1	`npm run atx '--' run --suite Asset --env Staging`	`[FAIL] Retire asset` with a link to its report, then FAILED suite Asset
2	`$LASTEXITCODE`	`1`
3	Make `Retire asset` a Draft again (New revision)	—

What happened: exit code 1 means “the application broke”: the pipeline step fails. Pipeline runs include Drafts, like the Run button.

Check yourself: what exit code means “the command itself was wrong”?

Exercise 21: Mistakes the CLI catches

#	Action	Expected result
1	`npm run atx '--' run --suite Nowhere`	The suite “Nowhere” has no shared tests to run. Exit code `2`
2	`npm run atx '--' run --suite Asset --env prod`	Environment “prod” not found in this project. Exit code `2`
3	`npm run atx '--' run --suite Asset --token abc`	Refused: the token belongs in `ATX_TOKEN`, not on the command line. Exit code `2`
4	`npm run atx -- run --suite Asset` (no quotes)	npm kept –suite for itself, so atx never received it. with the two forms that work. Exit code `2`
5	`$env:ATX_TOKEN = ""`, then the command from Exercise 19	set ATX_TOKEN to an API token (Project → API tokens). Exit code `2`. Set the token again afterwards

What happened: exit code 2 means “fix the command”. It is never confused with a failing suite.

Check yourself: why may a token not be given with --token?

Exercise 22: Interrupting a pipeline

#	Action	Expected result	✓
1	Start `npm run atx '--' run --suite Asset`, and press Ctrl+C within a second	interrupted: stopping the suite on the server…
2	Test Suites → Recent suite runs	That run ends CANCELLED, and nothing is left running

What happened: a cancelled pipeline stops its suite on the server too, the same way as Stop on the page.

Check yourself: what happens to a test that was already running when you pressed Ctrl+C?

Module 5: Schedule limits in depth

These exercises go further into what a plan allows. The organization is on Team (P1), the Asset schedule exists, and Retire asset is a Draft.

Exercise 23: Pause to make room

Goal: see that only active schedules count against the plan, and that resuming is checked.

#	Action	Expected result
1	Pause on the Asset schedule	Schedule paused
2	New schedule → suite Asset, At 14:00 → Create schedule	Created: the paused schedule does not count. Asset has two rows, one paused
3	Resume on the paused row	Refused: This plan allows 1 schedule per suite, and “Asset” already has one… It stays paused
4	Delete the 14:00 schedule → confirm. Then Resume the first one	Schedule resumed. The next run is back

What happened: your plan counts active schedules only, so pausing is how you make room. Resuming is checked like creating, which answers the question in Exercise 8.

Check yourself: on Team, Asset has one active and two paused schedules. Can you create another Asset schedule?

Exercise 24: Who may manage schedules

Before you start: the Tester from P1, signed in in a private window.

#	Action	Expected result
1	Private window: Test Suites	The Schedules card and the Asset row, with its next and last run
2	Look along the row and the card header	No New schedule, Run now, Pause, Edit or Delete
3	Click the schedule’s last run	The run opens: everyone in the project can read results

What happened: managing schedules needs the permission Create, change, pause and delete scheduled runs, which Admins and Test Managers have.

Check yourself: a Test Developer wants their suite to run every night. Who sets it up?

Exercise 25: Custom timings on Business

Before you start: platform administrator: change the plan to Business (P1).

#	Action	Expected result
1	New schedule → Custom (cron)	It can be chosen now: no badge
2	Suite Asset, expression `0 6 * * 1-5`	Preview Cron 0 6 * 1-5 (your zone), with runs on weekdays* at 06:00
3	Change it to `nonsense`	A red message that the expression is not valid, asking for five fields (minute hour day month weekday). Create schedule is greyed out
4	Change it to `/5 * * ` → Create schedule*	Refused: That runs every 5 minutes. Schedules can run at most every 15 minutes.
5	Change it to `/15 * * ` → Create schedule*	Created. Asset now has two active schedules

What happened: Business allows cron timings and any number of schedules per suite, but none more often than every 15 minutes, so one schedule cannot keep the runner busy for everyone.

Check yourself: write a cron expression for 8 PM on the first day of every month.

Exercise 26: A cron schedule runs

#	Action	Expected result	✓
1	Read the next run of the `*/15` schedule	Within the next 15 minutes
2	Wait until then, plus up to 30 seconds. Reload	Its last run is PASS (2 passed, 1 skipped), and the next run is 15 minutes later

What happened: a cron schedule runs every time its expression matches, in the schedule’s time zone.

Check yourself: one run of the */15 schedule takes 20 minutes. What happens at the slot in between?

Exercise 27: Changing to a smaller plan

Goal: see that a plan change never chooses which schedules stop.

#	Action	Expected result
1	Platform administrator: Change plan → Team → Save	Refused: Before changing the plan, pause 1 cron schedule or change it to a daily time. Paused schedules are kept and do not count. The plan is still Business
2	Test Suites → Pause the `*/15` schedule	Paused
3	Change plan → Team → Save	Plan updated
4	Resume the `*/15` schedule	Refused: This plan schedules a suite once a day at a fixed time. Custom timings (cron) come with the Business plan.
5	Delete the `*/15` schedule → confirm	One Asset schedule is left

What happened: a downgrade waits until you have paused, changed or deleted what the smaller plan does not allow. Paused schedules never block it, and they are kept for an upgrade.

Check yourself: on Business, Asset has three daily schedules. What must happen before the organization can move to Team?

Exercise 28: Deleting a suite that has a schedule

#	Action	Expected result
1	Add a suite `Spare`, a test `Spare check` in it (any steps), and a daily schedule for `Spare`	A Spare row in Schedules
2	Delete the test `Spare check`, so `Spare` has no tests	—
3	Test Suites → Delete `Spare`	Refused: “Spare” has 1 scheduled run. Delete it first.
4	Delete the Spare schedule, then delete the suite `Spare`	The suite is deleted

What happened: a suite can be deleted only when nothing uses it: no tests and no schedules.

Check yourself: why did step 1 need a test in Spare before you could schedule it?

Exercise 29: Back after a short outage

Only if you run the server yourself and can stop it.

#	Action	Expected result
1	Edit the Asset schedule: At = now + 3 minutes → Save	First run in 3 min
2	Stop the server before that time	—
3	Start it again 1-2 minutes after that time. Sign in, open Test Suites	Within seconds, the last run is PASS (just now), and the next run is tomorrow

What happened: on start, the server runs a schedule that fell due while it was down, once, if it is less than two hours late. A deploy or a restart does not lose the nightly run.

Check yourself: the server was down for 10 minutes at the scheduled time. How many runs start when it is back?

Exercise 30: Too late to catch up

Optional: it needs the server stopped for more than two hours.

#	Action	Expected result	✓
1	Edit the Asset schedule: At = now + 3 minutes → Save. Stop the server before that time	—
2	Start it again more than two hours later. Open Test Suites	The last run is MISSED: The server was not running at the scheduled time. Nothing ran, and the next run is tomorrow

What happened: a run more than two hours late is recorded, not started: a nightly suite that starts at lunchtime only gets in people’s way.

Check yourself: the server was down from Friday evening until Monday at 9 AM, and the schedule runs daily at 02:00. How many runs start when it is back?

Good to know

Who may do what. Run and Stop need the Run tests permission (Tester and above). Creating, changing, pausing and deleting schedules needs Admin or Test Manager. Everyone in the project can see schedules and suite runs.
Only shared tests run in a suite. A private test is never part of a suite run, because everyone in the project can open a suite run.
Narrow screens. On a phone, each schedule becomes a small labelled block, and some columns of the tables are hidden.

Answers to “Check yourself”

Runs from the Run button include Drafts. Only scheduled runs are limited to Ready tests.
Under Test Runs, and in the test’s own run history.
No. The suite runs on the server. Open Test Suites again to see it.
Tests already running finish before the suite ends, so their reports are complete.
2 / 3 passed · 1 skipped, and the suite passes.
The time is set to PM. It should be 12:44 AM.
Custom timings (cron), and as many schedules per suite as you need.
Otherwise pausing, creating a second schedule and resuming the first would get round the plan’s limit.
A small scheduled marker next to the suite name.
An error: No Ready tests in this suite. It counts as failing and sends an email.
When it starts, the server catches up with one run, because it is less than two hours late.
Change the schedule to another environment, or delete the schedule.
No. People who left the project are skipped, and the form shows them as no longer in the project.
The step and its message usually say enough to judge the failure before you open the report.
The schedule’s Last run column, and Recent suite runs.
Two: one failing, one passing again.
Cancelled, skipped (previous run still going) and missed runs.
When it passes again. You will then get the passing again email.
PowerShell drops a bare --, and npm then keeps --suite and the other options for itself.
2.
Command-line options show up in process lists and CI logs. An environment variable comes from the pipeline’s secret store.
It finishes normally, so its report is complete. Only tests that had not started are cancelled.
No. Only active schedules count, and Asset already has its one.
An Admin or Test Manager of the project.
0 20 1 * *
It is skipped, because the previous run is still going.
Two of them must be paused or deleted, so that only one stays active.
The form offers only suites that have tests: an empty suite has nothing to run.
One, when the server starts: it is less than two hours late.
None. The run is more than two hours late, so it shows one MISSED entry (not one per day), and the next run is Tuesday at 02:00.