Course: CI and releases — 4. Tests running at the same time

Fourteen hands-on exercises on the plan limit for tests running at the same time: how many tests your organization can run at once, and what happens to the ones above the limit.

You will see:

• that runs above the limit wait in a queue and start by themselves, and are never refused;
• how the editor shows a waiting run: Queued… on the Run button and a notice under Run result;
• that the limit is shared by everything your organization runs: editor runs, suite runs, schedules and CI;
• that each row of a data-driven test counts as one test;
• what the Rows at once field tells you before you run.

The rules are in Data-driven testing → Rows and your plan.

Plan	Tests running at the same time (whole organization)
Proof of value	1
Team	2
Business	6
Enterprise	Custom

Each exercise has:

• Goal: what you will see;
• Before you start: the preparation;
• Steps: Where, Action, Expected result, and a box to tick;
• What happened: the explanation;
• Check yourself: a question to answer before moving on.

Do the exercises in order: the plan changes between modules.

---

Preparation

P1: Accounts, organization and plan

Who	Why
You, Test Developer or above in a project	To create and run tests
A platform administrator	Only a platform administrator can change an organization’s plan (modules 2 and 3). On your own server this is the account claimed at First sign-in. If you are both, you need only one account.

Your organization should start on the Proof of value plan, which every new organization starts on. Make sure nobody else in your organization is running tests during the exercises: their runs share the same limit and would change the timings.

To change the plan (platform administrator): Organization → Settings → the plan card → Change plan → choose the plan → Save. A plan applies to runs started after the change, so change it only when nothing is running.

P2: Practice tests

Every test below is Shared: only shared tests run in a suite. Each one only waits, so the timings are easy to follow. To add a wait: Add step → Wait, and set ms.

Name	Suite	Step	Data source
Slow A	Concurrency	Wait, ms 20000 (20 seconds)	None
Slow B	Concurrency	Wait, ms 20000	None
Slow C	Concurrency	Wait, ms 20000	None
Solo	(none)	Wait, ms 20000	None
Rows	(none)	Wait, ms 10000 (10 seconds)	Inline table, below

1. Test Suites → type Concurrency → Add suite.

2. Test Cases → New test for each row of the table. Save each one.

3. For Rows, open the right-hand Test panel → Data source → Inline table, and paste:

   n
   1
   2
   3
   4
   5
   6

   Click outside the box: the hint reads 6 rows. Tick Rows are independent — run them in parallel, set Rows at once to 4, and save.

P3: Two browser tabs and a stopwatch

Several exercises start two runs at once: open AutoTestX in two tabs. Several expected results are about time: keep a stopwatch at hand (your phone is fine).

---

Module 1: Proof of value (one test at a time)

Exercise 1: A run with a free slot starts at once

Goal: see a normal run, to compare with the queued ones that follow.

Before you start: P1-P3. Nothing is running.

#	Where	Action	Expected result	✓
1	AutoTestX	Open Solo → Run test	The button changes straight to Running… with a play icon. It never shows Queued…
2	AutoTestX	Look under the steps	Run result reads Executing…. There is no yellow notice
3	AutoTestX	Wait	The run passes after about 20 seconds

What happened: the organization’s one slot was free, so the run took it immediately.

Check yourself: on Proof of value, how many tests can your whole organization run at once?

Exercise 2: A second run waits, then starts by itself

Goal: see a run queue, and see the editor explain it.

Before you start: Exercise 1. Solo in tab 1, Slow A in tab 2.

#	Where	Action	Expected result	✓
1	Tab 1	Run test on Solo	Running…
2	Tab 2	Within 5 seconds: Run test on Slow A. Start the stopwatch	The button reads Queued… with a clock icon
3	Tab 2	Hover over the button	Queued: your plan runs 1 at a time. This run starts when a slot frees up.
4	Tab 2	Look under the steps	Run result reads Queued, and a yellow notice: Queued — waiting for a free slot. Your plan runs one test at a time across the organization, and that slot is in use by other runs. This run starts automatically as soon as one finishes.
5	Tab 2	Click Compare plans that run more at once at the end of the notice	The pricing page opens in a new tab, at the plan comparison. Tab 2 still shows the queued run. Close the pricing tab
6	Tab 2	Wait, without clicking anything	When Solo finishes (about 20 s), the button turns Running… by itself and the notice disappears
7	Tab 2	Wait for the end	Slow A passes about 40 s after you clicked. Its reported duration is about 20 s

What happened: the run was accepted and queued, not refused. It started as soon as the slot was free. The time spent waiting is not counted as run time, so the duration shows only how long the test itself took.

Check yourself: a colleague sees Queued… and clicks Run test again in another tab. What happens?

Exercise 3: A suite runs one test at a time

Goal: see the same limit in a suite run.

Before you start: nothing is running.

#	Where	Action	Expected result	✓
1	AutoTestX	Test Suites → Concurrency → Run	Toast Running 3 tests in Concurrency
2	AutoTestX	Look at the suite panel	0 of 3 finished · up to 1 at a time. One test is RUNNING, two are QUEUED
3	AutoTestX	Watch	They run one after another in name order: Slow A, Slow B, Slow C
4	AutoTestX	Wait for the end	About 60 s in total; 3 passed

What happened: a suite run uses the same organization limit as the editor. Schedules and CI pipelines start suite runs too, so they wait their turn the same way.

Check yourself: on Proof of value, about how long does a suite of ten 1-minute tests take?

Exercise 4: An editor run goes ahead of waiting suite tests

Goal: see the order in which waiting runs start.

Before you start: nothing is running. Tab 2 has Solo open.

#	Where	Action	Expected result	✓
1	Tab 1	Test Suites → Concurrency → Run. Wait until Slow A is RUNNING	Slow B and Slow C are QUEUED
2	Tab 2	Run test on Solo	Queued… and the queued notice
3	Both	Wait until Slow A finishes	Solo starts next (Running…), while Slow B and Slow C stay QUEUED
4	Both	Wait for the end	Solo finishes, then Slow B, then Slow C. About 80 s in total; nothing fails or is cancelled

What happened: a run started from the editor goes to the front of the queue, because a person is watching it. Suite tests are batch work and wait in suite order.

Check yourself: why does an editor run go ahead of a suite test that was queued before it?

Exercise 5: A data-driven test in a suite

Goal: see a data-driven test run inside a suite on a plan with one slot.

Before you start: nothing is running.

#	Where	Action	Expected result	✓
1	AutoTestX	Open Rows → Test panel → Suite Concurrency → save
2	AutoTestX	Test Suites → Concurrency → Run	up to 1 at a time; 4 tests listed
3	AutoTestX	When Rows is RUNNING, open its run from the suite panel	Its rows run one after another, even though Rows at once is 4
4	AutoTestX	Wait for the end	Rows takes about 60 s (6 rows × 10 s). The suite ends with 4 passed
5	AutoTestX	Open Rows → Test panel → Suite back to none → save

What happened: each row counts as one test. The test’s own slot goes to its first row, and the other rows wait for slots like any other test. So the test always finishes, even on a plan with a single slot.

Check yourself: a data-driven test with 6 rows runs on Proof of value. How many rows run at the same time?

Exercise 6: The Rows at once field explains the limit

Goal: see the editor warn you before you run.

Before you start: open Rows → Test panel. Rows at once is 4.

#	Where	Action	Expected result	✓
1	AutoTestX	Look under Rows at once	A grey hint: Your plan runs one test at a time across the organization, so at most 1 of these rows run together. The others wait for a free slot.
2	AutoTestX	Change it to 1	The hint disappears
3	AutoTestX	Clear the field	The placeholder reads platform default (4), and the grey hint is back
4	AutoTestX	Type 6	Only the grey plan hint, no orange hint
5	AutoTestX	Reload the page without saving	Rows at once is 4 again

What happened: two limits can keep Rows at once lower than you set. The editor shows only the one that applies: here the plan’s single slot. The orange hint (exercise 11) is about the server’s own ceiling.

Check yourself: why does an empty Rows at once still show the plan hint?

---

Module 2: Team (two tests at a time)

Before you start: a platform administrator changes the plan to Team (P1).

Exercise 7: Two runs at once, the third waits

Goal: see the limit move with the plan.

Before you start: open Solo, Slow A and Slow B in three tabs.

#	Where	Action	Expected result	✓
1	Tabs 1-3	Run test in each tab, one after another, within 5 seconds	Tabs 1 and 2: Running… at once. Tab 3: Queued…
2	Tab 3	Read the notice	Your plan runs 2 tests at a time across the organization, and all are in use by other runs…
3	Tab 3	Wait	It starts by itself after about 20 s, and passes about 40 s after the click

What happened: the same rules with a bigger limit. The notice always names your current plan’s number.

Check yourself: on Team, a suite of 2 tests is running. You click Run test in the editor. What do you see?

Exercise 8: A suite and an editor run share the slots

Goal: see that the limit counts the whole organization, whatever started the run.

Before you start: nothing is running. Tab 2 has Solo open.

#	Where	Action	Expected result	✓
1	Tab 1	Test Suites → Concurrency → Run	up to 2 at a time; 2 tests RUNNING, 1 QUEUED
2	Tab 2	Run test on Solo	Queued…; the notice says 2 tests… all are in use
3	Both	Wait for the first suite test to finish	Solo takes the free slot before Slow C
4	Both	Wait	Slow C starts when the next slot frees up. Everything passes

What happened: suite runs, editor runs, schedules and pipelines all draw from the same slots.

Check yourself: a nightly schedule and a CI pipeline both start at 2:00 on Team. What happens?

Exercise 9: Each row takes a slot

Goal: see a data-driven run limited by the plan, not by its own setting.

Before you start: nothing is running. Open Rows → Test panel.

#	Where	Action	Expected result	✓
1	AutoTestX	Look under Rows at once (4)	Grey hint: Your plan runs 2 tests at a time across the organization, so at most 2 of these rows run together…
2	AutoTestX	Run test. Start the stopwatch	At most 2 rows run at a time
3	AutoTestX	Look at the notice	4 rows waiting for a free slot, then 2. It disappears when rows 5 and 6 start
4	AutoTestX	Look at the Run button	Running…, not Queued…: some rows are already running
5	AutoTestX	Wait for the end	Passes after about 30 s (three rounds of two rows), 6 of 6 rows passed

What happened: Rows at once asked for 4, the plan allowed 2, and the lower number decided. The remaining rows waited and started by themselves.

Check yourself: on Team, a test has 6 rows and Rows at once 4. How many rounds does it take?

Exercise 10: A data-driven run with no free slot

Goal: see a data-driven run that cannot start any row yet.

Before you start: nothing is running. Tab 2 has Rows open.

#	Where	Action	Expected result	✓
1	Tab 1	Test Suites → Concurrency → Run	2 tests RUNNING
2	Tab 2	Immediately: Run test on Rows	The button reads Queued… (clock icon): no row has started
3	Tab 2	Look under the steps	Executing rows… 0 of 6 done, and the notice 6 rows waiting for a free slot
4	Both	Wait	When a suite test finishes, Rows gets the slot before Slow C, and the button turns Running…. Everything passes

What happened: the button says Queued… only while nothing at all has started. Once one row runs, the run is under way, and the notice explains the rows still waiting.

Check yourself: what is the difference between Queued… on the button and N rows waiting in the notice?

---

Module 3: Business (six tests at a time)

Before you start: a platform administrator changes the plan to Business (P1).

Exercise 11: Other limits can keep a run below the plan

Goal: see that the plan limit is a ceiling, not a target.

Before you start: nothing is running. Open Rows → Test panel.

#	Where	Action	Expected result	✓
1	AutoTestX	Look under Rows at once (4)	No hint: 4 is within every limit
2	AutoTestX	Type 6 (don’t save)	An orange hint: This server runs up to 4 rows at once per run, so this test runs 4 at a time. No grey plan hint
3	AutoTestX	Set it back to 4 and save. Run test	4 rows run at once, not 6. There is no queued notice
4	AutoTestX	Wait for the end	Passes after about 20 s (4 rows, then 2)

What happened: how many rows run at once is the lowest of three numbers:

1. Rows at once, the test’s own setting (4);
2. the server’s ceiling per run, set by whoever runs the server (default 4);
3. the plan’s free slots (6 on Business), shared with the rest of the organization.

The server’s ceiling keeps a single run from opening too many browsers at once, whatever the plan. The orange hint warns you because it lowers your setting without queuing. Exercise 14 shows how to raise it.

Check yourself: on Business, why does setting Rows at once to 6 still run 4 rows?

Exercise 12: Stopping a suite

Goal: see what stopping does to running and waiting tests.

Before you start: nothing is running.

#	Where	Action	Expected result	✓
1	AutoTestX	Test Suites → Concurrency → Run	up to 6 at a time: all 3 tests RUNNING together. The suite finishes in about 20 s
2	AutoTestX	A platform administrator changes the plan back to Proof of value
3	AutoTestX	Run the suite again. While Slow A is RUNNING, click Stop	Toast Stopping: queued tests are cancelled, running ones will finish.
4	AutoTestX	Watch	Slow B and Slow C turn CANCELLED at once. Slow A finishes normally. The suite ends CANCELLED: 1 passed, 2 cancelled
5	AutoTestX	Right after Slow A finishes: open Solo → Run test	Running… at once: the stopped suite gave its slot back

What happened: a test that is already running is never cut off halfway, so its report stays complete. Waiting tests are simply removed from the queue.

Check yourself: you stop a suite while 2 of its 10 tests are running. What happens to each group?

---

Module 4: Organizations and the server

Exercise 13: Each organization has its own slots

Goal: see that a busy organization never delays another one.

Before you start: the plan is Proof of value. Open the organization switcher at the top of the left menu → New organization, and create Other Org with a project Web. In it, create a shared test Other wait (Wait, ms 20000).

#	Where	Action	Expected result	✓
1	Tab 1	Your first organization: Run test on Solo	Running…
2	Tab 2	Switch to Other Org / Web: Run test on Other wait	Running… at once. Neither tab shows Queued…

What happened: both organizations are on Proof of value, but each has its own slot. The limit is per organization, never shared between customers.

Check yourself: another company on the same server runs a large suite. Does it slow your runs down?

Exercise 14: Raising the server’s row ceiling

Goal: see the plan limit decide once the server allows more rows per run.

Before you start: this exercise is for the person who runs the server. The plan is Business, and nothing is running.

#	Where	Action	Expected result	✓
1	Server	Set ATX_MAX_ROW_CONCURRENCY=8 (Configuration) and restart AutoTestX
2	AutoTestX	Open Rows → Test panel. Clear Rows at once	Placeholder platform default (8)
3	AutoTestX	Rows at once 6, save, Run test	No hint. All 6 rows run at once, with no queued notice. Passes after about 10 s
4	AutoTestX	Add rows 7 and 8 to the table, Rows at once 8, save	The grey plan hint: Your plan runs 6 tests at a time… at most 6 of these rows run together…
5	AutoTestX	Run test	6 rows run, and the notice reads 2 rows waiting for a free slot. Passes after about 20 s
6	Server	Put ATX_MAX_ROW_CONCURRENCY back to its previous value and restart

What happened: with the ceiling at 8, the plan’s 6 slots became the lowest limit. Raise the ceiling only if the server has the memory for it: each row is a browser. The Configuration reference gives the safe values per server size.

Check yourself: a Business customer needs 6 rows at once. What must be true on the server?

---

Good to know

• Reloading the page while a run is queued. The editor shows Run test again and no longer follows that run, but the run is not lost: it starts when a slot frees up, and you find it under Test Runs.
• A plan change applies to new runs. Runs already waiting or running keep the limit they started with.

---

Answers to “Check yourself”

1. One. The limit is for the whole organization, not per person.
2. A second run is queued behind the first. Nothing is refused, but each click adds another run to the queue.
3. About 10 minutes: one test after another.
4. A person is watching it. Suite tests are batch work, and nobody is waiting at the screen for them.
5. One. Each row counts as one test, and the plan has one slot.
6. An empty field means the server’s default, 4, which is still more than the plan’s 1.
7. Queued… and the notice Your plan runs 2 tests at a time… all are in use…
8. They share the 2 slots. Two tests run, and the rest wait in a queue and start as slots free up.
9. Three: 2 rows, 2 rows, 2 rows.
10. Queued… means nothing of the run has started. N rows waiting means the run has started and some of its rows are still waiting.
11. The server’s ceiling per run is 4. The lowest limit decides.
12. The 2 running tests finish normally. The 8 waiting tests are cancelled.
13. No. Each organization has its own slots.
14. The server’s ATX_MAX_ROW_CONCURRENCY must be at least 6, and the server needs the memory for 6 browsers at once.