Course: CI and releases — 2. API tokens for CI

Seventeen hands-on exercises on API tokens: the credentials a CI pipeline uses to start runs and read results, without a person’s password.

You will see:

• how tokens are created, shown once, and revoked;
• how many tokens each plan allows;
• what a token can and cannot reach;
• why a plan cannot be downgraded while the organization holds more tokens than the new plan allows.

The full rules are in the REST API reference.

Each exercise has:

• Goal: what you will see;
• Before you start: the preparation;
• Steps: Where (AutoTestX or PowerShell), Action, Expected result, and a box to tick;
• What happened: the explanation;
• Check yourself: a question to answer before moving on.

Do the exercises in order: several of them use the tokens an earlier one created.

---

Preparation

P1: Accounts, organization and plan

You need:

Who	Why
You, Admin or Test Manager of a project	Only these roles can see and manage API tokens
A platform administrator	Only a platform administrator can change an organization’s plan (exercises 2, 15, 16 and 17). On your own server this is the account claimed at First sign-in. If you are both, you need only one account.
A second person, Test Developer in the same project	Exercise 6 only

Your organization should start on the Proof of value plan, which every new organization starts on. The exercises call your project Default Project; use your own project’s name.

Plan	API tokens per organization
Proof of value	none
Team	2
Business	unlimited
Enterprise	unlimited

P2: A PowerShell window

Module 3 onwards acts like a CI pipeline from Windows PowerShell (built into Windows) or PowerShell 7. Open PowerShell and paste this. On your own server, replace the address with yours, for example http://127.0.0.1:4100/api. The helper sends one request and always prints the HTTP status and the answer. Windows PowerShell would otherwise stop with an error on every refusal, and many exercises expect a refusal.

$BASE = 'https://app.autotestx.com/api'

function Invoke-Atx {
  param(
    [Parameter(Mandatory)][string]$Method,
    [Parameter(Mandatory)][string]$Path,
    [string]$Token,
    [hashtable]$Body,
    [string]$Project
  )
  $headers = @{}
  if ($Token) { $headers['Authorization'] = "Bearer $Token" }
  if ($Project) { $headers['x-project-id'] = $Project }
  $request = @{ Uri = "$BASE$Path"; Method = $Method; Headers = $headers; UseBasicParsing = $true }
  if ($Body) { $request.ContentType = 'application/json'; $request.Body = ($Body | ConvertTo-Json -Compress) }
  try {
    $r = Invoke-WebRequest @request
    $status = [int]$r.StatusCode; $text = $r.Content
  } catch {
    if (-not $_.Exception.Response) { throw }
    $status = [int]$_.Exception.Response.StatusCode; $text = $_.ErrorDetails.Message
  }
  Write-Host "HTTP $status" -ForegroundColor $(if ($status -lt 400) { 'Green' } else { 'Yellow' })
  if ($text) { $text | ConvertFrom-Json | ConvertTo-Json -Depth 6 }
}

The helper exists only in the window you pasted it into. If you close the window, paste it again, together with the token values you saved.

P3: A test to run

Exercise 8 starts a run. In Default Project, create a small test, for example one Navigate step to https://example.com, and save it. Its id is in the address bar after #/tests/; it starts with tc_.

P4: Sign in from PowerShell

Exercises 11 and 13 also need your own signed-in session in PowerShell, next to the tokens. In the P2 window, run this. A Windows dialog asks for your AutoTestX password, and PowerShell then keeps the session cookie, like a browser does:

$cred = Get-Credential -UserName 'your sign-in email' -Message 'Your AutoTestX password'
$login = @{ email = $cred.UserName; password = $cred.GetNetworkCredential().Password } | ConvertTo-Json
Invoke-WebRequest -UseBasicParsing -Uri "$BASE/auth/login" -Method Post -ContentType 'application/json' -Body $login -SessionVariable browser | Out-Null

No output means you are signed in. A red error with 401 or 403 means the email or the password was wrong; run it again.

---

Module 1: Plans and limits

Exercise 1: The Proof of value plan includes no tokens

Goal: see where tokens live, and that the starting plan does not include them.

Before you start: P1. The organization is on Proof of value.

#	Where	Action	Expected result	✓
1	AutoTestX	Project Members (left menu, under the project)	Page Members & roles
2	AutoTestX	Scroll to API tokens for CI	A pipeline sends a token as Authorization: Bearer … to start runs and read results in Default Project. Tokens belong to the project, not to a person, so they keep working when their creator leaves.
3	AutoTestX	Look below that text	API tokens for CI are not included in this plan. They come with the Team plan. No New token button and no counter

What happened: tokens are part of the plan. On Proof of value the section explains this instead of offering a button that would only be refused.

Check yourself: where do you go to manage the tokens of a project?

Exercise 2: The Team plan allows two tokens

Goal: see the limit appear when the plan changes.

Before you start: a platform administrator does steps 1-2.

#	Where	Action	Expected result	✓
1	AutoTestX	Organization → Settings	A card Proof of value plan · AI generations this month, with Change plan (platform administrators only)
2	AutoTestX	Change plan → Plan Team → Save	Toast Plan updated; the card says Team plan
3	AutoTestX	Project Members → API tokens for CI	Counter 0 of 2 used in this organization; a New token button; No tokens in this project yet.

What happened: the limit belongs to the organization’s plan. Team allows two tokens. The counter shows how many are in use.

Check yourself: who can change an organization’s plan, and why not the organization itself?

---

Module 2: Creating and managing tokens

Exercise 3: A token is shown once

Goal: create a token, and see that its value is never shown again.

Before you start: Exercise 2. Open a text editor to keep the token in.

#	Where	Action	Expected result	✓
1	AutoTestX	New token	Dialog New API token: Name, It can = Run tests, Expires after = 90 days
2	AutoTestX	Leave Name empty	Create token is disabled
3	AutoTestX	Name GitHub Actions - main; leave Run tests and 90 days → Create token	Dialog Copy your token now: This is the only time GitHub Actions - main is shown… The value starts with atx_pat_
4	AutoTestX	Copy, then paste the value into your text editor and label it TOKEN_RUN	Toast Token copied
5	AutoTestX	I have stored it	One row: GitHub Actions - main · atx_pat_xxxx… · Run tests · Last used Never · Expires in about 90 days. Counter 1 of 2 used
6	AutoTestX	Reload the page	The row still shows only the first characters, atx_pat_xxxx…. There is no way to see the full value again

What happened: AutoTestX keeps only a fingerprint (a SHA-256 hash) of the token. It can recognise the token when a pipeline sends it, but it cannot show it again. If a token is lost, revoke it and create a new one.

Check yourself: a colleague asks you to look up a token’s value in AutoTestX. What do you tell them?

Exercise 4: Two tokens per organization, not per project

Goal: reach the Team limit and see that it counts the whole organization.

Before you start: Exercise 3.

#	Where	Action	Expected result	✓
1	AutoTestX	New token: Name Nightly reader, It can Read results only, Expires after Never → Create token	The value is shown once. Save it as TOKEN_READ → I have stored it
2	AutoTestX	Look at the section	Counter 2 of 2 used; second row Read results only, expires Never
3	AutoTestX	Hover over New token	Disabled; tooltip Your plan’s tokens are all in use
4	AutoTestX	Organization → Projects: create a project Mobile, open it, Project Members	Mobile lists no tokens, but the counter still says 2 of 2 used in this organization and New token is disabled
5	AutoTestX	Switch back to Default Project

What happened: each token belongs to one project, but the plan’s limit counts every project of the organization together.

Check yourself: on Team, can each of five projects have two tokens?

Exercise 5: Revoking a token

Goal: remove a token and free its place.

Before you start: Exercise 4.

#	Where	Action	Expected result	✓
1	AutoTestX	On Nightly reader: Revoke → Cancel in the browser dialog	Nothing changes
2	AutoTestX	Revoke → OK	The browser asked Revoke “Nightly reader”? Any pipeline using it stops working at once. Toast Revoked Nightly reader; the row is gone
3	AutoTestX	Look at the counter	1 of 2 used; New token is enabled again
4	Text editor	Rename the saved TOKEN_READ value to TOKEN_OLD	Exercise 12 uses this revoked value
5	AutoTestX	New token again: Nightly reader, Read results only, Never → Create token	A new value. Save it as TOKEN_READ → I have stored it. Counter 2 of 2 used

What happened: revoking takes effect immediately. The next request with that token is refused, and its place in the plan’s limit is free again. A new token with the same name is a different token, with a different value.

Check yourself: a laptop with a pipeline’s token on it is stolen. What do you do first?

Exercise 6: Who can manage tokens

Goal: see that only Admins and Test Managers manage tokens.

Before you start: a second person who is Test Developer in Default Project (P1).

#	Where	Action	Expected result	✓
1	AutoTestX	Sign in as the Test Developer, in another browser or a private window
2	AutoTestX	Project Members	The members and the role table are shown, but there is no API tokens for CI section

What happened: a token can start runs in the project’s name, so issuing one is a decision for those who manage the project’s members.

Check yourself: which two project roles can create tokens?

---

Module 3: Acting as a pipeline

From here on, PowerShell plays the part of a CI pipeline. It sends no browser cookie, only the token.

Exercise 7: A token identifies itself

Goal: check a token before doing anything with it, as a pipeline’s first step should.

Before you start: P2. Paste your token:

$TOKEN_RUN = 'paste the GitHub Actions - main value here'

#	Where	Action	Expected result	✓
1	PowerShell	Invoke-Atx GET /tokens/self -Token $TOKEN_RUN	HTTP 200; "name": "GitHub Actions - main", "role": "tester", and "project" = Default Project. No hash in the answer
2	PowerShell	Invoke-Atx GET /tests -Token $TOKEN_RUN	HTTP 200 and the project’s tests
3	PowerShell	Invoke-Atx GET /runs -Token $TOKEN_RUN	HTTP 200

What happened: the token knows its own project, so a pipeline does not have to name one. /tokens/self is a cheap way to fail early with a clear message if a secret was pasted wrong.

Check yourself: why does the pipeline not need to send a project id?

Exercise 8: Start a run and read its result

Goal: run a test the way a pipeline does.

Before you start: Exercise 7, and P3 for the test id:

$TEST_ID = 'paste the tc_ id here'

#	Where	Action	Expected result	✓
1	PowerShell	Invoke-Atx POST /runs -Token $TOKEN_RUN -Body @{ testId = $TEST_ID; screenshots = $false }	HTTP 202 and "runId": "run_…"
2	PowerShell	$RUN_ID = 'paste the run_ id here'
3	PowerShell	Invoke-Atx GET "/runs/$RUN_ID" -Token $TOKEN_RUN	HTTP 202, Run not finished, while it runs. Repeat after a few seconds: HTTP 200 and "status": "passed" (or failed, with the details)
4	AutoTestX	Runs	The run is listed
5	AutoTestX	Project Members, reload	Last used of GitHub Actions - main shows today

What happened: 202 means accepted, not finished. A pipeline repeats the request until it gets 200, then fails the build if the run failed. Last used tells you which tokens are still in use before you revoke one.

Check yourself: what should a pipeline do when it gets 202?

Exercise 9: A read-only token cannot start a run

Goal: see the token’s role at work.

Before you start: Exercise 8. Paste the new Nightly reader value from exercise 5:

$TOKEN_READ = 'paste the TOKEN_READ value here'

#	Where	Action	Expected result	✓
1	PowerShell	Invoke-Atx GET "/runs/$RUN_ID" -Token $TOKEN_READ	HTTP 200: it can read results
2	PowerShell	Invoke-Atx POST /runs -Token $TOKEN_READ -Body @{ testId = $TEST_ID }	HTTP 403, Your role in this project does not allow running tests.

What happened: Read results only is enough for a reporting script, and harmless if it leaks. Give a pipeline Run tests only when it starts runs.

Check yourself: a dashboard script only shows last night’s results. Which role should its token have?

---

Module 4: Security boundaries

Exercise 10: What a token can never do

Goal: see the endpoints closed to tokens.

Before you start: Exercise 7.

#	Where	Action	Expected result	✓
1	PowerShell	Invoke-Atx GET /members -Token $TOKEN_RUN	HTTP 403, "code": "token_not_allowed", API tokens cannot use this endpoint. Sign in as a member instead.
2	PowerShell	Invoke-Atx POST /orgs -Token $TOKEN_RUN -Body @{ name = 'Token Org' }	The same 403
3	PowerShell	Invoke-Atx POST /tokens -Token $TOKEN_RUN -Body @{ name = 'child'; role = 'tester' }	The same 403
4	AutoTestX	Project Members and the organization switcher	No organization Token Org and no token child were created

What happened: a token reaches only what a pipeline needs: reading tests, suites, environment names and runs, and starting runs. It can never manage people, secrets or other tokens, so a leaked token cannot be used to create more access.

Check yourself: why must a token not be able to create another token?

Exercise 11: A token stays in its own project

Goal: see that a token cannot reach another project, even in the same organization.

Before you start: Exercise 4 (the Mobile project) and P4 (signed in from PowerShell). Look up Mobile’s id with your session (not a token), and keep it in $MOBILE:

$projects = (Invoke-RestMethod -UseBasicParsing -Uri "$BASE/context" -WebSession $browser).organizations.projects
$projects | Format-Table name, id
$MOBILE = ($projects | Where-Object name -eq 'Mobile').id

It lists your projects with their ids (prj_…), including Mobile.

#	Where	Action	Expected result	✓
1	PowerShell	Invoke-Atx GET /tests -Token $TOKEN_RUN -Project $MOBILE	HTTP 403, You are not a member of this project

What happened: the attempt is refused and recorded in the audit log as a cross-tenant attempt by the token (exercise 14 shows it).

Check yourself: a pipeline needs to run tests in two projects. What do you give it?

Exercise 12: Wrong, revoked and missing tokens

Goal: see how a pipeline is told its token does not work.

Before you start: Exercise 5. Paste the revoked value you saved as TOKEN_OLD:

$TOKEN_OLD = 'paste the TOKEN_OLD value here'

#	Where	Action	Expected result	✓
1	PowerShell	Invoke-Atx GET /tokens/self -Token 'atx_pat_this-is-not-real'	HTTP 401, "code": "invalid_token", Invalid or revoked API token.
2	PowerShell	Invoke-Atx GET /tokens/self -Token $TOKEN_OLD	The same 401: the revoked token stopped working
3	PowerShell	Invoke-Atx GET /tests (no token)	HTTP 401, Sign in to continue

What happened: a revoked token and one that never existed get the same answer, so the answer reveals nothing about which tokens exist. A token past its expiry date answers 401 with "code": "expired_token" and the date it expired.

Check yourself: a nightly pipeline started failing with 401. Name two likely reasons.

Exercise 13: A wrong token never falls back to a signed-in session

Goal: see that a request carrying a token is judged by that token alone.

Before you start: P4 (signed in from PowerShell). Run this block:

function Test-Cookie([string]$Label, [hashtable]$Headers) {
  try { $status = [int](Invoke-WebRequest -UseBasicParsing -Uri "$BASE/tests" -WebSession $browser -Headers $Headers).StatusCode }
  catch { $status = [int]$_.Exception.Response.StatusCode }
  "$Label $status"
}
Test-Cookie '1. Cookie only        - expect 200:' @{}
Test-Cookie '2. Wrong Bearer token - expect 401:' @{ Authorization = 'Bearer atx_pat_wrong' }
Test-Cookie '3. Basic auth         - expect 200:' @{ Authorization = 'Basic dXNlcjpwYXNz' }

Expected:

1. Cookie only        - expect 200: 200
2. Wrong Bearer token - expect 401: 401
3. Basic auth         - expect 200: 200

If line 1 shows 401, the session from P4 is missing or has ended. Run P4 again before reading lines 2 and 3.

What happened:

• Line 2: the request has a valid session and a wrong token, and it is refused. If the server fell back to the session, a pipeline whose token was revoked could quietly keep running as whoever is signed in on the same machine.
• Line 3: Basic authorization (for example from a password-protected proxy in front of a staging server) is not a token, so it is ignored and the browser session keeps working.

Check yourself: why is “refuse” safer than “fall back” here?

Exercise 14: The token is not stored, and the audit log names it

Goal: see what AutoTestX keeps about tokens. This exercise is for the person who runs the server, because it reads the data folder (Data, backup and seeding).

Before you start: Exercises 3-11. In PowerShell, go to the AutoTestX folder. The data folder is .data unless ATX_DATA_DIR says otherwise.

#	Where	Action	Expected result	✓
1	PowerShell	Select-String -Path .data\api-tokens.json -Pattern $TOKEN_RUN -SimpleMatch	No output: the value is not in the file. Each token has a 64-character hash instead
2	PowerShell	The command below	One row per action (see below)

Select-String -Path .data\audit.log -Pattern 'api-token|plan-changed|cross-tenant' | ForEach-Object { $_.Line | ConvertFrom-Json } | Format-Table at, event, userId, name, plan -AutoSize

Expected rows:

• api-token-created for each token, with its name; userId is the person who created it (usr_…);
• api-token-revoked for the first Nightly reader;
• cross-tenant-denied from exercise 11, with userId = the token (tok_…);
• plan-changed for the change to Team.

What happened: a copy of the data folder is not a set of working tokens. And whatever a token does is recorded under the token’s own id, not a person’s, so you can tell “the pipeline did it” from “a person did it”.

Check yourself: a backup of the data folder leaked. Do the pipelines’ tokens need to be replaced?

---

Module 5: Changing plans

Exercise 15: Upgrading removes the limit

Goal: see the counter change on an unlimited plan.

Before you start: a platform administrator does step 1.

#	Where	Action	Expected result	✓
1	AutoTestX	Organization → Settings → Change plan → Business → Save	Plan updated
2	AutoTestX	Project Members → API tokens for CI	The counter says 2 in this organization, with no “of”
3	AutoTestX	Create Deploy hook (any role, any expiry)	Created; New token stays enabled. There are now 3 tokens

What happened: Business and Enterprise have no token limit.

Check yourself: what does the counter look like on a plan without a limit?

Exercise 16: A downgrade waits until the tokens fit

Goal: see that a downgrade never chooses which pipelines stop working.

Before you start: Exercise 15 (Business, 3 tokens). A platform administrator does the plan changes.

#	Where	Action	Expected result	✓
1	AutoTestX	Change plan → Team → Save	Refused: This organization holds 3 API tokens and the new plan allows 2. Revoke 1 of them (Project Members → API tokens for CI), then change the plan.
2	AutoTestX	Reload Organization → Settings	Still Business
3	PowerShell	Invoke-Atx GET /tokens/self -Token $TOKEN_RUN	HTTP 200: every token keeps working while the downgrade waits
4	AutoTestX	Revoke Nightly reader (keep GitHub Actions - main and Deploy hook)	2 in this organization
5	AutoTestX	Change plan → Team → Save	Plan updated; the token counter shows 2 of 2 used

What happened: the server cannot know which pipeline matters most to you, so a downgrade waits for you to decide which tokens to revoke. Nothing stops working by surprise.

Check yourself: an organization on Business holds 7 tokens and wants Team. What must happen first?

Exercise 17: Back to Proof of value only with no tokens

Goal: finish the cycle: a plan without tokens needs every token revoked.

Before you start: Exercise 16 (Team, 2 tokens). A platform administrator does the plan changes.

#	Where	Action	Expected result	✓
1	AutoTestX	Change plan → Proof of value → Save	Refused: …the new plan allows none. Revoke 2 of them… The plan stays Team
2	AutoTestX	Revoke Deploy hook, then try again	Still refused, now Revoke 1 of them
3	AutoTestX	Revoke GitHub Actions - main, then try again	Plan updated. Project Members shows API tokens for CI are not included in this plan.
4	PowerShell	Invoke-Atx GET /tokens/self -Token $TOKEN_RUN	HTTP 401, "code": "invalid_token"

What happened: the same rule as exercise 16, down to zero. Every token that exists is always one the plan allows.

Check yourself: after this exercise, what would a pipeline still using TOKEN_RUN see?

---

Answers to “Check yourself”

1. Project Members, in the section API tokens for CI.
2. A platform administrator (the AutoTestX operator), because the plan is what the organization pays for.
3. It cannot be looked up: only a fingerprint is stored. Revoke the lost token and create a new one.
4. No. Team allows two tokens for the whole organization.
5. Revoke its token in Project Members at once. It stops working on the next request.
6. Admin and Test Manager.
7. The token belongs to one project, and the server uses that project.
8. Wait a few seconds and ask again, until the answer is 200.
9. Read results only.
10. Otherwise one leaked token could be turned into many, including tokens nobody knows about.
11. One token per project.
12. The token was revoked, or it expired (expired_token). A secret pasted wrong in the CI system gives the same invalid_token.
13. Falling back would let a pipeline with a revoked token keep working as a signed-in person, unnoticed and in that person’s name.
14. No. The data folder holds only hashes, which cannot be turned back into tokens. (Rotating them anyway is a reasonable precaution.)
15. N in this organization, with no “of”.
16. Revoke 5 tokens, so 2 are left, then change the plan.
17. HTTP 401 with invalid_token.