Run after deployment

Run a suite automatically every time you deploy, and stop the release if it fails.

A nightly schedule finds a break hours after it happened, by which time several deploys may have gone out. A suite that runs straight after each deploy points at one deploy and one commit. Because the pipeline waits for the answer, it can also fail the build and stop that deploy from reaching the next environment.

 your pipeline                         AutoTestX
 ─────────────                         ─────────
 build ─► deploy to staging ─► POST /api/executions  { suite, environment, deployment }
                               ◄─ 202 { executionId }
                               GET /api/executions/:id?wait=60   (repeat until done)
                               ◄─ { status: "passed" | "failed" | … }
          passed? ─► deploy to production
          failed? ─► stop, red build, email to the team

What you need

1. An API token with the Tester role. A project Admin or Test Manager creates it under Project Members → API tokens for CI (API tokens). Store it in your CI system’s secret store, for example as AUTOTESTX_TOKEN. A Viewer token can read results but cannot start runs.
2. A suite with shared tests, for example Smoke.
3. An environment in AutoTestX that points at the deployed system, for example Staging.

The easy way: atx run --suite

The AutoTestX CLI starts the suite, prints each test as it finishes, and exits 0 only when the suite passed. It needs Node and nothing else. No browser runs on the CI machine.

The CLI ships with AutoTestX (packages/cli). The examples below assume the pipeline has a checkout that contains it. If yours does not, skip to Without the CLI: two HTTP calls do the same with nothing to install.

In GitHub Actions, GitLab CI, Azure Pipelines, Bitbucket Pipelines and Jenkins, the CLI also sends the commit, branch, build number and a link to the pipeline run automatically. The suite run then shows which deploy it tested.

When the pipeline builds a git tag (v2.14.0), the tag is also sent as the release. For any other build, name the release yourself with --release or the ATX_RELEASE variable, e.g. ATX_RELEASE=${{ needs.build.outputs.version }}. AutoTestX never guesses a release from a branch name: release/2.14 builds many commits before 2.14 ships.

GitHub Actions

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh staging

  smoke-test:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4      # a checkout that contains the AutoTestX CLI
      - run: npm ci
      - run: npm run atx -- run --suite Smoke --env Staging --ready --json smoke.json
        env:
          ATX_SERVER: https://app.autotestx.com
          ATX_TOKEN: ${{ secrets.AUTOTESTX_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: smoke-result, path: smoke.json }

  deploy-production:
    needs: smoke-test                  # only runs when the smoke suite passed
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh production

GitLab CI

smoke-test:
  stage: verify
  needs: [deploy-staging]
  image: node:22
  script:
    - npm ci
    - npm run atx -- run --suite Smoke --env Staging --ready --json smoke.json
  variables:
    ATX_SERVER: https://app.autotestx.com
    # ATX_TOKEN: set as a masked CI/CD variable
  artifacts:
    when: always
    paths: [smoke.json]

Azure Pipelines

- stage: Verify
  dependsOn: DeployStaging
  jobs:
    - job: Smoke
      pool: { vmImage: ubuntu-latest }
      steps:
        - script: npm ci
        - script: npm run atx -- run --suite Smoke --env Staging --ready --json smoke.json
          env:
            ATX_SERVER: https://app.autotestx.com
            ATX_TOKEN: $(AUTOTESTX_TOKEN)     # a secret pipeline variable

Jenkins

A declarative Jenkinsfile. Store the token as a Secret text credential, here with the ID autotestx-token. The docker agent needs the Docker Pipeline plugin; on an agent that already has Node installed, use agent any instead.

pipeline {
  agent none
  stages {
    stage('Deploy to staging') {
      agent any
      steps { sh './deploy.sh staging' }
    }
    stage('Smoke test') {
      agent { docker { image 'node:22' } }   // a checkout that contains the AutoTestX CLI
      environment {
        ATX_SERVER = 'https://app.autotestx.com'
        ATX_TOKEN  = credentials('autotestx-token')   // masked in the build log
      }
      steps {
        sh 'npm ci'
        sh 'npm run atx -- run --suite Smoke --env Staging --ready --json smoke.json'
      }
      post {
        always { archiveArtifacts artifacts: 'smoke.json', allowEmptyArchive: true }
      }
    }
    stage('Deploy to production') {      // skipped when the smoke suite failed
      agent any
      steps { sh './deploy.sh production' }
    }
  }
}

A failed suite makes the atx step exit with 1, which fails the stage, and Jenkins skips the production deploy. The CLI reads GIT_COMMIT, GIT_BRANCH (or BRANCH_NAME in a multibranch pipeline), BUILD_NUMBER and BUILD_URL, so the suite run links back to the Jenkins build.

A repository without the CLI uses the one-file gate script instead: the complete setup, from the credential to the Test Result page, is in Jenkins pipeline.

Options worth knowing

Option	Why
--ready	Runs Ready tests only. A Draft someone is still editing is listed as skipped and cannot block a release. Recommended for every deploy gate.
--env Staging	The AutoTestX environment the deploy went to, by name
--timeout 20	Fail (and cancel the suite) if it takes longer than 20 minutes. Default 60.
--release v2.14.0	The version being tested. Shown first on the suite run, and filterable in Recent suite runs. Or set ATX_RELEASE.
--note "hotfix INC-1234"	One line shown on the suite run
--build, --commit, --branch	Override what was read from the CI system
--json smoke.json	Keep the result as a build artifact
--junit smoke.xml	Write the suite run as JUnit XML, so the pipeline shows each test in its own test tab (below)

Exit codes: 0 passed, 1 failed or error, 2 the request was refused (wrong token, unknown suite or environment), 130 cancelled. All options: CLI → Running a suite on the server.

Show the tests in the pipeline’s test tab

Add --junit smoke.xml to the atx command, then hand the file to the CI system’s own test report step. Each AutoTestX test appears as a test case, with the failed step and its error as the failure message and a link to the run report. A data-driven test is one case per row. The run with if: always() / when: always matters: the report is wanted most when the suite failed. How statuses map: Report exports → JUnit XML.

GitHub Actions has no test tab of its own. A report action turns the file into a check on the commit:

      - run: npm run atx -- run --suite Smoke --env Staging --ready --junit smoke.xml
        env:
          ATX_SERVER: https://app.autotestx.com
          ATX_TOKEN: ${{ secrets.AUTOTESTX_TOKEN }}
      - uses: mikepenz/action-junit-report@v5
        if: always()
        with: { report_paths: smoke.xml, check_name: AutoTestX Smoke }

The action needs permissions: checks: write on the job.

GitLab CI shows it in the merge request and the pipeline’s Tests tab:

  script:
    - npm ci
    - npm run atx -- run --suite Smoke --env Staging --ready --junit smoke.xml
  artifacts:
    when: always
    reports:
      junit: smoke.xml

Azure Pipelines shows it in the run’s Tests tab:

        - script: npm run atx -- run --suite Smoke --env Staging --ready --junit smoke.xml
          env:
            ATX_SERVER: https://app.autotestx.com
            ATX_TOKEN: $(AUTOTESTX_TOKEN)
        - task: PublishTestResults@2
          condition: always()
          inputs: { testResultsFormat: JUnit, testResultsFiles: smoke.xml, testRunTitle: AutoTestX Smoke }

Jenkins shows it under Test Result with the JUnit plugin:

      steps {
        sh 'npm ci'
        sh 'npm run atx -- run --suite Smoke --env Staging --ready --junit smoke.xml'
      }
      post {
        always { junit allowEmptyResults: true, testResults: 'smoke.xml' }
      }

Jenkins marks a build with test failures unstable. The atx step’s exit code 1 still fails the stage, so the production deploy is still skipped.

GitHub Actions without the CLI: the gate script

For a repository that does not contain AutoTestX, use the one-file gate script in examples/github-actions/autotestx-gate.mjs. Copy it into the repository and call it from a step. It needs only Node, reads the commit, branch, run number and link from GitHub’s own variables, waits for the suite, writes a results table to the run’s summary page, and fails the job when the suite does not pass.

- uses: actions/checkout@v4
- run: exec node .github/autotestx/autotestx-gate.mjs   # exec: a cancelled run reaches the gate
  env:
    ATX_SERVER: ${{ vars.ATX_SERVER }}
    ATX_TOKEN: ${{ secrets.AUTOTESTX_TOKEN }}
    ATX_SUITE: Smoke
    ATX_ENVIRONMENT: Staging

A complete workflow (deploy to staging → Smoke → deploy to production only when it passes), the full script, and a step-by-step setup for a real GitHub repository are in GitHub Actions pipeline.

Without the CLI: two HTTP calls

Any tool that can send HTTP requests can do the same: a shell step, PowerShell, a release tool, an internal deploy service.

1. Start the suite.

curl -sS -X POST "$ATX_SERVER/api/executions" \
  -H "Authorization: Bearer $ATX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "suite": "Smoke",
        "environment": "Staging",
        "scope": "ready",
        "deployment": { "release": "v2.14.0", "source": "release-tool", "build": "4711",
                        "commit": "a1b2c3d", "branch": "main",
                        "url": "https://ci.example.com/builds/4711" }
      }'
# → 202 { "executionId": "exec_…", "tests": 12, "status": "running",
#         "url": "https://app.autotestx.com/#/suites/exec_…",
#         "statusUrl": "https://app.autotestx.com/api/executions/exec_…" }

2. Wait for the answer. ?wait=60 holds the request until the suite finishes, or 60 seconds at most, so the loop asks about once a minute rather than every few seconds.

#!/usr/bin/env bash
set -euo pipefail
start=$(curl -sS -X POST "$ATX_SERVER/api/executions" \
  -H "Authorization: Bearer $ATX_TOKEN" -H "Content-Type: application/json" \
  -d "{\"suite\":\"Smoke\",\"environment\":\"Staging\",\"scope\":\"ready\",
       \"deployment\":{\"commit\":\"$GIT_COMMIT\",\"branch\":\"$GIT_BRANCH\"}}")
id=$(echo "$start" | jq -r .executionId)
[ "$id" != "null" ] || { echo "AutoTestX refused: $start"; exit 2; }

while :; do
  result=$(curl -sS "$ATX_SERVER/api/executions/$id?wait=60" -H "Authorization: Bearer $ATX_TOKEN")
  status=$(echo "$result" | jq -r .status)
  [ "$status" = "running" ] || break
done

echo "$result" | jq -r '"Smoke: \(.status) - \(.counts.passed)/\(.counts.total) passed - \(.url)"'
[ "$status" = "passed" ]

PowerShell:

$h = @{ Authorization = "Bearer $env:ATX_TOKEN" }
$body = @{ suite = 'Smoke'; environment = 'Staging'; scope = 'ready'
           deployment = @{ commit = $env:BUILD_SOURCEVERSION; branch = $env:BUILD_SOURCEBRANCHNAME } } | ConvertTo-Json
$start = Invoke-RestMethod -Method Post "$env:ATX_SERVER/api/executions" -Headers $h -ContentType 'application/json' -Body $body
do {
  $r = Invoke-RestMethod "$env:ATX_SERVER/api/executions/$($start.executionId)?wait=60" -Headers $h
} while ($r.status -eq 'running')
"Smoke: $($r.status) - $($r.counts.passed)/$($r.counts.total) passed - $($r.url)"
if ($r.status -ne 'passed') { exit 1 }

Every field of the request and response, and every refusal, is in the REST API reference → Suite executions.

Hosting platforms’ “deploy hooks” (the outgoing webhooks of Vercel, Netlify, Render and similar) cannot add an Authorization header or choose the request body, so they cannot call AutoTestX directly. Add a step to the pipeline that deploys instead, or have a small function receive the hook and make the call above.

Label nightly runs with the deployed version

A pipeline that runs a suite with deployment and an environment also records that deploy as what the environment now runs. Every later suite run there that does not name a deploy of its own is labelled with it: the nightly schedule, a person’s Run suite, an AI assistant’s run. A failing nightly run then says Deployed v2.14.0 · … · recorded 9h ago without anyone typing a version.

If your deploy job does not run tests itself, record the deploy on its own:

npx atx deployed --env Staging --release v2.14.0

In CI, the commit and build are read from the CI system as with atx run --suite; on a tag build the tag is the release. Without the CLI:

curl -sS -X PUT "$ATX_SERVER/api/environments/Staging/deployed"   -H "Authorization: Bearer $ATX_TOKEN" -H "Content-Type: application/json"   -d '{ "release": "v2.14.0", "commit": "a1b2c3d" }'

PowerShell:

Invoke-RestMethod -Method Put -Uri "$env:ATX_SERVER/api/environments/Staging/deployed" `
  -Headers @{ Authorization = "Bearer $env:ATX_TOKEN" } -ContentType 'application/json' `
  -Body (@{ release = 'v2.14.0'; commit = 'a1b2c3d' } | ConvertTo-Json)

Environments shows what each environment runs and when it was recorded. Forget deploy there stops labelling runs with it - for an environment that was rebuilt, or that a pipeline no longer reports on. A recorded deploy is only as current as the last pipeline that said so, which is why runs show recorded … ago beside it.

What you see in AutoTestX

• Test Suites → Recent suite runs lists the run marked deploy, with its release beside the suite. Hover over it for the commit. Pick a release in All releases to see only its runs.
• The suite run shows a Deploy line: Deploy v2.14.0 · github-actions · main @ a1b2c3d · build 4711 · pipeline run, with a link back to the pipeline.
• A run labelled from its environment shows Deployed instead of Deploy, with when the deploy was recorded, and is not marked deploy in the history.
• A pipeline can ask the same: GET /api/executions?release=v2.14.0 or ?commit=a1b2c3d.
• Every test has its own run report, as with any suite run.
• Failure emails: see Notifications. People on the project’s pipeline runs list get one email when a suite starts failing on an environment and one when it passes again.

Good practice

• Keep the deploy gate small and fast. A Smoke suite of the 10–20 most important flows that finishes in a few minutes. Run the full regression on a schedule.
• Use --ready / "scope": "ready", so only reviewed tests can stop a release.
• One token per pipeline, named after it (GitHub Actions - web main). You can revoke one without breaking the others, and runs show which pipeline started them.
• Set a timeout shorter than your pipeline’s own, so AutoTestX cancels the suite cleanly instead of the job being killed half way.

Limits

• One API token can have 5 suite runs in progress at once. One more is refused with 429 (too_many_running) until one finishes. This stops a pipeline stuck in a loop from filling the organization’s runners. The server administrator can change it with ATX_MAX_RUNNING_PER_TOKEN.
• How many tests run at the same time is set by the organization’s plan. Tests beyond that wait in the queue.
• ?wait= holds a request for 120 seconds at most.

Troubleshooting

You see	Why	Fix
401 invalid_token / expired_token	Token wrong, revoked or expired	Create a new token and update the CI secret
403 does not allow running tests	A Viewer token	Use a token with the Tester role
404 suite has no shared tests	Wrong suite name, or its tests are private	Check the name on Test Suites; share the tests
404 Environment “…” not found	The environment name is different in AutoTestX	Use the name shown under Environments
429 too_many_running	Earlier runs from this token are still going	Wait, cancel them, or lower how often the pipeline triggers
Every test skipped	--ready and no test in the suite is Ready	Mark the reviewed tests Ready

To try everything step by step, with a pipeline played from PowerShell, see the exercises. For release labels and what is deployed on an environment, see the release metadata exercises.

Practise this

• CI and releases — 5. Run after deployment, notifications and MCP: a pipeline played from PowerShell, deploy emails, the MCP server
• CI and releases — 6. Release metadata: 14 exercises: release, build and commit on a suite run