Jenkins pipeline

Set up a Jenkins pipeline that deploys to staging, runs the AutoTestX Smoke suite, and deploys to production only if Smoke passes. Each AutoTestX test also appears under the build’s Test Result.

 Build Now / a push to main
      │
      ▼  Jenkins (your server)
 ┌───────────────────┐   ┌──────────────────────────────┐   ┌──────────────────────┐
 │ Deploy to staging │──►│ Smoke test                   │──►│ Deploy to production │
 │ (your deploy)     │   │ node ci/autotestx-gate.mjs   │   │ only when Smoke      │
 └───────────────────┘   │   POST /api/executions ──────┼─┐ │ passed               │
                         │   GET  …?wait=60 (repeat) ───┼─┤ └──────────────────────┘
                         │   JUnit XML → Test Result    │ │
                         │   exit 0 = pass / 1 = fail   │ │ HTTPS + API token
                         └──────────────────────────────┘ ▼
                                             https://app.autotestx.com
                                             runs Smoke on Staging, emails on failure

You need two files. Download them here, or copy them from this page:

• Download Jenkinsfile: the pipeline (shown below)
• Download autotestx-gate.mjs: the gate script, the same one the GitHub Actions pipeline uses

File	Goes in your repository as	What it is
Jenkinsfile	Jenkinsfile (the repository root)	The pipeline: three stages
autotestx-gate.mjs	ci/autotestx-gate.mjs	Starts the suite, waits, prints the results, saves them as JUnit XML, and fails the stage if the suite fails. One file, no dependencies, needs only Node.

Why a gate script and not atx run --suite? The AutoTestX CLI ships inside the AutoTestX project and is not published to npm, so an ordinary repository cannot install it. The gate script does the same job in one file you copy in. If your Jenkins checkout does contain AutoTestX, the CLI works too: see Run after deployment → Jenkins.

Time needed: about 30 minutes.

---

1. What Jenkins needs

Network	The agent that runs the build must reach https://app.autotestx.com over HTTPS (outbound port 443). AutoTestX never connects to Jenkins.
Node.js	18 or newer on that agent (node --version). No npm install: the gate script has no dependencies.
Plugins	Pipeline, Git, Credentials Binding and JUnit. All four come with Jenkins’ suggested plugins.

No Node on your agents? Run the stage in a container instead, with the Docker Pipeline plugin: agent { docker { image 'node:22' } } on the Smoke test stage. Or install Node with the NodeJS plugin and add tools { nodejs 'node-22' } (the name you gave it under Manage Jenkins → Tools).

2. Prepare AutoTestX (in the browser, on app.autotestx.com)

1. Plan: the organization must be on Team or higher (API tokens are not in the proof of value).
2. Environment named Staging, pointing at your staging site.
3. Suite Smoke with your key shared, Ready tests. To try the pipeline first, use one test that passes and one that fails on purpose, such as Open checkout and Apply discount code from the exercises, step 0.7.
4. API token: Configurations → Project Members → API tokens for CI → name it after the job, e.g. Jenkins - shop, role Run tests (Tester), expiry 90 days. Copy the value: it is shown once.
5. Notifications (Test Suites page): leave Deploy pipeline runs on its default, or add yourself.

3. Store the token in Jenkins

Manage Jenkins → Credentials → System → Global credentials (unrestricted) → Add Credentials:

Field	Value
Kind	Secret text
Scope	Global
Secret	the atx_pat_… value from step 2.4
ID	autotestx-token (the Jenkinsfile asks for this ID)
Description	AutoTestX API token (Tester)

The Jenkinsfile reads it with credentials('autotestx-token'), and Jenkins masks it as **** in the build log. The address https://app.autotestx.com is not secret, so it sits in the Jenkinsfile.

4. Add the two files to your repository

1. Jenkinsfile at the root of the repository: all of the Jenkinsfile.
2. ci/autotestx-gate.mjs: the gate script, unchanged.

Commit and push both to main.

5. Create the pipeline job

1. New Item → a name, e.g. shop-deploy → Pipeline → OK.
2. Under Pipeline: Definition Pipeline script from SCM → SCM Git → your Repository URL (and its credentials if the repository is private) → Branch Specifier */main → Script Path Jenkinsfile.
3. To build on every push, tick a trigger under Triggers: GitHub hook trigger for GITScm polling when your Git host sends a webhook to Jenkins, or Poll SCM with H/5 * * * *.
4. Save → Build Now.

A Multibranch Pipeline works the same way: it finds the Jenkinsfile on each branch.

6. Read the first build

In Jenkins: the build’s stages show Deploy to staging ✅, Smoke test ❌ and Deploy to production skipped, because Smoke failed. That is the gate working. The Smoke test stage’s console output:

AutoTestX: Smoke on Staging - 2 tests · exec_…
Follow it: https://app.autotestx.com/#/suites/exec_…
[FAIL] Apply discount code                      3/4 steps 1.7s
         https://app.autotestx.com/#/runs/run_…
[PASS] Open checkout                            2/2 steps 0.8s
JUnit XML saved to autotestx-smoke.xml

FAILED  suite Smoke - 1/2 passed, 1 failed
Error: AutoTestX Smoke failed: 1/2 passed, 1 failed. https://app.autotestx.com/#/suites/exec_…

The build’s Test Result lists both tests. The failed one shows the failed step and its error, with a link to the AutoTestX run report. Because a test failed, the JUnit plugin marks the build unstable; the gate’s exit code 1 still fails the stage, so production is not deployed.

In AutoTestX: Test Suites → Recent suite runs → the new run is marked deploy:

Deploy · jenkins · origin/main @ 9f3c2e1a7b4d · #42 · pipeline run shop-deploy build 42

pipeline run opens the Jenkins build. The people on the Deploy pipeline runs list get an email “Smoke is failing on Staging after a deploy” with the same details.

Fix it and build again: fix the failing test in AutoTestX (Apply discount code → New revision → step 4 expected text Code expired → Status Ready → Save), then Build Now. All three stages go green, and AutoTestX emails “Smoke is passing again on Staging”.

7. Using it for a real application

• Replace the two echo deploy stages with your real deploy steps.
• Keep Smoke small and fast (10–20 key flows). Run the full regression on an AutoTestX schedule.
• One token per Jenkins job, so each can be revoked alone.
• On a Windows agent, use bat 'node ci\\autotestx-gate.mjs' instead of sh.
• Optional: put an input step before Deploy to production, so a person confirms even after Smoke passes.
• Aborting the build while Smoke test runs sends the gate a stop signal, and it cancels the suite on AutoTestX: tests not yet started are skipped and the suite run ends CANCELLED.

Settings of the gate script

Set them in the stage’s environment { } block.

Variable	Required	Default	Meaning
ATX_SERVER	yes		AutoTestX address
ATX_TOKEN	yes		API token with the Tester role, from credentials('…')
ATX_SUITE	yes		The suite to run
ATX_ENVIRONMENT		none	Environment name or id
ATX_SCOPE		ready	ready = Ready tests only; all = Drafts too
ATX_TIMEOUT_MIN		30	Cancel the suite and fail after this long
ATX_NOTE			One line shown on the suite run
ATX_JUNIT			Save the finished suite run as JUnit XML to this file, for the junit step

Read automatically from Jenkins: GIT_COMMIT (commit), GIT_BRANCH or BRANCH_NAME (branch), BUILD_NUMBER (build number) and BUILD_URL (the link back). Jenkins sets the GIT_* ones when the pipeline comes from SCM. Nothing is sent to Jenkins.

Exit codes: 0 passed · 1 failed or error · 2 refused (token, suite, environment, settings) · 130 cancelled.

Troubleshooting

In the console output	Why	Fix
node: not found	No Node on the agent	Install Node 18+, or use a node:22 docker agent or the NodeJS plugin (step 1)
Could not find credentials entry with ID ‘autotestx-token’	The credential ID differs	Use the ID autotestx-token, or change it in the Jenkinsfile
Error: Set ATX_SERVER, ATX_TOKEN…	A setting is missing from environment { }	Compare with the Jenkinsfile
Cannot reach https://…: fetch failed	The agent cannot reach AutoTestX (firewall, proxy)	Run curl -I https://app.autotestx.com on the agent
401 Invalid or revoked API token	Wrong, revoked or expired token	New token → update the credential
403 … does not allow running tests	A Read-only token	Use a Run tests token
404 The suite … has no shared tests	Suite name differs, or its tests are private	Use the name shown on Test Suites; share the tests
404 Environment "…" not found	Environment name differs	Use the name shown under Environments
Every test SKIP	ATX_SCOPE is ready and no test is Ready	Mark the reviewed tests Ready
Could not save the JUnit XML (…)	The report could not be downloaded	The build result is not affected. Check the address and the token’s access.
No commit or branch on the suite run	The job’s script is not from SCM, so Jenkins sets no GIT_* variables	Use Pipeline script from SCM, or describe the build with ATX_NOTE
429 … 5 suite runs in progress	Many builds at once with one token	Keep disableConcurrentBuilds()

How the pipeline, the API and the emails fit together is explained in Run after deployment and Notifications.

The Jenkinsfile

Jenkinsfile (download):

// Deploy to staging, gate on the AutoTestX Smoke suite, then deploy to production.
//
// Put this file at the root of your repository as Jenkinsfile, and the gate script
// (examples/github-actions/autotestx-gate.mjs) as ci/autotestx-gate.mjs. In Jenkins, add a
// "Secret text" credential with the ID autotestx-token: an AutoTestX API token with the
// Tester role.
//
// The agent needs Node 18 or newer. The two deploy stages only print a line: replace them
// with your real deploy.
pipeline {
  agent any

  options {
    disableConcurrentBuilds()          // one deploy at a time; a newer build waits
    timeout(time: 40, unit: 'MINUTES') // longer than the gate's 30, so the gate cancels cleanly first
  }

  environment {
    ATX_SERVER = 'https://app.autotestx.com'
  }

  stages {
    stage('Deploy to staging') {
      steps {
        sh 'echo "Deploying ${GIT_COMMIT} to staging (replace with your real deploy)"'
      }
    }

    stage('Smoke test') {
      environment {
        ATX_TOKEN       = credentials('autotestx-token') // masked as **** in the build log
        ATX_SUITE       = 'Smoke'
        ATX_ENVIRONMENT = 'Staging'
        ATX_SCOPE       = 'ready'
        ATX_TIMEOUT_MIN = '30'
        ATX_NOTE        = "${env.JOB_NAME} build ${env.BUILD_NUMBER}"
        ATX_JUNIT       = 'autotestx-smoke.xml'
      }
      steps {
        sh 'node ci/autotestx-gate.mjs'
      }
      post {
        always {
          // Each AutoTestX test under the build's Test Result, even when Smoke failed.
          junit allowEmptyResults: true, testResults: 'autotestx-smoke.xml'
        }
      }
    }

    stage('Deploy to production') { // skipped when the Smoke suite fails
      steps {
        sh 'echo "Smoke passed - deploying ${GIT_COMMIT} to production (replace with your real deploy)"'
      }
    }
  }
}

Part	What it does
agent any	Runs on any agent. It needs Node 18+ for the Smoke test stage.
disableConcurrentBuilds()	One deploy at a time. A newer build waits instead of racing the older one.
timeout(time: 40, …)	Jenkins’ own limit for the build. It’s longer than the gate’s 30 minutes, so the gate cancels the suite cleanly first.
environment { ATX_SERVER = … }	The AutoTestX address, for every stage
Deploy to staging	Your deploy to staging. Replace the echo with your real deploy.
Smoke test, environment { }	The token from the credential, the suite, environment, scope, time limit, a note, and where to save the JUnit XML
Smoke test, post { always { junit … } }	Publishes the tests under Test Result, also when Smoke failed
Deploy to production	Runs only when Smoke test succeeded. A failing Smoke suite fails the stage, and Jenkins skips the rest.

Practise this

• CI and releases — 5. Run after deployment, notifications and MCP: a pipeline played from PowerShell, deploy emails, the MCP server
• CI and releases — 6. Release metadata: 14 exercises: release, build and commit on a suite run