Course: Diagnosing failures — 2. Logs, video and trace
Thirteen hands-on exercises on the evidence a run keeps to tell an application failure from a test failure (Running tests and reading reports):
- Module 1: what the application reported around a failed step: server errors, crashes, refused and missing APIs, and the console and network logs;
- Module 2: the video and Playwright trace of a run, and which runs keep them;
- Module 3: secrets in recordings;
- Module 4: the server settings, and how long recordings are kept;
- Module 5: the same evidence as hints, for AI assistants.
Each exercise has:
- Goal: what you will see;
- Before you start: the preparation it needs;
- Steps: Action, Expected result, and a box to tick;
- What happened: the explanation;
- Check yourself: a question to answer before moving on. The answers are at the end.
Allow about 1 hour. Do the exercises in order: each one starts from what the one before left.
Three things a run keeps
Section titled “Three things a run keeps”| Evidence | Kept for | Where you see it |
|---|---|---|
| Console and network logs | every run | A failed step’s What the application reported box, and Console log / Network log in the run summary |
| Video | failed runs | The Recording card |
| Trace | failed runs, and passed runs where a step needed a retry | Download trace in the Recording card |
Preparation
Section titled “Preparation”The exercises use a small demo shop whose pages fail in known ways. The browser that runs your tests has to reach it, so do these exercises on your own installation (Deployment): a hosted server cannot reach a shop running on your computer. Module 4 also restarts the server.
P1: Start the demo shop
Section titled “P1: Start the demo shop”In a PowerShell window in the AutoTestX folder:
node examples/failure-evidence/shop.mjsIt prints shop on http://127.0.0.1:4420/. Leave it running. Its pages:
| Page | What is wrong with it |
|---|---|
/ Checkout |
The order API answers 500, the payment service is down, and the Pay now button crashes |
/ok Welcome |
Nothing: it works |
/slow Saving |
“Saved” appears 2.5 seconds after the page loads |
/noisy Noisy dashboard |
Logs 3,000 lines to the console, then one error |
/session My account |
Its API calls answer 401 and 404 |
/form Sign in |
A password field and a plain text field |
P2: Start AutoTestX with its own data
Section titled “P2: Start AutoTestX with its own data”In a second PowerShell window, so your normal data is not touched:
npm run buildnpm run build:web$env:ATX_DATA_DIR = ".data-evidence-training"; npm run startOpen the link printed after set the system administrator’s password, choose a password and sign in as the system administrator.
P3: Practice tests and environment
Section titled “P3: Practice tests and environment”- Test Cases → Import the seven files in
examples/failure-evidence/. They land in the suite Failure evidence:
| File | Test | Result |
|---|---|---|
01-checkout-crash.json |
Checkout fails | fails at “See the receipt” |
02-passes.json |
Welcome page | passes |
03-flaky-save.json |
Save is slow | passes, after retries |
04-noisy-dashboard.json |
Noisy dashboard | fails at “See the chart” |
05-session-expired.json |
Session expired | fails at “See my profile” |
06-secret-in-password.json |
Secret in a password field | fails at “See the dashboard” |
07-secret-in-text-field.json |
Secret in a text field | fails at “See the dashboard” |
- Configurations → Environments → New environment: name
Local shop, Variables (JSON){ "baseUrl": "http://127.0.0.1:4420" }, Save environment. - In Local shop, under Add a secret: NAME
APP_PASSWORD, valueTr1al-Pa55word!, Add secret.
Before every run, check that the environment selector above the steps says Local shop.
To open a run’s report: Test Runs → click the run. Its ID (run_…) is in the header, next to
the date.
Module 1: What the application reported
Section titled “Module 1: What the application reported”Exercise 1: A failure the application caused
Section titled “Exercise 1: A failure the application caused”Goal: see, under the failed step, the errors that say the application broke.
Before you start: P1-P3.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open Checkout fails → Run test | Steps 1 and 2 pass; step 3 See the receipt fails after about 5 seconds | |
| 2 | Open the run’s report and expand step 3 | Under the red error, a box What the application reported around this step | |
| 3 | Read its rows, oldest first | 500 POST 127.0.0.1:4420/api/orders · Console order request failed with 500 · Crash TypeError: Cannot read properties of undefined (reading ‘id’) · Failed GET 127.0.0.1:4421/payments/sdk.js · net::ERR_CONNECTION_REFUSED | |
| 4 | Look at the time at the end of each row | About −0.5s: before step 3 started, during the click | |
| 5 | Expand steps 1 and 2 | No such box: only a failed step has one |
What happened: the step’s own error, Expected the element to exist, but it was not found, reads like a broken test. The box tells the real story: the click sent an order, the server answered 500, and the page crashed trying to read the order it never got. The receipt was never drawn. The box lists only problems from the failed step and the 10 seconds before it: errors, crashes, failed requests and 4xx/5xx answers.
Check yourself: the page also logged checkout ready and a warning. Why are they not in the box?
Exercise 2: The full logs
Section titled “Exercise 2: The full logs”Goal: read everything the page logged, not only the problems.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | In the same report, look at Run summary | Two new rows: Console log 2 and Network log 2, with red dots | |
| 2 | Click Console log | One JSON line per entry: checkout ready, the legacy pricing warning, order request failed with 500, and the TypeError with its stack. The first two have "stepId":"open", the last two "stepId":"pay" |
|
| 3 | Go back, click Network log | Three lines: the page itself (status 200), the order call (500), the payment script (net::ERR_CONNECTION_REFUSED). No request or response bodies, no headers |
|
| 4 | Run Welcome page and open its report | It passes. Run summary: Console log 0, Network log 0, with green dots |
What happened: the logs are kept for every run, passed or failed. The number next to each is how many problems it holds. Each entry is filed under the step that was running, so you can tell what the click caused from what the page did on its own. The network log has addresses, statuses and timings only: bodies and headers are where passwords and personal data live.
Check yourself: a passed run’s logs are rarely read. Why keep them?
Exercise 3: Session expired, or an API missing
Section titled “Exercise 3: Session expired, or an API missing”Goal: tell a test-user or environment problem from an application bug.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Run Session expired; open the report; expand step 2 | 401 GET 127.0.0.1:4420/api/me and 404 GET 127.0.0.1:4420/api/v2/cart | |
| 2 | Look at Run summary | Network log 2, and no Console log row |
What happened: a 401 or 403 says the session expired or the test user lacks a permission: check the environment’s credentials before the test. A 404 from an API says that API is not deployed here, or the environment’s URL is wrong. There is no Console log row because the page logged nothing at all.
| You see | It points at |
|---|---|
| 500, 502, 503… | The application’s server |
| Crash | The application, in the browser |
| 401 or 403 | The session or the test user |
| 404 on an API call | An API not deployed here, or the wrong environment URL |
| Failed (no answer) | The environment or the network |
| Nothing | The application reported no problem: look at the test |
Check yourself: a step fails and the box shows only a 403. What do you check first?
Exercise 4: A log that hits its limit
Section titled “Exercise 4: A log that hits its limit”Goal: see what happens when a page logs too much.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Run Noisy dashboard; open the report | It fails at See the chart. Run summary: Console log 1, then a grey 1323 not kept (give or take a few) | |
| 2 | Hover over 1323 not kept | The log reached its size limit | |
| 3 | Click Console log and scroll to the end | It ends around widget 1678 refreshed. The file is just under 512 KB |
What happened: each log keeps at most 512 KB, so a page logging in a loop cannot fill the server’s disk. The page’s 3,000 lines did not fit: the log kept the first entries and counted the rest as not kept. The 1 is the page’s final error, dashboard failed…: it was counted, but it arrived after the log was full, so it is not in the file or under the step.
Check yourself: what does not kept tell you when you read the report of a run like this?
Module 2: Video and trace
Section titled “Module 2: Video and trace”Exercise 5: Watch the failure, step through it
Section titled “Exercise 5: Watch the failure, step through it”Goal: see how the page got to the failure.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open the report of Checkout fails (Exercise 1) | In the right-hand column, above Run summary, a Recording card: a video, and Download trace and open it at trace.playwright.dev… | |
| 2 | Play the video, then drag its progress bar | About 5 seconds of the Checkout page waiting for a receipt that never comes. It jumps at once to wherever you drag | |
| 3 | Click Download trace | trace.zip (about 30 KB) downloads |
|
| 4 | Open https://trace.playwright.dev and drop trace.zip on it |
A timeline of screenshots along the top, and the actions the test took on the left | |
| 5 | Click the click on #pay. Look at Before and After, then the Console and Network tabs |
The page before and after the click; the error and the TypeError; POST /api/orders with 500 |
What happened: a screenshot shows the moment a step failed. The video shows how the page got there: a spinner that never cleared, a dialog that closed, a redirect. The trace goes further: every action with the page as it was before and after, plus its console and network. trace.playwright.dev opens the file in your browser and uploads nothing.
Check yourself: when would you open the trace rather than the video?
Exercise 6: A passed run keeps no recording
Section titled “Exercise 6: A passed run keeps no recording”Goal: see that recordings are made on every run but kept only when they explain something.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open the report of Welcome page (Exercise 2) | No Recording card | |
| 2 | List its files: Get-ChildItem .data-evidence-training\runs\<run ID>\artifacts |
console.jsonl and network.jsonl only |
|
| 3 | Do the same for Checkout fails | Also video.webm, trace.zip and the failure screenshot |
What happened: a recording cannot start after the failure it should show, so every run is recorded from its first step. At the end, the runner decides: for a run that passed first time, both are deleted.
Check yourself: what does recording every run cost, if passed runs keep nothing?
Exercise 7: A flaky pass keeps its trace
Section titled “Exercise 7: A flaky pass keeps its trace”Goal: see why a run that passed only after retrying keeps a trace, but no video.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Open Save is slow. Click step 2 See Saved and open Run and failure in its properties | Timeout 500 ms, Retries 8: up to 9 tries of half a second each | |
| 2 | Run test | It passes in about 3 seconds | |
| 3 | Open the report | Step 2 has a badge such as 4 attempts (it can vary by one) | |
| 4 | Look at the right-hand column | A Recording card with Download trace only: no video | |
| 5 | Open the trace on trace.playwright.dev | The screenshots show Saving your changes… for about 2.5 seconds, then Saved |
What happened: the page adds Saved 2.5 seconds after it loads. Each attempt looks for it for up to half a second; when it is not there, the attempt fails and the next one starts at once. Only the step is retried: the page is not reopened.
| Time after the page loads | What happens |
|---|---|
| 0 – 0.6 s | Step 1: the page loads and the network goes quiet |
| 0.6 – 1.1 s | Attempt 1: not there yet → retry |
| 1.1 – 1.7 s | Attempt 2: not there yet → retry |
| 1.7 – 2.3 s | Attempt 3: not there yet → retry |
| from 2.3 s | Attempt 4: Saved appears at 2.5 s and is found → passed |
At the end of the run the runner decides what to keep:
| Rule | This run | |
|---|---|---|
| Video | Kept when the run failed | It passed: the video is deleted |
| Trace | Kept when the run failed or a step needed more than one attempt | 4 attempts: the trace is kept |
A pass that needed retries is a warning: the next time Saved takes 5 seconds, the test fails. The trace shows what you need to judge it, when each attempt ran and when the page changed, and its screenshots already show the page over time, so a video would add little.
Check yourself: the test passes every time. Why look at it at all?
Module 3: Secrets in recordings
Section titled “Module 3: Secrets in recordings”Exercise 8: A secret in a password field
Section titled “Exercise 8: A secret in a password field”Goal: see that a secret typed during a run does not end up in its recording.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Run Secret in a password field; open the report | It fails at step 3. The Recording card has a video and a trace | |
| 2 | Play the video | The Password field shows dots | |
| 3 | Download the trace. In PowerShell, in your Downloads folder: Expand-Archive trace.zip trace-08; Select-String -Path trace-08\* -Pattern 'Tr1al-Pa55word' -SimpleMatch |
No output: the secret is nowhere in the trace | |
| 4 | Select-String -Path trace-08\* -Pattern 'REDACTED' -SimpleMatch |
Lines with ***REDACTED***, where the value was typed |
What happened: a trace records what was typed and the page’s content. After the run, every
text file inside it has the run’s secrets replaced with ***REDACTED***. A trace that cannot be
cleaned this way is deleted, never kept as it was.
Check yourself: the step that typed the secret has sensitive ticked. What else does that do?
Exercise 9: A secret in a field that shows it
Section titled “Exercise 9: A secret in a field that shows it”Goal: see what happens when a recording cannot be cleaned.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Run Secret in a text field; open the report | The Recording card has no video and no trace, only: Step “Type the password into Note” typed a sensitive value into a field that shows it, so this run’s video and trace were not kept. | |
| 2 | List the run’s files | network.jsonl and the failure screenshot: no video.webm, no trace.zip |
What happened: a video is pictures: a secret typed into a visible field cannot be masked in it, and the trace’s screenshots show it too. So after a sensitive value is typed into anything but a password field, the run keeps neither, and says which step caused it.
Check yourself: how would you change the test so its failures keep a video again?
Module 4: Server settings and retention
Section titled “Module 4: Server settings and retention”Each exercise here restarts the server: press Ctrl+C in its window, then start it as shown. A setting stays in that PowerShell window until you remove it (Exercise 11, step 5).
Exercise 10: Video and trace settings
Section titled “Exercise 10: Video and trace settings”Goal: choose what the server records.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | $env:ATX_VIDEO = "off"; npm run start, then run Checkout fails |
Recording card: a trace, no video | |
| 2 | $env:ATX_VIDEO = ""; $env:ATX_TRACE = "always"; npm run start, then run Welcome page |
It passes and has a Recording card with a trace | |
| 3 | $env:ATX_TRACE = "off"; $env:ATX_VIDEO = "off"; npm run start, then run Checkout fails |
No Recording card. The What the application reported box and the logs are still there |
What happened: ATX_VIDEO (retain-on-failure or off) and ATX_TRACE
(on-retry-or-failure, always or off) decide what is recorded
(Configuration). The logs do not depend on them.
Video is the expensive one: every run is encoded so that the failures can be kept, which costs
CPU on every run.
Check yourself: your server is small and long suites slow it down. Which setting do you change first?
Exercise 11: Low disk, and watched runs
Section titled “Exercise 11: Low disk, and watched runs”Goal: see when the server keeps no recording, and says why.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | $env:ATX_TRACE = ""; $env:ATX_VIDEO = ""; $env:ATX_RECORDING_MIN_FREE_MB = "100000000"; npm run start |
The server starts | |
| 2 | Run Checkout fails | The Recording card says only: The server is low on disk space, so this run’s video and trace were not kept. | |
| 3 | $env:ATX_RECORDING_MIN_FREE_MB = ""; npm run start. Open Checkout fails, turn on Watch, Run test |
A browser window shows the run | |
| 4 | Open its report | Recording card: a trace, no video | |
| 5 | Remove-Item Env:ATX_VIDEO, Env:ATX_TRACE, Env:ATX_RECORDING_MIN_FREE_MB -ErrorAction SilentlyContinue; npm run start |
The server starts with the defaults |
What happened: below ATX_RECORDING_MIN_FREE_MB of free disk (2,048 by default), a run keeps
no recording: a full disk would stop the whole server, not only recordings. 100,000,000 MB is
100 TB, more than any disk has. A run you Watch keeps no video: you saw it happen.
Check yourself: where would you look if failed runs suddenly stop having videos?
Exercise 12: Old recordings are deleted
Section titled “Exercise 12: Old recordings are deleted”Goal: see recordings go after 14 days, while the report stays.
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Take the run ID of the Checkout fails run from Exercise 1. Get-ChildItem .data-evidence-training\runs\<run ID>\artifacts\* | ForEach-Object { $_.LastWriteTime = (Get-Date).AddDays(-20) } |
Nothing is printed | |
| 2 | Restart the server | A start-up line ends removed 2 video/trace file(s) older than 14 days | |
| 3 | List the run’s files | video.webm and trace.zip are gone; the logs and the screenshot are still there |
|
| 4 | Open its report | No Recording card; everything else is unchanged |
What happened: step 1 made that run’s files look 20 days old. The server deletes videos and
traces older than ATX_RECORDING_DAYS (14 by default) when it starts and every 6 hours.
Screenshots, logs and the report itself are never deleted this way.
Check yourself: why delete only videos and traces?
Module 5: Evidence for AI assistants
Section titled “Module 5: Evidence for AI assistants”Exercise 13: The failure context
Section titled “Exercise 13: The failure context”Goal: see the same evidence as an AI assistant reads it (AI assistants (MCP)).
| # | Action | Expected result | ✓ |
|---|---|---|---|
| 1 | Run Checkout fails again. In the browser, open http://127.0.0.1:4100/api/runs/<run ID>/failure-context |
JSON: failedStep.evidence lists the four problems from Exercise 1; logs and recording hold links to the logs, the video and the trace |
|
| 2 | Read hints |
The first: The server answered 500 to POST 127.0.0.1:4420/api/orders around step 3 (“See the receipt”). A server error points at the application or its back end, not at the test. Then the uncaught exception, then the unreachable payment service. was not found comes after them | |
| 3 | Do the same for the Session expired run (Exercise 3) | Hints about the 401 (the session expired or the test user lacks a permission) and the 404 (an API that is not deployed here, or the wrong environment URL) | |
| 4 | And for the Secret in a text field run (Exercise 9) | recording holds only withheld, with the reason |
What happened: an assistant asked “why did this fail?” reads this through the MCP server. The hints are rules, not a model, so the answer is the same with AI turned off. What the application reported comes first, because it outranks any guess from the step’s own error.
Check yourself: why does the 500 hint come before was not found?
Clean up
Section titled “Clean up”Stop both windows (Ctrl+C) and delete the practice data:
Remove-Item -Recurse -Force .data-evidence-trainingAlso delete trace.zip and trace-08 from your Downloads folder.
Answers
Section titled “Answers”- They are not problems. The box lists only errors, crashes, failed requests and 4xx/5xx answers; ordinary logs and warnings are in the Console log.
- Comparing a failed run with the last passed one: what the page logged when it worked is what tells you what changed.
- The test user’s credentials in the environment, or its permissions: a 403 means the server refused that user.
- That the log is incomplete: the entries you need may be among the ones left out, and the run’s own counts are the only record of them.
- When you need what the page was, not what it looked like: the elements, the values, the requests and their answers at each action.
- CPU: every run’s video is encoded, and the trace recorded, even though most are thrown away.
- A pass that needed retries can fail next time. Raise the step’s timeout or wait for the page to finish, before it turns into a real failure.
- It keeps the value out of the report, the logs and the step details, and it is what makes the runner check the field afterwards.
- Type the secret into a password field, or into a field the page hides, or untick sensitive if the value is not secret.
ATX_VIDEO=off: it removes the cost that every run pays, and the traces still explain failures.- The Recording card: it says why nothing was kept. Then the server’s free disk, and
ATX_RECORDING_MIN_FREE_MB. - They are by far the largest files. The report, screenshots and logs still explain an old failure.
- A 500 says the application broke. The missing element is a consequence of that, not the cause.