Writing an Evidence

An Evidence is a short test that fails on the Proposal and passes once the Defect is fixed. The failure is the proof: it shows the problem in the running application, not only in the diff.

It is a black-box test. It talks to the application only through HTTP or a browser, as a user or another program would, and never reads its code or its database. Write it in one of two formats:

• Hurl, for HTTP requests: an API, a redirect, a header, a status code.
• Playwright, for pages in a browser: what a user sees, fills in and clicks, and accessibility.

The Evidence never hard-codes an address, an id or a password: it gets them as variables.

• base_url is the address of the application during the run. It changes on every run.
• The other variables come from the data the application starts with: the ids of its content and the credentials of its test accounts. For example facility_a_slot_1 is the id of a slot and customer_a_basic is the HTTP Basic credential of a customer, already encoded.

Open the application of the Exercise to see them: the Environment page lists each variable with its value, and so does the Evidence form on the Exercise page once the Environment is ready. The values are the same on every run, because each run starts from the same data.

A Hurl file is a list of HTTP requests in plain text. Each request names a method and an address, then optional headers and a body. Under it, HTTP and a status code say what the response must be, and an [Asserts] section can check more. Write variables between double braces: {{base_url}}, {{customer_a_basic}}.

# Customer A already holds a confirmed booking of this slot: a second one is refused.
POST {{base_url}}/api/slots/{{facility_a_slot_1}}/book
Authorization: Basic {{customer_a_basic}}
HTTP 409

# Control: a slot that customer A has not booked yet can still be booked.
POST {{base_url}}/api/slots/{{facility_a_slot_2}}/book
Authorization: Basic {{customer_a_basic}}
HTTP 201

The requests run in order and stop at the first one whose response does not match. Cookies carry over from one request to the next, so a session started by a login form stays open. Lines that start with # are comments.

Check more than the status code with [Asserts]:

GET {{base_url}}/api/bookings
Authorization: Basic {{customer_a_basic}}
HTTP 200
[Asserts]
header "Content-Type" contains "application/json"
jsonpath "$.bookings" count == 1
jsonpath "$.bookings[0].email" not exists

Save a value from a response with [Captures] and use it in the next requests as a variable, for example the token of a form:

GET {{base_url}}/user/login
HTTP 200
[Captures]
form_build_id: xpath "string(//input[@name='form_build_id']/@value)"

The Evidence runs with Hurl 7.1 in test mode. Upload it as a .hurl file or paste it with the format Hurl.

A Playwright Evidence is one Playwright test file in TypeScript. It drives Chromium: it opens pages, fills in fields, clicks, and checks what the page shows with expect.

• baseURL is the address of the application, so page.goto('/user/login') opens a page of it.
• The variables are in the JSON file whose path is in process.env.FIXTURES_PATH.
• It can import @playwright/test, @axe-core/playwright for accessibility checks and the modules of Node.js.

import { test, expect, type Page } from '@playwright/test';
import { readFileSync } from 'node:fs';

const fixtures = JSON.parse(readFileSync(process.env.FIXTURES_PATH!, 'utf8'));

// Signs in through the login form, as a customer does in the browser.
async function signIn(page: Page, basic: string): Promise<void> {
    const [name, password] = Buffer.from(basic, 'base64').toString().split(':');
    await page.goto('/user/login');
    await page.getByLabel('Username').fill(name);
    await page.getByLabel('Password').fill(password);
    await page.getByRole('button', { name: 'Log in' }).click();
}

test('a wrong phone number is reported on the phone field', async ({ page }) => {
    await signIn(page, fixtures.customer_a_basic);
    await page.goto(`/slots/${fixtures.facility_a_slot_2}/book`);
    await page.getByLabel('Phone').fill('call me');
    await page.getByRole('checkbox').check();
    await page.getByRole('button', { name: 'Book' }).click();

    await expect(page.getByText('Enter a valid phone number')).toBeVisible();
    await expect(page.getByLabel('Phone')).toHaveAttribute('aria-invalid', 'true', { timeout: 2000 });
});

An accessibility check with axe, in the same file:

import AxeBuilder from '@axe-core/playwright';

test('the booking form has no form-label violations', async ({ page }) => {
    await page.goto(`/slots/${fixtures.facility_a_slot_2}/book`);
    const results = await new AxeBuilder({ page }).withRules(['label']).analyze();
    expect(results.violations).toEqual([]);
});

Fail on an expect. A run counts as failed only when every failed test stopped on an expect. A test that stops on an action, such as a click on a button that does not exist, ends in an error, because it shows that the test went wrong, not the application. Locate elements by role, label or text, as a user finds them.

Give a short timeout to the expect that proves the Defect. It fails on the Proposal, and by default it waits 5 seconds before it does. Each test has 40 seconds and the whole file 50.

The tests run one at a time, with no retries. Upload the Evidence as a .spec.ts file or paste it with the format Playwright.

Run on my Environment runs the Evidence against your Environment, with its data as it is now. Run it as often as you want while you write it: it does not count for the Review. If an earlier run or your own clicks changed the data, choose Reset data on the Environment page first.

When you Submit an Attempt, the Evidence runs again on new copies of the Proposal and of the fix of each Defect, all started from the same data. A Finding proves a Defect when its Evidence fails on the Proposal and passes on the fix of that Defect.

Each run has a limit of 60 seconds and no network access other than the application.

• Fails: an assertion does not hold. On the Proposal, this is what you want.
• Passes: every assertion holds. On the Proposal, the Evidence does not show the problem yet: check that it tests the rule that the Proposal breaks.
• Error: the Evidence could not run. The cause is in Output: a syntax error, an address that does not answer, a timeout, a Playwright test that stopped on an action instead of an expect, a file with no test.

• Test the rule, not the code. Start from the sentence of the Task, the requirement or the Checklist item that the Proposal breaks, and check what a user or a client would observe.
• Add a control. A step that must work on both versions, such as booking a free slot, keeps the Evidence from passing on a fix that only blocks everything.
• One problem per Evidence. An Evidence that checks two Defects cannot pass on the fix of only one of them.
• Use the variables for every address, id and credential, so the Evidence works on every copy of the application.
• Keep it short. A few requests or a single test say more than a long scenario, and they fail where the problem is.