Writing First Test with Playwright

Let’s see how we can write a simple test that would use Playwright to open DuckDuckGo, search for a Mercury element, ensure that search results are loaded, include the Wikipedia article about the element, and check that its symbol is presented correctly on the page.

Install Dependencies

Make sure you have Alumnium installed in your project.

$ pip install alumnium pytest-playwright
$ playwright install chromium

$ npm install alumnium

Now initialize Playwright in your project if you haven’t done it yet. Leave all the options as default and remove example test after initialization.

$ npm init playwright
$ rm tests/example.spec.ts

We’ll also need to install Docker to run the Alumnium server. Please, refer to Docker installation guide for your platform.

Setup Browser

Start by creating a test file for your tests and instantiating a browser instance:

from playwright.sync_api import Page


def test_search(page: Page):
    page.goto("https://duckduckgo.com")

Now run the test. You should see a Chrome browser window with DuckDuckGo opened.

$ pytest --quiet --headed
...
1 passed in 1.32s

import { test, expect } from "@playwright/test";

test("should search", async ({ page }) => {
  await page.goto("https://duckduckgo.com");
});

Now run the test. You should see a Chrome browser window with DuckDuckGo opened.

$ npx playwright test --project chromium --headed

Running 1 test using 1 worker
  1 passed (4.0s)

Setup Alumnium

The next thing we need to do is to start the Alumnium server. It is responsible for communication between the browser and the AI model and must be running for the tests to work. You can start it using Docker:

$ docker run --rm --detach \
      --publish 8013:8013 \
      --env ALUMNIUM_MODEL \
      --env OPENAI_API_KEY \
      alumnium/alumnium

Now let’s add the code that would instantiate Alumnium using the browser. We can use a test fixture to ensure proper setup and teardown of the Alumnium instance:

from alumnium import Alumni
from playwright.sync_api import Page
from pytest import fixture


@fixture
def al(page: Page):
    al = Alumni(page)
    yield al
    al.quit()


def test_search(al: Alumni, page: Page):
    page.goto("https://duckduckgo.com")

The test should still work fine, let’s re-run it to make sure:

$ pytest --quiet --headed
...
1 passed in 7.81s

import { test as base } from "@playwright/test";
import { Alumni } from "alumnium";

export const test = base.extend<{ al: Alumni }>({
  al: async ({ page }, use) => {
    const al = new Alumni(page);
    await use(al);
    await al.quit();
  },
});

export { expect } from "@playwright/test";

import { test, expect } from "@playwright/test";
import { test, expect } from ".";

test("should search", async ({ page, al }) => {
  await page.goto("https://duckduckgo.com");
});

The test should still work fine, let’s re-run it to make sure:

$ npx playwright test --project chromium --headed

Running 1 test using 1 worker
  1 passed (8.2s)

Add Actions

Now let’s add some actions that Alumnium should do on the page. Our test needs to search for a Mercury element, so let’s use this exact command:

from alumnium import Alumni
from playwright.sync_api import Page
from pytest import fixture


@fixture
def al(page: Page):
    al = Alumni(page)
    yield al
    al.quit()


def test_search(al: Alumni, page: Page):
    page.goto("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")

Run the test, you should now see “Mercury element” typed in the search box and the page with results loaded.

$ pytest --quiet --headed
...
1 passed in 11.34s

import { test, expect } from ".";

test("should search", async ({ page, al }) => {
  await page.goto("https://duckduckgo.com");
  await al.do("search for 'Mercury element' and press Enter");
});

Run the test, you should now see “Mercury element” typed in the search box and the page with results loaded.

$ npx playwright test --project chromium --headed
...
  1 passed (8.6s)

Add Verifications

The next step is to add some verifications that Alumnium should check on the page. We are going to add two of them, one that checks that the page title contains the search keyword and one to see if the Wikipedia article is present in the results.

Common wisdom says to never trust a test you haven’t seen fail. Let’s add the first verification and see it fail!

from alumnium import Alumni
from playwright.sync_api import Page
from pytest import fixture


@fixture
def al(page: Page):
    al = Alumni(page)
    yield al
    al.quit()


def test_search(al: Alumni, page: Page):
    page.goto("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Aluminium word")

Now let’s run our test:

$ pytest --quiet --headed
F
...
E       AssertionError: The requested information is whether the page title contains the word 'Aluminium'. The page title is 'Mercury element at DuckDuckGo', which does not include the word 'Aluminium'. Therefore, the statement is false.
...
1 failed in 25.21s

import { test, expect } from ".";

test("should search", async ({ page, al }) => {
  await page.goto("https://duckduckgo.com");
  await al.do("search for 'Mercury element' and press Enter");
  await al.check("page title contains Aluminium word");
});

Now let’s run our test:

$ npx playwright test --project chromium --headed
...
Error: The requested information is whether the page title contains the word 'Aluminium'. The page title is 'Mercury element at DuckDuckGo', which does not include 'Aluminium'. Therefore, the statement is false.
...
  1 failed

Our test failed as we expected and provided a meaningful explanation of what went wrong.

Let’s fix the first check and add another one to fail again:

from alumnium import Alumni
from playwright.sync_api import Page
from pytest import fixture


@fixture
def al(page: Page):
    al = Alumni(page)
    yield al
    al.quit()


def test_search(al: Alumni, page: Page):
    page.goto("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Aluminium word")
    al.check("page title contains Mercury word")
    al.check("search results do not contain Wikipedia articles")

Time to re-run the test:

$ pytest --quiet --headed
F
...
E       AssertionError: The search results do contain links to Wikipedia articles, specifically mentioning 'Mercury (element) - Wikipedia' and providing a link to the Wikipedia page for Mercury. Therefore, the statement is false.
...
1 failed in 37.28s

import { test, expect } from ".";

test("should search", async ({ page, al }) => {
  await page.goto("https://duckduckgo.com");
  await al.do("search for 'Mercury element' and press Enter");
  await al.check("page title contains Aluminium word");
  await al.check("page title contains Mercury word");
  await al.check("search results do not contain Wikipedia articles");
});

Time to re-run the test:

$ npx playwright test --project chromium --headed
...
Error: The search results do contain Wikipedia articles, as indicated by the presence of links to Wikipedia in the accessibility tree. For example, there is a link with the text 'Mercury (element) - Wikipedia' and another link labeled 'More at Wikipedia'. Therefore, the statement is false.
...
  1 failed

Ok, the test failed as we wanted it to, let’s fix it:

from alumnium import Alumni
from playwright.sync_api import Page
from pytest import fixture


@fixture
def al(page: Page):
    al = Alumni(page)
    yield al
    al.quit()


def test_search(al: Alumni, page: Page):
    page.goto("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Mercury word")
    al.check("search results do not contain Wikipedia articles")
    al.check("search results contain Wikipedia articles")

Now, re-run to make sure it passes:

$ pytest --quiet --headed
...
1 passed in 38.24s

import { test, expect } from ".";

test("should search", async ({ page, al }) => {
  await page.goto("https://duckduckgo.com");
  await al.do("search for 'Mercury element' and press Enter");
  await al.check("page title contains Mercury word");
  await al.check("search results do not contain Wikipedia articles");
  await al.check("search results contain Wikipedia articles");
});

Now, re-run to make sure it passes:

$ npx playwright test --project chromium --headed
...
  1 passed (14.7s)

Add Data Retrieval

Finally, let’s verify some data that Alumnium can get from the page. We are going to ensure the Mercury element card shows its symbol correctly.

Let’s add a failing verification first:

from alumnium import Alumni
from playwright.sync_api import Page
from pytest import fixture


@fixture
def al(page: Page):
    al = Alumni(page)
    yield al
    al.quit()


def test_search(al: Alumni, page: Page):
    page.goto("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Mercury word")
    al.check("search results contain Wikipedia articles")
    symbol = al.get("chemical symbol")
    assert symbol == "Al"

Time to re-run the test:

$ pytest --quiet --headed
...
E       AssertionError: assert 'Hg' == 'Al'
E
E         - Al
E         + Hg
...
1 failed in 49.14s

import { test, expect } from ".";

test("should search", async ({ page, al }) => {
  await page.goto("https://duckduckgo.com");
  await al.do("search for 'Mercury element' and press Enter");
  await al.check("page title contains Mercury word");
  await al.check("search results contain Wikipedia articles");
  const symbol = await al.get("chemical symbol");
  expect(symbol).toBe("Al");
});

Time to re-run the test:

$ npx playwright test --project chromium --headed
...
    Expected: "Al"
    Received: "Hg"
...
  1 failed

The test failed as expected, let’s fix it and re-run to make sure it’s passing:

from alumnium import Alumni
from playwright.sync_api import Page
from pytest import fixture


@fixture
def al(page: Page):
    al = Alumni(page)
    yield al
    al.quit()


def test_search(al: Alumni, page: Page):
    page.goto("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Mercury word")
    al.check("search results contain Wikipedia articles")
    symbol = al.get("chemical symbol")
    assert symbol == "Al"
    assert symbol == "Hg"

$ pytest --quiet --headed
...
1 passed in 64.50s (0:01:04)

import { test, expect } from ".";

test("should search", async ({ page, al }) => {
  await page.goto("https://duckduckgo.com");
  await al.do("search for 'Mercury element' and press Enter");
  await al.check("page title contains Mercury word");
  await al.check("search results contain Wikipedia articles");
  const symbol = await al.get("chemical symbol");
  expect(symbol).toBe("Al");
  expect(symbol).toBe("Hg");
});

$ npx playwright test --project chromium --headed
...
  1 passed (19.5s)

Congratulations, we have completed our first test!