Writing First Test with Appium

Let’s see how we can write a simple test that would use Appium to open DuckDuckGo, search for a Mercury element, ensure that search results are loaded, include the Wikipedia article about the element, and check that its symbol is presented correctly on the page.

Note

Alumnium can be used with both iOS and Android platforms, web and native applications, as well as any test runner you like. For the examples in this guide, we’ll use iOS with Safari and Pytest.

Note

Alumnium can be used with both iOS and Android platforms, web and native applications, as well as any test runner you like. For the examples in this guide, we’ll use iOS with Safari and WebdriverIO.

Install Dependencies

Make sure you have Alumnium installed in your project.

Installing dependencies

$ pip install alumnium pytest

Now let’s start Appium server to communicate with the mobile device. You can do it using npm in a terminal:

Installing Appium

$ npx appium driver install xcuitest
$ npx appium

Installing dependencies

$ npm install alumnium

Now initialize WebdriverIO in your project if you haven’t done it yet. Select the options to test mobile applications on iOS platform with Mocha framework. You don’t need to use visual testing and it’s better to remove autogenerated test files.

Initializing WebdriverIO

$ npm init wdio
$ rm test/specs/test.e2e.ts

We’ll also need to install Docker to run Alumnium server. Please, refer to Docker installation guide for your platform.

Setup Browser

Start by creating a test file for your tests and instantiating a browser instance:

test_search.py

from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


def test_search(driver: WebDriver):
    driver.get("https://duckduckgo.com")

Now run the test. You should see an iOS simulator with a browser window with DuckDuckGo opened.

Running test

$ pytest --quiet
...
1 passed in 3.73s

test/specs/search.e2e.ts

import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
  });
});

You might need to adjust your wdio.conf.ts file to include the desired capabilities for iOS Safari:

wdio.conf.ts

export const config: WebdriverIO.Config = {
  // other options
  capabilities: [
    {
      platformName: "iOS",
      browserName: "Safari",
      "appium:automationName": "XCUITest",
      "appium:deviceName": "iPhone 16",
      "appium:platformVersion": "18.5",
    },
  ],
  // other options
};

Now run the test. You should see an iOS simulator with a browser window with DuckDuckGo opened.

Running test

$ npx wdio run wdio.conf.ts

[Safari iOS #0-0] Running: Safari on iOS
[Safari iOS #0-0] Session ID: 29eb39f3-1ca6-4349-bb07-6ea545e8a3e9
[Safari iOS #0-0]
[Safari iOS #0-0] » test/specs/search.e2e.ts
[Safari iOS #0-0] TestSearch
[Safari iOS #0-0]    ✓ should search
[Safari iOS #0-0]
[Safari iOS #0-0] 1 passing (1.6s)

Spec Files:      1 passed, 1 total (100% completed) in 00:00:09

Setup Alumnium

The next thing we need to do is to start the Alumnium server. It is responsible for communication between the browser and the AI model and must be running for the tests to work. You can start it using Docker:

Starting server

$ docker run --rm --detach \
      --publish 8013:8013 \
      --env ALUMNIUM_MODEL \
      --env OPENAI_API_KEY \
      alumnium/alumnium

Now let’s add the code that would instantiate Alumnium using the browser. We can use a test fixture to ensure proper setup and teardown of the Alumnium instance:

test_search.py

from alumnium import Alumni
from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


@fixture
def al(driver: WebDriver):
    al = Alumni(driver)
    yield al
    al.quit()


def test_search(al: Alumni, driver: WebDriver):
    driver.get("https://duckduckgo.com")

The test should still work fine, let’s re-run it to make sure:

Running test

$ pytest --quiet
...
1 passed in 3.70s

Now let’s add the code that would instantiate Alumnium using the browser.

test/specs/search.e2e.ts

import { Alumni } from "alumnium";
import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  let al: Alumni;

  before(async () => {
    al = new Alumni(browser);
  });

  after(async () => {
    await al.quit();
  });

  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
  });
});

The test should still work fine, let’s re-run it to make sure:

Running test

$ npx wdio run wdio.conf.ts
...
Spec Files:      1 passed, 1 total (100% completed) in 00:00:08

Note

If you are getting errors about missing API keys, make sure you configured Alumnium and try again!

Add Actions

Now let’s add some actions that Alumnium should do on the page. Our test needs to search for a Mercury element, so let’s use this exact command:

test_search.py

from alumnium import Alumni
from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


@fixture
def al(driver: WebDriver):
    al = Alumni(driver)
    yield al
    al.quit()


def test_search(al: Alumni, driver: WebDriver):
    driver.get("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")

Run the test, you should now see “Mercury element” typed in the search box and the page with results loaded.

Running test

$ pytest --quiet
...
1 passed in 26.88s

test/specs/search.e2e.ts

import { Alumni } from "alumnium";
import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  let al: Alumni;

  before(async () => {
    al = new Alumni(browser);
  });

  after(async () => {
    await al.quit();
  });

  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
    await al.do("search for 'Mercury element' and press Enter");
  });
});

Run the test, you should now see “Mercury element” typed in the search box and the page with results loaded.

Running test

$ npx wdio run wdio.conf.ts
...
Spec Files:      1 passed, 1 total (100% completed) in 00:00:33

Add Verifications

The next step is to add some verifications that Alumnium should check on the page. We are going to add two of them, one that checks that the page title contains the search keyword and one to see if the Wikipedia article is present in the results.

Common wisdom says to never trust a test you haven’t seen fail. Let’s add the first verification and see it fail!

test_search.py

from alumnium import Alumni
from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


@fixture
def al(driver: WebDriver):
    al = Alumni(driver)
    yield al
    al.quit()


def test_search(al: Alumni, driver: WebDriver):
    driver.get("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Aluminium word")

Now let’s run our test:

Running test

$ pytest --quiet
F
...
E       AssertionError: The requested information is whether the page title contains the word 'Aluminium'. The title of the webpage is 'Mercury element at DuckDuckGo', which does not include the word 'Aluminium'. Therefore, the statement is false.
...
1 failed in 45.21s

test/specs/search.e2e.ts

import { Alumni } from "alumnium";
import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  let al: Alumni;

  before(async () => {
    al = new Alumni(browser);
  });

  after(async () => {
    await al.quit();
  });

  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
    await al.do("search for 'Mercury element' and press Enter");
    await al.check("page title contains Aluminium word");
  });
});

Now let’s run our test:

Running test

$ npx wdio run wdio.conf.ts
...
AssertionError: The requested information is about whether the page title contains the word 'Aluminium'. The page title is 'Mercury element at DuckDuckGo', which does not include the word 'Aluminium'. Therefore, the statement is false.
...
Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:35

Our test failed as we expected and provided a meaningful explanation of what went wrong.

Let’s fix the first check and add another one to fail again:

test_search.py

from alumnium import Alumni
from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


@fixture
def al(driver: WebDriver):
    al = Alumni(driver)
    yield al
    al.quit()


def test_search(al: Alumni, driver: WebDriver):
    driver.get("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Aluminium word")
    al.check("page title contains Mercury word")
    al.check("search results do not contain Wikipedia articles")

Time to re-run the test:

Running test

$ pytest --quiet
F
...
E       AssertionError: The accessibility tree contains multiple links to Wikipedia articles related to the Mercury element, indicating that search results do include Wikipedia articles. Therefore, the statement is false.
...
1 failed in 76.08s (0:01:16)

test/specs/search.e2e.ts

import { Alumni } from "alumnium";
import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  let al: Alumni;

  before(async () => {
    al = new Alumni(browser);
  });

  after(async () => {
    await al.quit();
  });

  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
    await al.do("search for 'Mercury element' and press Enter");
    await al.check("page title contains Aluminium word");
    await al.check("page title contains Mercury word");
    await al.check("search results do not contain Wikipedia articles");
  });
});

Time to re-run the test:

Running test

$ npx wdio run wdio.conf.ts
...
AssertionError: The search results include multiple links to Wikipedia articles, such as 'More at Wikipedia' and 'Mercury (element) - Wikipedia'. Therefore, the statement that search results do not contain Wikipedia articles is false.
...
Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:36

Ok, the test failed as we wanted it to, let’s fix it:

test_search.py

from alumnium import Alumni
from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


@fixture
def al(driver: WebDriver):
    al = Alumni(driver)
    yield al
    al.quit()


def test_search(al: Alumni, driver: WebDriver):
    driver.get("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Mercury word")
    al.check("search results do not contain Wikipedia articles")
    al.check("search results contain Wikipedia articles")

Now, re-run to make sure it passes:

Running test

$ pytest --quiet
...
1 passed in 91.62s (0:01:31)

test/specs/search.e2e.ts

import { Alumni } from "alumnium";
import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  let al: Alumni;

  before(async () => {
    al = new Alumni(browser);
  });

  after(async () => {
    await al.quit();
  });

  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
    await al.do("search for 'Mercury element' and press Enter");
    await al.check("page title contains Mercury word");
    await al.check("search results do not contain Wikipedia articles");
    await al.check("search results contain Wikipedia articles");
  });
});

Now, re-run to make sure it passes:

Running test

$ npx wdio run wdio.conf.ts
...
Spec Files:      1 passed, 1 total (100% completed) in 00:00:37

Add Data Retrieval

Finally, let’s verify some data that Alumnium can get from the page. We are going to ensure the Mercury element card shows its symbol correctly.

Let’s add a failing verification first:

test_search.py

from alumnium import Alumni
from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


@fixture
def al(driver: WebDriver):
    al = Alumni(driver)
    yield al
    al.quit()


def test_search(al: Alumni, driver: WebDriver):
    driver.get("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Mercury word")
    al.check("search results contain Wikipedia articles")
    symbol = al.get("chemical symbol")
    assert symbol == "Al"

Time to re-run the test:

Running test

$ pytest --quiet
...
E       AssertionError: assert 'Hg' == 'Al'
E
E         - Al
E         + Hg
...
1 failed in 68.00s (0:01:07)

test/specs/search.e2e.ts

import { Alumni } from "alumnium";
import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  let al: Alumni;

  before(async () => {
    al = new Alumni(browser);
  });

  after(async () => {
    await al.quit();
  });

  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
    await al.do("search for 'Mercury element' and press Enter");
    await al.check("page title contains Mercury word");
    await al.check("search results contain Wikipedia articles");
    const symbol = await al.get("chemical symbol");
    expect(symbol).toEqual("Al");
  });
});

Time to re-run the test:

Running test

$ npx wdio run wdio.conf.ts
...
Expected: "Al"
Received: "Hg"
...
Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:45

The test failed as expected, let’s fix it and re-run to make sure it’s passing:

test_search.py

from alumnium import Alumni
from appium.options.ios import XCUITestOptions
from appium.webdriver.webdriver import WebDriver
from pytest import fixture


@fixture
def driver():
    options = XCUITestOptions()
    options.automation_name = "XCUITest"
    options.bundle_id = "com.apple.mobilesafari"
    options.device_name = "iPhone 16"
    options.no_reset = True
    options.platform_name = "iOS"
    options.platform_version = "18.5"
    driver = WebDriver(options=options)
    yield driver


@fixture
def al(driver: WebDriver):
    al = Alumni(driver)
    yield al
    al.quit()


def test_search(al: Alumni, driver: WebDriver):
    driver.get("https://duckduckgo.com")
    al.do("search for 'Mercury element' and press Enter")
    al.check("page title contains Mercury word")
    al.check("search results contain Wikipedia articles")
    symbol = al.get("chemical symbol")
    assert symbol == "Al"
    assert symbol == "Hg"

Running test

$ pytest --quiet
...
1 passed in 108.67s (0:01:48)

test/specs/search.e2e.ts

import { Alumni } from "alumnium";
import { browser, expect } from "@wdio/globals";

describe("TestSearch", () => {
  let al: Alumni;

  before(async () => {
    al = new Alumni(browser);
  });

  after(async () => {
    await al.quit();
  });

  it("should search", async () => {
    await browser.url("https://duckduckgo.com");
    await al.do("search for 'Mercury element' and press Enter");
    await al.check("page title contains Mercury word");
    await al.check("search results contain Wikipedia articles");
    const symbol = await al.get("chemical symbol");
    expect(symbol).toEqual("Al");
    expect(symbol).toEqual("Hg");
  });
});

Running test

$ npx wdio run wdio.conf.ts
...
Spec Files:      1 passed, 1 total (100% completed) in 00:00:55

Congratulations, we have completed our first test!