Caching

Caching system allows to significantly increase performance and stability of tests written with Alumnium. It works by persisting the AI responses in a storage, allowing to skip LLM communications unless UI of the application changes. It works best on the CI system which often re-run tests without any underlying changes.

By default, Alumnium provides in-memory cache that is used during tests execution, but discarded at the end. In order to persist the cache database, you need to add a hook in your test runner that saves the cache every time the test passes and discards otherwise:

Behave

def after_scenario(context, scenario):
    # Assuming `context.al` is an instance of `Alumni`.
    if scenario.status == "passed":
        context.al.cache.save()
    else:
        context.al.cache.discard()

Pytest

from pytest import hookimpl


@hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item):
    outcome = yield
    report = outcome.get_result()
    if report.when == "call":
        # Assuming `al` is a fixture that provides an instance of `Alumni`.
        al = item.funcargs["al"]
        if report.passed:
            al.cache.save()
        else:
            al.cache.discard()

Once the cache is saved, you should see storage files in the root of your project. These files contain all the AI responses that Alumnium has used during the test execution. They can be added to .gitignore or committed to the repository, depending on your needs.

The cache will be updated automatically when the tests or UI of the application change.

Providers

There are two main cache providers at the moment. You can select the provider by setting the ALUMNIUM_CACHE environment variable.

Provider	Description	Storage
filesystem (default)	Stores the cache in the filesystem directory.	.alumnium/cache/
sqlite (deprecated)	Stores the cache in a SQLite database file. Does no work well with parallel test execution.	.alumnium-cache.sqlite

You can also disable the cache completely by setting the provider to none or false.

GitHub Actions

While cache maybe less useful for local test execution, it’s a good idea to use it in CI systems like GitHub Actions. This way, you can avoid unnecessary LLM calls and speed up your test runs. Saving and restoring the cache can be done using the actions/cache action:

- uses: actions/cache@v4
  with:
    path: .alumnium/cache/
    key: alumnium-cache-${{ github.sha }}
    restore-keys: alumnium-cache-

This configuration will keep updating the cache on every commit, falling back to the latest cache in the current or default branch.