We let the bots out

There’s something cozy about a GitHub Actions workflow: push some code, let a few named steps do their thing, and come back to a reassuring row of green ticks.

- name: Enable RGW
  run: ~/actionutils.sh enable_rgw

- name: Exercise RGW
  run: ~/actionutils.sh testrgw

Those steps came from MicroCeph’s old cluster-testing job. RGW is Ceph’s RADOS Gateway, which provides S3-compatible object storage. The tests enable it, exercise it, and move on.

But wait, what the heck does “Exercise RGW” even mean? And what do I have to do to run the same test locally in my dev environment?

Unfolding the test

The workflow calls actionutils.sh, which is a shared Bash script with helpers for installing MicroCeph, bootstrapping clusters, creating storage devices, waiting for services, and checking that everything works.

For example, testrgw checks Ceph and service status, creates an S3 test user if needed, writes a file, creates a bucket, uploads the file with s3cmd, and fetches it with curl to check its contents.

These are all sensible operations, but that’s quite a lot of work hiding behind two lines of YAML.

Some checks live directly in the workflow too. Take this cluster-list check:

if ! sudo microceph cluster list | grep -q $HOSTNAME; then
    echo "Failed: cluster list does not include $HOSTNAME"
    exit 1
fi
if ! sudo microceph cluster list -f json | jq '.[]["name"]' | grep -q $HOSTNAME; then
    echo "Failed: cluster list is improper json"
    exit 1
fi

Before any of these checks can run, the cluster needs storage. The workflow adds OSDs, the Ceph daemons responsible for storing data, and calls wait_for_osds to wait for them to become ready. Here’s part of its polling loop:

res=$(sudo microceph.ceph -s -f json 2>/dev/null | jq -r '.osdmap.num_in_osds // 0')
[[ "$res" =~ ^[0-9]+$ ]] || res=0

if (( res >= expect )); then
    echo "Found >=${expect} OSDs"
    break
else
    printf '.'
    sleep 5
fi

By this point, we’re reading both the workflow and the helper script to understand one scenario. Some assertions live in YAML, others in Bash functions, and the setup depends on earlier workflow steps.

To run this locally, I have to piece those steps together and figure out what the workflow expects to be installed already.

Let the bots out

We wanted to run these tests locally without reconstructing a GitHub Actions job. That’s what prompted the move to Robot Framework. We kept Actions for CI and continued using shell commands where we needed them.

Robot Framework is a keyword-driven test runner. We write the test using named steps such as Enable RGW, and Robot runs the code behind each step.

Remember the two workflow steps from the opening? Here’s their counterpart in the Robot cluster suite:

*** Test Cases ***
Test Enable RGW And Exercise
    [Documentation]    Enables Rados Gateway and verifies S3 upload/download.
    [Tags]    rgw
    Enable RGW
    Exercise RGW

The steps look much like the YAML we started with. Robot also knows which setup to run before them and which teardown to run afterwards. We can select this test by name and get a report of its execution without involving a GitHub job.

What changed?

BeforeNow
Scenario spread across workflow steps and Bash helpersNamed test cases in .robot suites
Installation and preparation wired into each jobEach suite calls shared helpers to prepare its environment
Assertions in shell snippets and helper functionsTests and shared helpers check the results, with some shell checks still underneath
Local reproduction meant reconstructing the workflowLocal and CI runs use the same tox entry point
GitHub step logs as the main execution trailRobot keyword logs and reports, uploaded by CI

When adding a scenario now, we can put it in a .robot suite and reuse the setup and execution helpers imported by that suite. We don’t need to add its individual steps to the GitHub workflow.

Following a keyword into the harness

We still have to open the harness to see exactly how Exercise RGW works. The difference is that we can now run its suite directly, with the setup and teardown included.

The harness combines Robot resource files with Python libraries. Its keywords use shared helpers to execute commands in the test environment. For example, Robot exposes the Python method run_in_vm_and_check as:

Run In VM And Check    sudo microceph.ceph status    30

That helper executes the command inside an LXD test VM, captures the output, and fails on a nonzero exit code. The 30 is the timeout in seconds.

There is also a plain Run In VM, which returns the result without automatically failing on a nonzero exit code. That’s useful when failure is the expected outcome, but it means we need to check how the test handles the returned result rather than assume the command succeeded.

For shared polling logic, the Python harness provides _poll_until. Pure parsers and polling helpers have their own pytest tests, so checking the test machinery doesn’t require booting a cluster.

There are still shell pipelines and sleeps in the tests. Moving shared parsing and polling into Python lets us test that code separately and reuse it across suites.

Somewhere for the tests to live

For the cluster suite, setup launches an LXD VM, copies in the snap and supporting scripts, installs tools, bootstraps MicroCeph, and adds three loop-backed OSDs.

This exercises the installed snap, including its services and interaction with Ceph.

The standard multi-node setup provisions four containers inside the VM:

Host running Robot
└── Outer LXD VM
    └── Nested LXD
        ├── MicroCeph node
        ├── MicroCeph node
        ├── MicroCeph node
        └── MicroCeph node

That gives us a place to exercise joins, node removal, service migration, and networking without reserving four physical machines.

The test cases within a suite generally share this environment to avoid provisioning a new cluster for each case. If we select just one test, we need to check whether it relies on a previous test having enabled a service or created some data.

At teardown, the harness gathers cluster status and service logs, then attempts to remove the environment.

Running it locally and in CI

Locally, the cluster suite runs like this:

tox -e robot -- \
  --snap-path /absolute/path/to/microceph.snap \
  --test-suite cluster-tests

You’ll need tox installed, initialised LXD, internet access from the guests, and a built snap. Tox creates an isolated Python environment with the test dependencies. The repository’s wrapper then passes the selected suite and snap path to Robot.

In CI, GitHub Actions builds or restores a cached snap and distributes suites across a matrix. Those jobs invoke the same tox entry point.

Each run produces report.html for the summary, log.html for the keyword-by-keyword execution trail, and output.xml for machine-readable results. CI uploads those reports even when a suite fails.

Host-only Robot suites also invoke Go tests, shell tests, Python tests, and static analysis. Their results appear in the same report format, which helps when looking through CI failures.

Trying it out

If you’d like to explore the suite without launching any infrastructure, start with a dry run:

tox -e robot -- --test-suite cluster-tests --dryrun

That checks test structure and keyword resolution without executing the integration actions. When you’re ready to launch the test environment, use the earlier command with --snap-path and leave out --dryrun. The Robot test README covers the prerequisites.

Now I can work on a failing test in my dev environment, using the same suite that CI runs, without pushing a workflow change just to try it again.

microcephcephtestingrobot-frameworklxd