Enterprise Software Development System

›relman CLI: VirtualBox Click-Tests

relman CLI: VirtualBox Click-Tests

← Back to the relman CLI command overview

VirtualBox click-tests

Unlike almost every other command group, the click-test object tree (virtual machine, test, testcase, teststep, testrun) uses flat, globally-unique tokens instead of an organization/project-group/project scope chain: a test suite is really owned by a build, not by a project directly, and a testcase or step has no single natural ancestor chain a caller would already be holding the way they hold an org/pg/project chain before looking up a task. So every token below is self-sufficient – no ancestor tokens needed once you have it.

Full path from nothing to a passing test: list vm → list build + list snapshot → add test → add testcase → add teststep + edit teststep (repeat) → run test → describe testcase / describe teststep to check progress → download screenshot.

Discovery

list vm (alias: vms)

Fetches every virtual machine’s id, name, running state and snapshot count. Takes no arguments – lists every VM the server knows about.

list build <org-token> <pg-token> <project-token> (alias: builds)

Fetches the id and a label of every build belonging to the project identified by <project-token>. A build has no name field of its own, so the label is its configured goals’ mojos joined with a space (e.g. “clean install”). Needed to pass a build token to add test – this is the one command in this group that is org/pg/project-scoped, since a build sits directly under a project.

list snapshot <vm-token> (alias: snapshots)

Fetches every snapshot of the virtual machine identified by <vm-token> – id, name and description. Needed to pass a snapshot token to add test.

list test <vm-token> (alias: tests)

Fetches every existing test suite whose snapshot belongs to the virtual machine identified by <vm-token>, newest first – id, name and snapshot name. Needed to pass a test token to add testcase (or to reach a test’s testcases via list testcase).

Building a test

add test <org-token> <pg-token> <project-token> <build-token> <snapshot-token> <name>

Creates a new test suite named <name>, bound to the build identified by <build-token> and the snapshot identified by <snapshot-token> (any virtual machine’s – not scoped to the build’s project). Prints the new test’s id, needed by add testcase to attach testcases to it.

list testcase <vm-token> (alias: testcases)

Fetches every testcase belonging to any test suite of the virtual machine identified by <vm-token>, newest test first – id, name, owning test’s id/name and step count.

describe testcase <testcase-token>

Fetches and prints the testcase’s name, description, owning test id/name, and every step (id and a short human-readable action summary, e.g. “type: hello”, “mouse click button 1”).

add testcase <test-token> <name>

Creates a new, empty testcase (description left blank) named <name> under the test identified by <test-token>. Prints the new testcase’s id, needed by add teststep.

list teststep <testcase-token> (alias: teststeps)

Fetches every step of the testcase identified by <testcase-token>, in execution order – id and a short human-readable action summary each.

describe teststep <teststep-token>

Fetches and prints every raw property of the step (mouse move/click, key down/up, double-click interval, literal text to type, CPU-usage wait, whether it has a required screenshot). edit teststep‘s field names match this command’s output field names exactly.

add teststep <testcase-token>

Creates a new, entirely empty step (every action property null) under the testcase identified by <testcase-token>. Prints the new step’s id – follow up with edit teststep to give it an actual action.

edit teststep <teststep-token> <field> <value>

Sets one of the step’s ten properties to <value>, or clears it if <value> is the literal string null. <field> is one of: mousemovex, mousemovey, mousedownbutton, mouseupbutton, keydowncode, keyupcode, doubleclickms, stringwrite, waitcputimeoutus, waitcpupercent. Prints the step’s full detail afterwards, the same shape describe teststep returns.

Attaching a reference screenshot for a “compare screenshot” step is not one of the ten fields – that needs a freshly created screenshot object to point at, which is out of scope for this command; use the web UI for that step.

Running & results

run test <testcase-token>

Queues a new run for the whole test suite that owns the given testcase (every testcase belonging to it, not just the one referenced) – the same as the web UI’s click-test “play” button. Prints the new run’s id and the test id it was queued for. The run executes asynchronously on whichever runner instance is watching the target virtual machine; re-fetch list testcase/describe testcase to see it progress.

download screenshot <teststep-token> <output-file>

Saves the reference screenshot PNG attached to the step identified by <teststep-token> to <output-file>, overwriting it if it already exists. Fails with an error (exit code 1) if the step has no screenshot attached yet.

Examples

1. Listing virtual machines

Global flags omitted below – see Utility & global flags. No org/project chain needed – the whole click-test tree uses flat tokens.

$ relman list vm
- id: vm-Fq82Nx
  name: win11-chrome
  running: true
  snapshotCount: 3
- id: vm-Hn73Kp
  name: ubuntu22-firefox
  running: false
  snapshotCount: 1

Edge case: running: false doesn’t mean the machine is broken – it just means the runner watching it hasn’t reported it as up right now; run test against a testcase on it still queues successfully, the run simply waits.

2. Building a step, then trying an invalid edit

$ relman add teststep vc-Jm62Tx
id: vs-Rq93Ln

$ relman edit teststep vs-Rq93Ln stringwrite "hello world"
testCaseId: vc-Jm62Tx
mouseMoveX: null
mouseMoveY: null
mouseDownButton: null
mouseUpButton: null
keyDownCode: null
keyUpCode: null
doubleClickHavingPeroidInMs: null
stringWrite: hello world
waitCpuUsageTimeoutMicroseconds: null
waitCupUsagePercent: null
hasRequiredScreenshot: false

$ relman edit teststep vs-Rq93Ln mousemovex forty
Error: The server responded with HTTP 400 for https://.../cli/teststep-edit?...

Edge case: the numeric fields (mousemovex, keydowncode, etc.) are strictly parsed server-side – a non-numeric value like forty is rejected with a bare HTTP 400 rather than being coerced or ignored. Note also that setting stringwrite did not clear it back to null automatically when unrelated – each field is independent, but a step is normally meant to carry only one action at a time by convention, not by enforcement.

3. Running a test – asynchronously

$ relman run test vc-Jm62Tx
runId: vr-Wp84Zc
testId: vt-Nx72Bq

$ relman describe testcase vc-Jm62Tx
name: Login with valid credentials
description: ""
testId: vt-Nx72Bq
testName: nightly-smoke
steps:
  - id: vs-Rq93Ln
    action: "type: hello world"

Edge case: run test returns immediately once the run is queued, not once it’s finished – there is no pass/fail field in its own response. The run executes asynchronously on whichever runner instance is watching the target VM; re-fetching describe testcase a bit later is currently the only way from the CLI to see whether it passed, and even then only indirectly, by checking back with the web UI or the VM’s activity – this command’s own output does not surface run results directly.

Top