Orcraft comes out of nearly two decades of validation work: software, hardware, firmware, drivers, applications and whole systems. The idea behind it is simple. You and your AI agents write the test logic, in plain Python, and nothing else. The Orcraft framework takes care of the rest: the GUI, the configuration, the machines the test runs on, getting the code to them, the logs, the results and the CI run.

The testing operation

This article goes around that loop once, with code from the tutorial suite. Every line of code below is from it.

What you give your AI agent

Two things. The first is what your product already has: its documentation. A README, a user manual, an API specification or the code itself will do, and later its issues and feature requests. The second is one link to the Orcraft instructions, which tell the agent how a suite is laid out.

Then you ask in plain words:

Create a suite for the maze application from its README. 

The agent writes an ordinary suite. To run it, it also gets the list of your lab's machines, a file the controller saves, and the requests stay as plain:

Run TCP Transmit on any available Linux endpoint. Explain why TCP Receive failed on lab-linux-2 in the last run. 

Any agent that can read a folder, write Python and run a command works with it, such as Claude Code, OpenAI Codex or Gemini CLI. The AI Agents pages list the inputs and more example prompts. There is no AI inside Orcraft, and it calls no AI provider.

Every part of this you can do yourself as well: write the suite, configure a run, run it and read the results. A person has the GUI and the command line for it, an agent the code and the command line. Either way the suite is the same plain Python, and the rest of this article opens one up.

A step is a Python class

The smallest unit is a step. This one runs the hostname command and checks the answer:

class StepHostname(OrcraftTestStep): def __init__(self, config: StepHostnameConfig | None = None) -> None: super().__init__(config or StepHostnameConfig()) def _run_step(self) -> OrcraftTestStepResult: self.result = StepHostnameResult() # The step's body runs on the endpoint it was sent to, so the command # answers with that machine's name. completed = subprocess.run(["hostname"], capture_output=True, text=True) hostname = completed.stdout.strip() self.checkpoint( completed.returncode == 0 and bool(hostname), f"hostname answered: {hostname}", f"hostname failed with exit code {completed.returncode}: {completed.stderr.strip()}", ) Logger.log(f"Hello from {hostname}", color=Logger.Color.BLUE) self.result.hostname = hostname return self.result 

Nothing in it says where it runs. A flow sends the step out, and a test owns the flow. The whole structure is suites, groups, tests, flows and steps, and it is the same in every suite.

The GUI comes from the code

The test that uses this step declares what can be configured:

class TestHelloEndpointConfig(OrcraftTestConfig): name = OrcraftConfigField.Text(default="Hello Endpoint") test_endpoint = OrcraftConfigField.Endpoint(default="localhost", label="Test Endpoint") 

That one Endpoint field is all it takes. The controller reads the class and builds the run panel from it, with a Test Endpoint field that lists the machines of the lab, Windows, Linux and macOS alike. There is no configuration file to write and no plugin to register.

The Hello Endpoint run panel with Test Endpoint set to a Windows machine

With the default, localhost, the step runs on your own computer. Pick another machine and press Run: the controller sends the step there, and the log comes back with that machine's name.

The controller after the run, with a Pass status in the tree and the line Hello from GMG-TEST-1 in the log

The test code was never installed on that machine. The controller uploads the test package to every endpoint a run uses, before the run starts. An endpoint machine needs only the Orcraft endpoint application, installed once; your suites never have to be copied, cloned or installed there.

Every machine has its role

Orcraft works with a lab both ways: with machines that all do the same job, and with a setup where every machine has its own role. Roles first. Where something produces and something consumes, a server and a client, a host and the device attached to it, a test declares one Endpoint field per role. This is the tutorial's Multiple Endpoints and Files Transfer test, with a generator and a solver:

endpoint_1 = OrcraftConfigField.Endpoint(default="localhost", label="Endpoint 1") endpoint_2 = OrcraftConfigField.Endpoint(default="localhost", label="Endpoint 2") 

The test fixes the roles. Which machine plays which role you choose in the run panel, for each run, and the code stays as it is: today the Windows machine generates and the Mac solves, tomorrow the other way round.

Inside the flow, each step names the role it belongs to, and the framework sends it to whichever machine holds that role. Moving a file from one role to the other is a single call:

transfer_result = self.transfer( src_endpoint=self.config.endpoint_1, dst_endpoint=self.config.endpoint_2, paths=step_2_result.maze_file_path, ) 

The step after it runs on the second role, with the file that arrived:

step_3_result = self.run_step( StepLoadSolve, endpoint=self.config.endpoint_2, maze_executable_folder=self.config.maze_executable_folder, maze_file=transfer_result.dst_paths[0], ) 

The results record the split, step by step:

The Details view of the test, with the generate steps marked GMG-TEST-1 and the solve steps marked GMG-TEST-5

The same test on every machine

In the other mode the machines all do the same job, and one test runs on each of them. For that, declare the field as an EndpointFanout in place of an Endpoint. This line is from the tutorial's Endpoint Fanout test:

test_endpoint = OrcraftConfigField.EndpointFanout(default=("localhost",), label="Test Endpoint") 

The same test also declares a matrix of cases:

matrix = OrcraftConfigField.Matrix( OrcraftConfigField.Dimension("width", (5, 10)), OrcraftConfigField.Dimension("height", (8, 12)), ) 

Two widths and two heights make four cases. In the run panel the Test Endpoint field now takes several machines. Here it holds three, so the counter reads twelve cases:

The Endpoint Fanout run panel with three machines in the Test Endpoint field and Total Cases 12

Press Run, and the test starts once per machine, all three at the same time. Each machine gets its own run row, its own log and its own results.

The execution tree with three passing runs of the same test, one per machine

From the GUI to CI

A run that works in the GUI is exported as a snapshot, and the snapshot runs unattended from the command line or a CI pipeline, with the same configuration and the same results. Every run also writes a history file beside its logs, which the controller opens, so a run made in CI can be looked at in the same tree as a run made by hand. The tutorial page Exporting a test batch for CI shows the whole path.

What else it does

The walkthrough shows one path through the controller. The rest is the framework's side of the work too, with no code of yours behind it:

  • Lambda values. A parameter or a matrix dimension takes its choices from a function that runs when the test is configured, such as the releases published at that moment.
  • Randomization. Matrix dimensions draw their values at random while the number of cases stays fixed, and are redrawn from the controller.
  • Endpoint pools. A test takes any free machine that matches a label, such as an operating system or attached hardware.
  • Queues. A run that needs a busy machine waits for it, and starts by itself when the machine is free.
  • Endpoint updates. The controller updates the endpoint software on a machine whenever its version differs from its own.
  • Step-level results. Live logs and results per machine, down to the single step, with export to a spreadsheet.
  • Sessions. A session holds the suites, the plan and the runs. Save it to a file, reopen it later, share it, and run it again with the same plan.
  • Rerun. Each test run keeps the exact configuration it ran with, and reruns with it at a click.
  • Secure connection. Optional mutual TLS between the controller and the endpoints, with token-based access control.
  • Shared endpoints. On the Team Plan, endpoint machines are shared by every team member, CI runner and AI agent.

Try it

Install with:

pip install orcraft 

Then start the controller:

orcraft-controller run 

It needs Python 3.14 or newer, on Windows, Linux or macOS. Running tests on your own computer needs no account.

To run tests on another machine of your lab, install Orcraft on it the same way and start the endpoint application there:

orcraft-endpoint run 

Then add the machine in the controller, sign in and activate its license. The Free Plan covers two remote machines with no card and no time limit. Adding endpoints shows each step, and the tutorial goes through the suite used here, test by test.

The 16-minute demo shows all of it in motion:

Orcraft is a commercial, closed-source product; the plans and prices are on the pricing page.

We would like to hear from people who run test or validation labs.