I maintain CI Bench, an open benchmark of GitHub Actions runners. I'm also the founder of Vitko, which sells one of the products measured (Vitko Runners, private beta). This post is about cost per job.
A runner's list price per minute is not fully correlated with cost per job. Runners take different amounts of time and round differently. For example, GitHub rounds every job up to a whole minute, while Vitko Runners bills per second. So we measured the same twelve real open-source CI jobs on nine configurations and priced each run.
Cost per job against GitHub-hosted 2 vCPU, at 100,000 jobs a month:
- RunsOn 88% lower (self-hosted in your own AWS account, so excluding setup and operating work)
- Vitko Runners 84% lower
- Ubicloud 83% lower
- Blacksmith 52% lower.
- A GitHub-hosted 4 vCPU runner cost 39% more per job and ran 1.6x as fast.
- Self-hosted options spread fixed monthly costs over the volume so their figure moves with it, the calculator shows that.
Links to the benchmark results:
- Full cost per job by workload, with intervals.
- The full ranking.
- List prices with source and read date for twelve providers.
- Method and every raw record.
Corrections from any provider measured are welcome, please submit here.
