Custom Buildkite agents on fresh VMs, without the Elastic CI Stack

The Buildkite agent is a single Go binary, buildkite-agent, that polls Buildkite for jobs on one queue and runs them on its host. Buildkite gives you two ways to supply those agents: run them yourself, or use Buildkite's hosted agents. A custom Buildkite agent pool makes sense when you need hardware or images the hosted shapes do not offer and you would rather not maintain the Elastic CI Stack or the Kubernetes agent stack.

Status: Buildkite is on the cirunner.dev roadmap. Our early access starts with GitHub Actions and GitLab CI. This page describes how Buildkite support will work.

Buildkite hosted agents: check them first

Buildkite's hosted agents are a strong option, and you should compare against them before anything else. Per Buildkite's documentation at the time of writing:

• Linux agents come in amd64 and arm64 shapes from 2 up to 64 vCPU, plus macOS and Windows VMs.
• Cache volumes on local disks and transparent Git mirroring come included.
• Billing counts vCPU-minutes, measured to the second, without startup and shutdown time.

If those shapes and images cover your builds, use them. They run inside Buildkite's own platform, and you skip a second vendor. The docs we read do not list GPU shapes, and you work within the agent images and configuration Buildkite supports. Teams outside those bounds end up self-hosting.

What a self-hosted Buildkite agent fleet involves

The Buildkite agents guide walks through it. The short version:

• Create a cluster and a self-hosted queue, then a cluster agent token. Buildkite shows the token once.
• Install buildkite-agent, set tags="queue=..." and the token in buildkite-agent.cfg, and add hooks for secrets.
• For one job per machine, start agents with --disconnect-after-job and have your provisioning layer destroy the VM when the process exits.
• For autoscaling, deploy the Elastic CI Stack for AWS into your AWS account, or the Agent Stack for Kubernetes into a cluster.

Then somebody owns AMI updates, agent versions, Spot interruptions, disk cleanup on long-lived hosts and the day the autoscaler stops picking up jobs.

How a custom Buildkite agent from cirunner.dev will work

1. You create a self-hosted queue in your cluster and a cluster agent token, and give us the token and the queue name.
2. You choose the machine in the configurator: more cores and memory, x86 or ARM, a GPU if you need one, and your own base image.
3. When jobs wait on that queue, we boot a fresh VM and start a one-job agent with --disconnect-after-job. Your steps target it with agents: { queue: "your-queue" }.
4. We destroy the VM after the job. A build cache persists between jobs.

You keep Buildkite for pipelines, logs and the UI, and you pay us per compute minute for the machines.

Buildkite agent options compared

Who it is for

Good fit:

• Teams that need GPU queues for CUDA builds or tests next to their CPU queues.
• Teams with custom base images or toolchains (game engines, embedded SDKs) and no appetite for running the Elastic CI Stack.
• Teams without an AWS account or Kubernetes cluster they want to dedicate to CI.

Not a fit:

• Teams whose builds fit Buildkite's hosted shapes and images. Stay there.
• Queues that must reach databases or services inside your private network.

FAQ

Does Buildkite have hosted agents?

Yes. Buildkite offers hosted Linux (amd64 and arm64), macOS and Windows agents with cache volumes and Git mirroring, billed per vCPU-minute. Check Buildkite's docs for current shapes and plans.

Can I run a custom Buildkite agent on my own machine?

Yes. Install buildkite-agent, give it a cluster agent token and a queue tag, and target that queue from your pipeline. The setup guide covers install, config and one-job agents.

How do I autoscale Buildkite agents?

Buildkite maintains the Elastic CI Stack for AWS and the Agent Stack for Kubernetes. Both run in your own account and scale with queue activity. cirunner.dev plans a managed alternative; vote for it on the early-access form.

Can a Buildkite agent listen to several queues?

No. In a cluster, each agent joins one queue. Run separate agents for separate queues.