Solana’s SBF Toolchain Is Becoming Independent: Reproducible Builds and CI Risks

For as long as most Solana developers can remember, installing the validator’s command-line tools also installed the program compiler. From Agave 4.5 it doesn’t. After the upgrade, cargo build-sbf either stops working or quietly runs whatever older copy is left on the machine.

Agave #15415, merged at the start of 24 October, removes cargo-build-sbf and cargo-test-sbf from the release. We tested what that means in practice: which copy cargo runs when there are two, how far the default compiler has moved between releases of the tool, and whether the same program still builds to the same bytes.

What changed

Until now, agave-install and the release.anza.xyz install script put both tools into ~/.local/share/solana/install/active_release/bin, next to solana and the validator. The Agave 4.2.1 release we have installed carries cargo-build-sbf 4.1.0. The commit deletes that step from the install script, drops the stable-branch pin of the tool’s version, and adds a breaking-change note to the 4.5.0 changelog, which says the commands “will fail with no such command, or silently use an older copy found elsewhere on your PATH”.

The tools now ship on their own, from the cargo-build-sbf repository and crates.io, on their own schedule: 4.0.0 in February, then 4.1.0 in May, 4.2.0 and 4.3.0 in August and September, 4.4.0 on 22 September and 4.5.0 on 23 October, the day before the merge. One cargo install cargo-build-sbf --force --locked installs both commands.

Which copy cargo runs

cargo build-sbf is not built into cargo: cargo looks for an executable called cargo-build-sbf and runs it. During the transition many machines have two, the bundled one in the Solana directory and a fresh one in ~/.cargo/bin. We put a stand-in script in each and asked cargo 1.99.0 to run the command:

When ~/.cargo/bin is on PATH, PATH order decides, so an older bundled copy that comes first keeps running after you install a new one. When ~/.cargo/bin is not on PATH, cargo searches it first anyway, while the shell’s which cargo-build-sbf names the Solana copy: the check the changelog suggests gives the wrong answer in exactly that case. cargo build-sbf --version goes through cargo’s own lookup, so it shows the copy that will build your program.

Version 4.5.0 of the tool adds a warning when it finds itself running from the Solana CLI’s directory, telling you to put ~/.cargo/bin first or remove the bundled file. Versions 4.1.0 to 4.4.0 have no such check, and 4.1.0 is the copy bundled with Agave 4.2.1.

The default compiler moves with the tool

cargo-build-sbf downloads a platform-tools bundle, Anza’s Rust and LLVM for SBF, and each release has a default version compiled into it. v1.54 and v1.56 ship LLVM 20, v1.57 LLVM 22:

A machine still on the bundled 4.1.0 and a CI runner that ran cargo install cargo-build-sbf this week compile the same source with compilers six Rust releases and two LLVM releases apart. The next jump is already out: platform-tools v1.58, released on 11 October, lists code-generation changes in its notes, and no release of the tool defaults to it yet. When one does, an unpinned install changes compilers without anyone changing a line.

Same source, same toolchain, same bytes; new toolchain, new bytes

We built one small Pinocchio 0.11.2 program with the bundled cargo-build-sbf 4.1.0, its lockfile fixed, in several ways:

With the toolchain fixed, the build was fully reproducible on this machine: the project path and the registry location did not leak into the binary. Changing only the platform-tools version produced a different program. That matters wherever someone rebuilds a program to check it against what is deployed, as verified-build checks do: the rebuild has to use the same toolchain version, and now that the tool is installed and updated separately, that version is something you have to record yourself.

What to change in CI

# install the tool on its own, at an exact version
cargo install cargo-build-sbf --version 4.5.0 --locked --force

# pin the compiler too, instead of trusting the tool's default
cargo build-sbf --tools-version v1.57

# log what actually ran (this goes through cargo's own lookup)
cargo build-sbf --version

On a clean runner, a workflow that installs a 4.5 Agave release and then calls cargo build-sbf fails with “no such command”, which at least is loud. The quiet failures are the ones above: an image that keeps an old bundled copy ahead of ~/.cargo/bin on PATH, or an unpinned install that picks up a new default compiler. Caching ~/.cache/solana, where the platform-tools bundles live, keeps runners from downloading about 1.3 GB per version.

What this does not show

Agave 4.5 was not released when this change merged, so we did not install it; we read the change and the 4.5.0 tool’s source. The builds used the two platform-tools versions already on our machine, one program, and one Apple Silicon Mac: we did not compare builds across operating systems or processors, which is where reproducibility usually breaks next. The PATH test used stand-in scripts, not the real binaries, to show cargo’s lookup on its own.

SBF Toolchain: Questions People Actually Ask

The one-line version

From Agave 4.5 the program compiler is installed on its own, so pin both cargo-build-sbf and its platform-tools version, check what runs with cargo build-sbf --version rather than which, and expect a different binary whenever the toolchain changes.

Sources: Agave #15415, merged as 3e5220f0; cargo-build-sbf 4.1.0 to 4.5.0 from crates.io; platform-tools release notes for v1.54 to v1.58. Builds and the PATH test are our own.