Solana’s One-Slot Deployment Edge Case at Epoch Boundaries

A Solana program is checked twice: once when it is deployed, and again each time a validator loads it to run. Almost always both checks apply the same rules. In one slot per epoch, the last, they may not: a program deployed there first runs in the next epoch, and the next epoch can switch on a feature that changes what a valid program looks like.

Agave keeps a second runtime environment for that one slot. SIMD-0688 drops it, and Agave #15280, merged on 27 October, puts that behind a feature gate. We verified real programs, built for SBPFv0 and SBPFv3, against Agave’s deployment and loading rules for two epochs, using the same VM library Agave pins, to see what changes in that slot and what does not.

Why there is a second environment

A program runtime environment is the list of syscalls a program may call plus the VM and verifier settings, such as which SBPF versions load. It is derived from the epoch’s feature set, so it can change only at an epoch boundary. A program deployed in slot S becomes callable in slot S + 1. When S is the last slot of an epoch, the program’s first run belongs to the next epoch.

For that slot only, Agave verifies deployments against the next epoch’s environment. Getting that environment right is its own problem: Agave prepares the next environment ahead of the boundary, and the comment above the last-slot override says that preparation “could have latched on the incorrect environment”, depending on which fork reached the window first.

What makes the second environment unnecessary is Agave #14314, from August. A deployment now stores an unloaded cache entry, so the first time the program runs, the cache loads and verifies it again under the environment of that slot. A program that no longer verifies becomes a failed entry, and every call to it fails with “Program is not deployed”. No unverified code runs either way, which is the SIMD’s security argument.

Two checks with different strictness

The two checks are not the same check. Deployment uses a stricter copy of the environment: it rejects malformed ELFs, drops the deprecated sol_alloc_free_, and can be limited to SBPFv3. Loading for execution is lenient, so programs deployed under older rules keep working.

The difference that matters here is syscall names. An SBPFv0 program, the default output of cargo-build-sbf, links each syscall by name. The ELF loader in solana-sbpf 0.26.0 rejects an unknown name only when broken ELFs are rejected, which is at deployment. At execution the name is accepted and the call fails when the program reaches it.

Four programs, two epochs

We built two small Pinocchio programs with cargo-build-sbf and platform tools v1.57, each for SBPFv0 and SBPFv3: one that only logs, and one that calls sol_big_mod_exp, a syscall behind its own feature. Then we ran Agave’s two checks on each ELF with solana-sbpf 0.26.0, the version Agave pins, using its configuration and its syscall names for two epochs:

Each change is one real Agave feature, disable_sbpf_v0_execution and enable_big_mod_exp_syscall, which we put at one boundary to show both cases at once. The first is the case Agave’s own test for this PR uses.

program               today (rules of N+1)              SIMD-0688 (rules of N)
logs, SBPFv0          rejected: UnsupportedSBPFVersion  accepted
logs, SBPFv3          accepted                          accepted
big_mod_exp, SBPFv0   rejected: UnsupportedSBPFVersion  rejected: UnresolvedSymbol
big_mod_exp, SBPFv3   accepted                          accepted

program               in epoch N                    in epoch N+1
logs, SBPFv0          returns 0                     load fails: UnsupportedSBPFVersion
logs, SBPFv3          returns 0                     returns 0
big_mod_exp, SBPFv0   loads, fails at the call      load fails: UnsupportedSBPFVersion
big_mod_exp, SBPFv3   loads, fails at the call      returns 0

Only the first row changes outcome. Today the SBPFv0 program is rejected in the last slot, and the deployer finds out at once. Under SIMD-0688 it is accepted, the rent and fees are paid, and one slot later, in the new epoch, it fails to load: every call returns “Program is not deployed” until it is upgraded with a build the new epoch accepts. Agave’s test asserts both outcomes, and that in the new epoch the program fails either way.

The case the SIMD describes, and the one it misses

The SIMD’s impact section names a second effect: a program that uses a newly activated feature must wait for the activation before it can deploy. Our third row bears that out for SBPFv0: the sol_big_mod_exp build is rejected under the current epoch’s rules with UnresolvedSymbol.

It does not hold for SBPFv3. A v3 program calls syscalls by number, and the verifier Agave runs on chain, RequisiteVerifier, does not look those numbers up; the check that does sits in LocalVerifier, which solana-sbpf describes as the verifier for a local solana deploy. Firedancer’s validator does not look the number up either. So the v3 build deploys in any slot under either rule, before its syscall exists, and fails only when it reaches the call. In our run that took 70 instructions; after the activation the same program finished in 84.

The Solana CLI closes part of that gap. solana program deploy runs both verifiers locally against the cluster’s current features before it sends anything, unless it is told to skip feature verification, in which case it assumes every feature is on. Tools that build deploy transactions some other way get only the on-chain check.

How often the slot matters

The environment changes only when a feature that shapes it activates. Of the features that still do so in Agave’s code, mainnet’s feature accounts record seven activations, each at its own epoch boundary:

None followed in the more than eighteen months up to SIMD-0688. Features since cleaned out of the code are not in this list, so it is a lower bound. Each boundary affected one slot, about 0.4 seconds of a two-day epoch, and only deploys whose verdict differs between the two environments. The one change Agave’s test is built around, disabling SBPFv0, cannot be scheduled yet: its feature still carries a placeholder key.

There is also a way to add a capability that never moves the boundary. P-256 support adds a curve to the existing curve syscalls and checks its feature when the call runs, so the environment, and with it the deployment check, stays the same before and after.

What deployers can do

After SIMD-0688 activates, a deploy that succeeds is a statement about the current epoch only. If a deploy, upgrade or ExtendProgram lands in the last slot of an epoch with a pending runtime feature, call the program once in the next epoch before relying on it. “Program is not deployed” after a successful deploy means the load failed, and an upgrade with a build the new rules accept fixes it. For SBPFv3 builds, a deploy that succeeds says nothing about whether the syscalls it calls are active, in any slot; the CLI’s local check against the cluster’s features is the place that catches it.

What this does not show

We ran Agave’s checks through solana-sbpf directly, with Agave’s configuration and syscall names at f51751cd and stub syscall bodies, not a validator crossing a real boundary; the pair of epochs is constructed. Agave’s own test covers the SBPFv0 case end to end in a bank. Activation dates come from mainnet’s feature accounts and block times. SIMD-0688’s own feature was added to Agave’s master on 27 October and was in no release.

Last-Slot Deployments: Questions People Actually Ask

The one-line version

SIMD-0688 lets a deploy in an epoch’s last slot pass under the current rules and fail to load one slot later, which in our tests changes the outcome only for SBPFv0 builds when the next epoch changes the environment, while SBPFv3 builds already deploy regardless of whether the syscalls they call exist.

Sources: SIMD-0688 at 749dc863; Agave #15280 and #14314, and master at f51751cd; solana-sbpf v0.26.0; Firedancer at 52de5f17; mainnet feature accounts. Programs and harness are our own, built with platform tools v1.57.