Solana Validator Memory: Transparent Huge Pages Without a Reserved Pool

An Agave validator reads its account storage files through io_uring into large buffers, 10 MiB each for account storage, and copies accounts out of them, for example while it builds its accounts index at startup. Until 22 October those buffers almost always sat in ordinary 4 KiB pages. The code asked for huge pages in a way that only works on hosts few operators set up.

Agave #16137, merged late on 22 October, switches to transparent huge pages. We reproduced its allocation in a C program on Linux, confirmed the kernel backs the buffers with 2 MiB pages, and measured what that changes for copying data out of them. The mechanism works as described. How much it speeds things up depends heavily on the machine.

Two ways to get huge pages

Every memory access goes through a virtual-to-physical translation cached in the TLB. With 4 KiB pages a 10 MiB buffer is 2,560 translations; with 2 MiB pages it is five. Code that sweeps through big buffers misses the TLB far less often with huge pages. Linux offers them two ways:

hugetlbfs hands out pages from a pool the administrator reserves in advance with vm.nr_hugepages; with no pool, the mmap fails. Transparent huge pages come from ordinary memory: in the kernel’s madvise mode, any 2 MiB-aligned range that asks with MADV_HUGEPAGE gets them when free huge pages are available. Agave’s old code tried MAP_HUGETLB first, rounding the buffer up to a power of two (a 10 MiB buffer became 16 MiB), and on the usual host without a pool fell back to the normal allocator without logging anything; in THP’s madvise mode, that means base pages.

What the change does

For buffers of 2 MiB or more, the new code maps one huge page more than it needs, starts the buffer at the first 2 MiB boundary inside the mapping and advises the kernel:

let map_len = aligned_size + HUGE_PAGE_SIZE;           // over-map by 2 MiB
let map = mmap(null, map_len, READ | WRITE, PRIVATE | ANONYMOUS, -1, 0);
let head = map.next_multiple_of(HUGE_PAGE_SIZE) - map;  // skip to a 2 MiB boundary
let start = map + head;
if madvise(start, aligned_size, MADV_HUGEPAGE) != 0 {
    // warn once: "io_uring buffers use base pages ... the kernel likely
    // lacks transparent huge page support (CONFIG_TRANSPARENT_HUGEPAGE)"
}
// the unused head and tail stay mapped until drop: untouched, they take no memory

Later commits in the same pull request settled two details. The buffer keeps its requested length instead of being rounded to a power of two. And the code no longer unmaps the spare head and tail right away: splitting a mapping can fail with ENOMEM when the kernel’s limit on mappings is reached, and unmapping the whole range after a partial failure could race with another thread mapping into it. Smaller buffers still use the normal allocator.

The pull request reports index generation in agave-ledger-tool verify taking 30.55 s in both runs against 32.07 and 31.60 s before, about 4% faster, and the whole run 132.0 and 131.5 s against 133.5 and 133.5 s. In its profiles, copying out of the read buffers was 60% cheaper and reading account headers 30% cheaper, with page faults unchanged. It does not name the hardware.

What we measured

We had no validator host, so we ran Linux 7.0 in a virtual machine on an Apple M1 Pro, with 4 KiB pages. A C program allocates 10 MiB buffers exactly as above, marked MADV_HUGEPAGE or, for comparison, MADV_NOHUGEPAGE, and copies data out of them. Medians of seven interleaved runs:

The kernel backed every advised buffer entirely with huge pages, the same 10,240 kB the pull request reports. Faulting a buffer in got seven times faster, but Agave keeps its buffers for the life of the reader, so that cost is paid once, consistent with the unchanged page faults in the pull request. Sequential copies got 15% faster. Random account-sized copies and header reads gained nothing within one 10 MiB buffer and 4 to 7% once eight or thirty-two buffers (80 to 320 MiB) were in play: the gain appears when the working set outgrows what the TLB can cover with base pages.

That is far below the 60% in the pull request’s profiles. TLB sizes differ between processors, and a virtual machine adds a second level of address translation, so neither number transfers to another machine. What does transfer is the mechanism: on any Linux host where transparent huge pages are on, these buffers now get them.

What an operator should check

# must show [always] or [madvise], not [never]
cat /sys/kernel/mm/transparent_hugepage/enabled

# with the validator running: non-zero means huge pages are in use
grep AnonHugePages /proc/$(pgrep -f agave-validator | head -1)/smaps_rollup

A kernel built without transparent huge page support now produces a warning in the log once, instead of a silent fallback. And a host that reserved a hugetlbfs pool for these buffers no longer uses it for them: pages in the pool stay reserved and cannot serve ordinary allocations, so a pool kept only for Agave’s io_uring buffers is now memory that nothing uses.

What this does not show

We did not run a validator. Our numbers come from a synthetic copy workload in a Linux virtual machine on Apple Silicon, not from validator hardware, and our random access patterns are not the order in which index generation reads accounts. The pull request’s numbers are its author’s. Agave’s separate work on jemalloc arenas for its replay threads is outside this change.

Huge Pages: Questions People Actually Ask

The one-line version

Agave’s io_uring read buffers now ask for transparent huge pages instead of a hugetlbfs pool most hosts never reserved, so they get 2 MiB pages wherever THP is on: we confirmed the pages and measured modest gains on our machine, the pull request reports larger ones on its own.

Sources: Agave #16137, merged as c4793187, and its description for the index-generation and profile numbers; the Linux kernel documentation on transparent huge pages and hugetlbfs. Our benchmark is our own.