When Solana WebSockets Stay Connected but Miss Updates

Your indexer subscribes to a program’s logs at finalized, because it only wants data that can never roll back. The socket stays open. No error arrives, no disconnect, no lag warning. And some transactions that are finalized on chain never reach it.

We measured how often on 5 November 2026, against the public mainnet RPC endpoint, which reported Agave solana-core 4.4.0. Over half an hour, a finalized logsSubscribe stream never looked at about 3% of slots and missed 61 of 1,803 finalized transactions that getSignaturesForAddress returns for the same slots. A processed stream on the same connection missed none.

The cause is a seven-year-old design decision meeting a consensus protocol that has run mainnet for two weeks. It was reported on 3 November, and a fix was proposed the same day. Here is what the stream reads, why it skips, which subscriptions are exposed, what the fix changes for clients, and the reconciliation loop that catches the gap today.

One finalized transaction in thirty never arrived

The setup was three programs and three commitment levels. The Address Lookup Table program and the first Memo program land roughly one transaction every slot or two, so each of their transactions is an independent test of whether a stream looked at its slot. The second Memo program lands a transaction in almost every slot, so the slots a stream reported for it are the slots that stream scanned. Ground truth was getSignaturesForAddress at finalized, polled over the same window.

All nine subscriptions shared one WebSocket connection, so one node served all three commitment levels. The run covered 05:36 to 06:07 UTC, slots 465,589,591 to 465,598,091. The endpoint dropped our connection about once a minute under the high-volume Memo stream (close code 1006, no close frame), so each level is judged only on slots strictly inside a stretch where its connection was live.

A second run, with one connection per commitment level and so most likely three different nodes behind the load balancer, agreed. Finalized never scanned 93 of 3,152 slots (2.95%) and missed 49 of 1,365 transactions; confirmed skipped 8 slots (0.25%) and 2 of 2,415 transactions; processed skipped none. Across both runs, finalized missed 61 of 1,803.

programSubscribe behaves the same way. A third connection subscribed to the Lookup Table program’s accounts at all three levels for 30 minutes. In the stretch all three covered, the confirmed stream delivered 37 table updates (3.0% of 1,253) that the finalized stream never sent, and the finalized stream delivered none that confirmed did not.

Each update reads exactly one slot

Agave’s PubSub service is driven by commitment updates. When the processed or finalized slot moves, the notifier thread visits every subscription at that level. For each one, it loads one bank, the one at the new commitment slot, and runs a query against it. What the query returns depends on the subscription type, and for two of them the answer is “only what happened in this slot”.

// rpc/src/rpc_subscriptions.rs — one bank: the new commitment slot's
let bank = bank_forks.read().unwrap().get(slot);

// programSubscribe's query against that bank
bank.get_program_accounts_modified_since_parent(&params.pubkey)

// runtime/src/bank.rs — which scans only that slot's own storage
self.rc.accounts.load_by_program_slot(self.slot(), self.bank_id(), Some(program_id))

// logsSubscribe reads the same bank's log collector: its own transactions only
bank.get_transaction_logs(pubkey)

programSubscribe sees the accounts written in the slot it was handed, and logsSubscribe sees the transactions executed in it. Nothing records which slots a subscription has already been shown: the “last notified slot” passes through these two filters untouched. So if the finalized slot moves from 5 to 7, slot 7 is scanned and slot 6 is not, by this update or by any later one.

The one-slot read for program subscriptions dates to March 2019. It is not a regression. The change is in how the commitment slot moves.

Three ways the commitment slot jumps

The issue names three paths, and each one is in the source.

• Coalescing. The commitment service drains its queue with try_iter().last(): if two updates are waiting when it wakes, it keeps the newer one and the older slot is never announced. Both the TowerBFT and the Alpenglow inputs are read this way.
• A skip vote the cluster overrules (processed). Under Alpenglow, processed follows the node’s own notarize votes. If the node votes to skip a slot the cluster then notarizes, it never notarizes that slot, and a parent check stops it notarizing the rest of that leader window. Processed then jumps to the next window.
• One root for many slots (finalized). Votor roots the highest finalized block it can root, and a single Rooted message moves the node’s finalized and root slots to that slot at once. It is published only after the bank tree has been rooted and pruned, so the banks in between are gone by the time the notifier runs.

The third path is the one our data shows. Alpenglow has decided finality on mainnet since 21 October 2026, and under it a block is rootable only once the node has voted on it and replayed it. A finalized block that is not rootable yet when a later one is gets folded into the later root. The processed paths exist in the code, but they did not fire on this node during our half hour.

Which subscriptions are exposed, and why confirmed is not a cure

The one-slot read is specific to two subscription types. The others either read through ancestors or walk back over the slots they missed:

Confirmed takes a different route, which is why it did twelve times better. Under Alpenglow there is no optimistic confirmation in the TowerBFT sense, so votor sends the RPC layer an “optimistically confirmed” notification for each new root, with a comment explaining it is critical for compatibility with the RPC API. The tracker that receives it walks every ancestor above the last confirmed slot and issues one update per slot. On the same node, confirmed and finalized notifications for a slot arrived a median of 9 milliseconds apart: on an Alpenglow Agave node, confirmed is already root-timed.

It still skipped 0.25% of slots, always the same window position as the finalized gaps. Our reading is a race with pruning. The confirmed walk follows parent links that squashing a root clears, and each slot it queues is then loaded from a bank tree that rooting prunes. Both happen at the moment a slot becomes final. We have not proven which of the two loses in our samples.

The fix on the table, and what it changes for clients

Pull request 16689 changes the read from “one slot” to “every slot since the last one this subscription was shown”:

• Snapshots before rooting. Each bank’s account writes and transaction logs are copied into a retained cache before rooting runs, so pruning and accounts cleaning cannot erase an update that has not been delivered yet. Rooted banks are captured from the same function that rooted them.
• Delivery in order, once. On each commitment update, the notifier walks from the last bank this subscription was shown to the new slot and delivers each in order, deduplicating frozen banks by bank ID. The confirmed path gets the same retained chain.
• A visible change in context.slot. Today a notification’s context.slot is the commitment slot that triggered it. With the fix it is the slot where the change happened, so a jump from 5 to 7 arrives as an update for 6 and then one for 7. Code that treats context.slot as “the current finalized slot” will see it lag behind.
• Bounded, and loud when exceeded. The cache is capped at the PubSub queue budget (--rpc-pubsub-queue-capacity-bytes, 256 MiB by default) and at 512 banks, under two minutes at the 4.7 slots a second we measured. A subscription that needs a bank already evicted is disconnected through the same path a lagging client takes today, instead of being skipped silently.

As of 5 November the pull request is open with no reviews, and no release contains it. The public endpoint we measured runs 4.4.0, which has the one-slot read, the coalescing and the root-timed confirmed path exactly as described above.

Catching the gap in your own pipeline

No subscription setting makes a PubSub stream complete today, so the dependable move is to diff it against the ledger. For a logs stream, getSignaturesForAddress at finalized is the list of everything that should have arrived. This is the loop we ran:

// ws: an open WebSocket to your RPC node · rpc(): a plain JSON-RPC POST helper
const PROGRAM = 'AddressLookupTab1e1111111111111111111111111';
const GRACE_SLOTS = 150; // ~30 s for the stream to deliver before we call it missed

const seen = new Set(); // signatures the stream delivered
let fromSlot = 0;       // first slot the stream was live for
let cursor;             // newest signature already reconciled

ws.send(JSON.stringify({
  jsonrpc: '2.0', id: 1, method: 'logsSubscribe',
  params: [{ mentions: [PROGRAM] }, { commitment: 'finalized' }],
}));
ws.onmessage = ({ data }) => {
  const msg = JSON.parse(data);
  if (msg.method !== 'logsNotification') return;
  const { context, value } = msg.params.result;
  fromSlot ||= context.slot;
  seen.add(value.signature);
};

async function reconcile() {
  if (!fromSlot) return;
  const sigs = await rpc('getSignaturesForAddress', [
    PROGRAM,
    { commitment: 'finalized', limit: 1000, ...(cursor && { until: cursor }) },
  ]);
  if (!sigs.length) return;
  // Only judge signatures the stream has had time to deliver.
  const settled = sigs.filter((s) => s.slot <= sigs[0].slot - GRACE_SLOTS);
  if (!settled.length) return;
  cursor = settled[0].signature;
  for (const s of settled) {
    if (s.slot > fromSlot && !seen.has(s.signature)) {
      // fetch with getTransaction and apply it
      console.log('missed', s.signature, 'in slot', s.slot);
    }
  }
}
setInterval(reconcile, 30_000);

Against mainnet from 07:57 UTC for three and a half minutes, it checked 239 finalized signatures and flagged 5 that the stream had skipped, every one of them in the third slot of a leader window. The grace period matters: the HTTP call and the WebSocket can land on different nodes, and without it a signature the stream is about to deliver looks missed.

• Subscribe at confirmed to shrink the backlog. On an Alpenglow Agave node it arrived with finalized and skipped a twelfth as often. Gate on rootSubscribe if you need finality to be explicit rather than a property of today’s implementation: the source says the confirmed notification may be deprecated. Make that gate idempotent. On the same endpoint, rootSubscribe sent each root twice, once on its own and again just before the next one (127 notifications for 66 roots in 15 seconds).
• Treat every disconnect as a gap. Reconcile from the last slot you finished. After the fix, falling behind the retention window is exactly how a gap will announce itself.
• For account state, re-read what the missed transactions touched. getMultipleAccounts with minContextSlot, or accountSubscribe for a known set, which reads through ancestors.
• Know which path you are on. Geyser plugins are fed from replay, not from this notifier, so a Geyser-based stream does not go through the one-slot read. A provider’s own WebSocket product may or may not be Agave’s PubSub underneath; the reconciliation loop works either way.

What this does not show

One endpoint, one morning, half an hour per run, and three programs. The rate will move with load and with the node. The logs numbers are checked against the ledger; the programSubscribe number is a comparison with the confirmed stream on the same connection, because a lookup table can change without an account notification under that owner (a closed table, for one).

We did not build or run the patched node, so section 05 describes the fix from its code and tests, not from measurement. Processed showed no gaps here, although the paths that would skip it are real. TowerBFT is not measured at all: mainnet no longer runs it. The one-slot read is as old under TowerBFT as under Alpenglow, so the exposure is not new, but we cannot say how often it fired before 21 October. The leader-window clustering is unexplained.

Missed WebSocket Updates: Questions People Actually Ask

The one-line version

On Agave today, a finalized programSubscribe or logsSubscribe stream is a sample of the finalized slots, not the set. On mainnet it is about 97% of them.

Subscribe at confirmed, reconcile against the ledger, and treat a disconnect as a gap. When the fix ships, a gap will close your socket instead of passing unnoticed.

The commitment levels themselves are explained in how many confirmations a deposit needs, and the certificates behind them in Alpenglow finality in numbers. A pipeline built on these streams is the subject of ingesting the Solana firehose with Rust, and the errors a reconnect can return are in the RPC error codes.

Sources: Agave master at commit 32f5b38fd6, rpc_subscriptions.rs, bank.rs, commitment_service.rs, votor’s event_handler.rs and optimistically_confirmed_bank_tracker.rs; the same paths checked in the v4.4.0 release; issue 16654 and pull request 16689 as of commit cfa411b669. Measurements: our own WebSocket and JSON-RPC calls to the public mainnet endpoint on 5 November 2026, 05:34 to 06:09 and 07:57 to 08:01 UTC.