Zero-Data-Loss Analytics: Scaling NATS and ClickHouse for Web3

In Part 1, we established our cloud economics and architecture blueprint. In Part 2, we built a resilient, zero-copy Rust ingestion service that decodes raw transactions and pumps structured swap events into NATS.

Now, we have to solve the final piece of the puzzle: Storage and Real-Time Delivery.

Piping millions of events into a traditional relational database like PostgreSQL will instantly lock tables, bloat disk space, and crash under heavy read/write concurrency. To analyze billions of historical swaps while simultaneously pushing live market feeds to end users with sub-second latency, we need a two-tier consumption layer: ClickHouse for analytical storage, and Axum WebSockets for live event fanout.

Consuming NATS JetStream & ClickHouse Schema Design

When consuming high-velocity data streams, database maintenance, schema migrations, or unexpected container restarts are inevitable. If your consumer reads directly from an in-memory queue, restarting your database means dropping millions of incoming swap events.

NATS JetStream: Persistence as a Safety Net

To guarantee zero data loss, we configure NATS with JetStream persistence. JetStream writes incoming messages to an encrypted, distributed disk stream. If our database goes offline for 15 minutes, NATS retains the stream and seamlessly replays every missing message the moment the database re-establishes its connection.

The ClickHouse Schema

For storage, ClickHouse is the clear winner. As a column-oriented DBMS, ClickHouse compresses data by up to 80% and can execute aggregate queries across billions of rows in milliseconds — the exact performance profile a swap analytics dashboard demands.

-- Production ClickHouse Schema for Solana Swaps
CREATE TABLE IF NOT EXISTS solana_swaps
(
    signature      String,
    slot           UInt64,
    block_time     DateTime CODEC(DoubleDelta, ZSTD),
    program_id     LowCardinality(String),
    signer         String,
    token_in       String,
    token_out      String,
    amount_in      UInt64,
    amount_out     UInt64,
    created_at     DateTime DEFAULT now()
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(block_time)
ORDER BY (program_id, token_in, token_out, block_time)
SETTINGS index_granularity = 8192;

By using the MergeTree engine, partitioning by month, and selecting an ordering key optimised for token pair filtering — (program_id, token_in, token_out, block_time) — ClickHouse skips 99% of the disk data during typical analytical queries.

High-Throughput Batch Ingestion

The single most common mistake engineers make with ClickHouse is executing single-row INSERT statements for every incoming event. ClickHouse is designed for bulk writes; single-row inserts create millions of tiny data parts on disk, triggering severe write amplification and bringing the cluster to its knees.

Dual-Threshold Flushing with tokio::select!

Our Rust consumer service pools incoming NATS messages into memory buffers and executes bulk inserts using two thresholds:

tokio::select! races both futures concurrently. Whichever fires first triggers the flush. This means during peak volume the pipeline flushes at 10k-row boundaries (maximising throughput), and during quiet periods it flushes every second (minimising latency for dashboards).

use clickhouse::Client;

async fn run_clickhouse_consumer(
    mut nats_sub: JetStreamSubscription,
    client:   Client,
) {
    let mut buffer   = Vec::with_capacity(10_000);
    let mut interval = tokio::time::interval(Duration::from_secs(1));

    loop {
        tokio::select! {
            // Priority 1: Fill buffer from the NATS stream
            Some(msg) = nats_sub.next() => {
                if let Ok(swap) = bincode::deserialize::<DecodedSwap>(&msg.payload) {
                    buffer.push(swap);
                    msg.ack().await.ok();
                }

                if buffer.len() >= 10_000 {
                    flush_to_clickhouse(&client, mut buffer).await;
                }
            }
            // Priority 2: Flush periodically during low volume
            _ = interval.tick() => {
                if !buffer.is_empty() {
                    flush_to_clickhouse(&client, mut buffer).await;
                }
            }
        }
    }
}

By decoupling row-by-row arrivals into structured bulk inserts, we maintain near-zero CPU usage on the database cluster while keeping end-to-end swap visibility latency under one second at all volume levels.

The Real-Time Interface: Decoupled WebSocket Fanout

Frontend applications — live trading dashboards, arbitrage bots, portfolio trackers — need to see swaps the exact millisecond they happen. Polling ClickHouse repeatedly to check for new rows creates unnecessary read pressure on your analytical storage and introduces artificial latency.

Instead, we build a dedicated, lightweight WebSocket service using Axum that listens directly to the NATS subject — completely independent of the ClickHouse consumer.

Because NATS supports pub/sub topics out of the box, our Axum WebSocket service acts as a pure fanout router:

• When a client connects to wss://api.xroot.dev/ws/swaps?pair=SOL-USDC, Axum subscribes them to the corresponding NATS subject (e.g. solana.swaps.raydium).
• When the Rust Ingester publishes a decoded swap to NATS, NATS broadcasts it simultaneously to all active WebSocket clients and the ClickHouse batch consumer — no polling, no secondary query.
• Thousands of concurrent frontend WebSocket connections consume live data feeds with zero impact on database performance, since ClickHouse is never touched by the fanout path.

System Summary

We have successfully engineered an end-to-end, high-throughput Web3 data pipeline from scratch. Here is the complete picture:

• Filtered RPC Ingestion: Optimised Helius WebSocket subscriptions cap costs at under 8M credits/day on a $999/month budget — no wasted credit burn.
• Zero-Copy Rust Engine: carbon + bounded tokio channels on a cost-effective 4-CPU pod handle thousands of events per second with no GC pauses and no memory leaks.
• Guaranteed Delivery: NATS JetStream persists the stream to disk, ensuring zero data loss during downstream outages and enabling seamless replay on reconnect.
• Analytical Storage & Fanout: A partitioned ClickHouse MergeTree table delivers sub-second analytical queries over billions of rows. A decoupled Axum WebSocket service fans out live swap events to frontend clients with zero database pressure.