🛠️Tuning a Solana Node [ENG]

🎯 Finer tuning of a Solana validator

Steps:

  1. Choose optimal CPU Pick a modern AMD EPYC (e.g., 9275F) with high base clock and boost. Prioritize single-core perf for PoH. Reference: SolanaHCL Hardware Guide

  2. Enable AMD P-State Scaling Driver

    • Use kernel 6.1+ (preferably 6.5 or higher for Zen4/5 support)

    • Recommended mode: amd_pstate=active instead of passive Add to /etc/default/grub:

      GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_pstate=active nohz_full=2,26 isolcpus=2,26 rcu_nocbs=2,26 irqaffinity=0-1,3-25,27-47 idle=poll mitigations=off nvme_core.default_ps_max_latency_us=0 pcie_aspm=off"

      Then:

      sudo update-grub && reboot

  3. Set CPU governor to performance

    sudo apt install -y cpufrequtils
echo 'GOVERNOR="performance"' | sudo tee /etc/default/cpufrequtils
sudo systemctl disable --now ondemand
sudo systemctl enable --now cpufrequtils

  4. Apply kernel & sysctl tweaks See: Solana Tuning Guide - Sysctl

  5. Disk layout

    • Format NVMes as:

      • XFS with noatime for ledger and snapshots

      • ext4 with noatime for accounts

    • Example:

      mkfs.xfs -f -m reflink=1 -L ledger -d su=512k,sw=1 /dev/nvmeXn1
mkfs.ext4 -F -O ^has_journal -L accounts /dev/nvmeYn1

  6. Use multiple NVMes

    • Store ledger, snapshots, and accounts on three separate Gen4+ NVMe devices

    • Split accounts across 3+ paths using --accounts /mnt/a1,/mnt/a2,/mnt/a3

  7. Enable RamDisk for Index & Hash cache

    • Use --accounts-index-path /dev/shm/idx and --accounts-hash-cache-path /dev/shm/hash

    • Ensure /dev/shm is large enough (sysctl -w kernel.shmmax if needed)

  8. Enable unified scheduler for catch-up

    --block-verification-method unified-scheduler

  9. Optimize accounts hash threads

    --accounts-db-hash-threads 2

    If snapshots are disabled:

    --accounts-db-hash-threads 1

  10. Snapshot tuning for HA setup

    --snapshot-interval-slots 0

  11. Non-voting hot spare setup See Pumpkin Pool's blog post on high-availability validator setups.

  12. Disable Ubuntu 24.04's automatic restarts

    sudo systemctl disable systemd-reboot.service
sudo systemctl mask systemd-reboot.service

  13. Check PCIe link speeds

    sudo nvme list
sudo lspci -vv | grep -i width

  14. Pick a data center with excellent peering & low latency

  15. Use mods and tweaks at your own risk

🔧 Core Isolation for PoH

Why:

Dedicated PoH thread = higher priority, lower latency, less jitter.

How:

  1. Know your topology

    lstopo

  2. Find core & hyperthread Example for core 2:

    cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list

  3. Edit GRUB Example (core 2 + HT core 26):

    GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_pstate=active nohz_full=2,26 isolcpus=domain,managed_irq,2,26 irqaffinity=0-1,3-25,27-47"

    Then:

    sudo update-grub && reboot

  4. Pin PoH thread Add to validator args:

    --experimental-poh-pinned-cpu-core 2

🧠 solPohTickProd Affinity Script

Because of this bug, core pinning doesn’t always work if cores are isolated. Use this workaround:

Script:

#!/bin/bash
solana_pid=$(pgrep -f "^agave-validator --identity")
if [ -z "$solana_pid" ]; then
    logger "set_affinity: agave_validator_404"
    exit 1
fi

thread_pid=$(ps -T -p $solana_pid -o spid,comm | grep 'solPohTickProd' | awk '{print $1}')
if [ -z "$thread_pid" ]; then
    logger "set_affinity: solPohTickProd_404"
    exit 1
fi

current_affinity=$(taskset -cp $thread_pid 2>&1 | awk '{print $NF}')
if [ "$current_affinity" == "2" ]; then
    logger "set_affinity: solPohTickProd_already_set"
    exit 0
fi

sudo taskset -cp 2 $thread_pid
logger "set_affinity: set_done"

  • Wait until the ledger is loaded before executing.

  • You can call it manually or via cron, ExecStartPost, or systemd.timer.

Example logs:

pgrep agave-validator
ps -o spid,psr,comm -T -p <pid> | grep 'solPohTickProd'

📊 Benchmarking PoH performance

Check in logs:

grep "PoH speed check" /path/to/solana.log
grep "hashes/sec" /path/to/solana.log

🧪 Other user scripts

PreviousRunning Firedancer (Solana Testnet)

Last updated