🛠️Tuning a Solana Node [ENG]

🎯 Finer tuning of a Solana validator

Steps:

1. Choose optimal CPU Pick a modern AMD EPYC (e.g., 9275F) with high base clock and boost. Prioritize single-core perf for PoH. Reference: SolanaHCL Hardware Guide

2. Enable AMD P-State Scaling Driver

   • Use kernel 6.1+ (preferably 6.5 or higher for Zen4/5 support)

   • Recommended mode: amd_pstate=active instead of passive Add to /etc/default/grub:

     GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_pstate=active nohz_full=2,26 isolcpus=2,26 rcu_nocbs=2,26 irqaffinity=0-1,3-25,27-47 idle=poll mitigations=off nvme_core.default_ps_max_latency_us=0 pcie_aspm=off"

     Then:

     sudo update-grub && reboot

3. Set CPU governor to performance

   sudo apt install -y cpufrequtils
   echo 'GOVERNOR="performance"' | sudo tee /etc/default/cpufrequtils
   sudo systemctl disable --now ondemand
   sudo systemctl enable --now cpufrequtils

4. Apply kernel & sysctl tweaks See: Solana Tuning Guide - Sysctl

5. Disk layout

   • Format NVMes as:

     • XFS with noatime for ledger and snapshots

     • ext4 with noatime for accounts

   • Example:

     mkfs.xfs -f -m reflink=1 -L ledger -d su=512k,sw=1 /dev/nvmeXn1
     mkfs.ext4 -F -O ^has_journal -L accounts /dev/nvmeYn1

6. Use multiple NVMes

   • Store ledger, snapshots, and accounts on three separate Gen4+ NVMe devices

   • Split accounts across 3+ paths using --accounts /mnt/a1,/mnt/a2,/mnt/a3

7. Enable RamDisk for Index & Hash cache

   • Use --accounts-index-path /dev/shm/idx and --accounts-hash-cache-path /dev/shm/hash

   • Ensure /dev/shm is large enough (sysctl -w kernel.shmmax if needed)

8. Enable unified scheduler for catch-up

   --block-verification-method unified-scheduler

9. Optimize accounts hash threads

   --accounts-db-hash-threads 2

   If snapshots are disabled:

   --accounts-db-hash-threads 1

10. Snapshot tuning for HA setup

    --snapshot-interval-slots 0

11. Non-voting hot spare setup See Pumpkin Pool's blog post on high-availability validator setups.

12. Disable Ubuntu 24.04's automatic restarts

    sudo systemctl disable systemd-reboot.service
    sudo systemctl mask systemd-reboot.service

13. Check PCIe link speeds

    sudo nvme list
    sudo lspci -vv | grep -i width

14. Pick a data center with excellent peering & low latency

15. Use mods and tweaks at your own risk

---

🔧 Core Isolation for PoH

Why:

Dedicated PoH thread = higher priority, lower latency, less jitter.

How:

1. Know your topology

   lstopo

2. Find core & hyperthread Example for core 2:

   cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list

3. Edit GRUB Example (core 2 + HT core 26):

   GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_pstate=active nohz_full=2,26 isolcpus=domain,managed_irq,2,26 irqaffinity=0-1,3-25,27-47"

   Then:

   sudo update-grub && reboot

4. Pin PoH thread Add to validator args:

   --experimental-poh-pinned-cpu-core 2

---

🧠 solPohTickProd Affinity Script

Because of this bug, core pinning doesn’t always work if cores are isolated. Use this workaround:

Script:

#!/bin/bash
solana_pid=$(pgrep -f "^agave-validator --identity")
if [ -z "$solana_pid" ]; then
    logger "set_affinity: agave_validator_404"
    exit 1
fi

thread_pid=$(ps -T -p $solana_pid -o spid,comm | grep 'solPohTickProd' | awk '{print $1}')
if [ -z "$thread_pid" ]; then
    logger "set_affinity: solPohTickProd_404"
    exit 1
fi

current_affinity=$(taskset -cp $thread_pid 2>&1 | awk '{print $NF}')
if [ "$current_affinity" == "2" ]; then
    logger "set_affinity: solPohTickProd_already_set"
    exit 0
fi

sudo taskset -cp 2 $thread_pid
logger "set_affinity: set_done"

• Wait until the ledger is loaded before executing.

• You can call it manually or via cron, ExecStartPost, or systemd.timer.

Example logs:

pgrep agave-validator
ps -o spid,psr,comm -T -p <pid> | grep 'solPohTickProd'

---

📊 Benchmarking PoH performance

Check in logs:

grep "PoH speed check" /path/to/solana.log
grep "hashes/sec" /path/to/solana.log

---

🧪 Other user scripts

• Core pinning workaround by RadiantAeon: core-pin.sh

• More examples for affinity and validation tweaks in 1000xsh’s repo