Thunder

Product Introduction

1. Definition: Thunder (distributed via Docker image sbansal793/thunderstrike) is a blazing-fast, persistent, and scalable key-value database server built in Rust. It functions as a high-throughput, low-latency data storage engine optimized for modern, data-intensive applications utilizing NVMe storage and advanced Linux kernel subsystems.
2. Core Value Proposition: Thunder exists to eliminate I/O bottlenecks in high-performance computing environments by combining the rapid speed of modern NVMe drives with intelligent memory management. Key primary keywords include "persistent key-value database," "Rust-built database," "shard-per-core architecture," "TLS 1.3 encrypted database," and "io_uring storage engine."

Main Features

1. Shard-Per-Core Architecture: How it works: Thunder utilizes a fixed internal layout of 1,024 logical shards distributed dynamically across derived reactor and helper pairs mapped to physical CPU cores. This architecture scales linearly with hardware capabilities while minimizing CPU cache contention and thread synchronization overhead.
2. Asynchronous I/O via io_uring: How it works: Both disk and network operations bypass traditional kernel context-switching bottlenecks by leveraging the Linux io_uring subsystem. This maximizes throughput for values ranging from a few bytes to 512 MiB using locked-memory arena allocations.
3. End-to-End Security and Encryption: How it works: Security is enforced natively through mandatory TLS 1.3 transport encryption and mandatory encryption-at-rest for every stored value. The database requires a secure user-provided keystore passphrase and mutual certificate provisioning before initializing reactor loops.

Problems Solved

1. Pain Point: Traditional in-memory databases suffer from massive memory overhead, volatile storage risks, or thread contention under heavy multi-core workloads, while legacy disk-based databases fail to saturate the raw IOPS capacity of modern NVMe hardware.
2. Target Audience: Backend Engineers, Systems Architects, Rust Developers, Infrastructure DevOps Engineers, and Enterprise Data Architects managing high-throughput distributed systems.
3. Use Cases: Real-time caching layers, financial transaction ledger states, high-frequency telemetry ingestion, session stores, and large-scale metadata management requiring strict data durability and sub-millisecond latencies.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):
   • Strengths: Exceptional throughput and low latency via native io_uring and Rust memory safety; strict security defaults (mandatory TLS 1.3 and encryption-at-rest); predictable scaling via physical-core budgeting.
   • Trade-offs/Limitations: Requires a modern Linux kernel (6.8 or newer) on the Docker host; proprietary licensing terms (free to run, but not open-source); strict hardware constraints requiring manual CPU topology configuration and memory lock limits (memlock: -1).
2. Key Alternatives & Differentiation:
   • Compared to Redis: Thunder provides native persistence with encrypted-at-rest NVMe backing and shard-per-core scaling, avoiding Redis's single-threaded event loop limitations or complex cluster-sharding overhead.
   • Compared to RocksDB-backed engines: Thunder eliminates C++ memory management pitfalls by being built natively in Rust and using direct io_uring kernel interfaces instead of standard POSIX file system calls.
   • Compared to enterprise NoSQL databases (e.g., MongoDB, Cassandra): Thunder specializes purely as a low-level, hyper-optimized key-value server rather than a heavy document store, yielding drastically lower memory footprints and predictable tail latencies.

Frequently Asked Questions (FAQ)

1. What are the host kernel and hardware requirements to run Thunder? Thunder requires a Linux host running kernel version 6.8 or newer, an even physical-core budget (at least 2 cores), and sufficient memlock privileges (memlock: -1) to anchor its memory arenas. On macOS or Windows, Docker Desktop must be configured with a compatible Linux VM kernel and sufficient hardware resources.
2. How does client routing work across the database shards? Clients connect to a single bootstrap address provided by the application. The client fetches the DISCOVER table containing the 1,024 logical shards and their derived reactor ports. It then computes CRC32C(key) % 1024 locally to route requests directly to the appropriate reactor port over TLS 1.3 without centralized proxy bottlenecks.
3. Is Thunder open-source and free for commercial production use? Thunder is proprietary software owned by SHORTSQUEEZE PRIVATE LIMITED. It is free to download, deploy, and run in production environments under its specific licensing terms, but its source code is proprietary rather than open-source.
4. How do I handle crash recovery and data durability? Thunder utilizes a built-in write-ahead journal (WAL) that makes every write durable before acknowledging the transaction to the client. Crash recovery automatically replays the journal on startup, while orderly shutdowns via SIGTERM drain in-flight requests and checkpoint the index.