ds4

Product Introduction

1. Definition: ds4 is a specialized, native inference engine designed for local deployment of the DeepSeek V4 Flash and PRO large language models (LLMs). It is a high-performance, self-contained software application that provides a complete inference stack, including model loading, prompt rendering, tool calling, and a built-in HTTP server.
2. Core Value Proposition: ds4 solves the critical problem of expensive cloud API dependency and high infrastructure costs for running state-of-the-art LLMs by enabling efficient, high-speed local inference on consumer and professional hardware. Its primary value is delivering capable AI performance on local machines like high-end MacBooks, multi-GPU workstations, and AMD Strix Halo systems.

Main Features

1. Multi-Platform GPU Acceleration: ds4 is optimized for three primary compute backends. On Apple Silicon Macs, it uses the Metal Performance Shaders (MPS) framework for maximum performance. For NVIDIA GPUs, it supports CUDA with advanced features like multi-GPU tensor parallelism and pipeline parallelism. On AMD systems like the Framework Desktop with Strix Halo, it utilizes the ROCm platform. This cross-platform support ensures users can leverage their specific hardware's full potential.
2. Aggressive, High-Quality Model Quantization: The engine supports specialized GGUF model files that use an asymmetric quantization strategy. The vast majority of the model's parameters—specifically the routed Mixture-of-Experts (MoE) layers—are compressed to 2-bit (IQ2_XXS) or 4-bit (Q4_K) formats. Critical components like shared experts and routing logic remain at higher precision (Q8/F32) to preserve model quality and reasoning capability, resulting in models that are dramatically smaller yet remain highly capable for tasks like coding and tool use.
3. SSD Streaming for Large Models: For systems with insufficient RAM to hold a full model, ds4 implements an intelligent SSD streaming mode. It keeps non-routed model weights resident in memory while dynamically caching frequently accessed routed experts from the SSD. This feature, combined with modern fast NVMe storage, makes it practical to run multi-hundred-billion parameter models like DeepSeek V4 Pro on hardware like 128GB MacBooks, trading some generation speed for vastly increased model capacity.
4. Integrated Distributed Inference: ds4 natively supports model parallelism across multiple machines. Tensor Parallelism splits the computational graph across GPUs in a cluster (e.g., two MacBooks) for faster inference. Pipeline Parallelism aggregates the memory of multiple systems to run models larger than any single machine's RAM, such as the full DeepSeek V4 Pro in 4-bit quantization across several nodes.
5. Production-Ready Server with Micro-Batching: The included ds4-server binary supports micro-batching of decoding and generation requests. This allows a multi-GPU server (e.g., with 8x NVIDIA L40S cards) to efficiently handle multiple concurrent user sessions, transforming older GPU architectures into viable multi-user LLM serving platforms for enterprise use.

Problems Solved

1. Pain Point: High cost and latency of cloud-based LLM APIs, along with data privacy and control concerns associated with sending prompts to external services.
2. Pain Point: The complexity and overhead of general-purpose inference frameworks (like vLLM) which may not support older GPU architectures or require significant engineering effort to optimize for specific model families.
3. Target Audience: AI Researchers and Hobbyists who want to experiment with cutting-edge MoE models on personal hardware. Software Developers building local AI-powered applications that require privacy, low latency, or offline functionality. Enterprises seeking to deploy a private, cost-controlled LLM inference endpoint on their existing on-premise GPU infrastructure.
4. Use Cases: Running a private coding assistant on a MacBook Pro. Deploying a company-internal chat agent on a cluster of NVIDIA GPUs. Conducting research on long-context model behavior using local SSD-streaming to handle 128K token contexts. Serving multiple departments with a single, efficient inference server using micro-batching.

Unique Advantages

1. Differentiation: Unlike general-purpose inference engines, ds4 is a narrow, specialized engine built exclusively for DeepSeek V4 and GLM 5.2 architectures. This allows for deep, model-specific optimizations in the kernel, graph compilation, and memory layout that general frameworks cannot match, resulting in higher performance per watt and per dollar.
2. Key Innovation: The asymmetric, routed-expert-only quantization methodology is a key innovation. By aggressively quantizing only the sparse, data-specific expert weights (which constitute most of the model's size) while preserving high precision for the shared, reasoning-critical weights, ds4 achieves an unparalleled balance of extreme model compression (e.g., ~30GB for a 600B+ parameter model) and maintained output quality for complex tasks.

Frequently Asked Questions (FAQ)

1. What models are compatible with the ds4 inference engine? ds4 is specifically designed for the DeepSeek V4 Flash, DeepSeek V4 Pro, and GLM 5.2 model families. It requires specially prepared GGUF files that use its supported asymmetric quantization layout (e.g., routed experts in IQ2_XXS/Q2_K). It is not a general GGUF loader and will not work with arbitrary model files from the broader ecosystem.
2. Can I run DeepSeek V4 Pro on a Mac with 128GB of RAM? Yes, using the SSD streaming feature with a 2-bit quantized Pro model. While generation speed will be slower than a fully resident model, it enables interactive use and long-context processing that would otherwise be impossible on that hardware class. The automatic cache budgeting (--ssd-streaming) helps configure this optimally.
3. How does ds4's performance compare to llama.cpp? ds4 is built upon concepts and kernels from the llama.cpp/GGML ecosystem and acknowledges this debt. Its primary advantage is specialization: by focusing only on DeepSeek V4 and GLM 5.2, it can implement faster, model-specific kernels, advanced features like integrated tensor parallelism, and a streamlined server. For these specific models, it often provides higher inference speeds, especially on Metal and in multi-GPU configurations.
4. What is DSpark speculative decoding and how do I use it? DSpark is an auxiliary draft model released by DeepSeek that predicts future tokens to accelerate generation. In ds4, you enable it by downloading the separate DSpark support GGUF and running with --mtp <file.gguf> --dspark. It works by having the draft model propose a sequence of tokens which are then verified in a single pass by the main model, committing only the correct prefix. This can significantly speed up greedy decoding for predictable text like code.
5. Is the code in ds4 written by AI? The development process uses AI assistants (GPT, Claude) extensively for implementation, but the core architecture, design decisions, testing, and debugging are led by humans. The project is transparent about this AI-assisted development methodology. The foundational quantization logic and kernels are adapted from the hand-written llama.cpp codebase under the MIT license.