Soup CLI

Product Introduction

1. Definition: Soup CLI is an open-source, Apache-2.0 licensed command-line interface (CLI) tool for the full-stack fine-tuning and alignment of large language models (LLMs). It is a comprehensive AI workbench that automates the post-training pipeline from data curation to model deployment.
2. Core Value Proposition: It enables developers and researchers to efficiently fine-tune and align state-of-the-art LLMs (like Llama, Qwen, Mistral) on consumer-grade hardware, specifically by overcoming VRAM limitations through its proprietary layer streaming technology. Its primary value is democratizing advanced LLM training by making it possible to run an 8B parameter model on a 4 GB laptop GPU.

Main Features

1. Layer Streaming (Exact Layer Streaming): This is the flagship technology. Instead of loading the entire frozen base model into GPU VRAM, Soup CLI keeps it in system RAM (or NVMe storage) and streams individual decoder layers into a small, reusable VRAM buffer pool on-demand. Combined with 4-bit NF4 quantization, this reduces peak VRAM consumption to the size of approximately one model layer. This allows for Llama-3.1-8B fine-tuning on a 4 GB GPU.
2. Automated Configuration & Training Pipeline: The tool features an intelligent autopilot that analyzes your dataset and automatically recommends the optimal training method (SFT, DPO, ORPO, SimPO, KTO, GRPO), writes the soup.yaml configuration file (including learning rate, epochs, and quantization), and performs pre-flight VRAM checks. It includes a data doctor for linting datasets and catching common template errors.
3. Comprehensive Model Lifecycle Management: Soup CLI extends beyond training. The soup ship command provides a definitive SHIP/DON'T SHIP verdict by evaluating models across seven bundled offline evaluation suites. It supports LoRA task arithmetic (merging, diffing, bisecting adapters), model pruning (soup shrink), and exports to formats like GGUF and ONNX. It also includes tools for security and compliance, such as generating regulation-ready configs (HIPAA, EU AI Act) and software bills of materials (ML-BOMs).

Problems Solved

1. Pain Point: The high cost and hardware barrier to entry for LLM fine-tuning and alignment. Traditional tools require high-VRAM GPUs, making experimentation and development expensive and inaccessible for individuals or small teams.
2. Target Audience: Independent AI researchers, startup ML engineers, students, and developers who need to customize LLMs but are constrained by hardware (e.g., laptop GPUs, single consumer cards). It also targets production engineers needing robust pipelines for evaluation, deployment, and compliance.
3. Use Cases: Supervised Fine-Tuning (SFT) of a coding assistant on a local dataset; Direct Preference Optimization (DPO) for aligning a chatbot with human preferences on a single GPU; quickly testing and iterating on model adapters without cloud costs; ensuring model safety and performance before deployment with automated evaluation suites.

Unique Advantages

1. Differentiation: Unlike other fine-tuning frameworks (LLaMA-Factory, Axolotl, Unsloth) that focus primarily on the training step, Soup CLI provides an integrated, opinionated pipeline from data to deployment. It uniquely solves the hardware constraint problem via layer streaming, a feature not available in other mainstream tools. Its soup migrate command allows seamless switching from these competitors.
2. Key Innovation: The bit-exact layer streaming protocol for LoRA training. The technical breakthrough is not just streaming layers, but doing so with guaranteed numerical correctness verified against a resident model run. Furthermore, its implementation of streaming for preference losses (DPO, ORPO, etc.) is unique—it derives the reference model from the same streamed base with adapters disabled, eliminating the need for a second model copy and preserving the memory savings.

Frequently Asked Questions (FAQ)

1. How does Soup CLI enable fine-tuning Llama-3.1-8B on a 4GB GPU? It uses a technique called exact layer streaming, where the frozen base model weights are kept in CPU RAM or on an NVMe drive and are dynamically loaded into the GPU's VRAM one decoder layer at a time during the forward and backward passes. This, combined with 4-bit quantization, drastically reduces the peak VRAM required, bounding it by the size of a single layer rather than the entire model.
2. Can I use Soup CLI for DPO and other alignment methods on a small GPU? Yes, starting with v0.72.4, Soup CLI supports Direct Preference Optimization (DPO), ORPO, SimPO, and KTO over the same layer-streamed base. It cleverly uses the same streamed model weights for both the policy and reference models in DPO by simply disabling the LoRA adapters for the reference pass, adding no extra weight memory cost.
3. Is Soup CLI free to use and open source? Yes, Soup CLI is completely free and open-source, released under the permissive Apache-2.0 license. There is no vendor lock-in, and it is designed to work offline on your own hardware.
4. How does Soup CLI compare to LLaMA-Factory or Axolotl? While LLaMA-Factory and Axolotl are excellent training libraries, Soup CLI offers a more automated, full-stack pipeline including data validation, automated config generation, model evaluation/gating (soup ship), and advanced features like layer streaming for low-VRAM training. It also provides one-command migration from these tools' configs.
5. What models and architectures does Soup CLI support? It supports nine core architecture families including Llama 3/4, Qwen 2.5/3.5, Mistral, Gemma, Phi, DeepSeek, and GPT-OSS, totaling over 200 models from Hugging Face. It also supports vision, audio (Whisper fine-tuning), and MoE models.