Free LLM API

Product Introduction

1. Definition: Free LLM API is an open-source, self-hosted OpenAI-compatible proxy and intelligent router designed to aggregate free-tier API keys from over 14 leading AI providers. Technically categorized as an LLM Gateway (or AI Proxy), it provides a unified /v1/chat/completions endpoint that abstracts the complexity of multiple provider SDKs and rate limits into a single, cohesive interface.

2. Core Value Proposition: The project exists to solve the fragmentation of the "free-tier" AI landscape. While individual providers like Google, Groq, and Mistral offer limited free usage, Free LLM API stacks these tiers to provide an estimated 1.3 billion tokens of monthly inference capacity. By implementing automatic failover, per-key rate tracking, and protocol translation, it enables developers to perform extensive personal experimentation and local LLM development without incurring API costs or managing 14 different authentication schemes.

Main Features

1. Intelligent Multi-Provider Routing and Failover: The system utilizes a sophisticated router (built in TypeScript) that selects the highest-priority available model based on real-time health status and rate limits. If a provider returns a 429 (Rate Limit Reached), 5xx (Server Error), or a timeout, the proxy automatically initiates a failover sequence. It places the failing key on a temporary cooldown and retries the request against the next available provider in the user-defined fallback chain, supporting up to 20 consecutive retry attempts to ensure request fulfillment.

2. Unified OpenAI-Compatible Tool Calling: Free LLM API supports complex tool calling (function calling) across all integrated providers. It translates standard OpenAI-style tools and tool_choice JSON objects into the native formats required by upstream providers, such as Google Gemini’s functionDeclarations. This allows for multi-step agentic workflows where assistant tool_calls and tool-role follow-up messages round-trip seamlessly, regardless of which provider is currently serving the request.

3. Secure Key Management and Admin Dashboard: Security is handled via AES-256-GCM envelope encryption for all API keys stored in the local SQLite database. Decryption occurs only in-memory during request execution. The product includes a built-in administrative interface built with React, Vite, and shadcn/ui, providing a centralized "Playground" for prompt testing, real-time analytics for monitoring token throughput (RPM/TPM), and a management console for reordering the provider priority list.

Problems Solved

1. Fragmented API Management: Developers often struggle to manage different SDKs, authentication headers, and response formats for various AI labs. Free LLM API eliminates this "integration debt" by providing a single "base_url" and "api_key" that works with any OpenAI-compatible library, including LangChain, LlamaIndex, and Continue.

2. Restrictive Free-Tier Rate Limits: Individual free tiers (e.g., Groq's RPM or Google’s daily caps) are often too restrictive for complex testing. This product solves the "429 problem" by pooling resources; when one provider's limit is hit, the traffic is transparently routed to another, effectively creating a high-availability cluster from free resources.

3. Target Audience: The primary users are Indie Developers, AI Researchers, and Hobbyists building local-first applications. It is specifically optimized for users running low-power hardware, such as a Raspberry Pi 4, who require a cost-effective way to power personal assistants, coding agents, or home automation scripts.

4. Use Cases: Ideal scenarios include local development of RAG (Retrieval-Augmented Generation) systems, powering AI-enabled IDE extensions (like VS Code "Continue") with high-intelligence models like Llama 3.3 70B or Gemini 2.5 Pro, and running long-running data processing scripts that would otherwise be cost-prohibitive on paid API tiers.

Unique Advantages

1. Massive Aggregate Token Capacity: By stacking 14 providers (including NVIDIA NIM, SambaNova, Cerebras, and GitHub Models), the proxy grants access to an unprecedented volume of free inference—roughly 1.3 billion tokens per month—which is far beyond what any single provider offers.

2. Provider-Agnostic "Auto" Modeling: Users can specify "model": "auto" in their request payloads. The proxy then dynamically determines the most "intelligent" healthy model available at that moment, optimizing for reasoning capability during the start of the day and falling back to faster, high-limit models as daily quotas are consumed.

3. Local-First Privacy and Performance: Unlike cloud-based aggregators, Free LLM API is self-hosted. It features a lightweight footprint (~40MB RAM usage at idle) and ensures that sensitive API keys never leave the user's local environment except to communicate directly with the upstream AI provider.

Frequently Asked Questions (FAQ)

1. Is Free LLM API compatible with the official OpenAI Python and Node.js SDKs? Yes. Because the project replicates the OpenAI v1 REST API structure, you can simply change the "base_url" in your OpenAI client configuration to point to your local Free LLM API instance. It supports both standard JSON responses and Server-Sent Events (SSE) for streaming completions.

2. Which AI models can I access through this proxy? The proxy provides access to a wide range of state-of-the-art open and closed models available via free tiers, including Google's Gemini 1.5 Pro and Flash, Meta's Llama 3.3 70B, Alibaba's Qwen series, Mistral's La Plateforme models, and Microsoft's Phi series via GitHub Models.

3. How does the system handle security for my API keys? All provider keys are encrypted using industry-standard AES-256-GCM before being stored in an on-disk SQLite database. The proxy itself requires a unique "freellmapi-..." bearer token for authentication, ensuring that only authorized local clients can access your aggregated AI resources.

4. Can I run Free LLM API on low-power hardware like a Raspberry Pi? Absolutely. The application is optimized for efficiency, running on Node.js 20+. It has a very low memory footprint (approximately 40MB RSS at idle) and is designed to run under process managers like PM2 behind an Nginx reverse proxy, making it perfect for a home server or "always-on" AI gateway.

5. What happens when all free-tier providers hit their rate limits? The system tracks Rate Limits Per Day (RPD) and Tokens Per Day (TPD) in an internal ledger. If all providers in your fallback chain are exhausted or rate-limited, the API will return a 429 error to the client. Limits typically reset at UTC midnight, at which point the router automatically marks the keys as healthy again.